Things on a Heap2023-02-09T12:10:42+00:00https://chjdev.comChristian JunkerTask management with my little red book2023-02-09T00:00:00+00:00https://chjdev.com/2023/02/09/journal<p>I’ve been asked what I’m scribbling in my little red book I’m carrying around and thought I’d share. Maybe someone will find it useful.</p>
<p>I really became enamored with the idea of https://bulletjournal.com/ a few years ago. However, I just could never get it to work for my workflow. It just is too powerful for my needs and always leads me back into the todo list death spiral of trying to optimize the tool instead of actually using it.</p>
<p>Two years ago I finally stumbled into a custom simplified version that works for me. Instead of premature optimization I started with “lets just take notes”. I soon needed a way to signify the status of those notes… are they even notes or something else? Do I need to worry about them now or later? etc. By doing this for a while I arrived at the system in the photo.</p>
<p>One key difference to the bullet journal is that I strictly use it for day to day planning only and force myself to rewrite unfinished tasks when I start a new day. If that gets too annoying I ^ move them to an external tool for long term planning (trello, linear, whatever).</p>
<p><img style="width: 75%; max-width: 45rem; margin: auto;" alt="Task management with my little red book" src="/assets/2022/02/little_red_book.png" /></p>
Global Population Density using Rust + Wasm + Webgl2020-03-25T00:00:00+00:00https://chjdev.com/2020/03/25/populationdensity<p>In this little experiment I combined Rust with wasm-bindgen to create a WebGL visualization of global population density.</p>
Docker overrules UFW!2016-06-08T00:00:00+00:00https://chjdev.com/2016/06/08/docker-ufw<p>Docker, by default, directly manipulates <code class="language-plaintext highlighter-rouge">iptables</code> in order to work its network magic. It thus completely circumvents UFW rules! In this post I show strategies to deal with this.</p>
<h1 id="problem">Problem</h1>
<p>While setting up my <a href="https://www.docker.com">Docker</a> server I noticed something
weird. I started up all my containers to see if everything is building and
running correctly, enabled ports 80 and 443 on my firewall, and everything
worked fine. However, after disabling the ports again my server was still
reachable. Even explicitly <code class="language-plaintext highlighter-rouge">REJECT</code>-ing the ports still kept them open! I’m
using <a href="https://wiki.ubuntu.com/UncomplicatedFirewall">UFW</a> as firewall to keep
the base system as simple as possible.</p>
<p>Huh? What the hell is going on? Well, as we can see in the
<a href="https://docs.docker.com/engine/userguide/networking/default_network/binding/">docs</a>,
Docker directly manipulates the systems <code class="language-plaintext highlighter-rouge">iptables</code> to forward trafffic to
containers!</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">sudo </span>iptables <span class="nt">--list</span>
<span class="o">[</span>...]
Chain DOCKER <span class="o">(</span>2 references<span class="o">)</span>
target prot opt <span class="nb">source </span>destination
ACCEPT tcp <span class="nt">--</span> anywhere 172.18.0.6 tcp dpt:https
ACCEPT tcp <span class="nt">--</span> anywhere 172.18.0.6 tcp dpt:http
<span class="o">[</span>...]</code></pre></figure>
<p>Well, I wasn’t aware that this completely circumvents rules set in UFW. UFW in
turn (obviously) doesn’t show you the whole <code class="language-plaintext highlighter-rouge">iptables</code> state but only its
subset of rules.</p>
<h1 id="strategies">Strategies</h1>
<p>If you want to disable this behaviour you can disable it in your daemon.json
(or via flags) and rely on the userland proxy:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">{</span>
<span class="s2">"iptables"</span>: <span class="nb">false</span>,
<span class="s2">"userland-proxy"</span>: <span class="nb">true</span>,
...
<span class="o">}</span></code></pre></figure>
<p>Now you face a different problem though. Since Docker uses bridged networking
by
<a href="https://docs.docker.com/v1.8/articles/networking/#container-networking">default</a>
you now lose the original source IP, which, depending on your use-case, can be
quite important, e.g. to log it as part of the access log of your web servers.</p>
<p>To solve this issue you have two options: switch the networking stack to <code class="language-plaintext highlighter-rouge">host</code>
either globally or the involved containers (mixed linking of containers doesn’t
work), or bite the bullet and deal with the iptables behaviour and disable the
userland proxy. I opted for the latter because of the <a href="https://github.com/docker/docker/issues/6401">security
implications</a> of <code class="language-plaintext highlighter-rouge">host</code>
networking. According to the docs:</p>
<blockquote>
<p>Note that this does not let the container reconfigure the host network stack
— that would require –privileged=true — but it does let container processes
open low-numbered ports like any other root process. It also allows the
container to access local network services like D-bus. This can lead to
processes in the container being able to do unexpected things like restart your
computer.</p>
</blockquote>
<p>Also it is apparently the more
<a href="https://github.com/docker/docker/issues/15086#issuecomment-125678120">canonical</a>
approach, with discussion of
<a href="https://github.com/docker/docker/issues/14856">disabling</a> the userland-proxy
by default.</p>
TLS secured Docker on Ubuntu 16.04 Quickstart2016-06-07T00:00:00+00:00https://chjdev.com/2016/06/07/docker-ubuntu<p>Installing a Vanilla-Docker environment on <a href="http://releases.ubuntu.com/16.04/">Ubuntu
16.04</a> was surprisingly non-straightforward
for me. Turns out it would have been actually quite simple, but the docs and
tutorials seem to be still assuming 14.04/non-systemd and led me in circles a
bit.</p>
<p><a href="/assets/2016/06/docker_ubuntu.png"><img src="/assets/2016/06/docker_ubuntu.png" /></a></p>
<p>The problem is that 16.04 is now based on
<a href="https://wiki.ubuntu.com/SystemdForUpstartUsers">systemd</a> and not upstart
anymore, so config files moved around. Previously you would pass docker daemon
options via <code class="language-plaintext highlighter-rouge">/etc/default/docker</code>, this however doesn’t work
anymore. So here is a quick write up on how to set up <a href="https://docs.docker.com/engine/security/https/">Docker with
TLS</a> on 16.04 with systemd.</p>
<h1 id="setting-up-docker-with-tls-on-systemd">Setting up Docker with TLS on systemd</h1>
<p>First a quick edit of the docker systemd service file is necessary. It is
located at <code class="language-plaintext highlighter-rouge">/lib/systemd/system/docker.service</code>. By default it
will launch the docker daemon with a unix socket bound to <code class="language-plaintext highlighter-rouge">-H fd://</code>,
however in my opinion it’s nicer to have all config options at a central place,
namely the <code class="language-plaintext highlighter-rouge">daemon.json</code> config file that is read by the docker daemon
on start.</p>
<p>So first we’ll remove the <code class="language-plaintext highlighter-rouge">-H fd://</code> flag from the
<code class="language-plaintext highlighter-rouge">ExecStart</code> line of the <code class="language-plaintext highlighter-rouge">docker.service</code> file and leave
it plain without flags:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ cat /lib/systemd/system/docker.service
[...]
ExecStart=/usr/bin/docker daemon
[...]
</code></pre></div></div>
<p>Next we create the <code class="language-plaintext highlighter-rouge">/etc/docker/daemon.json</code> file and add our settings
there:</p>
<figure class="highlight"><pre><code class="language-json" data-lang="json"><span class="p">{</span><span class="w">
</span><span class="nl">"tlsverify"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span><span class="w">
</span><span class="nl">"tlscacert"</span><span class="p">:</span><span class="w"> </span><span class="s2">"/etc/docker/YOUR_DOMAIN/ca.pem"</span><span class="p">,</span><span class="w">
</span><span class="nl">"tlscert"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"/etc/docker/YOUR_DOMAIN/cert.pem"</span><span class="p">,</span><span class="w">
</span><span class="nl">"tlskey"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"/etc/docker/YOUR_DOMAIN/key.pem"</span><span class="p">,</span><span class="w">
</span><span class="nl">"hosts"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">"fd://"</span><span class="p">,</span><span class="w"> </span><span class="s2">"tcp://BIND:2376"</span><span class="p">]</span><span class="w">
</span><span class="p">}</span></code></pre></figure>
<p>As you can see I stored my TLS files in the subdirectory <code class="language-plaintext highlighter-rouge">YOUR_DOMAIN</code> in
the <code class="language-plaintext highlighter-rouge">/etc/docker/</code> directory. The certificates are usually specific
to a domain name so it makes it more obvious. I bound the daemon to two hosts,
first the default unix socket for nicer docker management on the server itself
and the TLS secured TCP address. I use my domain name for <code class="language-plaintext highlighter-rouge">BIND</code> for
the same reason stated previously.</p>
<p>And that should be it, <code class="language-plaintext highlighter-rouge">reload</code> and <code class="language-plaintext highlighter-rouge">restart</code> your docker service and use <code class="language-plaintext highlighter-rouge">enable</code> to
boot it on startup and you should be able to connect:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo systemctl daemon-reload
sudo systemctl restart docker
sudo systemctl enable docker
</code></pre></div></div>
<p>You could add your server user to the <code class="language-plaintext highlighter-rouge">docker</code> group (the unix socket is owned
by this group), however since I manage docker via TCP I opt not to use it and
run docker via sudo if necessary.</p>
<p>Note: don’t forget to distribute certificates to your clients, and set the
<code class="language-plaintext highlighter-rouge">DOCKER_HOST</code> and <code class="language-plaintext highlighter-rouge">DOCKER_TLS_VERIFY</code> environment
variables.</p>
<p>Happy dockering!</p>
<h1 id="creating-the-certificates">Creating the certificates</h1>
<p>This is somewhat beyond the scope of this quick post and I’d like to redirect
you to the <a href="https://docs.docker.com/engine/security/https/">official docs</a> for
that.</p>
<p>A really neat tool I encountered though is this
<a href="https://gist.github.com/sheerun/ccdeff92ea1668f3c75f">GIST</a> by <a href="https://github.com/sheerun">Adam
Stankiewicz</a> that can generate certificates quickly:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ gem install certificate_authority
$ ruby certgen.rb YOUR_DOMAIN
</code></pre></div></div>
<p>This will generate self signed client and server certificates and store them in
your <code class="language-plaintext highlighter-rouge">~/.docker</code> folder. Copy the server certificates from
<code class="language-plaintext highlighter-rouge">~/.docker/YOUR_DOMAIN</code> to your server and set the proper
permission:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ sudo ls -la /etc/docker/YOUR_DOMAIN/
total 20
dr-------- 2 root root 4096 Jun 6 14:29 .
drwx------ 3 root root 4096 Jun 6 15:42 ..
-r-------- 1 root root 1151 Jun 6 14:29 ca.pem
-r-------- 1 root root 1155 Jun 6 14:29 cert.pem
-r-------- 1 root root 1679 Jun 6 14:29 key.pem
</code></pre></div></div>
<h1 id="update">Update</h1>
<p>Please take note that docker (by default) overrules your UFW firewall rules!
I wrote a post on this which you can find <a href="/2016/06/08/docker-ufw/">here</a>.</p>
HyperLogLog - Big Data™ in your Browser2016-01-23T00:00:00+00:00https://chjdev.com/2016/01/23/hyperloglog<p>In this post we’ll take a look at HyperLogLog, a probabilistic data structure
that allows you to estimate the cardinality of huge data sets with minuscule
memory requirements. This page is running an implementation JavaScript and uses
<strong>4kb</strong> to estimate a cardinality of up to <strong>10,000,000 unique visitors</strong> based on
50,000,000 randomly generated visits.</p>
<p>What is cardinality? It is the amount of unique elements in a set. And is
highly useful for a many problems, e.g. finding the top shared urls or
most popular geo locations etc., all based on the amount of how many unique
events occurred there.</p>
<h2 id="probabilistic-data-structures">Probabilistic Data Structures</h2>
<p>A nice list of popular probabilistic data structures with case studies can be
found in <a href="https://highlyscalable.wordpress.com/2012/05/01/probabilistic-structures-web-analytics-data-mining/">Probabilistic Data Structures for Web Analytics and Data
Mining</a>.
They can answer a diverse set of problems:</p>
<ul>
<li>find most popular elements, e.g. a top 100 list</li>
<li>estimate the frequency of these elements</li>
<li>how many elements are in a specified range of data?</li>
<li>does the data set contain a specific element?</li>
<li>and, the topic of this post and basis for many of such questions: estimate the cardinality of elements in a set</li>
</ul>
<p>They work by making smart observations about the internal structure of the data
and its correlation with statistical insights. For example HyperLogLog is based
on the likelihood of encountering a specific pattern of 0s in the datum and
relating that to the size of the cardinality <em>probably</em> necessary to find it.
More on that <a href="#hyperloglog">later</a>.</p>
<p>All of them work slightly differently, but what they share is that they trade
precision for speed and a (significantly) smaller memory footprint. They are
not able to give you the <em>precise</em> answers to the questions above, however,
they give acceptable answers at a <em>fraction</em> of the cost.</p>
<p>What is this <em>fraction</em> I talk about? Well in the case of this blog post
it’s a <strong>20,000 times smaller</strong> memory footprint!</p>
<h3 id="when-and-where">When and Where?</h3>
<p><small>Jump to the <a href="#tldr">TL;DR</a> if you’re already familiar with big data systems.</small></p>
<p>They usually only make sense when you have high requirements on speed and/or
volume. The volume problem can be easily solved via technologies like
MapReduce. However when speed is important, specifically with (near) real time
requirements one needs to make trade-offs, i.e. either throw more money at the
problem or get smarter. This is where probabilistic data structures enter the
picture, they are a <em>specific</em> mechanism with <em>specific</em> draw backs and
advantages for <em>specific</em> use cases!</p>
<p>First let’s take a quick look at the outline of big data systems. A very nice
read on the topic of designing such systems is <a href="https://www.manning.com/books/big-data">Big Data: Principles and
best practices of scalable realtime data
systems</a> by Marz N. and Warren J. Here
such a system is categorized broadly as:</p>
\[\text{query} = f(\text{all data})\]
<p>Everything else is optimization. And this is good starting point.
This idea is highly robust, handles human error well and can use
simple to implement and precise algorithms. However, it’s also
very slow and expensive! For example, calculating the cardinality
like this is pretty straight forward, you just go ahead and count
them, without counting the same element twice.</p>
<p>However, in order to deal with real time requirements and make the system
usable this view is to simplistic and needs to be augmented. The solution
proposed in the book and others is the so called Lambda architecture. This
architecture augments the slow but precise Batch layer with an imprecise but
efficient Speed layer and a Serving layer that integrates both results to
generate a seamless view of the query.</p>
<p>You can find a quick primer and resources on (the aptly named)
<a href="http://lambda-architecture.net">lambda-architecture.net</a> website that
dedicates itself to this architecture.</p>
<p><a href="/assets/2016/01/la-overview_small.png"><img src="/assets/2016/01/la-overview_small.png" /></a>
<small>Image owned by <a href="http://lambda-architecture.net">lambda-architecture.net</a><small></small></small></p>
<ul>
<li>All data entering the system is dispatched to both the batch layer and the
speed layer for processing.</li>
<li>The batch layer has two functions: (i) managing the master dataset (an
immutable, append-only set of raw data), and (ii) to pre-compute the batch
views.</li>
<li>The serving layer indexes the batch views so that they can be queried in
low-latency, ad-hoc way.</li>
<li>The speed layer compensates for the high latency of updates to the serving
layer and deals with recent data only.</li>
<li>Any incoming query can be answered by merging results from batch views and
real-time views.</li>
</ul>
<p>This architecture is not without
<a href="http://radar.oreilly.com/2014/07/questioning-the-lambda-architecture.html">critique</a>.
But it is a good model to keep in your head even if you deviate from this
approach.</p>
<h4 id="tldr">TL;DR</h4>
<p>So, when and where to use probabilistic data structures? If you need to
optimize for speed and memory based on (paying) user requirements then add them
as mechanism in the <strong>speed layer</strong> of your architecture. You could argue that
doing some preliminary filtering in the serving layer might be a good use case
as well, however in my opinion this layer should stay as agnostic to the actual
data processing as possible.</p>
<h2 id="hyperloglog">HyperLogLog</h2>
<p>Theory and implementation notes can be found in <a href="http://research.google.com/pubs/pub40671.html">HyperLogLog in Practice:
Algorithmic Engineering of a State of The Art Cardinality Estimation
Algorithm</a> from Research at
Google and <a href="http://algo.inria.fr/flajolet/Publications/FlFuGaMe07.pdf">HyperLogLog: the analysis of a near-optimal cardinality estimation
algorithm</a> from
Flajolet P., et al., the original authors.</p>
<p>The implementation used in this post is based on <a href="http://stackoverflow.com/a/6107232">this great
answer</a> on stackoverflow (what would life
be without it) by <a href="http://stackoverflow.com/users/36174/actual">“actual”</a> which
gave me some <em>Aha!</em> moments.</p>
<p>Although the mathematical reasoning behind it is a little bit intricate, the
core idea of HyperLogLog is actually quite intuitive. It is based on the
likelihood of encountering a specific pattern of 0s in the hash of a datum and relating
that to the size of the cardinality <em>probably</em> necessary to find it.</p>
<p>The easiest way to understand it is a slightly different view of the data
(without loss of generality). Let’s say the values are viewed as random natural
numbers instead of randomized bit vectors, e.g.:</p>
\[\begin{align}
1 &= 0b0000000000000001 \\
11593 &= 0b0010110101001001 \\
32768 &= 0b1000000000000000 \\
44266 &= 0b1010110011101010 \\
65535 &= 0b1111111111111111
\end{align}\]
<p>Half of the randomly chosen numbers will be \(\geq 2^{15}\) (\(2^{16} / 2 = 2^{16} * 2^{-1} = 2^{15}\)).
So, the chance to hit a low rank number is very high, e.g. for \(\text{rank}(n) =
1\) (i.e. no leading zero) the chance to hit a number is \(2^{15} / 2^{16} =
50\%\) (a 1 in the front leaving 15 bits). However to find one of \(\text{rank(n)} = 15\) only 1 number is possible, i.e. \(1 =
0b0000000000000001\), so the chance to hit that number is \(1 / 2^{16} =
0.0015\%\).</p>
<p>This is the key observation, the higher the ranks are that you observe the
higher the cardinality has to be in order to have enough chances to find it!</p>
<p>Using only this one estimation is very crude however You split your observation
into multiple estimators and base the calculation on a mean value of them, more
on that in the Implementation.</p>
<p>Now, as I said that example was just a special case. The hashes really are
randomly set bit vectors and you can use whatever pattern you want for your rank
implementation. However, the shown one is easy to implement and used by the
authors.</p>
<p>If you’d rely only on a single measurement you’d introduce a very high
variability. Remember, in our example above you had a chance of \(50\%\) to
hit a \(rank(n) = 1\), this doesn’t tell you much, cardinality could be
anything in \(0 < \text{cardinality} < 2^{15}\). To solve this, a technique
known as <a href="http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.12.7100">stochastic
averaging</a>
used. To that end, the input stream of data elements \(S\) is divided into \(m\) sub streams \(S_i\) of roughly equal size, using the first \(p\) bits
of the hash values, where \(m = 2^p\).</p>
<p>This leads us to the actual formula to estimate the cardinality:</p>
\[\begin{align}
E &= \alpha_m * m^2 * \left(\sum_{j=1}^{m} 2^{-M[j]}\right)^{-1} \\
M[i] &= \max_{x \in S_i} \text{rank}(x)
\end{align}\]
<p>The factor \(\alpha_m\) is approximated in real world implementations. For the mathematical
details I’d like to refer you to the papers.</p>
<h3 id="implementation-notes">Implementation notes</h3>
<p>The full code (ECMAScript and transpiled version using
<a href="https://babeljs.io">Babel</a>) can be found on
<a href="https://github.com/chjdev/chjdev.github.io/tree/master/assets/2016/01/hll/">Github</a>.</p>
<p>You’ll see that a different version for the \(\text{rank}(x)\) function is
used. Instead of using the most significant bits, the order is flipped and the
least significant bits are used. The implementation of this version is easier
and doesn’t make a difference since the hash value is random. Speaking of hash,
the <a href="https://en.wikipedia.org/wiki/Fowler–Noll–Vo_hash_function">FNV-1a</a> hash
is used. It is fast and has good randomness.</p>
<p>How did we arrive at the “20,000 times less memory” figure earlier?
The implementation used for the gold standard (actual cardinality) is based on
a hash table: <code class="language-plaintext highlighter-rouge">gold[id] = true;</code> with the actual cardinality then being simply
<code class="language-plaintext highlighter-rouge">Object.keys(gold).length</code>. Although not the most efficient implementation
memory wise it’s actually well behaved and grows linearly with cardinality. So
assuming a 32 bit architecture:</p>
\[\text{sizeof}(\text{gold}) = \text{cardinality} * (4\text{b} + 4\text{b})\]
<p>Why \(4\text{b}\) twice? Well once for the key and once for the value that
(a boolean, is represented with an int on x86 when unpacked).
So for our cardinality the gold standard needs \(10,000,000 * 8\text{b}
\approx 80\text{mb}\). (Note: this is the theoretical minimal size, without
taking memory for the object into account.)
The HyperLogLog data structure for a standard error of 4% however is using only:</p>
\[\approx 2^{\left\lceil{\log_2\left(\left(\frac{1.04}{0.04}\right)^2\right)}\right \rceil} * 4\text{b} = 4\text{kb}\]
<p>Hence \(80\text{mb} = 20,000 * 4\text{kb}\). The size of the HyperLogLog is
basically constant, however it doubles as soon as you need to use a 64bit hash
for representing bigger (practically unlimited) cardinalities.</p>
<p>The browser experiment uses a <a href="https://www.w3.org/TR/workers/">Web Worker</a> as
driver that generates random visitors and tries to add them as fast as possible
to the HyperLogLog and (if the option is checked) the gold implementation. The
gold implementation uses the hash table version described earlier. The driver
sends updates about the state after specified amount of steps. An interval
function is refreshing the graph every second based on the data generated.</p>
<link href="/assets/2016/01/hll/hll.css" rel="stylesheet" />
<script src="//d3js.org/d3.v3.min.js"></script>
<script src="/assets/2016/01/hll/hll_chart.js"></script>
<script>
var unique_visitors = 10000000,
std_error = 0.04,
debug_step = 10000,
data = [],
update = function () {},
worker = null,
updater = null;
function graph_reset() {
update = initGraph(data, document.getElementById('chart').offsetWidth * 0.85, std_error);
}
function start_worker() {
stop_worker();
data = [];
graph_reset();
updater = window.setInterval(function () { update(data); }, 1000);
worker = new Worker('/assets/2016/01/hll/hll_worker.js');
worker.addEventListener('message', function(e) {
if (e.data[0] == debug_step) { //first
data.push([0,0,0,e.data[3]/2]);
}
data.push(e.data);
}, false);
worker.postMessage([std_error, unique_visitors, debug_step, document.getElementById('with_gold').checked]);
}
function stop_worker() {
if (worker) {
worker.terminate();
worker = null;
}
if (updater) {
window.clearInterval(updater);
updater = null;
}
}
window.onload = window.onresize = graph_reset;
</script>
<div class="message"><b>Warning!</b> This experiment is quite intense and might
slow down your browser quite a bit! A modern browser is necessary, the
best performing one while testing was Firefox.</div>
<table>
<thead>
<tr>
<th>Experiment Parameter</th>
<th style="text-align: right">Value</th>
</tr>
</thead>
<tbody>
<tr>
<td>Max. Unique Visitors</td>
<td style="text-align: right">10,000,000</td>
</tr>
<tr>
<td>Standard Error</td>
<td style="text-align: right">4%</td>
</tr>
<tr>
<td>Data point every x visits</td>
<td style="text-align: right">10,000</td>
</tr>
<tr>
<td>Total visits</td>
<td style="text-align: right">50,000,000</td>
</tr>
<tr>
<td>Hash function</td>
<td style="text-align: right"><a href="https://en.wikipedia.org/wiki/Fowler–Noll–Vo_hash_function">FNV-1a</a></td>
</tr>
</tbody>
</table>
<p><input type="checkbox" id="with_gold" name="with_gold" value="with_gold" checked="true" />Run with gold standard? <br />
<button name="stop" onclick="stop_worker();">Stop Experiment</button>
<button name="start" onclick="start_worker();">Run Experiment</button></p>
The k-Means Algorithm Visualized2016-01-18T00:00:00+00:00https://chjdev.com/2016/01/18/kmeans<p>This quick post started as a challenge to myself: “I wonder if I can bang out a
<a href="https://en.wikipedia.org/wiki/K-means_clustering">k-means</a> implementation with
visualization from memory in less than 2 game of thrones episodes…” The
result is an immutable, functional implementation in ES6 including a
visualization in <a href="https://d3js.org">D3.js</a>.</p>
<blockquote>
<p>k-means clustering is a method of vector quantization, originally from signal
processing, that is popular for cluster analysis in data mining. k-means
clustering aims to partition n observations into k clusters in which each
observation belongs to the cluster with the nearest mean, serving as a
prototype of the cluster. This results in a partitioning of the data space into
Voronoi cells.</p>
</blockquote>
<p>The naïve algorithm is pretty simple:</p>
<ul>
<li>decide on a distance measure (e.g.
<a href="https://en.wikipedia.org/wiki/Euclidean_distance">Euclidean</a> distance)</li>
<li>decide a-priori how many clusters you want and randomly position the centroid
in the feature space.</li>
<li>now iteratively
<ul>
<li>assign each point to a cluster based on minimum distance to the centroids</li>
<li>move the centroids to the actual cluster centroid positions</li>
<li>repeat until change below threshold</li>
</ul>
</li>
</ul>
<p>The Euclidean version is well suited to find circular clusters, the choice of
measure always depends on the problem at hand. Also there are approaches to
infer the optimal number of clusters, but that is left to a post in the future.</p>
<p>The full ECMAScript code can be found
<a href="/assets/2016/01/kmeans.es">here</a> or viewed directly on
<a href="https://github.com/chjdev/chjdev.github.io/tree/master/assets/2016/01/kmeans.es">Github</a>.
A transpiled version using <a href="https://babeljs.io">Babel</a> can be found
<a href="/assets/2016/01/kmeans.js">here</a>.</p>
<!-- rendering code -->
<script src="//d3js.org/d3.v3.min.js"></script>
<script src="/assets/2016/01/kmeans.js"></script>
What's a Decision Tree?2016-01-16T00:00:00+00:00https://chjdev.com/2016/01/16/decision-tree<p>This post is based on the chapter
in the great book: <a href="http://www.amazon.com/Artificial-Intelligence-Modern-Approach-3rd/dp/0136042597/">Artificial Intelligence: A Modern
Approach</a></p>
<p>So what’s a decision tree? Simply put a way to make predictions based on
observations from a knowledge base. Let’s say we observed the following
behavior data set for deciding whether to wait for a table at a restaurant. It
contains twelve observations in total and is based on 10 attributes.</p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="kd">const</span> <span class="nx">attrib_strs</span> <span class="o">=</span> <span class="p">[</span><span class="dl">"</span><span class="s2">Alt</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">Bar</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">Fri</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">Hun</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">Pat</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">Price</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">Rain</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">Res</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">Type</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">Est</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">WillWait</span><span class="dl">"</span><span class="p">],</span>
<span class="nx">attribs</span> <span class="o">=</span> <span class="nb">Object</span><span class="p">.</span><span class="nx">keys</span><span class="p">(</span><span class="nx">attrib_strs</span><span class="p">).</span><span class="nx">slice</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="o">-</span><span class="mi">1</span><span class="p">),</span>
<span class="nx">examples</span> <span class="o">=</span> <span class="p">[</span> <span class="p">[</span><span class="dl">"</span><span class="s2">true</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">false</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">false</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">true</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">1</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">3</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">false</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">true</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">french</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">10</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">true</span><span class="dl">"</span><span class="p">]</span>
<span class="p">,</span> <span class="p">[</span><span class="dl">"</span><span class="s2">true</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">false</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">false</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">true</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">2</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">1</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">false</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">false</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">thai</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">40</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">false</span><span class="dl">"</span><span class="p">]</span>
<span class="p">,</span> <span class="p">[</span><span class="dl">"</span><span class="s2">false</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">true</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">false</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">false</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">2</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">1</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">false</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">false</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">burger</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">10</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">true</span><span class="dl">"</span><span class="p">]</span>
<span class="p">,</span> <span class="p">[</span><span class="dl">"</span><span class="s2">true</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">false</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">true</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">true</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">2</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">1</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">true</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">false</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">thai</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">30</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">true</span><span class="dl">"</span><span class="p">]</span>
<span class="p">,</span> <span class="p">[</span><span class="dl">"</span><span class="s2">true</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">false</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">true</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">false</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">2</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">3</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">false</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">true</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">french</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">100</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">false</span><span class="dl">"</span><span class="p">]</span>
<span class="p">,</span> <span class="p">[</span><span class="dl">"</span><span class="s2">false</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">true</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">false</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">true</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">1</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">2</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">true</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">true</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">italian</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">10</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">true</span><span class="dl">"</span><span class="p">]</span>
<span class="p">,</span> <span class="p">[</span><span class="dl">"</span><span class="s2">false</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">true</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">false</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">false</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">0</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">1</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">true</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">false</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">burger</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">10</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">false</span><span class="dl">"</span><span class="p">]</span>
<span class="p">,</span> <span class="p">[</span><span class="dl">"</span><span class="s2">false</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">false</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">false</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">true</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">1</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">2</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">true</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">true</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">thai</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">10</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">true</span><span class="dl">"</span><span class="p">]</span>
<span class="p">,</span> <span class="p">[</span><span class="dl">"</span><span class="s2">false</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">true</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">true</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">false</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">2</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">1</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">true</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">false</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">burger</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">100</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">false</span><span class="dl">"</span><span class="p">]</span>
<span class="p">,</span> <span class="p">[</span><span class="dl">"</span><span class="s2">true</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">true</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">true</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">true</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">2</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">3</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">false</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">true</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">italian</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">30</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">false</span><span class="dl">"</span><span class="p">]</span>
<span class="p">,</span> <span class="p">[</span><span class="dl">"</span><span class="s2">false</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">false</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">false</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">false</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">0</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">1</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">false</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">false</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">thai</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">10</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">false</span><span class="dl">"</span><span class="p">]</span>
<span class="p">,</span> <span class="p">[</span><span class="dl">"</span><span class="s2">true</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">true</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">true</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">true</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">2</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">1</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">false</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">false</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">burger</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">60</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">true</span><span class="dl">"</span><span class="p">]</span>
<span class="p">];</span></code></pre></figure>
<p>The goal now is to see if there is a pattern in this data that allows us to predict whether a new
observation based on these attributes will yield a positive or negative decision for the “will wait?”
question.</p>
<p>But how can we model this decision? Decision Trees to the rescue! In this model we try to build a
tree that leads us through intermediate decisions (stored in internal nodes) to a definite decision
(stored in it’s leaves) with the minimum steps required. Trivially one might just create a branch
for each example, however this is very inefficient and has terribly predictive performance for new
unobserved examples.</p>
<p>This whole idea actually more intuitive than it sounds, let’s look at the algorithm as
defined in the
<a href="http://www.amazon.com/Artificial-Intelligence-Modern-Approach-3rd/dp/0136042597/">book</a>:</p>
<blockquote>
<ol>
<li>If there are some positive and some negative examples, then choose the
best attribute to split them.</li>
<li>If all the remaining examples are positive (or all negative), then we are
done: we can answer Yes or No.</li>
<li>If there are no examples left, it means that no such example has been
observed, and we return a default value calculated from the majority
classification at the node’s parent.</li>
<li>If there are no attributes left, but both positive and negative examples,
we have a problem. It means that these examples have exactly the same
description, but different classifications. This happens when some of the
data are incorrect; we say there is noise in the data. It also happens
either when the attributes do not give enough information to describe the
situation fully, or when the domain is truly nondeterministic. One simple
way out of the problem is to use a majority vote.</li>
</ol>
</blockquote>
<p>Alright, so basically it’s a recursion that tries to find the attribute that best splits the remaining
examples in each step.</p>
<p>Let’s translate this into code! The full ECMAScript code can be found
<a href="/assets/2016/01/decision_tree.es">here</a> or viewed directly on
<a href="https://github.com/chjdev/chjdev.github.io/tree/master/assets/2016/01/decision_tree.es">Github</a>.
A transpiled version using <a href="https://babeljs.io">Babel</a> can be found
<a href="/assets/2016/01/decision_tree.js">here</a>.</p>
<p>The final (live) rendering that you can see at the bottom uses
<a href="http://d3js.org">D3.js</a> and is based on <a href="https://gist.github.com/d3noob/8323795">this
example</a>.</p>
<p>First of the main <code class="language-plaintext highlighter-rouge">decision_tree_learning</code> function. It’s basically a 1:1 translation of the
informal description above.</p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="kd">function</span> <span class="nx">decision_tree_learning</span><span class="p">(</span><span class="nx">examples</span><span class="p">,</span> <span class="nx">attribs</span><span class="p">,</span> <span class="nx">attrib_strs</span><span class="p">,</span> <span class="nx">def</span><span class="p">)</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">examples</span><span class="p">.</span><span class="nx">length</span> <span class="o">===</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="nx">def</span><span class="p">;</span>
<span class="p">}</span> <span class="k">else</span> <span class="k">if</span> <span class="p">(</span><span class="nx">examples</span><span class="p">.</span><span class="nx">every</span><span class="p">(</span><span class="nx">example</span> <span class="o">=></span> <span class="nx">example</span><span class="p">.</span><span class="nx">slice</span><span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span> <span class="o">===</span> <span class="nx">examples</span><span class="p">[</span><span class="mi">0</span><span class="p">].</span><span class="nx">slice</span><span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">)[</span><span class="mi">0</span><span class="p">]))</span> <span class="p">{</span>
<span class="c1">// classification is same for all</span>
<span class="k">return</span> <span class="nx">examples</span><span class="p">[</span><span class="mi">0</span><span class="p">].</span><span class="nx">slice</span><span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">)[</span><span class="mi">0</span><span class="p">];</span>
<span class="p">}</span> <span class="k">else</span> <span class="k">if</span> <span class="p">(</span><span class="nx">attribs</span><span class="p">.</span><span class="nx">length</span> <span class="o">===</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="nx">majority_value</span><span class="p">(</span><span class="nx">examples</span><span class="p">);</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="kd">const</span> <span class="p">[</span><span class="nx">best_attrib</span><span class="p">,</span> <span class="nx">best_values</span><span class="p">]</span> <span class="o">=</span> <span class="nx">choose_attribute</span><span class="p">(</span><span class="nx">attribs</span><span class="p">,</span> <span class="nx">examples</span><span class="p">),</span>
<span class="nx">tree</span> <span class="o">=</span> <span class="p">{},</span>
<span class="nx">m</span> <span class="o">=</span> <span class="nx">majority_value</span><span class="p">(</span><span class="nx">examples</span><span class="p">);</span>
<span class="nx">tree</span><span class="p">[</span><span class="nx">attrib_strs</span><span class="p">[</span><span class="nx">best_attrib</span><span class="p">]]</span> <span class="o">=</span> <span class="p">{};</span>
<span class="k">return</span> <span class="nx">best_values</span><span class="p">.</span><span class="nx">reduce</span><span class="p">(</span><span class="kd">function</span> <span class="p">(</span><span class="nx">tree</span><span class="p">,</span> <span class="nx">v_i</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">const</span> <span class="nx">examples_i</span> <span class="o">=</span> <span class="nx">examples</span><span class="p">.</span><span class="nx">filter</span><span class="p">(</span><span class="nx">example</span> <span class="o">=></span> <span class="nx">example</span><span class="p">[</span><span class="nx">best_attrib</span><span class="p">]</span> <span class="o">===</span> <span class="nx">v_i</span><span class="p">),</span>
<span class="nx">attribs_i</span> <span class="o">=</span> <span class="nx">attribs</span><span class="p">.</span><span class="nx">filter</span><span class="p">((</span><span class="nx">_</span><span class="p">,</span> <span class="nx">idx</span><span class="p">)</span> <span class="o">=></span> <span class="nx">idx</span> <span class="o">!==</span> <span class="o">+</span><span class="nx">best_attrib</span><span class="p">),</span>
<span class="nx">subtree</span> <span class="o">=</span> <span class="nx">decision_tree_learning</span><span class="p">(</span><span class="nx">examples_i</span><span class="p">,</span> <span class="nx">attribs_i</span><span class="p">,</span> <span class="nx">attrib_strs</span><span class="p">,</span> <span class="nx">m</span><span class="p">);</span>
<span class="nx">tree</span><span class="p">[</span><span class="nx">attrib_strs</span><span class="p">[</span><span class="nx">best_attrib</span><span class="p">]][</span><span class="nx">v_i</span><span class="p">]</span> <span class="o">=</span> <span class="nx">subtree</span><span class="p">;</span>
<span class="k">return</span> <span class="nx">tree</span><span class="p">;</span>
<span class="p">},</span> <span class="nx">tree</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span></code></pre></figure>
<p>The function <code class="language-plaintext highlighter-rouge">majority_value</code> returns the <code class="language-plaintext highlighter-rouge">WillWait</code> value that occured most often for this
subset of examples.</p>
<p>Next of is the <code class="language-plaintext highlighter-rouge">choose_attribute</code> function that selects the attribute that
yields the best split of the example subset. That’s pretty much it for the
decision tree itself. the <code class="language-plaintext highlighter-rouge">choose_attribute</code> function is generic and can use
different heuristics. Here we’ll use a heuristic that is based on which
attribute provides the highest “information gain”.</p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="kd">function</span> <span class="nx">choose_attribute</span><span class="p">(</span><span class="nx">attribs</span><span class="p">,</span> <span class="nx">examples</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="nx">attribs</span><span class="p">.</span><span class="nx">map</span><span class="p">(</span><span class="nx">attribute</span> <span class="o">=></span> <span class="p">[</span><span class="nx">attribute</span><span class="p">,</span> <span class="nx">distinct</span><span class="p">(</span><span class="nx">examples</span><span class="p">.</span><span class="nx">map</span><span class="p">(</span><span class="nx">example</span> <span class="o">=></span> <span class="nx">example</span><span class="p">[</span><span class="nx">attribute</span><span class="p">]))])</span>
<span class="p">.</span><span class="nx">map</span><span class="p">(</span><span class="nx">A</span> <span class="o">=></span> <span class="p">[</span><span class="nx">A</span><span class="p">,</span> <span class="nx">gain</span><span class="p">(</span><span class="nx">A</span><span class="p">,</span> <span class="nx">examples</span><span class="p">)])</span>
<span class="p">.</span><span class="nx">reduce</span><span class="p">((</span><span class="nx">acc</span><span class="p">,</span> <span class="nx">cur</span><span class="p">,</span> <span class="nx">idx</span><span class="p">)</span> <span class="o">=></span> <span class="nx">cur</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">></span> <span class="nx">acc</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="p">?</span> <span class="nx">cur</span> <span class="p">:</span> <span class="nx">acc</span><span class="p">,</span> <span class="p">[[],</span> <span class="o">-</span><span class="mi">1</span><span class="p">])[</span><span class="mi">0</span><span class="p">];</span>
<span class="p">}</span></code></pre></figure>
<p><code class="language-plaintext highlighter-rouge">distinct</code> is a helper function returning an array with the distinct values
of the input array, in this case the different values
for an attribute.</p>
<p>As stated the heuristic in this example is “information gain” the mathematical
details are beyond the scope of this quick post. You can find the details in
the book or <a href="https://en.wikipedia.org/wiki/Information_gain_ratio">here</a>. In
general, the information gain from the attribute test is the difference between
the original information requirement (in bits) and the new requirement.</p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="kd">function</span> <span class="nx">gain</span><span class="p">(</span><span class="nx">A</span><span class="p">,</span> <span class="nx">examples</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">var</span> <span class="p">[</span><span class="nx">p</span><span class="p">,</span> <span class="nx">n</span><span class="p">]</span> <span class="o">=</span> <span class="nx">split_pn</span><span class="p">(</span><span class="nx">examples</span><span class="p">);</span>
<span class="k">return</span> <span class="nx">I</span><span class="p">([</span><span class="nx">p</span><span class="o">/</span><span class="p">(</span><span class="nx">p</span><span class="o">+</span><span class="nx">n</span><span class="p">),</span> <span class="nx">n</span><span class="o">/</span><span class="p">(</span><span class="nx">p</span><span class="o">+</span><span class="nx">n</span><span class="p">)])</span> <span class="o">-</span> <span class="nx">remainder</span><span class="p">(</span><span class="nx">A</span><span class="p">,</span> <span class="nx">examples</span><span class="p">);</span>
<span class="p">}</span>
<span class="kd">function</span> <span class="nx">I</span><span class="p">(</span><span class="nx">P</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="nx">P</span><span class="p">.</span><span class="nx">filter</span><span class="p">(</span><span class="nx">v_i</span> <span class="o">=></span> <span class="nx">v_i</span> <span class="o">></span> <span class="mi">0</span><span class="p">)</span>
<span class="p">.</span><span class="nx">map</span><span class="p">(</span><span class="nx">v_i</span> <span class="o">=></span> <span class="o">-</span><span class="nx">v_i</span><span class="o">*</span><span class="nb">Math</span><span class="p">.</span><span class="nx">log2</span><span class="p">(</span><span class="nx">v_i</span><span class="p">))</span>
<span class="p">.</span><span class="nx">reduce</span><span class="p">((</span><span class="nx">a</span><span class="p">,</span><span class="nx">b</span><span class="p">)</span> <span class="o">=></span> <span class="nx">a</span> <span class="o">+</span> <span class="nx">b</span><span class="p">,</span> <span class="mi">0</span><span class="p">);</span>
<span class="p">}</span>
<span class="kd">function</span> <span class="nx">remainder</span><span class="p">(</span><span class="nx">A</span><span class="p">,</span> <span class="nx">examples</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">const</span> <span class="p">[</span><span class="nx">p</span><span class="p">,</span> <span class="nx">n</span><span class="p">]</span> <span class="o">=</span> <span class="nx">split_pn</span><span class="p">(</span><span class="nx">examples</span><span class="p">);</span>
<span class="k">return</span> <span class="nx">A</span><span class="p">[</span><span class="mi">1</span><span class="p">].</span><span class="nx">reduce</span><span class="p">(</span><span class="kd">function</span> <span class="p">(</span><span class="nx">prev</span><span class="p">,</span> <span class="nx">cur</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">const</span> <span class="nx">examples_cur</span> <span class="o">=</span> <span class="nx">examples</span><span class="p">.</span><span class="nx">filter</span><span class="p">(</span><span class="nx">example</span> <span class="o">=></span> <span class="nx">example</span><span class="p">[</span><span class="nx">A</span><span class="p">[</span><span class="mi">0</span><span class="p">]]</span> <span class="o">===</span> <span class="nx">cur</span><span class="p">),</span>
<span class="p">[</span><span class="nx">p_i</span><span class="p">,</span> <span class="nx">n_i</span><span class="p">]</span> <span class="o">=</span> <span class="nx">split_pn</span><span class="p">(</span><span class="nx">examples_cur</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">examples_cur</span><span class="p">.</span><span class="nx">length</span> <span class="o">===</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="nx">prev</span> <span class="o">+</span> <span class="mi">0</span><span class="p">;</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="k">return</span> <span class="nx">prev</span> <span class="o">+</span> <span class="p">((</span><span class="nx">p_i</span><span class="o">+</span><span class="nx">n_i</span><span class="p">)</span><span class="o">/</span><span class="p">(</span><span class="nx">p</span><span class="o">+</span><span class="nx">n</span><span class="p">)</span> <span class="o">*</span> <span class="nx">I</span><span class="p">([</span><span class="nx">p_i</span><span class="o">/</span><span class="p">(</span><span class="nx">p_i</span> <span class="o">+</span> <span class="nx">n_i</span><span class="p">),</span> <span class="nx">n_i</span><span class="o">/</span><span class="p">(</span><span class="nx">p_i</span> <span class="o">+</span> <span class="nx">n_i</span><span class="p">)]));</span>
<span class="p">}</span>
<span class="p">},</span> <span class="mi">0</span><span class="p">);</span>
<span class="p">}</span></code></pre></figure>
<p><code class="language-plaintext highlighter-rouge">split_pn</code> is just a simple function that splits the <code class="language-plaintext highlighter-rouge">examples</code> into positive (<code class="language-plaintext highlighter-rouge">"true"</code>) and negative (<code class="language-plaintext highlighter-rouge">"false"</code>) instances.</p>
<p>And that’s it. The resulting tree for running this code can be seen at the bottom.</p>
<p><strong>Note:</strong> the tree we arrived at here is slightly different than Russel’s & Norvig’s, it’s
actually slightly smaller! If that’s due to a bug in my implementation please let me know ;)</p>
<!-- rendering code -->
<script src="//d3js.org/d3.v3.min.js"></script>
<script src="/assets/2016/01/decision_tree.js"></script>
<style>
.node circle {
fill: #fff;
stroke: steelblue;
stroke-width: 3px;
}
.node text { font: 12px sans-serif; }
.link {
fill: none;
stroke: #ccc;
stroke-width: 2px;
}
</style>
<script>
function convert_to_d3(tree) {
var mroot = Object.keys(tree)[0];
return (function _convert_to_d3(name, par, tree) {
if (typeof tree ==='object') {
return { 'name': name
, 'parent': par
, 'children': Object.keys(tree).map(function (key) {return _convert_to_d3(key, name, tree[key]);})
};
} else {
return { 'name': name
, 'leaf': tree
, 'parent': par
};
}
})(mroot, 'null', tree[mroot]);
}
function draw_tree() {
//adapted from: https://gist.github.com/d3noob/8323795)
var d3_tree = convert_to_d3(tree_data),
margin = {top: 20, right: 50, bottom: 20, left: 50},
aspect = 16 / 9;
orig_width = document.getElementsByTagName("main")[0].offsetWidth,
width = orig_width - margin.right - margin.left,
orig_height = orig_width / aspect,
height = orig_height - margin.top - margin.bottom;
var i = 0;
var tree = d3.layout.tree()
.size([height, width]);
var diagonal = d3.svg.diagonal()
.projection(function(d) { return [d.y, d.x]; });
d3.selectAll("svg").remove();
var svg = d3.select("#tree_diagram").append("svg")
.attr("width", width + margin.right + margin.left)
.attr("height", height + margin.top + margin.bottom)
.append("g")
.attr("transform", "translate(" + margin.left + "," + margin.top + ")");
update(d3_tree);
function update(source) {
// Compute the new tree layout.
var nodes = tree.nodes(d3_tree).reverse(),
links = tree.links(nodes);
// Declare the nodes…
var node = svg.selectAll("g.node")
.data(nodes, function(d) { return d.id || (d.id = ++i); });
// Enter the nodes.
var nodeEnter = node.enter().append("g")
.attr("class", "node")
.attr("transform", function(d) {
return "translate(" + d.y + "," + d.x + ")"; });
nodeEnter.append("circle")
.attr("r", 10)
.style("fill", function(d) {
if (d.children || d._children) return "#ccc";
else if (d.leaf) return d.leaf === 'true' ? "#55ff55" : "#ff5555";
else return "#fff";
});
nodeEnter.append("text")
.attr("x", -15)
.attr("dy", ".35em")
.attr("text-anchor", "end")
.text(function(d) { return d.name; })
.style("fill-opacity", 1);
// Declare the links…
var link = svg.selectAll("path.link")
.data(links, function(d) { return d.target.id; });
// Enter the links.
link.enter().insert("path", "g")
.attr("class", "link")
.attr("d", diagonal);
}
}
window.onresize = draw_tree;
window.onload= draw_tree;
</script>
The World according to Flickr2016-01-12T00:00:00+00:00https://chjdev.com/2016/01/12/world-according-to-flickr<p><a href="/experiments/vienna_acyclic/world_flickr.png"><img src="/experiments/vienna_acyclic/world_flickr.png" /></a></p>
<p>Part of project proposal I’m working on currently is the network
(re)construction of geographical locations. For a first experiment I used a
publicly available data set provided by <a href="https://www.flickr.com/">Flickr</a> that
contains 100 million photos of which 49 million are geotagged. It can be found
<a href="http://yahoolabs.tumblr.com/post/89783581601/one-hundred-million-creative-commons-flickr-images">here</a>.
The dataset consists of 10 TSV files holding the information (like id, geotag,
user, etc.) of 100.000 photos each.</p>
<p>I used <a href="http://spark.apache.org/">Apache Spark</a> for the preprocessing of these
files and converted it into a <a href="http://spark.apache.org/graphx/">GraphX</a> graph.
An Edge between locations exists if at least 2 persons visited both and
locations are less then 10 degree apart.</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">val</span> <span class="nv">geoTriplets</span> <span class="k">=</span> <span class="n">flickrRowsWithGeo</span>
<span class="o">.</span><span class="py">map</span><span class="o">(</span><span class="n">frow</span> <span class="k">=></span> <span class="o">(</span><span class="nv">frow</span><span class="o">.</span><span class="py">user</span><span class="o">,</span> <span class="nc">List</span><span class="o">(</span><span class="nf">makeVIdx</span><span class="o">(</span><span class="nv">frow</span><span class="o">.</span><span class="py">latitude</span><span class="o">.</span><span class="py">get</span><span class="o">,</span> <span class="nv">frow</span><span class="o">.</span><span class="py">longitude</span><span class="o">.</span><span class="py">get</span><span class="o">))))</span>
<span class="c1">// build a that contains all localities for the user, e.g. (christian, [Innsbruck, Vancouver, Montanita, ...])</span>
<span class="o">.</span><span class="py">reduceByKey</span><span class="o">((</span><span class="n">a</span><span class="o">,</span> <span class="n">b</span><span class="o">)</span> <span class="k">=></span> <span class="n">a</span> <span class="o">++</span> <span class="n">b</span><span class="o">)</span>
<span class="c1">// transform this list into links between the localities [((Innsbruck, Vancouver), christian), ...] and flatten it</span>
<span class="o">.</span><span class="py">flatMap</span><span class="o">(</span><span class="n">nodeList</span> <span class="k">=></span> <span class="nv">nodeList</span><span class="o">.</span><span class="py">_2</span><span class="o">.</span><span class="py">combinations</span><span class="o">(</span><span class="mi">2</span><span class="o">)</span>
<span class="o">.</span><span class="py">filter</span><span class="o">(</span><span class="n">lst</span> <span class="k">=></span> <span class="nf">lst</span><span class="o">(</span><span class="mi">0</span><span class="o">)</span> <span class="o">!=</span> <span class="nf">lst</span><span class="o">(</span><span class="mi">1</span><span class="o">))</span>
<span class="o">.</span><span class="py">map</span><span class="o">(</span><span class="n">lst</span> <span class="k">=></span> <span class="o">((</span><span class="nf">lst</span><span class="o">(</span><span class="mi">0</span><span class="o">),</span> <span class="nf">lst</span><span class="o">(</span><span class="mi">1</span><span class="o">)),</span> <span class="nc">List</span><span class="o">(</span><span class="nv">nodeList</span><span class="o">.</span><span class="py">_1</span><span class="o">))))</span>
<span class="c1">// we sanitize for regionality, i.e. localities that are too far away from each other are excluded</span>
<span class="o">.</span><span class="py">filter</span><span class="o">(</span><span class="n">tup</span> <span class="k">=></span> <span class="o">{</span>
<span class="nf">val</span> <span class="o">(</span><span class="n">lat1</span><span class="o">,</span> <span class="n">lon1</span><span class="o">)</span> <span class="k">=</span> <span class="nf">parseVIdx</span><span class="o">(</span><span class="nv">tup</span><span class="o">.</span><span class="py">_1</span><span class="o">.</span><span class="py">_1</span><span class="o">)</span>
<span class="nf">val</span> <span class="o">(</span><span class="n">lat2</span><span class="o">,</span> <span class="n">lon2</span><span class="o">)</span> <span class="k">=</span> <span class="nf">parseVIdx</span><span class="o">(</span><span class="nv">tup</span><span class="o">.</span><span class="py">_1</span><span class="o">.</span><span class="py">_2</span><span class="o">)</span>
<span class="nf">shortestAngleDist</span><span class="o">(</span><span class="n">lat1</span><span class="o">,</span> <span class="n">lat2</span><span class="o">)</span> <span class="o"><</span> <span class="mi">10</span> <span class="o">&&</span> <span class="nf">shortestAngleDist</span><span class="o">(</span><span class="n">lon1</span><span class="o">,</span> <span class="n">lon2</span><span class="o">)</span> <span class="o"><</span> <span class="mi">10</span>
<span class="o">})</span>
<span class="c1">// we now basically inverted the list and gather up the users per location pair ((Innsbruck, Vancouver), [christian, marti])</span>
<span class="o">.</span><span class="py">reduceByKey</span><span class="o">((</span><span class="n">a</span><span class="o">,</span> <span class="n">b</span><span class="o">)</span> <span class="k">=></span> <span class="n">a</span> <span class="o">++</span> <span class="n">b</span><span class="o">)</span>
<span class="c1">// now we transform the user list into an edge weight and build a triplet (actually quadruplet) of the form (Innsbruck, 2, Vancouver, [christian, marti])</span>
<span class="o">.</span><span class="py">map</span><span class="o">(</span><span class="n">withUserList</span> <span class="k">=></span> <span class="o">(</span><span class="nv">withUserList</span><span class="o">.</span><span class="py">_1</span><span class="o">.</span><span class="py">_1</span><span class="o">,</span> <span class="nv">withUserList</span><span class="o">.</span><span class="py">_2</span><span class="o">.</span><span class="py">size</span><span class="o">,</span> <span class="nv">withUserList</span><span class="o">.</span><span class="py">_1</span><span class="o">.</span><span class="py">_2</span><span class="o">,</span> <span class="nv">withUserList</span><span class="o">.</span><span class="py">_2</span><span class="o">))</span>
<span class="c1">// we only keep edges with a weight > 1 meaning more than 2 person visited both</span>
<span class="o">.</span><span class="py">filter</span><span class="o">(</span><span class="nv">_</span><span class="o">.</span><span class="py">_2</span> <span class="o">></span> <span class="mi">2</span><span class="o">)</span>
<span class="k">def</span> <span class="nf">vertices</span><span class="o">()</span><span class="k">:</span> <span class="kt">RDD</span><span class="o">[(</span><span class="kt">VertexId</span>, <span class="kt">Unit</span><span class="o">)]</span> <span class="k">=</span> <span class="o">{</span>
<span class="nv">geoTriplets</span><span class="o">.</span><span class="py">map</span><span class="o">(</span><span class="nv">_</span><span class="o">.</span><span class="py">_1</span><span class="o">).</span><span class="py">union</span><span class="o">(</span><span class="nv">geoTriplets</span><span class="o">.</span><span class="py">map</span><span class="o">(</span><span class="nv">_</span><span class="o">.</span><span class="py">_3</span><span class="o">)).</span><span class="py">distinct</span><span class="o">().</span><span class="py">map</span><span class="o">((</span><span class="k">_</span><span class="o">,</span> <span class="o">()))</span>
<span class="o">}</span>
<span class="k">def</span> <span class="nf">edges</span><span class="o">()</span><span class="k">:</span> <span class="kt">RDD</span><span class="o">[</span><span class="kt">Edge</span><span class="o">[</span><span class="kt">Int</span><span class="o">]]</span> <span class="k">=</span> <span class="o">{</span>
<span class="nv">geoTriplets</span><span class="o">.</span><span class="py">map</span><span class="o">(</span><span class="n">triplet</span> <span class="k">=></span> <span class="nc">Edge</span><span class="o">(</span><span class="nv">triplet</span><span class="o">.</span><span class="py">_1</span><span class="o">,</span> <span class="nv">triplet</span><span class="o">.</span><span class="py">_3</span><span class="o">,</span> <span class="nv">triplet</span><span class="o">.</span><span class="py">_2</span><span class="o">))</span>
<span class="o">}</span>
<span class="k">def</span> <span class="nf">main</span><span class="o">(</span><span class="n">args</span><span class="k">:</span> <span class="kt">Array</span><span class="o">[</span><span class="kt">String</span><span class="o">])</span><span class="k">:</span> <span class="kt">Unit</span> <span class="o">=</span> <span class="o">{</span>
<span class="k">val</span> <span class="nv">graph</span> <span class="k">=</span> <span class="nc">Graph</span><span class="o">(</span><span class="nf">vertices</span><span class="o">(),</span> <span class="nf">edges</span><span class="o">())</span>
<span class="nv">graph</span><span class="o">.</span><span class="py">vertices</span><span class="o">.</span><span class="py">map</span><span class="o">(</span><span class="nv">_</span><span class="o">.</span><span class="py">_1</span><span class="o">.</span><span class="py">toString</span><span class="o">).</span><span class="py">saveAsTextFile</span><span class="o">(</span><span class="s">"/path/to/out_vertices"</span><span class="o">)</span>
<span class="nv">graph</span><span class="o">.</span><span class="py">triplets</span><span class="o">.</span><span class="py">map</span><span class="o">(</span><span class="n">triplet</span> <span class="k">=></span> <span class="nc">List</span><span class="o">(</span><span class="nv">triplet</span><span class="o">.</span><span class="py">srcId</span><span class="o">,</span> <span class="nv">triplet</span><span class="o">.</span><span class="py">dstId</span><span class="o">,</span> <span class="nv">triplet</span><span class="o">.</span><span class="py">attr</span><span class="o">).</span><span class="py">map</span><span class="o">(</span><span class="nv">_</span><span class="o">.</span><span class="py">toString</span><span class="o">)).</span><span class="py">saveAsTextFile</span><span class="o">(</span><span class="s">"/path/to/out_edges"</span><span class="o">)</span>
<span class="o">}</span></code></pre></figure>
<p>Using Spark the resulting vertices and edges where written to disk and
transformed to a JSON object holding these values. Using
<a href="http://d3js.org/">D3.js</a> (world rendering, top) and
<a href="http://processingjs.org/">Processing.js</a> (acyclic
<a href="https://en.wikipedia.org/wiki/Breadth-first_search">BFS</a> rendering with
starting point Vienna, bottom) this JSON object was loaded and the network
projected onto a <a href="http://bl.ocks.org/mbostock/3734333">Miller</a> and
<a href="http://bl.ocks.org/mbostock/3757110">azimuthal</a> projection respectively.
Vertices where drawn as translucent circles whose overlap creates different
color intensities on the map according to the popularity of the location. Edges
where drawn as translucent arcs with weighted line width that highlight highly
connected vertices via overlapping. The final renderings are based on ~170.000
vertices and over 4 million edges.</p>
New blog incomming2016-01-10T00:00:00+00:00https://chjdev.com/2016/01/10/new-blog<p>Currently working on migrating my Wordpress blog to jekyll on Github Pages.
In the meantime I just wanted to say: Lorem ipsum dolor sit amet, consectetur
adipiscing elit. Pellentesque tempor nulla ligula, vel euismod eros sodales
eu. Praesent at accumsan arcu, sed egestas dui. Praesent suscipit lorem ac mi
aliquam, suscipit semper purus sagittis. Lorem ipsum dolor sit amet, consectetur
adipiscing elit. Pellentesque tempor nulla ligula, vel euismod eros sodales
eu. Praesent at accumsan arcu, sed egestas dui. Praesent suscipit lorem ac mi
aliquam, suscipit semper purus sagittis.</p>
Read Scientific Papers on Your Kindle2015-02-08T00:00:00+00:00https://chjdev.com/2015/02/08/read-scientific-papers-kindle<p>Reading scientific on your Kindle (or other eBook
reader) usually sucks. The text is usually only available as PDF or PS files
and formatted in a way that is meant for printing in A4, or US Letter. A
two-column layout is also very common, which further complicates things. In
this post I show you a simple way to get these papers on your eBook reader for
comfortable reading.</p>
<!--
<div style="text-align: center;"><a
href="http://chjdev.com/wp-content/uploads/2015/02/original.png"><img
src="http://chjdev.com/wp-content/uploads/2015/02/original-232x300.png"
alt="original" width="232" height="300" class="alignnone size-medium
wp-image-153" /></a></div>
<div style="text-align: center;"><iframe style="width: 120px; height: 240px;"
src="//ws-na.amazon-adsystem.com/widgets/q?ServiceVersion=20070822&OneJS=1&Operation=GetAdHtml&MarketPlace=US&source=ss&ref=ss_til&ad_type=product_link&tracking_id=chjdev-20&marketplace=amazon&region=US&placement=B00I15SB16&asins=B00I15SB16&linkId=SPXKVXHNXUMSCTRA&show_border=true&link_opens_in_new_window=true"
width="300" height="150" frameborder="0" marginwidth="0" marginheight="0"
scrolling="no"> </iframe> <iframe style="width: 120px; height: 240px;"
src="//ws-na.amazon-adsystem.com/widgets/q?ServiceVersion=20070822&OneJS=1&Operation=GetAdHtml&MarketPlace=US&source=ss&ref=ss_til&ad_type=product_link&tracking_id=chjdev-20&marketplace=amazon&region=US&placement=B00JG8GOWU&asins=B00JG8GOWU&linkId=2KX4UWXTFMHOSJ5J&show_border=true&link_opens_in_new_window=true"
width="300" height="150" frameborder="0" marginwidth="0" marginheight="0"
scrolling="no"> </iframe> <iframe style="width: 120px; height: 240px;"
src="//ws-na.amazon-adsystem.com/widgets/q?ServiceVersion=20070822&OneJS=1&Operation=GetAdHtml&MarketPlace=US&source=ss&ref=ss_til&ad_type=product_link&tracking_id=chjdev-20&marketplace=amazon&region=US&placement=B00IOY8XWQ&asins=B00IOY8XWQ&linkId=QFDRMH3SYROOYXL5&show_border=true&link_opens_in_new_window=true"
width="300" height="150" frameborder="0" marginwidth="0" marginheight="0"
scrolling="no"> </iframe></div>
-->
<h2 id="step-1-preprocessing-with-briss">Step 1: Preprocessing with <code class="language-plaintext highlighter-rouge">BRISS</code></h2>
<p>First we will preprocess the file a bit to make the next step easier / more
successful. Using the cool little <a href="http://briss.sourceforge.net/"><code class="language-plaintext highlighter-rouge">BRISS</code></a>
tool we will crop out unnecessary parts and only leave the main text area. The
idea is to get rid of line numbers, notes in the margin (e.g. the arXiv line in
our test document), etc.</p>
<p><code class="language-plaintext highlighter-rouge">BRISS</code> is a graphical tool. You can use the menu to load the PDF or
just start it from the terminal: <code class="language-plaintext highlighter-rouge">briss Text\ Understanding\ from\
Scratch.pdf</code> You will be prompted to enter the range of pages that will
be analyzed to find the main text body. Usually it’s fine to just leave it at
the default. <code class="language-plaintext highlighter-rouge">BRISS</code> now tries to find the main text area.</p>
<!--
<div style="text-align: center;"><a
href="http://chjdev.com/wp-content/uploads/2015/02/briss.png"><img
src="http://chjdev.com/wp-content/uploads/2015/02/briss-300x216.png"
alt="briss" width="300" height="216" class="alignnone size-medium wp-image-150"
/></a></div>
-->
<p>Tweak the boxes until they only cover the relevant text and crop the PDF by
clicking <code class="language-plaintext highlighter-rouge">Action > Crop PDF</code>. We now have a PDF document with all
possibly misleading fluff cut out and can move on to the next step.</p>
<!--
<div style="text-align: center;"><a
href="http://chjdev.com/wp-content/uploads/2015/02/cropped.png"><img
src="http://chjdev.com/wp-content/uploads/2015/02/cropped-225x300.png"
alt="cropped" width="225" height="300" class="alignnone size-medium
wp-image-151" /></a></div>
-->
<h2 id="step-2-optimizing-with-k2pdfopt">Step 2: Optimizing with <code class="language-plaintext highlighter-rouge">k2pdfopt</code></h2>
<p>To optimize the cropped PDF for our Kindle we’ll use the
<a href="http://www.willus.com/k2pdfopt/"><code class="language-plaintext highlighter-rouge">k2pdfopt</code></a> tool. It has a
plethora of <a href="http://www.willus.com/k2pdfopt/help/options.shtml">options</a>
suiting many needs, but the default modes usually work fine.</p>
<p><code class="language-plaintext highlighter-rouge">./k2pdfopt -ppgs -dev kpw -mode 2col Text\ Understanding\ from\ Scratch_cropped.pdf</code></p>
<p>And that’s it, now you have a Kindle optimized PDF!</p>
<!--
<div style="text-align: center;"><a
href="http://chjdev.com/wp-content/uploads/2015/02/optimized.png"><img
src="http://chjdev.com/wp-content/uploads/2015/02/optimized-209x300.png"
alt="optimized" width="209" height="300" class="alignnone size-medium
wp-image-152" /></a></div>
-->
<p><strong>Warning</strong> the default modes include the <code class="language-plaintext highlighter-rouge">-n</code> flag, which will
enable native PDF output. This is the preferable mode since it leads to
smaller, better files because it uses native PDF instructions instead of
rendering the pages to bitmaps. However, (at least the 1st gen Paperwhite) may
crash opening files generated with this option, because it runs out of memory.
This forced me to factory reset my device a couple of times during first
experiments.</p>
<p><strong>Solution</strong> either disable native output by specifying <code class="language-plaintext highlighter-rouge">-n-</code>
leading to bigger, uglier files, or install Ghostscript (if you haven’t
already) and include the <a href="http://www.willus.com/k2pdfopt/help/faq.shtml""><code class="language-plaintext highlighter-rouge">-ppgs</code></a>
option. This will post process the file using Ghostscript and fix the issue.</p>
Universal Python Server2015-01-03T00:00:00+00:00https://chjdev.com/2015/01/03/universal-python-server<p>I recently read a cool post by <a href="https://joearms.github.io/">Joe Armstrong</a> in
which he showcases his favorite <a href="http://www.erlang.org/">Erlang</a> program: <a href="https://joearms.github.io/2013/11/21/My-favorite-erlang-program.html">the
universal server.</a>
The program creates an easily distributable generic server, that can perform
any concrete task you tell it to by sending it a function. E.g. it could act as
an HTTP server, RPC server, etc.</p>
<p>This little program “suitable for a 10min talk” highlights how awesomely simple
Erlang can be for the tasks it was designed to do best. I was intrigued and
wanted to see if a comparably simple universal Python server could be created
by using only the standard library.</p>
<p>To make the program distributable the <a href="https://docs.python.org/3/library/multiprocessing.html">multiprocessing</a>
module of the standard library will be used. It contains the tools necessary
for creating multiple processes and inter process communication.</p>
<h2 id="architecture">Architecture</h2>
<p>This version deviates a little bit from Armstrong’s program. Instead of a
<code class="language-plaintext highlighter-rouge">become</code> message, here the function to be performed is included for
every task to be performed. It is implemented using 3 components: a
<code class="language-plaintext highlighter-rouge">manager</code> as central connection point, a <code class="language-plaintext highlighter-rouge">worker</code> which
will receive generic tasks to be performed and a <code class="language-plaintext highlighter-rouge">boss</code> which will
schedule said tasks.</p>
<p>Disclaimer: Obviously this is only a simple little experiment, none of this
code is intended to be used in anything respectable.</p>
<h4 id="manager">Manager</h4>
<p>First here is the <code class="language-plaintext highlighter-rouge">manager</code> service. It’s responsible to provide the
communication channels. Both the <code class="language-plaintext highlighter-rouge">worker</code> - as well as the
<code class="language-plaintext highlighter-rouge">boss</code> processes connect to it for discovery.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">from</span> <span class="nn">multiprocessing.managers</span> <span class="kn">import</span> <span class="n">SyncManager</span>
<span class="kn">import</span> <span class="nn">marshal</span>
<span class="kn">import</span> <span class="nn">os</span>
<span class="k">class</span> <span class="nc">Manager</span><span class="p">(</span><span class="n">SyncManager</span><span class="p">):</span>
<span class="k">pass</span>
<span class="n">Manager</span><span class="p">.</span><span class="n">register</span><span class="p">(</span><span class="s">'get_job_q'</span><span class="p">)</span>
<span class="n">Manager</span><span class="p">.</span><span class="n">register</span><span class="p">(</span><span class="s">'get_res_q'</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">connect</span><span class="p">(</span><span class="n">ip</span><span class="p">,</span> <span class="n">port</span><span class="p">,</span> <span class="n">auth</span><span class="p">):</span>
<span class="n">manager</span> <span class="o">=</span> <span class="n">Manager</span><span class="p">(</span><span class="n">address</span><span class="o">=</span><span class="p">(</span><span class="n">ip</span><span class="p">,</span> <span class="n">port</span><span class="p">),</span> <span class="n">authkey</span><span class="o">=</span><span class="n">auth</span><span class="p">)</span>
<span class="n">manager</span><span class="p">.</span><span class="n">connect</span><span class="p">()</span>
<span class="k">return</span> <span class="n">manager</span>
<span class="k">def</span> <span class="nf">remote</span><span class="p">(</span><span class="n">func</span><span class="p">,</span> <span class="n">manager</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">inner</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="n">manager</span><span class="p">.</span><span class="n">get_job_q</span><span class="p">().</span><span class="n">put</span><span class="p">((</span><span class="n">marshal</span><span class="p">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">func</span><span class="p">.</span><span class="n">__code__</span><span class="p">),</span> <span class="n">args</span><span class="p">,</span> <span class="n">kwargs</span><span class="p">))</span>
<span class="k">return</span> <span class="n">inner</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span><span class="p">:</span>
<span class="kn">from</span> <span class="nn">multiprocessing</span> <span class="kn">import</span> <span class="n">Queue</span>
<span class="n">job_q</span> <span class="o">=</span> <span class="n">Queue</span><span class="p">()</span>
<span class="n">res_q</span> <span class="o">=</span> <span class="n">Queue</span><span class="p">()</span>
<span class="n">Manager</span><span class="p">.</span><span class="n">register</span><span class="p">(</span><span class="s">'get_job_q'</span><span class="p">,</span> <span class="nb">callable</span><span class="o">=</span><span class="k">lambda</span><span class="p">:</span> <span class="n">job_q</span><span class="p">)</span>
<span class="n">Manager</span><span class="p">.</span><span class="n">register</span><span class="p">(</span><span class="s">'get_res_q'</span><span class="p">,</span> <span class="nb">callable</span><span class="o">=</span><span class="k">lambda</span><span class="p">:</span> <span class="n">res_q</span><span class="p">)</span>
<span class="n">manager</span> <span class="o">=</span> <span class="n">Manager</span><span class="p">(</span><span class="n">address</span><span class="o">=</span><span class="p">(</span><span class="s">''</span><span class="p">,</span> <span class="mi">1234</span><span class="p">),</span> <span class="n">authkey</span><span class="o">=</span><span class="sa">b</span><span class="s">'abc'</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="s">'Manager startet with PID='</span><span class="p">,</span> <span class="n">os</span><span class="p">.</span><span class="n">getpid</span><span class="p">())</span>
<span class="n">manager</span><span class="p">.</span><span class="n">get_server</span><span class="p">().</span><span class="n">serve_forever</span><span class="p">()</span></code></pre></figure>
<p>The first thing that’s necessary is to create our own subclass of the
<a href="https://docs.python.org/3/library/multiprocessing.html#multiprocessing.managers.SyncManager">SyncManager</a>
provided by the multiprocessing module. This is necessary to <code class="language-plaintext highlighter-rouge">register</code> the
custom methods used. Here 2 methods are registered: a method to access the job
queue (which will contain the tasks to be performed) and a method to access the
result queue which will communicate the results.</p>
<p>Note: this is a very primitive setup and for example doesn’t play nice with
multiple bosses. <code class="language-plaintext highlighter-rouge">Manager.register</code> is only called with the name of these
functions to tell it that they exist, so if the manager is subsequently
imported it will be ready to use.</p>
<p>Next up is the <code class="language-plaintext highlighter-rouge">connect</code> function. This function is used by the workers and
the boss to connect to the same session. This is akin to joining multiple
running Erlang beam VMs and allows us to communicate with remote processes.</p>
<p><code class="language-plaintext highlighter-rouge">remote</code> is a decorator for functions that allows to seamlessly send them to
remote processes in order to be executed.
It works by taking the decorated function and putting a tuple of its
<a href="https://docs.python.org/3/library/marshal.html">marshalled</a> pseudo-compiled
code (<code class="language-plaintext highlighter-rouge">func.__code__</code>) and arguments in the job queue. The tuple thereby
contains everything a remote process needs to know to reconstruct the task.</p>
<p>Note: the marshal module has to be used because it appears that
<a href="https://docs.python.org/3/library/pickle.html">pickle</a> module is not powerful
enough to handle this specific use cases. Trying to directly pickle the
function and loading it in the remote process will lead to an
<code class="language-plaintext highlighter-rouge">AttributeError</code> because the function is not part of this process, and
pickling of the <code class="language-plaintext highlighter-rouge">__code__</code> builtin unfortunately doesn’t work.</p>
<p>Finally running the manager script will initiate the
<a href="https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Queue">queues</a>
used for communicating and reregister them. Then a manager object is created
and instructed to serve forever.</p>
<h4 id="worker">Worker</h4>
<p>The worker process will perform the tasks scheduled by the boss. It works by
getting tasks from the job queue, reconstructing the function of the task,
invoking this new function and putting its result into the result queue.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">from</span> <span class="nn">manager</span> <span class="kn">import</span> <span class="n">connect</span>
<span class="kn">import</span> <span class="nn">marshal</span>
<span class="kn">import</span> <span class="nn">types</span>
<span class="kn">import</span> <span class="nn">os</span>
<span class="k">def</span> <span class="nf">universal</span><span class="p">(</span><span class="n">job_q</span><span class="p">,</span> <span class="n">res_q</span><span class="p">):</span>
<span class="k">while</span> <span class="bp">True</span><span class="p">:</span>
<span class="n">funcs</span><span class="p">,</span> <span class="n">args</span><span class="p">,</span> <span class="n">kwargs</span> <span class="o">=</span> <span class="n">job_q</span><span class="p">.</span><span class="n">get</span><span class="p">()</span>
<span class="n">func_code</span> <span class="o">=</span> <span class="n">marshal</span><span class="p">.</span><span class="n">loads</span><span class="p">(</span><span class="n">funcs</span><span class="p">)</span>
<span class="n">func</span> <span class="o">=</span> <span class="n">types</span><span class="p">.</span><span class="n">FunctionType</span><span class="p">(</span><span class="n">func_code</span><span class="p">,</span> <span class="nb">globals</span><span class="p">(),</span> <span class="s">'loaded_func'</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="s">'processing '</span><span class="p">,</span> <span class="n">func</span><span class="p">,</span> <span class="n">args</span><span class="p">,</span> <span class="n">kwargs</span><span class="p">)</span>
<span class="n">res_q</span><span class="p">.</span><span class="n">put</span><span class="p">((</span><span class="n">os</span><span class="p">.</span><span class="n">getpid</span><span class="p">(),</span> <span class="n">func</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)))</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span><span class="p">:</span>
<span class="n">manager</span> <span class="o">=</span> <span class="n">connect</span><span class="p">(</span><span class="s">'127.0.0.1'</span><span class="p">,</span> <span class="mi">1234</span><span class="p">,</span> <span class="sa">b</span><span class="s">'abc'</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="s">'Worker started with PID='</span><span class="p">,</span> <span class="n">os</span><span class="p">.</span><span class="n">getpid</span><span class="p">())</span>
<span class="n">universal</span><span class="p">(</span><span class="n">manager</span><span class="p">.</span><span class="n">get_job_q</span><span class="p">(),</span> <span class="n">manager</span><span class="p">.</span><span class="n">get_res_q</span><span class="p">())</span></code></pre></figure>
<p>The <code class="language-plaintext highlighter-rouge">universal</code> function is the main driver. It takes the job - and result
queue as arguments. As stated above the job queue contains triplets with the
marshalled function code and the arguments. First the marshalled code is
unmarshalled giving us the pseudo-compiled code back. Next a new function is
created from this code, it is now part of the process and can be called.
Finally a tuple containing this processes id and the result of the invocation
is put into the result queue.</p>
<p>When the script is run, it joins the specified manager using <code class="language-plaintext highlighter-rouge">connect</code> and
then enters the <code class="language-plaintext highlighter-rouge">universal</code> function.</p>
<h4 id="boss">Boss</h4>
<p>Now this setup can be put to use for arbitrary tasks, for example here is a
boss that schedules the calculation of factorials.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">from</span> <span class="nn">manager</span> <span class="kn">import</span> <span class="n">remote</span><span class="p">,</span> <span class="n">connect</span>
<span class="kn">import</span> <span class="nn">os</span>
<span class="k">def</span> <span class="nf">factorial</span><span class="p">(</span><span class="n">n</span><span class="p">):</span>
<span class="n">res</span> <span class="o">=</span> <span class="mi">1</span>
<span class="k">while</span> <span class="n">n</span> <span class="o">></span> <span class="mi">1</span><span class="p">:</span>
<span class="n">res</span><span class="p">,</span> <span class="n">n</span> <span class="o">=</span> <span class="n">res</span> <span class="o">*</span> <span class="n">n</span><span class="p">,</span> <span class="n">n</span> <span class="o">-</span> <span class="mi">1</span>
<span class="k">return</span> <span class="n">res</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span><span class="p">:</span>
<span class="n">manager</span> <span class="o">=</span> <span class="n">connect</span><span class="p">(</span><span class="s">'127.0.0.1'</span><span class="p">,</span> <span class="mi">1234</span><span class="p">,</span> <span class="sa">b</span><span class="s">'abc'</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="s">'Factorial started with PID='</span><span class="p">,</span> <span class="n">os</span><span class="p">.</span><span class="n">getpid</span><span class="p">())</span>
<span class="n">inputs</span> <span class="o">=</span> <span class="nb">range</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">10</span><span class="p">)</span>
<span class="k">for</span> <span class="n">num</span> <span class="ow">in</span> <span class="n">inputs</span><span class="p">:</span>
<span class="n">remote</span><span class="p">(</span><span class="n">factorial</span><span class="p">,</span> <span class="n">manager</span><span class="p">)(</span><span class="n">num</span><span class="p">)</span>
<span class="k">for</span> <span class="n">_</span> <span class="ow">in</span> <span class="n">inputs</span><span class="p">:</span>
<span class="k">print</span><span class="p">(</span><span class="n">manager</span><span class="p">.</span><span class="n">get_res_q</span><span class="p">().</span><span class="n">get</span><span class="p">())</span></code></pre></figure>
<p>As can be seen the factorial function is a completely normal function. If the
boss is run, it again connects to a running manager and then uses the
<code class="language-plaintext highlighter-rouge">remote</code> decorator to schedule the factorial calculations to remote
processes.</p>
<p>This is arguably a pretty simple example, so here is something a bit cooler.
With this boss the workers will be turned into socket servers that can handle
requests on their own!</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">from</span> <span class="nn">manager</span> <span class="kn">import</span> <span class="n">connect</span><span class="p">,</span> <span class="n">remote</span>
<span class="kn">import</span> <span class="nn">os</span>
<span class="k">def</span> <span class="nf">server</span><span class="p">(</span><span class="n">port</span><span class="p">):</span>
<span class="kn">import</span> <span class="nn">socket</span>
<span class="n">host</span> <span class="o">=</span> <span class="s">''</span>
<span class="n">backlog</span> <span class="o">=</span> <span class="mi">5</span>
<span class="n">size</span> <span class="o">=</span> <span class="mi">1024</span>
<span class="n">s</span> <span class="o">=</span> <span class="n">socket</span><span class="p">.</span><span class="n">socket</span><span class="p">(</span><span class="n">socket</span><span class="p">.</span><span class="n">AF_INET</span><span class="p">,</span> <span class="n">socket</span><span class="p">.</span><span class="n">SOCK_STREAM</span><span class="p">)</span>
<span class="n">s</span><span class="p">.</span><span class="n">bind</span><span class="p">((</span><span class="n">host</span><span class="p">,</span><span class="n">port</span><span class="p">))</span>
<span class="n">s</span><span class="p">.</span><span class="n">listen</span><span class="p">(</span><span class="n">backlog</span><span class="p">)</span>
<span class="n">done</span> <span class="o">=</span> <span class="bp">False</span>
<span class="k">print</span><span class="p">(</span><span class="s">'start serving'</span><span class="p">)</span>
<span class="k">while</span> <span class="ow">not</span> <span class="n">done</span><span class="p">:</span>
<span class="n">client</span><span class="p">,</span> <span class="n">address</span> <span class="o">=</span> <span class="n">s</span><span class="p">.</span><span class="n">accept</span><span class="p">()</span>
<span class="n">data</span> <span class="o">=</span> <span class="n">client</span><span class="p">.</span><span class="n">recv</span><span class="p">(</span><span class="n">size</span><span class="p">)</span>
<span class="k">if</span> <span class="n">data</span><span class="p">:</span>
<span class="n">client</span><span class="p">.</span><span class="n">send</span><span class="p">(</span><span class="n">data</span><span class="p">)</span>
<span class="n">client</span><span class="p">.</span><span class="n">close</span><span class="p">()</span>
<span class="n">done</span> <span class="o">=</span> <span class="n">data</span><span class="p">.</span><span class="n">decode</span><span class="p">().</span><span class="n">strip</span><span class="p">()</span> <span class="o">==</span> <span class="s">'QUIT'</span>
<span class="k">print</span><span class="p">(</span><span class="s">'done serving'</span><span class="p">)</span>
<span class="k">return</span> <span class="n">port</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span><span class="p">:</span>
<span class="n">manager</span> <span class="o">=</span> <span class="n">connect</span><span class="p">(</span><span class="s">'127.0.0.1'</span><span class="p">,</span> <span class="mi">1234</span><span class="p">,</span> <span class="sa">b</span><span class="s">'abc'</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="s">'Echo started with PID='</span><span class="p">,</span> <span class="n">os</span><span class="p">.</span><span class="n">getpid</span><span class="p">())</span>
<span class="n">remote</span><span class="p">(</span><span class="n">server</span><span class="p">,</span> <span class="n">manager</span><span class="p">)(</span><span class="mi">5001</span><span class="p">)</span>
<span class="n">remote</span><span class="p">(</span><span class="n">server</span><span class="p">,</span> <span class="n">manager</span><span class="p">)(</span><span class="mi">5002</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="n">manager</span><span class="p">.</span><span class="n">get_res_q</span><span class="p">().</span><span class="n">get</span><span class="p">())</span>
<span class="k">print</span><span class="p">(</span><span class="n">manager</span><span class="p">.</span><span class="n">get_res_q</span><span class="p">().</span><span class="n">get</span><span class="p">())</span></code></pre></figure>
<p>Note: the import in the function is necessary, for it to be reconstructable in the remote process</p>
<p>All in all, this architecture is pretty usable. Using a fleet of these
universal servers you basically can build a simple cluster environment that can
compute anything. Not only that, it can be dynamically patched with arbitrary
new behavior without the need for a restart.</p>
<h2 id="conclusion">Conclusion</h2>
<p>As you can see, implementing the universal server in Python is quite a bit more
involved than it is in Erlang, but that is understandable because it wasn’t
specifically designed with this problem in mind. That being said, it is still
remarkable how simple this concept can be implemented using only standard
library modules. (Production code of course would be quite a bit more complex.)</p>
From Monoids to Monads in Python2014-12-09T00:00:00+00:00https://chjdev.com/2014/12/09/monoids-monads-python<p>Monads are a concept you will run into sooner or later when you learn about
functional programming. In the most general sense, Monads are a smart way to
structure a functional program and handle side effects. Although the math
concepts can be intricate, using them in your programs is quite straight
forward. One of the most trivial Monads for example will simply execute
functions in a specified order.</p>
<p><a href="/assets/2014/12/First_matryoshka_museum_doll_open.jpg"><img src="/assets/2014/12/First_matryoshka_museum_doll_open.jpg" alt="Monads and Monoids are kind of like a Matryoshka doll. Photo By: RK812, Doll carved by Zvezdochkin, painted by Malyutin (Sergiev Posad Museum of Toys, Russia) [Public domain], via Wikimedia Commons" /></a></p>
<p>Why is that interesting? Well, in purely functional programming languages the
concept of statements that get executed sequentially doesn’t really apply. Why
is that? A functional program basically consists of calling a function which is
composed of smaller functions, which are composed of smaller functions, and so
on. This makes sense because “pure” functions don’t have any side effects, e.g.
updating a global state, which means that it is guaranteed to return the same
result each time it is called with the same parameters. So calling different
functions in a sequence basically equates to running different programs, since
they have no way of influencing each other.</p>
<p>However, in real world use cases handling side effects is unavoidable, e.g. you
probably want to handle user input. Handling these side effects and still
complying with the basic structure of a functional program is where Monads come
in.</p>
<p>In this article we will build the concept of a Monad from the ground up using a
very pragmatic high level view, that focuses on the application instead of the
math. To cause less confusion we will use Python to implement the examples
because the syntax is probably the most familiar, the concepts however don’t
change in other languages. Monads are a general concept that can be implemented
in one way or another in any language.</p>
<h2 id="function-composition">Function Composition</h2>
<p>If you start reading about Monads you will probably see the word <a href="https://en.wikipedia.org/wiki/Monoid">Monoid</a> being thrown around. The
mathematical definition of a Monoid is: <em>an algebraic structure with a single
associative binary operation and an identity element</em>. It sounds complicated
but is pretty simple actually, an example would be the <code class="language-plaintext highlighter-rouge">+</code> operator: ``a + b</p>
<ul>
<li>c = (a + b) + c + 0 = (a + (b + c)) + 0 + 0<code class="language-plaintext highlighter-rouge">, with</code>0<code class="language-plaintext highlighter-rouge">being the identity element in
this specific case. Everything that behaves this way is a Monoid. Note:
although</code>+`` is <a href="https://en.wikipedia.org/wiki/Commutative_property">commutative</a> it is
not a requirement. The Monoid we will use is <a href="https://en.wikipedia.org/wiki/Function_composition">“Function
Composition.”</a></li>
</ul>
<p>Let’s work our way up. Let’s say the only things you can do in functional
programs is defining and calling functions. But how can you structure your
program like that? First lets define some functions to work with:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">def</span> <span class="nf">f</span><span class="p">(</span><span class="n">x</span><span class="p">:</span> <span class="nb">int</span><span class="p">)</span> <span class="o">-></span> <span class="nb">int</span><span class="p">:</span> <span class="k">return</span> <span class="o">~</span><span class="n">x</span>
<span class="k">def</span> <span class="nf">g</span><span class="p">(</span><span class="n">x</span><span class="p">:</span> <span class="nb">int</span><span class="p">)</span> <span class="o">-></span> <span class="nb">int</span><span class="p">:</span> <span class="k">return</span> <span class="n">x</span> <span class="o">**</span> <span class="mi">2</span>
<span class="k">def</span> <span class="nf">h</span><span class="p">(</span><span class="n">x</span><span class="p">:</span> <span class="nb">int</span><span class="p">)</span> <span class="o">-></span> <span class="nb">int</span><span class="p">:</span> <span class="k">return</span> <span class="mi">2</span> <span class="o">*</span> <span class="n">x</span></code></pre></figure>
<p>Note: I’m using <code class="language-plaintext highlighter-rouge">int</code> here but any types can be used as long as the
functions can be legally combined, more on that later.</p>
<p>The straight forward way to create a program from these functions is calling
the functions with the respective result of calling the other functions:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">x</span> <span class="o">=</span> <span class="mi">1</span>
<span class="k">assert</span> <span class="n">f</span><span class="p">(</span><span class="n">g</span><span class="p">(</span><span class="n">h</span><span class="p">(</span><span class="n">x</span><span class="p">)))</span> <span class="o">==</span> <span class="o">-</span><span class="mi">5</span>
<span class="k">assert</span> <span class="n">h</span><span class="p">(</span><span class="n">g</span><span class="p">(</span><span class="n">f</span><span class="p">(</span><span class="n">x</span><span class="p">)))</span> <span class="o">==</span> <span class="mi">8</span></code></pre></figure>
<p>In the first example we start by calling h with the variable x, then call g
with the result of h and finally call f with the result of g.</p>
<p>This notation however is not very flexible and quickly becomes
unwieldy. Function composition to the rescue! Let’s say we have two functions,
<code class="language-plaintext highlighter-rouge">f</code> and <code class="language-plaintext highlighter-rouge">g</code>, which we want to combine into one function <code class="language-plaintext highlighter-rouge">c</code>. The
mathematical notation for this composition is <code class="language-plaintext highlighter-rouge">g ∘ f</code>.</p>
<p><code class="language-plaintext highlighter-rouge">c(x) = (g ∘ f)(x) = g(f(x))</code></p>
<p>In Python this translates to the following:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">def</span> <span class="nf">combine</span><span class="p">(</span><span class="n">x</span><span class="p">:</span> <span class="s">'(int) -> int'</span><span class="p">,</span> <span class="n">y</span><span class="p">:</span> <span class="s">'(int) -> int'</span><span class="p">)</span> <span class="o">-></span> <span class="s">'(int) -> int'</span><span class="p">:</span>
<span class="k">return</span> <span class="k">lambda</span> <span class="n">_</span><span class="p">:</span> <span class="n">x</span><span class="p">(</span><span class="n">y</span><span class="p">(</span><span class="n">_</span><span class="p">))</span></code></pre></figure>
<p>Note: unfortunately Python doesn’t have a good way to type hint function
objects, so i use <code class="language-plaintext highlighter-rouge">(int) -> int</code> to say: a function that takes an <code class="language-plaintext highlighter-rouge">int</code> and
returns an <code class="language-plaintext highlighter-rouge">int</code>. Again it doesn’t have to be <code class="language-plaintext highlighter-rouge">int</code> and can be anything as
long as the types match up.</p>
<p>If you want to compose more than 2 functions, simply use <code class="language-plaintext highlighter-rouge">combine</code> repeatedly
with new functions and the result of a previous <code class="language-plaintext highlighter-rouge">combine</code>. Since Monoids
follow associativity rules the order in which they are combined doesn’t matter,
as long as the general order of operations is preserved.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">assert</span> <span class="n">combine</span><span class="p">(</span><span class="n">f</span><span class="p">,</span> <span class="n">combine</span><span class="p">(</span><span class="n">g</span><span class="p">,</span> <span class="n">h</span><span class="p">))(</span><span class="n">x</span><span class="p">)</span> <span class="o">==</span> \
<span class="n">combine</span><span class="p">(</span><span class="n">combine</span><span class="p">(</span><span class="n">f</span><span class="p">,</span> <span class="n">g</span><span class="p">),</span> <span class="n">h</span><span class="p">)(</span><span class="n">x</span><span class="p">)</span> <span class="o">==</span> <span class="o">-</span><span class="mi">5</span></code></pre></figure>
<p>Still pretty cumbersome to write but at least it is more general. Let’s take a
quick detour and use a cool Python hack which allows us to create <a href="http://code.activestate.com/recipes/384122/">infix operators</a>.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">ø</span> <span class="o">=</span> <span class="n">Infix</span><span class="p">(</span><span class="n">combine</span><span class="p">)</span>
<span class="k">assert</span> <span class="p">(</span><span class="n">f</span> <span class="o">|</span><span class="n">ø</span><span class="o">|</span> <span class="n">g</span> <span class="o">|</span><span class="n">ø</span><span class="o">|</span> <span class="n">h</span><span class="p">)(</span><span class="n">x</span><span class="p">)</span> <span class="o">!=</span> <span class="p">(</span><span class="n">h</span> <span class="o">|</span><span class="n">ø</span><span class="o">|</span> <span class="n">g</span> <span class="o">|</span><span class="n">ø</span><span class="o">|</span> <span class="n">f</span><span class="p">)(</span><span class="n">x</span><span class="p">)</span>
<span class="k">assert</span> <span class="p">(</span><span class="n">f</span> <span class="o">|</span><span class="n">ø</span><span class="o">|</span> <span class="n">g</span> <span class="o">|</span><span class="n">ø</span><span class="o">|</span> <span class="n">h</span><span class="p">)(</span><span class="n">x</span><span class="p">)</span> <span class="o">==</span> <span class="n">f</span><span class="p">(</span><span class="n">g</span><span class="p">(</span><span class="n">h</span><span class="p">(</span><span class="n">x</span><span class="p">)))</span></code></pre></figure>
<p>The <code class="language-plaintext highlighter-rouge">ø</code> symbol was chosen because it’s valid Python and resembles the mathematical notation.
If you squint, this almost looks like statements in an imperative language. You
start with an <code class="language-plaintext highlighter-rouge">x</code> then you do <code class="language-plaintext highlighter-rouge">h</code> followed by <code class="language-plaintext highlighter-rouge">g</code> followed by <code class="language-plaintext highlighter-rouge">f</code> and
arrive at a result.</p>
<p>Monads behave very similar to this concept. You probably already heard the
line: “A monad is just a monoid in the category of endofunctors, what’s the
problem?” We already know how Monoids work. Now however we will change the type
of the functions that will be used.</p>
<h2 id="down-the-rabbit-hole-monads">Down the rabbit hole: Monads</h2>
<p>Previously we used function of the form <code class="language-plaintext highlighter-rouge">(a) -> a</code> (for <code class="language-plaintext highlighter-rouge">a</code> meaning any
type). For Monads however we will now use functions of the form <code class="language-plaintext highlighter-rouge">(a) -> M a</code>.
The <code class="language-plaintext highlighter-rouge">M a</code> indicates a type constructor. Basically it’s just a wrapper that
wraps a type and needs to be evaluated to create (in this case) an <code class="language-plaintext highlighter-rouge">a</code>.</p>
<p>Our approach for function composition doesn’t work any more since we can’t
simply combine two <code class="language-plaintext highlighter-rouge">(a) -> M a</code> functions. E.g. if we change the type of our
functions <code class="language-plaintext highlighter-rouge">f</code> and <code class="language-plaintext highlighter-rouge">g</code> to use a type constructor, <code class="language-plaintext highlighter-rouge">g(f(x))</code> doesn’t work
since both functions expect an <code class="language-plaintext highlighter-rouge">a</code> as an argument, the result of <code class="language-plaintext highlighter-rouge">f</code>
however is an <code class="language-plaintext highlighter-rouge">M a</code>. It’s better to look at some real code to understand
this.</p>
<h3 id="the-trivial-monad">The Trivial Monad</h3>
<p>Let’s build the simplest Monad: Trivial. This Monad will simply chain functions
analog to our <code class="language-plaintext highlighter-rouge">combine</code> example. First let’s create our type constructor:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">class</span> <span class="nc">Trivial</span><span class="p">():</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">value</span><span class="p">):</span>
<span class="bp">self</span><span class="p">.</span><span class="n">_value</span> <span class="o">=</span> <span class="n">value</span>
<span class="k">def</span> <span class="nf">__call__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="p">.</span><span class="n">_value</span></code></pre></figure>
<p>This wrapper is pretty useless, it just wraps a value and spits it back out
when the object is called. However it is useful to have this abstraction for
more intricate Monads that allow e.g. I/O or concurrency. We’ll later see a
Monad that’s a bit more useful.</p>
<p>Alright, let’s create some functions, that are using this type constructor.
We’ll just use our previous functions and wrap them in the type constructor:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">def</span> <span class="nf">tf</span><span class="p">(</span><span class="n">x</span><span class="p">:</span> <span class="nb">int</span><span class="p">)</span> <span class="o">-></span> <span class="n">Trivial</span><span class="p">:</span> <span class="k">return</span> <span class="n">Trivial</span><span class="p">(</span><span class="n">f</span><span class="p">(</span><span class="n">x</span><span class="p">))</span>
<span class="k">def</span> <span class="nf">tg</span><span class="p">(</span><span class="n">x</span><span class="p">:</span> <span class="nb">int</span><span class="p">)</span> <span class="o">-></span> <span class="n">Trivial</span><span class="p">:</span> <span class="k">return</span> <span class="n">Trivial</span><span class="p">(</span><span class="n">g</span><span class="p">(</span><span class="n">x</span><span class="p">))</span>
<span class="k">def</span> <span class="nf">th</span><span class="p">(</span><span class="n">x</span><span class="p">:</span> <span class="nb">int</span><span class="p">)</span> <span class="o">-></span> <span class="n">Trivial</span><span class="p">:</span> <span class="k">return</span> <span class="n">Trivial</span><span class="p">(</span><span class="n">h</span><span class="p">(</span><span class="n">x</span><span class="p">))</span></code></pre></figure>
<p>It’s obvious now, that you can’t just compose the functions because you can’t
just call e.g. <code class="language-plaintext highlighter-rouge">tf</code> with a <code class="language-plaintext highlighter-rouge">Trivial</code> object. In order to compose these
functions we need a bit more plumbing. We need 2 things: a mechanism that
evaluates our type constructor and calls a function with the result, and a way
to chain multiple functions. This sounds a bit contrived, but will become
clearer once you see it in code.</p>
<p>First, the function to evaluate the type constructor and call a second function - we’ll call it <code class="language-plaintext highlighter-rouge">bind</code>:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">def</span> <span class="nf">bind</span><span class="p">(</span><span class="n">what</span><span class="p">:</span> <span class="n">Trivial</span><span class="p">,</span> <span class="n">other</span><span class="p">:</span> <span class="s">'(a) -> Trivial'</span><span class="p">)</span> <span class="o">-></span> <span class="n">Trivial</span><span class="p">:</span>
<span class="k">return</span> <span class="n">other</span><span class="p">(</span><span class="n">what</span><span class="p">())</span></code></pre></figure>
<p>This <code class="language-plaintext highlighter-rouge">bind</code> operator is different for each Monad, it depends on the type
constructor and what you want to do with it, e.g. you could parallelize the
execution. In this trivial example however, we just dig out the value and pass
it on to the function.</p>
<p>Now, how can we use this operator to compose multiple functions?</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">programA</span> <span class="o">=</span> <span class="p">(</span><span class="k">lambda</span> <span class="n">a</span><span class="p">:</span> <span class="n">bind</span><span class="p">(</span><span class="n">th</span><span class="p">(</span><span class="n">a</span><span class="p">),</span>
<span class="k">lambda</span> <span class="n">b</span><span class="p">:</span> <span class="n">bind</span><span class="p">(</span><span class="n">tg</span><span class="p">(</span><span class="n">b</span><span class="p">),</span>
<span class="k">lambda</span> <span class="n">c</span><span class="p">:</span> <span class="n">tf</span><span class="p">(</span><span class="n">c</span><span class="p">))))(</span><span class="n">x</span><span class="p">)</span></code></pre></figure>
<p>This looks a bit complicated but what it does is actually simple, it just
creates some intermediary variables to pass the input through all the functions
that will be bound. This probably will take a little time to grok, but just try
to run through it a couple of times and it should become clearer.</p>
<p>What have we gained with this mechanism? Well we can now create arbitrary
Monads that allow a natural structuring of a functional program and the
introduction of side effects.</p>
<p>For fun, let’s add some syntactic sugar:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">def</span> <span class="nf">_do_block</span><span class="p">(</span><span class="n">bind</span><span class="p">,</span> <span class="o">*</span><span class="n">stmts</span><span class="p">):</span>
<span class="k">assert</span> <span class="nb">len</span><span class="p">(</span><span class="n">stmts</span><span class="p">)</span> <span class="o">></span> <span class="mi">0</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">stmts</span><span class="p">)</span> <span class="o">==</span> <span class="mi">1</span><span class="p">:</span>
<span class="k">return</span> <span class="k">lambda</span> <span class="n">y</span><span class="p">:</span> <span class="n">stmts</span><span class="p">[</span><span class="mi">0</span><span class="p">](</span><span class="n">y</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">return</span> <span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">bind</span><span class="p">(</span><span class="n">stmts</span><span class="p">[</span><span class="mi">0</span><span class="p">](</span><span class="n">x</span><span class="p">),</span>
<span class="n">do_block</span><span class="p">(</span><span class="n">bind</span><span class="p">,</span> <span class="o">*</span><span class="n">stmts</span><span class="p">[</span><span class="mi">1</span><span class="p">:]))</span>
<span class="k">def</span> <span class="nf">trivial</span><span class="p">(</span><span class="o">*</span><span class="n">stmts</span><span class="p">):</span> <span class="k">return</span> <span class="n">_do_block</span><span class="p">(</span><span class="n">bind</span><span class="p">,</span> <span class="o">*</span><span class="n">stmts</span><span class="p">)</span>
<span class="n">programB</span> <span class="o">=</span> <span class="n">trivial</span><span class="p">(</span>
<span class="n">th</span><span class="p">,</span>
<span class="n">tg</span><span class="p">,</span>
<span class="n">tf</span><span class="p">)(</span><span class="n">x</span><span class="p">)</span>
<span class="k">assert</span> <span class="n">programA</span><span class="p">()</span> <span class="o">==</span> <span class="n">programB</span><span class="p">()</span></code></pre></figure>
<p>There, looks way better, doesn’t it?</p>
<h2 id="a-variation-of-the-maybe-monad">A Variation of the Maybe Monad</h2>
<p>To finish off, we’ll create a Monad that is actually useful. This is a
variation of the <a href="http://hackage.haskell.org/package/base-4.7.0.1/docs/Data-Maybe.html">Maybe
Monad</a> which is found in pretty much all functional languages in one form or
another.</p>
<p>Our Monad will allow us to work with functions that may fail to calculate a
result, and if they do, stop the remaining computation immediately. Let’s look
at the code:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">class</span> <span class="nc">Maybe</span><span class="p">():</span> <span class="k">pass</span>
<span class="k">class</span> <span class="nc">Just</span><span class="p">(</span><span class="n">Maybe</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">value</span><span class="p">):</span>
<span class="bp">self</span><span class="p">.</span><span class="n">_value</span> <span class="o">=</span> <span class="n">value</span>
<span class="k">def</span> <span class="nf">__call__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="p">.</span><span class="n">_value</span>
<span class="k">class</span> <span class="nc">Nothing</span><span class="p">(</span><span class="n">Maybe</span><span class="p">):</span> <span class="k">pass</span>
<span class="o">@</span><span class="n">singledispatch</span>
<span class="k">def</span> <span class="nf">bind</span><span class="p">(</span><span class="n">what</span><span class="p">:</span> <span class="n">Just</span><span class="p">,</span> <span class="n">other</span><span class="p">:</span> <span class="s">'def f(a) -> Trivial: pass'</span><span class="p">)</span> <span class="o">-></span> <span class="n">Maybe</span><span class="p">:</span>
<span class="k">return</span> <span class="n">other</span><span class="p">(</span><span class="n">what</span><span class="p">())</span>
<span class="o">@</span><span class="n">bind</span><span class="p">.</span><span class="n">register</span><span class="p">(</span><span class="n">Nothing</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">bind_nothing</span><span class="p">(</span><span class="n">what</span><span class="p">:</span> <span class="n">Nothing</span><span class="p">,</span> <span class="n">_</span><span class="p">)</span> <span class="o">-></span> <span class="n">Nothing</span><span class="p">:</span>
<span class="k">return</span> <span class="n">what</span>
<span class="k">def</span> <span class="nf">maybe</span><span class="p">(</span><span class="o">*</span><span class="n">stmts</span><span class="p">):</span> <span class="k">return</span> <span class="n">_do_block</span><span class="p">(</span><span class="n">bind</span><span class="p">,</span> <span class="o">*</span><span class="n">stmts</span><span class="p">)</span></code></pre></figure>
<p>Alright, everything in place: a type constructor, a <code class="language-plaintext highlighter-rouge">bind</code> operator and some syntactic sugar.
The type constructor <code class="language-plaintext highlighter-rouge">Maybe</code> can either be a <code class="language-plaintext highlighter-rouge">Just</code> object which wraps a
value, or a <code class="language-plaintext highlighter-rouge">Nothing</code> object.
The bind method for this Monad will either pass on the wrapped value or
immediately return <code class="language-plaintext highlighter-rouge">what</code> (i.e. the <code class="language-plaintext highlighter-rouge">Nothing</code> object).</p>
<p>Now let’s use this Monad. We will create a program that checks for different
conditions and print the result.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">def</span> <span class="nf">mfun1</span><span class="p">(</span><span class="n">x</span><span class="p">:</span> <span class="nb">int</span><span class="p">)</span> <span class="o">-></span> <span class="n">Maybe</span><span class="p">:</span>
<span class="k">print</span><span class="p">(</span><span class="s">'mfun1'</span><span class="p">,</span> <span class="n">x</span><span class="p">)</span>
<span class="k">return</span> <span class="n">Just</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="k">if</span> <span class="p">(</span><span class="n">x</span> <span class="o">%</span> <span class="mi">2</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span> <span class="k">else</span> <span class="n">Nothing</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">mfun2</span><span class="p">(</span><span class="n">x</span><span class="p">:</span> <span class="nb">int</span><span class="p">)</span> <span class="o">-></span> <span class="n">Maybe</span><span class="p">:</span>
<span class="k">print</span><span class="p">(</span><span class="s">'mfun2'</span><span class="p">,</span> <span class="n">x</span><span class="p">)</span>
<span class="k">return</span> <span class="n">Just</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="k">if</span> <span class="p">(</span><span class="n">x</span> <span class="o">%</span> <span class="mi">3</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span> <span class="k">else</span> <span class="n">Nothing</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">mfun3</span><span class="p">(</span><span class="n">x</span><span class="p">:</span> <span class="nb">int</span><span class="p">)</span> <span class="o">-></span> <span class="n">Maybe</span><span class="p">:</span>
<span class="k">print</span><span class="p">(</span><span class="s">'mfun3'</span><span class="p">,</span> <span class="n">x</span><span class="p">)</span>
<span class="k">return</span> <span class="n">Just</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="k">if</span> <span class="p">(</span><span class="n">x</span> <span class="o">%</span> <span class="mi">4</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span> <span class="k">else</span> <span class="n">Nothing</span><span class="p">()</span>
<span class="n">programC</span> <span class="o">=</span> <span class="n">maybe</span><span class="p">(</span>
<span class="n">mfun1</span><span class="p">,</span>
<span class="n">mfun2</span><span class="p">,</span>
<span class="n">mfun3</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="n">programC</span><span class="p">(</span><span class="mi">12</span><span class="p">)())</span>
<span class="k">try</span><span class="p">:</span>
<span class="k">print</span><span class="p">(</span><span class="n">programC</span><span class="p">(</span><span class="mi">11</span><span class="p">)())</span>
<span class="k">except</span><span class="p">:</span>
<span class="k">print</span><span class="p">(</span><span class="s">'as expected'</span><span class="p">)</span></code></pre></figure>
<p>The first example will work since 12 is a valid input for all functions. The
second example will fail immediately on the first function, since 11 is not
even and the attempt of evaluating <code class="language-plaintext highlighter-rouge">Nothing</code> will raise an <code class="language-plaintext highlighter-rouge">Exception</code>.</p>
<p>This concludes our little foray into Monads. I hope you now have an overview
about what people are talking about when they mention Monads and see the
usefulness of the concept.</p>