endafarrell.net

I've been looking at how Pelican works, and was working with many different themes, and of course, after a while, it became annoying to have to make repeated calls to see what the content looks like in different themes. So I put this together:

for theme in $(
    find themes/pelican-themes \
        -type d \
        -not -empty \
        -depth 1 \
    | grep -v ".git/" \
    | cut -f3 -d/ \
    | sort
)
do 
    printf "${theme}: "
    pelican content \
        -o output/${theme} \
        -s pelicanconf.py \
        -t themes/pelican-themes/${theme}
done 2>&1 | tee log

Mermaid is a very useful way of writing a form of Markdown that defines graphs - like this:

    graph TD
    A[Christmas] -->|Get money| B(Go shopping)
    B --> C{Let me think}
    C -->|One| D[Laptop]
    C -->|Two| E[iPhone]
    C -->|Three| F[fa:fa-car Car]

After you run a document with the above through Mermaid, you'll get an image like this:

Being able to draw simple diagrams like this is clearly very useful when documenting software systems and data flows. I often use Sphinx when writing docs and it comes with a Makefile so that you can simply make docs …

There are good instructions on how to tether an Edison to Bluetooth in the docs but I've found them a little difficult to follow, and I'm not certain that it worked "as is" for me, so here's my version.

-2: Comment-out the local WiFi from /etc/wpa_supplicant/wpa_supplicant.conf. The rig prefers WiFi over BT - a decision I very much agree with - but you'll need to ensure that you don't have any WiFi going on for a while until you get this sorted out.

-1: bluetoothd --version If you don't have the correct bluetoothd you're going to be in trouble …

I use git via the terminal and not through UI tools (though I do use Sourcetree to understands the flow of commits), and I recently found a very helpful change for my prompt:

It's turned out to be really helpful and not difficult to set up. All of the changes are in my .profile:

COLOR_RED="\033[0;31m"
COLOR_YELLOW="\033[0;33m"
COLOR_GREEN="\033[0;32m"
COLOR_OCHRE="\033[38;5;95m"
COLOR_BLUE="\033[0;34m"
COLOR_WHITE="\033[0;37m"
COLOR_RESET="\033[0m"

function parse_git_branch {
     git branch 2> /dev/null | sed -e '/^[^*]/d' -e 's/* \(.*\)/\1/'
}

function git_color {
  local git_status="$(git …

At work, we use gerrit. It's different to git and to github, and we didn't get the right kind of training on howto use it. This is a self-assembled survival guide. If you can articulate the differences between a merge and a rebase then this is not for you!

Initial checkout

Your initial checkout should look something like this:

git clone ssh://you@gerrit.hostname:29418/project/sub-project && scp -p -P 29418 you@gerrit.hostname:hooks/commit-msg sub-project/.git/hooks/

You should then find yourself on the master branch.

You want to do some work

Create a local branch to …

Clearly, the correct way to deal with dates in Java is with Joda, but I've been looking at a project which doesn't have Joda installed already, and I haven't earned the right to rearrange that project's dependencies and build patterns. It's also a project that needs to run on lower-specced machines (I'm used to writing server code, but this is destined for things like watches), so there are likely to be reasons I don't understand why external modules will not be welcome.

Here's my current thinking:

import javafx.util.Pair;  // <<<- this woould be replaced with the Pair in the Android …

At work, we're in the process of moving datacentres and some of my (extensive) monitoring needs to be updated. In particular, I have a monitor on Hadoop jobs which looks for job names of a specific pattern and posts them to a TeamUp calendar so that we can see what's running where. Over the weekend, the old datacentre's Hadoop job-trackers were shut down - now my monitor is complaining that it can't connect.

The datacentre migration does not have a central documentation location showing the URLs of old and new (for reasons I cannot understand, the ops people decided to change …

You're not going to want to do this - but if you have to do it ...

I have a job that needs 200+ GB of disk to run. I have build servers which do have that kind of space, but as they are shared servers, other jobs can take some of that space, and occasionally I have jobs which run into the "No space left on device" problem.

Like with all build servers, it's easy to add new jobs when you need them - but we humans are not good at removing old jobs - and more importantly their old, unused workspaces - when …

I've created an new PGP key which you're most welcome to use. It's on the public PGP key servers, but it's also here, linked at the bottom of each page on this site, and below. It's only a 2k key, but should do for starters.

It's long for a 2k key as it also embeds an image - it turns out that there are at least 3 different "Enda Farrell"s with keys - you'll be wanting to know which is which.

-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG/MacGPG2 v2.0.30
Comment: GPGTools - http://gpgtools.org

mQENBFmhSDYBCADfCifYS+K76Qf9vHkGvY11meNeOwOHhQ3zERG6HJRGULznF4EM
+J4Gy1QrSWvQIShzsVfH2R6G6GpEukj4QvCG0btCj/gBOtKKe7aNYbW12g1RMvhu …

It's been a year since my original Getting started with Pelican post and clearly I didn't write at all - why not?

Part of it was that I'd little interesting to say, but part was that I had trouble with how I had Pelican set up and with how I published to github. I'd started using plugins - a good thing - but I foolishly started to customise them in their installed location and not in a source-code-managed local subdirectory. If I was only ever going to publish from that one machine then I'd have gotten away with my foolishness, but I wanted …

From https://www.grc.com/sn/sn-616.htm:

PADRE: That's why there's only two drives, two types of drives I use in my NASes right now. One is Western Digital Red because that's got the TLER control, so you can essentially say, don't figure it out, the NAS does everything, just let the NAS do it.

Steve: Time Limited Error Recovery.

PADRE: Right. And the second one is Seagate's IronWolf series, which is often because I've got a bunch of 8TB drives. They also run cool. I think they will do TLER reconfiguration, as well. But, yeah, I've had so …

I had an itch to be able to easily find the Facebook ID for Place pages that I was visiting. You can - usually - find the ID that the Facebook Graph API uses for a Place page by looking into the HTML source of the Facebook Place page and recognising some patterns. It was a slow business, so I wrote this instead.

Due to my not really knowing how to set things up properly, the installation says that:

It can:

• Read and change all your data on all facebook.com site
• Read your browsing history

And I guess it could, based …

There's a lot of crap advice about getting jupyter notebooks to play nicely with pyspark. I guess things have changed a lot over the last couple of years, but here's how I have things.

I use conda for my python envs, but I doubt that matters here.

SPARK_HOME=/workspace/pyspark-games/spark-1.5.0-bin-hadoop2.4 \
PATH=$SPARK_HOME/bin:$PATH \
PYTHONPATH=$SPARK_HOME/python/:$SPARK_HOME/python/lib/py4j-0.8.2.1-src.zip:$PYTHONPATH \
IPYTHON_OPTS="notebook" \
$SPARK_HOME/bin/pyspark

With this approach I don't need/want to play with Jupyter profiles and, it may (now) be unsupported with Jupyter. This way allows me to …

Getting started with Pelican since I'd quite like to have somewhere that I can refer to. Why not use the automatic pages from GitHub? I don't know really - other than I'd quite like to do it myself and who know's how long GitHub pages will be here?

This is now my fourth "blogging" platform and this time I'd quite like to have my own content here to hand when it's time to migrate to the fifth.

So: knowing I didn't want a platform tie-in, and I wanted a static site generator, why not use Jekyll? I simply couldn't get it …

We take quite a few photos and I'm forever reinventing how I look after them. Here's where I am at at the moment:

• Photos come from a few different cameras
• Aperture (now 3.2.4) is what we use to manage them
• All photos/videos are imported
• Good ones get a 3 start rating, very good a 4 and the best a 5.
• Rejected, one and two star photos are deleted - unrated get to live for another day
• I export full size JPGs to my ~/Pictures/ApertureExports into subdirectories for each year.
• After exporting I use (the very excellent) exiftool …

I've been processing log files recently to see how a live system is being used. When you have millions of hits daily, you need these processors to be fast.

Today the best way is to have your log files shipped over onto a Hadoop cluster and map-reduce them: but sometimes there's still a call for quick scripts. Here awk is a king :-)

Converting to a HTML table

I couldn't find a decent online version of converting a CSV into a HTML table - but with awk it's easy. Here's the awk script:

BEGIN {
    print "<table>" HDR;
} { 
    printf "<tr>";
    for (i=1 …

One really quite useful command is “grep -o” - it allows you to fire off something like this:

grep -o "Location supplier=\"\w*\"" locations.xml

And the output will be the phrases matching the regular expressions that start with “Location supplier=” with a double quoted word (\w) in the locations.xml file. :-)

Thanks to http://www.contentwithstyle.co.uk/content/4-ssh-config-tips-for-faster-remote-working/ I can avoid creating files which need to be scp’d: I can pipe the content directly:

local$ cat localfile.txt | ssh remote "cat - >> remotefile.txt"

For this to work, I have already got “remote” set up in my ~/.ssh/config. You can get much more involved in this: see

• http://backdrift.org/transparent-proxy-with-ssh
• http://www.cyberciti.biz/faq/howto-use-tar-command-through-network-over-ssh-session/
• http://scratching.psybermonkey.net/2011/02/ssh-how-to-pipe-output-from-local-to.html

My favourite example from the last site (execute commands on remote server but save the output to local) is:

ssh user@example.com …

Here’s another view of the Places in the Nokia Places Registry.

Each pixel is the location of one or more points of interest that we have as of the end of June 2011. Under each pixel there may only be one place or there may be thousands - as is the case in Beijing, Shanghai and São Paolo.

There’s a black & white version black & white version too.

Please feel free to use this image - but it would be nice if you acknowledge the source :-) And speaking of acknowledgement, this image comes with a large piece of inspiration from Erik …

Here’s a little visualisation of what I am up to at Nokia. You see a KML representation of the Points of Interest we currently have - the taller the tower the more active places we have in that spatial area.

I am about to change some of my insulin settings, but before I do I thought I’d write them here.

Below is a graph of how things are currently. As I am changing things you can deduce that the regime is not correct, but it isn’t too far off (I think). The four different lines are (from the bottom)

• The thick blue columns are the hourly dose in 100iU units (scale on the left in U/hr). The early morning spike is an attempt to handle my “dawn phenomenon” but it’s not quite working as I’d …

From listening to The Changelog for the first time today I should play with Yahoo’s YQL. It’s a server developer tool that allows you to query the web in a SQL-like format - and have the results come back in JSON. Where it gets really cool is when you mash-up different sites - then things become really useful.

We have recently started to use TestNG and I like it. One thing that is needed is runtime control over whether or not to run a group of tests.

We have both unit tests and integration tests to assure ourselves of the system’s health. Running the integration tests requires running databases (CouchDB in this case) but we don’t - by policy - have databases running on the continuous integration Hudson server. There are of course other approaches to dealing with this, but we have found this works well for us.

Oh - and we need to have just one POM. So …

I keep loosing these links, so I thought I’d spend a minute and gather them here:

• http://jsonformatter.curiousconcept.com/ - really excellent
• http://chris.photobooks.com/json/default.htm
• http://www.thomasfrank.se/downloadableJS/JSONeditor_example.html

These sites all do something a wee bit different from each other, and at different times I have used them all.

Update 2016-06-25 12:00: Over the last 6 years a different site has become my go-to for getting a better look at my content:

• http://jsbeautifier.org

I'd be surprised if this site is news to anyone, but just in case ;-)