Note to self: sharing the best photos

We take quite a few photos and I'm forever reinventing how I look after them. Here's where I am at at the moment:

• Photos come from a few different cameras
• Aperture (now 3.2.4) is what we use to manage them
• All photos/videos are imported
• Good ones get a 3 start rating, very good a 4 and the best a 5.
• Rejected, one and two star photos are deleted - unrated get to live for another day
• I export full size JPGs to my ~/Pictures/ApertureExports into subdirectories for each year.
• After exporting I use (the very excellent) exiftool to change the ctime of the file to be the photos's Original Datetime: (see the code block below)
• I have a ReadyNAS server for backups.
• I mount the NAS and rsync the exports: (see the code block below)

exiftool "-filemodifydate<datetimeoriginal" .

rsync -rv --delete --progress --times \
    /Users/Shared/ApertureExports/ \
    /Volumes/nasmedia/Pictures/BestApertureExports/

Gawking out

I've been processing log files recently to see how a live system is being used. When you have millions of hits daily, you need these processors to be fast.

Today the best way is to have your log files shipped over onto a Hadoop cluster and map-reduce them: but sometimes there's still a call for quick scripts. Here awk is a king :-)

Converting to a HTML table

I couldn't find a decent online version of converting a CSV into a HTML table - but with awk it's easy. Here's the awk script:

BEGIN {
	print "<table>" HDR;
} { 
	printf "<tr>";
	for (i=1; i <= NF; i++) {
		printf "<td>%s</td>", $i;
	}
	print "</tr>";
} END {
	print "</table>";
}

This makes use of the field seperator handling of awk: so a CSV can be converted into a simple HTML table like this:

cat stats.csv |\
awk -F',' -f 2Table.awk \
--assign HDR="<tr><th>A>/th>...</tr>" > stats.html

Summing distinct rows

So the best way to describe this is to show an example:

IP1 GET url1 200 32
IP2 GET url1 200 29
IP1 GET url1 200 21

The "distinct rows" here would be "IP1 GET url1 200" and "IP2 GET url1 200". You could think of these as just a few entries of the access_logs. Anyway, I wanted to sum together the times that these calls were taking (the last column). Sometimes the number of fields varied so a general solution was needed.

The desired output is:

IP1 GET url1 200 53
IP2 GET url1 200 29

and the script to do this is:

{
	# Grab the value
	num = 0 + $NF; 
	# Strip the value - $0 no longer has the value
	NF--;
	# Increments the value at hash[$0] or creates it
	# if not already present.
	hash[$0] = hash[$0] + num
}
END {
	for (ndx in hash) {
		print ndx,hash[ndx];
	}
}

N900 development using OSX

Tuesday, 8 May

Now using the Qt SDK v1.2.1 on a Mac. I don’t yet know if it will work well as a way of developing on an N900 - but I now know that I have to use Wi-Fi and not the USB in order to connect.

Monday, 7 May

This post is likely to be updated quite a few times …

1. See the earlier post about getting a USB connection
2. I’m now using Ubuntu 12.04 LTR desktop 32-bit 
3. I’m following wiki.maemo.org instructions 
4. On my Linux guest, I needed to add a new apt source: this is so far undocumented and then (and only then) does an apt update work.

sudo echo "\ndeb http://scratchbox.org/debian hathor main" >> /etc/apt/sources.list

Host IP when server brought up in a virtual host

I often need to bring up servers within my dev laptio, and use both VMWare Fusion and VirtualBox. One thing which I nice to have is the guest server’s IP address on screen shortly after startup so that I can ssh into it.

There’s two parts to this:

1. a script to show the IP information
2. an entry to call this script when the server (the guest) starts

the script

#/bin/sh
# Network interfaces can take a while to come up, esp with DHCP
sleep 5

# What is our current IP address?
CURRENT_IP=$(/sbin/ifconfig | \
  grep "inet addr" | \
  grep -v "addr:127.0.0.1" | \
  awk '{print $2}' | \
  cut -f2 -d':')

# If the current IP address is empty, we need to set up networking
if [ "x$CURRENT_IP" == "x" ]; then
  echo; echo "Networking is NOT configured! " >> /dev/tty1
  echo; echo "run \"system-config-network\" straight-away! " >> /dev/tty1
  echo; echo "then run \"service network restart\" to enable en0 " >> //dev/tty1
else
  OLD_IP=$(cat /root/.ip_info.dat 2>/dev/null)
  if [ "x$CURRENT_IP" != "x$OLD_IP" ]; then
    echo; echo "Update your ~/.ssh/config " >> /dev/tty1
    echo $CURRENT_IP > /root/.ip_info.dat
  fi
  /sbin/ifconfig | grep "inet addr" >> /dev/tty1
fi

I saved this as /root/.ip_info.sh and made it executable. Note that it’s Unixy.

Entry to call the script

If you want this on server start, then add the following line to /etc/rc.local /root/.ip_info.sh &

ssh into a Nokia N900 from a Mac over USB

I wanted to play with neasuring mobile network latency, and the Nokia N900 being a phone based on Debian is a great tool for this. As getting this to work as (a) not as simple as “plug it in” as is often the case on a Linux box and (b) a seemingly strange thing to want to do, I thought I’d write it up here.

First up: I needed to SSH from a Mac OSX Lion. It seems that older version of OSX allowed USB interfaces a lot more easily. In System Preferences, Netowrk I cannot create a USB interface (there isn’t an option) and I don’t want to install yet more new utilities as I have enough already thank you very much.

Second up: I am SSHing into a Nokia N900 as I happen to have one. There are a lot of articles about sshing into an iPhone and other devices, but the OS capabilities of the N900 is what I want to use.

Third up: I am sshing over teh USB cable: not over wifi. I will be playing with the interfaces - 3G and wireless - and I want my full keyboard. SSHing in over wifi is straight-forward and if you’re reading this page I’m going to assume you’ve already done that ;-) If not, check out scratching.psybermonkey.net’s how-to - please read the part about changing the default user password as you will that later.

the bad news

OSX: I couldn’t get the Mac to give me a USB interface over which to SSH. I probably wasted 2 hours trying to get OSX Lion to give me a USB networking interface and I couldn’t manage it.

Virtual Box: I couldn’t get the N900 to connect at all to a CentOS 5.6 instance in Virtual Box (yes, I know it’s an old CentOS, but it wasn’t a CentOS issue) due to this error:

Failed to attach the USB device Nokia N900 (PC-Suite Mode) [0211] to the virtual machine CentOS56.

USB device ‘Nokia N900 (PC-Suite Mode)’ with UUID {856379de-9fd3-4682-a9d8-7ec4665cd26b} is in use by someone else.

Result Code:	NS_ERROR_FAILURE (0x80004005)
Component:	HostUSBDevice
Interface:	IHostUSBDevice {173b4b44-d268-4334-a00d-b6521c9a740a}
Callee:	IConsole {1968b7d3-e3bf-4ceb-99e0-cb7c913317bb}

However, for other reasons, I also have VMware Fusion and that was able to connect.

the good news

• Fire-up the N900
• Fire-up (any) unix - in my case  a CentOS 5.7 within VMWare Fusion (Ubunto 12.04LTS works exactly the same too)
• Read http://wiki.maemo.org/N900_USB_networking
• Connect the N900 to the Mac USB port (it doesn’t seem to matter which) and connect as “PC Suite Mode”
• VMWare Fusion | Virtual Machine | USB | Connect Nokia N900 (PC Suite mode)
• Over on the guest CentOS you’ll see something like the following -which doesn’t matter as I am not trying to use the N900 as a modem for the laptop :-): 
  

  drivers/usb/class/cdc-acm.c: This device cannot do calls on its own. It is no modem.

• On the N900, you need to do the following:
  

  sudo gainroot
  ifup usb0

• On the guest unix server, become root
• ifconfig usb0 192.168.2.14 
• ssh user@192.168.2.15

And if you’ve followed the instructions and first changed the default user password on the N900 as shown in the scratching.psybermonkey.net article, you’re in :-)

vim and too many spaces

When pasting code into vim you can often get some really annoying spacing.

This first command re-indents the file as vim thinks it should be indented. Of course you gotta teach vim the file type you’re working with but that’s another story.

<esc>gg=G

This one deals with repteated (defined here are two or more) spaces characters within a line by reducing them to one for the whole file

:%s/\(\s\)\{2,\}/\1/g

To stop the problem happening in the first place, you may want to consider using ":noai" (no auto indenting). Some more info can be found here.

Nokia Lumia 800

Be careful! This starts the Diagnostic Field Test and makes the “Diagnostics” app visible too for next time.

##634#

Coder's Toolbox

Here’s a nice handy site: coderstoolbox.net

It convers:

• Time
  • Local vs UTC
  • Unix, ISO 8601, RFC 2822
• Strings
  • Base64, XML, URL, ECMAScript, Character set
  • Encode, decode
  • Output: US-ASCII, ISO-8859-1, UTF-8
• Numbers
  • Decimal, Hex, Octal, Binary
• Networks
• Bandwidth
• XPath (beta)

"grep -o" - really quite useful

One really quite useful command is “grep -o” - it allows you to fire off something like this:

grep -o "Location supplier=\"\w*\"" locations.xml

And the output will be the phrases matching the regular expressions that start with “Location supplier=” with a double quoted word (\w) in the locations.xml file. :-) 

Piping content through SSH

Thanks to http://www.contentwithstyle.co.uk/content/4-ssh-config-tips-for-faster-remote-working/ I can avoid creating files which need to be scp’d: I can pipe the content directly:

local$ cat localfile.txt | ssh remote "cat - >> remotefile.txt"

For this to work, I have already got “remote” set up in my ~/.ssh/config. You can get much more involved in this: see

• http://backdrift.org/transparent-proxy-with-ssh
• http://www.cyberciti.biz/faq/howto-use-tar-command-through-network-over-ssh-session/
• http://scratching.psybermonkey.net/2011/02/ssh-how-to-pipe-output-from-local-to.html

My favourite example from the last site (execute commands on remote server but save the output to local) is:

ssh user@example.com "mysqldump -u DB_username -pDB_password DB_name | gzip -c" > /local/dir/DB_backup.gz

N8 + micro USB cable + APN + Mac OSX = mobile connectivity

The Nokia N8 has a great mobile receiving radio which helps make it a great mobile modem. Here’s what’s needed when connecting a Mac with thanks to flixispy for details:

• N8 phone with SIM card and a really good data plan
• micro-USB cable to connect the phone to the Mac
• Watch for the mac to bring up the “Network connectivity configuration” screens to configure the phone as a modem (which should be in a Nokia/Ovi Suite mode - not the “modem” mode!)
• Confuguration: Default
• Telephone Number: <your-phone-number>
• Account Name: internet.t-d1.de
• Password: t-mobile
• Click Advanced | Modem
• Vendor: Nokia
• Model: GPRS (GSM/3G)
• APN: internet.t-mobile
• CID: 1
• Click the DNS tab
• Under the DNS Servers panel add 193.254.160.1
• Everythings else can be left alone, so click OK
• Click on Connect and enjoy!

This article has been written using this connection: and it’s not bad at all :-)

 Update: from theunlockr.com/internet-mms-settings/ we have more details:

T-Mobile - Germany

• 3G Frequency – 1900mhz/2100mhz
• Name: T-Mobile
  APN: internet.t-mobile
  Proxy:
  Port:
  Username: t-mobile
  Password: tm
  Server:
  MMSC: http://mms.t-mobile.de/servlets/mms/
  MMS Proxy: 172.28.23.131
  MMS Port: 8008
  MCC:
  MNC:
  Authentication Type:
  APN Type:
  APN Protocol:
  APN Roaming Protocol:

Read more at http://theunlockr.com/internet-mms-settings/#VV6QCtYXR3cOE3bi.99

pip - installing python packages

I’ve found that pip is nice when installing python packages: and this discussion here is close to what I think about it too.

A Places Registry view of the world

Here’s another view of the Places in the Nokia Places Registry.

Each pixel is the location of one or more points of interest that we have as of the end of June 2011. Under each pixel there may only be one place or there may be thousands - as is the case in Beijing, Shanghai and São Paolo.

There’s a black & white version too.

Please feel free to use this image - but it would be nice if you acknowledge the source :-) And speaking of acknowledgement, this image comes with a large piece of inspiration from Erik Fischer (@enf)’s brilliant “See something or say something” Flickr set showing Tweets and Flickr images - especially how these hi-res images need custom programs to render them.

Catalina logs - time of error

Often, expecially when you are monitoring, you need to have the date/time of an error message appear on the same line as the exception itself. Some log files give you this easily - most notably the “normal” access logs, but in the case of many installations of Tomact, the logs are written like this:

Jul 14, 2011 12:46:26 AM org.apache.solr.common.SolrException log
SEVERE: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@/data/solr/index/lucene-0a8b0319342c625714c04d6e6fcc50e8-write.lock

...

Jul 14, 2011 12:46:26 AM org.apache.solr.update.SolrIndexWriter finalize
SEVERE: SolrIndexWriter was not closed prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!
...

As the date/time is on the preceding line (see the line numbers), we would have to make “iteresting” changes to many log processors. Of course you could ask your devs to change the log configuration of the app - but this example may side step the problem (and open up your old logs to processing too):

cat catalina.2011-07-14.log | grep -B 1 SEVERE | grep -v '^--$' | sed -e 'N;s/\n/ /'

The output from here is:

Jul 14, 2011 2:17:44 AM org.apache.solr.common.SolrException log SEVERE: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@/data/solr/index/lucene-0a8b0319342c625714c04d6e6fcc50e8-write.lock
Jul 14, 2011 2:17:45 AM org.apache.solr.common.SolrException log SEVERE: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@/data/solr/index/lucene-0a8b0319342c625714c04d6e6fcc50e8-write.lock
Jul 14, 2011 2:17:45 AM org.apache.solr.update.SolrIndexWriter finalize SEVERE: SolrIndexWriter was not closed prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!

You can see by the line numbers that this is easier to process later.

MySQL date difference

MySQL does weird things with dates and with date maths. I think it’s trying to be helpful, but it took ages to find out how to get the number of seconds between two date-times reliably, but in the end I got this:

SELECT
    timestampdiff(SECOND, max(dateLastUpdated), UTC_TIMESTAMP())
FROM 
    Entity;

rsync

The simpliest rsync ever:

rsync --progress -avz -e/usr/bin/ssh SOURCE:production-logs/ /Volumes/My Book/production logs/

Nokia Place Registry visualisation

Here’s a little visualisation of what I am up to at Nokia. You see a KML representation of the Points of Interest we currently have - the taller the tower the more active places we have in that spacial area. 

MacFusion

MacFusion is a great tool for mounting file systems over FTP or SSH. However, you may need to add an extra option when connecting to clean up UID inconsistencies between your Mac and the server you’re connecting to. In “SSH Advanced” add the following to the “Extra Options:”:

-o defer_permissions

This allows the remote server to correct UID inconsistencies based on the user you are connecting as.

Insulin regime

I am about to change some of my insulin settings, but before I do I thought I’d write them here.

Below is a graph of how things are currently. As I am changing things you can deduce that the regime is not correct, but it isn’t too far off (I think). The four different lines are (from the bottom)

• The thick blue columns are the hourly dose in 100iU units (scale on the left in U/hr). The early morning spike is an attempt to handle my “dawn phenomenon” but it’s not quite working as I’d like. The afternoon dip was a reflection of many lows which I had had in the autumn of 2010. In total I take 18.59U basal daily.
• The thin blue line is the average basal of 0.77 hourly (scale on the left in U/hr).
• The green line is my Carb Ratio in units per 10g (scale on the left in U/10g). A ratio that varies from 1.0 to 1.2 back to 1.1 is quite low - most of these ratios go from 1.0 to 2.5.
• The red line is drawn with the scale on the right - my sensitivity in mmol/L for 1 U of insulin. These sensitivity are very high - they are normally 2 to 3 - so the 3 to 4 as I have here are unusual.

My HbA1C as of 5 Jan 2011 is 7.3 which suggests that the overall insulin dose is quite close to being correct (as a value of 7.0 is quite good, with 6.5 being really very good) - but - there are often quite a lot of corrective doses of extra insulin. This mean that either more basal is needed or there is a problem with my ratios and/or carb estimation. 

The first set of changes relate to some morning readings - as shown here. Though only 3 days readings are shown, this pattern has I believe been going on for about 6 weeks. I have anecdotal evidence of post breakfast lows after giving corrective doses first thing - but foolishly I hadn’t been taking more glucose readings at breakfast. I had been assuming that the correction was still making its way through the system and so didn’t retest.

This chart says a few things to me:

• I need more basal on the run up to 6:00/6:30 in the morning. These elevated readings of 12 or 13 are not the result of a pre-dawn hypo rebound - there is no way the readings could drop so quickly afterwards.
• Either my sensitivity in the morning is off the scale at 6 mmol/L per 1U or I have too much basal between 6:30 and 7:30 (I never eat at 06:30, my first food of the day is at 07:30)

On the morning of the 8th I tried “playing” with these and instead of giving the normal corrective dose I gave only half and this explains why the drop in glucose levels took twice as long as before. Neither of the likely explanations (that I have too much basal or an off-the-chart sensitivity) are what I would call “attractive” but I think it makes more sense to deal with the too-much-basal - which when combined with knowing that I do need more basal in the hours up to 6 am - can be as simple as to move the dawn phenomenon basal spike earlier in the morning.

Moving it by two hours seems too much (though I can’t support this statement - it’s just a gut feeling) so I’ll start by moving it by one to see the effect. I’ll report on that in a week or so. 

jconsole SOCKS proxy issues within a corporate network?

jconsole won’t connect to a process running on your own machine? Try this:

jconsole -debug -J-DsocksProxyHost

It turns out that Java on a Mac doesn’t necessarily respect the System Preferences, but this might help.