Alex Tomkins

Avoiding cached datetimes with Django querysets

2018-07-21T13:14:00+01:00

In Django we often need to filter out objects from a queryset which shouldn't be visible to public users, a typical example of this would be a news post in a blog. A staff user could edit a news post to have a publish date in the future, allowing it to be automatically published by the site without having to log back in and publish it.

A simple model for such a news post could look like:

from django.db import models


class Post(models.Model):
    title = models.CharField(max_length=100)
    content = models.TextField()
    published_at = models.DateTimeField(db_index=True)

    class Meta:
        ordering = ('-published_at',)

    def __str__(self):
        return self.title

In this example, we're using a typical ListView, filtering out any posts which haven't yet been published:

from django.utils import timezone
from django.views.generic import ListView

from .models import Post


class PostListView(ListView):
    queryset = Post.objects.filter(published_at__lte=timezone.now())

Note - we could use an ArchiveIndexView instead, which by default excludes objects from the future. However for this example, we're sticking with ListView to show a simplified version of the problem for other use cases.

When we first load the page, looking through the SQL queries generated for the request, we can see the posts being filtered by their publish date:

SELECT "news_post"."id",
       "news_post"."title",
       "news_post"."content",
       "news_post"."published_at"
  FROM "news_post"
 WHERE "news_post"."published_at" <= '2018-07-21T11:02:12.998079+00:00'::timestamptz
 ORDER BY "news_post"."published_at" DESC

At first this all seems okay, however at some point later on you'll realise that new posts aren't being shown. Looking at the SQL queries generated for the following requests, we can see that the timestamp doesn't change between requests:

SELECT "news_post"."id",
       "news_post"."title",
       "news_post"."content",
       "news_post"."published_at"
  FROM "news_post"
 WHERE "news_post"."published_at" <= '2018-07-21T11:02:12.998079+00:00'::timestamptz
 ORDER BY "news_post"."published_at" DESC

Why? The queryset gets evaluated when the Django server starts, as the queryset is an attribute of the generic view.

One solution for this is to move the queryset into the get_queryset method for the generic view:

class PostListView(ListView):

    def get_queryset(self):
        return Post.objects.filter(published_at__lte=timezone.now())

By using a method we're creating a new queryset for every request - using the current timestamp when the request is generated. Problem solved!

However, since Django 1.9 there's a better way - let the database figure out the current time stamp.

Instead of using timezone.now(), we can switch the view code to the Now() database function:

from django.db.models.functions import Now
from django.views.generic import ListView

from .models import Post


class PostListView(ListView):
    queryset = Post.objects.filter(published_at__lte=Now())

Looking at the SQL queries generated for the request, we can see that STATEMENT_TIMESTAMP() is being used by Postgres to filter out any news posts

SELECT "news_post"."id",
       "news_post"."title",
       "news_post"."content",
       "news_post"."published_at"
  FROM "news_post"
 WHERE "news_post"."published_at" <= (STATEMENT_TIMESTAMP())
 ORDER BY "news_post"."published_at" DESC

The same SQL query will be used for every request, which will now work as the current timestamp gets evaluated by the database - and we don't need to create a get_queryset method for every generic view!

RUNLEVEL=1 apt-get install package alternative

2018-03-31T17:25:00+01:00

An old documented way of preventing services from starting immediately after installation in Debian/Ubuntu is using the RUNLEVEL environment variable to trick the runlevel helper into returning a response that the system isn't fully running, such as:

# RUNLEVEL=1 apt-get install nginx

Sadly this doesn't work in newer versions of Debian/Ubuntu, the official way is to use the policy helper script /usr/sbin/policy-rc.d and return a 101 exit code. This is a a bit more inconvenient - having to temporarily create this file only to remove it after installing the package.

Fortunately there is an alternative - policyrcd-script-zg2. Install the package:

$ sudo apt-get install policyrcd-script-zg2

Create a new script which returns a 101 exit code, I've created it as /usr/local/sbin/policy-donotstart:

#!/bin/sh
exit 101

Then make it executable:

$ sudo chmod 755 /usr/local/sbin/policy-donotstart

When installing packages where you don't want the service to immediately start, use the POLICYRCD environment variable:

# POLICYRCD=/usr/local/sbin/policy-donotstart apt-get install nginx

The service will install, but you'll a message similar to:

invoke-rc.d: policy-rc.d denied execution of start.

For Ansible, you can add environment variables to any task:

- name: Install nsd
  apt: pkg=nsd install_recommends=no
  environment:
    POLICYRCD: /usr/local/sbin/policy-donotstart

Now you can safely install a package, configure it, and then start it once you've got all the correct files in place - all with a convenient environment variable.

Fixing GDAL and GEOS for Django on macOS

2017-08-06T15:32:00+01:00

As a user of MacPorts for all the additional packages needed when working with Django on macOS, a recent upgrade to GEOS managed to break all my projects which used GeoDjango:

$ ./manage.py help
Traceback (most recent call last):
  File "./manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)

  ...

  File "/Users/tomkins/.virtualenvs/greendale/lib/python3.5/site-packages/django/contrib/gis/geos/libgeos.py", line 147, in geos_version_info
    raise GEOSException('Could not parse version info string "%s"' % ver)
django.contrib.gis.geos.error.GEOSException: Could not parse version info string "3.6.2-CAPI-1.10.2 4d2925d6"

$ port installed geos
The following ports are currently installed:
  geos @3.6.2_0 (active)

Highly annoying!

This will be fixed in PR #8817 for Django master, which will be released in Django 2.0 later this year, and PR #8841 for the upcoming Django 1.11.5 release. However the fix won't be backported to older versions of Django, such as the Django 1.8 LTS branch. So to continue using GeoDjango on macOS, we need to use an older working version of GEOS.

KyngChaos packages

Fortunately KyngChaos has a variety of Unix Compatibility Frameworks available for download, including GDAL and GEOS. Fortunately it's an older version of GEOS (3.6.1) which will still work with older versions of Django. Also the older version of GDAL (1.11) works with Django 1.8, as newer versions of GDAL also cause problems with Django 1.8.

Download and install:

GDAL 1.11 Complete
GDAL 2.1 Complete

Although we won't be using GDAL 2.1 in this example, you can easily switch to it if you're only running Django 1.11.

Add the following to your .bash_profile:

export GDAL_LIBRARY_PATH="/Library/Frameworks/GDAL.framework/Versions/1.11/GDAL"
export GEOS_LIBRARY_PATH="/Library/Frameworks/GEOS.framework/Versions/3/GEOS"

Then add the following to your Django settings file:

# GeoDjango fixes
GDAL_LIBRARY_PATH = os.environ.get('GDAL_LIBRARY_PATH')
GEOS_LIBRARY_PATH = os.environ.get('GEOS_LIBRARY_PATH')

On the server you're deploying these environment variables won't be set, so the setting will default to None - in which case Django will automatically find the installed versions of GDAL and GEOS.

Now you should have a fully functioning GeoDjango project on macOS!

The cost of Dirty Fields

2016-12-04T16:13:00+00:00

After installing Django Dirty Fields on projects a few months ago and seeing a dramatic reduction in the number of writes to our main Postgres database - everything seemed fine. However on a brand new project, something wasn't quite right performance wise:

$ siege --concurrent=1 --reps=10 "http://127.0.0.1:8000/map/?lat=51.4995&lng=0.1248"
...

Transactions:                     10 hits
Availability:                 100.00 %
Elapsed time:                  11.85 secs
Data transferred:               2.27 MB
Response time:                  0.88 secs
Transaction rate:               0.84 trans/sec
Throughput:                     0.19 MB/sec
Concurrency:                    0.75
Successful transactions:          10
Failed transactions:               0
Longest transaction:            0.96
Shortest transaction:           0.83

Painfully slow! Although it wasn't the most optimised code possible, an average of 880ms per request wasn't acceptable.

Investigating the cause

As the view for this request generates a JSON response, using Django Debug Toolbar wasn't a viable option - as all the debugging output gets attached to HTML responses only. So instead I decided to run through the code with shell_plus from Django Extensions and IPython.

After going through parts of the code, one of the querysets seemed slower than expected:

>>> %timeit venue_list = list(Venue.objects.all()[:100])
10 loops, best of 3: 101 ms per loop

Over 100ms to go through a fairly small queryset? This is far too slow. Just to see if it's a database problem, we'll change it to use values_list instead, which just returns a list of tuples:

>>> %timeit venue_list = Venue.objects.values_list('id', 'name', 'location')
10000 loops, best of 3: 80.3 µs per loop

And testing another model from another app as a quick sanity check to ensure there's no problems with other models:

>>> %timeit permission_list = list(Permission.objects.all())
100 loops, best of 3: 2.29 ms per loop

So something is obviously wrong with the queryset/model.

After seeing that this model had DirtyFieldsMixin, which was one obvious difference between this model and all the others, the next test was to remove it and see if that made any difference:

>>> %timeit venue_list = list(Venue.objects.all()[:100])
100 loops, best of 3: 8.04 ms per loop

From 101ms to 8ms.

After removing all instances of DirtyFieldsMixin, another performance test showed an improvement:

$ siege --concurrent=1 --reps=10 "http://127.0.0.1:8000/map/?lat=51.4995&lng=0.1248"
...

Transactions:                     10 hits
Availability:                 100.00 %
Elapsed time:                   7.98 secs
Data transferred:               2.27 MB
Response time:                  0.40 secs
Transaction rate:               1.25 trans/sec
Throughput:                     0.28 MB/sec
Concurrency:                    0.50
Successful transactions:          10
Failed transactions:               0
Longest transaction:            0.44
Shortest transaction:           0.36

From 880ms to 400ms - a big difference.

Testing django-model-utils

As Django Dirty Fields wasn't great for performance, an alternative which seems to offer similar functionality is the tracker field from django-model-utils.

Let's test by adding a FieldTracker field to a model:

>>> %timeit venue_list = list(Venue.objects.all()[:100])
10 loops, best of 3: 21.5 ms per loop

Much faster than Django Dirty Fields!

Some of the code when saving/updating objects needs updating for django-model-utils, as it doesn't have the same convenience methods:

if venue.id is None:
    venue.save()
else:
    changed_fields = venue.tracker.changed().keys()
    if changed_fields:
        venue.save(update_fields=changed_fields)

One subtle difference between the two packages is that you'll need to ensure the data you enter is the same type. It's possible to give an IntegerField a string value, and Django will still save it.

With Django Dirty Fields:

>>> venue = Venue.objects.get(id=14132)
>>> venue.grid_ref_x
442478
>>> venue.grid_ref_x = '442478'
>>> venue.is_dirty()
False
>>> venue.save()
>>> venue.grid_ref_x
'442478'
>>> venue.grid_ref_x = '123'
>>> venue.is_dirty()
True
>>> venue.save()
>>> venue.grid_ref_x
'123'
>>> venue.refresh_from_db()
>>> venue.grid_ref_x
123

With django-model-utils:

>>> venue = Venue.objects.get(id=14132)
>>> venue.grid_ref_x
442478
>>> venue.grid_ref_x = '442478'
>>> venue.tracker.changed()
{'grid_ref_x': 442478}

Which would result in a lot of additional saves for data, even though the saved data will end up being the same.

Using proxy models

Given that none of the code in the Django views used the dirty fields methods, and won't use the tracker field either - this seems like an ideal case for proxy models in Django. Instead of adding the tracker to the Venue model, we'll create a proxy model instead:

class VenueTracker(Venue):
    tracker = FieldTracker()

    class Meta:
        proxy = True

Now we've got two versions of the same model:

>>> %timeit venue_list = list(Venue.objects.all()[:100])
100 loops, best of 3: 8.85 ms per loop
>>> %timeit venue_list = list(VenueTracker.objects.all()[:100])
10 loops, best of 3: 21.2 ms per loop

So by default we'll use the standard Venue model in views for speed, however if we need a version with tracking for our update scripts, we simply change the imports to point to the model with a tracker:

from .models import VenueTracker as Venue

Now we can have fast views as normal, but with the option to switch to the tracked version if needed.

Modern Django with Ubuntu Trusty

2016-10-02T22:57:00+01:00

At this point Ubuntu 14.04 Trusty Tahr is nearly 2.5 years old, with another 2.5 years support left until it reaches end of life for support. However for those of us still working with Trusty, it's often desirable to try and get a few backported modern packages to make development and hosting of Django apps a bit easier to work with.

These days you could use containers to deploy bleeding edge applications with all the new stuff bundled inside, maybe with either Docker or rkt. However in this post we'll be going through a couple of available APT repositories to help modernise a slightly older distribution.

Python

As of October 2016, Python 3.5 is the latest stable version of Python. It's supported in Django 1.8 to 1.10, and for anyone wanting to develop an app for the long term - going for Python 2.7 at this point in time is a dead end. Ubuntu Trusty only has Python 2.7 and 3.4, although 3.4 is well supported - we want to try and go for the latest possible Python version.

Deadsnakes

The Old and New Python Versions repository (or deadsnakes) provides multiple Python versions which aren't included in a particular version of Ubuntu. Consider the support of this repository carefully, as it isn't an official repo (stick with Python 3.4 if this bothers you).

Installation is quick and easy:

$ sudo add-apt-repository ppa:fkrull/deadsnakes
$ sudo apt-get update
$ sudo apt-get install python3.5

As deadsnakes provides even more versions - we could also install Python 3.3 as well, which could help in testing code under multiple Python versions with tox.

PostgreSQL

Ubuntu Trusty comes with PostgreSQL 9.3, which is missing JSONB support. So if we want the JSONField which is new since Django 1.9 - we'll need a more modern version of Postgres. As of October 2016 the latest release of Postgres is 9.6 - which is what we want to aim for.

PostgreSQL Apt Repository

Fortunately postgresql.org provides official packages of all the supported Postgres versions for quite a few supported distributions. As this is provided by the official Postgres site, packages are updated with every new release - so support is good.

To install:

$ sudo add-apt-repository "deb http://apt.postgresql.org/pub/repos/apt/ trusty-pgdg main"
$ curl -sL https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
$ sudo apt-get update
$ sudo apt-get install postgresql-9.6

Now we can use JSONField thanks to the updated Postgres.

Conclusion

Just because you're stuck on an older release of Ubuntu doesn't mean you're stuck with all the old tools - go upgrade!

Easier PostgreSQL Extension Installations

2016-09-24T20:32:00+01:00

Postgres has a number of extensions which require superuser access for a user to install them, PostGIS being one of them. If you're wanting to install a GeoDjango app, you'll probably encounter an error such as:

$ ./manage.py migrate
Traceback (most recent call last):
  ..
django.db.utils.ProgrammingError: permission denied to create extension "postgis"
HINT:  Must be superuser to create this extension.

For security reasons you don't want to give the user superuser access, and for convenience you don't want to go and manually go into each database just to add PostGIS support.

PostgreSQL Extension Whitelisting

The pgextwlist extension can allow regular users to use the CREATE EXTENSION command, allowing them to install the extension without getting the permission denied error.

To install on Ubuntu 16.04 (Xenial Xerus):

$ sudo apt-get install postgresql-9.5-pgextwlist

To install on Debian 8 (Jessie):

$ sudo apt-get install postgresql-9.4-pgextwlist

After installing, edit the relevant postgresql.conf file for your Postgres instance and add the following lines:

# Allow PostGIS to be loaded easily
local_preload_libraries = 'pgextwlist'
extwlist.extensions = 'postgis'

Then run:

$ sudo service postgresql restart

Now you should be able to use a GeoDjango app/database without any errors:

$ ./manage.py migrate
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, sessions
Running migrations:
  ..

Testing cloud-init with Vagrant

2016-09-18T21:22:00+01:00

cloud-init is a standard tool in most cloud computing environments as it's an easy way to provide configuration for provisioning server instances. You provide a YAML config file and it'll use the data to configure the server as needed.

Testing cloud-init in a public can be a bit problematic and time consuming, having to relaunch servers to test the config from a pristine state. It's possible to reset the cloud-init config on the server - but if you want to test that your config file works as expected, it's best to test from a freshly booted server.

Thanks to the Ubuntu Vagrant boxes which are built with cloud-init support, we can test our cloud-init configuration with Vagrant easily. We'll be using the Ubuntu 16.04 Xenial Xerus box to test with - as it's the latest Ubuntu LTS release.

Creating a cloud-init ISO

cloud-init can support multiple data sources for configuration, one of them being an CD image. We'll be using this as an easy way to provide our config files. To generate this we can use genisoimage/mkisofs.

To install on Debian/Ubuntu:

$ sudo apt-get install genisoimage

To install on OS X Macports:

$ sudo port install cdrtools

Next up we'll need our user-data file, the main cloud-init config file:

#cloud-config
chpasswd:
  list: |
    ubuntu:RANDOM
  expire: False

manage_etc_hosts: localhost

ssh_authorized_keys:
  - ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA6NF8iallvQVp22WDkTkyrtvp9eWW6A8YVr+kz4TjGYe7gHzIw+niNltGEFHzD8+v1I2YJ6oXevct1YeS0o9HZyN1Q9qgCgzUFtdOKLv6IedplqoPkcmF0aYet2PkEDo3MlTBckFXPITAMzF8dJSIFo9D8HfdOV0IAdx4O7PtixWKn5y2hMNG0zQPyUecp4pzC6kivAIhyfHilFR61RGL+GPXQ2MWZWFYbAGjyiYJnAmCP3NOTd0jMZEnDkbUvxhMmBYSdETk1rRgm+R4LOzFUGaHqHDLKLX+FIPKcF96hrucXzcWyLbIbEgE98OHlnVYCzRdK8jlqm8tehUc9c9WhQ== vagrant insecure public key

This can be customised with the configuration you need to test, however in this example we're adding the default Vagrant insecure public SSH key. Don't worry - Vagrant will replace it shortly after rebooting.

Now we need a meta-data file:

instance-id: iid-0123456789abcdef
local-hostname: ubuntu-xenial

This is just the bare minimum - cloud providers usually provide more data. If you depend on other data provided as metadata then just add it as needed.

To create the ISO image we'll create a Makefile:

nocloud.iso: meta-data user-data
    mkisofs \
        -joliet -rock \
        -volid "cidata" \
        -output nocloud.iso meta-data user-data

Don't forget - a Makefile must be indented with tabs and not spaces!

Now to create the ISO file, all we have to do is run:

$ make

And you'll have a brand new nocloud.iso ISO image.

cloud-init Vagrantfile

Now we can create a Vagrantfile to test our newly generated ISO image:

$ vagrant init ubuntu/xenial64

This will create a sample Vagrantfile, which we'll need to edit:

# -*- mode: ruby -*-
# vi: set ft=ruby :


CLOUD_CONFIG_PATH = File.join(File.dirname(__FILE__), "nocloud.iso")


Vagrant.require_version ">= 1.8.0"

Vagrant.configure(2) do |config|
    config.vm.box = "ubuntu/xenial64"

    # Disable SSH password for 16.04 - we'll add the insecure Vagrant key
    # (don't worry, it's just an example and gets replaced anyway)
    config.ssh.password = nil

    # Disable shared folders
    config.vm.synced_folder ".", "/vagrant", disabled: true

    # Tweak virtualbox
    config.vm.provider :virtualbox do |vb|
        # Attach nocloud.iso to the virtual machine
        vb.customize [
            "storageattach", :id,
            "--storagectl", "SCSI",
            "--port", "1",
            "--type", "dvddrive",
            "--medium", CLOUD_CONFIG_PATH
        ]

        # Speed up machine startup by using linked clones
        vb.linked_clone = true
    end
end

This creates a new virtual machine based on the xenial64 Vagrant box, but replaces the CD/DVD port with our nocloud.iso file.

All you have to do at this point is:

$ vagrant up

Vagrant will start up a virtual machine, customised with your cloud-init config file. If anything goes wrong you can easily vagrant destroy the machine and try again - without the cost of starting a new server in a public cloud.

Using your own SSH keys

If you're developing a user-data file and want to use your own SSH keys instead of Vagrant replacing it on every boot, edit the Vagrantfile and add these lines:

config.ssh.private_key_path = File.expand_path("~/.ssh/id_rsa")
config.ssh.insert_key = false

Don't forget to edit the user-data file and add your own public key to ssh_authorized_keys, and run make to generate a new ISO image.

Conclusion

Testing cloud-init with Vagrant is quick and easy once you've got everything setup.

Source code for this is available in the cloud-init-vagrant repo on GitHub.

Automating Ansible groups with Vagrant

2016-09-10T21:57:00+01:00

Vagrant is a great tool for creating development environments and testing deployment scripts, especially with Ansible. Testing an Ansible playbook is can be as simple as vagrant up.

The problem

Ansible uses an inventory file to define the SSH details for each server, what groups a server belongs to - which then allows you to run certain roles/playbooks on specific servers. However the two options with Vagrant in dealing with the inventory file are rather limited.

The first option is the auto generated inventory, Vagrant will create a fresh inventory file based on the servers listed in the Vagrantfile which are currently up and running. This inventory file gets populated with the correct SSH details, however it means it starts with an empty inventory file which is only populated by setting ansible.groups.

The second option is the static inventory, which Vagrant will use instead of creating a dynamic inventory file. The downside with this would be duplicating the existing hosts file just to edit the SSH details, as well as hardcoding the IP addresses of all your Vagrant virtual machines.

The solution

We can use the ansibler Ruby gem to read the hosts file to automatically populate ansible.groups.

Installation is easy, as Vagrant maintains plugins separately from other Ruby libraries:

$ vagrant plugin install ansibler
Installing the 'ansibler' plugin. This can take a few minutes...
Installed the plugin 'ansibler (0.2.2)'!

Now we need to update the Vagrantfile. First of all we're going to read the existing hosts file, and populate the ANSIBLE_GROUPS variable with a hash of all the groups:

inventory = Ansible::Inventory.read_file('hosts')

ANSIBLE_GROUPS = {}

inventory.groups.each do |group|
    ANSIBLE_GROUPS["#{group.name}"] = group.hosts.map{ |host| host.name }
end

Next up is updating the Ansible provisioner config:

config.vm.provision "ansible" do |ansible|
    # ...
    ansible.groups = ANSIBLE_GROUPS
end

And that's it!

Now that ansible.groups contains an exact copy of all the group config from your hosts file, every time Vagrant creates a new dynamic inventory file it'll be populated with all the group configuration from your main hosts file.

Upgrading packages in all Python virtual environments

2016-09-03T14:32:00+01:00

If you're working with a lot of Python virtual environments in a wide variety of projects, over time you'll end up with outdated packages in some of the older ones - potentially missing out on new features.

For most of the packages you'll want to keep them at a specific version for each virtual environment, however for packages such as pip, you may want to upgrade it to the latest version in all your local virtual environments. If you're running virtualenvwrapper, all you have to do is use the allvirtualenv command:

$ allvirtualenv pip install --upgrade pip setuptools wheel

And inside every virtual environment you'll have the latest version of pip and the other essential package installation tools updated in your virtual environment.

Speed up Django static files

2016-08-28T21:16:00+01:00

For a fairly easy performance win, and for something which makes dealing with old cached CSS something a thing of the past - enable ManifestStaticFilesStorage.

First of all you'll need to edit settings.py:

# Use manifest static storage
STATICFILES_STORAGE = 'django.contrib.staticfiles.storage.ManifestStaticFilesStorage'

Then you'll need to edit any templates which aren't using the static template tag. Instead of using:

<img src="{{ STATIC_URL }}images/hello.jpg" %}" alt="Hello">

You'll need to use:

{% load static from staticfiles %}
<img src="{% static 'images/hello.jpg' %}" alt="Hello">

Now when you run django-admin collectstatic, Django will include the MD5 hash of the file as part of the file name. Now when you use the {% static %} tag you'll see the file name with the hash.

If you're running nginx, update your site configuration so that browsers will cache static files for as long as possible:

location /static/ {
    expires max;
}

Then reload nginx.

Now you should be up and running with a good performance boost, and you won't have to ask users to refresh a page to get updated static files.

Be careful with Django's .create_or_update()!

2016-08-21T18:44:00+01:00

Although using Model.objects.create_or_update() with a Django model is extremely convenient, sometimes you might want to consider using it carefully with certain usage patterns.

Excessive Postgres WAL files

To allow point-in-time recovery (PITR), I usually setup Barman. Set it up on a remote server, get your Postgres instance to rsync files over to the Barman server - and you've got a set of backups which should allow you to recover your database from an earlier point in time.

For every UPDATE/INSERT, Postgres will write data to the WAL (Write-Ahead Log). This isn't a problem if you're not doing anything else with the finished WAL files, although you'll have increased disk I/O - it's probably just a minor increase you won't notice too much.

However if you're keeping the WAL and archiving it for backups, every update to a table row will be backed up:

$ barman list-backup all
golestandt 20160613T063007 - Mon Jun 13 06:36:06 2016 - Size: 4.5 GiB - WAL Size: 12.7 GiB

Ouch.

Finding the problem databases

Assuming you've got statistics enabled, use psql to show the statistics for each database:

postgres=# SELECT datname,tup_updated,tup_inserted,tup_deleted FROM pg_stat_database ORDER BY tup_updated DESC;
                datname                | tup_updated | tup_inserted | tup_deleted
---------------------------------------+-------------+--------------+-------------
 site1                                 |     1138475 |       191605 |      136569
 site2                                 |      153224 |        46650 |       12385

Now we've got a list of databases which are excessively updating and inserting new rows.

How did we get here?

By being too lazy with .create_or_update() in applications doing regular syncs.

If you've got Django code syncing with a third party service, it's easy to write code similar to:

for tweet in tweet_list:
    Tweet.objects.create_or_update(tweet_id=tweet['id'], defaults={
        'user': tweet['user']['screen_name'],
        'text': tweet['text'],
        'retweet_count': tweet['retweet_count'],
    })

If you're only updating a few objects occasionally - this is fine and does the job. However if you're updating hundreds of objects every hour - you could end up with hundreds of thousands of rows being updated on a weekly basis, even though most of the data is likely to stay the same.

Every field for the object gets updated every time the object gets saved. If you've got a model with more fields, this will add up very quickly to additional WAL data.

Avoiding updates

If only some of the synced data is actually being updated - just update the fields which will receive any updates:

for tweet in tweet_list:
    obj, created = Tweet.objects.get_or_create(tweet_id=tweet['id'], defaults={
        'user': tweet['user']['screen_name'],
        'text': tweet['text'],
        'retweet_count': tweet['retweet_count'],
    })

    if not created:
        # Update counts, but try to avoid excessive updates
        update_fields = []

        if obj.retweet_count != tweet['retweet_count']:
            obj.retweet_count = tweet['retweet_count']
            update_fields.append('retweet_count')

        if update_fields:
            obj.save(update_fields=update_fields)

In this example we're assuming tweet IDs or usernames won't ever change. This is probably a safe assumption for IDs, and slightly less so for usernames.

By switching to .get_or_create(), and using .save(update_fields=list) - we've significantly reduced the number of updates. If the object already exists and none of the fields have been changed - update_fields will be an empty list and we don't even bother with .save().

The only downside using this method is that it can be tedious having to check every single field for updates. We could probably improve this with a more generic solution - but at this point it's easier to use a third party package which solves the problem.

Django Dirty Fields

Django Dirty Fields is an easy alternative to writing your own code to check all of the fields of an object. Just add a DirtyFieldsMixin to your model and you can simplify your code:

for tweet in tweet_list:
    try:
        obj = Tweet.objects.get(tweet_id=tweet['id'])
    except Tweet.DoesNotExist:
        obj = Tweet(tweet_id=tweet['id'])

    obj.user = tweet['user']['screen_name']
    obj.text = tweet['text']
    obj.retweet_count = tweet['retweet_count']

    # Only save if needed
    if not obj.id:
        obj.save()
    elif obj.is_dirty():
        obj.save_dirty_fields()

Considerably easier to deal with!

The one huge advantage of this approach is that you can add as many fields as you want, and Django Dirty Fields will do the job of figuring out which fields to update for you. As before if none of the fields have been updated - the object won't be saved.

The Results

After trying to fix several projects:

$ barman list-backup all
golestandt 20160704T063009 - Mon Jul  4 06:33:29 2016 - Size: 4.6 GiB - WAL Size: 822.0 MiB

There's still a few old projects which haven't been optimised - however we've now managed to reduce the size of the weekly WAL to be smaller than the weekly snapshot.

Improving a Pelican blog with gulp

2016-08-13T22:49:00+01:00

Pelican is a Python powered static site generator, which is useful for blogs or other small static sites. Although a respectable number of Pelican plugins are available - the Node.js/JavaScript ecosystem for build systems is currently much stronger and updated far more frequently.

In this example I'm going to use gulp.js to take the output from Pelican, process it and output something slightly better.

Installing gulp

We'll quickly create a package.json file to make things easier later:

$ npm init

Then we can install the packages we need. In this small example we'll use gulp-htmlmin to minify the HTML output from Pelican:

$ npm install --save gulp gulp-htmlmin

To make running gulp a bit easier, update the package.json file and add an entry into scripts:

{
    "scripts": {
        "gulp": "gulp"
    }
}

Now we can run gulp with npm run gulp - however we'll need to create a gulpfile first.

gulpfile.js

Create a file named gulpfile.js:

var gulp = require('gulp');
var htmlmin = require('gulp-htmlmin');


gulp.task('copy', function() {
    return gulp.src(['./output/**', '!./output/**/**.html'])
        .pipe(gulp.dest('dist'));
});

gulp.task('html', function() {
    return gulp.src('./output/**/*.html')
        .pipe(htmlmin({
            collapseWhitespace: true
        }))
        .pipe(gulp.dest('dist'));
});

gulp.task('default', ['copy', 'html']);

This gulpfile has a copy task, which will just copy all files which don't have an HTML extension from output to dist, so any images, CSS, or RSS feed files will be copied across without any modifications.

We've also got an html task, which takes any files with an .html extension and runs it a minifier before being output to the dist directory.

Updating the Makefile

A few tweaks are needed for publishing to

Add a DISTDIR variable at the top of your Makefile, this is where we're going to output files from gulp:

DISTDIR=$(BASEDIR)/dist

Update the publish target to run gulp every time we regenerate the site:

publish:
    $(PELICAN) $(INPUTDIR) -o $(OUTPUTDIR) -s $(PUBLISHCONF) $(PELICANOPTS)
    npm run gulp

Update the service_upload target for the provider you upload your Pelican site with, replace OUTPUTDIR with DISTDIR. In this example I've updated the rsync_upload target:

rsync_upload: publish
    rsync -e "ssh -p $(SSH_PORT)" -P -rvzc --delete $(DISTDIR)/ $(SSH_USER)@$(SSH_HOST):$(SSH_TARGET_DIR) --cvs-exclude

Publish!

Now we've got gulp installed, created our gulpfile, and updated the Makefile, updating the published content should be as simple as:

$ make publish

Pelican will run first, then gulp will process all the files into the dist directory. Any other targets will perform both of these tasks first before uploading the content to your preferred provider.

What's next?

Improve your gulpfile and add more tasks to it! You might find some inspiration from Gulp Starter, or from hundreds of other gulp tutorials online.

Alternatively you could use Grunt, or an entirely different build system which isn't JavaScript powered. Thanks to the Makefile you can easily change the build process for your site.

MySQL UNIX socket authentication

2014-07-26T23:49:00+01:00

It's not currently a default feature, but since MySQL 5.5.10 you can easily enable UNIX socket authentication.

Add the following to your MySQL config file:

[mysqld]
plugin-load=auth_socket=auth_socket.so

Restart MySQL, then you'll be able to use IDENTIFIED WITH auth_socket as a replacement for IDENTIFIED BY 'password'.

Makes things easier for local installations for services running under a specific account, or for limiting access to the root account to the local root user.

DigitalOcean Debian kernel

2013-11-16T16:38:00+00:00

DigitalOcean offers some fantastic KVM powered virtual machines for low prices, however the technical decisions they've made in various places leaves something to be desired.

One of the problems which you may face at some point is that you're stuck with whatever kernel they provide, you can't use your own bootloader. They've attempted to solve this by providing kernels from every Linux distribution they support - but sadly this isn't the solution which I'd expect a KVM provider to use.

To work around this, we can use kexec to change the kernel on boot. Install kexec:

$ sudo apt-get install kexec-tools

Move /sbin/init to /sbin/init.distrib:

$ dpkg-divert --add --local --rename /sbin/init creates=/sbin/init.distrib

And create a script to replace /sbin/init

#!/bin/bash

if grep -qv " kexeced$" /proc/cmdline; then
    /sbin/kexec --load /vmlinuz \
        --reuse-cmdline \
        --initrd=/initrd.img \
        --append="init=/sbin/init.distrib kexeced" \
    && /bin/mount -o ro,remount / \
    && /sbin/kexec --exec
fi

exec /sbin/init.distrib "$@"

Make it executable:

$ sudo chmod 755 /sbin/init

And then you can reboot. Next time your server restarts, the script will run before any other init scripts and will change the running kernel with kexec. You can change the kernel parameters in this file if you need custom options, or just leave it as is - and you'll be able to stick with your standard Debian kernel.

Credit to Joe Johnston for the original script.

Update

A couple of years later, DigitalOcean eventually created new images which use the virtual machines bootloader - which means this hack is no longer needed for newer Linux distributions.

Hello Pelican!

2013-09-08T19:53:00+01:00

After nearly 5 years of using Wordpress to power this blog, I've finally switched to an alternative - Pelican, a Python powered static site generator.

Why the switch? Maintaining a Wordpress site is annoying in the long term. Upgrades are needed on a regular basis for security patches, which is to be expected from a fairly large web application. However having to deal with styling issues from using a custom theme, and dealing with plugin incompatibilites every time isn't something I want to deal with for a simple site.

Pelican includes the features I want (themes, code highlighting, etc) without the hassle of dealing with a site which I need to upgrade every few months.

Android fragmentation must be addressed

2010-05-04T00:10:00+01:00

According to the Platform Version statistics at the Android developer site, 37% of devices are running 1.5, 29% are on 1.6, with 32% running 2.1 - the latest version. Android 2.1 has been available since January 2010, however the number of users on older phones is shockingly worrying and is harmful for the future of Android.

I'm not too bothered about the amount of different devices which use the Android operating system, which certain people seem to focus on. What does concern me are the slow handset manufacturers who aren't updating their phones to the latest version within a reasonable amount of time. The Motorola Droid was one of the more successful Android 2.0 devices which has been updated to 2.1, which probably explains why there are still a few users still on the 2.0 series (inexperienced users who don't know how to update), but could be much worse if this device wasn't updated.

Manufacturers need to update all their recent phones within a reasonable amount of time, otherwise the future of the platform is going to be an awkward one. Users are often spending hundreds of pounds on phones which they consider to be smart phones, yet their manufacturer is treating them like the existing dumb phones - where you get updates if you're lucky. Customers are paying a premium for what they think is an alternative smart phone to the likes of the Apple iPhone. Smart phones deserve a respectable lifespan of updates after the release.

At the moment my HTC Hero is due an update in June 2010, this is 5 months after the release of 2.1 on other platforms. I love Android, however I no longer like HTC. I'm sure there are others who love the operating system, but hate the way their manufacturer is treating them with updates.

Just remember HTC - other phone manufacturers are releasing Android powered devices too.

Sockets with JavaScript

2009-03-10T23:14:00+00:00

We've seen a great push towards using JavaScript in modern websites, using AJAX and JSON for dynamic websites which start to feel more like applications instead of plain boring pages. One of the problems which more interactive sites face is the constant polling of the web server due to the way HTTP works. Various web apps these days tend to poll for data with the server keeping the connection alive without sending any data, once data has been sent then a new request is sent asking for more data.

One particular web app which isn't that great is IRC web chat. Most IRC servers are keen on users connecting from their real connection, as it's hard to track abuse when connections are being proxied by a web server. Using AJAX or JSON with a web server isn't that ideal either, browsers have to constantly poll and refresh with the server to receive the data, this typically requires a light weight web server as Apache with PHP or Perl is hardly efficient for multiple users. The most common approaches over the years has been CGI:IRC, but this is just yet another inefficient Perl script which keeps the connection open. One of the oldest methods is using a hefty Java application, hardly friendly for users.

So, why not use sockets with JavaScript? Obviously JavaScript can't do it directly, but with Adobe Flash and something like socketBridge (or this improved version) you could control a socket with mostly JavaScript. The only problem with Flash and sockets is that it would have to connect to a policy port before being authorised to connect to that particular port, however it would be a much lighter solution for the server compared to all the polling techniques.

I'm tempted to try and create a web IRC client using JavaScript and the Flash socket bridge. Even if other IRC networks won't open a port for Flash policy requests, you could do something similar to the other web chat apps - use your server as a proxy.

Forwarding all mail with Gmail

2009-03-09T17:59:00+00:00

If you've got multiple Gmail accounts, such as different Google Apps Gmail accounts (for hosted domains) and gmail.com accounts, being able to forward them all to one account saves a lot of time and effort. Gmail does allow you to forward email under the Forwarding and POP/IMAP section of the control panel, however this only forwards non-spam emails. As most people know, sometimes the spam filters aren't quite perfect and suffer from false positives occasionally.

To forward all mail to another account, simply remove your existing forward, create a new filter from * (matches any sender), then forward it to your main email address - making sure you've got Never send to spam selected. You'll probably want to choose Delete it as well, this means any incoming mail on that specific account will be forwarded and deleted on the original account - but you'll still have 30 days until it gets removed from trash.

This way you'll get all mail forwarded from Gmail - including spam. Now you can see all the spam from any other Gmail accounts incase there are any false positives.

Expansions in MMORPGs

2009-01-02T11:57:00+00:00

After resubscribing to Dark Age of Camelot, there is one thing which has been slightly bothering me whilst reminiscing about the game - the fact that expansions and updates in MMORPGs tend to destroy any use of old areas.

Trials of Atlantis rendered a huge amount of armour and weapons entirely useless from the original version and the first expansion, thanks to their overpowered nature. Catacombs comes along and it had some of the best areas to level up characters in - until they increased the bonuses for other areas making it entirely pointless. The more recent Dragon's Revenge updates makes the older armour less useful, so everyone changes to the latest and greatest. Certain things tend to remain useful for quite a long time if there isn't any better alternative, however once something better has been added the players will flock to it.

Why does it bother me? There is a lot of quality content in the game which people won't touch and will avoid as much as possible because newer things are better. Why would I bother doing the very old quests which give virtually no reward? Instead I can do the latest expansions quests and get 500 gold just for completing one of steps. Many hours must have been spent on the design, the coding, the graphics, the storyline and everything else involved in an expansions content - yet a few updates a year later will render that content pointless.

I'd like to see more MMORPGs updating their older areas at the same time as working on their expansions. This is something Mythic has been doing, although it has only been a fairly recent and minor effort. Update the stats on drops for old items, though not retroactively as to encourage people to go back for the new bits. Add more quests and content to existing areas instead of putting the entire focus on new areas, maybe more of the old content could be updated to point to new areas within the storyline - instead of just having a portal to switch to new bits.

Players like to reminisce about the old days, take advantage of your old content and update it slightly to make them think they're still playing the game they once loved.

Adverts in RSS feeds

2008-12-29T20:06:00+00:00

RSS feeds are great for those people who want to follow up on news sites without having to constantly revisit the same site multiple times a day. Google FeedBurner offers nice statistics to any sites with RSS feeds, allowing them to see how many subscribers they've got by hosting the RSS feed. What I'm finding increasingly annoying is the other service which FeedBurner offers - their advertising programme.

I don't mind if RSS feeds are either partial feeds to show users the first paragraph to attract users to visit the site, or full feeds to show users the entire article, however putting in adverts is highly annoying. To me RSS feeds are there to entice me to go and visit the site itself. RSS feeds are mostly used by fairly technical users - putting in adverts where the people reading them are most likely to ignore them seems a bit pointless. Adding other little buttons/icons to enhance the feed is annoying as well, I just want the article and nothing more!

I don't usually bother with advert blocking, but after seeing more feeds with adverts in I've finally installed Adblock Plus for Firefox. Good riddance to pointless adverts!

Cables not included

2008-12-25T17:56:00+00:00

To me this is one of the most annoying things manufacturers decide on when releasing their products – not including all the cables necessary to use the product!

Various USB printers have done this for quite a while, you have to buy the cable separately. Whilst it's sometimes understandable with cheap products, it's still slightly annoying that the product you buy isn't usable until you go and buy yet another cable. What I find really annoying is spending £300 on a console such as the PlayStation 3 and they're too cheap to include an HDMI cable. This is a console designed for high definition gaming, yet they don't even bother including a cable for it.

It feels like the modern equivalent of "batteries not included", certainly not something which should happen if you spend large amounts of money on a new device.

Poor Windows Vista support in games

2008-11-19T16:39:00+00:00

As much as people might dislike Vista, it's the operating system bundled with new machines and it's here to stay. It's been around 2 years since Vista was launched, whilst I can understand certain software developers might not offer support from the immediate release – it should be one of their high priorities. MMORPG developers seem to be among the worst for poor Vista compatability, along with other game developers.

Certain games decide that they need to run as Administrator from the moment the patcher starts up, even though the patcher might not need to download any file at all – therefore not needing to write any files to the Program Files directory. So the entire game runs under admin privileges, just incase the patcher might need to update files – Mythic Entertainment are guilty of this with Dark Age of Camelot.

Other games are still living in the old days where they still write any configuration files to the game directory for all players, instead of the relevant application data directory for the user. This is just extremely lazy and makes things awkward for users, instead of a clean split between game data and user data – everything is just lumped together in one directory. If I wanted to make a backup of my computer, it would be far easier to just backup my user profile which would contain any important data – however keeping this data along with the game itself makes it difficult. If I wanted to restore my machine from a backup, I'd lose any settings from the game.

There are other games which are complete failures, getting players to go with workarounds such as changing the permissions on the installation directory. After installing Lord of the Rings Online I didn't even get a shortcut to the game from my start menu.

If game developers could just put a bit of extra effort into game installation it would make things so much easier. I know the aim of these developers is probably just to get the game running on these platforms, but a fully functional game which works properly would save users so many headaches.

Server Name Indication being held back

2008-10-17T20:10:00+01:00

Ever since we've had websites available over HTTPS (HTTP over SSL), there has always been the problem that the host always requires an IP per site as each site needs a new certificate. With SSL the server couldn't just switch certificate according to the site, as SSL is negotiated before any HTTP request is sent.

Server Name Indication is a TLS extension which sends the hostname during the TLS negotiation, which means the server can switch to the appropriate certificate – allowing a web host to potentially have as many HTTPS sites as they want on a single IP address. It's a great solution, however the only problem is that any chance of it's currently being held back.

Firefox 2.0, Opera 8.0 and even Google Chrome supports SNI, however Safari on OS X currently doesn't support it and Internet Explorer 7 only supports it on Windows Vista and not Windows XP. So SNI is being held back due to a lack of support from IE and Safari, these are two fairly significant browsers which many people use – admins can't ignore either of them.

SNI is only just becoming available in Linux distributions with Apache 2.2.8 and the latest versions of OpenSSL, so it isn't available to everyone just yet. However being unable to use it for many years due to the fact that any Internet Explorer user on Windows XP won't be able to view any SNI sites is going to be a huge hinderance towards adopting it.

3ware 8006-2LP RAID with CentOS 5

2008-10-14T12:50:00+01:00

The Hetzner DS-8000 servers feature a 3ware RAID card (8006-2LP) for hardware RAID 1, a nice feature – although the default CentOS setup is slightly broken with SELinux. After several hours of wondering why smartd wouldn't work, /var/log/audit/audit.log reveals the following:

type=AVC msg=audit(1223922183.059:22): avc:  denied  { getattr } for  pid=2654 comm="smartd" path="/dev/twe0" dev=tmpfs ino=8940 scontext=user_u:system_r:fsdaemon_t:s0 tcontext=user_u:
object_r:device_t:s0 tclass=chr_file

A good administrator will probably want to add some custom SELinux policies, but the lazy admin might just want to disable SELinux. I went with the disabling of SELinux as the easier option! It's not a recommended solution though.

Eventually I got smartd working again, so I've now got decent monitoring of the disks on the machine.

Using Linux but never giving back

2008-09-30T20:55:00+01:00

One great thing we see with Linux is that it's used for servers quite frequently, it's probably one of the strongest points about it. A strong and stable server platform which many companies rely on to run many parts of their infrastructure for an organisation.

A common situation we'll see are MMORPGs, games with thousands of players mostly with Windows clients connecting to servers powered by Linux. Occasionally you'll see people trying to run their game client under WINE, but it isn't quite the same as running a native game. These companies use Linux to run their gaming servers, web servers and possibly many other parts of their infrastructure, but when it comes to building a Linux client for their game they'll rarely consider it or even dismiss it without even considering the feasibility of it.

I don't think anyone expects companies who use Linux to start contributing hundreds of lines of source code back to various projects, but it would be nice if they considered a Linux client of their product where it's viable as a way of saying thank you. They're getting the benefits of many hours of effort being placed into a freely available operating system, but they're not helping this operating system by refusing to release their product for it.

In an ideal world we'd see multi platform clients, maybe we'll see more if it one day.

How not to launch an MMORPG

2008-09-29T10:20:00+01:00

The past few weeks I've been enjoying one of the latest MMORPG games – Warhammer Online. The main annoying thing about the European WAR experience is the fact that GOA are running the servers for all players over here. As expected they managed to make several mistakes which many players knew were bad decisions, yet GOA still went full steam ahead. Tim Jenks has listed some of them in his How not to launch a beta for a WoW-Killer MMO, I'm going to a list a few of the things which have annoyed me about the entire launch.

Don't leave registration to the last minute. GOA managed to utterly fail the open beta release as their system couldn't cope with the large numbers of players trying to register their open beta codes. Apparently it was an unforeseen problem and they did test their systems beforehand, however this wouldn't be so much of an issue if they spotted this a week earlier by letting players register in advance.

Don't open your servers in the afternoon. As with all MMORPG games you need to kill something in order to gain experience to advance your character, this is an absolute nightmare if you're fighting with hundreds of other players in the starting zone. If the servers were up at around 6AM then the hardcore players would've been able to rush through the starting zones allowing other players who get up at a more reasonable time to get started with a zone which isn't so busy.

Don't launch with a limited number of servers. Players with the collectors edition or standard edition who ordered from certain retailers were able to get head start codes which allowed them to play 3-4 days earlier than those who didn't pre-order. Certain servers became overpopulated rather quickly, as players who didn't pre-order wanted to join their friends in guilds on the servers which were open for head start, instead of the newer servers which were open on the official launch day.

The entire point of all the different head start dates for collectors edition, standard edition and the official launch should've allowed GOA to spread out registrations and people joining servers – however if they leave everything to the last minute then disasters are bound to happen, and they did.

Overall I'm enjoying the game quite a lot, there are various concerns I've got with various aspects of the game, but for now I'm having fun!

Ubuntu on the Aspire One

2008-08-26T01:40:00+01:00

Okay, I didn't last long. After 24 hours of sticking with Linpus I eventually decided to install Ubuntu on the laptop, even though adjusting it for the Aspire One was going to involve some tweaking – sticking with Linpus would've ended up annoying me.

The AspireOne page on the Ubuntu wiki explains how to get you Ubuntu up and running. The Aspire One supports PXE booting, which I found far easier to setup than fiddling about with a USB stick just to get the install going.

The Ubuntu Netbook remix includes some great packages which are useful for running GNOME on a small screen, I did try them all however I wasn't too happy with the UME launcher. I'd highly recommend Maximus, an application which maximises open windows and removes the top bar from running programs, window-picker-applet which is ideal with Maximus and doesn't hog too much space for switching between programs, and human-netbook-theme for a few visual tweaks.

Overall I'm fairly happy, now I've got the power of Ubuntu for this laptop. Hopefully the weird quirks and bugs which the Aspire One has will be ironed out in future releases.

Acer Aspire One

2008-08-23T00:24:00+01:00

After resisting the EeePC for so long in the hope that better ultra portable laptops would appear with better specs, I finally decided to go for an Acer Aspire One. The 8GB flash storage and 512MB RAM are perfectly adequate for basic use in Linux, it's a great little device for browsing a few sites without having to take a huge laptop around with you.

The Aspire One comes with Linpus Lite, a Fedora based distribution aimed at lightweight devices. After using it for a day it seems reasonably adequate for an average user, but for someone who understands Linux well might get a little bit annoyed at the quality of this particular distribution.

This version of Linpus Lite is based on Fedora 8, slightly out of date as it was released back in November 2007. Firefox 2 is included, which is also a little bit old and support is ending shortly as Firefox 3 has been out for quite a while. I suppose this is sort of expected with the fast pace of change, but it's slightly annoying. The one thing I did try to do is run yum upgrade from the command line, however the dependencies are broken so it won't update – not exactly great.

I'm keen on replacing Linpus with Ubuntu in the future, however the support for the Aspire One isn't quite complete at the moment with some functionality not quite working. I'll keep track of it for the future, however I want ease of use instead of having to deal with command line tweaks to fix broken functionality.

Overall a thumbs up for the Acer Aspire One, but a bit of a thumbs down for Linpus Lite. I just wish they could've sorted out Ubuntu for this device.

Hello world!

2008-08-10T21:04:00+01:00

It's taken over a year for me to finally get my blog up and running, I registered the domain back in April 2007 but I've only recently managed to get the urge to finish it. Many thanks to Joni for helping me finish the design off.

Some of you may recognise that the avatar at the top of the page is my current Mii avatar from the Wii. The only problem is that it's taken so long that it isn't entirely accurate – I've recently had my eyes lasered, so no more glasses for me. Looks like I'll have to update it eventually, hopefully it won't take me another year to get that sorted though!

I'll be using this place for my various thoughts and rants, so check back often for updates.