A simple model for such a news post could look like:

from django.db import models

class Post(models.Model):
    title = models.CharField(max_length=100)
    content = models.TextField()
    published_at = models.DateTimeField(db_index=True)

    class Meta:
        ordering = ('-published_at',)

    def __str__(self):
        return self.title

In this example, we're using a typical ListView, filtering out any posts which haven't yet been published:

from django.utils import timezone
from django.views.generic import ListView

from .models import Post

class PostListView(ListView):
    queryset = Post.objects.filter(published_at__lte=timezone.now())

Note - we could use an ArchiveIndexView instead, which by default excludes objects from the future. However for this example, we're sticking with ListView to show a simplified version of the problem for other use cases.

When we first load the page, looking through the SQL queries generated for the request, we can see the posts being filtered by their publish date:

SELECT "news_post"."id",
       "news_post"."title",
       "news_post"."content",
       "news_post"."published_at"
  FROM "news_post"
 WHERE "news_post"."published_at" <= '2018-07-21T11:02:12.998079+00:00'::timestamptz
 ORDER BY "news_post"."published_at" DESC

At first this all seems okay, however at some point later on you'll realise that new posts aren't being shown. Looking at the SQL queries generated for the following requests, we can see that the timestamp doesn't change between requests:

SELECT "news_post"."id",
       "news_post"."title",
       "news_post"."content",
       "news_post"."published_at"
  FROM "news_post"
 WHERE "news_post"."published_at" <= '2018-07-21T11:02:12.998079+00:00'::timestamptz
 ORDER BY "news_post"."published_at" DESC

Why? The queryset gets evaluated when the Django server starts, as the queryset is an attribute of the generic view.

One solution for this is to move the queryset into the get_queryset method for the generic view:

class PostListView(ListView):

    def get_queryset(self):
        return Post.objects.filter(published_at__lte=timezone.now())

By using a method we're creating a new queryset for every request - using the current timestamp when the request is generated. Problem solved!

However, since Django 1.9 there's a better way - let the database figure out the current time stamp.

Instead of using timezone.now(), we can switch the view code to the Now() database function:

from django.db.models.functions import Now
from django.views.generic import ListView

from .models import Post

class PostListView(ListView):
    queryset = Post.objects.filter(published_at__lte=Now())

Looking at the SQL queries generated for the request, we can see that STATEMENT_TIMESTAMP() is being used by Postgres to filter out any news posts

SELECT "news_post"."id",
       "news_post"."title",
       "news_post"."content",
       "news_post"."published_at"
  FROM "news_post"
 WHERE "news_post"."published_at" <= (STATEMENT_TIMESTAMP())
 ORDER BY "news_post"."published_at" DESC

The same SQL query will be used for every request, which will now work as the current timestamp gets evaluated by the database - and we don't need to create a get_queryset method for every generic view!

$ ./manage.py help
Traceback (most recent call last):
  File "./manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)

  ...

  File "/Users/tomkins/.virtualenvs/greendale/lib/python3.5/site-packages/django/contrib/gis/geos/libgeos.py", line 147, in geos_version_info
    raise GEOSException('Could not parse version info string "%s"' % ver)
django.contrib.gis.geos.error.GEOSException: Could not parse version info string "3.6.2-CAPI-1.10.2 4d2925d6"

$ port installed geos
The following ports are currently installed:
  geos @3.6.2_0 (active)

Highly annoying!

This will be fixed in PR #8817 for Django master, which will be released in Django 2.0 later this year, and PR #8841 for the upcoming Django 1.11.5 release. However the fix won't be backported to older versions of Django, such as the Django 1.8 LTS branch. So to continue using GeoDjango on macOS, we need to use an older working version of GEOS.

KyngChaos packages

Fortunately KyngChaos has a variety of Unix Compatibility Frameworks available for download, including GDAL and GEOS. Fortunately it's an older version of GEOS (3.6.1) which will still work with older versions of Django. Also the older version of GDAL (1.11) works with Django 1.8, as newer versions of GDAL also cause problems with Django 1.8.

Download and install:

• GDAL 1.11 Complete
• GDAL 2.1 Complete

Although we won't be using GDAL 2.1 in this example, you can easily switch to it if you're only running Django 1.11.

Add the following to your .bash_profile:

export GDAL_LIBRARY_PATH="/Library/Frameworks/GDAL.framework/Versions/1.11/GDAL"
export GEOS_LIBRARY_PATH="/Library/Frameworks/GEOS.framework/Versions/3/GEOS"

Then add the following to your Django settings file:

# GeoDjango fixes
GDAL_LIBRARY_PATH = os.environ.get('GDAL_LIBRARY_PATH')
GEOS_LIBRARY_PATH = os.environ.get('GEOS_LIBRARY_PATH')

On the server you're deploying these environment variables won't be set, so the setting will default to None - in which case Django will automatically find the installed versions of GDAL and GEOS.

Now you should have a fully functioning GeoDjango project on macOS!

$ siege --concurrent=1 --reps=10 "http://127.0.0.1:8000/map/?lat=51.4995&lng=0.1248"
...

Transactions:                     10 hits
Availability:                 100.00 %
Elapsed time:                  11.85 secs
Data transferred:               2.27 MB
Response time:                  0.88 secs
Transaction rate:               0.84 trans/sec
Throughput:                     0.19 MB/sec
Concurrency:                    0.75
Successful transactions:          10
Failed transactions:               0
Longest transaction:            0.96
Shortest transaction:           0.83

Painfully slow! Although it wasn't the most optimised code possible, an average of 880ms per request wasn't acceptable.

Investigating the cause

As the view for this request generates a JSON response, using Django Debug Toolbar wasn't a viable option - as all the debugging output gets attached to HTML responses only. So instead I decided to run through the code with shell_plus from Django Extensions and IPython.

After going through parts of the code, one of the querysets seemed slower than expected:

>>> %timeit venue_list = list(Venue.objects.all()[:100])
10 loops, best of 3: 101 ms per loop

Over 100ms to go through a fairly small queryset? This is far too slow. Just to see if it's a database problem, we'll change it to use values_list instead, which just returns a list of tuples:

>>> %timeit venue_list = Venue.objects.values_list('id', 'name', 'location')
10000 loops, best of 3: 80.3 µs per loop

And testing another model from another app as a quick sanity check to ensure there's no problems with other models:

>>> %timeit permission_list = list(Permission.objects.all())
100 loops, best of 3: 2.29 ms per loop

So something is obviously wrong with the queryset/model.

After seeing that this model had DirtyFieldsMixin, which was one obvious difference between this model and all the others, the next test was to remove it and see if that made any difference:

>>> %timeit venue_list = list(Venue.objects.all()[:100])
100 loops, best of 3: 8.04 ms per loop

From 101ms to 8ms.

After removing all instances of DirtyFieldsMixin, another performance test showed an improvement:

$ siege --concurrent=1 --reps=10 "http://127.0.0.1:8000/map/?lat=51.4995&lng=0.1248"
...

Transactions:                     10 hits
Availability:                 100.00 %
Elapsed time:                   7.98 secs
Data transferred:               2.27 MB
Response time:                  0.40 secs
Transaction rate:               1.25 trans/sec
Throughput:                     0.28 MB/sec
Concurrency:                    0.50
Successful transactions:          10
Failed transactions:               0
Longest transaction:            0.44
Shortest transaction:           0.36

From 880ms to 400ms - a big difference.

Testing django-model-utils

As Django Dirty Fields wasn't great for performance, an alternative which seems to offer similar functionality is the tracker field from django-model-utils.

Let's test by adding a FieldTracker field to a model:

>>> %timeit venue_list = list(Venue.objects.all()[:100])
10 loops, best of 3: 21.5 ms per loop

Much faster than Django Dirty Fields!

Some of the code when saving/updating objects needs updating for django-model-utils, as it doesn't have the same convenience methods:

if venue.id is None:
    venue.save()
else:
    changed_fields = venue.tracker.changed().keys()
    if changed_fields:
        venue.save(update_fields=changed_fields)

One subtle difference between the two packages is that you'll need to ensure the data you enter is the same type. It's possible to give an IntegerField a string value, and Django will still save it.

With Django Dirty Fields:

>>> venue = Venue.objects.get(id=14132)
>>> venue.grid_ref_x
442478
>>> venue.grid_ref_x = '442478'
>>> venue.is_dirty()
False
>>> venue.save()
>>> venue.grid_ref_x
'442478'
>>> venue.grid_ref_x = '123'
>>> venue.is_dirty()
True
>>> venue.save()
>>> venue.grid_ref_x
'123'
>>> venue.refresh_from_db()
>>> venue.grid_ref_x
123

With django-model-utils:

>>> venue = Venue.objects.get(id=14132)
>>> venue.grid_ref_x
442478
>>> venue.grid_ref_x = '442478'
>>> venue.tracker.changed()
{'grid_ref_x': 442478}

Which would result in a lot of additional saves for data, even though the saved data will end up being the same.

Using proxy models

Given that none of the code in the Django views used the dirty fields methods, and won't use the tracker field either - this seems like an ideal case for proxy models in Django. Instead of adding the tracker to the Venue model, we'll create a proxy model instead:

class VenueTracker(Venue):
    tracker = FieldTracker()

    class Meta:
        proxy = True

Now we've got two versions of the same model:

>>> %timeit venue_list = list(Venue.objects.all()[:100])
100 loops, best of 3: 8.85 ms per loop
>>> %timeit venue_list = list(VenueTracker.objects.all()[:100])
10 loops, best of 3: 21.2 ms per loop

So by default we'll use the standard Venue model in views for speed, however if we need a version with tracking for our update scripts, we simply change the imports to point to the model with a tracker:

from .models import VenueTracker as Venue

Now we can have fast views as normal, but with the option to switch to the tracked version if needed.

These days you could use containers to deploy bleeding edge applications with all the new stuff bundled inside, maybe with either Docker or rkt. However in this post we'll be going through a couple of available APT repositories to help modernise a slightly older distribution.

Python

As of October 2016, Python 3.5 is the latest stable version of Python. It's supported in Django 1.8 to 1.10, and for anyone wanting to develop an app for the long term - going for Python 2.7 at this point in time is a dead end. Ubuntu Trusty only has Python 2.7 and 3.4, although 3.4 is well supported - we want to try and go for the latest possible Python version.

Deadsnakes

The Old and New Python Versions repository (or deadsnakes) provides multiple Python versions which aren't included in a particular version of Ubuntu. Consider the support of this repository carefully, as it isn't an official repo (stick with Python 3.4 if this bothers you).

Installation is quick and easy:

$ sudo add-apt-repository ppa:fkrull/deadsnakes
$ sudo apt-get update
$ sudo apt-get install python3.5

As deadsnakes provides even more versions - we could also install Python 3.3 as well, which could help in testing code under multiple Python versions with tox.

PostgreSQL

Ubuntu Trusty comes with PostgreSQL 9.3, which is missing JSONB support. So if we want the JSONField which is new since Django 1.9 - we'll need a more modern version of Postgres. As of October 2016 the latest release of Postgres is 9.6 - which is what we want to aim for.

PostgreSQL Apt Repository

Fortunately postgresql.org provides official packages of all the supported Postgres versions for quite a few supported distributions. As this is provided by the official Postgres site, packages are updated with every new release - so support is good.

To install:

$ sudo add-apt-repository "deb http://apt.postgresql.org/pub/repos/apt/ trusty-pgdg main"
$ curl -sL https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
$ sudo apt-get update
$ sudo apt-get install postgresql-9.6

Now we can use JSONField thanks to the updated Postgres.

Conclusion

Just because you're stuck on an older release of Ubuntu doesn't mean you're stuck with all the old tools - go upgrade!

$ ./manage.py migrate
Traceback (most recent call last):
  ..
django.db.utils.ProgrammingError: permission denied to create extension "postgis"
HINT:  Must be superuser to create this extension.

For security reasons you don't want to give the user superuser access, and for convenience you don't want to go and manually go into each database just to add PostGIS support.

PostgreSQL Extension Whitelisting

The pgextwlist extension can allow regular users to use the CREATE EXTENSION command, allowing them to install the extension without getting the permission denied error.

To install on Ubuntu 16.04 (Xenial Xerus):

$ sudo apt-get install postgresql-9.5-pgextwlist

To install on Debian 8 (Jessie):

$ sudo apt-get install postgresql-9.4-pgextwlist

After installing, edit the relevant postgresql.conf file for your Postgres instance and add the following lines:

# Allow PostGIS to be loaded easily
local_preload_libraries = 'pgextwlist'
extwlist.extensions = 'postgis'

Then run:

$ sudo service postgresql restart

Now you should be able to use a GeoDjango app/database without any errors:

$ ./manage.py migrate
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, sessions
Running migrations:
  ..

First of all you'll need to edit settings.py:

# Use manifest static storage
STATICFILES_STORAGE = 'django.contrib.staticfiles.storage.ManifestStaticFilesStorage'

Then you'll need to edit any templates which aren't using the static template tag. Instead of using:

<img src="{{ STATIC_URL }}images/hello.jpg" %}" alt="Hello">

You'll need to use:

{% load static from staticfiles %}
<img src="{% static 'images/hello.jpg' %}" alt="Hello">

Now when you run django-admin collectstatic, Django will include the MD5 hash of the file as part of the file name. Now when you use the {% static %} tag you'll see the file name with the hash.

If you're running nginx, update your site configuration so that browsers will cache static files for as long as possible:

location /static/ {
    expires max;
}

Then reload nginx.

Now you should be up and running with a good performance boost, and you won't have to ask users to refresh a page to get updated static files.

Excessive Postgres WAL files

To allow point-in-time recovery (PITR), I usually setup Barman. Set it up on a remote server, get your Postgres instance to rsync files over to the Barman server - and you've got a set of backups which should allow you to recover your database from an earlier point in time.

For every UPDATE/INSERT, Postgres will write data to the WAL (Write-Ahead Log). This isn't a problem if you're not doing anything else with the finished WAL files, although you'll have increased disk I/O - it's probably just a minor increase you won't notice too much.

However if you're keeping the WAL and archiving it for backups, every update to a table row will be backed up:

$ barman list-backup all
golestandt 20160613T063007 - Mon Jun 13 06:36:06 2016 - Size: 4.5 GiB - WAL Size: 12.7 GiB

Ouch.

Finding the problem databases

Assuming you've got statistics enabled, use psql to show the statistics for each database:

postgres=# SELECT datname,tup_updated,tup_inserted,tup_deleted FROM pg_stat_database ORDER BY tup_updated DESC;
                datname                | tup_updated | tup_inserted | tup_deleted
---------------------------------------+-------------+--------------+-------------
 site1                                 |     1138475 |       191605 |      136569
 site2                                 |      153224 |        46650 |       12385

Now we've got a list of databases which are excessively updating and inserting new rows.

How did we get here?

By being too lazy with .create_or_update() in applications doing regular syncs.

If you've got Django code syncing with a third party service, it's easy to write code similar to:

for tweet in tweet_list:
    Tweet.objects.create_or_update(tweet_id=tweet['id'], defaults={
        'user': tweet['user']['screen_name'],
        'text': tweet['text'],
        'retweet_count': tweet['retweet_count'],
    })

If you're only updating a few objects occasionally - this is fine and does the job. However if you're updating hundreds of objects every hour - you could end up with hundreds of thousands of rows being updated on a weekly basis, even though most of the data is likely to stay the same.

Every field for the object gets updated every time the object gets saved. If you've got a model with more fields, this will add up very quickly to additional WAL data.

Avoiding updates

If only some of the synced data is actually being updated - just update the fields which will receive any updates:

for tweet in tweet_list:
    obj, created = Tweet.objects.get_or_create(tweet_id=tweet['id'], defaults={
        'user': tweet['user']['screen_name'],
        'text': tweet['text'],
        'retweet_count': tweet['retweet_count'],
    })

    if not created:
        # Update counts, but try to avoid excessive updates
        update_fields = []

        if obj.retweet_count != tweet['retweet_count']:
            obj.retweet_count = tweet['retweet_count']
            update_fields.append('retweet_count')

        if update_fields:
            obj.save(update_fields=update_fields)

In this example we're assuming tweet IDs or usernames won't ever change. This is probably a safe assumption for IDs, and slightly less so for usernames.

By switching to .get_or_create(), and using .save(update_fields=list) - we've significantly reduced the number of updates. If the object already exists and none of the fields have been changed - update_fields will be an empty list and we don't even bother with .save().

The only downside using this method is that it can be tedious having to check every single field for updates. We could probably improve this with a more generic solution - but at this point it's easier to use a third party package which solves the problem.

Django Dirty Fields

Django Dirty Fields is an easy alternative to writing your own code to check all of the fields of an object. Just add a DirtyFieldsMixin to your model and you can simplify your code:

for tweet in tweet_list:
    try:
        obj = Tweet.objects.get(tweet_id=tweet['id'])
    except Tweet.DoesNotExist:
        obj = Tweet(tweet_id=tweet['id'])

    obj.user = tweet['user']['screen_name']
    obj.text = tweet['text']
    obj.retweet_count = tweet['retweet_count']

    # Only save if needed
    if not obj.id:
        obj.save()
    elif obj.is_dirty():
        obj.save_dirty_fields()

Considerably easier to deal with!

The one huge advantage of this approach is that you can add as many fields as you want, and Django Dirty Fields will do the job of figuring out which fields to update for you. As before if none of the fields have been updated - the object won't be saved.

The Results

After trying to fix several projects:

$ barman list-backup all
golestandt 20160704T063009 - Mon Jul  4 06:33:29 2016 - Size: 4.6 GiB - WAL Size: 822.0 MiB

There's still a few old projects which haven't been optimised - however we've now managed to reduce the size of the weekly WAL to be smaller than the weekly snapshot.