Warwickshire County Council Election Map -

This is a map of Warwickshire County Council’s election results for the 4th of June 2009, built as part of a mini-project to show off the possibilities of our opendata before our Hack Warwickshire competition. (You really should enter you know, you might win an iPad.).

Anyway, for comparison, here’s the result for 2005 – it’s easy to see that Labour lost share over the four years. It’s a strong indication of how Labour generally do well in cities and the Conservatives do better in rural areas – compare with BBC map of 2010 UK General Election results.

I really enjoyed putting this together, despite the stress of not knowing anything about this stuff at the start, and it wasn’t that difficult in the end.

The outlines of the electoral divisions are available from the Ordnance Survey as part of the recently opened-up Boundary Line product, and comes as an ESRI Shapefile, a popular file format for geographic vector information.

The full dataset is 46MB in size, and contains polygons for all of the county councils in England, so I’ll need to chuck a load of that data away. Here’s the full dataset mapped out in open-source GIS tool Quantum GIS – Warwickshire is the thing in yellow in the middle.

I used the command line utilty ogr2ogr from the Geospatial Data Abstraction Library (GDAL), and converted the .SHP file from eastings/northings into latitude/longitude:

ogr2ogr -s_srs EPSG:27700 -t_srs EPSG:4326 destination.shp source.shp

…and then converted the resulting file into a .KML:

ogr2ogr -f "KML" destination.kml source.shp

Each area, called a placemark in the KML, came out of the sausage machine looking something like this, only 108MB of it.

<Placemark>
  <Style>
    <LineStyle>
      <color>ff0000ff</color>
    </LineStyle>  
    <PolyStyle>
      <fill>0</fill>
    </PolyStyle>
  </Style>
  <ExtendedData>
    <SchemaData schemaUrl="#electoral_division_latlng">
      <SimpleData name="FID">1558</SimpleData>
    </SchemaData>
  </ExtendedData>
  <Polygon>
    <outerBoundaryIs>
      <LinearRing>
       <coordinates>-1.667473197956685,52.164132540593961 (and lots more....)</coordinates>
      </LinearRing>
    </outerBoundaryIs>
  </Polygon>
</Placemark>

At this point I came to the all-too obvious realisation that the polygons really really were just made out of a shitload of co-ordinates. I knew this would be the case, but I kind of expected it to be something a bit cleverer. I’m not sure what else I could’ve been expecting, really.

So all good so far, but I had no idea which electoral division was which. Alongside the .SHP file was a .DBF file (so dBase, right?) – this can be opened up in Excel. It took me a while to work it out, but the FID number in the KML corresponded to the row number (which is C in the table below) of the area details.

DBF file displayed in Excel

Once I’d chopped out all the non-WCC electoral divisions from the KML file, I was left with about 3MB of data. I took this and ran it through a very simple PHP routine (leaning heavily on the SimpleXML library) to load the data into a MySQL database as field type geometry. The original data was accurate to at least 16 decimal places, which is lovely and all, but overkill on Google Maps, so I took it down to six decimal places, which I think is to within 10cm. Which should be enough.

From there I imported the names of the areas, and with a bit of fiddling, matched those up against the names of the electoral divisions I’d scraped from the original Notes election systems.

I’d had a warning previously that web browsers were a bit rubbish at dealing with the large amount of data it can take to map areas, and I was a bit concerned when I found that the full map of Warwickshire electoral divisions as supplied by the Ordnance Survey contains over 80000 points. It sounded like a lot.

Some rooting around in Stack Overflow brought up a mention of the Douglas-Peucker algorithm, which is a method for simplifying lines. Developer Anthony Cartmell has written an implementation of the algorithm in PHP, and has a demo of it in action too. I used Anthony’s class to simplify the electoral division polygons. Here’s a screenshot showing an example of the line generalisation using Anthony’s class on the Aston Cantlow electoral division – I’ve chosen badly with the colours on the map, the pink line has 813 points, and the yellow line has 74 points.

Comparison between original and simplified lines, using the Aston Cantlow electoral division as an example

I also wrote a routine to export the whole dataset as a simplified KML file – this is at setting 3000, which results in a 176 KB KML file, and 6647 points (just over 8% the size of the full version):

Warwickshire Electoral Division boundary line, generalised

Here’s a link to the KML in Google Maps, with 6647 points – notice on the left that the areas aren’t labelled yet.

After all that, I did an export of the dataset at 99% of the points in the full version, resulting in a 1.8 MB KML (view on Google Map) – which still works quickly in Mac OS X Safari, and was actually OK in IE7 too. I was interested that IE could cope with this many points – it made me wonder if something was being done Google server side to smooth out the points at a particular zoom level.

I also wrote a variation on this to output back to the database rather than out as a KML file – this meant I could quickly experiment with different generalisations of the data to check performance.

Once I’d finally finished fiddling and settled on a generalisation I was happy with (around 8000 points for the entire set), I built up a single GPolygon using the Google Maps API on the results page map, and coloured it in with the winning party colour:

Alcester Electoral Division

For the main map it was just a matter of building up all the GPolygons for all the areas, and listeners to add pop-up bubbles when clicking on an area.

Showing the election pop-up bubble

…and we’re done. Performance is presumably dependent on Javascript performance – it’s definitely fastest in the WebKit-based browsers, as you’d expect, with Firefox 3 being ok, and IE7/8 being very slow. Chrome was actually usable with the full 80,000 points.

Surprisingly, Opera 10 throws a psychedelic fit, refusing to remove previous shapes as you zoom in, which is pretty but a bit rubbish.

All I have for mobile testing is my much loved 1st gen iPod Touch, which crawls but does work, including pinch zoom, interestingly. I’d like to see it on an iPhone 3GS.

Just to compare the experience, here’s the map running within Chrome 4 on a Windows 7 x64 VM on my Macbook.

And here’s the same map in Internet Explorer 8.

I had high hopes for the IE9 preview, which is said to boast much better Javascript performance and SVG, but in my testing it didn’t seem much quicker than IE8.

There are alternatives to building the polygons client-side, but with the experiments I’m doing right now, I’m trying to keep things as simple as possible, running outside of the corporate network using my cheap shared host and free web services to get a quick leg-up and show what can be done without spending a fortune. The shapes could be built server-side, which given a big enough server – however big that is – would be more usable across a wider range of browsers. This would need infrastructure putting in place, and I’m too cheap to get myself a VPS or dedicated server at the moment. Also building the polygons client-side also gives you a level of interactivity (…alright, they’re clickable) which has further possibilities.

So for now, this method works for simple shapes, but for presenting more complicated outlines to a general audience, I’m hoping the future will catch up with us and Microsoft release a blindingly quick version of IE9 which somehow automatically replaces all previous versions in a flash.

Yeah, that’s gonna happen.

But if not, I’ll get round to looking at GeoServer. I noted that KML files displayed quickly when viewed directly in Google Maps in IE7 – this could also be something to explore.

(I should say that this page uses Ordnance Survey data © Crown copyright and database right 2010.)