OpenEcoMaps is back!

OpenEcoMaps, eco-living maps using OpenStreetMap data, is now working again. Hooray! I decided to sit down and work out why the OpenLayers interface wasn’t working and it turned out to be quite simple to fix.

You can now browse around maps of low carbon energy generators in London, veggie restaurants in Edinburgh, allotments in Exeter, recycling facilities in Glasgow and more! The data is updated every hour, direct from OpenStreetMap, and is available on maps and downloadable/reusable KML and GeoJSON files. The code is also in Github, so you could set-up your own version for another country if you like.

OpenEcoMaps is back!

There are still some of the layers that aren’t working because the underlying data isn’t being extracted from OpenStreetMap properly. But I’m very glad that, after well over six months with it completely broken, the web site basically works again!

OpenEcoMaps halfway back

For almost a year now, my pet project OpenEcoMaps has been broken. The vagaries of unreliable XAPI servers meant the system couldn’t download OpenStreetMap data to create all the KML files, and (I think) some changes to OpenLayers meant the web maps also stopped working. It has taken me a long time to work up the energy to fix these.

Today I can happily say one half of the system is now working again, and the underlying code is much improved.

OpenEcoMaps halfway back

OpenEcoMaps KML files, and now GeoJSON files, are being created again. Hooray! I switched from XAPI to the Overpass API; grabbed JSON which enabled me to write a more powerful function to turn this into usable objects (for example building a complete Python object for an allotment merging data from relevant nodes, ways and relations); wrote a new library to create GeoJSON files; refactored everything else to fit with these changes; and made numerous other small improvements.

You can browse, download and use the KML files and GeoJSON files with those links. To see an example, look at this KML file of low/zero carbon energy generators overlaid on Google Maps.

Now I just need to fix the web maps so you can see the lovely features on the main web site, and so people can easily embed the maps on their own web sites. I did dabble with using Leaflet before Christmas but I got stuck trying to get the icons to match styles defined in the GeoJSON file. I had a quick look at the OpenLayers code and quickly decided I had better things to do with my time! If anyone fancies giving it a go, the code is all in Github and is all released under the General Public License.

Analysing Southwark’s natural geography

Following my map of London’s green and blue infrastructure, I have been working on some analysis of the land uses.

I was inspired and encouraged to try this by Liliana’s interesting work called “imagining all of Southwark“. Lili and Ari have managed to get the council to release lots of data on properties and car parking, and they are producing analysis of this data by postal code area and by street. They haven’t managed to get anything on land uses, so I thought, why not produce this with OpenStreetMap data?

A few evenings later, here is the result shared on Google docs (direct link) covering the eight postal code areas that between them cover most of the borough (SE1, SE5, SE15, SE16, SE17, SE21, SE22, SE24):

What the data means

The “summary” worksheet shows the total land area, expressed in hectares (10,000 m2), for various different types of land coverage. I have also calculated the percentage of that postal code area that the land uses represent, which gives an interesting insight into the differences between the areas.

Some of the land uses will overlap, for example miscellaneous bits of green space are often mapped on top of residential areas. So the numbers aren’t supposed to add up to anything like 100%.

The spreadsheet also contains worksheets for each postal code area. These contain a dump of all the objects in OpenStreetMap in those postal code areas, and this is the raw data the summary spreadsheet uses to get the totals.

Flaws in the data

You should use this data with a large spoonful of salt. Here are the significant flaws I have noticed:

Postal code areas are approximate, for example the boundary between SE15 and SE22 should mark the boundary between Peckham Rye Common (SE15) and Peckham Rye Park (SE22). In my data both the park and the common show up in both of the postal codes, because the boundary isn’t quite right. Read down to my method to see why. The errors introduced are pretty tiny in most places (plus or minus a few meters along the full boundary), and probably cancel themselves out for big land uses like residential, but they probably also introduce some significant errors for parks where the boundaries go awry by 20-30m in places. Sadly there aren’t any accurate open data polygons I can use.

Data is missing because OpenStreetMap contributors haven’t mapped it. Of course the easy solution here is to get more of it mapped and up to date! My estimate of the different types is as follows:

  • Allotments: complete for the whole borough.
  • Parks and commons: all major and district parks complete.
  • Misc green spaces: very poor coverage of, for example, large areas of grass on estates, especially in SE5, the north pat of SE15 and SE17.
  • Woods/forest: all major woods complete, coverage of big clumps of trees e.g. on a housing estate or in a park is very uneven.
  • Residential: complete except for SE16.
  • Industrial, retail, commercial: large areas are complete, but small shopping parades, industrial parks and rows of offices are very patchy.
  • Brownfield/construction: patchy across the borough and sometimes out of date as sites are built on.

Data is also sometimes missing because of flaws in the Geofabrik shapefiles, not all of which I have corrected. For example, I noticed they were missing commons so I manually added those in, but I may have missed other land uses. One major omission, a shame given the interest in them, is the humble sports pitch/playing field.

How I produced this

After a lot of experimentation – I’ve never been trained to use GIS tools – I worked out this method. If you know of an easier way I’d love to hear about it.

  1. Prepare the boundary data:
    1. Extract a polygon for the London Borough of Southwark from the OS Boundary-Line data.
    2. Download the OS Code-Point-Open data, open the spreadsheet for the SE area in QGIS and use the ftools ‘Voronoi polygons’ plugin to infer polygons for the postal codes from the centroids. Post code centroids are very dense in the middle of residential areas, so the boundary between SE15 4HR and SE22 9BD is only going to be out by a few meters, but are quite far apart with large parks and commons, so the inferred boundaries get less accurate in those areas. See this map for an illustration of the Peckham Rye Park / Common problem mentioned above.
    3. Merge together postal codes into the areas (e.g. SE22 9QF, SE22 4DU etc. into SE22) by quering the shapefile for all objects with postal codes starting with SE22, then using the mmqgis merge tool to merge them into single polygons. Clean up the attributes so the shapefile just has one attribute for the correct postal code area.
    4. Clip the postal codes by the Southwark polygon and save the result – finally – as the postal codes shapefile for Southwark.
  2. Prepare the land use data:
    1. Download the  OpenStreetMap shapefiles from Geofabrik for Greater London.
    2. Download common and marsh ways/relations using the Overpass API (with the meta flag on), import the data into QGIS using the OpenStreetMap plugin, and save the data as a Shapefile.
    3. Merge together the Geofabrik natural and landuse shapefiles with my Overpass-derived shapefile into one land use shape file using the mmqgis plugin.
    4. Clip the land use file by the Southwark polygon and save the result – finally – as the land uses shapefile for Southwark.
  3. Produce the postal code stats; for each postal code:
    1. Select the postal code, and clip the land use layer to that selected code, saving it as a new shapefile.
    2. Open that shapefile, then save it in a new projection that will be in meters rather than degrees (I used  EPSG:32631 – WGS 84 / UTM zone 31N).
    3. Open the new shapefile, then run the ftools ‘Export/add geometry columns’ tool (in Vector/Geometry Tools) to add two attributes to the objects for the area and perimeter.
    4. Save the layer again as a CSV file.
  4. Produce the stats for the area of each postal code so we can calculate % of the area as well as ha for each land use:
    1. Save the Southwark postal codes polygon in the meters projection, add the geometry columns, and save as a CSV file.
  5. Collate all the data
    1. Tidy up and copy the data from each CSV file into a spreadsheet, then add in the formulae to tot everything up. You’re done!

For reference, some of the totals in the summary work off more than one land use type so here are the categories and the corresponding OpenStreetMap tags:

  • Allotments – landuse=allotments
  • Parks and commons – leisure=park / leisure=common
  • Misc green spaces – landuse=conservation / landuse=farm / leisure=garden / landuse=grass / landuse=greenfield / landuse=greenspace / landuse=meadow / landuse=orchard / landuse=recreation_ground
  • Woods and forest – landuse=forest / natural=wood
  • Residential, industrial, retail, commercial, brownfield, construction – corresponding landuse tags

Future ideas

One obvious improvement would be to get more data in. Perhaps this first analysis will encourage people to help out with that? I have also emailed Geofabrik about the flaws I have discovered in their shapefiles, so I hope those get fixed.

Another thought is to produce the stats by council ward. But given that there are far more wards, I’d like to find a quicker way of producing the stats for each ward (step three above) first.

It would also be interesting to do it by town/suburb, for example comparing Peckham to East Dulwich. But we don’t have any meaningful boundaries for those natural areas. It would be really interesting to do a mass version of “this isn’t fucking Dalston” for a whole borough, using the Voronoi polygons method to infer areas from surveys at thousands of locations around the borough. One day…

London’s natural geography

I’ve been playing around with open data from OpenStreetMap and Natural England to make a pretty map of “green and blue infrastructure” in London. Here’s the result:

You can download a PDF version suitable for printing here: natural_london.

I’m pretty happy with the result, my first real attempt to produce something useful with QGIS. The data I used was:

There’s no reason the Natural England data couldn’t be manually added to OpenStreetMap, giving us a complete dataset of natural features. I just chose to get on and do it this way rather than wait, or try to add all the data across areas of the city I don’t know well and am not going to visit any time soon. I also didn’t really need to use the Ordnance Survey data for boundaries, but it’s slightly more accurate and complete than OpenStreetMap data.

The map is probably missing lots of smaller patches of green space, including grass verges, green roofs and biodiverse brownfield sites. The biggest omission is the humble private garden. They cover 24% of London’s land!

But the map at least shows the more obvious, visible, public green spaces, and is a nice example of what a geek with no GIS training (but years of playing with OpenStreetMap) can do with free software and free data these days.

Sitting around the data campfire

Similar to Gail Ramster, I went along to the Friday afternoon part of UK GovCamp 2012 without really knowing why. I suspect most people would say the same thing. You go because… well, you never know which useful people you might bump into, and what interesting things you might hear about. Plus a colleague Janet Hughes was going, and I’d cleared my desk of essential work for the week.

Here are a few takeaway thoughts from my afternoon.

1. I barely knew anyone

It’s years since I was a fish in a geeky pool, active in the free culture movement, the KDE community, software patent activism and other odds and sods.

For the past five years or so I’ve moved onto land, or perhaps a coral reef, to be more involved with issues around the environment, housing and pay inequality. The past two or so have been working as a local government employee at the GLA, supporting Green Party Members of the London Assembly. They have pushed for open data, but it’s not exactly a hot topic in our weekly meetings. My only remaining connection has been OpenStreetMap, my one geeky obsession.

Still, it didn’t matter, go along even if you know no-one at all.

2. It was nice to reconnect with optimistic techies

The event reminded me of one of the things I most like about these crowds: they’re all optimistic about the future and enthusiastic about the common interest.

I’m glad I managed to quickly chat to a few people I did know, sort of… Gail via Twitter, and Giles Gibson from the Herne Hill Forum, but sadly I only said as much as “hello” to people like Emer Coleman and Chris Osborne. That’s what you get for arriving late and leaving early.

3. It’s more meaty than you’d think

That’s “meatspace” as in “the real physical world”, compared to “cyberspace” online. Compared to events a few years ago on open data and technology, most of the discussion I heard was about councils and companies working on staff structures and consultation processes, and then thinking about how technology and data could help.

I used to get frustrated with discussions that started with the assumption that open data and technology was going to revolutionise the world. That seemed upside down to me. So I was pleasantly surprised at this.

4. There’s a lot of “we”

Somebody pointed this out in one session – it’s very easy to apply “we” to the wider population when you really mean “we sort of people in this room”.

Often “we” are innovators or early adopters of ideas that become more mainstream, like using a smart phone to access services. Sometimes “we” are set to be a significant minority, like journalists, bloggers and politicians who use data to enhance their investigatory work. Just as often “we” are a world unto our own.

It’s fine, innocent mostly, typical of any event with like-minded people. It just grated on me when people talked about reconfiguring public services or management around their preferences, as though the rest of the world will thank them.

I might make a badge for myself if I go again, with the slogan “we’re not normal” or similar!

5. Theres a lot going on out there

Cocooned in City Hall, working on affordable housing or the pay gap, it’s hard to keep even a toe dipped in this pool. It was great hearing from so many people in so many walks of work and life doing so many useful things.

Sometimes when I map an area for OpenStreetMap, walking down a street noting house numbers, I feel a bit bewildered by all these people living here! London feels impossibly enormous. I left UKGovCamp feeling similarly bewildered by the enormity of work going on in this field, relative that is to my own small bits and pieces in my job and my free time.

Political philosophy in Open Street Map

Thea Clay made the killer point in Chris Osborne‘s “What’s wrong with OpenStreetMap” session (video here). It was even better than Mikel Maron‘s observation that people should agree with founder Steve Coast just a little bit less!

Foundation Board member Henk Hoff (a very nice-sounding chap) was describing the classic technocratic argument that in a “do-ocracy”, those who get on with doing things make decisions by default. Steve must have loved it, you just get on with useful stuff and avoid getting bogged down in pointless debates. Right?

Thea pointed out that in a community of tens of thousands, only a few hundred can “do” stuff like making amazing tools and creating useful maps from the tags they’ve invented. I’m in the bigger mass of people who want to contribute data, see lots of good uses and ways in which OpenStreetMap can improve, but lack the skills to “just get on and do” much that I’d like to with the data. I’ve been at it for five years.

I spend a lot of time involved in local groups trying to improve their town centre, grow food on their housing estate, represent tenants or establish a community-owned renewable energy company. They’ve come up with some great ideas for using OpenStreetMap, but none of us have the skills or money to “just do it”. I’ve tried, for two years, to get others to help with this with very little success. So is OpenStreetMap just not for us? Do I leave them to use Google Maps and miss out on the potential of such an amazing open platform?

Henk’s suggestion that the decisions of the “do-ers” are generally accepted is also just plain wrong. Nobody has ever been able to agree on the appropriate difference between marking a feature as a footway or a path, after years of debate. The same goes for wood/forest and countless other features.

That’s a pretty sad inditement as it makes for even more confusing map detail (the map key is absurdly long, and why should the average map user care if it’s a footway or a path?) that also makes applications like routing needlessly complicated.

Anarchy gives OpenStreetMap real energy and the space for innovation by the do-ocracy elite. But it leads to very inconsistent data and the exclusion of all but a technically hyper-literate few.

Most mappers come from countries run by democratic governance; they use software created by communities with clear and functional democratic structures like this, this and this; there’s a great study on Debian that drives the point home. Many mappers have engaged with the pseudo-democratic process of voting on new tagging proposals. I suspect that most mappers – if you canvassed beyond the extremely vocal elite – would support a more democratic approach to basic decisions about core tags, use of funding, etc.

That won’t stop people “just doing” what they want to do, but it will make the data more useful and the project more accessible for the vast majority of people outside of the technical elite.

Finally, just as we let people who prove their technical skill loose on the technology, why not let people with proven organisational and political skills loose on these problems? An audience member suggested OpenStreetMap was going through puberty. Well as much as teenagers hate to admit it, sometimes authority borne of experience and training is worth listening to.

For these reasons and more we need to hear much more from the likes of Thea and Mikel, and a lot less from techno-junkies like Steve.

Map geeks in the bowels of City Hall

Following past discussions with staff at Southwark Council and the Greater London Authority, I helped organise a technical  workshop in City Hall this weekend. We brought some key OpenStreetMap geeks together with some keen potential early adopters from the GLA, Southwark and Brent to talk open maps and hack on data & tools.

Harry Wood will post a useful rundown of some of the stuff people hacked on over the two days.

The only practical thing I did was to work with Rob Scott (OSM) and Scott Day (Southwark Council) to try and extract buildings from the open Ordnance Survey maps so we could merge them into OpenStreetMap without having to manually trace every single one (yawn). For the nerds, I’ve written up our initial results on the wiki.

Talking with Scott, it was clear that Southwark probably have a lot of data we could benefit from, and that they would really like to use OpenStreetMap in lots of places where they don’t need the pinpoint accuracy for road shapes, gardens, etc. that they get with commercial products like UK Map and Ordnance Survey’s Mastermap.

Alisdair Maclean from Brent brought the whole road network on a USB stick. Robert did a quick visual analysis of differences with OSM data using his “musical chairs” app, and Grant overlayed the two sets of data to spot roads he had missed over the years. Again, I hope Alisdair really benefited from the opportunity to chat.

The trick is really just to keep experimenting and helping government people to try things out. Expecting local authorities to just come and use OSM without any decent documentation or personal contact is… well, it won’t happen. Without those relationships, open data initiatives like the London Datastore could just end up dumping data on the public without benefiting from our capacity to correct and improve upon that data.

Why I could never vote for the Pirates

I’ve a lot of respect for anyone who steps up to run for election with a manifesto that, they genuinely hope, will improve the lot of their constituents. But aside from my obvious partisan reasons, I don’t think I could ever vote for a Pirate Party candidate in these forthcoming national and local elections.

I suspect I’m like the majority of people in that I really get put off by politicians saying “don’t vote for Party X or you’ll let Party Y in”, as though they’ve nothing more compelling to offer voters than “we’re not that lot”. Ultimately I would always want people to vote for the party they most support, give or take some tactical voting if they prefer. So if the Pirates are your bag then get involved with them.

But the Pirates are an unashamed single issue party. Their manifesto lays out a radical agenda for copyright, patents and online privacy. That’s an interesting proposition for an MEP who can take that militant approach in a very large Parliament. But MPs and councillors are constituency politicians, they need to represent and support people on every issue on the books with an open statement of their approach. If I were to vote for a Pirate, I’d want to know that they are concerned about the need for affordable homes, better partnership working to improve my town centre and urgent action on climate change. Even if I weren’t a committed Green, I’d want a local Pirate candidate to set her/his personal stall on those issues before they got my vote.

We Greens also have some good policy on these subjects, tying them into our wider approach to the economy, culture and government. When their grassroots and youth wing mobilise, the Lib Dems sometimes take these issues up, the Conservatives sometimes talk the talk and the split personality Labour government have made some good recent moves. I’d much rather vote for a candidate from a rounded party who also took the Pirates’ concerns seriously in these elections.

Boris the culture commie?

Are free photos evil? I’m going to stick my neck out and defend the Greater London Authority for setting up a Flickr group where Londoners can submit photos to be used on the GLA web site. A few photographers are upset that anyone can now get decent photos for free from citizens who donate them. Shocker. These photographers want the GLA to use our taxes to pay them for their hard work.

I’m sorry, but that’s just plain ridiculous. Should we condemn the GLA for using free software for their web site, instead of paying for a proprietary content management system? Dearie me. Look, the web has changed many creative industries and bust the business models of those few who were charging for stuff that lots of us will happily share quite freely. Get over it.

This storm-in-a-lens-pouch has been picked up by the venerable Boris Watch and the Telegraph, who both seem to sympathise with the photographers. They echo the photographers and conflate this issue with the way that the police and the More London security guards act stupidly towards photographers, as though this has anything to do with the GLA (a different organisation) inviting its 7.56 million citizens to contribute their lovely photos to the GLA web site.

The great collection of photos in the group – including a few of mine – suggest that most people are quite happy donating their work.

European Parliament elections, the Green Party and free stuff

Open AccessGavin Baker, a really fantastic free data activist from the US, nudged me recently about his post on the position of different European parties on open access to research. Scott Redding, one of our Green Party candidates in the elections and also doing amazing work getting more online activism for the party, gave a fairly strong statement on his intention to support the cause of open/free data.

So where do we stand? It’s easier to explain by talking about our wider policy and activism around intellectual property. So here’s an update on a previous post:

The Green Party in England & Wales hasn’t done much work specifically on open access to research but I guess the message to Gavin and other activists is: we have strong, clear policy on this, and we’re an open door. Green MEPs will always listen to you, turn up to vote on directives, write letters and within reason support the cause!