From the Open Street Map wiki “OpenStreetMap creates and provides free geographic data such as street maps to anyone who wants them. The project was started because most maps you think of as free actually have legal or technical restrictions on their use, holding back people from using them in creative, productive, or unexpected ways.”
In many ways Open Street Map is one of the most interesting and exciting sources of data currently available to low cost map makers. Sadly in some other ways is it also the most frustrating data sources too; it is developed by smart people who just want to get on with it and it allows unprecedented flexibility in adding data [1] which is great until you come to try and style the map or use modern GIS tools to handle it.
I found a mirror of the Open Street Map planet file at http://planet.king-nerd.com/ and fetched planet-100721.osm.bz2. It has to be said that Open Street Map could make it easier to find the complete data set on their wiki, every time I go to fetch data I have to do a Google search to find it. The planet file is huge (10Gb) so allow some time for the download.
The Open Street Map people don’t like standards so you need to download and build a specific tool to import the planet file into a PostGIS database. I found this helpful page which explains the process. First I had to set up some dependencies using:
yum install geos-devel proj-devel postgresql-devel libxml2-devel bzip2-devel gcc-c++ autoconf
Then I fetched the sources to osm2pgsql from the subversion repository:
svn export http://svn.openstreetmap.org/applications/utils/export/osm2pgsql/
Then I had to build the configure script and edit the makefile [2]
./autogen.sh
./configure
sed -i 's/-g -O2/-O2 -march=native -fomit-frame-pointer/' Makefile
and then finally make the program.
Once osm2pgsql is installed I had to work out how to use it. Step 1 seemed to be creating a new database (I was already using the default gis for the Natural Earth data).
createdb osm
createlang plpgsql osm
psql -d osm -f /usr/share/pgsql/contrib/postgis-1.5/postgis.sql
psql -d osm -f /usr/share/pgsql/contrib/postgis-1.5/spatial_ref_sys.sql
Then I could add the data to the database:
osm2pgsql --slim -d osm -l planet-100721.osm.bz2
This was unable to find the projections, so I uninstalled the packaged proj files and rebuilt from source <http://download.osgeo.org/proj/proj-4.7.0.tar.gz>. I also added the postgresql-contrib package
yum install postgresql-contrib
psql -d osm -f /usr/share/pgsql/contrib/_int.sql
Then I left it running overnight. The next day I came in and followed the instructions at http://www.pubbs.net/200812/fedora/15560-moving-varwww-with-selinux-enforcing.html about how to move /var/lib/pgsql to a bigger physical disk, and started it running again. Then 6 days later we had a thunderstorm and the power went out. Sadly it looks like osm2pgsql doesn’t do any sort of check pointing.
I tried fetching the smaller sub-planet for Pennsylvania from Cloudmade.com <http://downloads.cloudmade.com/north_america/united_states/pennsylvania#downloads_breadcrumbs> but osm2pgsql still took several hours to complete the upload which seemed disproportionally long to me [3]. So I started looking for other import options.
The next program I tried was osmosis which has the advantage of being written in Java so I have a better chance of working what is going on. I fetched the program from the Osmosis wiki <http://wiki.openstreetmap.org/wiki/Osmosis> and then went to the usage examples <http://wiki.openstreetmap.org/wiki/Osmosis/Detailed_Usage> to see what it could do.
Footnotes
| [1] | There was (and may still be) a location tagged horse=yes |
| [2] | and the Open Street Map crowd complain about complex GIS tools - at least we ship compiled code. |
| [3] | there is also no way (I can see) to automatically download all of the planet files for the US, let alone the whole world. |