Scraping GPX files out of RunKeeper into Postgres and QGIS

I’ve been using RunKeeper to track my walking. I wanted to work with the 40+ GPS tracks they’ve logged for me and didn’t want to download them one at a time. Some quick notes on how to scrape them all out.

  1. Log into the website in your browser and go to your Activities page. It has a URL like http://runkeeper.com/user/USERNAME/activity/70000177
  2. Save that big HTML page to a file named activities.html
  3. Capture the IDs of all your activities from that page with a regexp like this:
    printMatches  ‘link=”/user/USERNAME/activity/([0-9]+)'< activities.html > ids
  4. You can now download a GPX for that ID with a URL like http://runkeeper.com/download/activity?activityId=70000177&downloadType=gpx. However, the download requires a URL, at least if your activities are private.
  5. Look in your browser for your JSESSIONID cookie. That’ll stand in for a login.
  6. Use wget with that cookie. Here’s my little shell script to get them all
cat ids | while read id; do
  wget --no-cookies \
       --header 'Cookie: JSESSIONID=C~2BF2D2156284732CE8D8BDA900000000' \
       -O $id.gpx \
       "http://runkeeper.com/download/activity?activityId="$id"&downloadType=gpx";
done

Set up a PostGIS database with all the data

  1. createdb walks
  2. psql -d walks -f postgis.sql
  3. psql -d walks -f spatial_ref_sys.sql
  4. psql -d walks -f postgis_comments.sql
  5. for f in *gpx; do echo $f; ogr2ogr -append -f “PostgreSQL” PG:dbname=walks $f; done

ogr2ogr will create a fancy schema for you that contains every field ever seen in a GPX file. For RunKeeper data, the salient things are:

  • tracks.wkb_geometry: one row per track, multiline string
  • track_points: one row per point with a point, elevation, timestamp, etc.

QGIS did a fine job rendering the tracks table for me. I also pulled in a raster OSM map for a base layer from http://wms.latlon.org/.

Here’s a quicky query to get the length of a track in miles. Need to understand the ST_Transform better, also join to something else to get proper data about each imported track.

select name, ST_Length(ST_Transform(wkb_geometry, 2163)) /1609 as miles from tracks order by name;

A quicky view to simplify the tracks a bit, plus a query that compares the length of the original and simplified views.

create or replace view simple as
select
 ogc_fid,
 ST_Simplify(wkb_geometry,0.00005) as track,
 name
from tracks;

select
  tracks.name,
  ST_Length(ST_Transform(simple.track, 2163)) /1609 as miles,
  ST_Length(ST_Transform(tracks.wkb_geometry, 2163)) /1609 as miles
from simple inner join tracks
on simple.name = tracks.name order by name;