Happy multiprocessing

I managed to get multiprocessing working for OpenAddress job processing. Looks like it’s working pretty well. We use multiprocessing.Pool to manage worker processes, then set up a SIGALRM to abort jobs that run too long. (Using a nifty decorator to add the timeout). I also added a SIGUSR1 handler in the master job process that lets us abort the run. At least once I’ve had the pool fail where there was work being done but none of the worker processes was trying to do it.

Here’s a happy screenshot of htop showing a run going. Note the nifty custom process names courtesy of setproctitle.

Screen Shot 2015-01-18 at 3.30.17 PM

OpenAddress Machine job management

Our OpenAddresses system wants to run N jobs at once while downloading and processing our ~1000 sources. Mostly to do a bunch of network requests to various servers in parallel, but also for some CPU parallelism in multi-core systems. Normal jobs use very little RAM (100M?) but occasionally a big GeoJSON file blows up to 2G or 4G. Here’s 4 options for job management.

Unix processes. I’ve been running stuff in GNU parallel, forking off a new Python process for each job. 8 simultaneous processes on a 8 CPU system. That’s worked pretty well, see previous blog posts for notes. CPU usage around 250%, memory usage all over the place but generally under 2G. Full job finishes in 3, 4 hours.

Python threads. I just tried openaddr-process for the first time to run a full go (on the same Linux box). That code works by spawning 16 threads. So one big Python process. That’s not working quite so well. Running at a steady 120% CPU; because of Python’s GIL we can’t get more parallelism. The process also has 11G resident. I suspect much of that could be reclaimed, probably left over from when 2 or 3 big GeoJSON jobs ran at once. But that stresses the garbage collector; there’s something to be said about killing a process. (Update: the resident set size got as big as 17G at one point, enough to force swapping. It didn’t seem to thrash though. Also Python is releasing the memory again later, the RSS has gotten as small as 9G again.)

Python multiprocessing. I want to try this. It should be a relatively easy change from threading, but I am hoping gives some CPU parallelism and memory housekeeping. Also we need a way to kill jobs asynchronously and Python’s threading module is designed to make that possible. With multiprocessing you just send a SIGTERM and are done.

Decoupled task architecture. openaddr-process is structured to do the job as one big long-lived Python process. Even with multiprocessing, when an individual job finishes it’s passing Python objects back to the task manager, which waits 3+ hours to collect them all before formatting the crucial report. This is brittle. Long term I’d like to redesign the system to be more decoupled, so that individual jobs run as completely separate processes and autonomously store state in some persistent storage; files and maybe a row or two in a database. Then the overall Machine monitor can just fork off jobs as it needs and collect data off the database. That’s much more like a task queue architecture, a robust thing. I know how to build this with my own servers, less clear how to match it to the EC2 model of cloud computing.

Fiona vs Unicode

Following on my OGR/Python vs Unicode work, I took a look at what Fiona is doing. Fiona is a nice Pythonic interface to OGR with a fair amount of Cython code to bridge the gap from the OGR C library to Python idioms. Fiona explicitly returns all strings as u’Unicode Strings’ in both Python 2 and 3 and has code to handle encodings.

Fiona tries to guess encodings. In theory you can override the guess with an encoding=’foo’ parameter when you open a source, but in practice that seems to do nothing. I tried passing in obviously wrong encodings like ‘ascii’ or ‘shift-jis’ and didn’t see any of the expected errors.

Below is the repr of the string Cassiopée as it comes through from the Shapefile in Fiona, with its own guessing. These all appear correct to me. Note that Python 2 encodes Unicode string reprs as ASCII using Python \x escaping, hence the \xe9 in the output. That’s a single character, the expected Unicode codepoint U+00e9 or é.

Good file (ca-qc-gatineau)

Python2. Fiona: u’Cassiop\xe9e’  OGR: b’76 Rue de Cassiop\xc3\xa9e’
Python3: ‘Cassiopée’  OGR: u’Cassiopée’.

Bad file (be-flanders)

Python2. Fiona: u’Now\xe9lei’  OGR: b’Now\xe9lei’
Python3. Fiona: u’Nowélei’  OGR: exception.

Summary

Fiona seems to be doing the right thing with both my inputs, returning proper Unicode strings in both Python 2 and Python 3. In Python 2 OGR seems to basically not be decoding at all, just returning byte strings that I’m supposed to decode myself. In Python 3 OGR is trying to return Unicode strings but throws an exception on my bad file.

I think from here, for our OGR code I should add something to explicitly decode strings to Unicode in Python 2 and file a bug in Python 3.

Fiona code

'Extract fields from shapefiles using Fiona, a Unicode test'

import sys, fiona, logging
from pprint import pprint

logging.getLogger().setLevel(logging.DEBUG)
logging.basicConfig(stream=sys.stderr, level=logging.DEBUG)

enc = sys.argv[2] if len(sys.argv) > 2 else None
with fiona.open(sys.argv[1], 'r', encoding=enc) as source:
    for f in source:
        for field in f['properties'].values():
            sys.stdout.write('%s %r ' % (type(field), field))
        sys.stdout.write('\n')

OGR/Python vs. Unicode

I’ve been trying to figure out how OGR deals with Unicode, particularly in shapefiles, to solve this issue with OpenAddress conversion. I haven’t found clear docs. There’s something about a SHAPE_ENCODING environment variable and something else about OGR trying to guess. More details in that linked GitHub issue. Anyway, here’s what I discovered.

Good file

I have a working non-ASCII shapefile that seems to be in UTF-8. It’s ca-qc-gatineau for us, stored at http://gatineau.ca/donneesouvertes/telechargement/ADRESSE.zip, and has street names like Rue de Cassiopée that our code is working for us.

When OGR/Python opens the file, the capability OLCStringsAsUTF8 is set to True.

In Python 2, GetField() is returning Python strings of type str, which means “bytes”. Those strings appear to be sequences of UTF-8 code points, the repr() of that non-ascii street name is ’76 Rue de Cassiop\xc3\xa9e’. Basically OGR isn’t trying to handle Unicode at all for us, or rather if it is it’s returning UTF-8 encoded byte strings and all is well.

In Python 3, GetField() is also returning Python strings of type ‘str’, which now means ‘unicode’. Their repr in Py3 is a Unicode string, ‘Rue de Cassiopée’, which makes sense.

In both cases OGR/Python is doing the right thing, or at least something consistent and sensible.

Bad file

I have a non-ASCII shapefile that seems to be in Latin 1. It’s be-flanders for us, stored at https://downloadagiv.blob.core.windows.net/crab-adressenlijst/Shapefile/CRAB_Adressenlijst.zip. It has street names like ‘Nowélei (Jean Baptiste)’ although that é isn’t coming through right.

When OGR/Python opens the file, the capability OLCStringsAsUTF8 is set to False.

In Python 2, GetField still returns ‘str’. And the repr for that street is ‘Now\xe9lei (Jean Baptiste)’. That’s actually not awful; if I know to expect that behavior, I can decode that myself with ISO-8859-1 and get the Unicode string.

In Python 3, GetField() throws an exception.

Traceback (most recent call last):
  File /home/nelson/src/oa/shpenc.py, line 17, in <module>
    field = in_feature.GetField(i)
  File /usr/lib/python3/dist-packages/osgeo/ogr.py, line 3033, in GetField
    return self.GetFieldAsString(fld_index)
  File /usr/lib/python3/dist-packages/osgeo/ogr.py, line 2362, in GetFieldAsString
    return _ogr.Feature_GetFieldAsString(self, *args)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 3: invalid continuation byte

I don’t quite understand what is happening here, but I guess something inside the OGR code is trying to decode that sequence assuming it’s UTF-8 and failing. Note my code is not really in the stack trace here, I”m just asking for the field contents. I think this is an OGR bug. Or else I think the Shapefile is corrupt. Setting SHAPE_ENCODING before invoking my code doesn’t help.

Worth noting QGIS displays the string as Nowlei, skipping the non-ascii character.

Summary

It appears that what OGR is doing is trying to guess and cope with the source encoding, presenting UTF-8 byte strings to Python 2 code and Unicode strings to Python 3 code. Which is great! Only things go wrong with the be-flanders file. OGR presents an ISO-8859-1 byte string to Python 2, which you can cope with. OGR crashes in Python 3.

Test code

Here’s the little OGR/Python program I’m using to examine shapefiles. It runs in both Python2 and Python3.

import sys
from osgeo import ogr, osr
ogr.UseExceptions()

in_datasource = ogr.Open(sys.argv[1], 0)
in_layer = in_datasource.GetLayer()
inSpatialRef = in_layer.GetSpatialRef()

in_layer_defn = in_layer.GetLayerDefn()
in_feature = in_layer.GetNextFeature()
print('OLCStringsAsUTF8? %r' % in_layer.TestCapability(ogr.OLCStringsAsUTF8))
while in_feature:
    row = dict()
    for i in range(0, in_layer_defn.GetFieldCount()):
        field = in_feature.GetField(i)
        sys.stdout.write('%s %r ' % (type(field), field))
        in_feature = in_layer.GetNextFeature()
    sys.stdout.write('\n')

Second full OpenAddress run

After the first run against 600+ sources we patched up a bunch of bugs, improved the running infrastructure. Then I ran another set last yesterday, amended a little bit today to re-run jp-* and kr-* after one more bug fix. I wasn’t quite as careful in measuring everything, but here’s some notes on output.

  • 579 input sources: 484 out.csv files, 483 sample.json files
    84% success rate; compare 66% from previous run.
  • 102M output rows.
    Compare 80M from previous run, 100M from Node
  • Roughly 3.5 hours running time, maybe less.
    3 threads stalled on bad servers so it’s not a good measure.

Throwing out the 3 stalled runs, the average source took 137s to execute. Standard deviation of 585s, so it’s a very broad distribution. That first run averaged 104s (SD 265). But a lot more of those runs failed in 1 second!

I’m curious if the new ESRI code is faster or slower than the old code. It’s definitely better in that it works with many more sources. Don’t really have data for it. The ESRI time is probably dominated by slow servers anyway.

Here’s the slowest non-ESRI sources. Given that these times include download times from some slow servers I’m pretty OK with them.

dk            CSV   2000s     3.4M rows
au-victoria   SHP   1620s     3.4M rows
es-25830      CSV   1173s     8.8M rows
nl            CSV   1156     14.8M rows

95 sources failed to produce an out.csv. Of those, 7 did succeed last time we ran, a potential regression. Here’s the cause of each failure:

  • us-il-tazewell: ESRI source, JSON parsing error after many calls
  • us-tx-colleyville us-tx-dallas us-tx-hurst: ESRI sources, found no records
  • us-nc: missing “file” attribute for multiple shapefiles
  • us-dc: bad download
  • us-ca-san_diego: bad zip download

And finally here’s the list of all 95 sources that didn’t produce an out.csv. I should go through and hand-classify the failures again, but I bet most of 88 of them are for the same reason as last time.

au-queensland
be-flanders
ca-ab-calgary
ca-ab-strathcona-county
ca-bc-kelowna
ca-bc-langley
ca-bc-nanaimo
ca-bc-okanagan_similkameen
ca-bc-surrey
ca-bc-west_kelowna
ca-ns-halifax
ca-on-niagra
ca-sk-regina
us-al-shelby
us-ar
us-ca-san_diego
us-co-sanmiguel
us-ct-avon
us-dc
us-fl-alachua
us-fl-collier
us-ga-glynn
us-ia-linn
us-ia-polk
us-id-canyon
us-il-tazewell
us-in-hamilton
us-in-madison
us-in-st_joseph
us-la-st_james
us-ma
us-mi-muskegon
us-mn-dakota
us-mn-metrogis
us-mn-polk
us-mn-pope
us-mn-yellow_medicine
us-mo-barry
us-mo-st_louis_county
us-ms-madison
us-nc
us-nc-1
us-nc-10
us-nc-2
us-nc-3
us-nc-4
us-nc-5
us-nc-6
us-nc-7
us-nc-8
us-nc-9
us-nc-charlotte
us-nc-columbus
us-nc-davie
us-nc-wake_county
us-ne-omaha
us-nm-san_juan
us-nv-henderson
us-nv-lander
us-nv-nye
us-nv-washoe_county
us-oh-clinton
us-oh-hamilton
us-pa-beaver
us-ri
us-tn-memphis
us-tx-colleyville
us-tx-dallas
us-tx-denton
us-tx-el_paso
us-tx-hurst
us-tx-keller
us-tx-north_richland_hills
us-tx-round_rock
us-va-alexandria
us-va-augusta
us-va-city_of_falls_church
us-va-city_of_petersburg
us-va-richmond_city
us-va-roanoke
us-va-stafford
us-wa-san_juan
us-wa-snohmish
us-wi-adams
us-wi-calumet
us-wi-crawford
us-wi-dodge
us-wi-jefferson
us-wi-juneau
us-wi-lincoln
us-wi-oneida
us-wi-richland
us-wi-sauk
us-wi-superior
us-wy-laramie

Python: logging different threads to different files

The OpenAddresses code includes a Python task manager thats spins off a lot of Python threads, one per input file describing a job. We’d like to capture the debug log records for each thread to a separate file, for presentation in a dashboard. How to do that?

This demo program I wrote shows how. The key concept here is a Python logging.Handler() object that uses some per-Thread state to decide where to write the message. In this case I use the Thread name itself, but you could also use some state stored in Thread.local().

Also this feels related.. log4j had a way to store information in ThreadLocal variables that you could then write out to log files. Ie: you could add a special per-thread variable like “job name” or “HTTP session ID” and have the formatter print it out where appropriate. It was quite handy in some use cases. Python doesn’t exactly have that, but the cookbook has notes on doing something similar with Filters.

Update: see also Mike’s alternate approach. He creates a new Handler for every thread, then uses a Filter to only show messages from the thread he cares about. I like how the approach makes more use of logging’s machinery, it is a bit weird how I had to make a whole Handler that did something odd with the output. I think Mike’s approach means for N threads we’ll end up with N Handlers seeing every log message. That’d be inefficient but if there’s not 100+ threads I doubt it matters.

OpenAddress run: time to process

Here’s how long it took to process each source. No idea how much of this time was network, how much processing, etc. Painful irony that many of the slowest ones were the ones that didn’t work. There’s no job-level timeout.

2162.325   us-tx-dallas
1970.223   us-al-huntsville
1843.934   us-nc-alexander
1826.809   au-victoria
1639.524   us-tx-houston
1511.907   us-ca-san_bernardino_county
1423.542   es-25830
1385.237   pl-dolnoslaskie
1255.783   nl
1238.894   us-mn-anoka
1115.263   us-va-fairfax
1100.821   be-flanders
1087.482   pl-lubelskie
932.043   pl-slaskie
852.481   pl-podlaskie
817.774   pl-pomorskie
803.326   pl-malopolski
789.957   pl-mazowieckie
781.457   pl-lubuskie
779.531   pl-kujawsko-pomorskie
779.183   us-wa-seattle
772.016   pl-swietokrzyskie
768.572   us-wa-king
766.155   pl-podkarpackie
736.996   us-tx-denton
726.582   pl-lodzkie
712.528   us-ne-omaha
681.070   us-pa-philadelphia
680.825   us-ms-hinds
639.296   pl-opolskie
611.966   us-va
608.180   us-ca-los_angeles_county
592.763   us-nc-3
591.630   pl-wielkopolskie
581.695   us-co-jefferson
576.934   pl-zachodniopomorskie
559.960   us-fl-bay
557.759   us-oh-hamilton
554.983   us-ca-solano_county
521.117   us-fl-pinellas
520.235   us-oh-franklin
496.750   us-fl-manatee
491.657   pl-warminsko-mazurskie
427.657   ca-on-welland
415.115   us-fl-palm_beach_county
394.993   us-ca-san_diego
378.344   us-or-portland_metro
356.776   us-nc-mecklenburg
356.656   us-nc-5
340.884   es-25829
340.322   us-tn-rutherford
319.549   us-mt
310.424   ca-nb
293.255   us-va-loudoun
284.713   us-fl-charlotte
284.701   us-nm-san_juan
280.506   us-ms-madison
279.436   us-nc-7
273.664   ar-ba-buenos_aires
260.210   us-fl-clay
259.266   us-ut
252.911   es-25831
252.718   no
250.454   us-nc-10
243.587   us-in-hamilton
239.786   ca-on-toronto
236.674   us-va-city_of_chesapeake
233.260   us-id-canyon
227.365   us-nm-alburquerque
226.377   us-oh-cuyahoga
224.207   us-vt
221.414   us-nc-9
213.543   us-ca-san_mateo_county
210.970   us-ca-kern
210.303   us-fl-miami
207.909   us-fl-hillsborough
197.618   us-mn-dakota
197.151   us-fl-brevard
190.312   us-me
189.663   us-il-tazewell
179.207   us-tx-el_paso
179.126   us-pa-allegheny
175.989   us-nc-6
166.787   us-nc-4
165.742   us-co-larimer
155.927   us-ca-santa_clara_county
155.580   us-ga-muscogee
155.365   us-fl-sarasota
152.921   jp-aichi
152.023   us-nc-burke
150.282   us-ca-sacramento_county
142.847   us-dc
138.299   us-nc-1
136.860   us-oh-stark
136.565   us-ca-el_dorado_county
130.320   us-ga-chatham
125.109   us-tx-north_richland_hills
123.697   us-oh-lucas
122.766   us-ca-marin_county
119.822   us-va-stafford
119.769   us-oh-butler
118.186   us-wi-kenosha
110.447   us-ca-orange
110.149   jp-hyogo
103.195   us-nc-cary
102.859   us-tx-austin
99.213   us-ca-santa_barbara_county
98.967   us-il-vermilion
98.502   us-mt-flathead
96.210   us-ms-copiah
95.000   us-nc-iredell
94.719   us-co-arvada
92.375   us-nc-2
91.884   us-fl-volusia
88.892   us-ca-contra_costa_county
88.881   us-co-denver
87.736   us-mo-st_louis
86.178   us-tx-waco
84.266   es-32628
81.907   us-tx-colleyville
80.419   us-oh-greene
78.767   us-oh-delaware
78.532   us-ca-palo_alto
77.562   us-ca-fresno
77.328   jp-nagano
76.621   us-oh-lake
76.158   jp-shizuoka
75.286   us-nc-orange
73.651   us-nc-avery
72.853   us-mt-gallatin
72.545   us-nc-montgomery
72.190   au-tasmania
69.991   us-or-portland
69.405   us-tn-franklin
69.186   be-wa-brussels-nl
68.425   jp-kanagawa
67.906   us-nc-8
65.562   ca-bc-vancouver
65.465   us-il-macoupin
65.074   us-ca-ventura_county
63.371   jp-fukuoka
62.161   ca-bc-kamloops
61.934   us-tx-hurst
60.508   us-wa-city_of_spokane
60.384   us-tx-keller
60.120   us-oh-clark
57.881   us-mt-yellowstone
57.463   us-oh-mahoning
57.326   us-nc-cabarrus
56.569   is
55.605   us-il-jackson
55.457   jp-osaka
55.022   jp-gifu
54.289   us-oh-columbiana
52.161   us-mo-greene
51.536   us-va-arlington
51.290   us-mn-roseau
51.153   us-co-westminster
50.761   jp-tokyo
50.466   us-oh-jefferson
49.832   us-oh-ashtabula
48.766   us-mt-cascade
48.682   ca-on-hamilton
46.963   us-fl-citrus
46.023   jp-ishikawa
46.003   us-oh-trumbull
45.902   us-ca-bakersfield
45.794   us-co-arapahoe
45.746   us-nd-cass
45.516   us-wa-skagit
44.664   us-id-post_falls
44.443   us-oh-richland
44.222   jp-nara
44.064   us-oh-wood
42.650   us-nc-buncombe
42.276   us-co-mesa
42.112   us-nc-alleghany
41.743   jp-mie
41.643   us-fl-lake
41.412   jp-niigata
41.096   us-tx-williamson
40.901   ca-qc-gatineau
40.548   us-va-james_city
40.118   us-tn-jefferson
39.818   us-mn-wadena
39.400   us-va-louisa
39.240   us-oh-tuscarawas
39.075   us-wy-carbon
38.982   us-mt-missoula
38.972   us-md-carroll
38.865   au-queensland
38.796   us-ca-carson
38.714   jp-hokkaido
38.390   us-oh-licking
37.633   jp-kyoto
37.205   us-oh-portage
37.197   be-wa-brussels-fr
37.180   jp-kagoshima
37.177   us-oh-allen
37.004   us-oh-muskingum
36.500   us-il-menard
36.194   us-il-morgan
36.190   us-ak-anchorage
36.092   us-wy-albany
35.545   us-md-howard
35.249   us-ga-gordon
35.069   us-oh-wayne
34.263   us-ca-san_luis_obispo_county
34.092   us-mn-aitkin
34.065   us-nc-moore
34.036   us-wi-green
33.545   jp-kagawa
33.531   jp-okayama
33.191   jp-miyagi
33.138   us-mn-clay
33.088   us-oh-knox
33.008   jp-ehime
32.773   jp-yamanashi
32.620   jp-toyama
32.552   jp-hiroshima
32.001   jp-kumamoto
31.514   us-va-amherst
30.983   us-oh-seneca
30.331   us-nc-alamance
30.328   jp-shiga
30.269   us-oh-madison
30.212   us-wi-trempealeau
29.878   us-mt-lake
29.704   us-wi-calumet
29.582   us-oh-miami
29.470   us-mt-ravalli
29.048   us-ca-placer_county
28.687   ca-on-windsor
28.651   us-oh-hancock
27.639   ca-on-kitchener
27.455   us-oh-sandusky
27.420   jp-miyazaki
26.766   us-oh-lawrence
26.677   us-oh-fairfield
26.407   us-mt-lake_county
26.376   us-mt-madison
25.683   us-oh-huron
25.242   us-oh-erie
24.702   us-il-jersey
24.532   us-oh-guernsey
24.509   ca-bc-north_cowichan
24.407   jp-yamaguchi
24.311   us-nc-lincoln
24.006   jp-aomori
23.981   us-il-mcdonough
23.644   us-il-clark
23.476   us-il-putnam
23.437   jp-nagasaki
22.700   us-oh-ashland
22.108   us-il-schuyler
22.053   us-mt-lincoln
21.906   us-ca-napa
21.409   us-ca-santa_cruz_county
21.344   jp-wakayama
21.323   us-oh-williams
21.184   us-va-nelson
21.170   us-oh-shelby
21.101   us-oh-darke
21.098   us-ca-butte_county
21.091   us-co-fort_collins
20.880   jp-okinawa
20.805   jp-oita
20.732   ca-on-waterloo
20.546   us-oh-ross
20.122   us-oh-athens
20.068   us-wa-chelan
19.955   us-mn-becker
19.913   us-oh-putnam
19.363   us-ky-lexington-fayette
19.221   us-nc-haywood
19.079   jp-shimane
18.733   us-nc-chatham
18.615   us-oh-fulton
18.565   us-md-st_marys
18.356   us-il-massac
18.256   us-mt-silverbow
17.888   us-wy-laramie
17.864   us-oh-ottawa
17.811   us-mn-mille_lacs
17.628   us-oh-gallia
17.486   us-oh-crawford
17.345   us-nc-granville
17.086   us-oh-washington
16.881   us-oh-defiance
16.858   us-oh-marion
16.675   us-ca-shasta_county
16.354   ca-ab-calgary
16.302   us-oh-morrow
16.300   us-id-teton
16.095   jp-saga
16.057   us-oh-highland
15.799   us-va-halifax
15.565   jp-kochi
15.555   us-oh-logan
15.456   us-oh-pike
15.177   us-oh-holmes
15.073   us-oh-henry
15.065   us-nh-jaffrey
14.664   us-oh-clinton
14.541   us-oh-preble
14.537   ca-yk-whitehorse
14.442   us-nc-union
14.295   us-ca-berkeley
14.214   us-co-aurora
13.914   us-nc-city_of_elizabeth_city
13.876   us-oh-perry
13.676   jp-tottori
13.539   us-co-loveland
13.351   ca-bc-kelowna
13.335   us-nc-davie
13.325   jp-tokushima
13.276   us-mt-sanders
13.198   jp-fukui
13.157   us-nc-caswell
13.154   us-nc-city_of_king
13.120   us-mt-teton
12.972   us-co-boulder
12.967   us-mt-blaine
12.919   us-md-cecil
12.675   us-mt-jefferson
12.447   ca-on-guelph
12.382   us-mt-phillips
12.360   ca-bc-prince_george
12.336   us-mt-stillwater
12.278   us-mt-powell
12.269   us-oh-pickaway
12.214   us-oh-harrison
12.183   us-nc-blowing_rock
11.910   us-oh-wyandot
11.700   ca-on-burlington
11.690   us-va-city_of_radford
11.661   us-mo-cole
11.637   us-oh-hardin
11.628   us-mt-broadwater
11.604   us-ca-nevada_county
11.486   us-oh-mercer
11.400   ca-bc-okanagan_similkameen
11.281   us-oh-hocking
11.195   us-oh-meigs
11.153   us-oh-jackson
11.105   us-mt-fergus
10.942   us-oh-carroll
10.685   jp-saitama
10.585   us-oh-paulding
10.562   us-mt-big_horn
10.338   us-oh-van_wert
10.256   us-oh-brown
10.208   us-mt-valley
10.177   us-mt-deer_lodge
10.039   ca-on-oakville
9.806   us-mt-custer
9.804   us-mt-chouteau
9.730   us-va-salem
9.624   us-mt-roosevelt
9.514   us-mt-granite
9.391   us-oh-fayette
9.380   us-mt-carbon
9.232   us-mt-hill
9.210   us-mn-pope
9.134   us-ca-yuba_county
9.118   us-va-greene
9.055   us-oh-monroe
8.944   us-oh-vinton
8.440   us-oh-noble
8.099   us-mt-glacier
8.032   it-er-rimini
8.016   us-ia-pottawattamie
7.963   us-ia-marshall
7.956   us-mn-renville
7.882   us-mt-rosebud
7.764   ca-pe
7.546   us-va-blacksburg
7.546   us-mt-toole
7.523   us-mt-judith_basin
7.382   jp-tochigi
7.328   us-oh-auglaize
7.275   us-oh-adams
7.253   us-mt-musselshell
7.058   us-sd
6.987   us-mt-dawson
6.959   us-va-madison
6.927   us-nc-madison
6.911   us-mt-wheatland
6.894   us-mt-richland
6.827   us-ia-dallas
6.812   us-va-clarke
6.694   us-co-pitkin
6.667   us-ia-dickinson
6.585   us-ia-webster
6.560   us-nc-city_of_carolina_beach
6.412   us-mt-liberty
6.314   us-ia-marion
6.286   us-oh-morgan
6.271   us-ia-sioux
6.261   us-mt-petroleum
6.253   us-ia-crawford
6.188   us-al-shelby
6.133   us-ia-boone
6.026   us-wa-douglas
5.980   us-ia-sioux_county
5.860   jp-fukushima
5.552   us-ia-madison
5.505   us-va-city_of_petersburg
5.420   us-mt-garfield
5.335   us-mt-sheridan
5.186   us-ia-jasper
5.139   us-wi-iron
5.108   us-ia-carroll
5.078   kr-seoul-yeongdeungpogu-old
5.047   us-nm-dona_ana-2
5.013   us-mt-park
5.005   us-va-city_of_emporia
4.999   us-ia-winneshiek
4.996   us-mt-mineral
4.915   us-ia-palo_alto
4.892   us-wa-snohmish
4.869   kr-seoul-seongbukku-new
4.812   us-mt-fallon
4.801   us-mt-pondera
4.767   jp-ibaraki
4.594   us-ia-montgomery
4.562   us-in-st_joseph
4.530   us-mt-sweet_grass
4.496   us-ia-monona
4.491   us-ia-guthrie
4.443   us-ia-buchanan
4.368   kr-seoul-yangcheongu-new
4.308   kr-seoul-tobonggu-new
4.281   us-mt-meagher
4.225   jp-chiba
4.214   us-mt-carter
4.213   kr-seoul-mapogu-old
4.210   us-ia-buena_vista
4.198   kr-seoul-kangseogu-old
4.188   us-ia-wright
4.152   us-mt-prairie
4.141   jp-akita
4.135   jp-yamagata
4.127   kr-seoul-tongjakku-new
4.121   us-ia-shelby
4.120   us-nv-las_vegas
4.111   us-fl-alachua
4.079   us-ia-harrison
4.064   us-mt-powder_river
4.019   jp-gunma
4.016   kr-seoul-seodaemungu-new
4.009   us-ia-clarke
4.000   us-ia-benton
3.970   kr-seoul-yeongdeungpogu-new
3.960   us-ia-greene
3.950   us-ia-hardin
3.937   us-ia-franklin
3.911   us-ia-bremer
3.871   us-mt-daniels
3.865   us-ia-winnebago
3.757   us-pa-beaver
3.747   us-va-fluvanna
3.731   us-mt-mccone
3.518   ca-ab-grande_prairie
3.516   us-mi-ottawa
3.351   us-va-city_of_falls_church
3.319   us-ia-cass
3.311   kr-seoul-kangnamgu-new
3.309   us-ia-kossuth
3.306   us-mn-otter_tail
3.296   us-ia-delaware
3.294   kr-seoul-tongdaemungu-old
3.288   us-tn-memphis
3.243   us-ia-monroe
3.236   us-ia-cherokee
3.232   us-mt-golden_valley
3.223   us-ia-ida
3.207   us-ia-sac
3.195   us-ia-hamilton
3.194   us-ia-howard
3.180   us-va-middlesex
3.134   kr-seoul-songpagu-new
3.115   ca-bc-victoria
3.111   us-ia-calhoun
3.077   us-ia-decatur
3.071   kr-seoul-tongdaemungu-new
3.032   kr-seoul-kangbukku-new
2.957   us-mt-wibaux
2.898   kr-seoul-mapogu-new
2.890   us-ia-mills
2.851   us-ia-emmet
2.801   us-ia-lucas
2.752   kr-seoul-seongbukku-old
2.601   us-co-sanmiguel
2.595   kr-seoul-tongjakku-old
2.552   us-ia-pocahontas
2.546   kr-seoul-yangcheongu-old
2.498   us-ia-humboldt
2.490   kr-seoul-keumcheongu-old
2.433   us-ia-lyon
2.430   kr-seoul-kangnamgu-old
2.424   us-ia-clarke_county
2.412   us-ia-fremont
2.407   kr-seoul-kwangjingu-old
2.405   us-co-gilpin
2.375   us-tx-round_rock
2.365   us-mt-treasure
2.342   kr-seoul-kwanakku-new
2.328   us-mi-kent
2.325   us-ga-glynn
2.325   kr-seoul-kangseogu-new
2.325   jp-iwate
2.320   kr-seoul-seochogu-old
2.312   us-fl-collier
2.240   kr-seoul-kwanakku-old
2.200   kr-seoul-noweongu-new
2.190   kr-seoul-yongsangu-new
2.170   kr-seoul-noweongu-old
2.159   us-ak-matanuska_susitna_borough
2.117   kr-seoul-seochogu-new
2.032   us-mn-wilkin
2.026   kr-seoul-yongsangu-old
2.024   kr-seoul-kwangjingu-new
1.954   kr-seoul-chungnanggu-new
1.950   us-in-marion_county
1.875   kr-seoul-chongnogu-new
1.805   us-va-augusta
1.805   kr-seoul-kangbukku-old
1.798   ca-bc-west_kelowna
1.792   kr-seoul-chunggu-new
1.779   us-wi-oneida
1.774   kr-seoul-songpagu-old
1.767   kr-seoul-seongdonggu-new
1.753   kr-seoul-kurogu-new
1.750   us-wi-superior
1.733   us-nc-wake_county
1.724   us-mn-yellow_medicine
1.706   kr-seoul-kangdonggu-old
1.705   ca-bc-vernon
1.694   kr-seoul-chungnanggu-old
1.692   kr-seoul-chunggu-old
1.673   kr-seoul-seodaemungu-old
1.671   kr-seoul-seongdonggu-old
1.667   kr-seoul-kurogu-old
1.654   us-wi-adams
1.649   kr-seoul-kangdonggu-new
1.566   us-ct-lyme
1.565   kr-seoul-chongnogu-old
1.532   us-ct-watertown
1.519   kr-seoul-tobonggu-old
1.463   us-va-richmond_city
1.407   kr-seoul-keumcheongu-new
1.149   us-la-st_james
1.141   us-sc-lexington
1.141   us-nv-nye
1.137   us-ia-polk
1.134   us-ca-san_francisco
1.127   us-ct-haddam
1.121   us-mo-st_louis_county
1.120   us-mi-muskegon
1.115   us-mo-columbia
1.113   us-ri
1.108   us-wi-sauk
1.108   us-va-city_of_norton
1.105   us-wi-dodge
1.091   us-co-gunnison
1.091   us-ca-alameda_county
1.090   ca-bc-nanaimo
1.089   us-la-acadia
1.088   us-wa-san_juan
1.081   us-va-alexandria
1.079   us-va-roanoke
1.077   us-sc-berkeley
1.073   us-mn-polk
1.072   us-nv-henderson
1.070   dk
1.067   us-wi-fond_du_lac
1.064   us-in-madison
1.057   us-wi-juneau
1.040   us-mo-barry
1.003   ca-ns-halifax
0.986   us-al-calhoun
0.980   us-ca-amador
0.979   us-nv-washoe_county
0.953   us-ky-oldham
0.889   nz
0.863   us-va-essex
0.780   us-wi-jefferson
0.754   us-wi-vernon
0.720   us-nc-columbus
0.676   us-ny-nyc
0.658   us-ia-linn
0.654   us-ar
0.645   us-wi-richland
0.615   ca-ab-strathcona-county
0.609   ca-bc-surrey
0.547   us-wi-crawford
0.545   us-ma
0.532   us-wi-lincoln
0.529   us-ct-avon
0.524   us-va-accomack
0.524   us-nc-charlotte
0.516   ca-bc-langley
0.513   us-nv-lander
0.483   ca-sk-regina