Happy multiprocessing

I managed to get multiprocessing working for OpenAddress job processing. Looks like it’s working pretty well. We use multiprocessing.Pool to manage worker processes, then set up a SIGALRM to abort jobs that run too long. (Using a nifty decorator to add the timeout). I also added a SIGUSR1 handler in the master job process that lets us abort the run. At least once I’ve had the pool fail where there was work being done but none of the worker processes was trying to do it.

Here’s a happy screenshot of htop showing a run going. Note the nifty custom process names courtesy of setproctitle.

Screen Shot 2015-01-18 at 3.30.17 PM