PS4 5.50+ USB hard drive, copying games

A tale of failures. I wanted to copy a PS4 game I have off the internal hard drive onto a USB drive, so I could copy it onto a second console instead of redownloading 60GB+ worth of stuff. This turns out to be impossible.

Since version 5.50 the PS4 has gotten a lot friendlier about supporting USB drives. Among other things you can relatively easily plug in a USB drive and copy user data (like game saves) to and from it. The PS4 will do this with any exFAT or FAT32 disc, like your garden variety flash drive.

You can also move a game from the internal hard drive to a USB. Key point there: move, not copy. Copy is explicitly not allowed. You can only move a game to disc if the disc is formatted as “extended storage” for the PS4. I thought I’d be clever and move the game to an external drive, make a clone, then move it back. But you can’t do that easily. The extended storage volume doesn’t even have a meaningful partition table, no filesystem I can mount. I suppose I could make a block level clone of the disk but that’s too much trouble. (It’s probably just some slightly bent version of a standard FreeBSD or Windows filesystem, but I’m not going to waste time researching that.)

One last option PS4 supports is to back up your whole console to an external disc, then restore it to a different console. You can also clone one PS4 to another directly over a LAN. However in both cases the destination PS4 gets entirely wiped and reconfigured, it’s not suitable for making a copy of a game.

The stupid thing about all this is Sony probably still thinks it’s doing some useful copy protection. It’s not, the copy protection is entirely keyed now to your online account. They’re just making it awkward for power users to do reasonable things.


Linux I/O schedulers

My Linux server is mostly idle, but every few hours I run a backup and then the disks get very busy. I’m also trying to rebuild a corrupted Duplicati database now, a second heavy I/O load. If both run at once the system does not run very well. Worse, interactive performance it terrible; like 30 seconds to log in and get a shell.

I think a fix for this is changing the Linux I/O schedule. I checked and all my hard drives were set to the deadline scheduler. I changed them to CFQ and interactive performance seems better, although I didn’t test carefully.

I’m not clear why CFQ isn’t something you’d just use all the time. I guess there’s some overhead associated with it vs. noop but I have to think it’s not very much under normal workloads. Deadline seems like a bad idea unless you’re in a very special environment. But I don’t really understand this stuff.

Zip codes vs census tracts

A lot of digital maps use zip codes as a binning feature. Election maps, property value maps, pollution maps. But while zip codes are convenient and familiar there’s a much better set of polygons for mapping most US data: census tracts. Zip codes are pretty unusual polygons(*), drawn mostly to make the process of mail sorting and delivery simpler. They are quite arbitrary. Census tracts are lovingly and carefully drawn to respect demographic and political reality.

I have a great example of the difference: River Oaks in Houston, one of the wealthiest neighborhoods in America. It’s the place where Saudi princes buy a 10,000 square foot house and glass in the backyard so they have air conditioning. Also plenty of local Houston money, mostly old money.

But you’d never know it looking at zip code averages. Because River Oaks shares 77019 with a bunch of people who live east of Shepherd, in a significantly denser, less wealthy, more diverse neighborhood. It also includes big chunks of the Fourth Ward which used to be quite a poor neighborhood and was recently redeveloped. By contrast River Oaks is almost perfectly described by census tracts 4112 and 4114. (Not exactly; the chunk in the SW of 4114 is not technically River Oaks, nor is it in 77019, but demographically it’s quite similar to the rest of 4114.)

Census tracts are part of a hierarchy of census polygons. Census tracts aim to have about 4000 people in them, (zip codes average 8000 people, but vary greatly). Block groups are smaller and have ~1500 people each. Census blocks are the smallest division, about 40 blocks per group, but unlike census tracts they are highly variable (in terms of statistical demographics). There are 11M census blocks in the US, although nearly half of them are uninhabited.

Census tracts are frequently named by 4 digit numbers like 4112 or 4114, with a county name or something else to disambiguate them. But some tracts have numbers with decimals or other complicating factors, like 32 or 204.01. (I believe canonically census tracts are named with 6 digit numbers, with the decimal and leading zeros frequently omitted).

FIPS codes are the larger naming scheme; the full name of 4112 is sometimes given as 48201411200. 48 is Texas, 201 is Harris County, 4112 the census tract, and the 00 is the block group. Census blocks get their own 4 digit number at the end, making for a full 15 digit code to name the most precise polygon.

(*) technically speaking zip codes are actually just lists of points. The polygons are properly called zip code tabulation areas and there are some important differences.

Ubuntu 18, Samba, and ntlmv1

I have a security camera that writes videos to my Ubuntu 18 machine, mounting the disk via Samba. it stopped working recently. Turns out Bionic Beaver shipped a tiny little security patch which includes this:

  * SECURITY UPDATE: Weak authentication protocol allowed
    - debian/patches/CVE-2018-1139-*.patch: Do not allow ntlmv1 over SMB1
      and add tests.
    - CVE-2018-1139

They’re not wrong to change this; ntlmv1 is a terrible ancient insecure authentication protocol. But it’s just the kind of thing some old firmware is likely to require, like my camera.

The simple workaround is to restore ntlmv1 by adding this to the smb.conf:

ntlm auth = yes

Don’t do this if you don’t trust your network. For that matter don’t run Samba at all if you don’t trust your network. I mean it’s supposed to work, but I sure wouldn’t.


Munin and NVMe SSD drives

Summary: Munin seems to have no support for monitoring the life of NVMe SSD drives.

NVM Express is the new hotness for SSD drives.  Those are the little M.2 cards you plug in that are actually hard drives, but with more direct access to the system bus than the usual SATA interface. They are fast and compact and great. I’d like to use Munin to graph some stats on my drives, to see how busy they are and how close they are to wearing out.

Munin has an excellent SMART plugin for monitoring important drive health measures. Unfortunately it doesn’t work for NVMe drives. smartctl only started supporting NVMe drives at all last year, and they did so with a brand new output format that looks totally different from the old tables that smartctl knows how to parse.

It’d be simple enough to make a new plugin to grab data from the new format. Main thing holding me back is I don’t really understand what all the values in the SMART report really mean. I think “Available Spare” and “Data Units Written” are probably the most interesting fields along with the various error columns, but I dunno.

Here’s an example output

 # smartctl -A /dev/nvme0
smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.15.0-23-generic] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke,

SMART/Health Information (NVMe Log 0x02, NSID 0xffffffff)
Critical Warning:                   0x00
Temperature:                        33 Celsius
Available Spare:                    100%
Available Spare Threshold:          10%
Percentage Used:                    0%
Data Units Read:                    16,973 [8.69 GB]
Data Units Written:                 488,334 [250 GB]
Host Read Commands:                 463,571
Host Write Commands:                16,861,501
Controller Busy Time:               19
Power Cycles:                       13
Power On Hours:                     80
Unsafe Shutdowns:                   1
Media and Data Integrity Errors:    0
Error Information Log Entries:      0
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0
Temperature Sensor 1:               33 Celsius
Temperature Sensor 2:               33 Celsius

Firefox Sync

Firefox Sync is nice. Nice enough I’ve switched to using Firefox on iOS. Almost entirely for the ability to send a tab to my mobile device. This happens to me all the time; I have something interesting or useful on my desktop browser and I want to look at it on my iPhone. Now instead of emailing myself a URL awkwardly I can just send the tab to the iPhone and it’s done.

The other feature I like is the ability to open URLs that are or were recently open in another browser. That way I can grab something of interest on my iPhone even if I forgot to send it from the other computer. This feature’s less useful to me since I tend not to leave tabs open. Also the sync seems to be minutes out of date, so sometimes what I want isn’t available.

Firefox Sync shares a lot of state: bookmarks, saved passwords, browser history, installed addons, etc. It all seems pretty useful and it all seems to work well.

Update: by default it syncs every 15 minutes. You can configure it.

Twitter streaming API workarounds: lists and following

Twitter is shutting off their streaming API. After months of warning, the actual deadline is in three days, Thursday the 16th. If you were using the streaming API to follow at most a few thousand accounts, there’s a workaround: Twitter lists.

The idea is you create a Twitter list containing all the accounts you want to follow. Then you poll lists/statuses for the timeline of tweets. Polling isn’t awesome but the default rate limit is once a second, so it’s not too bad. Twitter lists are limited to 5000 accounts, so if you need more than that this won’t work (but consider multiple lists).

The main thing you’ll lose this way is replies; Twitter lists don’t include replies not to list members. (Or maybe not to people you follow, I’m confused on that point.) But otherwise it seems to work. If you’re working with old Twitter code be sure you understand about extended_tweets; you almost certainly want to be getting them.

One wrinkle: creating a list of thousands of people isn’t easy. It looked to me like there was a hard limit of adding 1000 users to a list every day. Also maybe a soft limit of hundreds in a short period of time.

The other option I tried was instead of a list, create an account and follow the users of interest. Then look at their home timeline. This doesn’t work as well for various reasons. First, Twitter mucks with a home timeline more than a list timeline, more risk of reordering and stuff. Also the rate limit is lower (once a minute). Finally it seems harder to quickly follow thousands of new accounts, I ran into more draconian rate limits.

One more wrinkle: listing or following a user notifies them you are interested in them. Following people via the streaming API was completely passive, now users have a hint they’re being watched. It’s not a big deal but some users may choose to block you when they get listed / followed.