PS4 6.02: more external storage woes

As I wrote in an earlier post, on a PS4 there’s no way to make a copy of a game’s download files to an external drive. You can move the files to a drive but they are then deleted from the source. Which is a huge PITA when the game requires a 90 GB download.

But it gets better. If you plug an external drive with a copy of a game into a PS4 that already has a copy of that game, the software freaks out. It insists you delete one of the two copies before you can use the external drive. There’s no way to tell it to ignore the duplicate to, say, let you get at some other game on the external drive. You must delete first.

So not only can you not create copies of games, but if you screw up you’ll be forced to delete a copy you downloaded. Argh!

(I reiterate; none of this is about copy protection; the PS4 online DRM will prevent you from playing the game if your login doesn’t have a license for it, whether you have a copy on the drive or not.)

PS: the external drive copying is awfully slow. a 50GB game image is taking 18 minutes, or about 50 MB/s. That’s just about USB 2.0 speeds. The drive, cable, and the PS4 itself are all supposed to support USB 3.0. Maybe it’s the spinning drive speed limiting things.

Yet more Python ORMs

I continue to fail to use ORMs with Python. They’re just so complicated and mysterious. But maybe that’s really just SQLAlchemy and I should look elsewhere.

PeeWee popped up on my radar recently, an ORM explicitly designed to be small and simple. It looks pretty good. Also have heard a couple of people mentioning PonyORM recently. It seems far too magic.

Going even simpler, I should circle back and spend more time with requests and/or dataset. They’re not even really ORMs, just convenience wrappers for database rows, and that seems fine by me. Still bugs me they both depend on SQLAlchemy even if the actual interaction is minimal.


PS4 5.50+ USB hard drive, copying games

A tale of failures. I wanted to copy a PS4 game I have off the internal hard drive onto a USB drive, so I could copy it onto a second console instead of redownloading 60GB+ worth of stuff. This turns out to be impossible.

Since version 5.50 the PS4 has gotten a lot friendlier about supporting USB drives. Among other things you can relatively easily plug in a USB drive and copy user data (like game saves) to and from it. The PS4 will do this with any exFAT or FAT32 disc, like your garden variety flash drive.

You can also move a game from the internal hard drive to a USB. Key point there: move, not copy. Copy is explicitly not allowed. You can only move a game to disc if the disc is formatted as “extended storage” for the PS4. I thought I’d be clever and move the game to an external drive, make a clone, then move it back. But you can’t do that easily. The extended storage volume doesn’t even have a meaningful partition table, no filesystem I can mount. I suppose I could make a block level clone of the disk but that’s too much trouble. (It’s probably just some slightly bent version of a standard FreeBSD or Windows filesystem, but I’m not going to waste time researching that.)

One last option PS4 supports is to back up your whole console to an external disc, then restore it to a different console. You can also clone one PS4 to another directly over a LAN. However in both cases the destination PS4 gets entirely wiped and reconfigured, it’s not suitable for making a copy of a game.

The stupid thing about all this is Sony probably still thinks it’s doing some useful copy protection. It’s not, the copy protection is entirely keyed now to your online account. They’re just making it awkward for power users to do reasonable things.


Linux I/O schedulers

My Linux server is mostly idle, but every few hours I run a backup and then the disks get very busy. I’m also trying to rebuild a corrupted Duplicati database now, a second heavy I/O load. If both run at once the system does not run very well. Worse, interactive performance it terrible; like 30 seconds to log in and get a shell.

I think a fix for this is changing the Linux I/O schedule. I checked and all my hard drives were set to the deadline scheduler. I changed them to CFQ and interactive performance seems better, although I didn’t test carefully.

I’m not clear why CFQ isn’t something you’d just use all the time. I guess there’s some overhead associated with it vs. noop but I have to think it’s not very much under normal workloads. Deadline seems like a bad idea unless you’re in a very special environment. But I don’t really understand this stuff.

Zip codes vs census tracts

A lot of digital maps use zip codes as a binning feature. Election maps, property value maps, pollution maps. But while zip codes are convenient and familiar there’s a much better set of polygons for mapping most US data: census tracts. Zip codes are pretty unusual polygons(*), drawn mostly to make the process of mail sorting and delivery simpler. They are quite arbitrary. Census tracts are lovingly and carefully drawn to respect demographic and political reality.

I have a great example of the difference: River Oaks in Houston, one of the wealthiest neighborhoods in America. It’s the place where Saudi princes buy a 10,000 square foot house and glass in the backyard so they have air conditioning. Also plenty of local Houston money, mostly old money.

But you’d never know it looking at zip code averages. Because River Oaks shares 77019 with a bunch of people who live east of Shepherd, in a significantly denser, less wealthy, more diverse neighborhood. It also includes big chunks of the Fourth Ward which used to be quite a poor neighborhood and was recently redeveloped. By contrast River Oaks is almost perfectly described by census tracts 4112 and 4114. (Not exactly; the chunk in the SW of 4114 is not technically River Oaks, nor is it in 77019, but demographically it’s quite similar to the rest of 4114.)

Census tracts are part of a hierarchy of census polygons. Census tracts aim to have about 4000 people in them, (zip codes average 8000 people, but vary greatly). Block groups are smaller and have ~1500 people each. Census blocks are the smallest division, about 40 blocks per group, but unlike census tracts they are highly variable (in terms of statistical demographics). There are 11M census blocks in the US, although nearly half of them are uninhabited.

Census tracts are frequently named by 4 digit numbers like 4112 or 4114, with a county name or something else to disambiguate them. But some tracts have numbers with decimals or other complicating factors, like 32 or 204.01. (I believe canonically census tracts are named with 6 digit numbers, with the decimal and leading zeros frequently omitted).

FIPS codes are the larger naming scheme; the full name of 4112 is sometimes given as 48201411200. 48 is Texas, 201 is Harris County, 4112 the census tract, and the 00 is the block group. Census blocks get their own 4 digit number at the end, making for a full 15 digit code to name the most precise polygon.

(*) technically speaking zip codes are actually just lists of points. The polygons are properly called zip code tabulation areas and there are some important differences.

Ubuntu 18, Samba, and ntlmv1

I have a security camera that writes videos to my Ubuntu 18 machine, mounting the disk via Samba. it stopped working recently. Turns out Bionic Beaver shipped a tiny little security patch which includes this:

  * SECURITY UPDATE: Weak authentication protocol allowed
    - debian/patches/CVE-2018-1139-*.patch: Do not allow ntlmv1 over SMB1
      and add tests.
    - CVE-2018-1139

They’re not wrong to change this; ntlmv1 is a terrible ancient insecure authentication protocol. But it’s just the kind of thing some old firmware is likely to require, like my camera.

The simple workaround is to restore ntlmv1 by adding this to the smb.conf:

ntlm auth = yes

Don’t do this if you don’t trust your network. For that matter don’t run Samba at all if you don’t trust your network. I mean it’s supposed to work, but I sure wouldn’t.


Munin and NVMe SSD drives

Summary: Munin seems to have no support for monitoring the life of NVMe SSD drives.

NVM Express is the new hotness for SSD drives.  Those are the little M.2 cards you plug in that are actually hard drives, but with more direct access to the system bus than the usual SATA interface. They are fast and compact and great. I’d like to use Munin to graph some stats on my drives, to see how busy they are and how close they are to wearing out.

Munin has an excellent SMART plugin for monitoring important drive health measures. Unfortunately it doesn’t work for NVMe drives. smartctl only started supporting NVMe drives at all last year, and they did so with a brand new output format that looks totally different from the old tables that smartctl knows how to parse.

It’d be simple enough to make a new plugin to grab data from the new format. Main thing holding me back is I don’t really understand what all the values in the SMART report really mean. I think “Available Spare” and “Data Units Written” are probably the most interesting fields along with the various error columns, but I dunno.

Here’s an example output

 # smartctl -A /dev/nvme0
smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.15.0-23-generic] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke,

SMART/Health Information (NVMe Log 0x02, NSID 0xffffffff)
Critical Warning:                   0x00
Temperature:                        33 Celsius
Available Spare:                    100%
Available Spare Threshold:          10%
Percentage Used:                    0%
Data Units Read:                    16,973 [8.69 GB]
Data Units Written:                 488,334 [250 GB]
Host Read Commands:                 463,571
Host Write Commands:                16,861,501
Controller Busy Time:               19
Power Cycles:                       13
Power On Hours:                     80
Unsafe Shutdowns:                   1
Media and Data Integrity Errors:    0
Error Information Log Entries:      0
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0
Temperature Sensor 1:               33 Celsius
Temperature Sensor 2:               33 Celsius