NFS vs Samba

My music stops playing when I write a large video file to my server.

More technically, when I write 10MB/sec via NFS to my Ubuntu 14.04 server, the Samba server on the same host (and same disk) stops serving files reliably to my Sonos system.

The server’s definitely not CPU bound, nfsd is taking 5% of one CPU. It’s probably IO bound though. The NFS client doing the writing is MacOS with the automounter, so I’m guessing it’s UDP. I wonder if I’m simply flooding my 100Mbps ethernet with NFS traffic and the TCP Samba packets can’t get through?

This happens rarely enough I’m not going to put more time into figuring it out. Sure is an interesting failure mode though.

 

Setting up motion and an Axis camera

A few months back I bought a simple network camera for low-key home security and then never set it up. Well I set it up now and it works. Some notes.

The camera is an Axis M1034-W. It mostly works well. Right out of the box pretty much; acquires an address via DHCP (ethernet first, until you configure wifi). Immediately serves a video stream once it’s up and has a straightforward web configuration interface. That’s the good part. The bad part is the only video stream that works is Motion JPEG (ie: a bunch of JPEGs). There’s support for H.264 but it requires a Quicktime plugin to view, a nearly dead technology. Also a video stream is all I can get. The camera has its own motion detection stuff but it requires a Java applet (!) to configure. So that’s all dead web technology. But the basic camera and HTTP viewing works.

I ended up using motion to monitor the camera and publish images and movies of motion. My original idea was to use ZoneMinder, but that install uses PHP and MySQL and seemed like overkill. motion just writes image files to a directory (and optionally, a database). It does depend on avconv / ffmpeg, but that’s all.

Configuration of avconv was pretty simple. Install it in ubuntu, set up a /home/motion, set up a /home/motion/.motion/motion.conf with the config. The key configuration directive was “netcam_url http://192.168.0.XXX/mjpg/video.mjpg”, to tell it how to find the Motion JPEG stream. You can also have it poll static JPEGs via a URL like http://192.168.0.XXX/axis-cgi/jpg/image.cgi?resolution=1280×800 but I’m imagining the motion JPEG stream is better. I also had to set up the username / password for the camera, and some output options. There’s a lot of choices for tuning the motion detection algorithm, file writing frequency, etc.

The main tuning I did was set the Axis camera itself to only serve 640×480 @ 10fps. Sad to lose the quality, but I don’t want to flood my wireless network with 18 mbps. We’re pushing about 2 mbps as it is; motion JPEG is a really stupid way to send video when you expect nothing to change. My Mac Mini is chewing about 6% CPU processing this data.

When I have more time I’m going to look to see if the Axis camera itself can usefully do the motion detection and video encoding. I think it can, just gobstopped by the Java applet requirement. I mean, really?

Non-cryptographic hashes

 

About once a year I need to hash some data to give it a name / short checksum. I don’t need a cryptographic hash. But I end up using SHA-1 or MD5 anyway because it’s so available. A non-crypto hash should be ~100x faster to process data, which could actually matter. But finding a standard, portable, non-cryptographic hash with optimized implementations for Python, Node, etc is a challenge.

A lot of history of hash functions is producing 32 bit values on 32 bit CPUs. That stuff is mostly obsolete. 32 bit hashes are not useful for many applications; you expect hash collisions after just 2^16 ~= 65,000 objects. And 32 bit CPUs are largely irrelevant now. You really want a micro-optimized machine code implementation for AMD64 that is aware of cache geometry, pipeline optimizations, etc. It may be now that you want a GPU implementation; you certainly do for high throughput crypto hashes. Not clear if it matters as much for non-crypto hashes and deployment realities mean you may not have a GPU at all.

Stock Python has the hashlib library, which includes crypto hashes like MD5 and SHA-1. It’s highly optimized. There’s also CRC32 and Adler32 hiding in the zlib library, but they are only 32 bit hashes and therefore not useful.

MurmurHash3 is the usual recommendation for a non-cryptographic hash. It’s been around a long time and widely implemented. It’s also a bit confusing, with both 32 and 128 bit versions and multiple Python implementations.

There’s a bunch of “nextgen” hash functions out there. Metrohash and xxHash both look promising on first glance.

SMHasher is the standard test and benchmark suite for hash performance. I’d love to read a carefully done comparison of hash functions out there in contemporary languages and deployment environments.

some ffmpeg bash functions

I’m doing a lot of trashy video editing on the command line. I’m taking FLV files from a screen recorder, trimming them to a time frame and converting them to MP4. And then I’m concatenating multiple MP4 files. This is all very confusing to remember how to do in ffmpeg so here are two shell functions I use to do the work for me. There’s nothing particularly clever here except the <() process substitution for the catmp4 program, which creates a temporary file (or named pipe or something) with the output of a command.


function tomp4 {
    ffmpeg \
      -i "$1" \
      -c:v libx264 \
      -preset slower \
      -crf 19 \
      -ss "$2" \
      -to "$3" \
      "${1%.flv}".mp4
}

function catmp4 {
    ffmpeg \
      -f concat \
      -safe 0 \
      -i <(for f in "$@"; do echo "file '$PWD/$f'"; done) \
      -c copy \
      out.mp4
    ls -l out.mp4
}

Ubiquiti cranky customer

Very happy with my Ubiquiti network hardware. Unimpressed with the software experience. Just sent this cranky email to their support, where I’m sure I will get a canned “thanks for the note” reply and nothing will improve.

Your products have three problems that harm the perception of security. Y’all make excellent network hardware, but your software should be as excellent. I’ve already worked around these problems, just registering a complaint.

The Unifi Controller Mac application is not signed. MacOS refuses to launch it until I disable Gatekeeper security.

The Unifi Controller does not come with a valid SSL certificate. I realize this is complicated, but it’s a bad experience to require users to disable Chrome SSL checks to use the product.

Your reset password links from ZenDesk are being flagged as spam by Gmail. They say “Why is this message in Spam? It has a from address in ubnt.com but has failed ubnt.com’s required tests for authentication. Learn more”.

Thanks for listening,
Nelson

Bash arithmetic syntax

For years I’ve been doing bash arithmetic like this

echo $[2+2]

TIL I’m supposed to be doing it like this

echo $((2+2))

Who knew? Everyone, and apparently since 1996 with the release of bash 2.0. The $(()) syntax goes back to the 1988 Korn Shell, which predates when I learned Unix. But I learned most of my fancy shell scripting in the 1989–1994 time frame, when bash was the new hotness and it felt edgy using advanced features unimagined by Bourne (blessed be his name). And I preferred bash to ksh.

I learned this when someone edited one of my StackOverflow karma factories, on how to generate random numbers in bash. I updated the syntax in my answer. (Stack’s moderation review decided it was a bad edit. They were wrong.)