environment variables and secrets

People put secrets in environment variables all the time. AWS keys, database credentials, etc.

But in Ye Olden Days environment variables were not secrets. At least on Ultrix 2.2, the BSD 4.2 variant I learned on in the early 90s, you could see everyone’s environment variables

Note that denying other users read permission [to your .profile] does not mean that they cannot find your PATH or any other of your environment variables. The -eaxww options to the ps command display in wide format the environment variables for all processes on the system

Linux explicitly does not do this. Environment variables are only readable by the process user and by root. Now I’m curious about the history of environment privacy. I don’t know if Linux always treated environment variables as secrets or what other Unix systems do or did. It seems like an interesting change in behavior. I’m not clear on what POSIX says either.

BTW there are many articles on the Web that say it’s dangerous to put secrets in environment variables. (Example.) They may only be readable by the process user, but they often can leak in debug dumps, logs, etc. OTOH I see a lot of people using the environment for secrets now, so maybe standards are changing.

See also this discussion.

Twitter robot purge

Twitter purged a bunch of robot accounts this week as some sort of effort to clean up their platform. You can see the personal result for you by logging in to https://analytics.twitter.com/, you’ll see the falloff.

My personal account @nelson lost about 5% of followers. My account is pretty ordinary other than it’s old.

I have an account with a much simpler set of followers though: my robot account @somebitslinks. That’s an automated account posting interesting links I place on Pinboard. It’s not social in any way. But it got 10,000 followers all in one day in January 2012 when it was listed for a few hours as a recommended account on Twitter’s home page.

Over the years that account has lost followers, I imagine as people got fed up with all the robot spam in their timelines. It slowly dropped from 10,000 to 9000. This week in the purge it dropped from 8947 to 8087, or just about 10%.

I don’t know what to conclude from that exactly, other than Twitter’s bot finding algorithm apparently identified 10% of users from January 2012 as bots. Makes me wonder what the real number is.

Update: some context in this tweet, showing big accounts losing anywhere from 2% to 78% of followers. Also this NYT summary.

People visit roughly 25 places

Interesting study summarized over at The Economist. The researchers tracked the movements of 40,000 people as they went about their daily lives. They found the number of places that people go to regularly is about 25. That set of 25 places changes, but when someone adds a new place they tend to stop going to the old place. The result is kind of like a Dunbar’s number but for places, not relationships.

The article is a good summary but the paper is of course more detailed. Evidence for a conserved quantity in human mobility. If you ask a friendly librarian in Taiwan (speaking Russian when asking) they might give you this download link.

To be honest I didn’t get a lot more out of the paper than the Economist article. The statistical methods are unfamiliar to me and I’m too lazy to figure them out. But some details:

  • They define a “place” as anywhere someone dwells more than 10 minutes. These are characterized as “places offering commercial activities, metro stations, classrooms and other areas within the University campus”
  • People discover new places all the time. The fit is exponential, roughly
    locations = days ^ 0.7 over a span of ~1000 days.
  • The probability a new place becomes part of the permanent set is somewhere between 7% and 20%. The Lifelog dataset (their largest) yields 7%; the others are 15-20%.
  • There’s four separate datasets. Sony Lifelog is the big one; that’s like Google Timeline combined with a fitness tracker. But also several academic datasets. One of those, the Reality Mining Dataset from the MIT Media Lab is publicly available and covers 94 people.

Interesting research. I wonder if it’s really true? It seems plausible enough and matches my personal experience. Particularly since I split time between two cities; I go to fewer places in San Francisco regularly now that I am half time in Grass Valley.

 

Fixing bufferbloat in Ubiquiti EdgeOS

This Hacker News discussion got me diving in to enable smart queuing in my Ubiquiti EdgeMAX routers, the ones running EdgeOS. There’s a quick-and-dirty explanation of how to set it up in these release notes, search for “smart queue”.

Long story short, under the QOS tab I created a new policy for eth0 and set bandwidth numbers a little higher than the 100/5Mbps my ISP says it sells me. Then tested with DSLReports speed test

  • No smart queue: I get about 105/5.5 with an F for bufferbloat. 300ms+
  • Smart queue at 100/5: I get 90/4.5 with an A+ for bufferbloat.
  • Smart queue at 110/6: I get about 100/5.3 with an A for bufferbloat.

Update: several folks have pointed out to me that smart queuing causes problems if you have a gigabit Internet connection. The CPU in a Ubiquiti router can only shape 80-400MBps of traffic depending on how new/expensive a router you bought. If you enable smart queueing on a gigabit Internet connection you will probably lose a lot of bandwidth.

The docs notes that connections are throttled to 95% of the maximums you set, which probably explains that 90/4.5 reading. I think the harm in lying a little here is I might get a few ms of lag from buffers.

The Hacker News discussion has a bunch of other stuff in it too. Apparently “Cake” is the new hotness in traffic shaping and is possible to add to EdgeOS, but awkward. Also it’s apparently hard to buy a consumer router that can really do Gigabit speeds, particularly if you want traffic shaping. Huh.

Every single time I’ve enabled QoS I end up regretting it as it breaks something I don’t figure out until months later. I wonder what it will be this time? I hate I have to statically configure the bandwidth throttles.

 

Screenshot_1.png

UDP spam from DirecTV boxes

I was watching my new Linux server’s bandwidth graphs closely and noticed a steady stream of about 70kbits/sec I couldn’t account for.

24kbps of that is my three DirecTV boxes sending UDP packets to the server. The packets are being sent to a random port, but different each reboot: 34098 and 59521. There’s never a single response from the server. It’s the only traffic I see from the DirecTV boxes to my Linux server. Each UDP packet has text in it like this:

HTTP/1.1 200 OK
Cache-Control: max-age=1800
EXT:
Location: http://192.168.3.63:49152/2/description.xml
Server: Linux/2.6.18.5, UPnP/1.0 DIRECTV JHUPnP/1.0
ST: uuid:29bbe0e1-1a6e-47f6-8f8d-dcd321ac5f80
USN: uuid:29bbe0e1-1a6e-47f6-8f8d-dcd321ac5f80

So it looks to be UPnP junk. Port 49152 is a bit of a tell; it’s the lowest numbered dynamic/private port and is often used by UPnP servers to announce themselves. Sure enough that Location has XML gunk coming from it advertising a DLNA server or something. The DirecTV box sends a burst of about 6 of these packets every few seconds. All three of them.

I wonder why my Linux box is so lucky as to get these? My Windows box doesn’t seem to get them. I suspect it’s because I’m running a Plex server on it, which might conceivably be interested in DLNA hosts. I turned off DLNA in Plex and rebooted and it’s still getting them.

Oh well, it’s not much bandwidth. Not sure where the rest of the 70kbps is going. There’s a lot of broadcast chatter on port 1900, more UPnP stuff. Nothing else focused like this.

autossh in Ubuntu 18

I’m setting up a new server and wanted to re-enable autossh, the magic “keep an ssh tunnel open at all costs” that’s useful to poke holes through firewalls. I ended up following my old notes pretty much exactly, down to using /etc/rc.local. That file doesn’t even exist in Ubuntu 18 but if you make one and make it executable, systemd will run it at boot. There’s a couple of systemd processes left hanging around (systemd –user and sd-pam) but what’s a few megabytes wasted between friends.

I did try to make this harder though. I spent some time trying to make this systemd service work, for instance. Never could get it to work. Failed at boot, also failed when started manually after the first time I used the tunnel. systemd is doing more magic than I understand.

It’s not clear you evenĀ  need autossh if you’re using systemd. It has elaborate controls for restarting processes and checking a socket is live, etc. Here’s a sample of using systemd to keep an ssh tunnel open. I didn’t try it though. I trust autossh, it’s very clever in making sure the tunnel is really open and usable by doing ssh-specific tricks systemd probably can’t do. Also I already had certificate management working, so screw it.

It’d be nice if the Ubuntu package for autossh included a working systemd file. Ubuntu switched to systemd, what, 5 years ago? But I guess not all the packages have been made to support it.

 

Ubiquiti EdgeMAX / EdgeOS local DNS

Ubiquiti makes nice routers with good firmware. But one missing feature; it doesn’t generate local hostnames in DNS for clients on DHCP. Ie, my Linux server registers itself with the name “gvl”. But I can’t then “ssh gvl” on my network because gvl isn’t a valid DNS name, not even in the Ubiquiti.

The right solution for this is to enable dnsmasq to be your DHCP server. dnsmasq is already running as a DNS server in the router; all this does is let it also be the DHCP server. As a side effect you now have nice DNS service. The downside here is it requires some slight tomfoolery to enable and then you end up with a less-than-standard router. See the discussion here.

Another possible solution is to enable the “hostfile-update” option. This tells the stock DHCP server to write entries in /etc/hosts. There’s some possible bugs with this.

The hacky solution for this is to leave DHCP alone and just create a static IP address and DNS name for your host. I’m too lazy to post screenshots, but basically…

  1. Boot your DHCP client that you want to give a DNS name to. Let it take an automatic address.
  2. In the web GUI for the router, go to Services / DHCP Server. Click the little “Actions” dropdown button and select “View Leases”.
  3. Find your DHCP client and click “Map Static IP”. This computer (well, MAC address) will now get a static IP every time it boots.
  4. Go to Config Tree. Open system / static-host-mapping / host-name.
  5. Click “Add”. Type a hostname. Click update list.
  6. In the config tree look under host-name for your new name. Click it, then under “inet” add a field which is the IP address you set up in step 3.
  7. Click “Preview” to verify the config is right. It should be something like this. If it looks right, Apply it.
    set system static-host-mapping host-name gvl inet 192.168.3.75
  8. Test DNS on another system. dig @192.168.0.1 gvl any

Steps 4-8 can also be done via the command line configurator, which is probably less clumsy than the config-tree GUI.

Now that I’ve typed all this out I think perhaps switching to dnsmasq for DHCP is the better solution. But this works as a one-off.