Anyway, long story short, my friend did some measurement and thinking and realized the problem. His server makes API calls that get a big JSON response from some other server. He extracts some data from that JSON and then throws the rest of the response away. Only it looks like the responses were never getting GCed. Best guess; the data he kept is a substring of the original JSON response, and V8 implements substrings as pointers into memory for the original interned string. So that original string can’t be collected.
His solution was to gzip the data he needed. Which is smaller in memory, but even better means that it’s a real new string and not a copy. It seems to have fixed the problem.
Update: this may not have been the problem or a correct diagnosis at all. He’d been setting –max_old_space_size incorrectly; fixing that may have fixed everything.