Hacker News Reader

Comment by monkeybutton | Hacker News Reader

Only tangentially related, but I solved a problem at my work this week with a couple of cleverly crafted MILP programs. Its really exciting in the nerdiest way possible and I need to share it with someone, anyone! I can't write a blog post about it because I don't have a personal website, plus it'd be a trade secret anyways. Most of my coworkers aren't technical so they're happy to get an answer to their problem, but they don't really care about the how part.

View on HN

The coolest instance of MILP I have encountered "in the wild" at work was a disaggregation problem where we could only receive vendor reports that contained aggregated statistics, each report aggregating the original data along a different dimension (say one by customer zip code, another by age range, etc). By lining up these aggregated reports, you could turn that into an MILP system that could be solved to unwind all of the aggregation and get back to the original data. This gave us way more flexibility in deciding how to use the vendor data.

Was the purpose of aggregating the data to "anonymize" it? If so, this just illustrates how useless anonymizing data is.

Yup, k-anonymity is hard. State of the art is rapidly evolving, requiring higher and higher k (i.e. more coarse/less useful reports). Most companies out there prefer to, um, not notice that.

Yes, and I mostly agree. Most companies just randomize an ID or do some group-bys and consider it "anonymized", which are susceptible to a variety of correlation techniques (such as setting up this MILP problem). And the more data sets you have with the same underlying masked primary key (e.g. person), the easier it can get. I have seen these weak attempts at hiding information in multiple industries. Just for the record, in my use case there was never an intent to map data to individuals, just to get the most granular information possible about key metrics.

Hello, would this have been on vendor reports from a certain large online advertising and search company?

Ha yes, I probably know you.

Write it up. Even if you can’t publish it now, you may not recall later. It’s quite likely circumstances will change such that your current employer has no interest in blocking publication so if you already have a document you’ll be able to advance human knowledge.

Doesn't hurt to ask for written approval from your manager to write about it! I'd love to read it as well. Getting to use optimization algorithms at your job is super rare (unless you're in specific industries).

I actually found a few use cases recently.

Solving for the funding plan of a bank to balance the balance sheets and meet all the regulatory constraints (it actually uses a linearisation of a piecewise linear convex problem that I had learned in college decades ago, but completely forgot about, I asked for guidance from one of my professors who asked for permission to use the case as an exercise for his students).

Assigning cash flows to changes in balances when the two feeds are inconsistent and the timing is off (you need to do some complex combinations to match them).

But same as OP, you get a sense of self-satisfaction but that won’t get you any credit internally.

Super rare indeed! For me, it seems to be around one every couple of years. Just long enough that it becomes a novelty again.

Please do write it up! It's always interesting to read how others solve problems using similar tools. If you have a github account you can always gist it. If not, perhaps one of the prolific writers in HN would be willing to publish it for you?

> I don't have a personal website

It's so easy to make a free blog on github (like sub 5 minutes) with some version of jekyl. You should probably just do it.

Was it a classic scheduling problem (e.g. distributing resources)?

I suppose so. A first pass on which resources to acquire with different costs and different (and diminishing) rates of return. Then a second pass to assign those resources to workers who have different capacities to do work, and also can't do every kind of work (so bin packing?).

Reply on news.ycombinator.com