Uber Eats Canada Ran Production in a MacBook Air | Hacker News Reader

Uber Eats Canada Ran Production in a MacBook Air

twitter.com

133 pointsdvrp98 comments

OP — I’m not that privy to the details of what in the world was running on that MacBook Air, but I’m 100% sure it wasn’t “the production servers for Uber eats Canada.” It was some kind of hack doing some kind of thing on a recurring basis to make sure the service continued. These things were often hacked together by engineers, and sometimes ops people, in the middle of outages, as a stop gap.

yep definitely pulled this move before. Probably on a laptop because it is the only thing that has permissions to do whatever insanely janky thing it’s doing.

Sounds like something probably in the System keychain, that's not getting unlocked when the user is logged out.

Incidentally, the command to unlock the system keychain (eg in a cronjob) is:

  systemkeychain -vt

That doesn't seem to be widely known, and is very useful for things like build jobs. ;)

Not that I condone this sort of thing, but folks who do need to do it should know about caffeine [0] which allows a Mac to disable it's lid switch and keep running with the lid closed.

[0] https://www.macupdate.com/app/mac/24120/caffeine

Side note, Caffeine[1][2] does not disable it's lid switch, but instead disables the automatic sleep behaviour while enabled.

[1] Author link (includes link to source): https://www.intelliscapesolutions.com/apps/caffeine

[2] Original author link (stopped updating before OSX Mojave): https://www.lightheadsw.com/caffeine/

i switched from caffeine to amphetamine due to compat issues and amphetamine does have a mode to disable the lid switch

Or you can use the built-in and more powerful commands: pmset and caffeinate.

E.g. you can prevent sleeping until another process exists, or initiate sleep or shutdown.

There's a difference between "hosting all of Eats Canada" and "hosting some component or service until we can fix prod." I think it's more likely similar to the latter

That was my thoughts as well. Maybe someone from Uber Eats Canada can clarify.

    while true; do ssh intern@eats.uber.ca 'find /var/log -type f -delete'; sleep 300; done

This tracks. At a former employer, one of their acquired startups (that I joined on to help guide technical direction for) ran their entire order flow by calling a url every minute via a cron task from the primary database server.

At one point, the newer devops organization migrated to a bigger, beefier MySQL (this is when fast NVMe was first available in specialized EC2 instances).

Nobody remembered to migrate the cron script at that moment, so for about an hour I ran all of customer order flow with a bash script curling a public facing (yikes) url to initialize those jobs.

Probably was running a script that ssh'ed into production boxes and restart some service at some specified interval or cleared some log files to stop the drive from filling up.

“until we can fix prod”, but then other stuff comes up, and just look at the backlog! And, I mean, it works pretty well for now…and before you know it, it’s been there forever

that’s nothing! how about air traffic control in a text editor? https://www.reddit.com/r/emacs/comments/lly7po/comment/gnvzi...

What a gem!!

Ouch. German ATC software in the 90s seems to have been a special place. (My favorite war story is "Hey, we have somebody 10 miles out from FRA airport, low on fuel, ATC software is down, can we escalate" - yes, yes we can.)

Mayday, mayday! Hello, can you hear us? Can you hear us? Can you ... Over... We are sinking! We are sink...

Hello, this is the German Coast Guard.

We are sinking! We're sinking!

What are you 'sinking' about?

For those who didn’t get this gem - https://youtu.be/gmOTpIVxji8

woah

emacs is practically an operating system.

So no surprise here really.

It surely would have run more smoothly as an elaborate vimrc file considering how much cleaner and simpler it is to use.

^[[^[[^[[^[[:wq^[[^[[::wq!^[[:wq!

It ran something that would cause Uber eats Canada to go down if it got disconnected, but not all of Uber eats production in one laptop (imo)

So basically: all of the horror with none of the impressiveness.

I mean if the options you’re facing are Uber Eats Canada running because of something happening on a laptop or Uber Eats Canada not running, it’s not a hard choice to make.

Fix the underlying problem later — or don’t.

I’ve worked for big companies in the past where mission critical cron jobs were run on a Linux vm running off an old little desktop tucked away on a benchtop in the corner that no one ever noticed. Done to bypass various network controls to allow us to fix problems as they popped up. Absolute cowboy stuff, but effective.

One place I worked at had some data science stuff that pulled data from prod on a cron and re-uploaded the results every day, on some core2duo in the back of the office. AFAIK it was still being used in 2017 and might still be lol. It was a pretty powerful machine because someone got it to see how much money he could get the company to spend.

Maybe Ian wanted to ensure that no one touched his laptop.

I’m not surprised. Amazon payment service’s reconciliation and settlement system was run off of a developer desktop.

My memory is hazy as it was 18 years ago but I think it was due to some IP restrictions by banks and Amazon didn’t have tech to migrate IPs across machines.

I totally believe the job was on a desktop, but I’ll call bullshit on the IP reason. The desktops don’t have direct external IPs allocated, so there’s a NAT with standard DHCP. There was no IP migration issues to deal with on the internal network.

It’s funny how people assume things. It seems odd that the assumption would be that this laptop runs production. I don’t think anyone who builds things with tech would assume this.

That title is quite the garden-path sentence - I got halfway through rationalizing how Uber would eat a Canada-ran production until I figured it out.

It’s called clickbait.

Any respectable company would've used a Mac Mini.

My current company, within the past few years, ran the production web server on a couple of mac minis (and like, not exactly new ones either).

Seemed to work ok, honestly. Our DB boxes, which do much more heavy lifting are not now (and weren't then) Mini's, or anything close, though, heh.

Why? The MacBook has a built-in UPS.

Was just kidding, but in all seriousness, the Mini has slightly better performance and cooling (not by some huge margin), and it's easier to connect accessories. But yeah, having a battery might be good for some use cases.

Legality aside I’d love how many people you could serve with $2000 worth of hardware (used or new) and a 1gbit up/down home internet connection With max p99 latency of 500ms.

It is strange that they didn’t at least use a Mac mini, but I assume they already had the hardware.

Built-in UPS and peripherals are nice advantages of a laptop as a server.

The problem with a laptop is terrible cooling that you can't sustain high performance. They say new Macbooks are better in that aspect but didn't have a chance to try one.

Yeah. M1/2 ones are a lottt better than Intel in this regard but still slower than the M1/2 MBP or mini.

Good point.

I wonder if it's because the laptop sorta has its own built in UPS.

I ran one of the popular web sites in Turkey on a AMD Duron 600Mhz 512MB RAM server until 2003. It used to serve 70 million pages per month on a Windows Server when it was retired. Because the site is a social platform, most of the content had to be dynamically generated too. I believe the PC had cost me around $500 when I bought it in 2000 or so, obviously a lower spec than even a Raspberry Pi4 today.

And then people started writing code in Python.

Exactly. Developers seems to be hell bent on making software as slow as possible and/or maximising the cost of running servers. I bet that 99% of businesses out there could comfortable run on a single server with lots of RAM. With another servers on warm stand-by.

It's more fun to try and figure out where the bottleneck would be.

Back in the early 00s I hosted websites on my local off the shelf Dell desktop on a cable internet connection.

Cloud computing wasn't really a thing and I didn't know better / have much better options (eventually I did migrate to some webhost bundled with the DNS registar).

For the longest time network bandwidth was my limiting factor. I remember finally upgrading my connection, NIC, etc. to the point where it was actually harddrive write speed that was the bottleneck.

Probably depends on your application but harddrive I/O does seem like a likely limiting factor to me today.

SSDs are pretty fast.

Hardware wise, kinda depending on the characteristics of the app. Network wise, I've seen this tried. Commercial internet, even if it is symmetrical 1Gibps has nearly no SLA tied to it and is subject to traffic shaping. Not only could you be turned off at the companies whim if you began to saturate the points up to your demarcation point, but they could also traffic shape you for a time (or indefinitely). This all largely has to do with how internet telephone companies and cable companies categorize and pool traffic in their regions up to backbones.

Also save for the fact that commodity hardware lacks a lot of the hardware safeties a server does. For instance, your m.2 under sustained load would not perform at the same rate that an enterprise server hard drive can; we're talking on an order of 180MiB/s vs 550MiB/s per drive. Once the m.2's cache is exhausted it will be comparable to a spinning disk in performance.

It seems that you might be assuming the entire stack ran on that laptop which is probably not the case.

I’m assuming it’s a service, so I’m curious how many people the Mac is responsible for serving. Of course more likely to be innocuous and maybe not in the serving path at all

I was just reading in "Losing the Signal"[1] about how Blackberry's relay service of the late 1990s was sometimes run on a sysadmins laptop just because they needed anything to help deal with the traffic they weren't equipped to deal with.

[1] https://us.macmillan.com/books/9781250096067/losingthesignal

...over wifi??

the picture is from like 8 years ago.

Thanks for clarifying...updated.

This was obviously temporary but is actually cool for showing how powerful modern machines actually are. People routinely underestimate the load that can be handled by one box properly configured and running efficient code.

I’ve found that modern developers almost always overestimate the amount of physical hardware needed to run things. I’m not sure why.

I was once interviewed by Lyft and I told them that a 64 GB Mac Book Pro is sufficient to run the core business.

The interviewer didn't believe me.

I did the calculations in front of him.

This was the time, I just learned Rust.

I did the full analysis and showed him that QPS is great.

Of course, my design didn't include 500 microservices, 50 deployments of Hadoop, Kafka, Elastic, Mongo DB, and 500 slightly different dashboards that different teams in Lyft probably use. Most modern tooling in ZIRP phenomenen.

Someone's Hadoop experience - https://adamdrake.com/command-line-tools-can-be-235x-faster-...

And did you get the job?

I did. I rejected them as I didn't believe in the future of taxi-app industry.

How do you define "core business"? Any laptop os would crumple purely by running out of resource handles on the connections, even without doing work. I've run into file descriptor / handle size limits on small clusters before, I can't imagine trying to deal with 1m+ concurrent tcp connections.

MacOs is a terrible server OS, unless you want to YOLO lack of synflood protections. That said, if 64GB macbookpro means 64GB of ram, that's enough for a lot more than 1M file descriptors. I'd still tend to pick other hardware; I don't see the sense in buying Apple hardware to run a different OS; especially as they move farther away from mainstream hardware.

The Mac Studio running Linux seems like it'd actually make a pretty decent server. Not the most cost-effective option but not terrible when you also factor in power consumption.

Apple should really license the M architecture for servers. They could even have someone else manufacture and deal with delivering it. They're leaving free money on the table. I don't think it would cannibalize their Mac sales at all.

"core bussiness" is finding and dispatcching drivers including payments. Everything in-app.

> I can't imagine trying to deal with 1m+ concurrent tcp connections. 2 million connections on a BSD mchine is doable though not trivial and that too even in 2012. https://blog.whatsapp.com/1-million-is-so-2011

You'd think that with all the advances in hardware, computers would get faster. No, this modern computer takes 20x longer to open the calculator then windows NT

...but, WiFi?

That part I agree is a bad idea. At least plug a USB Ethernet card into it.

Don't worry, they moved it to the opposite side of the kitchen from the microwave, so no connectivity problems.

This a reference to the weird signal from outer space that turned out to be a microwave?

This is a reference to the fact that the 2.4GHz Wi-Fi signal is in the same frequency emitted by the kitchen microwave oven.

Microwave means 0.3 to 300 GHz.

Oh, I was thinking about this, then https://www.theguardian.com/science/2015/may/05/microwave-ov...

You should probably also keep your very expensive and sensitive space equipment at a safe distance from any microwave too, what I was trying to say I guess

I'm perfectly fine with using laptops as edge servers in remote areas (Oh did you say Canada?). The thinkpad W series were workhorses, you could get 32GB of RAM and they had built in UPSs. Not that confident about Macbook Airs though, didn't they use to overheat?

Not even an MBP. Professionalism is dead.

Would love to hear the story behind this

Basically ran a macro to adjust how far restaurants would deliver by time of day / day of week.

The goal was to balance supply and demand of restaurants and orders.

Eats product didn't have any demand shifting levers at that time yet restaurants were getting blown up with demand at peak times (ex. top pizza restos at 7pm on a Friday).

The product did however allow ops teams to manually set a delivery radius for a given restaurant.

Clever ops teams scripted shrinking / expanding this delivery radius to encourage demand at non-peak times and reduce it during period where restaurants were overloaded and likely to ignore / cancel orders.

This laptop ran scripts for all cities in Canada to adjust delivery radii for each restaurant until a few different levers were built into product.

Ian (who is a _super nice_ guy) owned this process which kept things working smoothly while Eats was experiencing explosive growth.

Another fun fact - Eats Marketplace, which is the current Eats you see today, was built and launched in Toronto - SF engineers / product / design supported by Toronto ops teams. Quickly eclipsed the Instant Eats product which was the 3rd (and most successful to date) try by Uber at food delivery.

lmao... figured it was radius reduction or somesuch. i ended up working on Winterfell (the service that managed automated radius reduction) at some point during my time there.

When Maersk got ransomwared, they had to use a surface tablet for the domain controller of the company until things went back to normal.

That’s really interesting. Any links?

even for 2015 (when this was likely taken) Uber Eats would most certainly not have run from a single laptop. Core services like dispatch, matching, pricing etc in any case weren't geography specific. If i had to guess, this was some Ops person running a local driver incentive, payout query + email workflow from their laptop.

I ran a national self-moving company’s website in the early days off of an original iMac with Apache on Mac OS X Server 1.0 (not a typo). Mostly static pages with a bit of PHP/CGI/SSI back then, but still zippy with a ton of traffic thanks to our obsession with file size. Highly unprofessional!

As Morty would say, “Sounds like a cron job but with extra steps”

I'll be honest, while it probably didn't run the whole production system it probably did something important enough to at least get wired internet.

So all the microservices are running on that one system?

Not clear but very unlikely.

I too, have deployed my laptop to production

(Desktop app, but...) whenever I or another developer would say "it works fine on my computer", my boss would sarcastically reply "so, we should put your computer on the customer's site then?"

And that's how docker was built.

too bad docker for mac isn’t very good. maybe someday.

Uber's market cap is 93 billion right? What the fuck are they doing with all that money

The shares worth that amount are sat in members of the public’s trading accounts.

Given that Uber Eats is losing money on every order the evaluation might actually be that high because, not despite of the laptop given that it can thankfully only do so many orders at any given time

Uber is not making money. Which makes it not a business. More like a scam to extract $ from investors.

Semi off-topic, anyone else get annoyed when people use the word "ping" to mean "contact (me)"?

Sadly I've never had the chance to ask "Ah, are you ICMP compliant?" in response to a "ping me!"...

I assume someone does, but my friends and family will readily respond with "ack" or "pong".

my reply is usually something like

    64 bytes from eventhorizon77: icmp_seq=1 ttl=64 time=31m

I think the origin of that ping is the "ping" sound that various instant messengers made/make on receiving a message, not some nerd reference to internet protocols.

Ping me when you have a chance to use that comeback, I'm curious to know how it goes.

Jokes aside, I use it when I'm chatting (IM) with other net-ops people, it just basically means "get ahold of me however you think is best based on the urgency".

Blackberry Messenger had a ping built in. Nothing since has had an equivalent feature. It was absolutely amazing for getting someones attention urgently....

And for annoying friends in their meetings.

Such a strange way to run a kubernetes cluster.

Practical.

Reply on news.ycombinator.com