Hacker News Reader

Comment by dsl | Hacker News Reader

Answering the security question specifically: v8 is a runtime and not a security boundary. Escaping it isn't trivial, but it is common [1]. You should still wrap it in a proper security boundary like gVisor [2].

The security claims from Cloudflare are quite bold for a piece of software that was not written to do what they are using it for. To steal an old saying, anyone can develop a system so secure they can't think of a way to defeat it.

1. https://www.cvedetails.com/vulnerability-list/vendor_id-1224...

2. https://gvisor.dev/

View on HN

Docker went through an era of thinking they could built multitenant systems that amortized the cost of resources across customers. Java went through an era of thinking they could build multitenant VMs that amortized resources across customers. Unix went through an era of thinking they could build multitenant processes that amortized resources across customers. As did Windows.

I point out that Docker is trying - and failing - to offer us the same feature set we were promised by protected memory, preemptive multitasking operating systems 30 years ago. I'm still waiting. I don't recall if there were old farts complaining about this in the early 90's, but I suspect they exist.

Also Java had a bunch of facilities for protection and permissions that I've not heard anyone claim exist in V8. Not that feature equality is required (in fact that may seal your doom), but I don't see feature parity either.

edit: some of facilities Java investigated do exist in cgroups, so one might argue that there is a union of v8 and cgroups that is a superset of Java, but unless I am very mistaken, isolates don't work that way, so isolates are still not the way forward there.

>cost of resources across customers.

IMHO, this is all a solution chasing a problem. A server from Hetzner costs the equivalent of 30 minutes of my time a month. A VM on a shared machine can cost even less. There's just not cost there to save in order to justify the security risks and performance ghosts. It only makes sense for extremely bursty loads or background processing where latency isn't important. But those are pretty atypical scenarios and usually buying for max load is still cheaper than the engineering time to set up the system.

> A server from Hetzner costs the equivalent of 30 minutes of my time a month.

My toy code, which takes ~6ms (at p75) to exec, runs in 200+ locations, serves requests from 150 different countries (with users reporting end-to-end latencies in low 50ms). This costs well below $50/mo on Cloudflare, and $0 in devops. Make what you will of that.

how many request a month

It has seen everything between 10rps to 10000rps, and never once complained. Latencies remain steady, requests remain serviceable, abuse (DDoS/bots) still gets caught, throughput just seems to scale elastically even if not boundlessly.

What problem did that solve?

Have you seen Oceania and Africa egress prices?

One server may be 30 mins of your time. But whether that's cost efficient depends on your domain.

There's B2B work typically characterized by a low number of very high value transactions, so you run a few servers, and your math works out.

And then there are consumer services such as Instagram or Google maps or TikTok that see a huge amount of very low value requests that only contribute a small trifling in ad revenue. When your service needs to process millions of requests to pay for your service, your math no longer works out.

One of the beauties of modern computing is that our economies of scale enable us to deliver complex services such as Instagram etc at cost, for free. That's pretty amazing if you think of it, but engineers need to be very stingy for this to work out.

The math still works out because there's not much cost savings to be had in using V8 instead of VM/physical machine instances to share physical equipment cost. It's worth that small extra bit of money to pay to have your code running with the extra bit of protection.

> I point out that Docker is trying - and failing - to offer us the same feature set we were promised by protected memory, preemptive multitasking operating systems 30 years ago.

You were given them, by those facilities, 30 years ago.

Then someone started fiddling with CPU design, and things got weird.

I find it funny how we've ended up here:

"Let's use containers, that way we aren't running unnecessary redundant kernels in each VM!"

later

"Oh shit, our containers share a kernel, let's add another kernel to each container to fix this!"

Maybe the next step is that we realize there's so many CPU bugs, that we really just need to give each container their own hardware :)

That's what cloud-native processors are for.

> Maybe the next step is that we realize there's so many CPU bugs, that we really just need to give each container their own hardware :)

I am reasonably sure that most of the micro services I write would be very happy running on a 400mhz CPU with a couple hundred megs of RAM, if they were rewritten in native code, or even just compiled to native code instead of being ran on top of Node. Throw it all on a minimal OS that provides networking and some file IO.

How much does it cost to manufacture 400mhz CPUs with onboard memory? Those must cost a few pennies each, throw in a 4GB SSD, preferably using SLC chips for reliability, and a NIC, and sell 4 "machine" clusters of them for ~$100 a pop.

> [...] if they were rewritten in native code, [...] Throw it all on a minimal OS that provides networking and some file IO.

You may want to check out MirageOS[0]. It gives you a library OS with the primitives you say you need, and then all you have to do is import them in your application code as if you are writing your typical OCaml, build the virtual appliance and boot it up anywhere you want.

[0] https://mirage.io/docs/overview-of-mirage

There's also https://github.com/includeos/IncludeOS

However, how much is this kind of stuff is actually used at scale today?

Well we’ve kind of already started exploring that option with unikernels. Compile each app as its own complete software stack.

https://en.m.wikipedia.org/wiki/Unikernel

Nice! Thanks for the link. As I've been playing around building a toy OS for my Raspberry Pi and learning more about hardware virtualization, I was thinking exactly along these lines, i.e. instead of virtualizing an full OS, virtualizing a minimal image with just a network driver and a flat address space per process/app seems like it would have benefits (i.e. no virtual memory mapping or privilege ring transitions eating cycles). I wasn't aware it was already an area that had much research behind it.

From the article:

> For example, off-the-shelf applications such as nginx, SQLite, and Redis running over a unikernel have shown a 1.7x-2.7x performance improvement

That's a pretty healthy boost. Are there any significant efforts close to general use anyone is aware of in this space? (Or good reasons why it's a totally terrible idea and hasn't gone anywhere!)

Not first hand experience, but I was adjacent to a team that tried this approach. In short its playing on hard mode. Dev experience and tools, deployment chains, testing, etc are all possible but nowhere near your awesome userspace toolchains. Be prepared for lots of DIY action like building your own network or file io. The team also wasnt able to quite get down to the superfast ms start times and MB images that we were going for.

My memory is fuzzy but ISTR that the people behind seastar/osv moved on to an approach closer to what mrkurt is emphasizing with “container” composition plus firecracker for the environment.

To be fair to Clouldflare, they do a lot of very thoughtful work on sandboxing at all levels, including at the process level as well as above and below that. A lot of interesting details here:

https://blog.cloudflare.com/mitigating-spectre-and-other-sec...

The article focuses on Spectre, but it also mentions general stuff.

edit: It is true they are using V8 for a novel purpose, but a lot of their techniques mirror how Chrome sandboxes V8 (process sandboxing, an outer OS sandbox, etc.).

I have a lot of respect for Kenton, but the facts are: v8 security team recommends running untrusted code in separate processes [1], and as described in your link, Cloudflare doesn't do that. From what Kenton said earlier on this[2], it sounded like they had already committed to their architecture before that v8 recommendation was made. They thought through a lot of possible attacks and created some clever mitigations, and ultimately decided "its probably ok".

[1] https://v8.dev/docs/untrusted-code-mitigations#sandbox-untru...

[2] https://news.ycombinator.com/item?id=18280061

eta: Also, Chrome does use process isolation - every browser tab runs in a separate process. The whole point of Cloudflare isolates is to avoid the overhead of a fork() for each worker request.

Also Cloudflare has already broken the Internet once by doing unorthodox things.

Sometimes the guy missing a finger is exactly the one you want to take safety advice from. Sometimes it's just a matter of time before he's missing three fingers.

I don't think what Chrome is doing is quite that simple? When you click on a link in a tab, it might take you to a different site. Also, there can be iframes from different websites in the same browser tab.

Also see "limitations" on this page:

https://www.chromium.org/Home/chromium-security/site-isolati...

That link tells you this:

> Cross-site documents are always put into a different process, whether the navigation is in the current tab, a new tab, or an iframe (i.e., one web page embedded inside another). Note that only a subset of sites are isolated on Android, to reduce overhead.

Chrome's use of strict process isolation is a fairly new thing. For most of its history, it was trivially easy to open any other site in the same process as your own site (via window.open(), iframe, etc.), and V8's sandboxing was the thing protecting those sites from each other. So V8 was, in fact, designed for this.

When Spectre hit, Chrome concluded that defending the web platform from Spectre in userspace was too hard, so they decided to go all-in on process isolation so that Spectre was the kernel's problem instead. This is a great defense-in-depth strategy if you can spare the resources. On most platforms where Chrome runs, the additional overhead was deemed worth it. (Last I heard there were still some lower-powered Android phones where the overhead was too high but that was a while ago, maybe they've moved past that.)

That doesn't mean V8 has decided their own security doesn't matter anymore. Securely sandboxing the renderer process as a whole isn't trivial, especially as it interacts with the GPU and whatnot. So it's still very much important to Chrome that V8's sandbox remains tight, and the renderer sandbox provides secondary defense-in-depth.

When it comes to Cloudflare Workers, the performance penalty for strict process isolation is much higher than it is for a browser, due to the finer-grained nature of our compute. (Imagine a browser that has 10,000 tabs open and all of them are regularly receiving events, rather than just one being in the foreground...) But we have some advantages: we were able to rethink the platform from the ground up with side channel defense in mind, and we are free to reset the state of any isolate's state any time we need to. That lets us implement a different set of defense-in-depth measures, including dynamic process isolation. More details in the blog post linked earlier [0]. We also did research and co-authored a paper with the TU Graz team (co-discoverers of Spectre) on this [1].

I may be biased, but to be perfectly honest, the security model I find most terrifying is the public clouds that run arbitrary native code in hardware VMs. The VM software may be a narrower attack surface than V8, but the hardware attack surface is gigantic. One misimplemented CPU instruction allowing, say, reading physical addresses without access control could blow up the whole industry overnight and take years to fix. Spectre was a near miss / grazing hit. M1 had a near miss with M1racles [2]. I think it's more likely than not that really disastrous bugs exist in every CPU, and so I feel much, much safer accepting customer code as JavaScript / Wasm rather than native code.

[0] https://blog.cloudflare.com/mitigating-spectre-and-other-sec...

[1] https://blog.cloudflare.com/spectre-research-with-tu-graz/

[2] https://m1racles.com/

A few corrections:

1. Chrome has always had a multi-process architecture, with at least one dedicated process for each tab's renderer (which hosts the v8 runtime).

2. Chrome was already in the process of moving cross-origin iframes to separate processes when Spectre was announced. The announcement of Spectre style vulnerabilities just significantly increased the pace of implementation and deployment.

Yes, I didn't mean to imply that Chrome wasn't multi-process all along. But Chrome historically used it more for fault isolation (= crashing one tab doesn't crash them all) rather than security isolation. Historically window.open() would (usually) open a new tab that actually ran in the same process, and iframes ran in the same process, so getting any arbitrary site running in the same process as yours was easy.

1/ is not correct. I worked on Chrome for many years. Processes were frequently reused across tabs for many reasons, including the window.open() example that kenton mentioned earlier, but also just resource exhaustion.

As you scale up with R2, D3, Email Workers etc, is it possible that the _future_ scale of touching very sensitive, "ought-to-be-secure / separate" data / code help reconsider this decision?

Without process isolation, all it takes is one bad Chromium commit or a incorrectly allow-listed JS/V8 command for this model to fall through regardless of how thorough the defense-in-depth/vetted Workers may be.

"the public clouds that run arbitrary native code in hardware VMs." -> Isn't it double whammy then that a V8 isolate + HW attack surface in combination could provide an order of magnitude more hackability?

"Last I heard there were still some lower-powered Android phones where the overhead was too high" -> I believe a Zygote-process fork was made available to Chromium at some point (https://chromium.googlesource.com/chromium/src/+/HEAD/docs/l... ?).

> all it takes is one bad Chromium commit

All it takes is one bad Xen commit, or one bad design choice at Intel, for VM-based clouds to be broken. Every isolation mechanism has bugs, this is not unique to V8.

> for this model to fall through regardless of how thorough the defense-in-depth/vetted Workers may be.

No no, when I say "defense in depth" I mean we have fallback contingencies to contain the damage of a V8 bug.

> Isn't it double whammy then that a V8 isolate + HW attack surface in combination could provide an order of magnitude more hackability?

No, the opposite. Having to break two layers at the same time is much harder than breaking just one thing. VM-based clouds are largely counting on just one layer today (most of which is burned into silicon).

Browser requirements may differ from Cloudflare's, though. In particular, browsers run arbitrary code from anywhere - just clicking a link downloads and runs code off of anywhere on the planet, theoretically - while Cloudflare runs code from their customers.

They do have a free tier, so yes, there are risks here. But one mechanism they do according to that post, IIUC, is to keep paying customers separate from the free tier. So if you are a paying customer, any attacker against you is at least someone that Cloudflare has the credit card information of. Still dangerous, but very different than the browser risk model.

It’s worse than the browser threat model. In a browser, security-maximalist users can choose to only visit trusted sites, which reduces the attack surface somewhat, though far from completely (there’s still exploits hidden in ads and watering hole attacks to worry about). With CloudFlare, you’re keeping out… people without credit cards? What kind of even remotely serious attacker can’t obtain a stolen credit card, or else sign up for a credit card with false identification? At most the credit card restriction prevents attackers from creating large numbers of accounts, reducing the chance they’ll be co-located with you, but even that can probably be circumvented by sufficiently sophisticated attackers.

You're right that credit cards are far from guarantees. Still, I'd imagine Cloudflare do something like cluster users by risk level at some granularity. Brand-new paying accounts might be put together in the same process, while paying customers for over a year might be in another. You might be attacked even in the latter group, but the attacker can do that at a much lower frequency (the card isn't recently stolen; and they've been paying for this attack for a year - only to be kicked out now if found).

Also, the browser risk model is much worse because of the scale. If you manage to get malicious code onto a popular website then you get many, many opportunities to attack people. With Cloudflare you can only attack the other tenants of your process, and you're paying every time you do it (in a browser, it runs on the target's machine, free for you). And even then as the blogpost mentions they randomize pairings, limit the amount of time they spend together, etc.

Overall I think this is far less risky than the browser situation. And Cloudflare isn't the only one that thinks so - their competitors also put multiple tenants in the same process. Done carefully, it's a reasonable model.

V8 is still very hard to escape -- an escape is a remote hole in Chrome isn't it? To be honest, I'm happy to base my security on "as safe as Chrome".

If the patch gap is small, yes. But are you patching V8? Node generally isn't.

As a user of CF workers, the API is clearly designed to allow the runtime to redeploy e.g., for patches.

Example: there is no notification when your worker is unloading. It just dies. This is sometimes annoying as a developer, but it's the kind of decision a platform makes when it wants (needs) to be able to shut down code arbitrarily and restart it later on a different version of the platform.

Chrome runs each origin in a different process. Chrome also runs all the privileged code in separate processes, so like file, network, GPU, and many OS calls are not actually happening from within the same process as v8 executing web JS code. Process isolation is also necessary to mitigate CPU and OS level timing attacks between origins (and the privileged process).

v8 is a pretty good security boundary, but to be as secure as Chrome you need the other layers too.

Hi Elliot! I think it's more complex than that.

Chrome (all web browsers) have a massive and ancient attack surface -- all of HTML, JS, CSS, every DOM and web platform API, every image codec, video, and on and on and on.

CF workers have a comparatively minuscule attack surface that takes into account all the most recent lessons (high-res timers, for example).

So yeah, if we call Chrome of 2008 a baseline, then Chrome's more recent process isolation increases security. But so to does CF's much reduced attack surface.

I wouldn't say those are equivalent. Any exploit in v8 gets full access to the process, which in a multi tenant situation means access to other tenants. And v8 itself is hardly bug free.

Reducing the JS API surface might reduce the probability of finding a way out, but it doesn't reduce the blast radius. 2008 was also 14 years ago, I'm not sure that's the right bar for security!

The OP was also speaking to using v8 getting you the same level of security as Chrome, not to architectural decisions in CF beyond that.

> an escape is a remote hole in Chrome isn't it

Maybe. Chrome has a lot of extra security hardening you don't get if you are just running the V8 engine alone.

The going rate for a V8 escape is probably around $100k at the moment. Totally worth it financially to snarf credit cards from all the sites on Cloudflare.

Personally, I think I'd recommend SELinux and SecComp before I use something like gVisor. There's significant performance impact with application kernels.

Those are not mutually exclusive, but if you use a syscall blocklist only you will have to refuse certain workloads

I've been hoping a runtime would arise for this use case explicitly, and v8 is close. But the solution has to have multitenancy, limiting of memory and cpu, and security baked in from the beginning. The lua runtime seems promising, but again, not designed for that.

We are going the long way around to figure out that Tannenbaum was right and we need microkernels. The IPC costs cause sticker shock, but at the end of the day perhaps they are table stakes for a provably correct system.

I was reading the Wikipedia page about this debate and I find this low-level stuff so fascinating. Wish I had done CS instead of Physics.

At a low level and at a place of ignorance, I kind of wish monolithic kernels hadn't won, just like I wish the morbidly obese browsers hadn't either... They sound way more, well, sound in their theoretical basis

There's a few clever tricks that seem to make a pretty big difference with microkernels, but I know just enough to be dangerous.

I think one of the biggest ones I recall, and I think it was part of the L4 secret sauce, was the idea of throwing a number of small services into a single address space, so that a context switch requires resetting the read and write permissions, but not purging the lookup table, making context switches a couple times faster. With 64 bit addressing that gets easier, and allows other tricks. For instance recent concurrent garbage collectors in Java alias the same pages to 4(?) different addresses with different read/write permissions. Unfortunately this means your Java process can 'only' address 4 exabytes of memory instead of 16, but we will surely persevere, somehow. How that plays with Meltdown and Specter might put the kibosh on this though, and most processors don't actually support 64 bit addressing. Some only do 44, 48.

That's pretty old news though. The more interesting one to me is whether there's a flavor of io_uring that could serve as the basis for a new microkernel, allowing for 'chattier' interprocess relationships to be workable by amortizing the call overhead across multiple interactions.

I am still waiting for a seL4 based router, firewall, DNS server, file server, etc. Even better if we could get it into critical infrastructure (water treatment, electricity, etc). Something that is exposed to the insanity of the internet, but may not necessarily have high performance requirements. We have super computers in our pockets, we can take some performance hit if it eliminates large security concerns.

Something like MirageOS looks pretty cool

I know V8 from back to front and I feel that it is quite safe, after all, there's not much you can do with a javascript interpreter ...

Regarding the CVEs mentioned, the overwhelming majority of them have a "denial of service" effect, not sure how something like gVisor helps you mitigate that.

If you know it back to front, then you are likely aware of the number of TurboFan vulnerabilities that have been found being exploited in the wild.

Sure, and most of them have been addressed properly.

Honestly (as another commenter said), V8 being behind one of Google's largest revenue streams makes it feel "safe enough" for me, they have all the resources and incentive to make it as safe as realistically possible.

Of course it's not perfect, but is there anything better at the moment?

With all their resources and incentive, Google have chosen to rely on process isolation in Chrome, not just V8.

Process isolation (security) belongs to the OS so there's nothing stopping one from doing the same, also, it shouldn't be much difficult to implement.

Reply on news.ycombinator.com