I Hacked Google App Engine: Anatomy of a Java Bytecode Exploit | Hacker News Reader

I Hacked Google App Engine: Anatomy of a Java Bytecode Exploit

blog.polybdenum.com

281 pointsarkadiyt41 comments

This is impressive! For folks who are worried about what this says about App Engine's security - historically the JVM level sandboxing was one part of an overall sandboxing story (not surprisingly, very similar to Chrome's setup) and my understanding, although haven't been on the team for a decade at this point, is that the JVM-level restrictions have been effectively replaced by a gVisor-based solution.

In-process sandboxing is falling out of favour.

Code Access Security has been removed from the .NET framework [1]. And Java's SecurityManager is deprecated for removal in future versions of Java [2].

[1] https://docs.microsoft.com/en-us/dotnet/framework/misc/code-...

[2] https://openjdk.java.net/jeps/411

Yeah, the way we adopted threading and dynamic libraries for plugins is on the way out and we are back into multiple process with IPC, for more stability and security.

The downside is they are more memory hungry.

> In-process sandboxing is falling out of favour.

Javascript and webassembly still aim for for in process sandboxing, though browsers add process level security as defense-in-depth.

They do not, at least not in the way "in-process sandboxing" means in the context of Java's Security Manager, which is now being considered for deprecation and removal. Java's Security Manager does not create a single set of permissions for the entire Java program, but rather assigns different permissions to different Java classes running in the same program and even on the same thread, through a mechanism that assigns different permissions to different subroutines on the call-stack, composing them in some elaborate way. The same operation can be allowed or blocked depending on the call-stack by which it is reached. For example, writing to a specific file or socket could be allowed if the call stack contains a trusted subroutine that is known to ensure the access is done "safely" but blocked otherwise.

The reason this is now being considered for removal is that even though this mechanism is very flexible and powerful in theory, it is too elaborate to be used correctly by most programmers in practice.

There is a replacement in sight?

No, because Java's security hasn't been based on the Security Manager sandbox for years, and very few people use it (and almost no one uses it for sandboxing untrusted code -- what it was designed to do). There will be replacements, and in some cases there are already, for other (ab)uses of the Security Manager, such as tracking I/O events.

That actually sounds awesome... A bit like the setuid bit on unix for running the ping utility always as root (normal users can't receive ping packets), but with a whole lot more flexibility.

I can see how careless use of it can lead to an unintentionally rather massive security surface area though...

Yeah, it's a great idea and a clever design, but sophistication is often the enemy of security: http://www.cs.cmu.edu/~clegoues/docs/coker15acsac.pdf

Yes, it is pretty awesome. I am was sort of sad to hear that it is slated for deprecation and removal. After the Java focus shifted from running in the browser to running on the server, the security manager was seldom used.

There was a lot of interest in early days of Java and the internet in mobile code that could move between devices. It seems that interest in such systems has waned. I suppose this was more of a solution looking for a problem. However, I still think that there is potential there in some way.

Well now we have WebAssembly being marketed for such purposes as if it was the first of its kind.

Meanwhile the first security papers already started being shown.

There are a couple of WebAssembly efforts that run things in-process, such as Cloudflare Workers.

In process sandboxing was definitely one of the worst ideas attempted in the early days of Java. None of my jobs in the last 10 years have ran with the security manager enabled, as it's just useless. I'm suspicious of any modern technology attempting to do the same thing, as it's just hard to get right.

When IBM developed Websphere for the 1996 Atlanta Olympics, they wanted a mainframe, but you know, web stuff. The model was doomed from the start, as they designed the JVM to run as a process.

Sun attempted tried to get deeper kernel support for the JVM with Solaris, but ultimately just ended up developing the precursor to LXC Containers called Solaris Zones.

What we have today with containers is pretty awesome but it was definitely a bumpy road to figure it all out.

.NET Core doesn't support the .NET version of security manager (CAS), and the is actually a JEP for deprecating the security manager for removal.

The JAAS idea was quite good, however it was too complex for most developers to implement properly.

I see a big failure for .NET and Java not having been a JIT/AOT stack since the beginning, and now we are going through their reboots while trying to keep the existing ecosystems to fall apart in the process.

I believe it's JEP 411 [1], good ridance, there is nothing worst than a false sense of security.

[1] https://openjdk.java.net/jeps/411

> In process sandboxing was definitely one of the worst ideas attempted in the early days of Java. None of my jobs in the last 10 years have ran with the security manager enabled, as it's just useless. I'm suspicious of any modern technology attempting to do the same thing, as it's just hard to get right.

I think Android has done a very good job with their permissions, which seem quite similar to me. Am I missing something?

Yes, properly limiting permissions is difficult, and there can be bugs, but that's true regardless of what layer the security controls are applied at.

The comment from @pron in the sibling thread explains the difference. Android applications, in general, don't share process.

The Android API gives me the feeling that the original idea was to do in-process sandboxing in mostly same way as it is done in Java and run the whole thing as one giant JVM instance and only later it was changed to the process-for-app model (which probably was not that much of breaking change because lot of things that cross the process boundary had to be serializable/parcelable already because of how android activity lifecycle handles memory pressure).

That's true, but largely irrelevant.

> Yes, properly limiting permissions is difficult, and there can be bugs, but that's true regardless of what layer the security controls are applied at.

Yes, the fact that each application is usually a process, and a mix of Linux kernel features are used for sandboxing, namely LinuxSE, seccomp and eBPF.

Also despite being based on Linux kernel, since Project Treble Android has doubled down on its Binder mechanism for IPC across drivers, kernel and activities with multiple processes, which in the end makes it look like microkernel based architecture.

What's the link between WebSphere and the Olympic games?

Looks like the fix was:

https://gitlab.ow2.org/asm/asm/-/issues/316506

12 October 2013: ASM 4.2 (tag ASM_4_2) 316506 ByteVector doesn't validate byteLength in putUTF8().

https://asm.ow2.io/versions.html

Couldn't find a CVE

I think GAE isolation nowadays is done by gVisor[1]? Looking at the doc [2] however, it seems like the old "proprietary" mechanism is still used for old (first generation) runtime. Probably this post is about that version.

   [1] https://cloud.google.com/blog/products/containers-kubernetes/how-gvisor-protects-google-cloud-services-from-cve-2020-14386
   [2] https://cloud.google.com/appengine/docs/standard/runtimes

Even the first-generation runtimes no longer use this mechanism. Note https://cloud.google.com/appengine/docs/standard/java/migrat...: "All standard Java classes are now available, and there is no class allowlist."

That's good to know!

+1 impressive independent of anything else. +1 for doing this as an intern.

Security was the prime reason app engine couldn’t add features fast enough, and hence couldn’t gain the market fast enough, else it could have been the money maker for google.

Security for similar offerings in AWS and Azure is quite good, I guess that wasn't the reason, after all Google only hires creme de la crème.

This is a beautiful, impressive, hack. The blog post itself is a minor tour de force.

Impressive work, Google must have been interested in your skills after this?

This was done in one week during a college internship. Seriously what the hell.

A logic programming language like Prolog seems like it would have been a good fit for solving the bytecode packing with constraints.

Note that GAE has two flavors: - Standard (constrained, more shared bits) - Flex (supply your own container, get your own instance)

These days, Standard uses gVisor so there are fewer and fewer constraints. Most things just work.

Flex really is for the edge cases where you need really big instances or really long execution times.

(I used to work at GCP)

Or, annoyingly, websockets.

Yeah, it seems like all the new infrastructure features (websockets, gRPC, GBLB, VPC, etc) are coming to Cloud Run instead of App Engine.

Websockets are fine on flex, but yeah, Cloud Run is getting the love these days.

Would capability-based security be more effective than what Java attempted for in-process isolation?

Probably easier to reason about as a programmer, at least in my opinion

very nice write up. wondering if they kept detailed notes or could recall most of this from memory after 8yrs.

too long. lost interest halfway through the article

Reply on news.ycombinator.com