JEP 457: Class-File API for Parsing, generating, transforming Java classfiles | Hacker News Reader

JEP 457: Class-File API for Parsing, generating, transforming Java classfiles

openjdk.org

102 pointscarimura34 comments

This is pretty exciting... I've used them all libraries at this point in my career: CGLib, ASM, BCEL, ByteBuddy, Javassist, etc... each has its pluses and minuses. I've designed everything from profiling agents, to systems that pack decimals into EBCDIC and invoke COBOL programs on big IBM iron, to lightweight JIT compilers, all using these libraries.

> In 2002, the visitor approach used by ASM seemed clever

I couldn't agree more. The visitor pattern was very hard to explain/justify back then, and still difficult to explain to newbie programmers just entering the profession.

Looking at the examples, I think this is going to be an official replacement for ASM, meaning it's going to be pretty low level. The use of streams pretty straightforward.

If anyone from the JEP is reading this: I have two pieces of feedback!

First, take some inspiration from the way CDI Portable Extensions work. This is probably the most delightful extension API I've ever used. The @Observe callbacks are super simple to explain to people and it's really easy to write extensions for the framework.

Next, I wouldn't ignore the need for a higher-level API akin to ByteBuddy or Javassist. Sometimes I just want to write an interpreter or intercept a method call and thats it.

For example in my Junit/Mockito extension https://github.com/exabrial/mockito-object-injection I need to intercept a call to the class under test in order to lazily inject dependencies at the last possible moment. While I certainly could do this with ASM, Javassist makes this fairly simple with it's MethodHandler api.

Side note, it's a damn shame we don't have a mobile operating system that is JVM native :/ All this cool APIs simply never reach a huge number of devices.

I just want to share my support of CDI extensions. They just make sense!

>First, take some inspiration from the way CDI Portable Extensions work

It's way too late, the API is effectively locked in preview mode.

While I am not a fan of ASM's visitor pattern, explaining it to anyone would be the least of my concerns, ASM requires pretty extensive knowledge on class structure, method signatures, and most of the byte code instructions. Whoever ventures in byte code editing mode should be able to read the byte code directly.

How would something like CDI portable extensions fit in here? Are you thinking it would another way to replace ASM's visitors? Doesn't it lack all the useful structure that the design in the JEP as, around streaming and building and transformation and so on?

I meant the "style of" CDI portable extensions :)

When you write a CDI portable extension, you register a bunch of observation handlers. So as the CDI environment is discovering stuff, it calls your observers and you have the chance to make chances to the runtime. I was thinking something akin to that here... when reading a class, your observers get called and you have a chance to manipulate the class data.

I find this style intuitive.

Surely the problem you're objecting to is that Android is JVM native, so the Java APIs are a part of the OS itself.

I highly recommend you to watch this video by Brian Goetz if you want to learn more about this JEP.

https://youtu.be/pcg-E_qyMOI

And as a companion piece I really enjoyed Paul Sandoz's 'Code Reflection' talk https://www.youtube.com/watch?v=xbk9_6XA_IY, which considers how java code might understand java code. The sort of thing that enable pushing functions to SQL or GPUs for example.

Edit to add: slides here https://cr.openjdk.org/~psandoz/conferences/2023-JVMLS/Code-...

Interesting, but I don't see how operating on the code model is going to be any easier than working on the AST. It seems much more hard to reason about many transformations as compared to manipulating the AST. I do hope they take inspiration from scala and rust macros

(Late reply, sorry)

Well currently you can't get at the AST, only (with some hairy code) bytecode. (The use cases are post-compilation)

But suppose this work did enable that. What Paul is saying is that there are intermediate representations between bytecode and AST that are more helpful for these GPU / SQL / whatever runtime compilers. Representations that capture dataflows, for example. Chance are these compilers would transform an AST to something like this anyway.

See https://mlir.llvm.org/ which is referenced in that talk.

Thanks, that makes sense. I think I approached more from the perspective of a library writer that doesn't want to delve into the depth of language to implement transformations/macros. However it does seem reasonable to have a variety of abstractions to be able to express these transformations in a way closer to the target language

At round 8:50, Goetz talks about the design, in essence the quote is,

"We designed it as a functional library because functionally inspired libraries are successful at meeting these goals."

Ironic.

Why ironic? Java is a multi paradigm language (though obviously not at the same level as say C++). It has been getting "functional" features for a while now, including streaming libraries, pattern matching, immutable records, etc.

Because they have in the past declared "Java is not a functional language" and were hostile to the use of monads in the standard library. They are adding nice features, but the language will never be as clean and unified as Scala.

Monads don’t go by the name, they must satisfy the axioms. How many people do it right? Even in Haskell lots of folks forget to check the axioms. Also, monads are hard to compose. I think there’s no reason to be like Cats or Arrow. It’s all already there, the mental shift is minimal in my opinion.

"Monads don't compose" just means you should choose a single monad for most of your functions; in Scala that is usually Future, Either, Try, IO or ZIO. There are conversion methods between the most common monads.

Scala monads are fantastic IMO; Either-based programming is simple and eliminates tons of boilerplate. So is Cats, although the learning curve is steep.

Unfortunately with many gotchas to be aware of all the time…

This is the sort of thing that will have a LARGE impact on the ecosystem for updating from one JDK to the next.

One of the big headaches we see with moving up JDK version is bytecode generation/manipulation libraries choking on newer versions of the JDK. It's generally a simple update, but sometimes it's not (for example, when someone is shading asm).

Having bytecode generation as part of the JDK will mean all libraries, including asm, can migrate to that and benefit from a perpetually supported API that updates as the JDK does.

This is probably 80% of the headaches I've experienced going from the likes of Java 11 to 17 and 17 to 21.

I’ve always wondered why the APIs for reflection in Java were always so low-level. If you look at, say, Objective-C (a language with a very similar model of Java) almost nobody drops down to assembly to patch things, because the language is expressive enough that you don’t need to do this basically ever. Why not add the same to Java?

I wonder if this will be available as a standalone library for older JDKs, it looks very nice.

Will this enable lombok officially?

Lombok's headaches are because they are touching Java compiler internals to accomplish their magic. This won't fix that.

Lombok wouldn't be nearly as troubled if it was just doing simple bytecode manipulation.

If lombok wants to stop the pain, then they'll need to stop reaching into internal APIs. They'll possibly need to remove a few features (like some of the `private final` work they are doing).

In other words, you can expect lombok to be a headache for years to come. (Maybe consider not using it? That'd be swell. Speaking as someone that curses lombok everytime jdk updates roll around.)

Why isn't Lombok just written as a pre-javac pass? javac(lombok(srccode)). Just gotta keep that parser alive and up to date, which open source IDEs also must so they're probably available.

It's easier (read more reliable) to manipulate bytecode or other internal representation than the source.

Manipulating the AST would be fine, but Lombok pretends to be an annotation processor (which can generate new classes, but not alter the semantics of the class being processed). They could create lombokc and crack open the internals of the Java compiler as much as they like, but this would mean admitting they are really Java, and they don’t seem to want to accept that.

Lombok is a compiler extension, where this is an API for the JVM. So two very different things. Think of Lombok/compiler extensions as #include directives in C... things that happen at compile time. This API is for writing programs at runtime, past the compile phase.

Lombok is not a compiler extension. Compiler extensions, aka annotation processors, are offered only specific capabilities that ensure that they preserve the Java language specification. Particularly, code that compiles successfully with an extension also compiles without it (perhaps requiring other classes to be available) and it compiles down to the same bytecode. Annotation processors are used to implement pluggable type systems (e.g. https://checkerframework.org) or to generate other classes (e.g. https://immutables.github.io/).

Unlike compiler extensions, Lombok compiles source files that do not conform to the Java language specification. Lombok is an alternative Java Platform language, like Clojure or Kotlin or Scala, except that it's a superset of the Java language. However, rather than forking `javac` source code and modifying it to compile Lombok source files, the Lombok compiler modifies `javac`'s operation by hacking into its internals and modifying them as it runs to compile Lombok sources rather than Java sources.

Having alternative Java Platform languages is perfectly fine. The problem with Lombok is that it doesn't present itself as such but as a library or a compiler extension even though it violates the Java language specification in ways that compiler extensions are forbidden from doing.

Last time I checked, admittedly several years ago, Lombok was abusing the fact that the AST esposed to the annotation processors was mutable (exposing a deep immutable view over a complex data structure is not something that you do easily nor efficiently in Java to this day, so I'm not blaming javac here). The compiler estensionions were not really forbidden in any meaningful way to do what Lombok is doing, and given the difference in tooling and effort required for a new language I don't really blame them either. That said, going against the whishes of the JSL has its risks, but I can understand Lombok's choice. I still push against Lombok in the projects I work on since how it works makes me unconfortable.

There's nothing wrong with offering an alternative language or even with basing another language's compiler on javac. What's wrong is the misrepresentation of what they're doing and how they try to hide the technical risks involved from their users. Don't call yourself a Java library if you're really a different language that's very much not Java; don't say you're a compiler extension if you bypass the compiler extension API and reach into its internals (that can change at any time, BTW) to turn it into a compiler for your new language. Their choices are fine; telling users they've chosen something else is not.

Lombok should not be touched, esp. since records have been available. Morealso, lombok is just a sugar coating during compilation time.

Even before records, just use public final fields, and be done with the getter/setter nonsense. (IDEs do a good job of offering options for toString(), and c-tors)

Personally, I consider lombok one of the better anti-patterns widely used.

Lombok can be used for more than just getters/setters.

I personally use the @Builder annotation on records with more than 3-4 fields. I find it much more readable than a long list of arguments to the constructor.

It also makes it easy to return a copy of the record where only a few fields have changed:

   var r = book.toBuilder()         
           .lastUpdated(now)
           .title("...")
           .build()

I also use other annotations, but I could work without them if a future version of Java provides a builder-like pattern (or named arguments)

This design note could be of interest: https://github.com/openjdk/amber-docs/blob/master/eg-drafts/...

Totally, and the With annotation is a bliss. Hope to have it at a syntax level someday, just like in Kotlin. In addition, you can decompile Lombok annotations straight into your code proper and change what you need, Builder methods for validation, say. Lombok is great for DX, especially if you have experience with more recent languages.

Records are the way to go, but there are many situations where the ecosystem doesn't work with them e.g. if you're stuck using something like Hibernate.

Reply on news.ycombinator.com