Stepping down as Mockito maintainer after ten years | Hacker News Reader

Stepping down as Mockito maintainer after ten years

github.com

283 pointssaikatsg214 comments

My second project at Google basically killed mocking for me and I've basically never done it since. Two things happened.

The first was that I worked on a rewrite of something (using GWT no less; it was more than a decade ago) and they decided to have a lot of test coverage and test requirements. That's fine but they way it was mandated and implemented, everybody just testing their service and DIed a bunch of mocks in.

The results were entirely predictable. The entire system was incredibly brittle and a service that existed for only 8 weeks behaved like legacy code. You could spend half a day fixing mocks in tests for a 30 minute change just because you switched backend services, changed the order of calls or just ended up calling a given service more times than expected. It was horrible and a complete waste of time.

Even the DI aspect of this was horrible because everything used Guice andd there wer emodules that installed modules that installed modules and modifying those to return mocks in a test environment was a massive effort that typically resulted in having a different environment (and injector) for test code vs production code so what are you actually testing?

The second was that about this time the Java engineers at the company went on a massive boondoggle to decide on whether to use (and mandate) EasyMock vs Mockito. This was additionally a waste of time. Regardless of the relative merits of either, there's really not that much difference. At no point is it worth completely changing your mocking framework in existing code. Who knows how many engineering man-yars were wasted on this.

Mocking encourages bad habits and a false sense of security. The solution is to have dummy versions of services and interfaces that have minimal correct behavior. So you might have a dummy Identity service that does simple lookups on an ID for permissions or metadata. If that's not what you're testing and you just need it to run a test, doing that with a mock is just wrong on so many levels.

I've basically never used mocks since, so much so that I find anyone who is strongly in favor of mocks or has strong opinions on mocking frameworks to be a huge red flag.

I'm not sure I understand. "The solution is to have dummy versions of services and interfaces that have minimal correct behavior".

That's mocks in a nutshell. What other way would you use mocks?

Imagine you're testing a service to creates, queries and deletes users. A fake version of that service might just be a wrapper on a HashMap keyed by ID. It might have several fields like some personal info, a hashed password, an email address, whether you're verified and so on.

Imagine one of your tests is if the user deletes their account. What pattern of calls should it make? You don't really care other than the record being deleted (or marked as deleted, depending on retention policy) after you're done.

In the mock world you might mock out calls like deleteUserByID and make suer it's called.

In the fake world, you simply check that the user record is deleted (or marked as such) after the test. You don't really care about what sequence of calls made that happen.

That may sound trivial but it gets less trivial the more complex your example is. Imagine instead you want to clear out all users who are marked for deletion. If you think about the SQL for that you might do a DELETE ... WHERE call so your API call might look like that. But if the logic is more complicated? Where if there's a change where EU and NA users have different retention periods or logging requirements so they're suddenly handled differently?

In a mokcing world you would have to change all your expected mocks. In fact, implementing this change might require fixing a ton of tests you don't care about at all and aren't really being broken by the change regardless.

In a fake world, you're testing what the data looks like after you're done, not the specific steps it took to get there.

Now those are pretty simple examples because there's not much to do the arguments used and no return values to speak of. Your code might branch differently based on those values, which then changes what calls to expects and with what values.

You're testing implementation details in a really time-consuming yet brittle way.

I am unsure I follow this. I'm generally mocking the things that are dependencies for the thing I'm really testing.

If the dependencies are proper interfaces, I don't care if it's a fake or a mock, as long as the interface is called with the correct parameters. Precisely because I don't want to test the implementation details. The assumption (correctly so) is that the interface provides a contract I can rely on.

In you example, the brittleness simply moves from mocks to data setup for the fake.

The point is that you probably don't care that much how exactly the dependency is called, as long as it is called in such a way that it does the action you want and returns the results you're interested in. The test shouldn't be "which methods of the dependency does this function call?" but rather "does this function produce the right results, assuming the dependency works as expected?".

This is most obvious with complex interfaces where there are multiple ways to call the dependency that do the same thing. For example if my dependency was an SQL library, I could call it with a string such as `SELECT name, id FROM ...`, or `SELECT id, name FROM ...`. For the dependency itself, these two strings are essentially equivalent. They'll return results in a different order, but as long as the calling code parses those results in the right order, it doesn't matter which option I go for, at least as far as my tests are concerned.

So if I write a test that checks that the dependency was carried with `SELECT name, id FROM ...`, and later I decide that the code looks cleaner the other way around, then my test will break, even though the code still works. This is a bad test - tests should only fail if there is a bug and the code is not working as expected.

In practice, you probably aren't mocking SQL calls directly, but a lot of complex dependencies have this feature where there are multiple ways to skin a cat, but you're only interested in whether the cat got skinned. I had this most recently using websockets in Node - there are different ways of checking, say, the state of the socket, and you don't want to write tests that depend on a specific method because you might later choose a different method that is completely equivalent, and you don't want your tests to start failing because of that.

The fakes vs mocks distinction here feels like a terminology debate masking violent agreement. What you’re describing as a “fake” is just a well-designed mock. The problem isn’t mocks as a concept, it’s mocking at the wrong layer. The rule: mock what you own, at the boundaries you control. The chaos you describe comes from mocking infrastructure directly. Verifying “deleteUserById was called exactly once with these params” is testing implementation, not behavior. Your HashMap-backed fake tests the right thing: is the user gone after the operation? Who cares how. The issue is finding the correct layers to validate behavior, not the implementation detail of mocks or fakes… that’s like complaining a hammer smashed a hole in the wall.

In the SQL example, unless you actually use an SQL service as a fake, you cannot really quite get the fake do the right thing either. At which point, it's no longer a mock/fake test but an integration/DB test. Network servers are another such class and for most parts can be either mocked or faked using interface methods.

I would argue that (barring SQL), if there are too many ways to skin a cat, it is a design smell. Interfaces are contracts. Even for SQL, I almost end up using a repository method (findByXxx flavors) so it is very narrow in scope.

The general term I prefer is test double. See https://martinfowler.com/bliki/TestDouble.html for how one might distinguish dummies, fakes, stubs, spies, and mocks.

Of course getting overly pedantic leads to its own issues, much like the distinctions between types of tests.

At my last Java job I used to commonly say things like "mocks are a smell", and avoided Mockito like GP, though it was occasionally useful. PowerMock was also sometimes used because it lets you get into the innards of anything without changing any code, but much more rarely. Ideally you don't need a test double at all.

There are different kinds of mocks.

Check function XYZ is called, return abc when XYZ is called etc are the bad kind that people were bit badly by.

The good kind are a minimally correct fake implementation that doesn't really need any mocking library to build.

Tests should not be brittle and rigidly restate the order of function calls and expected responses. That's a whole lot of ceremony that doesn't really add confidence in the code because it does not catch many classes of errors, and requires pointless updates to match the implementation 1-1 everytime it is updated. It's effectively just writing the implementation twice, if you squint at it a bit.

The second way is usually referring to as "fakes", which are not a type of mocks but a (better) alternative to mocks.

In reflection heavy environments and with injection and reflection heavy frameworks the distinction is a bit more obvious and relevant (.Net, Java). In some cases the mock configuration blossoms to essentially parallel implementations, leading to the brittleness discussed earlier in the thread.

Technically creating a shim or stub object is mocking, but “faking” isn’t using a mocking framework to track incoming calls or internal behaviours. Done properly, IMO, you’re using inheritance and the opportunity through the TDD process to polish & refine the inheritance story and internal interface of key subsystems. Much like TDD helps design interfaces by giving you earlier external interface consumers, you also get early inheritors if you are, say, creating test services with fixed output.

In ideal implementations those stub or “fake” services answer the “given…” part of user stories leaving minimalistic focused tests. Delivering hardcoded dictionaries of test data built with appropriate helpers is minimal and easy to keep up to date, without undue extra work, and doing that kind of stub work often identifies early re-use needs/benefits in the code-base. The exact features needed to evolve the system as unexpected change requests roll in are there already, as QA/end-users are the systems second rodeo, not first.

The mocking antipatterns cluster around ORM misuse and tend to leak implementation details (leading to those brittle tests), and is often co-morbid with anemic domains and other cargo cult cruft. Needing intense mocking utility and frameworks on a system you own is a smell.

For corner cases and exhaustiveness I prefer to be able to do meaningful integration tests in memory as far as possible too (in conjunction with more comprehensive tests). Faster feedback means faster work.

Why is check if XYZ is called with return value ABC bad, as long as XYZ is an interface method?

Why is a minimally correct fake any better than a mock in this context?

Mocks are not really about order of calls unless you are talking about different return values on different invocations. A fake simply moves the cheese to setting up data correctly, as your tests and logic change.

Not a huge difference either way.

The point is to test against a model of the dependency, not just the expected behavour of the code under test. If you just write a mock that exactly corresponds to the test that you're running, you're not testing the interface with the underlying system, you're just running the (probably already perfectly understandable) unit through a rote set of steps, and that's both harder to maintain and less useful than testing against a model of the underlying system.

(And IMO this should only be done for heavyweight or difficult to precisely control components of the system where necessary to improve test runtime or expand the range of testable conditions. Always prefer testing as close to the real system as reasonably practical)

But mocks are a model of the dependency. I don't quite see how a fake is a better model than a mock.

In any case, I agree testing close to a real system, with actual dependencies where possible is better. But that's not done with a fake.

The kind of mocks the OP is arguing against are not really a model of the dependency, they're just a model of a particular execution sequence in the test, because the mock is just following a script. Nothing in it ensures that the sequence is even consistent with any given understanding of how the dependency works, and it will almost certainly need updating when the code under test is refactored.

My point is that a fake doesn't magically fix this issue. Both are narrow models of the underlying interface. I don't still quite understand why a mock is worse than a fake, when it comes to narrow models of the interface. If there is a method that needs to be called with a specific set up, there is no practical difference between a fake and a mock.

Again, none of this is a replacement for writing integration tests where possible. Mocks have a place in the testing realm and they are not an inherently bad tool.

Mocking is testing how an interface is used, rather than testing an implementation. That's why it requires some kind of library support. Otherwise you'd just on the hook for providing your own simple implementations of your dependencies.

Heavy mocks usage comes from dogmatically following the flawed “most tests should be unit tests” prescription of the “testing pyramid,” as well as a strict adherence to not testing more than one class at a time. This necessitates heavy mocking, which is fragile, terrible to refactor, leads to lots of low-value tests. Sadly, AI these days will generate tons of those unit tests in the hands of those who don’t know better. All in all leading to the same false sense of security and killing development speed.

I get what you are saying, but you can have your cake and eat it too. Fast, comprehensive tests that cover most of your codebase. Test through the domain, employ Fakes at the boundaries.

https://asgaut.com/use-of-fakes-for-domain-driven-design-and...

“The solution is to have dummy versions of services and interfaces that have minimal correct behavior”

If you aren’t doing this with mocks then you’re doing mocks wrong.

Martin Fowler draws a useful distinction between mocks, fakes, and stubs¹. Fakes contain some amount of internal logic, e.g. a remote key-value store can be faked with a hashmap. Stubs are a bit dumber—they have no internal logic & just return pre-defined values. Mocks, though, are rigged to assert that certain calls were made with certain parameters. You write something like `myMock.Expect("sum").Args(1, 2).Returns(3)`, and then when you call `myMock.AssertExpectations()`, the test fails unless you called `myMock.sum(1, 2)` somewhere.

People often use the word "mock" to describe all of these things interchangeably², and mocking frameworks can be useful for writing stubs or fakes. However, I think it's important to distinguish between them, because tests that use mocks (as distinct from stubs and fakes) are tightly coupled to implementation, which makes them very fragile. Stubs are fine, and fakes are fine when stubs aren't enough, but mocks are just a bad idea.

[1]: https://martinfowler.com/articles/mocksArentStubs.html

[2]: The generic term Fowler prefers is "test double."

In part, you’re right, but there’s a practical difference between mocking and a good dummy version of a service. Take DynamoDB local as an example: you can insert items and they persist, delete items, delete tables, etc. Or in the Ruby on Rails world, one often would use SQLite as a local database for tests even if using a different DB in production.

Going further, there’s the whole test containers movement of having a real version of your dependency present for your tests. Of course, in a microservices world, bringing up the whole network of dependencies is extremely complicated and likely not warranted.

I use test containers and similar methods to test against a "real" db, but I also use mocks. For example to mock the response of a third party api, can't very well spin that up in a test container. Nother example is simply time stamps. Can't really test time related stuff without mocking a timestamp provider.

It is a hassle a lot of the time, but I see it as a necessary evil.

You can use a library like [1] to mock out a real HTTP server with responses.

[1] https://www.mock-server.com/

I'd go a bit farther — "mock" is basically the name for those dummy versions.

That said, there is a massive difference between writing mocks and using a mocking library like Mockito — just like there is a difference between using dependency injection and building your application around a DI framework.

> there is a massive difference between writing mocks and using a mocking library like Mockito

How to reconcile the differences in this discussion?

The comment at the root of the thread said "my experience with mocks is they were over-specified and lead to fragile services, even for fresh codebases. Using a 'fake' version of the service is better". The reply then said "if mocking doesn't provide a fake, it's not 'mocking'".

I'm wary of blanket sentiments like "if you ended up with a bad result, you weren't mocking". -- Is it the case that libraries like mockito are mostly used badly, but that correct use of them provides a good way of implementing robust 'fake services'?

In my opinion, we do mocking the exact opposite of how we should be doing it — Mocks shouldn't be written by the person writing tests, but rather by the people who implemented the service being mocked. It's exceedingly rare to see this pattern in the wild (and, frustratingly, I can't think of an example off the top of my head), but I know Ive had good experiences with cases of package `foo` offering a `foo-testing` package that offers mocks. Turns out that mocks are a lot more robust when they're built on top of the same internals as the production version, and doing it that way also obviates much of the need for general-purpose mocking libraries.

I think the argument they're making is that once you have this, you already have an easy way to test things that doesn't require bringing in an entire framework.

The difference, IMO, between a mock and a proper "test" implementation is that traditionally a mock only exists to test interface boundaries, and the "implementation" is meant to be as much of a noop as possible. That's why the default behavior of almost any "automock" is to implement an interface by doing nothing and returning nothing (or perhaps default-initialized values) and provide tools for just tacking assertions onto it. If it was a proper implementation that just happened to be in-memory, it wouldn't really be a "mock", in my opinion.

For example, let's say you want to test that some handler is properly adding data to a cache. IMO the traditional mock approach that is supported by mocking libraries is to go take your RedisCache implementation and create a dummy that does nothing, then add assertions that say, the `set` method gets called with some set of arguments. You can add return values to the mock too, but I think this is mainly meant to be in service of just making the code run and not actually implementing anything.

Meanwhile, you could always make a minimal "test" implementation (I think these are sometimes called "fakes", traditionally, though I think this nomenclature is even more confusing) of your Cache interface that actually does behave like an in-memory cache, then your test could assert as to its contents. Doing this doesn't require a "mocking" library, and in this case, what you're making is not really a "mock" - it is, in fact, a full implementation of the interface, that you could use outside of tests (e.g. in a development server.) I think this can be a pretty good middle ground in some scenarios, especially since it plays along well with in-process tools like fake clocks/timers in languages like Go and JavaScript.

Despite the pitfalls, I mostly prefer to just use the actual implementations where possible, and for this I like testcontainers. Most webserver projects I write/work on naturally require a container runtime for development for other reasons, and testcontainers is glue that can use that existing container runtime setup (be it Docker or Podman) to pretty rapidly bootstrap test or dev service dependencies on-demand. With a little bit of manual effort, you can make it so that your normal test runner (e.g. `go test ./...`) can run tests normally, and automatically skip anything that requires a real service dependency in the event that there is no Docker socket available. (Though obviously, in a real setup, you'd also want a way to force the tests to be enabled, so that you can hopefully avoid an oopsie where CI isn't actually running your tests due to a regression.)

my time at google likewise led me to the conclusion that fakes were better than mocks in pretty much every case (though I was working in c++ and python, not java).

edit: of course google was an unusual case because you had access to all the source code. I daresay there are cases where only a mock will work because you can't satisfy type signatures with a fake.

Mockito, in every case I had to use it, was a last resort because a third party library didnt lend itself to mocking, or you were bringing legacy code under test and using it long enough to refactor it out.

It should never be the first tool. But when you need it, it’s very useful.

I dont use dummy services and I dont use mocking. I'm writing simulators to test things for HW or big services which are not available for testing.

Simulators need to be complete for their use cases or they cannot be used for testing.

Wow, there's a lot of anger in some of these posts.

I've been using Mockito for about 4 years, all in Kotlin. I always found it to be "plenty good" for like 99% of the cases I needed it; and things more complicated or confusing or messy were usually my fault (poor separation of concerns, etc).

I regularly found it quite helpful in both its spy() and mock() functionality.

I never found it meaningfully more or less useful than MockK, though I have heard MockK is the "one that's better for Kotlin". It's mostly just vocabulary changes for me, the user.

I'm going to have to monitor Mockito's future and see if I'll need to swap to MockK at some point if Mockito becomes unmaintained.

"Wow, there's a lot of anger in some of these posts."

If I was OP I'd retire happy knowing that a very thankless job is well done! Given what it does: the more outrage the better. Projects like Mockito call out the lazy and indolent for whom they are and the hissing and spitting in return can simply be laughed at.

10 years this bloke has given his time and effort to help people. He states: nearly a third of his life.

I'll raise a glass and say "was hale" or perhaps wassail as an Englander might.

Is it anger? The agent thing maybe. The other two points seem to boil down to:

1. Kotlin is a hack and 2. Rust is more fun.

Pretty understandable why one would simply want to move on to greener pastures.

I'm not sure Kotlin is a hack, but it's growth is directly related to Java not being able to deliver on simple features that would help the day to day lives of developers. So I also can't blame anyone for moving on from the language. I get frustrated with it all the time, especially when Kotlin shows that some feature could work perfectly fine.

Here are some examples that have hit the graveyard: It's been 2 years since exception handling in switch was proposed [0], 3 years since null-restricted types were proposed [1], 4 years since string templates [2], 8 years since concise method bodies [3], and 11 years for JSON parsing [4].

[0] https://inside.java/2023/12/15/switch-case-effect/

[1] https://openjdk.org/jeps/8303099

[2] https://openjdk.org/jeps/430

[3] https://openjdk.org/jeps/8209434

[4] https://openjdk.org/jeps/198

Sure there are pain points in Java, but I definitely wouldn't want the language to introduce 15 new features each having only a tiny benefit.

Based on your other comment you prefer "fat" languages like kotlin and C# - and that's fair. I find languages with less, but more powerful features far more elegant and while Java has its fair share of historic warts, modern additions are made in a very smart way.

E.g. switch expressions seamlessly support sum and product types, meanwhile kotlin's `when` really is just syntactic sugar.

All in all, with too many features you have to support and understand the complete matrix of their interactions as well, and that gets complicated quickly. And Kotlin is growing towards that.

What can switches finally do that when can't? Other than the terrible fall through behavior?

When used as a different way of writing an if else if can be a code smell. When used with an exhaustive list of an enum or sealed class prevents bugs in production.

Switch expressions don't do fall through. They are proper ML-like pattern matching.

I’m sorry, but null restricted types, string templates, basic json support, and fixing the broken exception system isn’t “fat”. They’re the basics of a functioning language.

Null restricted types are indeed important and while null check plugins exist for Java, this is a big step in the good direction by kotlin.

String templates are valuable syntactic sugar, but that's it.

There are a bazillion JSON libraries with at least 3-4 absolutely stellar ones. I don't really see it that big of a limiter.

And if you mean checked exceptions, that's controversial to claim it's all bad. But some ergonomics improvements would be better.

Handling JSON as a basics for language? Really? null restricted type will come with valhalla, string templates had a try, but people complained and they withdrawn them after 2 preview iterations. Not sure what issue people had with them and what issues maintainers noticed.

What do you mean by broken exception system?

Java is not a small language that you can throw features around, they have to take into consideration final goal of it and decades of development.

> Handling JSON as a basics for language?

Yes, even Mark Reinhold admitted that in the last "Ask the Architects" interview.

>null restricted type will come with valhalla

Will they? It's been 10+ years of Valhalla. Why is a compiler construct even behind project Valhalla? Kotlin has showed you don't need it to do them.

> string templates had a try

Yes they were over engineered and they failed to deliver a basic feature.

> Java is not a small language that you can throw features around, they have to take into consideration final goal of it and decades of development.

Yes I agree, but it shouldn't take YEARS to ship anything. 11 years for a JSON api?! Come on.

> Yes I agree, but it shouldn't take YEARS to ship anything. 11 years for a JSON api?! Come on.

Anything? Or maybe something that you specifically want?

It added records when I wanted them, it added streams and lambdas. I don't care for string templates (I thing those are ugly in any language).

null-restricted types would be nice but you have to understand that designing a language is not throwing every possible feature on top of it (like kotlin and resulting lack of readability it has and hard time updating to newer JDK), you have to design it, think of the possibilities, what users really want etc.

Valhalla type system results in addition of null-restricted types, as a natural evolution of the language.

Why did Kotlin prevent those features?

It didn't. They're examples of features that have driven Kotlin's growth. Thing's Java developers have been begging for. I'm using them as examples of why Java developers are starting to choose Kotlin. I'm a pretty die hard Java fan, I've built my career on it, and know way too much about the ecosystem. But, I've been writing F#, C#, Kotlin, and some other languages in my free time to see where I want to move to. I’m pretty sure C# is what I’ll settle on.

Oh I see, thank you for explaining.

PS: personally I really like Swift, I would suggest giving it a try for fun

Yeah swift is nice! I’ve ranted about how Java should add Swift’s try! and try? for checked exceptions before.

I don’t think it’s for me though. It’s very Apple centric and I’m a Windows gamer and PowerShell user at heart.

Kotlin is a thoughtful language that is a joy to write.

Java was stagnant and ripe for being kicked off the top.

Scala was the hack that showed why Java was starting to suck but Scala suffered from no direction. Every single idea a PhD ever had was implemented in the language with no thought other than it seems cool to be able to do that too. Which is why all the Scala codebases fell apart, because you could write anything, anyway you want, and it was impossible to maintain without extremely strict guidelines on what parts of the language you were allowed to use. Also the build times were atrocious.

Kotlin designers evaluated language features across the ecosystem and chose what made sense to make Java better when it came out and now has easily surpassed Java. They are still thoughtful in what they choose to add to the language and it is now very powerful. However they smartly imitate Python by having clear guidelines on what choices you should be making when writing code so that other developers can easily dive in.

I liked Groovy, but Kotlin was the better successor. Scala was beyond ridiculous as far as learning curve--crazy language. I always thought Java sucked and I spent far too much time working with Java (because money) and having to deal with framework on top of framework as well as all those idiotic design patterns that seemed so important only for me to realize years later that it was all basically a waste because OOP itself is a waste. I was writing better code in Python and Go five years ago without needing those all-important design patterns (okay, a few still translate). I just didn't know it at the time how horrible the situation was in Java. In hindsight, everything about Java was someone's money grab that I was dealing with. Mockito did help me a lot through those times, but I wish I had never gone through them at all!

I still miss dependency injection in Python

FastAPI rolled their own using annotations. It's... ...okay.

Unfortunately it also constructs dependencies per request instead of singletons, so you have to rely on hacks like `@lru_cache(maxsize=1)` on the factory functions.

I'd really like to see a Python framework that embraces class-based controllers to do away with this problem; beyond that writing a dependency injection system is not too difficult of a task (the last time I did it for a hobby project many years ago, it was around ~150 lines of code).

> Scala was the hack

> Kotlin designers evaluated language

Well, I like scala, and most of the time I understand it, but I can't say the same with Kotlin. I would say quite the opposite is true.

> Scala was the hack

Well, I absolutely disagree with this take. Scala actually builds on top of a couple of its powerful primitives. Especially Scala 3 is a beautiful language.

In retrospect - from an outside point of view - it seems they should have just declined to support Kotlin. There are other options for Kotlin, and it sounds like it makes sense for it to have something developed for that from the beginning. But probably they had no idea what they were getting into when it was first proposed.

At the end of the day I think there's nothing wrong with the tool itself. The problem is that mocking and spies make it easy to not bother properly isolating the effects of a function for testing and then you end up having a test where 95% of it is setting up an elaborate array of mocks to create the condition you wish to test, which are completely incomprehensible to the next person trying to read it.

I used it mainly because the code I inherited was untestable as written. I made it testable via those methods. Then it got refactored.

Well sure you sometimes haven’t got much choice. But I’m talking about people write new code this way

Mock is effective when developers keep applications at 4-5 layers deep (and honestly, you don't need more than that 97% of the time: initiators, controllers, services, transports, and cross cutting concerns).

The problem is, engineers love to solve problems, and the funnest types of problems are hypothetical ones!

Yep, I was guiltily of taking DI and writing a spider web of code. It's not the tools fault, it was my attitude. I _wanted_ something hard to work on, so I created something hard to work on. Nowadays, my team and I work on a 4-5 layer deep limit and our code base is tight, consistent, and has near 99% test coverage naturally. We use mock testing for testing the single class, and of course integration test for testing requirements. Not everyone will do it this way, and that's fine, but the most important thing is actually just creating a plan and sticking to it so as to be consistent.

In the end, don't blame the tool, when you (or your coworkers) simply lack discipline.

For those, like me, who haven't heard of it: Mockito is the "most popular mocking framework for Java".

It’s taken years off of my life dealing with the test mess people have made with it.

Absolutely the worst. 1000 line test setups that shatter into pieces the instant you try to make the simplest change to a function. Makes refactoring an absolute nightmare.

What's specifically bad about Mockito here? Poor defaults for mocks?

You've just built a calculator, and now you want to test it. Well, you have to push buttons on a calculator, so you build a robotic hand. And that hand needs a power source and some intelligence. And you need to photograph and OCR the result from the calculator screen.

This is kinda how we build software right? A little bit of "our logic" (calculation), represented as objects/actors/modules which "do things", but intermingled with million-LoC dependencies like databases and web servers.

After a while it gets frustrating setting up the robot hand and OCR equipment for each test case. Maybe it's just easier to test manually, or skip testing entirely.

At this point you can have an epiphany, and realise you only care about the numbers going in and out of the calculator, not the button pushes and pixels.

Mockito swoops in and prevents you from having that epiphany, by making it easier to keep doing things the stupid way.

Instead isolating the calculation from any IO, you can now write things like: when(finger.pushbutton(1)).then(calculator.setState(1)) when(calculator.setAnswer(3)).then(camera.setOcr(3))

(I've mostly worked in Java, but it seems like other languages typically don't let you intercept calls and responses this way)

I’ll answer: Nothing specific to Mockito, it happens in every language. Tests “solidify” code which makes refactoring hard. And yet, after refactoring, one can be happy to have tests to check whether there is any regression.

Testing is hard. I’ve tried with AI today: No, it is still not capable of handling that kind of (straightforward) task (Using Claude).

They also encourage/enable code that is less testable. If you use mockito to get your fake responses/assertions where you need them, you don't have to think about your class's dependencies to make your code testable and therefore better decomposed. I don't even do TDD, but I still find that thinking about how I'd test a class guides me toward better-factored code.

One alternative to make code with typing styles in the Java way (as opposed to the Typescript or Go way) is to have a whole lot of custom interfaces and then you end up with a whole bunch of:

  doTheThing(foo: Fooable) { ... }

when there's really only one Foo implementation in prod. It leads to (what feels like, to me) more code obfuscation in large projects, than the benefits that come out, at least for me.

So Mockito and friends are a nice alternative to that.

That is just my experience and opinion though, and there are definitely more valid or equally valid alternatives.

I don't think we have to choose. Naturally finding the "right division of labor" is as infinite as finding the "right level of abstraction", but I think the ideal situation is to strive toward code that is easy to test without having to introduce a lot of mocks or without infinite layers of abstraction.

IMO Mockito is fine, the problem I’ve encountered is people taking the easy way out and trying to test something that needs a real integration test with a convoluted series of mocks.

I've found that most of the time that if you need to mock, you probably just need to do an integration test.

I agree. I mostly only use mocks for external service dependencies (apart from primary application database), or library features that are very problematic to use in test. A lot of code simply should not be unit tested - if an integration test can cover the same paths its far better in the long run. Performance can become an issue but it is easier to deal with that than to spend so much time maintaining and debugging mock constructs.

I can’t concur with this enough.

I’ve been on projects where mocking _literally made the project less reliable_ because people ended up “testing” against mocks that didn’t accurately reflect the behavior of the real APIs.

It left us with functionality that wasn’t actually tested and resulted in real bugs and regressions that shipped.

Mocking is one of these weird programmer pop-culture memetic viruses that spread in the early 2000s and achieved complete victory in the 2010s, like Agile and OOP, and now there are entire generations of devs who it’s not that they’re making a bad or a poorly argued choice, it’s that they literally don’t even know there are other ways of thinking about these problems because these ideas have sucked all the oxygen out of the room.

> like Agile and OOP

Ha.

I think there's room to argue "Agile" is a popular bastardisation of what's meant by "agile software development", and with "OOP" we got the lame Java interpretation rather than the sophisticated Smalltalk interpretation. -- Or I might think that these ideas aren't that good if their poor imitations win out over the "proper" ideas.

With mocking.. I'm willing to be curious that there's some good/effective way of doing it. But the idea of "you're just testing that the compiler works" comes to mind.

For OOP, I'd say the issue isn't so much that it's not useful (it is very useful), but rather that it was treated as "common sense, the only way to do it".

Sometimes the right tool for the job is objects, sometimes it's functional, sometimes you do want encapsulation but it's better as structs and using composition over inheritance. When everything looks like a `class Hammer extends Tool`…

Agreed - I like your phrasing of it. There are some good ideas in OOP. I don't know that I'd go as far as to credit OOP for those ideas, but things like polymorphism and encapsulation can be very useful. What I objected to was, as you said, OOP became "the only way to do it". It became a dogma. I was very happy when functional programming started to break through and show that there were other ways that were not only viable, but often better.

Had my fill of both until I could not stand it anymore. So tired of the grifters--agile was sometimes good, but mostly a grift.

Mocking and integration tests are not mutually exclusive. I often use mocks in my integration tests. Some things can be integration tested, some things can't. Sone things you just need to mock.

People using it wrong. It definitely should not be that popular. 95% of times when I see it I consider it a tech debt.

You should be using it in rare cases when you want to verify very complex code that needs to be working with strict requirements (like calling order is specified, or some calls cannot be made during execution of the method).

Usually it is used for pointless unit tests of simple intermediary layers to test if call is delegated correctly to deeper layer. Those tests usually have negative value (they test very little but make any modification much harder).

It allows you to mock the whole universe so it becomes a hammer instead of nicely designed functions, interfaces.

Which is absurd that people use mocks considering the tests are supposed to help with refactoring but because of the mocks they can’t make a change without breaking the test.

I wish there are tests written to cover functionalities (& race conditions) instead of covering lines/branches.

It also translates to “small booger”, in Spanish, which always made me question who thought the name was a good idea over there.

Why? Every name you pick is likely to be weird in one language or another. Mockito does one thing well as a name, and that is hinting strongly at what it is (a mocking library).

>is likely to be weird in one language or another.

But this name is weird in the specific language it’s imitating (both the -ito termination for diminutives and the drink on which I assumed the name is based are Spanish).

Wasn't it a riff on "Mojito" which was a popular drink at the time?

Mojitos aren't popular anymore?!

Actually, no. "small booger" would be _moquito_ in spanish.

Look, as an English only speaker I don't care - I'm still stuck at "Haw haw, small booger library!"

Fair. The spelling is off, but the pronunciation is the same.

I'm Spanish and subconsciously pronounced the library as MOCKito, as opposed to moQUIto.

I’m bilingual(ish) and while I’ve always pronounced it MOCKito, I think I may start pronouncing it moQUIto instead now.

| My personal take is that folks involved with the change severely underestimated the societal impact that it had. The fact that proper build support is non-existent to this day shows that agents are not a priority. That's okay if it isn't a priority, but when it was communicated with Mockito I perceived it as "Mockito is holding the JVM ecosystem back by using dynamic attachment, please switch immediately and figure it out on your own".

Id like to hear the platform team's perspective on this. As it stands, it is a pretty sad state of affairs that such a prominent library in the ecosystem was made out to be the scapegoat for the adoption of a platform change. It is not a healthy thing to treat the library maintainer community like this.

The JDK team has advertised these changes for years. People who want to do wild & crazy dynamic stuff in their testing infrastructure still can, with a tiny amount of added setup. This is the correct tradeoff, vs having the JVM pessimistically unable to apply all sorts of optimizations because they never know when their users might have inadvertently opted into runtime shenanigans.

Running open source _anything_ looks exhausting from the outside.

I don't know why owners/maintainers show grace like this.

For me; I would say I'm done and hand it over or just archive the repo.

Hope Tim finds a measure of sanity after he steps down.

I think showing grace was the right way to do things. It preserves the value of all the effort he put into it. When you work on things that long they are part of you. Like your baby, you don't toss them aside carelessly, it's painful to do that I'm sure.

Well done for Tim for being one of those guys that worked so hard for others to benefit. Tim, you've done something you can be deeply proud of forever, no matter how it ended.

A lot of OSS burnout comes from a broken assumption: that publishing code creates an obligation.

Historically, open source meant "here's code, use it if it helps, fix it if it breaks." No support contracts, no timelines, no moral duty. GitHub-era norms quietly inverted that into unpaid service work, with entitlement enforced socially ("be nice", "maintainers owe users").

Intrinsic motivation is the only sustainable fuel here. Once you start optimizing for users, stars, adoption, or goodwill, pressure accumulates and burnout is inevitable. When you build purely because the work itself is satisfying, stopping is always allowed, and that's what keeps projects healthy.

Hard boundaries aren't hostility; they're corrective. Fewer projects would exist if more maintainers adopted them, but the ones that remain would be stronger, and companies would be forced to fund or own their forks honestly.

Open source doesn't need more friendliness. It needs less obligation

>Mockito 5 shipped a breaking change where its main artifact is now an agent. That's because starting JVM 22, the previous so-called "dynamic attachment of agents" is put behind a flag

Wouldn't this hold back enterprise adoption, the same way breaking changes meant that Java 8 was widely used for a long time?

JVM 22 stuff isn't going to be mainline at enterprise for years. Java 25 is the first LTS release that includes it and it's only just been released.

Most places are just about getting rid of 8 for 17.

I know!!!!!! I've been yelling about this for years how impossible to uplift JVM versions is because of how products were built! It's so crazy. Lots of obfuscated jars went unsupported after like Java 7.

It just means your test runs need to add the flag.

TimvdLippe: You've done an incredible job. You have incredible ideas and vision. Just wanted to say thank you.

Hey, fwiw, thank you for all of your hard work! Mockito and Powermock helped me be a big hero at one company I worked for 2011-2014. Before I arrived as a tech lead there, they had ZERO tests. The QA cycle was weeks, even months long. I instituted a whole new regime of unit and intergration testing for all the applications I was responsible for. Within one release cycle, the bug counts fell to nearly zero and the QA cycle basically became a verification step because there were no more bugs. The only way I was able to pull that off was using mocking and some clever powermock hacks because the code was otherwise untestable and thus un-refactorable. So, thanks!

Pretty sure Powermock is dead in the water now, it did some pretty gnarly things (like patching `Object`) that flat out didn't work in later versions of Java.

I (ironically enough) spent some time replacing usages of it in Kafka test code with Mockito because of this, IIRC there was only one situation where Mockito couldn't easily replace Powermock and I'm pretty sure it was mocking out a private static method, so the solution was to refactor the code under test so that you didn't need to mock a private static method to test it.

Mockito is fine if you know how to write tests. You can write bad tests with Mockito or many other frameworks if that’s what you do. You can even write bad tests while programming in Rust from high up in your ivory tower. You can write good tests also.

IMO mockito has a relatively good use experience. If you use MockitoExtension, especially, you write code that is relatively maintainable, and easy to mutate. The problem without MockitoExtension is you can throw all kinds of junk in there and it just sits there doing nothing, without you knowing it.

Spy's on the other hand, are a pain in the neck. I think they should be good in theory, but in practice they are difficult, the debuggers really don't work correctly, setting breakpoints, stepping, etc, is just broken in many cases. Would love to know if this is just a difficult bug, or something that is baked into spying.

This is a good opportunity to ditch mocking and use fakes with adapters. Not only mocks create brittle tests that often test only the framework itself, but they do so in an order of magnitude slower way.

Also, F Kotlin and their approach of "we'll reinvent the wheel with slightly different syntax and call it a new thing". Good riddance I say, let them implement their mockk, or whatever it is called, with ridiculous "fluent" syntax.

The whole mocks, fakes and the rest were always a terrible idea. Although people didn't realize it st the time, they were a band-aid to try to make poorly architected code testable.

90% of the time, needing to use a mock is one of the clearest code warning smells you have of there being an issue in the design your code.

It took a while, but the industry seems to be finally (although slowly) coming to this realization. And hopefully with it almost all of this can go away.

I agree. Tests relying on mocks rarely uncover or prevent issues. They also typically make it harder to make changes. Very bad idea that should have been left behind years ago.

How do you substitute for dependencies that you're not testing, or that you want to deliberately break?

90% of the time (or more): you don't. The real thing is perfectly fine in a test with the right setup. Fileio is fast, I just need a test file in a tempdir. databases are fast, I just need an easy way to settup my schema. Sometimes I need the isolation but normally I do not.

Well, no - you don't.

What you're describing is a very limited subset of testing, which presumably is fine for the projects you work on, but that experience does not generalise well.

Integration testing is of course useful, but generally one would want to create unit tests for every part of the code, and by definition it's not a unit test if hits multiple parts of the code simultaneously.

Apart from that, databases and file access may be fast but they still take resources and time to spin up; beyond a certain project and team size, it's far cheaper to mock those things. With a mock you can also easily simulate failure cases, bad data, etc. - how do you test for file access issues, or the database server being offline?

Using mocks properly is a sign of a well-factored codebase.

> Integration testing is of course useful, but generally one would want to create unit tests for every part of the code, and by definition it's not a unit test if hits multiple parts of the code simultaneously.

The common pitfall with this style of testing is that you end up testing implementation details and couple your tests to your code and not the interfaces at the boundaries of your code.

I prefer the boundary between unit and integration tests to be the process itself. Meaning, if I have a dependency outside the main process (eg database, HTTP API etc) then it warrants an integration test where i mock this dependency somehow. Otherwise, unit tests test the interfaces with as much coverage of actual code execution as possible. In unit tests, out of process dependencies are swapped with a fake implementation like an in-memory store instead of a full of fledged one that covers only part of interface that i use. This results in much more robust tests that I can rely on during refactoring as opposed to “every method or function is a unit, so unit tests should test these individual methods”.

Yeah I think that's a question of using the right tool for the job. Some projects are of a size that it's not really necessary to be more fine-grained, but as the number of moving parts increases, so too in my experience does the need to ensure those parts are individually working to spec, and not just the whole thing. A classic example might be something like a calculator that enshrines a complex piece of logic, and a piece of code that uses it. I would test both of those in isolation, and mock out the calculator in the second case so that I could generate a whole range of different return values and errors and prove that the calling code is also robust and behaves correctly. Separating them like this also potentially reduces the number of tests you need to write to ensure that you hit all possible combinations of inputs and outputs.

Why would I want to create tests for every part of the code? I did that for years because I was taught that, but I came to realize it never mattered - if a test breaks it is because of the last thing I changed. I have a few flakey tests from time to time, but they have been not too bad to track down nd often taught me enough about how the system really worked as to be worth the time anyway.

I'm sorry, I don't understand what you mean.

You seem to say it's not worth writing a lot of tests, but then you talk about tests breaking due to bad changes - if you don't write those tests in the first place, then how do you get into that situation?

I didn't word my earlier comment very well - I don't mean to advocate for 100% coverage, which I personally think is a waste of time at best, and a false comfort at worst. Is this what you're talking about? What I wanted to say is that unit tests should be written for every part of the code that you're testing, i.e. break it into bits rather than test the whole thing in one lump, or better, do both - unit tests and integration tests.

Write a lot of tests - mostly integration. Unit tests have proven more harmful than helpful - unit tests are great when the api is used so often it would be painful to change it so you don't anyway. Otherwise I want to change the api and the code that uses it as requirements change. When I'm writing string or a list I'd unit test that - but mostly that is in my standard library so I'm not. Instead I'm writing code that is only used a few places and those places both will change every few years as requirements change.

I'm sorry I don't really understand your argument; unit tests are bad because code changes? Surely if you're only testing small pieces then that works better when you make a change? You only need to update the tests for the part of code that changed, not everything that touches it. That's another great thing about mocks - your interfaces should be stable over time, so the mocks don't need to change that often, and when you're changing MyCalculator, all the dependencies to MyCalculator remain the same so you only have to update MyCalculatorTests.

This is what I mean about a well-factored code base; separate things are separate, and can be tested independently.

> Unit tests have proven more harmful than helpful

Why? I find them very useful. If I change a method and inadvertently break it somehow, I have a battery of tests testing that method, and some of them fail. Nothing else fails because I didn't change anything else, so I don't need to dive into working out why those other tests are failing. It's separation of concerns.

Let me put it a different way: what is a unit?

i have concluded a unit needs to be large. Not a single class or function but a large collection of them. When 'archtecture astronaughts' draw their boxes they ar drawing units. Often thousands of functions belong to a unit. even then though often it is easier to use the real other unit than a test double.

You can make your own definition of unit if you like, but the usual definition is the smallest whole piece of something - a unit of currency, a unit of measure, etc.

If your unit is thousands of functions wide then you have a monolith, and there are widely discussed reasons why we try to avoid those.

> Using mocks properly is a sign of a well-factored codebase.

Well-factored codebase doesn’t need mocks.

That's not a counter-argument. Why don't you need to mock out the interfaces that you're not testing?

why would I? Either the code works either way or I'm glad to know I broke things when the seemingly unrelated tests breaks. Remember if a test fails the fault is almost always the last thing I changed.

now I do take care to avoid writing tests that depend on other code's results that would be likely to change. This hasn't proven to be the problem I've so often been warned about though.

> Either the code works either way or I'm glad to know I broke things when the seemingly unrelated tests breaks

...or, you tested each thing individually, so only related tests break and you can quickly zero in on the problem. Isn't that easier?

I worked on a project where a dev wanted to mock out the database in tests "for performance, because the database is slow". I almost lost my shit.

Even funnier, this was all hypothetical and yet taken as gospel. We hadn't even written the tests yet, so it was impossible to say whether they were slow or not. Nothing had been measured, no performance budget had been defined, no prototype of the supposedly slow tests had been written to demonstrate the point.

We ended up writing - no joke - less than 100 tests total, almost all of which hit the database, including some full integration tests, and the entire test suite finished in a few seconds.

I'm all for building in a way that respects performance as an engineering value, but we got lost somewhere along the way.

tne knly reason i like having mocks os so you can define all the data objects the whole way through, and see the whole workflow as if you'd written it on paper, without having to open a bunch of files or jump around. just top to bottom every object that is supposed to be there in what order

I like Mockito, it’s useful, but I’ll never understand mocking out DB stuff.

Why fake it when an integration test tests the real thing.

I’ve seen what you clearly have. Mocked ResultSets, mocked JDBC templates. “When you get SQL, it should be this string. These parameters should be set. Blah blah.”

It’s so much work. And it’s useless. Where does that SQL to check come from? Copy and paste, so it won’t catch a syntax error.

Test data is faked in each result. You can’t test foreign keys.

Just a bad idea. You’re so right. I find it odd some people are so anti-mock. Yeah it gets abused but that’s not the tool’s fault.

But DB calls are not a good spot to mock out.

> I almost lost my shit.

Hopefully one day you'll back at that, and realise what an immature attitude that was.

I'm sorry you felt the need to post this in response to a figure of speech. Please keep your moralizing to yourself. Thank you.

Well of course it was a figure of speech - if you meant it literally then I would not by prodding you about your medical condition. I was posting in response to your story that a colleague proposed a way of working or technical solution to something, and that this in some way enraged or otherwise upset you. And then you thought that made a good story to share online. That is, as I said, an immature response to a disagreement.

Unless that whole paragraph was a figure of speech, in which case what I said doesn't apply, and we can both go about our days.

No thanks, I'll pass on the amateur psychoanalysis. Good luck to you sir.

Hopefully one day I won’t have to deal with clowns like you.

Aw don't say that, you'll make me sad! There's nothing worse than a sad clown.

What if you're calling an API? How do you test a tool that does a bunch of back-and-forth with another service without actually hitting that other service? Do you have to spin up your entire k8s ecosystem in a local cluster just to validate that the logic in one function is sound? Do you have to deliberately misconfigure it in order to test that your function handles errors properly?

More broadly, suppose foo() has an implementation that depends on Bar, but Bar is complicated to instantiate because it needs to know about 5 external services. Fortunately foo() only depends on a narrow sliver of Bar's functionality. Why not wrap Bar in a narrow interface—only the bits foo() depends on—and fake it?

I'm not a maximalist about test doubles. I prefer to factor out my I/O until it's high-level enough that it doesn't need unit tests. But that's not always an option, and I'd rather be flexible and use a test double than burden all my unit tests with the full weight of their production dependencies.

I have never done a full k8s but I have started dbus on a non-standard port for isolation.

how are you deliberately breaking those dependencies? or are you only testing the happy path?

you could extend this to say 85% of the tome just write the code directly to prod and dont have any tests. if you broke something, an alarm will go off

Why substitute dependencies? Is the isolation worth it?

For the same reason you isolate variables in a scientific experiment; to ensure you're controlling the test that you're running, and not accidentally testing something else.

To easily simulate failure cases, a range of possible inputs, bad data etc.

To make the testing process faster when you have hundreds or thousands of tests, running on multiple builds simultaneously across an organisation.

Off the top of my head :-)

I don’t think it’s worth doing that, and comparing it to scientific experiments doesn’t really apply.

You can do all that without mocks as well.

Making the tests run faster at the expense of better tests seems counterproductive.

Now you should think of reasons why you should not isolate.

> I don’t think it’s worth doing that

OK; it's your choice to do what you think is right.

> and comparing it to scientific experiments doesn’t really apply.

Why not? I think it's a fairly apt comparison; you have a theory ("this piece of code does the following things"), and write tests to prove it.

> You can do all that without mocks as well.

OK, but mocks make it easier and cleaner - so why wouldn't I do that?

> Making the tests run faster at the expense of better tests seems counterproductive.

Smaller, more focused, cleaner tests are better in my opinion; speed is a beneficial side effect.

> Now you should think of reasons why you should not isolate.

Why? That's your argument - it's not on me to prove it for you. If you can give me some good reason why mocking out the interfaces you are not testing is a bad idea, and some better alternative, then we can have a discussion about it.

I don’t want to take too much of a tangent, but in scientific studies, you are trying to understand some phenomena, and isolating variables can help with very complex things. A test is typically not that complex. Another example is the use of animals in testing medicine, it can help but it obviously would be much better to test directly on humans but we don’t for good reason.

Your position is reasonable and I do think isolation can be beneficial, but I still wouldn’t use mocking to do it.

>Smaller, more focused, cleaner tests are better in my opinion.

Cleaner is subjective. I can write “small” and “focused” functional tests that are also quick to run.

I am of the opinion that functional tests provide more value. They are testing more of the actual code than an approximation, which in turn gives a better indicator that it works. Functional tests are less likely to change unless the input/output changes.

Now let’s say you mock something in your function. Let’s say you make a change to that but the input and output are the exact same. Now you have to update your test.

> in scientific studies, you are trying to understand some phenomena, and isolating variables can help with very complex things.

Not to labour the point here, but no, the primary reason you isolate variables in a scientific experiment is that you want to ensure you're only testing the thing you intend to test. A medical study is a good example - you want to be sure that the effect you observed was due to the drug you're testing, and not some unrelated lifestyle factor.

Thanks for sharing your views on the rest; there was just one thing I wanted to expand on:

> Now let’s say you mock something in your function. Let’s say you make a change to that but the input and output are the exact same. Now you have to update your test.

I think the scenario you're describing here is: a function's dependencies have changed, but the inputs and outputs of that function have not; therefore even though the behaviour is the same, the tests still need to be updated. Is that right? In which case I would say: of course you need to update the tests - the dependencies have changed and therefore the behaviour of the function depends on different things and you need to model the behaviour of those new things in order to properly test the original function. To me this objection only holds if you are mainly focussed on code coverage; however, to me, good testing exercises the same code paths in multiple different ways to stress the code and ensure that the results are correct given all possible inputs. The dependencies of a function are also inputs of a kind.

I appreciate your thoughtful comments but we do disagree.

>Is that right? In which case I would say: of course you need to update the tests.

That is right. I think it is bad for you to need to update a test where the input and output are the same. Your mock is there for you to essentially ignore, but now you need to update the test. You now do not know if you introduced a bug.

You are losing out on encapsulation, the test should not know about the internals, generally speaking.

>The dependencies of a function are also inputs of a kind.

Typically that should not be a concern to the caller of the function.

Ok, can someone explain this to someone with double digit iq? Is it validating the adapter idea or something else?

What does Agent mean in this context? And what is "dynamic attachment of agents"?

Tools for profiling, debugging, monitoring, thread analysis, and test coverage analysis can attach to the Java Virtual Machine (JVM) using the 'Tool Interface'

If you've got a running java process on your local machine right now, you can use 'jconsole' to see the stack traces of all threads, inspect various memory statistics, trigger an immediate garbage collection or heap dump, and so on. And of course, if the tool is an instrumenting profiler - it needs the power to modify the running code, to insert its instrumentation. Obviously you need certain permissions on the host to do this - just like attaching gdb to a running process.

This capability is used not just by for profiling, debugging and instrumentation but also by mockito to do its thing.

Java 21 introduced a warning [1] saying this will be disabled in a forthcoming version, unless the process is started with '-XX:+EnableDynamicAgentLoading' - whereas previously it was enabled by default and '-XX:+DisableAttachMechanism' was used to disable it.

The goal of doing this is "platform integrity" - preventing the attachment of debugging tools is useful in applications like DRM.

[1] https://openjdk.org/jeps/451

A JVM agent is able to instrument and modify running JVM applications. Stuff like debugging, hot patching, etc rely on this. You used to be able to tell the JVM to listen on a port where you could connect debuggers (agents) dynamically at runtime, but that was deemed a security issue so now you can only declare specific agents at launch time through command-line flags.

See https://openjdk.org/jeps/451: JEP 451: Prepare to Disallow the Dynamic Loading of Agents, which has a lot of background on the topic.

I respect the maintainer's decision, but I don't understand the justification.

> but when it was communicated with Mockito I perceived it as "Mockito is holding the JVM ecosystem back by using dynamic attachment, please switch immediately and figure it out on your own".

Who did the communication? Why is dynamic attachment through a flag a problem, and what was the solution? Why is "enable a flag when running tests" not a satisfactory solution? Why do you even need a _dynamic_ agent; don't you know ahead of time exactly what agent you need when using Mockito?

> While I fully understand the reasons that developers enjoy the feature richness of Kotlin as a programming language, its underlying implementation has significant downsides for projects like Mockito. Quite frankly, it's not fun to deal with.

Why support Kotlin in the first place? If it's a pain to deal with, perhaps the Kotlin user base is better served by a Kotlin-specific mocking framework, maintained by people who enjoy working on those Kotlin-specific code paths?

Mockito was indeed a poor fit for Kotlin. MockK is the one. Except I suppose for shops that have projects that mix Java and Kotlin and already have a Mockito tests.

> Why support Kotlin in the first place?

Some complexities are discovered along the way, people don't know everything when they start.

They could also drop the support after some time, but then it would have created other set of problems for adoption and trustworthiness of the project.

Sounds like a problem that the Mockito team created for themselves, and like a problem only they can solve.

> That's because starting JVM 22, the previous so-called "dynamic attachment of agents" is put behind a flag.

Ok am I being stupid or is the pragmatic solution not to just to enable this flag for test runs etc. and leave it off in prod?

I think the problem is that it's on his users to enable this flag, not something that can be done by Mockito automatically.

Most people want their test suite to pass. If they ugprade java and mockito prints out a message that they need to enabled '--some-flag' while running tests they're just going to add that flag to surefire in their pom. Seems like quite a small speedbump.

> To me, it felt like the feature was presented as a done deal because of security.

Not security, but integrity, although security (which is the #1 concern of companies relying on a platform responsible for trillions of dollars) is certainly one of the primary motivations for integrity (others being performance, backward compatibility or "evolvability", and correctness). Integrity is the ability of code to locally declare its reliance on some invariant - e.g. that a certain class must not be extended, that a method can only be called by other methods in the same class, or that a field cannot be reassigned after being assigned in the constructor - and have the platform guarantee that the invariant is preserved globally throughout the lifetime of the program, no matter what other code does. What we call "memory safety" is an example of some invariants that have integrity.

This is obviously important for security as it significantly reduces the blast radius of a vulnerability (some attacks that can be done in JS or Python cannot be done in Java), but it's also important for performance, as the compiler needs to know that certain optimisations preserve meaning. E.g. strings cannot be constant-folded if they can't be relied upon to be truly immutable. It's also important for backward-compatibility or "evolvability", as libraries cannot depend on internals that are not explicitly exposed as public APIs; libraries doing that was the cause of the migration pain from Java 8 to 9+, as lots of libraries depended on internal, non-API methods that have changed when the JDK's evolution started picking up steam.

In Java, we've adopted a policy we call Integrity by Default (https://openjdk.org/jeps/8305968), which means that code in one component can violate invariants established by code in another component only if the application is made aware of it and allows it. What isn't allowed is for a library - which could be some fourth-level dependency - to decide for itself at some point during the program's execution, without the application's knowledge, that actually strings in this program should be mutable. We were, and are, open to any ideas as long as this principle is preserved.

Authors of components that do want to do such things find the policy inconvenient because their consumers need to do something extra that isn't required when using normal libraries. But this is a classic case of different users having conflicting requirements. No matter what you do, someone will be inconvenienced. We, the maintainers of the JDK, have opted for a solution that we believe minimises the pain and risk overall, when integrated over all users: Integrity is on by default, and components that wish to break it need an explicit configuration option to allow that.

> built on a solid foundation with ByteBuddy

ByteBuddy's author acknowledges that at least some aspects of ByteBuddy - and in particular the self-loading agent that Mockito used - weren't really a solid foundation, but now it should be: https://youtu.be/AzfhxgkBL9s?t=1843. We are grateful to Rafael for explaining his needs to us so that we could find a way to satisfy them without violating Integrity by Default.

Is there a point at which library maintainer feedback would meaningfully influence a by-default JVM change?

I keep a large production Java codebase and its deployments up-to-date. Short of upstreaming fixes to every major dependency, the only feasible way to continue upgrading JDK/JVM versions has often been to carry explicit exceptions to new defaults.

JPMS is a good example: --add-opens still remains valuable today for important infra like Hadoop, Spark, and Netty. If other, even more core projects (e.g. Arrow) hadn't modernized, the exceptions would be even more prolific.

If libraries so heavily depended upon like Mockito are unable to offer a viable alternative in response to JEP 451, my reaction would be to re-enable dynamic agent attachment rather than re-architect many years of test suites. I can't speak for others, but if this reaction holds broadly it would seem to defeat the point of by-default changes.

> Is there a point at which library maintainer feedback would meaningfully influence a by-default JVM change?

Of course, but keep in mind that all these changes were and are being done in response to feedback from other users, and we need to balance the requirements of mocking frameworks with those of people asking for better performance, better security, and better backward compatibility. When you have such a large ecosystem, users can have contradictory demands and sometimes it's impossible to satisfy everyone simultaneously. In those cases, we try to choose whatever we think will do the most good and the least harm over the entire ecosystem.

> JPMS is a good example: --add-opens still remains valuable today for important infra like Hadoop, Spark, and Netty. If other, even more core projects (e.g. Arrow) hadn't modernized, the exceptions would be even more prolific.

I think you have answered your own question. Make sure the libraries you rely on are well maintained, and if not - support them financially (actually, support them also if they are). BTW, I think that Netty is already in the process of abandoning its hacking of internals.

Anyone who has hacked internals agreed to a deal made in the notice we had in the internal files for many years prior to encapsulation [1], which was that the use of internals carries a commitment to added maintenance. Once they use the new supported mechanisms, that added burden is gone but they need to get there. I appreciate the fact that some open source projects are done by volunteers, and I think their users should compensate them, but they did enter into this deal voluntarily.

> If libraries so heavily depended upon like Mockito are unable to offer a viable alternative in response to JEP 451

But they have, and we advised them on how: https://github.com/mockito/mockito/issues/3037

The main "ergonomic" issue was lack of help from build tools like Gradle/Maven.

[1]: The notice was some version of:

    WARNING: The contents of this source file are not part of any supported API.
    Code that depends on them does so at its own risk: they are subject to change or removal without notice.

Of course, we did end up giving notice, usually a few years in advance, but no amount of time is sufficient for everyone. Note that JEP 451 is still in the warning period that started over two years ago (although probably not for long)

What am I missing that makes this change a massive headache if just setting a flag gets the old behavior?

The auto-attach flag isn’t a huge deal, since it’s a one-liner that can be statically documented and the fix works in all cases. The bigger issue is the JDK / runtime team’s stance that libraries should not be able to dynamically attach agents, and that the auto-attach flag might be removed in the future. You can still enable mockito’s agent with the —javaagent flag, but you have to provide the path to the mockito jar, and getting that path right is highly build-system-dependent and not something the mockito team can document in a way that minimizes the entry threshold for newcomers.

Nothing makes it a massive headache, but I think that Maven/Gradle don't make it as easy as they could and should.

I haven't read the discussion but this seems like the obvious answer considering the flag only needs to be set during test.

Presumably this might miss some edge case (where something else also needs the flag?) though an explicit allow of the mockito agent in the jvm arg would have solved for that.

> though an explicit allow of the mockito agent in the jvm arg would have solved for that.

You can and should explicitly specify Mockito as an agent in the JVM configuration, as it is one.

> some attacks that can be done in JS or Python cannot be done in Java

Examples?

In September there was a supply-chain attack on NPM where the payload code injected hooks into the DOM API. Changing the behaviour of encapsulated components, like Java's standard library, is not possible now without the application explicitly allowing code to break the integrity of the encapsulated component.

Sad to see an important project's core maintainer leave but their justification seems very understandable. It is sad so much of OSS is maintained by very few as they alluded to in the XKCD comment, an especially given they felt the JVM ecosystem was causing them pain with limited support or feedback possible. I think it is always a little irresponsible to cause a great deal of breakage and not be there to support those who you break downstream of your project.

As someone who is not in the Java world, why does Java need a mocking library? Interface based polymorphism is not enough?

Mocks make it easy to record and assert on method invocations. Additionally spys (instance mocks) are really useful when you need to forward to the real method or rely on some state.

At the moment I can't see anything Mokckito gives that you technically couldn't implement yourself via subclassing and overriding, but it'd be a lot of boilerplate to proxy things and record the arguments.

Subclasing and overriding is not a good idea. There is no compilation failure if you forget to override a function which can lead to flakey tests at best and prod data impact at worst.

your test environment should not have the credentials to write to prod data. yiiiiikes!

Credentials end up existing in prod because the person used Mochito and didn’t override the function for providing credentials :’c

Credentials should only be provided at the application root, which is going to be a different root for a test harness.

Mockito shouldn't change whether or not this is possible; the code shouldn't have the prod creds (or any external resource references) hard coded in the compiled bytecode.

I totally agree, I’m being tongue in cheek, but given how poor some codebases can be, the more precautions the better ie compilation failures on non-mocked functions.

Mockito allows one to write mocks in tests for code that doesn't use dependency injection and isn't properly testable in any other way.

On the one hand, you should just design things to be testable from the start. On the other... I'm already working in this codebase with 20 years of legacy untestable design...

Google API libraries mark every class as "final" so it's not trivial to mock-extend it for tests. But third-party IO is exactly the thing you'd want to mock.

Probably because they zealously followed "Effective Java" book.

> But third-party IO is exactly the thing you'd want to mock.

You write an adapter.

Once you start writing adapters you need a way to instantiate them to choose between implementations, and factories is often used for this. Then you might generalize the test suites to make the setup easier and you end up with the infamous FactoryFactory pattern.

No, some other library classes accept only their own, not my adapter.

Not mentioning of course needless copy-pasting dosens of members in the adapter. And it must be in prod code, not tests, even though it's documentation would say "Adapter for X, exists only for tests, to be able to mock X".

You wrap whole 3rd party dependency in an adapter.

That's a lot of upfront work and maintenance, not to mention the friction of needing to mentally translate every occurrence of OurFooAdapter to Foo in order to find documentation.

Yeah, well, good code takes some thought to produce. More news at 11.

The second argument still holds -- all those wrappers will exist in prod only for tests.

Moreover, that wrapper library is now a pretty large piece of code, and we'd want to maintain and test as well. But cannot without hacks.

I think it's easier to unpack/remove-"final"/re-compile the library .jar during the build time. Will be more stable to any changes than an adapter.

Mockito uses declarative matching style of specifying what should be mocked. You don't need to implement or even stub all of interface methods since Mockito can do it itself. It may be extremely concise. For example, interfaces may have tens methods or even more, but only one method is needed (say, java.sql.ResultSet). And finally probably the most important thing, interaction with mocks is recorded and then can be verified if certain methods were invoked with certain arguments.

That’s the seductive power of mocking - you get a test up and running quickly. The benefit to the initial test writer is significant.

The cost is the pain - sometimes nightmarish - for other contributors to the code base since tests depending on mocking are far more brittle.

Someone changes code to check if the ResultSet is empty before further processing and a large number of your mock based tests break as the original test author will only have mocked enough of the class to support the current implementation.

Working on a 10+ year old code base, making a small simple safe change and then seeing a bunch of unit tests fail, my reaction is always “please let the failing tests not rely on mocks”.

> Someone changes code to check if the ResultSet is empty before further processing and a large number of your mock based tests break as the original test author will only have mocked enough of the class to support the current implementation.

So this change doesn't allow an empty result set, something that is no longer allowed by the new implementation but was allowed previously. Isn't that the sort of breaking change you want your regression tests to catch?

I used ResultSet because the comment above mentioned it. A clearer example of what I’m talking about might be say you replace “x.size() > 0” with “!x.isEmpty()” when x is a mocked instance of class X.

If tests (authored by someone else) break, I now have to figure out whether the breakage is due to the fact that not enough behavior was mocked or whether I have inadvertently broken something. Maybe it’s actually important that code avoid using “isEmpty”? Or do I just mock the isEmpty call and hope for the best? What if the existing mocked behavior for size() is non-trivial?

Typically you’re not dealing with something as obvious.

What is the alternative? If you write a complete implementation of an interface for test purposes, can you actually be certain that your version of x.isEmpty() behaves as the actual method? If it has not been used before, can you trust that a green test is valid without manually checking it?

When I use mocking, I try to always use real objects as return values. So if I mock a repository method, like userRepository.search(...) I would return an actual list and not a mocked object. This has worked well for me. If I actually need to test the db query itself, I use a real db

The alternative to what? Using mocks?

For example, one alternative is to let my IDE implement the interface (I don’t have to “write” a complete implementation), where the default implementations throw “not yet implemented” type exceptions - which clearly indicate that the omitted behavior is not a deliberate part of the test.

Any “mocked” behavior involves writing normal debuggable idiomatic Java code - no need to learn or use a weird DSL to express the behavior of a method body. And it’s far easier to diagnose what’s going on or expected while running the test - instead of the backwards mock approach where failures are typically reported in a non-local manner (test completes and you get unexpected invocation or missing invocation error - where or what should have made the invocation?).

My test implementation can evolve naturally - it’s all normal debuggable idiomatic Java.

It doesn't have to be a breaking change -- an empty result set could still be allowed. It could simply be a perf improvement that avoids calling an expensive function with an empty result set, when it is known that the function is a no-op in this case.

If it's not a breaking change, why would a unit test fail as a result, whether or not using mocks/fakes for the code not under test? Unit tests should test the contract of a unit of code. Testing implementation details is better handled with assertions, right?

If the code being mocked changes its invariants the code under test that depends on that needs to be carefully re-examined. A failing unit test will alert one to that situation.

(I'm not being snarky, I don't understand your point and I want to.)

The problem occurs when the mock is incomplete. Suppose:

1. Initially codeUnderTest() calls a dependency's dep.getFoos() method, which returns a list of Foos. This method is expensive, even if there are no Foos to return.

2. Calling the real dep.getFoos() is awkward, so we mock it for tests.

3. Someone changes codeUnderTest() to first call dep.getNumberOfFoos(), which is always quick, and subsequently call dep.getFoos() only if the first method's return value is nonzero. This speeds up the common case in which there are no Foos to process.

4. The test breaks because dep.getNumberOfFoos() has not been mocked.

You could argue that the original test creator should have defensively also mocked dep.getNumberOfFoos() -- but this quickly becomes an argument that the complete functionality of dep should be mocked.

Jumping ahead to the comments below: obviously, I mentioned `java.sql.ResultSet` only as an example of an extremely massive interface. But if someone starts building theories based on what is left unsaid in the example for those from outside the Java world, one could, for instance, assume that such brittle tests are simply poorly written, or that they fail to mitigate Mockito's default behavior.

In my view, one of the biggest mistakes when working with Mockito is relying on answers that return default values even when a method call has not been explicitly described, treating this as some kind of "default implementation". Instead, I prefer to explicitly forbid such behavior by throwing an `AssertionError` from the default answer. Then, if we really take "one method" literally, I explicitly state that `next()` must return `false`, clearly declaring my intent that I have implemented tests based on exactly this described behavior, which in practice most often boils down to a fluent-style list of explicitly expected interactions. Recording interactions is also critically important.

How many methods does `ResultSet` have today? 150? 200? As a Mockito user, I don't care.

As someone who has been out of Java for close to 10 years now, you certainly could do without Mockito, but you'd be writing a lot of boiler plate code repetitively. There's also the case of third-party libraries that you don't control and Mockito has decent facilities for working with those, especially when you're working with a codebase that isn't pure DI and interfaces.

The point is to let you create mocks without having to go through the whole polymorphism rigmarole, without forcing classes to define a separate interface or anything like that.

because even supposing you have an interface for your thing under test (which you don't necessarily, nor do you necessarily want to have to) it lets you skip over having to do any fake implementations, have loads of variations of said fake implementations, have that code live somewhere, etc etc.

Instead your mocks are all just inline in the test code: ephemeral, basically declarative therefore readily readable & grokable without too much diversion, and easily changed.

A really good usecase for Java's 'Reflection' feature.

An anonymous inner class is also ephemeral, declarative, inline, capable of extending as well as implementing, and readily readable. What it isn't is terse.

Mocking's killer feature is the ability to partially implement/extend by having some default that makes some sense in a testing situation and is easily instantiable without calling a super constructor.

Magicmock in python is the single best mocking library though, too many times have I really wanted mockito to also default to returning a mock instead of null.

> What it isn't is terse

Yeah, it's funny, I'm often arguing in the corner of being verbose in the name of plain-ness and greater simplicity.

I realise it's subjective, but this is one of the rare cases where I think the opposite is true, and using the 'magic' thing that shortcuts language primitives in a sort-of DSL is actually the better choice.

It's dumb, it's one or two lines, it says what it does, there's almost zero diversion. Sure you can do it by other means but I think the (what I will claim is) 'truly' inline style code of Mockito is actually a material value add in readability & grokability if you're just trying to debug a failing test you haven't seen in ages, which is basically the usecase I have in mind whenever writing test code.

I cannot put my finger on it exactly either. I also often find the mocking DSL the better choice in tests.

But when there are many tests where I instantiate a test fixture and return it from a mock when the method is called, I start to think that an in memory stub would have been less code duplication and boilerplate... When some code is refactored to use findByName instead of findById and a ton of tests fail because the mock knows too much implementation detail then I know it should have been an in memory stub implementation all along.

Before Mockito, it was common (where I worked) to create an interface just to support testing. This is an anti-pattern in my opinion. To create interfaces just for testing complicates the code and it is one of my pet peeves. It also encourages the factory pattern.

I prefer Mockito's approach.

It’s definitely a bit annoying and verbose in Java but I think creating an interface to support testing is a net positive. That interface is the specification of what that concrete class requires it’s dependencies to do.

I think all the dependencies of a class should define behaviour not implementation so it’s not tightly coupled and can be modified in the future. If you have a class that injects LookUpService, why not put an interface LookUpper in front of it? It’s a layer of indirection but we have IDEs now and reading the interface should be easier or at least provide context.

There are many cases where you don't control the library code your code depends on that you want to test. Also, the FactoryFactoryFactory patterns can be quite cumbersome and simply mocking out something makes for a far simpler test. There are likely more common cases.

Arguably it doesn't. Mocking is over used and you should just use a real implementation or distributor provided fake whenever possible.

It doesn't. But good luck teaching hordes of enterprise "developers".

>Energy drain because of JVM agent change

So funny, he essentially works for free for 10 years, then finally burns out because he doesn't want to put up with a bunch of annoying work? This is why you shouldn't work on open source unless you have a business strategy to get paid. Tons of stuff in life is 100x more annoying and exhausting if you aren't making any money. If he was making $1 million per year from this I doubt his energy would be drained.

Disagree, its working for money that's draining. Extrinsic motivation is toxic to many things that might motivate an open source maintainer.

What about paying your mortgage, food, utilities etc.? Or is that all supposed to come from savings or inheritance?

I don't claim to have a recipe that sustains itself, I just find that my energy diminishes if I'm not doing a project for its own sake.

If after ten years the spark was gone we should be happy for that ten year contribution, but I don't think there's any reason to assume that money would've prevented the problems that motivated him to step down. Maybe it could incentivize him to muscle through after the magic was gone, but that's a different sort of thing.

> If he was making $1 million per year from this I doubt his energy would be drained.

People quit all the time from highly paid jobs due to burnout. This absolutely does not follow.

> burns out because he doesn't want to put up with a bunch of annoying work

It’s more than annoying work, it’s pointless work needlessly created by people other than him.

It’s like migrating from Java 8 to newer versions, the decision makers placed backwards compatibility at the back of their priority list. Literally a decade later it’s still griefing migrating users, all because “Jakarta not javax” nonsense. I’m greatly simplifying but that’s the essence of it.

Now we have some genius decision to I guess protect against untrusted code doing unexpected things. And at the same time Applets are gone and Security Manager is gone. And the reality is that Java applications aren’t run with untrusted code. The run scripts define all the jars/classes used. If there was some malicious code that wanted to run, I’m fairly confident it would also just modify the run scripts to include this new flag.

So all we’ve gained is support headache and pain, and no real net gain in practice.

I wouldn't make such a conclusion. I don't think there is any info about whether OP got financial incentives for his work or not. In fact, he posted on Mastodon, he's gonna be doing open source Rust work further on.

An excellent stepping down post, nothing more could be hoped for. Enjoy the next projects

Beware of a supply chain attack

thank you for all you hard work!

Reply on news.ycombinator.com