owickstrom

That sounds like a pretty ideal case for Bombadil. Yeah you can write custom generators for exactly this reason!

OK. Yeah so I made the decision early to go one level below Playwright and target CDP directly, because I believe I need the tight connection with Chrome/Chromium to make it as fast as possible. Also I really value the distribution aspects of writing this in Rust and compiling to a single executable with everything in it.

Do you mean test runners like JUnit, pytest, etc? Or browser test runners specifically?

Thanks! I don't know much about it, but from what I can tell from their site, it does differential testing based on screenshots and behavior, and mocks out network IO, focusing on the client side. I'm not sure what the AI part is, if it's for extracting information from the page, for generating actions, for smarter diffing, or some combination of those.

Bombadil doesn't mock the network (although I'm working with a contributor to add something like Playwright's Route system for manipulating responses). It runs on your full stack (if you want) and if you need determinism for perfect bug reproducibility, you should run it all in Antithesis.

On the AI front, my view is that Bombadil should be optimized for using LLMs to generate and refine specs over time (properties and action generators), but that it should be an offline process. It's not like a "browser testing agent". It's analogous to having the LLM generate a program, rather than having the LLM itself perform actions. For some problems, the latter makes more sense, but for many problems the former is better, and I believe that PBT (and things like proof systems, but that's not my domain) are suited to the former strategy.

I have some sketches on how the refinement-with-human-in-the-loop thing should work with Bombadil and LLMs, but it's only a roadmap item for now.

One smartass way to answer that for the general case is: you don't. And I mean that in the same sense as "you don't write example based tests using a PBT framework". You supply independent actions through generators and have it apply them "chaotically", not strictly in sensible sequences.

That said, for things like auth, you need to do some fixed steps to get anywhere at all. I haven't added support for this yet, but as I'm envisioning it, you'd either:

a) authenticate the browser using headers or cookies that you inject into the Bombadil browser, or b) use a "custom action" (essentially a stringifed JS function) that'd perform your particular steps

Or what you could do now: add preconditions to your actions so that they'd necessarily fill out the form before anything else could be done.

Another thing we might need in Bombadil is secrets, but for now I think it's fine to inject test credentials that are plain visible text.

Please let me know if you try it out and if you need any help!

I'll quote my lobste.rs reply to this question:

> I think his jolly chaotic good energy, along with how he's uncorruptible and sort of outside the normal world, fits a fuzzer in a spiritual sense. But it might all just be a good excuse to use one of my favorite names from my favorite book.

It's also my favorite part of the book, and definitely not something I'd skip. I know some people would object, but I think of it as an homage rather than stealing.

Cool. Is this publicly available?

Go for it! I've been meaning to do an "architecture of Bombadil" blog post that'd likely answer this question. It's not super advanced by any means, but it's a mindset shift to how you might think about browser testing coming from the mainstream frameworks like Playwright.

Yes, and in fact, capturing "prior" values with bindings and closing over them in temporal operator thunks is how you talk about some relation between s and s' in a Bombadil formula (not having that particular syntax though). It's a deliberate way of embedding this LTL flavor in JS/TS in the most natural and ergonomic way I could think of. I didn't want a deep EDSL or even a new bespoke spec language that people (and LLMs) would have to learn, and to have to write tools for. Now you can write Bombadil specs with a good LSP and be able to import packages off of npm or whathaveyou. Most web devs will probably be comfy with JS or TS, so that's why I chose that style.

I hope that makes sense?

Thanks for the nice feedback!

Hey, yeah, so actions are indeed independent and you can't express fixed sequences of them (e.g. fill out thus form by doing X, then Y, then Z). If you want to enforce certain sequentiality you'd need to precondition your generators. This is admittedly limited right now. I do want to add QoL things like `filter` on the generators to make that more ergonomic.

Also, as you note, you can't implement custom actions. And that's just something that I haven't gotten to yet. It'd be quite straightforward to add a plain function (toStringed) as one of the variants of the Action type and send that to the browser for eval. (Btw, we're taking contributions!)

Weighting is also important right now. There's no smart exploration in Bombadil yet, only blind random, so manual weighting becomes pretty crucial to have more effective tests (i.e. unlikely to run in circles). I'd like to both make the Bombadil "fuzzer" better at this, but eventually you might want to run Bombadil inside Antithesis instead to get a much better exploration, even in a single campaign.

The Until operator is also one of those things that I haven't gotten to yet. I actually didn't expect someone to hit this as a major limitation of the tool so quickly, which is why I've focused on other things. Surprising!

To add some more context, Bombadil is an OSS project with one developer (but we're hiring!) and it's 4 months old and marked experimental. I'm sorry you were disappointed by its current state, but it's very early days, so expect a lot of these things to improve. And your feedback will be taken into account. Thanks!

Fair enough! I'm probably doing a demo video that'd help with this. Maybe can condense that in text form into the readme as well.

Not yet but I definitely will record one soon. I've been working at the debugging UI for Bombadil (some hints at https://x.com/owickstrom/status/2034925537706020978) that will make for a nice cohesive demo.

Did you have a look at the intro in the manual? https://antithesishq.github.io/bombadil/1-introduction.html

If that's not clear, please let me know how we can improve it!

Cool. I think that is a very neat way of expressing properties of UIs (and stateful systems more generally) that works out nicely in testing. There are some gotchas related to the finiteness of testing, but it's manageable.

Let me know what bugs you find (in Bombadil itself or your next project!)

That's true, clean app state gets you far. And that's something I'm going to add to Bombadil once it gets an ability to run many tests (broad exploration, reruns, shrinking), i.e. something in the spec where you can supply reset hooks, maybe just bash commands.

Regarding state machines: yeah, it can often become an as-complex mirror of the system your testing, if the system has a large complicated surface. If on the other hand the API is simple and encapsulates a lot of complexity (like Ousterhout's "Deep Modules") state machine specs and model-based testing make more sense. Testing a key-value store is a great example of this.

If you're curious about it, here's a very detailed spec for TodoMVC in Bombadil: https://github.com/owickstrom/bombadil-playground/blob/maste... It's still work-in-progress but pretty close to the original Quickstrom-flavored spec.

I'd say it does! Bombadil is very new but its predecessor has found complicated and very real bugs in my work projects, and in the paper we wrote about it, we found bugs in more than half of the TodoMVC example apps: https://arxiv.org/abs/2203.11532

:) Thank you for trying it!

This is a great point. Bombadil _is_ also tied to the DOM much like those tools, but as you focus on providing just a set of generators (which can be largely the defaults already supplied by Bombadil), you get a lot of testing from a small spec. You might need to specify some parts in terms of DOM selectors and such, and that has coupling, but I think the power-to-weight ratio is a lot better because of PBT.

Good feedback! Short answer: a lot of stuff is remaining. It's a very new projects and I've been trying to cover the basics. There's a ton to do around better state space exploration, reporting/debugging (working on this now!), integration with other tools and platforms like CI, etc. But a living section in the README or the Manual for "planned but not yet built" probably makes sense.

Thanks, glad you like it! Do you mean the temporal logic aspect of it?

Sure, server-side or client-side generated UIs tend to have a lot more interesting complexity to test. But I do want to bring up that with the specification language being TypeScript, you can validate some basic properties even for statically generated sites. I wrote a spell checker for Bombadil that uses https://github.com/wooorm/nspell and a custom dictionary and found a bunch of old typos on my own blog.

OK. What kind of problems do you hit from third-party libraries with that?

Interesting. What kind of properties are you checking?

Glad you noticed! I've been putting quite some energy into keeping things this way. VERY worth it, IMO.

Hey, yeah the default specification includes a set of action generators that are picked from randomly. If you write a custom spec you can define your own action generators and their weights.

Rerunning things: nothing built for that yet, but I do have some design ideas. Repros are notoriously shaky in testing like this (unless run against a deterministic app, or inside Antithesis), but I think Bombadil should offer best-effort repros if it can at least detect and warn when things diverge.

Shrinking: also nothing there yet. I'm experimenting with a state machine inference model as an aid to shrinking. It connects to the prior point about shaky repros, but I'm cautiously optimistic. Because the speed of browser testing isn't great, shrinking is also hard to do within reasonable time bounds.

Thanks for the questions and feedback!

btw, some background on the project: https://wickstrom.tech/2026-01-28-there-and-back-again-from-...

Author here, happy to answer questions about Bombadil! :)

I'm using oxc_traverse and friends to implement on-the-fly JS instrumentation for https://github.com/antithesishq/bombadil and it has been awesome. That in combination with boa_engine lets me build a statically linked executable rather than a hodgepodge of Node tools to shell out to. Respect to the tools that came before but this is way nicer for distribution. Good times for web tech IMO.

I think it works nicely either at a table or desk with the stand, or from the sofa with a low table. Sometimes I zoom in a bit in termux. 80 columns fit anyway.