Show HN: Fuzz Map – a GUI fuzzer, interactive demo | Hacker News Reader

Show HN: Fuzz Map – a GUI fuzzer, interactive demo

fuzzmap.io

205 pointsjonathanyc33 comments

     > In the short term, I'd like to make a plug-and-play version of Fuzz Map that supports end-to-end React fuzzing. You won't have to change your build process or use a special browser—just run ./fuzzmap localhost:8080 -p 9090 and start fuzzing or live programming! A reverse proxy will add instrumentation on the fly.

Eagerly waiting for this. I need this. I don't write tests for a certain React codebase [1], and this could be a life saver. Amazing work!

1: Before you hate, I'm not against tests and I advise on writing them. There are certain situations in this case.

I am so excited to see this live! I always wanted something like this for testing games and complex SPA's. UI's are particularly problematic for writing tests -- there's usually much more churn in that area of an application versus the business logic, and no matter how many tests we did try to write, users (and ambitious internal testers that clicked everywhere like crazy) would find a creative way to break things. I think having UI fuzzing in the CI pipeline will go a long way to deploying UI updates with more confidence. Congrats on launching!

I imagine this would be a very useful tool to have if you want to communicate about the design or user flow of an existing app, in addition to the highlighted use case of finding unexpected behavior and crashes.

And, if you can save 'snapshots' of a given fuzzmap to detect potential regressions and / or changes in general to the flow.

It's a HTML/JS GUI fuzzer, but the author saw this as the "default" to the point they didn't even specify that. At this point I've given up hope that native libraries and frameworks will return to the mainstream of desktop apps, but I'm still going to continue working with native libraries.

Even native programs are moving towards web tech, for eg GNOME shell is written in JavaScript and the PolicyKit component of the GNU/Linux desktop world uses JavaScript as a scripting language to define policies in. Of course there are Electron/Tauri apps for wrapping web tech in a native porting layer.

> eg GNOME shell is written in JavaScript

Is that some way related to why almost every GNOME shell extension is a memory hog (due to memory leaks?), I was fed up of replacing shell extensions in the hope of finding better one and finally disabled shell extensions altogether.

I've hardly had any GNOME crash after disabling shell extensions.

P.S. Congratulations on the launch, OP. Seems like a very useful too, Would try to play with it later.

I've noticed the memory leaks too. Unfortunately I can't live without GNOME shell extensions and I need to reboot/logout for security updates every now and then, so I just got enough RAM to let me continue working while leaks are going on.

I was doing the same, But unexpected crashes due to OOM killing became too much to handle with 32GB memory.

I replaced the extensions I use with standalone applications and I've never been happier with GNOME. But its sad that the extensions ruin the experience of GNOME and possibly one of the main reasons for the hate it gets as a DE.

I have less RAM than that. I use psi-notify to get a warning when GNOME is starting to use too much RAM. That is usually early enough that I can close a few apps to avoid the crash, then logout/reboot to regain the RAM.

There is also a GNOME/FreeDesktop memory PSI notifier thing but it isn't in Debian yet.

https://gitlab.freedesktop.org/hadess/low-memory-monitor/ https://www.hadess.net/2019/12/gmemorymonitor-low-memory-mon... https://www.hadess.net/2019/08/low-memory-monitor-new-projec...

I really wish there was a way to debug where all the leakage is coming from, I don't think it has always been the case :(

You might know this, But for those who don't it's very easy to get the stack trace and debug GNOME (or any package) with Arch Linux now due to inclusion of debug packages via Debuginfod service[1] (From which the symbols are automatically downloaded when using gdb).

So for GNOME shell,

   coredumpctl gdb /usr/bin/gnome-shell

   (gdb) bt full

Then head on to GNOME's gitlab to post the issue, Only to be confronted by similar issues being open for past five years :P

P.S. I mean no offense to GNOME developers and contributors, I love GNOME and use it on all my devices(including phone!) but sometimes I wish they worked more on stability than creating under cooked mechanisms which abuses the system(GNOME extensions).

[1] https://wiki.archlinux.org/title/Debugging/Getting_traces

coredumpctl and gdb aren't very useful at debugging out-of-memory situations. I've had better luck running plasmashell under heaptrack (koute likes to advertise bytehound but it hasn't worked on the programs I tried it on, though I've never reported the bugs). Unfortunately heaptrack usually just indicates that allocations are coming from the QV4 JS engine (since plasmashell, like GNOME Shell, is written substantially in JS), rather than giving me a C++ stack trace. You can try heaptrack on GNOME Shell as well, but you may not have much better luck considering that shell extensions are written in JS (it's possible you'll get a usefully-sized C++ stack trace telling you what resource the JS code is leaking).

Thank you for sharing the information, I will be trying heaptrack.

Very nice!

Does this tool have a state model that you can check the application state against? In other words, does it have property-based testing? Here[1] is a demo of what that looks like.

[1] https://medium.com/criteo-engineering/detecting-the-unexpect...

Thank you! I'd like to support property-based testing in the future. For this demo I focused on visualization and more exploratory testing.

I figured that'd make the demo more approachable to people who weren't already into fuzzing or formal methods. But being able to automatically discover non-crashing bugs using property-based testing could be pretty powerful.

wow! this is very cool. I wonder how this would work with slightly more complicated UIs. Does it create too much noise? Does anyone need this kind of verbosity?

Thank you! Good question about complicated GUIs. Keeping the map simple enough to be usable ended up being the most interesting part of the project, engineering-wise--the end of the write up touches on map simplification! There's a lot more I'd like to do.

Do you plan to extend the support to other frameworks too, like Angular? A UI fuzzer would be handy to have in the toolbelt. Really enjoyed the write up, it answered some of the questions I had when thinking about fuzzers, for example the branch count blow up and optimizations, would read more if you decide to follow up with more development notes.

Wow, that is cool. How do you do code coverage? Console.log() every function call?

We add instrumentation at compile-time, just like conventional fuzzers!

    if (e) a else b

becomes

    if (e) { hit(1); a }
    else { hit(2); b }

There's more detail in the writeup if you curious, just scroll down!

Wow, that's really impressive. I'll probably start using this right away.

so is this a fuzzer for generating GUI elements to see what works or not? or one to generate input states that crash a GUI?

It runs GUI code (like React) and fuzzes the inputs not just to generate possible crashes but to discover different code paths and states.

The example they provide shows a simple food ordering system and the fuzzer eventually discovers how to place an order, but not before first encountering validation errors and fuzzing past them.

I'm not sure what you mean by "see what works." My understanding is that it's designed to automatically interact with a GUI and identify the possible different states that the UI could be in and the inputs required to enter those states. (Presumably this includes the state of being crashed.)

Super cool, thanks for sharing!

> Here's a 5-second video of real-time fuzzing on my laptop.

The space where the video is supposed to be is blank. Clicking on play instantly skips to 0:05 mark and nothing else happens. Firefox on Windows.

It's a bit pointless anyways imho (on Desktop). Just close the popup and click Fuzz.

i kind of like having the video. actually running the fuzzer uses a lot of CPU: it makes my fan startup.

it works fine here on Firefox + Debian GNU/Linux 11 stable/bullseye.

Plays fine in my Firefox for Windows, FWIW. Also works in Chrome desktop and mobile.

Why is this process called "Fuzzing"?

Isn't this a really uninformative/ misleading name?

Because it’s a fuzzy representation of input, in that it’s partially valid enough so to be able to use the system, but also invalid in a way that’s designed to find bugs.

https://en.wikipedia.org/wiki/Fuzzing

Thanks. Actually, if anything, the original source mentioned in that article strengthens my opinion. The term has nothing to do with 'fuzziness' as understood today. It was a fun, arbitrary term that was chosen for a student project on a whim rather than careful consideration. The 'fuzz' was referring to the 'fuzz generator', where 'fuzz' here seems to have been used as a synonym for 'crud', 'dirt', 'fluff', 'noise' etc, i.e as opposed to clean/useful/careful inputs. Nothing to do with the term 'fuzzy' as understood today (i.e. in terms of fuzzy logic or fuzzy set/measure theory).

Good to know though. I kept hearing this term and didn't know what it referred to. Thanks for the link!

Wow.

Reply on news.ycombinator.com