Stop Filing Bugs, File a Container
blog.runkit.com
blog.runkit.com
I think we've all been in a situation where we have a bug reproducing in front of us but when we ask for help by describe or even give code to someone else they say "hmm, it works on my machine". Since software rarely runs in a vacuum, having better system level access can sometimes be the only way to debug it.
Our hope here is that RunKit notebooks make it easier to see what's wrong, even when the problem is less obvious.
Nothing prevents you from 1) copy pasting the code to your own computer to run it exactly the same way as you would have previously done, or even better 2) hitting the "download" link on the left which downloads the code + shrink-wrap file so you are using the same dependencies as the user. On top of that, we have also made stack traces and other elements of the UI a lot friendlier.
From a reproducibility perspective, the notebook represents an undeniable instance that the bug did happen, along with all the background information you are usually asked (what version of node? what version of package? any other dependencies?). The goal is to make that all apparent in the code itself, vs extraneous other files like package.json or such. Here is an example: https://runkit.com/tolmasky/my-bug/1.0.0
If you are interested in the underlying technology, we've documented it here: http://blog.runkit.com/2015/09/10/time-traveling-in-node.js-...
That has not been my experience. Especially with software written under time constraints including the constraint of "this is FOSS software that I'm writing in my free time."
I was able to track it down once I realized that signal handlers were, technically speaking, another thread of execution in a special context ...
Not even close. Different classpaths, different versions of packages, some ENV setting like LOCALE causing havoc, a different configuration between the two installations of the programs that are debugged, a single core vs multi core cpu that masks some race conditions, there are literally MILLIONS of things that can, and in my experience, have, cause something to work in one machine and not another.
In fact memory access violations do not even register as a blip on the top-100 reasons...
docker container diffIt's like the live javascript/html/css editors, but for systems!
https://runkit.com/npm/lodash and poof, you can run queries against lodash and share and they can edit it. (which you could do in a plunkr too, if you loaded it). I didn't even need to sign up...
Looks like http://code.runnable.com/ does similar for more stacks, but you have to sign up for that.
I tried to play with docker, but haven't seen much use for it in for my current workflows.
I've always liked the idea of Nix but I've never used it. How easy is it to extract and understand the details of the environment you've constructed to build your project?
i also haven't used nix, but i hear it truly solves the problem.
> How easy is it to extract and understand the details of the [Nix] environment you've constructed to build your project?
The "extract" part is easy: Nix uses a declarative language to describe the environment, so in that sense it's like docker, but more fine-grained: each package is built in isolation, and installed to a read-only filesystem.
As far as "understanding the details", it depends on what you want to know. You can use "nix-shell" to enter a build environment in a shell, e.g.
nix-shell -E 'some Nix expression'
nix-shell -p 'some Nix expression'
For example, I'm currently tracking down a space leak in a Haskell program by running the following command after each edit of the source code:
nix-shell -p 'with import <nixpkgs> {}; profiledHaskellPackages.callPackage (runCabal2nix { url = ../myTester; }) { myDependency = profiledHaskellPackages.callPackage (runCabal2nix { url = ./.; }) {}; }' --run 'tester +RTS -M100M -xc'
This will be run in an environment containing Haskell packages built from the source code in "./." (the current directory, for "myDependency") and "../myTester" (for the "tester" command). These packages and all of their dependencies will be built with profiling enabled.
Running this command over and over will just re-use the previously-built environment. If I edit the source in ./. or in ../myTester, Nix will notice that the hashes have changed, rebuild the packages and their dependents and make a new environment using those. This makes it easy for me to test potential fixes.
All problems in computer science can be solved by another level of indirection, except of course for the problem of too many indirections. - David Wheeler
This is a great concept : remove the work to reproduce an environment, and use the freed resource to understand said env.
Major advantages are:
- the bug reporter may not know his/her case is special and won't provide you with the information you need to see it. Providing a running container with the bug just obliviate that.
- the running container can come with a docker file, which gives you everything: libs, versions, settings. It's basically a summary of the stuff you need for it to go wrong.
- you can try locally the stuff, even if your setup is completly different, without messing with your setup. Because complex bugs rarely are reproducible with a bunch or pip install, apt get and other yum incantations.
It's not good for all bugs. I would say it's actually bad for most bugs. But for the bugs that needs it, it seems fantastic.
If your codebase has significant undocumented environmental dependencies and these are hard to script in place, there's something far more fundamentally wrong with your development process than bug reporting.
I create Python libs that are used in so many various configurations by so many people I don't know. They come back and say "hey, I got this stack trace" (in the best case).
Now I have to play the guessing game. Is it me, is it you ? Wrong path ? Permission problem ? Bad conf ? Network is having trouble ? Server is this particular linux version and and there is something important about the SEL setup here ? Oh but upstart/systemd don't behave the same way. Output is redirected here look. Stuff is not the expected encoding. Na it was not a bug but you're file is corrupted. What the heck is this data format ?
Etc, etc.
You thing you though about every single thing ? Your error handling is perfect for all IO ? You deal with all encoding, all user inputs perfectly ? You know all the little OS peculiarities that will make your subprocess run in the exact way you think ?
Of course you don't, nobody is perfect, we don't have infinite resources. But there are infinite ways to fail.
And for many things, you can figure it out with with just your code base and the error because it's a simple cause. But from time to time arrive this terrible bug that is a mix between a strange LOCAL, this particular version of the VM you use but only in one time zone with this env variable set. And for that, yes, a good container with a reproducible bug in the proper env is an interesting idea.
Apparently, you don't ship software that is wildly used enough or you would not be that arrogant.
Deploying on your own 100 servers is hard. Try seing your code deployed on 1000 servers that you don't own, nor configure.
In short throwing your hands up in frustration at the complexity of software is not a solution and gets you nowhere. The way we deal with "infinite ways to fail" is to control the environment. These days, quality projects are expected to version control their environments and conduct test deployments within a representative set of environments using a representative set of configurations.
Docker provides an easy way to do this ("always deploy on <distro>-<os>-<version>"), but it's only one approach. Another free and relatively straightforward place to start getting up to speed would be automating build and test processes with Travis CI for an open source project.
Deploying to any number of tested environments is trivial.
If you create a Python lib, 10000 people will pip install it. You have no control on the env.
If you create a deb package, you will see ppa and it will be installed on many various env. You have no control on the env.
Again, you definitely have no experience in shipping software outside of your bubble.
A lib is not "a web project". A cmd line tools is not either. You still need to debug them. People will run them on windows, linux, mac, bsd, and who knows where. And they will come for you.
Now you can choose to simplify the problem and only support a limited number of env. But I guess I'm quite happy the guys who created apache, ffmpeg didn't force me to only used them on Linux with LOCALE set to accept only ascii and CEST.
For example, here is a library I maintain with 12,000+ installs per month that is tested on 7 different environments every commit using Travis: https://github.com/globalcitizen/php-iban
Don't get me wrong. It's a good thing. Most people I know don't even have unit tests. Having a CI is fantastic. 7 env is more than a lot of people do.
But that doesn't even scratch the surface of the combinations of factors you can get. It can't.
You can't install all the locals, all the lib versions, simulate all network conditions and all user inputs.
You will get strange bugs that your CI didn't take in consideration. And some of them will be very hard to reproduce. That's just a fact.
Now knowing you do have all this setup I just can't understand we still disagree. It's impossible you didn't run in those.
And we should be able to look at the problem probabilistically too: given the form of the test case, the value pipeline through the debugger, and the structure of the program, we should be able to narrow down places to look as much as possible using some form of bayesian search theory.
Compared to cut down repros, the packages are huge (200+MB) and debugging requires more gumption but we save a huge amount of time going back and forth with customers. It's not ideal but we'd rather have a "bad" repro than never hear about it.
We've been taking customer issues this way for almost 2 years and it's been a good thing to have in the tool belt.
[Disclaimer - I work at Runnable]
I think probably the biggest hurdle with something like this gaining traction is having the users remember that it's available.
Is it possible to fork from an existing container? If that's possible, you can probably get maintainers to include a link to a working container with the lib already setup in the project's new issue template.