hitchstory

It's not completely meaningless but it does tend to be something people recite and then never think about - even when they're inadvertently following it. It's performative - a bit like reciting the bible.

Take I as an example - "don't depend upon interfaces which you do not use". I've seen good developers cut out dependencies hundreds of times for this reason - not because they've trained themselves on SOLID but because it just feels right after years of experience. If I said "this is part of SOLID" many of them would go "... is it?" "hmmm. I guess it is..."

As with reciting the bible, just because somebody recites it doesn't mean that they took it to heart and people who take it to heart can't necessarily recite it.

Being able to recite it is just a ritual used as a social signaling mechanism. It's a bit like leetcode, an ability to recite big O notation or knowledge of git's internals - symbols of "developerness" that don't necessarily align with skill.

It's become almost a cliche as an interview question. That's the main reason I've thought about it enough to be able to critique it.

I think it's not necessarily a bad question assuming that the candidate can recite it. If they can, they probably should have thought about what it means.

I've met plenty of devs who can't recite it but who do it all instinctively though. That's why I'd rather see people demonstrate these principles while coding without using the term at all.

But, coding interviews are expensive and you need some way of sorting the 100 applicants into the 90 who can be ignored and the 10 who are worth an interview. This is a hard problem I've never found a really good answer to. OP's question isn't worse than most of the attempts I've seen.

This software also tends to be waterfalled to hell and back. Theyve already planned out their deliverables for the next 12 months and will be held accountable for meeting them. Your bugfix doesnt fit into that picture unless an exec treats it as a priority.

...in which case it'll be jammed into the devs' task queue with no expectation that the pre-agreed delivery dates will shift.

The problem is xUnit style tests are really bad at this and make it tedious - which is why people gravitate to LLMs for writing them. LLMs on the surface look like they can relieve the pain of using bad abstractions - but theyre still a band aid on a gaping wound. 20 years ago we'd be using them to write ugly raw PHP.

Characterization tests ideally need to not be written in code but defined in something resembling a configuration language - something without loops, conditionals, methods, etc. There then needs to be a strict separation of concerns kept between these definitions and code that executes them.

I wrote a testing framework (with the same name as my username) centered around this idea. Because it is YAML based, expected textual outputs can be automatically written into the test from actual inputs which saves tons of time and you can autogenerate readable stakeholder documentation that validates behavior.

It might seem unbelievable but with decent abstractions, writing tests and TDD stops being a chore and actually starts being fun - something you won't want to delegate to an LLM.

This pattern keeps repeating because the creators of an app think they need a configuration language (which some users do) and then when eventually other users end up needing an API or some feature the config language doesnt have. They both dont exist so they hack it with templating.

It's often quite hard to know what kind of problems need an API and what kind of problems need configuration. It's sometimes only obvious in retrospect - the same point when you also get held back from changing things by backwards compatibility and culture.

The reverse problem (no configuration, only turing complete code) carries its own set of problems and can be just as infuriating. Getting the balance just right is hard and often only clear in retrospect.

Kubernetes really should have done something about the templating horrors by now though.

I took the same "docs are tests and tests are docs" approach with integration testing when I created this library: https://github.com/hitchdev/hitchstory

I realized at some point that a test and a how-to guide can and probably should actually be the same thing - not just for doctests (https://docs.python.org/3/library/doctest.html), but for every kind of test.

It's not only 2x quicker to combine writing a test with writing docs, the test part and the docs part reinforce the quality of each other:

* Tests are more easily understandable when you attach written context - the kind that is used when generating a readable how-to guide.

* How to docs are better if they come attached to a guarantee that they're tested, not out of date and not missing crucial details.

* Integration test driven development where how-to docs are created as a side effect is really nice.

I have created a project for easily writing this type of test with YAML:

https://github.com/hitchdev/hitchstory

I dont think that this type of task is really appropriate for an LLM though. It is better to use hard, prewritten abstractions for the truly deterministic stuff (e.g. click).

For other situations where you may need to do subtle trade offs (e.g. choosing an HTML selector for the search bar) an LLM will generally do a bad job.

>The other kind of test is what I call a "storybook test." Table tests are less useful here

Storybook tests lend themselves well to forked tests. E.g.

  steps:
  ...
  - click: shopping basket

  variations:
    Empty basket:
      following steps:
      - click: empty basket
      - items in basket: 0

    Purchase basket:
      following steps:
      - click: purchase
      ...

>What I mean is that a big refactor might take 9 months to do atomically whereas it could have been done in 3 months as a massive single PR.

My experience with these 3 month-long PRs is that they have a tendency to not get finished. Then they get stale. Then they get abandoned. Then somebody starts another one.

In one case it was finished, but the CTO was just too afraid to deploy it because it was big and it might break something. So while we were waiting for the right deployment window it got stale, meaning more conflict fixes....

I also think most people overestimate the amount of extra work involved in breaking up a refactoring. It's usually no more than ~20% more coding, and that extra work almost always pays for itself by reducing deployment risk.

>Does anyone use this method in practice?

I used a very similar method to upgrade Django from a very low version to a very high version where I would periodically:

1) Upgrade the dependencies.

2) Find something that was broken and fix it so it worked on both versions - usually with an ugly if statement.

3) Merge to master and deploy.

4) Repeat 2 days later when I had another spare half hour.

Because it was a pressure cooker environment, the testing was thinner on the ground than I would have liked but mostly because this was an enormous change this whole process took about a year. I actually left the company before I finished off the upgrade and I thought this work had been abandoned.

However, my coworker picked it up after I left and finished off the upgrade in the same way.

Before I joined there were several aborted attempts to do big bang upgrade on the same thing - stale branches that just died by optimistic people who repeatedly thought they could do it all in one go.

>But if you have a good test suite, then there should be no problem shipping the big change in one go!

No, there is. If the change is large enough this is still a horribly bad idea - especially if there are structural changes to data involved. You might be able to make bigger chunks than you would otherwise, but you should still drip-feed large refactorings into production because something can always go wrong. The fewer changes you make at once, the lower chance something will go wrong and the easier it is to isolate and fix the problem if it does.

If you are starting out with a good test suite (something I find almost never exists on a project I'm joining), you should be upgrading everything via small increments and very frequently - negating the need for using this strategy on most upgrades.

I dont know much about nested text im afraid.

It is on my to do list to make a spec and a language independent test suite but both of these things take time.

>by being extremely strict about indentation, both in terms of the parser & also community convention. YAML hasn't

This is why I created StrictYAML. A lot of the pain of changing YAML goes away if you strictly type it with a schema but you keep the readability.

Counterintuitively that also includes most indentation errors - it's much easier to zero in on the problem if the error was "expecting status code or content on line 334, got response", for instance.

One example was when they decided to test out the idea of sending particular kinds of price change notification emails to users. They created one checkbox on a form our users could click and set up a trigger to fire notify somebody in the Philippines who would construct the emails manually from the database. They then iterated on those emails, creating them manually the whole time.

Once the email was iterated upon and the idea was proven (i.e. that users actually wanted it in the form that existed), it was handed back to me to automate properly, which in that case took about 7-10 days of dev work.

Or not, if it turned out the customers simply weren't interested. There were plenty of experiments like that where I created a checkbox or something and then we quietly removed it 2 weeks later.

It was a refreshing change from past jobs where I'd spent weeks working on a feature asked for by users that we were so sure would be a game changer only to see 1 or 2 people actually use it.

The escape hatch is where most low code products fall down, I think. This is because of market fit as you correctly point out, but not only that.

It's also because knowing where the escape hatch and how to use it requires greater than average development skill and finesse and it isn't at all obvious when this is required.

The people who use these platforms aren't usually able to tell what kind of problem requires a developer and what kind of problem requires them doing just a bit more research. They're usually vaguely aware that the limit is out there somewhere but in the specific instances when they hit it they often don't know it's happened.

I've also seen some low code platforms get a little excitable about the idea of nearly or even fully reinventing the turing complete programming language to introduce more flexibility to their platform and make that their escape hatch. This is when things go really downhill.

I've actually just been creating step by step markdown docs with screenshots with this approach but I love your idea of embedding annotations in videos with JS popups and a proxy cursor. Assuming playwright allows for it, I will experiment with this.

My experience has been that customers are usually just happy to get docs that are guaranteed up to date even if they're not very good. They seem receptive to the idea of writing YAML that generates this stuff too.

I think I might blow some minds if I combine this with your popups and maybe some AI generated voiceovers.

The most successful company I ever worked out figured out a way to perform and get high quality feedback on low cost user experiments with a quick turnaround.

About 4/10 failed, 5/10 did ok and then with 1/10 they would absolutely hit it out of the park - usually on something that looked about as promising as the other 10. They did this frequently enough that they clobbered the competition, who were mostly just trying to do normal product management stuff like user surveys, etc.

The most interesting thing was that most people if you'd have asked them before would have gone "I don't think you can turn this into an experiment" and then immediately adopted a different strategy. Somehow they always figured out a creative way of injecting a low cost experiment where most people would have imagined that it wasnt feasible to run any experiment at all. It required quite a lot of pressure/leadership from above.

>Docs should be tested - because you may be sure that everything is right, but without real users, like with any other product, you won't know for certain. Test it and test it again.

If you want to automate this, see: https://github.com/hitchdev/hitchstory

It is not a common approach, but generating how-to docs from tests can be almost unreasonably effective.

In Germany they vary aluminium production rates in response to electricity supply (i.e. produce much less when it's less sunny and windy). They just dont turn it off completely, ever.

There's probably a lot more low hanging electricity demand shifting fruit like this.

This sounds like TDD but with the humans writing the test and the machine writing the code.

For those interested in the concept of having guaranteed/permanently up-to-date documentation (with or without screenshots) I built this testing framework based upon autogenerating good how-to or reference docs from tests:

https://github.com/hitchdev/hitchstory

(the website example has screenshots generated by playwright, the REST API example has real tested JSON snippets, the command line example has command outputs, etc.)

>The reasons you would choose to include a 'before-all' method will vary from case to case.

Not really. I would always purge anything that would cause tests to share state. I wouldn't do it on a case by case basis.

>What is it about your application code that would make you think that addUser is flaky enough to need a clean slate to run against?

The user already existing in the database? The behavior of the app would change in that case. Something has to wipe the db clean to test that scenario.

Thats why tests shouldn't share databases.

>... constructing a particular set of circumstances which will never occur in the real-world system.

What do you find unlikely about a scenario where a test uses an app in a realistic way (e.g. with a browser) set up in a realistic context (e.g. with some fixed sample data) to reproduce a realistic scenario (e.g. a bug that already happened)?

I wouldn't say that isolation and realism are completely orthogonal but I find that well engineered integration tests are usually able to reproduce 90% of bugs sourced from production while unit tests can often manage only 10 or 15%. Bug in the SQL? Browser is involved? No can do.

Nice and clean in the context of an integration doesn't mean no state it just means no state outside of the context of that test.

If I set up an integration test that sets up a database from scratch and tears it down and tests only the behavior of the app in that rigidly defined context, then yes, it will be useful. It will tell you how the code behaves in the scenario you've created.

Bad integration tests will share state with each other - e.g. by using the staging DB.

Unit tests have lower up front costs but higher ongoing costs. By their very nature they couple more to implementation details, so they will not give you clear confirmation when code is broken.

Integration tests can give unclear signals when they are flaky, but when they are engineered well they will give a much clearer signal that things work when they pass and that something is broken when they fail.

It's harder to engineer a good integration test - this includes making them isolated and independent e.g. of test ordering or indeed, anything else.

It's readable, it just desperately needs type safety: https://github.com/crdoconnor/strictyaml

When I read "gbq enabled: yes" I don't worry about whether it's a string or a boolean when reading it, I worry about whether it's a string or a boolean when I want to change it and it gets parsed again.

I don't like gherkin. I think that it has awkward syntax, it lacks type safety, it's very verbose and doesn't have enough ability to abstract scenarios. Rather than being a source for generating the documentation it tries to be the documentation.

Nonetheless, there is a small number of projects where they either work around these constraints or it doesn't matter as much, so it can work. I find that most people that apply gherkin to their projects run into a brick wall though - usually for one of the above reasons.

I built https://github.com/hitchdev/hitchstory as an alternative that has straightforward syntax (YAML), very strict type safety (StrictYAML), low verbosity and the ability to abstract scenarios.

It is explicitly designed as a source for generating markdown documentation so that you can create richer docs like https://github.com/hitchdev/hitchstory/blob/master/examples/... for your users rather than more test-like documents like https://github.com/diaspora/diaspora/blob/develop/features/m...

I started doing documentation driven development a while back (with FOSS code) I wrote and I realized at some point that my tests and my how to docs were doing the same thing - one was just an easier to read version of the other.

Moreover, when one went out of date so did the other.

So, I tied them together - I wrote integration tests such that with a jinja2 template I could autogenerate readable markdown how-to docs from the tests and their artefacts.

Setting that up led me to the realization that writing docs isn't actually tedious at all. It's the writing things down 3 times (spec, test & docs) and keeping them in sync that is tedious. And that part can be automated.

If there is a framework to build the whole bundle of docs from integration tests, integration test metadata, integration test artefacts, schemas, code and an assortment of high level explanatory texts then it stops being a chore. It actually becomes fun.

It means that DDD becomes integration TDD. The only difference is that I will write an additional 3-4 explanatory sentences alongside the integration test I write before the feature and run a "docgen" script afterwards.

JSON is also supposed to be human readable in a pinch.

JSON seems more like a more appropriate format for this since it's an interchange format.