hitchstory

TDD != starting at unit level. It's about red-green-refactor.

Or, it should be anyway. People talk about how it "drives" design but I dont think it should. I think tests should be as design-agnostic as possible, providing freedom to redesign.

I always start with an end to end test. These dont force you to commit to anything.

This is why on greenfield projects I start with an end to end test and do red-green-refactor with it. This lets me defer many architectural decisions (those decisions are usually of a better quality when done during the refactor step) while locking in behavior up front.

Call it what you like. Some people would be insistent that this is TDD. Some would say it isnt because it's not test driven design. Some say it is TFD. I dont really care what it's called, for me it doesnt make sense to develop serious non-spike/non-POC code any other way.

It's one of the few things in tech where my opinion has hardened and not softened over the years.

"Unit tests" on the other hand, blech, sometimes useful, very overused.

>There are areas at the edges where the mocking/stubbing required to really follow TDD cause make changes harder but never find bugs

That's not TDD that's either a badly designed unit test or (more likely) you should have written an integration/e2e test instead.

>There are entire families of bugs that are far better handled via strong types than by building tests.

Tests and types are not solving the same problem and they work better in conjunction than alone. Types reduce the execution space, reducing what can go wrong and tests validate the execution space that does exist.

Yes, types reduce the need for tests by shrinking the execution space but this will be for scenarios which should never happen.

There are entire families of code and bugs that unit tests are unsuitable for but that's a whole different topic.

> but hard red-green-refactor mandates tests with negative value.

I have never seen an example of this. I've seen plenty of tests with negative value but they were simply bad tests - usually a unit test when an integration test was more appropriate.

I wrote a systems/integration testing framework for this purpose specifically and I wrote a few essays alongside it. It has the same name as my handle.

Not much traction, unfortunately. Id be interested in any comments you might have.

It's called TDD not UTDD.

I frequently do TDD with integration and end to end tests. The type of test should be dictated by the nature of the code, which TDD doesnt have an opinion on.

TDD is about following red-green-refactor.

If you do test-after you have to keep a mental tally of new scenarios to test when making code changes. This makes it more unreliable since it's easy to forget one of those scenarios or mix it up with an already tested scenario.

TDD lets you safely forget by tying the test pass/failure directly to the code.

So, yes, you can do test-after but why?

The only reason Ive ever heard for doing it after anyway is "I just prefer it that way".

Not writing tests at all makes sense (e.g. for a spike), but if I were going to start writing tests at any point I cant see any reason not to do it with TDD.

If it remains perfectly bug free until the end of time then yes it's fine /s

Tests are usually soul crushing and boring for the same reason PHP was in 1999 - a complete lack of structure, tooling and separation of concerns made it tedious and difficult.

I find it interesting that when people describe to me how they use LLMs to write code it's either short throwaway scripts or to write the kind of code that would make me retch (e.g. tests stuffed full of horrible mocks, spaghetti boilerplate).

This doesnt make a lot of sense. The set of things your software cant do is infinite. The set of potential errors is massive and many if not most are unrelated to domain logic.

That's just one example of it.

Other examples:

* Replacing automated tests with (quicker) type checking and running it on a git commit hook instead of CI.

* Replacing slower tests with faster tests.

* Running tests before merging a PR instead of after.

* Replacing a suite of manual tests with automation tests.

etc.

* investment (instead of dogma) driven approach to testing

* hermetic end to end tests - tests that test your code in a hermetically sealed environment using fakes.

* automatic generation of documentation from story driven tests

>I’ve been through this documentation story before. I can predict how it ends. By the time you read the documentation it’s out of sync with the code so you end up reading the code anyway. But then you feel bad, so you write more documentation & force yourself to update documentation whenever the code changes. But that takes time away from development & the pressure is on, so you stop writing/updating.

The story doesn't always end this way.

There are two types of documentation:

* Those that don't really go out of sync when you update the app - e.g. intros, overviews, contribution guidelines, justifications, etc.

* Those that can go out of date when you change the code - API references, how to guides, tutorials.

If the latter is autogenerated from tests + test metadata (how-to guides/tutorials) or from code (API references), it won't go out of date.

I did something similar but with my phone, the android orgzly app, termux, jinja2 and latex.

I cribbed a CV template from overleaf and put some jinja2 in it to take the content from CV note in my note taking app: https://hitchdev.com/orji/using/latex-cv/

With termux I can then hit a button to run a script and it will instantly generate a pretty PDF using latex from an orgzly note and fire off an android share intent.

The nice thing about this set up was that if a recruiter called me while I was out and wanted a CV quickly with a couple of tweaks made I could just do it on my note taking app and email an updated PDF in a few seconds.

In theory I could easily change the style of the CV but in practice I haven't felt the need to touch it in years.

I used to be really good at working with big balls of mud.

I actually even started enjoying it at some point - it felt like a mixture between a crossword puzzle and archaeology. There was also something nice about being good at something other people hated that was nonetheless very critical.

Somewhat ironically though, the better I got at it, the class of job I got the better the quality of the code I ended up working with.

Now my mud wrestling skills have atrophied a fair bit.

The skeleton test-generating-docs example I built to exhibit the framework actually does this. Here's an example:

https://github.com/hitchdev/hitchstory/blob/master/examples/...

It records a video while running the test and then at the end runs it through FFMPEG to make a smaller, slowed down GIF that can be embedded in the autogenerated docs.

It's quite rudimentary though. I've been meaning to try making something more sophisticated and even potentially do additional automated video editing to inject text from the steps or something.

That's very cool.

I think screenshotting needs to be integrated into the tests though - if a scenario involves a wizard or something, the latter screenshots will be dependent upon the actions in earlier steps.

It works for user facing documentation too. There are actually pretty good reasons for this - e.g. you can use the test to autogenerate up to date screenshots with playwright to put in the documentation.

I'm pretty convinced that there should be a single source of truth for specifications, tests and documentation but I think the industry will take a while to catch up to this idea.

I built a testing library centered around this (same as my username) but it's hard to get people to stop writing unit tests :)

I created a command line app for applying jinja2 to orgmode. I did it to scratch the itch of being able to generate pretty latex PDFs for my CV and for printed letters with content from my note taking app on my phone.

I published it on pypi/github out of habit, but it's primarily about scratching my itch and I doubt anybody except me will really use it. There's some interest in doing this type of thing with org mode but it all centers around LISP and emacs (neither of which I like).

I built a Python/YAML framework around the ability to build these tests and then generate documentation from them. E.g.

https://github.com/hitchdev/hitchstory/blob/master/examples%...

which generates

https://github.com/hitchdev/hitchstory/blob/master/examples%...

When you show this stuff to other people and gather feedback - that's essentially what BDD is, too.

Screenshots was one thing that didnt work that I needed but I think lots of control characters used by command line apps also messed up pexpect.

I built my own probably not very good equivalent of your thing called icommandlib. I'm going to investigate ripping it out and replacing it with your tool.

I wrote an integration testing framework which I wanted to integrate with a tool exactly like this so it could be used to, e.g. test a command line app like vim.

Expect is what I tried to integrate with first. It falls over quite quickly with any kind of app that does anything mildly complicated with the terminal.

I built a testing and documentation framework to solve precisely this problem (google for my username to find it).

You write the StrictYAML based tests structured according to your domain language/interactions and then you write one class that interprets them to execute the test and one template file that you apply to generate the markdown/HTML/whatever docs (with test artifacts like screenshots if you want them).

How to documentation then becomes a build artefact that, since it is generated from a test, never goes out of date by definition.

I usually combine this kind of docs with other types (e.g. explanatory) which can't be generated but also dont tend to go out of date.

The "busy work" of low level unit tests is perfectly valid, but TDD still works extremely well with high level integration tests so I don't think a similar criticism necessarily applies.

I built something like this too:

https://github.com/hitchdev/hitchstory/blob/master/examples%...

Thanks!

Thanks, that's very helpful.

1 and 2 make perfect sense and are easy do demonstrate but 3 seems to me to be incredibly difficult.

I haven't found an easy way to advertise convincingly to somebody who (quite reasonably) grants you a limited amount of attention that custom things won't be a nightmare. It's the kind of thing you only tend see when you get dug in the weeds and hence people will tend to make assumptions based upon surface details.

This is a problem I'm struggling with.

I think robot/cucumber could require less code if they were better abstractions (and would be more loved), but I find it hard to illustrate that an abstraction is going to be good or bad, particularly to people with limited attention and particularly to people who don't necessarily have the skills to recognize a good abstraction.

BTW, odd question - assume somebody built a low/less code test automation framework that didn't fail and was very effective but it also wasn't popular.

What could it do to stand out to you? What could it demonstrate in 15-20 seconds for you to think "OK this is different"?

Not just how quickly it can be changed but how risky it is to change.

If they include detailed plans and timelines for feature deliveries then yes.

>Theyve already planned out their deliverables for the next 12 months and will be held accountable for meeting them.