hitchstory

>as though framing problems "in a way coding agents can solve" is itself a well-understood or bounded problem.

It's not, but if you can A) make it cheap to try out different types of framings - not all of them have to work and B) automate everything else then the labor intensity of programming decreases drastically.

>At commercial quality, is it an accessible skill for inexperienced people

I'd expect the opposite, it would be an extremely inaccessible skill requiring high skill and high pay. But, if 2 people can deliver as much as 15 people at a higher quality and they're paid triple, it's still way cheaper overall.

I would still expect somebody following this development pattern to routinely discover a problem the LLM can't deal with and have to dive under the hood to fix it - digging down below multiple levels of abstraction. This would be Hard with a capital H.

The pyramid makes sense for certain types of application - very logic heavy with light integration touch points with everything cleanly dependency injected. rqlite fits this pattern.

If you're building a logic lite application which has a lot of integration touch points (I find most commercial code actually fits this pattern) then it makes zero sense - an integration test heavy test suite is what you want - an upside down pyramid if you will.

If you have a ball of mud on your hands then it doesnt matter what kind of app it is, E2E tests are the only thing that makes sense and you need to build very sophisticated fakes (requiring a lot of engineering skill) to avoid things like flakiness bugs and the token scraping problem you referred to.

If you're writing a parser, you probably want 100% unit tests and property tests for all those millions of parser edge cases. No pyramid of any shape is required, just a slab.

The testing pyramid reminds me of microservices: the people who came up with Their Grand Idea had no clue what it was about their specific circumstances that made their approach work for them but still managed to market it as The Way.

Realistically, the ultimate shape of your test suite should be an emergent property based upon a series of smart, local decisions - ideally TDD'ed tests which match the kind of code you are writing right now. Making a particular shape a goal is asinine.

That sounds like a reasonable average, yeah. Some tiny changes take 45 seconds, some harder changes take 40 minutes.

I find that working in working code increments that are as small as possible to be ideal.

People leaving is a great reason to write literate tests and other kinds of docs - the kind people actually look for and want to read.

If somebody asks a question that can only be asked by looking in a commit message that usually represents a failure in one of those docs.

I do squash them where appropriate.

I do use git stash where appropriate.

These arent nonobvious novelties, save perhaps to junior engineers.

I mostly just message the person who wrote it if I have questions when I look at a commit and have questions.

I suppose if I painstakingly write 1000 beautiful commit messages I could save myself from having that one conversation when somebody else has a question about one of those commits.

>can be good self-promotion. You may not realize

When I said nebulous and hard to pin down this is kinda what I meant.

Do you routinely monitor commit messages on adjacent teams' repos? I dont. I dont know anybody who does. When I want to know something about their code I ping them a slack message and vice versa.

I dont think the CTO is reading commit messages either, theyre too busy.

I think it's good to ask when and why people actually do read commit messages to make sure what you write aligns with what they want to see.

I'd estimate I probably commit ~30-50 times a day. I dig into and read a commit message once every 3 months.

A 90 seconds a commit that's an hour a day spent writing beautiful commits.

That one hour a day could be substituted with one conversation "hey, why did you do xyz in [ linktocommit ]?" every 3 months.

If you dont do these back of the envelope calculations in your head when trying to figure out if something is worth doing I highly encourage it.

>Already it’s pulling teeth just to get literate commit messages…

I usually push back on this because those commit messages almost never actually get read. It's an investment whose dividends are nebulous and hard to pin down.

It's rare that I look at a commit, and it's even rarer that I read it and wish that it had a better message.

There are all sorts of other documentation I routinely wish people put more effort into writing (comments, a "why" attached to every test, how to guides, tutorials), but rarely ever a commit message.

A drop in remote worker will still require their work to be checked and their access to the systems they need to do their work secured in case they are a bad actor.

My issue with the formal methods community is that it always tended to hand wave away the cost/benefit trade offs. It's not that they didnt see those trade offs (they constantly reiterated that they weren't suitable for every problem) - more that they saw it as someone else's problem to solve to decide which problem, when and why.

Im sure if somebody wrote a blog post that provided a sensemaking framework for WHEN formal methods are appropriate vs waste your time it would be popular.

I see hints of this all over the thread - e.g. some people seem to believe it works well for state machines (though I suspect they were nontrivial ones), but as a topic it rarely ever seems to be addressed head on. OP certainly doesnt.

I suspect this trope will reiterate again next year with "formal methods! useful! sometimes!". The community has been like this for decades.

>if it was really that helpful, more people would be applying it and the benefits would speak for themselves. They've been around a long, long time, though, and never caught on - it's hard not to conclude that there's probably a reason.

My impression is that there are actually not that many business domains where a large investment in time and money to get domain logic correctness from 98% to 99.99% correct is actually called for.

Formal methods are a large investment, too. No two ways around it.

Also, while they havent really caught on in general, some of their ideas have made it into modern type systems.

That depends on what the drawings are of.

>Engineers dabbling in documentation and feeling lost as they approach the field, on the other hand, are drawn to docs frameworks because frameworks are all they’re used to working with

Im drawn to frameworks because without them I cannot implemented an automated way to address the problem of docs going out of date.

I dont either, but this can be mitigated by adding guard rails (strictly validating input), double checking actions with the user and using it for tasks where a mistake isnt world ending.

Even then mistakes can slip through, but it could still be more reliable than a visual UI.

There are lots of horrible web UIs i would LOVE to replace with a conversational LLM agent. No #1 is jira and so is no #2 and #3.

Snapshot tests are even more sensitive to flakiness than E2E tests but flakiness is still a bug that can (and probably should) be fixed like any other.

As an example - the different browser you have running locally and in CI? That's a bug. The fix is to run the browser in a container - the same container in both environments. You can probably get away without this with an E2E test. Snapshot test? No way.

Ive seen people decry and abandon E2E tests for the same reason - simply because they couldnt get flakiness under control and thought that it was impossible to do so.

I dont think it's intrinsically impossible though. I think it's just a hard engineering problem. There are lots of nonobvious points of flakiness - e.g. unpinned versions, selects without order bys, nondeterministic JavaScript code, etc. but few that are simply IMpossible to fix.

Theres a big payoff if you get it right, too.

>the algo-heavy code is a small fraction of the total code they write

Wasnt the case in the trading firm I worked at.

Do you work at Jane Street? Have you worked elsewhere?

>in my opinion, because users don't care about "custom widgets," "queues," or "databases."

Users care quite a lot when these things break.

>They care about business functionality, which is quite easily expressible in model form.

They care about reliable applications that do the task they want. Users don't have a concept of "business functionality". If a widget they wanted to use is broken, they'll be pissed - they won't tell you that it's ok because the concept of the application was still good.

From your link:

>The main thing that I want to highlight here is how simple the model is. This is pure domain logic, and is probably as close to the essential complexity of the application that we can get

This is exactly the issue.

If I go down my recent list of bugs and categorize them, ~8 out of 10 of them are in areas which the author would have *abstracted away*. The other two probably could have been avoided by having a team member write one test to cover that functionality instead of zero.

What does it mean when you abstract something away from a test? Well, it means the test won't catch a bug in that thing.

If I went down my recent list of bugs and been able to see that 9 out of 10 of them would have been caught via adding property testing/model testing on top of regular testing/typing, etc. I'd have absolutely no second thoughts about implementing it but that simply isn't the case.

I have worked on problems in (not usual) environments where it is the case, and in those rare cases I have advocated for and used property testing but I have more often seen it advocated and used by people who see every bug as a nail and property testing as some kind of super advanced hammer.

>Why is PBT useful at Jane Street, at least more than in other places?

Because trading firms write a lot more algorithmic code than most businesses. Trading strategy code is intensely algorithmic and calculation heavy by its very nature as is a lot of the support code written around it.

At least, that's what it was like when I worked in a trading firm. Relatedly, it was one of the few projects Id worked on where having 95% unit tests and 5% integration tests made perfect sense. It fitted the nature of the code, which wasnt typical of most businesses.

Somebody else wrote that they wrote a lot of numerical code in another business for which property testing is extremely useful and again, I dont doubt that either. 95% is still != 100% though.

Trading companies are unusual in writing a lot of algo-heavy code. Did you assume every company was like this?

I can assure you they arent.

>Hah! Try to separate your domain logic from your interfaces

Im not an amateur.

The only time I dont do this is when there literally is no domain logic yet (e.g. a CRUD app).

>In my experience, these gaps are what's typically expensive, not the weird problem a junior had with properly using Redis or S3.

What can I say? Your experience might not be as broad as mine.

Redis is a source of almost no bugs because it is very well designed, but most interfaces I couple to have design qualities that are the exact opposite of redis's.

Those interfaces (e.g. wonky payment gateway APIs, weird microservice APIs) are the probably source of most bugs in enterprise systems I work on.

#2 is probably simple misspecifications (customer said code should do X, it should actually do Y which is almost the same but very slightly different).

#3 would be domain logic errors. And even most of those are uncovered and avoided with saner architecture or a couple of unit tests.

For the parsers I write at home, sure, property testing kicks ass. For your college degree algorithm coursework, sure, it helps a lot. For 95% of business logic? Pointless, the complexity isnt buried deep in the business logic.

If the arbitrary input is, say, a text box which takes a name and puts it into a database, it probably won't uncover any bugs.

It has some use if you build something like a complex pricing engine, numerical code or a parser for a mini DSL. I find that problems of this type don't crop up a lot though.

Property testing is a lot like formal methods - really cool, but almost entirely useless in ~95% of commercial contexts.

They're both extremely useful when, say, building a parser, but when the kind of code you write involves displaying custom widgets, taking data and pushing it onto a queue, looking up data in a database, etc. integration tests have a lot more bang for the buck.

I use it for:

* Reminders

* Checklists

* Idea collation

* Fleshing out and prioritizing personal tasks

* Reference notes - e.g. passport number, tax numbers, frequent flier numbers, anything else I might want to input into a form one day.

* Calendar

* As a place to draft emails/letters/documents.

There's so much crossover between all of these things that it helps to keep them all together.

It often protects me from things I miss. E.g. when I'm TDD'in a feature and I'm implementing the 5th-6th test I often find myself inadvertently breaking one of the earlier tests.

I do think there is such a thing as overtesting - i.e. regretted tests. TDD actually protects you from this to an extent by tying each test / modification of a test to a change in code.

Runtime assertions definitely give you more bang for the buck (they are ridiculously cheap) but they are complementary to tests, not a replacement. It attacks the same problem from the bottom up instead of top down.

I also find that when you combine the two, the tests become more useful - previously passing tests will fail when they trigger those assertions.

There isnt much consensus on the right kind of test to write with TDD, but when you get it right or wrong it makes or breaks TDD.

Recently Ive been writing mostly "end-to-end unit tests" - stateless, faking all external services (database, message queue, etc.) with TDD which works great.

There is a sweet spot on default test types - at a high a level as possible while being hermetic seems to be ideal.

The other un-talked about thing is that to be able to always write this kind of test you need test infrastructure which isnt cheap to build (all those fakes).

>Unit tests tend to have a lot of boilerplate sometimes, often not copy-pastable

When people use LLMs to write code and they find if helpful, invariably it is because they are spewing boilerplate.

If you dont systematically eliminate boilerplate the codebase eventually turns into an unmaintainable mess.

>Sometimes you do actually want to add a ton of unit tests even if they’re a little too implementation-focused.

Really? Id consider this an antipattern.

>I find it better to treat writing tests with LLMs tactically

I find the prospect of using them to write production code / tests pretty depressing.

The best thing that can be said is that they will create lots of jobs with the mess they make.

Using LLMs to generate tests is bizarre. If anything, humans should write the tests and the LLMs should make it pass.

Tests lose value the further away from the spec they drift. Writing a test after the code causes drift. Writing with an LLM causes even more drift.

It ends up being a ritual to appease the unit testing gods rather than an investment that is supposed to pay dividends.

>If you’ve ever heard someone claim that Test-Driven Development leads to zero-defects code

I havent. It's a lot more reliable, still not 100% reliable.

>If you can define the inputs, predict the outputs, and cover edge cases, TDD is like a superpower. But the real world is rarely that tidy.

shrug it ought to be that tidy if you specify your features before building them.

For writing code to test a sin function (a bit of a contrived example, but let's run with it) - well, that would be an excellent candidate for property test driven development because there are straightforward invariants you can define.

When edge cases crop up for other styles of code, well, talk about them with your PM and write test cases for them too.

If the spec is something like a UI which is visual and specified imprecisely with a sketch or something, snapshot test driven development more appropriate but that's still just a variant of TDD.

How do you do that?

Im curious because I wrote a framework for this (same name as my username) but when I talk about the idea I receive a sea of blank faces.