Hacker News Reader

what are the kinds of tests do people write in other areas of the industry?

Aerospace here. Roughly this would be typical:

- comprehensive requirements on the software behavior, with tests to verify those requirements. Tests are automated as much as possible (e.g., scripts rather than manual testing)

- tests are generally run first in a test suite in a completely virtual software environment

- structural coverage analysis (depending on level of criticality) to show that all code in the subsystem was executed by the testing (or adequately explain why the testing can't hit that code)

- then once that passes, run the same tests in a hardware lab environment, testing the software as it runs on the the actual physical component that will be installed on the plane

- then test that actually on a plane, through a series of flight tests. (The flight testing would likely not be as entirely comprehensive as the previous steps)

A full round of testing is very time-consuming and expensive, and as much as possible should be caught and fixed in the virtual software tests before it even gets to the hardware lab, much less to the plane.

I'll bite: How is your job improving or affected by this AI surge?

So far, very little.

Per corporate policy -- in the name of safety and legal reasons -- the most extensive public uses of AI, like vibe coding and agentic systems, aren't really options. The most common usage I have seen is more like consulting AI as a fancy StackOverflow.

Will this change? I personally don't expect to ever see pure vibe coding, with code unseen and unreviewed, but I imagine AI coding uses will expand.

The interesting bit is this: how much of what you wrote over all those years actually did what you wanted it to do, no more, no less?

This is where testing gets interesting: I took some old code I wrote 30 years ago or so and decided to put it literally to the test. A couple of hundred lines from a library that has been in production without every showing a single bug over all that time. And yet: I'm almost ashamed at how many subtle little bugs I found. Things you'd most likely never see in practice, but still, they were there. And then I put a couple of those bugs together and suddenly realized that that particular chain must have happened in practice in some program built on top of this. And sure enough: fixing the bugs made the application built on top of this more robust.

After a couple of weeks of this I became convinced: testing is not optional, even for stuff that works. Ever since I've done my best to stop assuming that what I'm writing actually does what I want it to. It usually does, for the happy path. But there are so many other paths that with code of any complexity, even if you religiously avoid side effects you can still end up with issues that you overlook.

I think you have the correct approach. Tests are signals, not proofs.

I do often hear on HN people advocating for Test Driven Development (TDD) but that I think is a different category of error. It encourages people to write to the tests, not to the spec. They'll claim the test is the spec but not understand that the spec can't be written down in full. The spec is the intent, the written spec is an approximation of that, the tests are an approximation of that.

The problem with TDD is it makes it easy to believe that by passing the tests your code is complete and bug free. It's easy to test tests as proofs.

But just because tests aren't proofs doesn't mean they're not useful. They narrow the space in which errors exist. Tests also are a form of communication. It can tell others about your assumptions, again, making the search space for debugging smaller.

Tests are useful, but no one can write tests that are complete.

The value of tests is when the fail they show you of something you broke that you didn't realize. 80% (likely more, but I don't know how to measure) of the tests I write could safely be thrown away because they fail again - but I don't know which tests will fail and thus inform me that I broke things.

The system I'm working on has been in production for 12 years - we have added a lot of new features over those years. Many of those needed us to hook into existing code, tests help us know that we didn't break something that used to work.

Maybe that helps answer the question of why they are important to me. They might not be to your problems.

How do you know you haven't unknowingly broken something when you made a change?

I think if: - the code base implements many code paths depending on options and user inputs and options such that a fix for code path A may break code path B - it takes a great deal of time to run in production such that issues may only be caught weeks or months down the line when it becomes difficult to pinpoint their cause (not all software is real-time or web) - any given developer does not have it all in their head such that they can anticipate issues codebase wide

then it becomes useful to have (automated) testing that checks a change in function A didn't break functionality in function B that relies on A in some way(s), that are just thorough enough that they catch edge cases, but don't take prod levels of resources to run.

Now I agree some things might not need testing beyond implementation. Things that don't depend on other program behavior, or that check their inputs thoroughly, and are never touched again once merged, don't really justify keeping unit tests around. But I'm not sure these are ever guarantees (especially the never touched again).

I think the whole concept of testing confuses a lot of people. I know I was (and still sometimes am) confused about the various "best practices" and opinions around testing. As as well as how/what/when to test.

For my projects, I mainly want to Get Shit Done. So I write tests for the major functional areas of the business logic, mainly because I want to know ASAP when I accidentally break something important. When a bug is found that a test didn't catch, that's usually an indicator that I forgot a test, or need to beef up that area of functional testing.

I do not bother with TDD, or tests that would only catch cosmetic issues, and I avoid writing tests that only actually test some major dependency (like an ORM).

If the organization you are in does not value testing, you are probably not going to change their mind. But if you have the freedom to write worthwhile tests for your contributions to the code, doing so will probably make you a better developer.

Yes

I worked for a company that had no tests.

I worked on the core software, new employee, the programmer who wrote the software gone...

Regularly released new features and found out, some days later, that I'd broken some peripheral, but important, business logic.

Drove me mad! I was not allowed to write tests, it was "unproductive"

Tests are important. Writing good tests even moreso. However, you can't do either of you don't have good product requirements and communication.

Most software has "bugs" simply because people couldn't communicate how it should work.

I think most programmers are on top of actual runtime errors or compilation errors. It's straightforward on how to fix those. They are not on top of logic issues or unintended behavior because they aren't the product designer.

Programmers just cook the burger. If you order it well done, don't complain when it doesn't come out medium rare.

Follow-up questions: Do you test manually? Why? Do you debug manually? Why?

You wanted examples: https://github.com/openjdk/jdk/tree/master/test/jdk/java/uti...

I do test manually in salesforce. Mainly its because you do not control everything and I find the best test is to log in as the user and go through the screens as they do. I built up some selenium scripts to do testing.

In old days, for the kinds of things I had to work on, I would test manually. Usually it is a piece of code that acts as glue to transform multiple data sources in different formats into a database to be used by another piece of code.

Or a aws lambda that had to ingest a json and make a determination about what to do, send an email, change a flag, that sort of thing.

Not saying mock testing is bad. Just seems like overkill for the kinds of things I worked on.