Chasing a Performance Regression with Erlang/OTP 22 | Hacker News Reader

Chasing a Performance Regression with Erlang/OTP 22

tech.nextroll.com

71 pointstodsacerdoti6 comments

The rule we (rather mature engineer organization) we try to follow is slightly more complex than the one in the post.

We run on second to the last version, but we keep testing on the latest and even beta release. So to catch problems immediately and if possible give our feedback to the developers of the platform.

Yes, this is the downside of their "use last version" approach - if the version has a regression you can't do anything about it. If you're using the current version then maybe you can have your problem treated as a bug and fixed in a maintenance release.

Your tests also need to execute on the same systems that will run your system.

And you need to test on a version back, your version and a version forward. When you move versions, you need to know exactly why and when you lose support for an older platform.

I don't think this changes the work involved that much. Either way, you have to investigate the regression and see what changed and why that causes an issue in your environment. It's a little easier to see what change is at issue if you're testing all changes as they come in of course.

They got support from upstream, and found out that they also needed to upgrade their environment to support the intended madvise semantics. Maybe if they were tracking master, they could have convinced upstream not to call madvise unless proper semantics were available; maybe not.

Sometimes you get lucky and other people find the regressions and the maintenance releases are missing bugs the .0 release had. It's of course nice to check if upstream still has the issue and let them know if so.

At the end of the day, if you need code for your system, you need to be able to maintain it. And this post shows they can.

It definitely helps.

It de-risk the changes. If you are tracking beta, latest or pre-release you know immediately what is ahead of you.

You will know ahead of time, that you will need to investigate this performance regression before to migrate.

In a big organizations where resources and time are limited, knowing a bit of what is ahead of you is invaluable.

This article was really great, and this gem stood out to me:

> When the hypothesis failed, we again rushed to move on with another work. We didn’t take the time to analyze why our theory was wrong. We didn’t question the experiment’s validity, which would have led us to realize our patch was wrong early on.

I've read dozens of postmortems where "someone did something to 'prove' that X wasn't the cause, but after ruling everything else out the team eventually realised that X was in fact the cause.", with 'X' being everything from "this patch can't have caused that behavior" to "I know it's not the kernel update because how could that have changed anything?"

In the face of a crisis, if you are in charge you really have to take all the "facts" you are presented with as being contingent on the level of panic that the presenter is feeling -- being a great engineer is more or less orthogonal to calm under pressure.

Reply on news.ycombinator.com