Hacker News Reader

Go sql/database uses its own connection pool. But still that shouldn't create any problems. I have seen the reverse where apps that assumed temporary tables stick around from statement to statement without an explict `txn` (which regular postgres connections don't need) clearly failed. But I have not seen the issue you talk about.

My wild guess would be that the Go code never closed the result/rows which caused either a pool exhaustion / left the connections hanging and eventually got a timeout. Consider this similar to how http client's need to close the response.Body or else connections can't be reused.

At a previous employer, I was forced to use a certain 3rd party ODBC library. Under certain circumstances, it would just do `exit(1)`, with nothing even to stderr. Very frustrating and annoying to debug/fix. Had I physically known the developer responsible for that behavior, I'd probably have faced murder charges and plead temporary insanity.

"Always code as if the person who ends up maintaining your code is a violent psychopath who knows where you live.

I usually maintain my own code, so the as-if is true. "

http://wiki.c2.com/?CodeForTheMaintainer

Lol

The database/sql package isn’t very magical. The comment about magical library makes me think it was some other package.

Any sufficiently advanced technology is indistinguishable from magic. If your source of knowledge is stack overflow or a youtube video posted on linkedin you already live in a world filled with magic.

They may have been using one of the PG specific Go libraries instead too. eg https://github.com/jackc/pgx

>> That ended up fixing the problem

Interesting take, since you're still stuck with "simple reports" that take hours to run. What about writing more efficient queries, adding indexes, normalizing the DB... ?

Wouldn't be the first developer to just SELECT * with a thousand joins and then sort data in app(!).

Or even worst/better, thousand of SELECT with joins returning one result and then sort in app.

This sounds like the Hibernate Execution plan I found one time when it wasn't configured properly.

It amazes me the complexity of solutions these days, when 20 years ago almost everything ran in relational databases and query tuning was usually the solution to performance problems.

Because we got rid of DBAs in favor of “big data” developers that never learned much about SQL in the first place.

Because 64gb of ram is really cheap these days.

It no longer makes sense to tune your queries, or to wait weeks\months for the vendor to tune their queries, when you can just slap a few sticks in and call it a day.

Tuning queries still makes plenty of sense. Slow/expensive queries need extra infrastructure like a memory cache server or more app servers to paper over the inefficiency.

That's a bunch of added effort that might as well be spent on understanding your database. Even something simple like using materialized views can significantly increase performance of expensive queries.

If you're like a company I was at before, you'd pay $10k+ for DB consultants to tune some queries and your prod database, and when migrating your DB to new hardware forget to re-tune it and waste the extra 64GB and even SSDs installed. There should still be a bare minimum floor of competence for actually developing against and maintaining databases organizationally whether it's a DBA, better engineers, etc. Throwing hardware at a problem is fine when you're sure that you are actually throwing it in the first place which I have seen surprisingly few places do well.

But we aren't doing something simple like that, we are building monstrosities based around the theory of micro-services in the cloud. Kubernetes. It takes hours to get a development environment put together to try and reproduce / debug a problem. We are adding complexity instead of keeping things simple.

And query tuning isn't that difficult. Spend a few hours on this site and you will be better than 90% of devs out there: https://use-the-index-luke.com/

> we are building monstrosities based around the theory of micro-services in the cloud. Kubernetes

This isn't something I'm doing and none of the people I personally know are doing this (disclaimer: im not in SV or the "startup scene")

> And query tuning isn't that difficult. Spend a few hours on this site and you will be better than 90% of devs out there: https://use-the-index-luke.com/

It used to be that companies had DBAs and could call out their developers\vendors on their sloppy queries. They have been replaced by 64gb of ram

The technology inept organizationally can’t even figure out how to use that 64GB of RAM properly is my point and more RAM won’t fix a full blown table scan on every other query coming in because people don’t have a way to stop queries without limits. However, you can add modern SSDs with millions of IOPS to paper over complete ineptitude at using a free query analyzer in your database. But to myself, inability to bring oneself to use that is about as incompetent as not being able to use a debugger or profiler. I can certainly understand a level of inability when hiring only junior devs or having one’s hands tied when dealing with production systems locked behind compliance and bureaucracy, but all these things are problems that hardware nor more people will fix honestly.

And TBF, I am mostly familiar with the guts of technology laggards and they’re adopting K8S at frighteningly fast rates compared to anything else I’ve seen in almost 2 decades. Very common to see pretty smart people familiar with the latest tech yet lacking some serious holes in fundamentals familiar to those with experience.

So, FWIW. The database I mentioned in my original post had 2TB of RAM. Food for thought!

> What about writing more efficient queries, adding indexes, normalizing the DB... ?

Adding indexes seems like the best effort/reward benefits of those.

In any case, I can easily imagine the following workflow:

1. Kick off report after first logging in in the morning

2. Get on with your day

3. Check report after first coffee break

I've heard about some database drivers/middleware being smart and optimizing/caching queries like "SELECT 1" to save the round trip, so now I do something like "SELECT now()" or similar to do health checks.

Go code often reinvents/reimplements a lot of things from scratch, reintroducing problems that have been addressed long ago in other systems.

It's like this new trend, let's rewrite everything in Go to be cool. Financially makes little to no sense.

Let’s not just target Go with that sentiment, it applies almost universally, just in varying degree.

Counterpoint: how is anyone supposed to learn, if not from their mistakes? We might worry about the blast radius, but there’s no compression algorithm for experience.

Well, for one, you can learn from other people's mistakes, which is better than learning from your own, because then you don't have to feel the pain.

Go is a deserving target for this criticism because the language itself deliberately made a lot of the mistakes other language communities made and learned from, like weak typing[1] and naive garbage collection algorithms. Literally if you opened an undergraduate textbook on either topic you'd see much better ways to do things. But early adopters argued vehemently that Go was simple and didn't need those things.

It does seem like Go is learning from their mistakes here: they've introduced precise garbage collection and it seems like some form of generic or template types are inbound in the next few releases. Perhaps in a few years Go will be a language I am willing to work in. But it would have been nice if a new language which already had these problems worked out had become popular, instead Go, which has reached popularity through hype rather than technical merit[2].

Tracking the history of template/generic types has been somewhat humorous: you can almost see it in this article[3] where the author starts in with the title "Who needs generics!", goes on to describe some frankly horrible ways to get around the lack of generics (it's amazing how complex Go's simplicity can be!) and finally backpedals in an update ("I am the last one to balk at the idea of generics in Go."). I don't mean this to be picking on the author here though--I've seen this history played out on other blogs and in the comments of Hacker News as well.

[1] Before you flame me on this, ask yourself if you can articulate the difference between strong and static types, because if you can't, you don't have the prerequisite knowledge to have an opinion on this.

[2] It's worth noting that the decisions made in Go probably have merit within the context of Google. The problem is that most Go users aren't at Google, and have different problems than Google, so the tradeoffs made by Go are nonsensical for their use cases.

[3] https://appliedgo.net/generics/

The question was and is rhetorical. People don’t inwardly digest the mistakes of others. And anthropomorphising a language? Most peculiar.

None of this makes NIH a less than widespread phenomenon.

The readers of this forum often do know their type theory. Gatekeeping otherwise won’t go over well, it just reads like an arrogant insult from someone utterly lacking in self-awareness and accustomed to presuming themselves the smartest person in the room with the only relevant opinion. Ironically, given the subtopic, much like Google often does.

> The question was and is rhetorical.

The fact that you ask a question not expecting a direct answer is not proof that a direct answer does not exist.

> People don’t inwardly digest the mistakes of others.

In my spare time, I'm a rock climber, and mistakes in my rope systems can kill me. The same is true in mycology, firearms, airplane piloting, civil engineering. If you really feel that you can only learn from your own mistakes then I guess it's lucky for you that you've chosen to learn in a field where the stakes aren't life and death.

> The readers of this forum often do know their type theory.

That's true. The same is not true for the many Gophers who repeatedly claim that Go has a strong type system, which is who that comment was directed at.

No, but it’s a trap. If someone answers an obviously rhetorical question, they’re inadvertently demonstrating a predilection for engaging the construction, not the substance, of a statement, and almost certainly missing the ironic subtext.

I’d be happy to repeat my assertion though. People don’t inwardly digest the mistakes of others, which is why educators on safety-critical topics such as those mentioned must go to extraordinary lengths to extract and convey the salient teachings, translated into better practices, drills, equipment etc.

Reading the archives of the Dropzone fatalities database, for example, won’t make me a better skydiver.

Conversely, the best structured educational processes I’ve experienced are essentially offering the student the opportunity to make their own mistakes, but under circumstances that don’t have consequences (other than pedagogical or scholastic)

> No, but it’s a trap. If someone answers an obviously rhetorical question, they’re inadvertently demonstrating a predilection for engaging the construction, not the substance, of a statement, and almost certainly missing the ironic subtext.

That's a pretty self-aggrandizing analysis of the situation.

From my perspective, I got you to make the statement, "People don’t inwardly digest the mistakes of others", which sounds a lot more absurd when you actually say what you mean plainly instead of hiding it in rhetoric.

Maybe some people go through life that way, but that's a pretty poor life strategy and I personally make a pretty big effort to learn from other peoples' mistakes. Maybe I haven't been successful always, but I can point to lots of examples of where I have.

> ? "I got you to make the statement" ?

That's quite the signal of bad faith debate. I don't think it's my own aggrandizement in play here. Quite the reverse. c.f. remarks passim re. hubris. So there the conversation must end.

The guy who thinks he "trapped" me with a rhetorical question accuses me of arguing in bad faith? Okay...

All I did was get you to say clearly what you believe. If what you believe is so embarrassing that it's a sign of bad faith to get you to say it in clearly, maybe believe better things?

Another option is to learn from the mistakes of others.

The problem is you don't even know where to learn about the mistakes of others until you make the mistake yourself and in making the mistake you get some clue was to what to search for that then uncovers the mistakes of others.

I've found that when I ask others what mistakes I should avoid, they tend to answer the question.

Another way to discover any mistakes is to ask people why they didn't do certain things which you think are good ideas. Often that reveals that they did do that, and it turned out poorly.

People rarely post their mistakes to stack overflow.

They surely write papers about it.

I bet anyone at Google is able to access ACM, SIGPLAN, HOPL, IEEE, USENIX papers.

Most programming language implementations don't try to avoid using libc.

OK. Rust, too.

I.e. https://fasterthanli.me/blog/2020/i-want-off-mr-golangs-wild...

You lose a lot of allure of Go if "the world" is not implemented in it (mostly safety) — I think that is the core reason everything is reimplemented in Go.

Of course, the real reason is the same reason even Go exists — we always think we can do something better the next time :) But hindsight is not 20/20 when it comes to software development.

> It's like this new trend, let's rewrite everything in Go to be cool. Financially makes little to no sense.

Ha. This is maybe the archetypal software dev story.

React? Java once? C++ for OO?

guess what other shiny new language is doing that, too...

Zig?

I was thinking Rust, but in reality it's probably all of them :)

Go also commits another un-Unix-y sin, in my opinion, in that it doesn't respect kernel keepalive. In other words, you can't use sysctl to configure it. Every single Go app has its own behaviour, requiring a recompile to change.

As far as remember, Go didn't enable TCP keepalives until 2018.

Nodejs Sequelize is transparently doing this when ping connection.

That aside, I wonder why you need to keep the connection alive for > 30 mins, while usually sql con is short lived. Why can't you just close and reopen them, is it temporary table?

Not the GP, but I assume transactional consistency was important for the report, hence the need to keep the connection alive. That's a pretty common situation.

Does it means the report take the tables hostage for 30+ minutes? How can transactional consistency be achieved without write lock? Temporary table?

Reporting can typically be split into two transactions. One long-running readonly transaction with snapshot consistency obtaining the data for the report and a separate transaction which publishes the result.

MVCC.

SQL connections can be arbitrarily long-lived. ETL processes and other data-intensive jobs often take multiple hours.

OP said it need to sleep for 30+ minutes. Is it common for connection to idle for half hours for etl processes?

Interesting I’ve run into similar issues when we put network load balancer from aws in front of a db. It has a fixed tcp connection timeout so similar if the query takes a long time it’ll disconnect. We fixed this at the socket level by ensuring our linux syscnf was set to keep connections alive at an interval below network load balancer. Was a tricky problem to figure out.

Tangential: any good sql libraries in go? gorm lacks in features like prepared statements, batch inserts and we end up knitting raw SQL queries.

which library?

Sounds like pgx.

Spent forever debugging this myself.

https://github.com/jackc/pgx/issues/494

was the go library fixed?

No clue!

> The most senior developer (24yo or so)

I see the problem there.

I've seen older developers that call themselves senior, but lack basic knowledge. I've seen younger developers, wise well beyond their years. Age simply isn't a big factor in how you judge a developer.

Just because you saw a few exception does not mean the rule does not hold in general. Or are you saying most people don't learn with time (a corollary of your theory that age is not a big factor)?

I would say that I have seen no discernible pattern, so I have learnt that it is imprudent to judge a developer by their age. People do learn over time, but some people learn more "cogently" than others, i.e. some get more out of one year's worth of learning than others get in 10.

The best thing I've seen on the topic is: "Some people have 10 years experience, some people have 1 years experience 10 times".

That's misleading because it's too simplistic. A smart person could spend 10 years gaining real, legitimate experience and they could still be eclipsed by someone with little experience but much more talent.

Talent is no substitute for experience.

I'd say it's mostly the other way around, experience can't substitute for high natural cognitive ability. Of course a person still needs experience, but people with high cognitive ability don't need nearly as much, and people with less cognitive ability will hit thresholds of capability much more quickly. A lot of people live in a bubble of people with similar ability so they don't grasp the true importance of ability. And fakers who learn nothing year over year but have "years of experience", extremely common in this industry, don't like being told there are 14 year olds way more capable than them at their own jobs.

I mean talent is no substitute for experience in the sense that experience teach you things about how the world works which you cant figure out just by thinking, however smart you are. But sure, cognitive ability is a multiplier, and if you don't learn from experience it is wasted.

A have met people much smarter than me who wrote bad code because they were working from a theoretical framework which just didn't correspond to realty.

But your point is only valid if those who do learn with time are either rare (so do not make a dent in the general trend) or they somehow also do not learn much after a certain age, both of which I find deeply problematic.

Programmers by the nature of the profession must be good at learning over time... and I find it difficult to believe someone who is good at learning when at 20, will be bad at learning when at 40...

> some get more out of one year's worth of learning than others get in 10.

Sure, but they probably continue learning the year after?? And the year after??? Or they too, stop learning (or are just too rare)?

> most people don't learn with time

I think this is correct. The curve flattens with age (in my opinion).

> you saw a few exception

I think the seniors you met (the knowledgeable ones) /were/ the exceptions. Most seniors I have seen have convinced themselves that they know all there is to know.

I am getting on a bit and its not that I think I know all there is to know, but I see so much new tech that seems like pointless unnecessary complexity. I reckon that more than 90% of apps could be done in a classical MVP, with a bit of JQuery and a well tuned database. Most new ideas are a rehash of something from decades ago. It gets rather tiring.

> Most seniors I have seen have convinced themselves that they know all there is to know.

Yeah, seems we've met different kinds of seniors... I don't know any senior who wouldn't understand that "the more you know, the more you understand just how much you still don't know".

That quote is very true!

Most people don’t magically grow wiser with time. They need to be in an environment where they can grow, otherwise they’ll emerge just as stupid as before.

If you start out with a cohort of individuals across the spectrum of ability, over time many of those of lesser ability will self-select out of the pool. It is my experience (oops, see what I did there) that people rarely spend a career doing what they're not very good at. The exceptions inevitably stand out.

So, combining the winnowing of the not-very-apt with the gaining of knowledge through experience, the end result is that you have a preponderance of wise old experienced contributors.

I you are one of the younger inexperienced ones who believe they know better it's likely you'll self-select at some point, secure in your Dunning-Kreuger knowledge, and move to some career in which your high level of competence is valued more.

The counterpoint is you need time to grow wise, and there's no way to get wise without many years of experience.

That’s also true. I guess that the relative percentage of people you’d call wise increases with age.

Shameless age discrimination. Very smart people exist at all ages. I've seen 21yo junior grad developers outperform 45yo 'senior' developers.

It's unlikely (but not impossible) that a 24 year old developer has had much of the experience needed to be classed as properly senior.

Seniority is not about how smart you are. And most programmers of any age are neither prodigies nor geniuses.

Do not accuse people without evidence! I'm all against age discrimination.

..and when you see a company where the age cluster around a very small area you can tell age discrimination is happening. Which might very well be the case of the parent post.

It's even more complex as you cant even say dev X > dev Y for all tasks. So the age thing is even more silly. After so many years you start forgetting the stuff you did (in detail) in the first years anyway.

You're bringing up the exceptions that prove the rule as if they disprove it.

Age is a just a number. John Carmack created Doom in his early 20s.

That's a weird take. You don't think Carmack 2020 is wiser than Carmack 1993?

He is currently working on artificial general intelligence.

Every good rule has an exception.

Why not judge people by the result of their work, instead of their age?

Because then it’s harder to deal with our own imposter syndromes if we can’t blame it on the youth and hold their heads in the toilet while giving them the professional-development equivalent of a wedgie.

This was discussed at length in last week’s “Grey Beard Weekly” newsletter.

So the "old people" are the bad guys for you, hmm.

Then again, it's mostly "old people" who are discriminated against when it comes to hiring.

So the "old people" are the bad guys for you, hmm.

Yea that was snark, bud. Doesn't always translate well over text, admittedly.

But no, not the "bad guys". That's an interesting conclusion you've arrived at.

It seems so, and the fact that people are accusing me of ageism as I pointed out that a company hires only people below 24 years of age is very telling...

You didn't point that out, you made a vague complaint about someones age, and the comment you replied to doesn't even give you a good basis to assume that, so it's no wonder nobody understood what you tried to hint at. Hint: in many companies, a project involves only a small part of the workforce.

Now that you have advertised the newsletter. Maybe share a link too.

There isn't a newsletter. It was tongue in cheek.

"that there was nothing in the middle managing connections"

Found the real bug, and it wasn't in the library.

Nothing about go is magical, and you probably solved the problem blindly.

Wow, okay. You alright over there?

Edit, because meh: I'm making no claims about go itself. No idea what makes you think that's what I'm saying, since I'm clearly talking about a library, and not even any stdlibs. "Magic" is just a term useful for describing systems that sweep much of their abstractions under the carpet in a way that probably has gotchas. Granted, the term itself is magical.

In terms of fixing the problem, I knew for a fact that the keepalives that I was seeing were nothing like what I've seen in the past, at many companies, across Oracle, postgres, and MySQL, all who've implemented "SELECT 1" for the sake keep alives, by devs who've been in the field for much longer than me. The suggestion was by no means blind, unless you consider implementing a widely used method for this exact purpose, "blind". Had I gone a different route in fixing the problem within the stable, existing system, it would have likely broken many of the other database connections by many teams. I'll pass on that, since frankly, even ignoring the risk of such a change, the dev should have done the investigation themselves.

Your post was unnecessarily aggressive and seems to come from me having struck a nerve somehow. Genuinely hoping you're doing alright. Peace.

I presume they took it as an attack on Go. Truth is, it's an attack on the library developer who themselves may have found their keep-alive solution by stumbling blindly on it.

Select 1 is a really hacky keepalive that relies on several assumptions, I'm not surprised it wasn't expected.

I have no affection towards go or any language. They're just tools. You sounded elitist calling something magic and pointing out someone's age as part of your point. And your passive aggressiveness to my response is proof of it. I "genuinely" hope you're doing alright too.

Heh, I wasn't being passive aggressive, I was being serious. Take a breath, man.

Also "magic" is not a new term: https://en.wikipedia.org/wiki/Magic_(programming)