Hacker News Reader

Mockery is never OK.

That said, their experience probably reflects my own: using SQLite on a personal project for config or something, loving the simplicity, loving the "ditch all that complexity" hype, then trying it on a more demanding project that according to their mental model it should have excelled at (a central job tracking process), watching it completely faceplant under load (20TPS? Surely it just needs an index or something? Maybe a ramdisk? Err.....), panicking when they realize that perf and monitoring and diagnostics and optimized backends were some of those "bells and whistles that nobody needs," deciding "fine I'll do it myself," throwing far too much time and effort at co-debugging python and C, recompiling SQLite with debug symbols, sprinkling printfs, running profilers, reading the source, and tracking down the issue... only to find that it was a bad default which people had been unsuccessfully begging the maintainers to change for years, which was guaranteed to torpedo perf under a wide variety of common workloads, and which the maintainers had been resisting based entirely on inertia and the misguided idea that this setting was easily discoverable. Yikes.

That's how someone picks up rules of thumb like "SQLite is for preferences, not perf, and definitely not prod."

Now, this experience is a decade stale. SQLite bit me in 2012 and the particular bug was fixed in 2015 IIRC. The people begging to fix the bad default finally won. Time passed, SQLite got better, and now, based on these new stories of SQLite doing well under load, I think it's time to lift my "once bit, twice shy" sanctions regarding perf. I had them in place for a reason, though, and if you had YOLO'd when I had YOLO'd, you would have faceplanted when I faceplanted. I fully defend the use of sanctions like this in general. They are the only tool we have against overenthusiastic marketing.

To be fair, the SQLite site is pretty good, it's the fans that tend to take things too far.

Speaking of potentially obsolete sanctions, conferences have been on hold so I might be hanging on to a few more of them past their due date. Does anyone have fresh experiences with these?

------------------

AMD GPUs (2014): Waiting for GPGPU support from Adobe and Blender that was supposed to be operational (Adobe) or getting there (Blender) but never landed.

AMD GPUs (2016): I burned an entire semester trying to port CUDA code to OpenCL only to run into bug after bug after bug. Lockups, black screens (no framebuffers were supposed to be involved but bugs find a way), leaks, stalls which were difficult to attribute, unscruitable errors leading to abandoned threads with pleading users from years past with no reply. I finally cracked when after a long day of "I swear this should work I have no idea how the changes I make correspond to the bugs I see" I ran my ailing OpenCL code on a weaker NVidia card, and it Just Worked. Not only did it just work, it was 10x faster. Even though the card was like 2 generations older, and I was using OpenCL, which NVidia had every reason to sandbag, but they were still winning at. Oh, and the debugging tools actually worked, too. I was persuaded to sell my red cards, eat the ebay tax, eat the nvidia tax, and buy green from then on. Time has passed since 2016, though. ROCm looks good and AMD has money now, so maybe they can afford to pay people to fix bugs. I want to see someone running Blender and a modern ML framework or two before I retire the sanctions, though. Anyone have fresh intuition here?

Numba CUDA (2018): I spent two weeks working through a succession of problems that I eventually resolved in two days by just using CUDA C++. Numba CUDA only worked with year-stale CUDA, it copied float8s over a float4 array despite having access to triply redundant type specs, the mechanism to pass buffers between kernels always forced copy off GPU, and one more that I forget. I made patches for the first three and gave up on the last one before fixing. Has anyone used it recently? Is it in better shape these days?

I was hired by AMD to work on ROCm in 2020. I'm not exactly an unbiased source, but IMO it has improved significantly. There's a number of working frameworks and Blender was recently ported [1]. Just be sure to choose appropriate hardware.

ROCm still has lots of rough edges, but we're slowly filing them down. The documentation is a bit patchy too. Overall, though, it's been a steady progression in the right direction.

[1]: https://www.phoronix.com/review/blender32-hip-cuda

For OLAP applications (heavy on reads, pretty much no writes) I find a local SQLite has far superior performance to eith cloud based Postgres or SQL Server instances. YMMV.

> went as far as to mock my engineering choices

That's not acceptable.

> Experienced software engineers

Measured in years, probably, not by ability.

I've heard supposedly 'experienced' engineers saying things like "No-one uses Lua in production!".

It’s amazing how otherwise competent engineers will look down on technology that so many use all around the world when generally their contribution to society is a few buggy Python classes in some internal codebase.

Are they competent, or have they only written a few buggy classes? Your strawman needs work :)

That said, I'd agree that mockery is rarely ever appropriate, and is often undertaken by those without the credentials to do so.

It's also possible that someone is expressing hard won knowledge/battle scars in an unskilful way. And it takes investigation to know the difference.

For a counterexample:

Some pretty big businesses have been built on the back of PHP. That doesn't make the language any less ridiculous.

Why?

Because there are objective measures by which a programming language can be judged and assesed to be better or worse than others of its peers, and those measures, like a powerful (not necessarily static) type system and consistency of the language's core design and libraries, put PHP at the bottom of a hierarchy of languages.

This need not imply condesenction to the language users or designers, not as individuals at any rate.

Yes. And you might quibble about the exact criteria. But by almost any metric, old-style PHP was an awful language.

(Facebook's re-invented Hack isn't nearly as bad. It's about the best language they could have made starting from PHP.)

There was an article about PHP that summarized some of its issues back in the day: https://eev.ee/blog/2012/04/09/php-a-fractal-of-bad-design/

I think that it was excellently written and actually mentioned concrete issues, rather than vague personal dislikes. That is good, because you can revisit those in 10 years or so and see how many of those have been fixed in the actual language.

Someone actually tried to address a few of those: http://maettig.com/2020-09-16-revisiting-a-fractal-of-bad-de...

Thanks for the second link! It's good to see someone (sympathetic to PHP) address the issues reasonably objectively.

That's part of why I carefully wrote 'old-style PHP'.

When I hear that sort of mockery, especially from so-called "thought leaders" and other loud voices, in conference talks, in long-winded YouTube videos (rants), etc., it's such a big red flag to me. Even though I might agree with the individual in a general sense, it makes me wonder what they're missing or getting wrong or biased against in the areas where I don't have the (supposed) expertise that they do.

I sometimes wonder how much of this objection to SQLite is purely technical, and how much is due to the fact that Dr. Richard Hipp would not exactly be a great "cultural fit" at places like the New York Times or Mozilla.

SQLite is great. But the simple fact that it runs locally makes it a poor technical fit in many scenarios.

For every technology there are scenarios in which it is a poor fit.

You don't want to install Postgresql or Oracle on an embedded system nor would you want the extra weight of administering them for a system that has few users.

The SQLite site runs on SQLite and it has considerable traffic.

> The SQLite site runs on SQLite and it has considerable traffic.

The forum and source code trees run on sqlite (via the Fossil SCM, authored by sqlite's Richard Hipp). The main sqlite site is static HTML with a tiny sprinking of JS.

"But the simple fact that it runs locally makes it a poor technical fit in many scenarios"

MySQL and PostgreSQL can run locally too. I assume you mean that SQLite doesn't provide a network server interface out of the box?

FWIW, i just almost wet my pants laughing that that :).

I would guess that at most 1 in 50 people familiar with SQLite even know Dr. Hipp's name, much less anything about him that might influence their perception of his work.

I've looked at his homepage, and only saw something about bible translations. Is there anything else that's iconically non-NYT about him?

Never underestimate programmer machismo. So many people seem to rule out SQLite because of programmer lore that it's not fast enough or doesn't scale. But ofc Postgres is a real database and therefore is the automatic correct choice.

A lot of things don't need to scale (and never will) so using a simpler solution is more economical and efficient.

PostgreSQL is now facing that too. Since the current fad is building everything as distributed micro-services(no matter the scale!) and PostgreSQL encourages a centralized database, it tends to get dismissed for other solutions (that often aren't anywhere near as good).

"Doesn't scale" gets thrown around often. Yeah sure, but sometimes it doesn't _have_ to. I've seen large fleets of NoSQL databases (won't name to avoid flamewars) running workloads they are not really optimal for. Or PostgreSQL databases completely bogged down and vertically scaled to ridiculous levels because teams don't bother to optimize their queries.

Query performance is not part of OKRs, but new releases usually are.

I've seen highly scalable distributed DBs used as poor choices, and have used Postgres as a replacement in some cases.

It really depends on the kind of problem and the scale, but a good implementation that matches the problem will sing compared to a mediocre implementation on inappropriate yet highly scalable infrastructure. Even if it's supposedly inferior because it's centralized, etc.

Knowing the constraints of the engineering challenge and choosing wisely within that is key. You can sometimes leverage constraints and simplifications in ways that can stack and remove many layers of complexity. Immediate consistency is a huge one, as are transactions.

I avoided relational tech early in my career, when I was really just turned off by examples of poorly tuned implementations, and badly written schemas and queries. Once I finally opened my mind, I was better able to appreciate its role as compared to other DB models.

I think that's a natural maturation as you progress in your career/get more exposure. It's too bad most of us seemsto have to walk the same path though.

"Mongo DB is a web scale DB and doesn't use SQL or Joins so it's high-performance. Everybody knows that relational DBs don't scale because they use Joins and Write-To-Disk."

https://www.youtube.com/watch?v=b2F-DItXtZs

I always love watching this.

I'm looking to develop a self-hosted, open source Node.js web app mainly to upgrade my resume (as well as to scratch an itch). I considered SQLite to be the perfect choice - the software will be mostly used locally in a single user setup, so that saves RAM and even if someone wanted to host it for many users it would still be able to handle a lot. Plus, the package will be so much more portable, with SQLite it can be distributed via npm install, rather than a Docker image. I initially wanted to use one of the "real" databases concerned that an interviewer may look down on it, but decided to scratch that idea and use the right tool for the job. Your comment makes me reconsider that choice. I think I will do SQLite anyway, because it's something I want to use myself and I also will be applying as a front-end, rather than full-stack.

Crazy to hear that.

Those conversations are my favorite when interviewing candidates because it gives you a view into how their brains work.

I would’ve nerded out about how so many macOS apps use SQLite under the hood because it’s so solid.

SQLite is even used in the Airbus A350 and they signed (approx something like) a 35 year support contact with Airbus for it, I remember hearing in a podcast a long time ago. (Please correct if I’m way wrong here)

> Experienced software engineers from The New York Times, for example,

I can think of a couple of scenarios where they wouldn’t want to use SQLite, such as high availability database to support their subscription system, or the back-end for their interactive website (which was a mean stack at one point, so not even SQL).