Comment by NikolaNovak | Hacker News Reader

Comment by NikolaNovak | Hacker News Reader

Parent Full thread

DBA is still a necessary job. I would say more and more so: Over my own now 25 year career in IT, the average developer seems to have less interest/knowledge in data modelling and administration. I don't think you'll find great DBA skills let alone a dedicated role in a small high velocity exciting startup - I feel they'll cloud brute force their way :). But in a large organization, a good DBA is vital. I have 2-3 on my project team and always looking for more.

It is astonishing how much a poorly tuned data request / SQL, or poorly designed data model / table, can impact performance by multiple orders of magnitude. It is also fascinating how much addition of specific index or tuning statistics can help execution. There comes a scale where throwing more hardware on performance problem is prohibitive over hiring a good DBA, as much as current thinking is that "hardware is cheap". But on a regular basis I'm seeing experienced, serious, dedicated developers create SQL that is functional correct and reads 10 billion rows into buffer in order to spit out a single row answer (sometimes of course that's necessary, but usually not :)

> Over my own now 25 year career in IT, the average developer seems to have less interest/knowledge in data modelling and administration.

Even when it's clear as day that the data model is incorrect they often just ignore it. They blame MySQL for not scaling while having ten joins per query and using ORs like they're going out of fashion. Say it can't be done in another way, even tho clearly it could.

Or say how the benchmarks for Elasticsearch are specicially crafted, you can say that when you get 30k per second and they get 50k, but when you get 50 per second and they get 50k per second it's not their benchmark that is specially crafted it's your data model is crap.

I honestly think the thing that costs most companies the most in their hosting is developers not using databases correctly and spiking the hosting costs. I literally saw one company have to spend 20,000 a month for 500 requests per second on elasticsearch.

>while having ten joins per query

Ten joins per query is quite nothing. I tend to see the opposite problem, where new devs denormalize a bunch of tables thinking it is going to improve performance when it fact it duplicates data everywhere in tables and indexes and makes things much less cache efficient, the CPU spending much more time stalled (https://en.wikipedia.org/wiki/CPU_cache#CPU_stalls) moving different copies of the same data in and out of ram and caches (or god forbid disk).

JOINS are a performance _enhancement_ in an efficient, indexed schema, especially with narrow tables when the joined pages are small enough to stick in caches longer.

This is why columnar stores like elasticsearch effectively only allow single column tables so every column always has to be joined. They also aggressively compress pages of these columns to make them more cache efficient. Its much faster for the cpu to compress/decompress/join on the fly than to move uncompressed denormalized data from different levels of cache.

In SQL world, if you keep your tables narrow and normalized and you get the same benefits plus the benefits of only having the indexes you need (also more cache efficient) rather than index everything and throw hardware at it like elastic. With SQL, you also avoid the issue of sending data over potentially congested and less reliable networks that distributed DBs like elastic have.

> JOINS are a performance _enhancement_ in an efficient, indexed schema, especially with narrow tables when the joined pages are small enough to stick in caches longer.

When used correctly nearly everything is good. But in reality most production applications in the world you'll find them doing joins where it's not performant. You'll find 1 to 1 relationships all over the place.

In many scenarios the application and database would run quicker if you do 11 queries instead of doing 1 query and doing joins to fetch the other data. In many real world scenarios. And you can literally demostrate that to some developers with a pull request that performs better and they'll continue to write their crappy SQL.

> I honestly think the thing that costs most companies the most in their hosting is developers not using databases correctly

100%. Data modeling and SQL has become a lost art.

Knowing how to data model is a skill I learned and got good at long before I was ever forced to write my first SQL statement. For all it’s power an usefulness SQL is to databases what JavaScript is to web development… some people are WAY too obsessed with using it for everything they can. And this is a bad thing. I find “DBA”s tend to reach for SQL first and can cause their own kind of chaos. From writing difficult to test code because they hand wrote SQL that only works on the production database (PostgreSQL vs in memory SQLite) to my pet peeve of constantly suggesting that a problem would be trivial to solve with stores procedures which would massively complicate (requirements completely replacing) the database migration system currently in use (because most frameworks, like Rails, Laravel and Django don’t have good built in tools for managing, versioning and updating stored procedures).

Adding a DBA to a team is like adding an opinionated senior developer, sometimes you need that experience and sometimes it’s just going to be a total fucking pain in your ass.

If you feel you need better data modelling skills in your developers, don’t hire a DBA, hire developers that know how to do data modelling.

This is a good caveat, and I agree with the thrust of your point. Especially about stored procedures. Sometimes DBAs, especially corporate oldschool ones, can really propose DBA-only solutions, and it's painful.

I think another way of thinking about this is hire "former DBAs."

Most of the developers I know who had really solid SQL, the greybeards I learned from, were either former DBAs or former sysadmins who had become full-stack developers.

edit: Another was a junior developer who interned two years as a DBA role. He was desperate to lose the DBA title, and I always respected that, but damn if his SQL wasn't better than mine.

Agreed; depends hugely on system and project/team.

I'm in the ERP space (Fun!:), which means a lot of code is effectively legacy and a lot of standards/methods are also effectively... old :-).

We are lucky on my current project to have developers who ARE senior and experienced and pretty good at data modelling; but there is still a lot of need for somebody to own the back end system and design maintenance and notice problems and optimize things holistically as opposed to program by program.

So I guess I should've clarified - a good developer certainly helps but does not obviate the need for DBA in many scenarios.

I would argue that a developer that can't structure and query data for efficiency and integrity, is probably wreaking havoc in the rest of the systems too. You don't need a DBA, you need a better developer. Managing state is an important skill outside of DBs too.

I'm ~7 years into my professional software career and it wasn't until the last year or two I took a huge interest in the database layer, I can attribute a lot of it to Designing Data Intensive Applications.

)

https://xkcd.com/541/

Reply on news.ycombinator.com