The Database Administrator is dead

> The successful DBA will understand that this new world means handling petabytes of data and billions of operations on thousands of logical databases.

is untrue. Most of our customers have DBs that are in the GB size. A few have TB size DBs, and none are on that scale.

datastore-as-a-service doesn't replace DBAs - we make good money being remote DBAs for people who are using datastore-as-a-service providers, because they still run into the same problems as everyone else.

Sorry to hear it, either way.

Yup. And with the supply being so low, it's hard to get a DBA (you'll probably have to steal one from another business), so people are flocking more and more to DBaaS, and DBaaS providers are more than happy to propagate the fiction that "you don't need a DBA, you have us!".

It honestly doesn't bother me much that they make these statements; it's marketing.

On the other hand, believing those statements harm our customers; they spend time and money to migrate to these providers and find out the hard way that they still need someone who can handle their DBs for them. That does bother me.

More memory? Increase the buffer pool size.

Faster HDD? Tweak the settings that determine how many disk operations are attempted every second.

Bigger CPU? Figure out the point of diminishing returns on the number of CPU cores for your DB, and start sharding onto multiple DBs to make sure you can use all of the cores.

SAN? But I thought you wanted performance. ;)

Plus, what gives you the best DB kit for the buck? I could probably tell you that (I am a DBA, and get paid to answer those questions), but do you know? Do you know where to find out?

But Oracle needs the finetuning, and I can't see that changing anytime soon, with Oracle some queries basically require that you use a IOT, other some are better with partitioned storage, and so on.

So your premise is extremely dependent on what DB you use.

Is ETL considered a DBA task?

In other words, efficiency and performance both depend heavily on the machine your DB is running on.

As such, a good DBA needs to be able to do sysadmin tasks. The business won't care that it was the sysadmin's fault for not realizing that a battery had gone dead on the raid controller, and a DBA shouldn't care either. Their purvue is the database, and everything that it entails.

Given that background I still think that the topic of databases is just too deep for a generalist. I know a lot about the MySQL database (and a little about PostgreSQL), enough to write failover software, automate deployments, write guardian crons which slap down problematic queries & pre-emptively, automate backups, do vip failover and haproxy configuration... and I still have to go to my boss for most of the hard questions.

His knowledge encapsulates 12 years of working with and around MySQL, and it's proven invaluable to our customers. Knowing when to force certain optimizations, how to make subqueries run O(1) vs O(n), how to rebuild a complete database from binary logs, how to configure MySQL to work with SSD caches... these problems don't come up often, but when they do, not having a DBA available to you means contracting out to one at exorbitant rates.

It's the difference between a few minutes of downtime when the proverbial dung hits the fan, versus a few hours or days.

Start with the book High Performance MySQL. [1]

Follow up with the whitepaper "Causes of Downtime". [2]

Then find a copy of the IMDB dataset, put that in a database, and write an app against it. Make that app perform well, then simulate load against the app (pretend it hit the top of Reddit and Hacker News simultaneously), and keep it performing well.

After that, it's a matter of practical practice.

[1] http://www.amazon.com/High-Performance-MySQL-Optimization-Re...

[2] http://www.percona.com/redir/files/white-papers/causes-of-do...

And who wants to do that?

People may assume whatever they wish.)

> Who compiles anything for PHP4?

It was long ago, but you probably wouldn't believe how many people are scared to touch anything, leave alone to perform even a necessary security update - "what if it stop working?!"

Unfortunately, frustration, which sometimes influenced wording of some of my comments, is based on a quite long time in the industry, and I never asked for it.

That's a pretty easy statement to defend. But, I'll respond by saying that most Companies running Oracle 11g with more than a couple terabytes of databases, require a competent DBA, particularly if Disaster Recovery/Transaction Rollback is important.

" the DBA often never has sufficient domain knowledge of the problem "

Of the half dozen or so truly high level DBAs I've worked with (and managed on occasion), I can say they had incredible domain knowledge of Oracle Database Server, and worked extraordinarily hard to have next to zero knowledge of the application running on it. Their focus was to keep the database running, defend it from engineers and users, and recover it when things went really awry.

"DBs performance is complicated, yes, but the vast majority of it is extremely simple."

Any time you see the phrase, "Extremely Simple" when discussing a domain in which the expert practitioners routinely make $250K/year or more without any form of market manipulation, you need to reconsider why, exactly, these technicians are being paid so much to do something, "Extremely Simple."

"What you need to do is teach your developers how to read those query plans. "

Completely agree here, but, there are two perspectives on this topic. There is the "Engineers are ultimately responsible for the efficiency of their query plans, and should be educated/trained to take that responsibility" and then there is, "We can't train our engineers to be query plan experts, just keep them from shooting themselves in the foot, and let Query Optimizer handle the rests - it's up to the DBA to manage stats gathering to keep DBM_STATS healthy"

I think we tend to see the second approach more frequently in the enterprise, where your engineers are likely making less money, and the company is keen to leverages their many 10s of millions of dollars of Oracle Technology.

Finally, when a company is paying 10s of millions of dollars a years in Oracle licenses, they consider it a worthwhile investment to have a few high-level DBAs to fully leverage that investment.

Without getting into the rest of your argument, I'd like to quickly address this.

No, it really isn't.

In your typical MySQL database, your performance for a simple "select * from x where y" is going to go through a lot of complicated machinery (most of which can be tuned for performance), a few points of which I will enumerate below.

    1) Acquire a query cache lock & see if this query is there
    2) Run the query through the optimizer
      2a) Perform multiple shallow dives into a table to look at the cardinality of the filtered columns
      2b) Identify the best indexes based on the shallow dives
      2c) Create a query plan
    3) Push the query plan down into InnoDB
    4) Load the index into memory, if its not already there
    5) Load the potential rows into memory, if they are not already there
      5a) If there's not enough memory, load a few into memory, and be ready to push those rows out of memory in favor of more rows when needed
      5b) Load rows that are still in the insert tree but not yet part of the regular buffer pool or pages on disk
    6) Loop through the candidate rows for matches to the filter
    7) Return the data to MySQL
    8) Acquire the query cache lock & update it
    9) Return the data to the client

As an example, I attended an introductory course to being a MySQL DBA; it lasted 5 days of 8-5 teaching & running examples. And it only scratched the surface of what I do on a daily basis.

"often never"?

I've met some DBAs who were quite versed in the business domain. And some that weren't. This is no different to the majority of the developers I run in to, so I'm not sure why you're drawing a line there, except that the article was about database stuff.