Oracle engineers caused five days software outage at U.S. hospitals

I took over support for a small ISP after the previous sysadmin died unexpectedly. He was a big fan of putting code in the PostgreSQL database, which I had never done before. That was fun to figure out and debug issues.

Haven’t worked with Oracle in over a decade, but SQL Server’s tooling is actually quite decent in that regard. You can attach the Visual Studio debugger to a T-SQL stored procedure, set breakpoints, step, inspect variables etc. There are rough edges, but it’s good enough to be useful.

Automated tests of T-SQL stored procedures are also possible. There are couple of ways you could do that, but in our case we just write “unit” tests in C# which call the procedures. This integrates nicely into Visual Studio’s Test Explorer, is debuggable etc. You can even step through a C# test, and then just “step into” T-SQL being called from there!

There are valid reasons for not putting business logic into stored procedures, but “not being a proper language” is probably not one of them, at least in the case of T-SQL. Then again, there are also reasons for using stored procedures. Everything is a tradeoff…

Curious if that is from recent experience? I used Oracle decades ago but these days it's all SQL Server.

I didn't need a header anyway.

Maybe. One thing I've found is the LLMs give me a lot of short term leverage. I can (get an LLM to) program in a language I know almost nothing about. Then I hit a wall where I don't know enough to get the LLM to fix something I don't know how to ask for, and then I'm stuck. When vibe coding, I'm not learning how to solve problems myself so much, and that means anything the LLM can't do, I also can't do. When I'm coding on my own, I'm picking up lessons along the way the help me build a foundation for incrementally harder projects.

I'm not so sure. I did an experiment last year. I was writing a prototype in a language I'm not well versed in, using unfamiliar tools and only somewhat familiar libraries. I made heavy use of ChatGPT and other tools to get the job done. (Well, actually, I wrote several pieces of software this way, but only one major for-work prototype).

Some observations:

Initial velocity was high. It felt like I was getting things done. This was exciting and satisfying.

The code I wrote was structurally unsound as it ended up having no overall plan. This became more and more evident as the prototype was completed. Yes, it was a prototype, but usually I use prototypes to think about structure. (Choosing the right way to structure a problem is a key part of solving the problem)

My retention of new knowledge was terrible. Ditto for developing any deep understanding of the tools and libraries I was using. I had skipped the entire process by which I usually learn and just had the LLMs tell me solutions. This provided a poor basis for turning the prototype into production code because I had to catch up on all the thinking that should have been part of producing the prototype. And I had to sit down and read the documentation.

LLMs are only as good as your prompts. You actually need to know quite a bit in order to formulate good prompts. Being a fairly experienced programmer (and also having a some knowledge of LLMs and a former career in web search) I had significant advantages novices do not have.

Now imagine a novice who lacks the experience to see the shortcomings of their code (hey, it works doesn't it!?) and the ability to introspect and think about why they failed.

Half of the time the LLM would lead me astray and pursue paths that resulted in poor solutions when better solutions would have been more obvious if I'd just read documentation. It is easy to become focused on exhausting paths that the LLM sent you down in the first place and poke it to give you something that works.

I have nearly 40 years of programming experience. It scared me how stupid relying too much on a LLM made me. Yes, it got me started quickly and it feels faster. However the inflection point that comes some time into the learning process, where things start to click, didn't materialize. Mostly I was just poking the LLM in various ways to make it spit out solutions to tiny parts of the problem at a time.

I still use LLMs. Every day. But I use it more as a fancy search engine. Sometimes I use it to generate ideas - to try detect blind spots (solutions I don't know about, alternative approaches etc). Having been burnt by LLMs hallucinating I consistently ask LLMs to list their sources and then go and look at those.

LLMs are *NOT* mentors. A mentor isn't someone who does the thinking and the work for you. A mentor is also not an alternative to reference material or the means by which you find information. You're expected to do that yourself. A mentor is not someone who eliminates the need to grind through things. You have to grind through things to learn. There is no alternative if you are going to learn.

A mentor is someone who uses their experience and insight to help your personal growth.

> If anything, I think the youngsters will learn faster from their mistakes because they already have good mentor for the easy stuff and will get grinded on the hard stuff sooner.

Maybe? Back when I had to troubleshoot coaxial network terminations uphill both ways in the snow, we had to learn how things actually work (e.g., what's in a tcp header, how to read an RFC) and that made debugging things a little more straightforward.

I'm pretty insulated from young developers these days, but my limited experience is that they might know the application, presentation and maybe session layers (if using the old OSI model) but everything below that is black magic.

Because some human errors are bound to happen, what is often missing is procedures to minimize them, catch them or prevent them from having catastrophic effects.

It is not even about being technical. Have a person put data on a spreadsheet and you can get into so many errors if the procedure of doing that is bad.

> It's a very common cop out by engineers/technical folks when something goes wrong somewhere else, to blame it on management/deadlines/customers/etc.

Look, I've been an executive and a management consultant for a long time now (started as a sysadmin and developer), and it's quite often the case that everything was late (decisions, budgets, approvals, every other dependency) but for some reason it's ok that the planned 4 months of development is now compressed in to 2.5 months.

I have been involved to some degree or another in probably close to 300 moderately complex to highly complex tech projects over the last 30 years. That's an extremely conservative number.

And the example I describe above is probably true for 85% of them.

Feels like the exact opposit story when I read engineers commenting online.

Oh come on, let's be honest. All of us have colleagues who we don't trust with mission critical stuff.

Are managers really not doing background checks about company behaviour before signing a contract?

> due to young people not taking an interest in COBOL and Mainframes

Having only fleeting professional experience with COBOL during a summer my view of it is that it is a mix of dataanalyst job and programming. Where the programming is horrible and the report making is ok though archaic. As long as you modify processes already available it is not so bad, but the developer experience was horrible.

With all that said I actually liked ideas in COBOL but it is an extremely niche language that does not serve you at all in the real world.

> young people not taking an interest in COBOL and Mainframes

It is more like young people not wanting to:

- throw their careers out of the window by pigeonholing themselves into zombieware tech

- experience high levels of stress trying to debug code older than their parents, writing code that can't be unit tested and pushing said code to production

> In most cases you're talking 50-60 years of "legacy" (still running, still working, still updated) code, and replacing a system that has evolved for half a century is not an easy undertaking, at least not if you plan on getting it 100% right the first time.

This seems like an application area in desperate need of applied A.I.: Analyze a codebase and replicate it (for the first few iterations at least) in the new target language.

Isn't there research on this ? Or am I full of balonie ?

Java has been a staple of university courses since before Oracle bought Sun. I guess I am getting old now, if the kids all want to program in Python and Javascript for 4 years...

As much as I hate Oracle, there are huge advantages to Java. Java is strongly typed, free, and runs damn near anywhere. It's pretty much the go-to language when people think about OOP and for good reason. It has some excellent open source IDE support and it's widely used in the industry. With the current OpenJDK setup, it's also free of Oracle's licensing issues. It runs fast and can support most algorithms you'll probably ever study, without the manual memory management troubles of something like C++.

I was taught C# in uni for very similar reasons except the entire uni ran on Windows and the Microsoft platform, which made doing assignments on Linux rather inconvenient. With the status of dotnet core, I'd say Java finally has a good competitor when it comes to teaching OOP languages.

I was at Uni in Australia well before Oracle bought Sun. Java was used quite extensively then too.

I think it’s from Sun Microsystems who had the friendly to edu strategy

Java was taught far before that: we got taught the first public beta version in the 90s in uni. Thought it was nonsense compared to c, prolog, lisp and Haskell.

Same.

I work with engineers and technical managers with 25+ years of experience, building and maintaining serious business software 'you could run a country with' - people here build react web apps or do scientific research, or work for a SaaS provider - completely different view than building highly complex, regulated, mission critical software that supposed to run for decades and be supported at this level.

OracleDB forbids benchmarking it in the license, so I just presume it's inefficient and too slow to compete.

Beats at what? Not mission critical multi decade setups that can be easily repeated by hiring someone from postgres consultancy. Oracle handles every use case you throw at it: maybe not optimally, but that is not what you care about at that level anyway. They suck but what are the actual alternatives: not postgres or supabase for most orgs of significant size.

Postgres is a better choice for 99% of the companies out there. But there are cases where you need ability to massively scale AND control your database cluster perfectly.

Postgres won't even let your force an execution plan and ignores hints (yes, there is an extension for that) so your optimized query can at some point just 10x in execution time and increase load in production when the plan changes.

In Oracle, I am told you can prioritize certain queries at certain time of day - it's crazy what it can do. Yes, it's slow and expensive. If you have money to throw at the problem, it's fast and it solves your problem, whatever the scale. Their Exadata cluster, for example, is wicked fast storage layer pre-filtering the data it sends to the database.

Of course, I despise their business practices - especially the abuse of customers via audits. As a database, it absolutely has its place regardless of lobbying, corruption, and whatever else they are doing.

The Oracle database is an amazing piece of software. The problem with it is as open source and SQL Server started eroding its share, Oracle pivoted to owning vertical line of business software.

My employer probably spends more money on databases for our learning management system than we do for one of our main customer facing apps with thousands of concurrent users. It’s literally a tally of training courses.

Or plastic, backed by electric heating!

https://www.totousa.com/washlet