Hacker News Reader

Comment by hinkley | Hacker News Reader

IBM has a fondness for complexity that makes the current situation with cloud offerings look like children's toys.

Their Global Services division could be summarized as a War on Kernighan's Law. Find clever people, and have them strain themselves to stand up a system so complex that you not only need to keep paying GS to show up and maintain it, but in fact you probably need to hire two more people just to keep the wheels on.

That has made them many enemies on the technical side of things. People like me go out of our ways to sabotage any inroads they try to make because we know what will happen if we don't. Presentations of systems so complex you get a headache trying to wrap your mind around it.

If we talk about it, it's not for long and with as few ears to overhear as possible. It's like guerilla warfare. Targeted attacks against a foe who could crush you like a bug if they knew where to find you.

Oracle sniffing around is merely an irritation compared to the spectre of IBM GS getting their hooks into your CTO.

View on HN

My last CEO was ex-IBM. He was pissed when we (his technical team), nixed his attempt to do a storage related deal with IBM. We went through endless meetings with dozens of people on the calls. They wanted to bring in some giant complex product and we just wanted boxes with disks. They couldn't understand that we didn't need all the striping tools they had, because our storage was just using full multiple copies across servers/disks. It was a total square peg / round hole situation and their sales team tactics were to try to discredit people individually with what amounted to as being complete lies. It really put a bad taste in my mouth to ever deal with those type of people again.

I was at a company where that exec won. In this case it was the Infosphere suite to replace a fully mature Informatica/Teradata environment and fix all the "issues". Obviously the issues were all design and management, not technical.

3 years a tens of millions later the whole project is shut down with no value delivered except a single data store and a couple API endpoints. That exec went back to his IBM sales job.

Last I heard they were still paying for and maintaining all of it because that endpoint was used in a key product offering.

This is like one of those "horror stories in 2 sentences" type things.

Most ex-IBMers want to "go home" in some form. It's some bizarre rationale about returning to the mothership when they were actually jettisoned. Don't get me wrong, working at IBM in the 90s was fascinating but a lot of them only had careers there.

I've known and know a not insignificant number of IBMers who were there on the order of 30 years. I've tended to stay in most jobs a fairly long time (decade plus or minus) but nothing like that.

A former manager at another fairly old-line company, it will presumably be the only company he will have ever worked for--albeit through a couple acquisitions and some stuff during dot-bomb.

A lot of this mentality has to do with wanting a throat to choke if something goes south. The bigger the throat, the more the C-Suite rests easy, despite the costs. The old "Nobody ever got fired buying IBM has morphed into nobody ever got fired buying Microsoft/Oracle/AWS/whatever the flavor of the day."

I've worked in places that refused to allow us (IT) to write programs to automate processes that would have saved enough money yearly to buy a house. They were "severely concerned" because "if you leave or get hit by a bus, who will maintain it?"

I work at a place now where I can automate away with permission and prototype to show use case. I've been at this job a year and have already automated a goodly portion of the grunt work. Some of my colleagues look askance at me, and one has said, "you're automating us out of jobs, eventually, you do know this, right?" These guys/girls fear AI. I don't. I don't use it because God gave me a brain and I'm expected to exercise it. AI also feels like cheating to me. Sure, I may have to read the docs more, hit up someone on Stack Exchange, debug my code a little more, but you know what? I enjoy the challenge. I'm basically getting paid to have fun, despite the daily grind.

> A lot of this mentality has to do with wanting a throat to choke if something goes south.

You're absolutely right about this on many levels. I also heard that quite a few times from my CEO, even though we often went with single smaller providers for things because he also wanted to cut corners.

In this case though, I'd say it was more about nepotism. The CEO wanted the deal with his old pals in exchange for access to the IBM sales pipeline so that we could sell their customers on storage deals on our hardware deployment. The only way to get access to the deals was to buy IBM hardware that was extremely over priced and far more "capable" than what we actually needed.

The fact that their sales teams outright lied about things they didn't even fully understand, was sickening for me. Thankfully other people on my team stepped up and helped prevent anything from moving forward.

I agree with you, automation is key.

>Some of my colleagues look askance at me, and one has said, "you're automating us out of jobs, eventually, you do know this, right?" These guys/girls fear AI. I don't. I don't use it because God gave me a brain and I'm expected to exercise it. AI also feels like cheating to me.

I'm... Not seeing the logical connection here. If you could automate more of the work faster using AI than not using AI, why wouldn't you?

I got into IT because I want to be challenged. Asking some generative AI bot about how to write a better hotpatch function or how to automate a terribly-detailed set of steps that require two people would defeat my own learning. I'd rather take 3 days to sort that out than solve it in minutes with AI. I get better by slogging it out and the mistakes I make help me avoid future mistakes.

I don't want the answers, per se, I want guidance. AI would solve the issue for me. I don't want some bot thinking for me. I'll retire when I cannot suss out how to write code that solves problems or makes life easier.

And yes, I'm one of those holdouts who has never had the desire to even test out generative AI. I'm not worried about it supplanting me. I worried that I will lose my own edge and curiosity.

> Asking some generative AI bot about how to write a better hotpatch function or how to automate a terribly-detailed set of steps that require two people would defeat my own learning. I'd rather take 3 days to sort that out than solve it in minutes with AI. I get better by slogging it out and the mistakes I make help me avoid future mistakes.

That is a good answer. Certain kinds of tools can become a crutch that limit you and stifle your own development. Maybe that's tolerable when it's not a core skill (e.g. never learning your way around your city because you only know how to react to GPS prompts), but when you're talking about a core job skill, it's a recipe for disaster (e.g. letting your skills atrophy or not developing them in the first place).

While I think there has been a lot of hype in generative AI and expecting it to replace programming as we know it is premature at least, programming has always become "less challenging" over time due to improvements in technology. The famous "Story of Mel" was about a programmer in the late 1950s refusing to use those newfangled optimized assemblers rather than writing machine code directly. Then there were compiled languages like Fortran and Cobol that were far easier than machine code/assembly. Then higher level languages, etc.

Same with (after HLLs, and in rough historical order, over a few decades), 4GLs, CASE tools, OOP / OOAD / UML / round-trip engineering, NoSQL, Functional Programming, what, not, ML, AI, ... ;)

What is that French quote about: the more things change, the more they stay the same?

Or that George Santayana quote.

https://en.m.wikipedia.org/wiki/George_Santayana

> Or that George Santayana quote.

Which one?

Good question, actually. Thanks for asking.

I initially kind of assumed that most people would know about it, because it is a somewhat famous one, or at least, I've read it in many places. But your question made me realise that may not be the case. So, here it is, from the top of the above Wikipedia article about him, just below the header:

[ Santayana is popularly known for aphorisms, such as "Those who cannot remember the past are condemned to repeat it" ] .

Yeah, I wondered whether that might be the one. But it's slightly different from the plus ça change, plus c'est la même chose quote, so I thought maybe you had a different one in mind. (Santayana was known for his aphorisms.)

Yes, they are two different quotes. I didn't mean to use them as equivalent. It was more like, I used both, because both were applicable in the context I used them in.

>The famous "Story of Mel" was about a programmer in the late 1950s refusing to use those newfangled optimized assemblers rather than writing machine code directly.

IIRC, K&R said something similar in the first edition of their C book (referring to others' imagined reactions about the C strcmp function:

"What, you mean I have to call a function to compare two strings?"

They went on to add that they had tried to make C function call overhead quite low, to handle this issue.

As an alternative, you can seek learning and opportunity to learn through other means even when assisted by AI.

For me, AI assistance has simply sped up the code I already know how to write (like structured test setups etc.) so I can focus more on the problems related to the business, which AI has no clue how to solve.

The way I use genAI, is exactly as you describe - it's my ultimate rubber duck, one that's actually smart and can talk back.

I use ChatGPT as a study partner in learning new direct-skills, as a soft-skills partner in navigating business dynamics, and a strategy partner in product work. It's very good at helping me learn to think better.

GPT 4 is basically useless for outputting the type of code (or writing) I need on a regular basis. It just doesn't have the right context to even be a viable tool beyond boilerplate/bootstrapping.

> I've worked in places that refused to allow us (IT) to write programs to automate processes that would have saved enough money yearly to buy a house. They were "severely concerned" because "if you leave or get hit by a bus, who will maintain it?"

I mean... they kinda have a point don't they?

I was the sole maintainer on an app used by a Fortune 500 company, and if I were to get hit by a bus, it would've been very difficult (possibly impossible) to support it.

There was very little documentation, because nobody but me was maintaining it.

> it would've been very difficult (possibly impossible) to support it.

People are reverse engineering binary blobs, I’d say your software can be deciphered. It would be an hassle, but doable.

But you wrote the docs and documented it’s place in the enterprise knowledge graph, because you’re a professional, right?

As an eng manager at an enterprise (not SV style tech) I differentiate between coders and developers.

Developers help our business by actively taking part in shaping software, automated workflows and integrations that makes us improve as a whole.

At lot of time is spent documenting and figuring out how that piece of software someone wrote 7 years ago, before us, works and what integrates.

This is why I firmly believe you need to tie documentation to running code in some way - at the very least a basic service graph that contains all services, dependencies and some metadata.

Tie this database to automatic firewall and proxy configuration and you’re on a good track.

We’ve on the other hand taken the strategic decision that tech and IT will help us win (is it the 90’s all over again?!) and we staff as such. A team is at least two, and no individual can own a service.

best CTO I ever had was someone who was honest about churn: people leave, for multitude of reasons, that's why they stressed documentation and testing, so others can pick up where you will inevitably leave off, eventually for good. It was baked into the culture that hand off was going to happen eventually

Not only one throat to choke, but most C-Suite had direct or indirect experience with what the old IBM (not sure about the current IBM) could do in those infrequent situations with high-impact, high-visibility, nasty technical problems. IBM had a well-defined process in place to handle these when they occurred. If the local team couldn't fix it, they could declare a "critsit" (critical situation) that launched a process that allowed them to pull in amazing resources and bypass red tape.

In one that I had direct experience in, they flew in one of the software's developers to do onsite troubleshooting. Also, being the hegemon of the tech world, IBM often would take the end-to-end role and not just stop at their boundary of a tech problem. I was involved in one in the late 1990's where we were running Netscape's iPlanet web server on an IBM JVM (on IBM's AIX) and it had a nasty memory leak in production where volumes were higher. We called Netscape and got what felt like a guy with a pager on the beach (it was outside business hours when we called); not helpful at all. We called IBM and things started to rock & roll. Eventually we had someone from their UK lab tracing the JVM until he found a JNI (Java Native Interface) call from external C-code (which would have been iPlanet's code) that had the wrong value for the flag that indicated who (JVM or C-code) should release the memory. He could have stopped there saying "not IBM's fault" but he just kept going reverse compiling the C-code to find the bad call. We were able to get back to Netscape saying in module yadayada.c, somewhere around line 47, there's a JNI call with flag=0, change it to flag=1. The patch from Netscape fixed the problem.

These infrequent critsits are something the C-suite remember for a very long time. They know that no way in hell their own staff could resolve these situations, especially in the days before the web, Stackoverflow, shared-knowledge, etc. If not for IBM squashing the bug, their bacon would be fried giving unsatisfactory explanations to their CEO.

The worst quality software I ever worked on as a software developer was an unholy mess of baldy indented, copy & pasted PL/SQL, spaghetti code JSPs, and dubious code generally ... written by IBM contractors (around Y2K timeframe). Luckily the CTO was smart enough to hire our own team to build and maintain it (and eventually rewrite it) rather than have IBM maintain it for $$$$.

I once saw the original architecture document and it was actually very nice -- though perhaps unorthodox. The implementation was like an encyclopedia of Worst Practices.

I also worked a summer for IBM Global Services doing some sysadmin and security work back in 1997... I'd characterize the whole thing as... mediocrity.

Another time, I worked a contract in another part of IBM and saw other parts and got to attend presentations and lectures there and see the patents on the walls, and be exposed to some of the really smart people and history there, and that product side of the organization (people who worked on DB2, etc.) seemed much better.

I’m old enough to have encountered Netscape contractors. The kid they sent to us, at $250/hr (that’s $470 inflation adjusted) no less, had the manual open in his lap. I’m pretty sure they were taught never to get caught doing that.

If memory serves I helped our IT guy replace the whole mess with Apache, so we’d never have to talk to them again. I used to compete with Netscape so there was no small amusement in costing them contractor hours.

> I’m pretty sure they were taught never to get caught doing that.

I've long ago learned that people misremember all the time. I trust the person who quickly looks up the correct answer in the manual over the person who thinks they din't need the manual and uses what they think is right. Sure often the second person gets things right, but when they get it wrong they waste a lot of time trying to figure out which thing is wrong.

Yeah. It's one thing if they obviously don't have the slightest clue. But if someone sort of knows how to do something or even thinks they know how, I'm not going to begrudge them checking a manual.

Opening a book to check something is very different from rearranging yourself so you can keep it open in your lap while you type.

Nothing they were doing was worth $250 an hour.

> The kid they sent to us, at $250/hr (that’s $470 inflation adjusted) no less, had the manual open in his lap. I’m pretty sure they were taught never to get caught doing that.

In my very first consulting gig, my employer intentionally buried all of the best documentation on an internal forum. I can't even count all the hours I burned up on troubleshooting, only to learn days later that the thing I was encountering was a known issue that was completely undocumented and the only way to know that was to be present in the office... which was 3000 miles away.

I tried to get management to open up the forums to public consumption, but they were dead set against it, said that their "intellectual property was too valuable."

You point out Oracle as akin (but less entirely awful). I'd throw Palantir on the pile. Sell management on bullshit, then, once they've got their hooks in, it turns out they'll need to be way up your ass (and billing the whole time) to make any of it happen, and the entire process will be miserable. Another candidate for "begin insurgent activities as soon as the name is overheard" responses.

Who would be a good name to suggest? I can’t imagine SAP would do a good job at it either.

AWS I suppose

Versus Palantir, I think was the intent of the question. IDK, from what I've seen they can't really do much more out-of-the-box than lots of other less-bullshitty data platforms, they're just far more aggressive about selling "consulting" services to get you all the features they told you they could do (and they can—with a bunch of custom development, just like any other platform)

I wouldn’t suggest a competitor in that line of business, because it’s not a line of business which should exist: fundamentally, it represents a firm wanting to bring in another firm to engage in its core line of business for it, because it no longer knows how to.

Well, if the firm doesn’t know how to engage in its core business, just fire the employees, close the offices, take the names off the buildings and return the capital to the owners!

How could anyone who has read or seen The Lord of the Rings think that a Palantir was anything but an existential threat?

Not all of the lost seeing stones have been accounted for. We don’t know who else may be listening.

> How could anyone who has read or seen The Lord of the Rings think that a Palantir was anything but an existential threat?

If they want to be Sauron.

That's Palantir Technologies' customer base.

Making thing simpler is hard.

Any technical excellence at IBM is probably an island and definitely doesn't represent the rest of the company.

In my time there, only one thing mattered: head count (or more precisely, body count). It didn't matter if those heads could think. It was all about closing outsourcing contracts and making a profit. With the occasional sprinkle of Watson propaganda sprinkled on top to make it look like you were at the pinnacle of computing.

I'm surprised IBM still exists. I can't remember a single company that I worked for where IBM would have helped anything.

IBM isn’t just accidentally leaving things complex, they sell tools that fix problems almost nobody has. What’s that XML RPC gateway they were peddling years ago? Jesus Christ just use nginx and learn to manage certs.

'Fond' memories for monstrosities like Websphere or MQ servers. Take trivial JMS or less-trivial-but-grokkable J2EE spec, add 10x more ultra-proprietary stuff that is very complex to grok compared to original spec, solves little on top in real world, makes you utterly vendor-locked in for every app until complete rewrite. You need additional Eclipse-based apps to work with it at all, and IIRC they were costly as hell too. Rinse and repeat.

Then there was Weblogic, that worked as well and mostly by standards. Devs I met everywhere loved it.

> Take trivial JMS or less-trivial-but-grokkable J2EE spec,

MQSeries predates Java itself and started out on MVS; it was already the way it is when the JMS spec was developed. There's no real excuse for WebSphere, though.

From painful personal experience there are about 3 people in all of IBM who understand PKI well enough to configure a server.

Maybe it's because I have very good spatial intelligence, but I always found PKI to be a lot simpler than most people made it out to be. People treating me like some sort of priest of the church of crypto when generating and signing keys is really not that hard.

PKI is like a kitchen knife. The hard part is not getting one, or picking it up, it's using it without cutting yourself. But people treat the 'knife' like all aspects are magical.

I've found it ok to understand maybe 70% of PKI, but that I always need to review that last 30% (OSCP stapling, cross signing certs, whatever the heck the formats are, different extensions, more that I'm forgetting).

I've put my favorite learning resources and tools into https://www.bbkane.com/blog/learn-ssl/

I was at IBM for about five months, a few years ago.

It took from ~2 weeks after my start date until ~1 week before my last day to get a vSphere cluster provisioned for my development work.

PKI? Sounds hard, by comparison.

Probably outsourced internal dev environments to Kyndryl too?

Surprisingly no, not for this project. It was an acquisition, and the developers had come with - so had their data center. I was hired to try to help move it out of their on-prem location. They'd hired me specifically saying it was going to AWS; then they went "whoops! all IBM Cloud" a couple weeks in, and they never actually got me a fully working environment to build against in the few months I was there. (The one nice thing I'd say is that teams from all over IBM were incredibly helpful; some externally-facing consultants actually volunteered a couple hours to try to help undo whatever Softlayer had open-palm slammed onto my virtual machines. They too were confused.)

LLMs are sure going to make the Watson advertising look outdated.

IBM is offering a Cobol to Java migration service and it is based on LLM (watsonx). Will it save them?

Years ago I worked for IBM, and have dealt with them in various contexts since.

The thing I've found that best captures the essence of IBM is how they love bringing *everyone* to every meeting, and then insist on full introductions. Every time.

I've been on a few different projects where we would routinely chew up 20 minutes of a 30 minute call hearing every last one of them announce their name and role. Thanks folks, I already know who you are. Can we talk about the project now?

On the clock of course.

My last interaction with them, I remember they kept claiming the sky was falling (aka schedule slip) if we didn’t rework the library to do exactly the API they worked out before any line of code was written and with no input from the four other teams that needed to consume it.

This culminated in 2 of our leads sitting in a room with a giant table, in a meeting of what had to be ten people at least, for three hours one afternoon while we dictated pseudocode and sequence diagrams for their next three months’ of work (if you can describe a month of work in an hour, it’s really not as complex as people are making it out to be)

So we paid six or seven of them to listen while we did half of their job for them, with our two most expensive employees.

This made me laugh hard. When I worked at Oracle they’d do this in internal meetings if it was outside your immediate team! A dozen people saying “I am a System Analyst II for Managed Services” like goddamn who fucking cares

I had a boss who was making giving me a lead position into a bigger deal than it needed to be. Like he wanted me and one of my peers to be secret leads before he announced it formally. The entire situation was odd.

I told him I didn't care what he called me, what mattered is whether people would listen to me when I asked them to do stuff, like use a better solution to a problem. If you want to be a lead, you have to do a lot of that yourself. But I was young (1st lead position) and his lack of backing that up were making it all more complicated than it needed to be.

Made all the weirder by the fact that he was good at selling people on ideas, to a fault (it took me 3 tries to quit that job, he was such a smooth talker)

Another story for that same project: this was a program where being two weeks behind schedule resulted in theatrics. Think English Parliament in period pieces, but toned down and with more ties.

That same other lead and I had figured out that IBM was perpetually 5 weeks behind, and always fishing for problems on other teams they could use to invoke day-for-day schedule slippage.

And the reason we knew it was five weeks is this: they would not take patches or hot fixes from us. Because doing so would indicate that we hadn’t fulfilled out contract to deliver on X date. So I’d give them version 1.0.4 (had to be exactly 1.0.4, if the CI/CD pipeline glitched and I gave them 1.0.5 they would demand an explanation, fuckers). And then I’d go off working on catching up with automating functional testing that we were still figuring out, trying to get it to run faster on one particular underpowered destination system, or get feedback from other teams about how some feature in the release was fundamentally broken, and I’d fix those problems and issue 1.0.5-7.

I’d tell them they needed the patch and they would say no. Then eight weeks after we gave it to them they’d complain it was broken, they are blocked, and the schedule will slip “day for day” until we fix the bugs - often bugs they should have seen week one if they were actually using the deliverables we gave them.

In one case it took me about 14 clock hours to fix their problem, but in all other cases I had already fixed that bug almost exactly six weeks before they found it. I just informed them that 1.0.7 already fixes that, upgrade. They were hoping for a week and every time they got less than a day, and only then if the meeting schedule aligned so the could stall for half a day. That was honestly one of the more satisfying aspects of that job.

Ten times they came at us with that schtick, and it worked once. Barely.

Near as I can tell they were spending a month at the beginning of each milestone cleaning up tech debt from stapling together the feature set for the previous milestone, and not getting around to our stuff until much later than they wanted the company to know.

https://en.wikipedia.org/wiki/Schedule_chicken

I am amazed there is an wiki article for this.

Yes. There was a ton of this going on as well.

We were almost always the first to say that we needed an extra week. Tons of harrumphing. We were, however, never (maybe 1 exception) the last team to ask for another week.

The thing is when we said we needed 1 week, it meant we needed 3-6 business days. When they said 1 week, it meant they were too scared to ask for three.

The inanity of the whole thing is that the project we were on was behind by a year that people would admit to (it was more like 30 months), so who the fuck cared if our division delivered our part three months late? It was going to collect a lot of cobwebs before it got properly exercised.

Bunch a drama with absolutely no purpose whatsoever.

This really hits home. It was absurd. 12 people on a zoom call, 1-2, at most, speak beyond the introductions.

> This really hits home. It was absurd. 12 people on a zoom call, 1-2, at most, speak beyond the introductions.

For a while there, I had a weekly meeting at a former employer that had 63 people on it. The goal of the project was to change the IP address on an Oracle server.

Nobody wanted to take responsibility for the change, because if it broke, they might be unemployed.

So it just dragged on, week after week after week.

I honestly don't know how you can work in an environment like that, it sounds like my idea of hell.

I once had a contracting job where nobody wanted to update-and-reboot a server because they were afraid of getting fired. As a contractor who already had another job anyway, I had no such fears. The server had been running for several years with no patches or updates. You'd think someone would've been fired for that?

Anyway, the "job" involved more meetings than actual, real work. I'd guess 4 hours of meetings and status updates for every 1 hour of work. I don't know how some places even function.

> Find clever people, and have them strain themselves to stand up a system so complex that you not only need to keep paying GS to show up and maintain it, but in fact you probably need to hire two more people just to keep the wheels on.

AS/400 can still be found in the checkout line at Costco. I don't think it is the main POS system but is used for things like looking up customer information. IIRC Oracle is taking a lot of that retailer backend business now.

I have a feeling that IBM, including all those pre-80 behemoths, are more like civil servants -- they always remind me of Sir Humphrey with tons of formality, planning and such. Maybe it's because they hired from academies heavily and actually did work on a lot of government projects?

Since this is the real "engineering" spirit that HNs love, I'm not going to argue. But I definitely prefer SpaceX' run and break things spirit.

Reply on news.ycombinator.com