Hacker News Reader

Comment by jacurtis | Hacker News Reader

It is also worth noting that Raj wasn't alone in the confusion on how to perform this task. Before he could complete the transfer his boss in India also had to look at what Raj did. Then their boss in Deleware also approved the transaction.

ALL THREE PEOPLE INVOLVED IN THIS TASK WERE CONFUSED by the interface and all three thought that this was the correct way to do this.

This wasn't the case of one person checked the wrong box. Everyone who looked at this was confused. This problem was not individual, but systemic.

View on HN

Totally NOT SURPRISED that this was an awful Oracle enterprise app.

These things were put together in haste in the 90's and Oracle keeps pushing them and their corporate customers keep these things going long past when the interface becomes utterly unfamiliar.

The blame has to be shared, however, with the battle-axe personalities of the people that use this stuff. They would like nothing more than to retire while still using the _exact_ same applications and dated interfaces they've used for 20+ years. Literally, these are default "grey-theme" java applications, with all kinds of inconsistent UI quirks and absolutely no shits given for discoverability or even being able to create hyperlinks so that people can share stuff without screenshots.

It's fine if it's the same people keep using these apps forever, but the problems start when new people get on-boarded. If it's like most places, the newbies will be given no training and the only help they will get would be from some dour in-house oracle-application jockey whose been doing the same thing for decades. That's when people make very, very expensive mistakes. We hear about this one because the sum was vast, but people make $10,000 mistakes on shit software from Oracle all the time-- that's partly why Oracle has a very deep bench of retained lawyers, I suppose.

I have to say Citibank deserves it.

It blows my mind that Oracle gets paid huge sums of money for the garbage they put out. I went to a CUNY school and they rolled out a "new" administration system called CUNYfirst. Literally a several $100 million dollar contract and Oracle provided a slightly modified version of Peoplesoft. The UI looks like something that should have never existed on this earth. Every student gets an Employee ID, you choose classes for a semester by adding then to a shopping cart and checking out. Minimal effort from them for maximum cost.

I went to QCC from 98-01 and you'd sit in the admin building on an IBM green screen terminal and setup your schedule. There was little to no prerequisite checking so I made all sorts of goofy schedules and took classes out of order. I don't remember having any issues with it as the interface was pretty strait forward.

I wound up going back several years later when I was thinking of a different field and they had that trashcan CUNYfirst system. By then the UI was already outdated and clunky. I wound up with the wrong class the first time around because of some issue and have to have it changed post payment which was time consuming. I would have rather used the IBM's.

That’s aggressive sales department, make no mistake about it.

It’s actually easy to sell shit software. The average person knows a lot more about cars, for example, thank they do software. I’m sure Oracle makes huge money on name alone. The IT manager doesn’t want to take a chance with some lesser name. Just get Oracle, then nobody can blame them when shit hits the fan.

I would be more inclined to blame the buyer, not the seller.

> Just get Oracle, then nobody can blame them when shit hits the fan.

Only incompetent people wouldn't blame the IT manager for choosing Oracle.

It's the old big business joke. "No one ever got fired for buying IBM". You can easily replace Oracle in that sentence.

yeah, a shopping cart existing on a webapp where you are not purchasing product is such a red flag

I absolutely loathe any time I see "shopping cart" in an app that isn't me buying something.

When you see that, it’s clear a system/framework/etc. is being used for something it was not designed for

In college, we had a two-week stretch at the beginning of a new semester when you could audition classes and add/drop them at will. This was called the "shopping period", so in my view, the shopping cart analogy on the registrar's web site doesn't sound too out of place here. Of course, the logic behind it would need to be different from an off-the-shelf e-commerce cart.

Eh, not necessarily. There might be a thought on the UX team that shopping cart is something anyone can relate to, so they use that as the metaphor.

Exactly. There are folders and a trash can on my computer desktop for the same reason.

Yes, but to me, a “shopping cart” implies a purchase. It’s what I use at a grocery store to put my food and stuff in. A “cart” would be better term IMO despite just being a shorter form of “shopping cart”. But idk.

But you signing up for classes is you buying something.

I used a class registration system that wasn't this and didn't mention a "shopping cart" metaphor anywhere, but the system was exactly the same. Why object to calling it a "shopping cart?

But in the case of university classes, you are in fact buying them, it ain't free. So the shopping cart actually does make sense even if its a terrible use of another use cases' interface.

The University of Wisconsin used (and still uses AFAIK) this system as well. "Shopping" for classes was horrendous, and there were at least three interfaces to search for classes.

I can only imagine how hard it was to populate the available classes. I wouldn't be surprised if part of the huge administrative bloat in the various departments was people who's only job was to force data into and out of PeopleSoft.

10 years ago I did exactly this for the University of Wisconsin. My job was entirely to facilitate data extraction out of oracle databases / PeopleSoft for longitudinal studies done by faculty or other interested parties.

There were a few dozen of us spread across the UW System who's sole job it was to extract and massage data from these databases... One, very large, central database that handled data for UW and every satellite college they had.

I have to use an Oracle product for timesheets and besides the UI which is as you can imagine reading the previous comments, contains javascript code for a 'finite state machine for controlling the notification popup component'... [edit: typo]

As shitty as it is, do you think that the CUNY school has the ability to write a spec for an administration system that handles all of the business needs and compliance requirements? Or do that and manage the delivery of those requirements?

The answer is of course no.

The Oracle pitch is "We do this for Michigan State and can do X, Y and Z". That is true.

Oracle gets alot of shit for this stuff because they own the market for enterprise financial software. Their software is gross, they bought out the competition and generally suck, but that is what the market demands.

My uncle is a locomotive engineer. His railroad makes a conscious decision to remove any creature comforts from the cab, like a chair cushion. It's needlessly uncomfortable and is awful, but that doesn't mean that GE (or whomever) sucks. They delivered what they are asked to do.

The scalability argument of the solution is becoming less relevant with readily available cloud architecture, no?

The school cannot handle the requirements or implementation. Of course, but maybe a smaller, more focused provider than oracle can?

And yes, you have to be at peace with your data being god-know-where.

To be fair, comfort can be a mixed bag for system safety, especially when humans are alone, safety critical and on weird schedules which can all be the case for train drivers (which is what I understand "locomotive engineer" to mean in this context) - because comfort is conducive to sleep.

One of the setups I come to almost expect in maritime accident reports is an officer taking a night watch, alone, while somewhat fatigued. Warm, dark, quiet. After a few hours the ship has drifted significantly off its intended course. For unknown reasons (the officer was probably asleep) no attempt is made to correct... When they do wake up, they often don't have good situational awareness, and their focus is on pretending nothing is wrong, even if they are in fact already in serious trouble and need all the help they can get.

Night-time railway accidents are less common in my country, perhaps because the relevant unions insist on proper "if you're too tired just go home you aren't fired" type rules to reduce fatigue errors, but I do recall a case where a freight driver was dipping in and out of consciousness. When he was conscious he could see a red lamp out of the window, signalling "Danger" ahead so he'd reduce the traction power. If he'd been less sleepy he'd have noticed those red lamps were not getting closer each time, they were moving gradually away from him. His train was on a steep climb, and he'd reduced power so much it was gently rolling backwards. Oops. Nobody died, that time.

> do you think that the CUNY school has the ability to write a spec for an administration system that handles all of the business needs and compliance requirements? Or do that and manage the delivery of those requirements?

WTF are the administrators paid for if not that? Isn't that literally their whole job?

Agreeing to bad requirements makes you a bad business

You can reference the Texas power outages. Just because the requirements were to fail in cold weather, doesn't mean it was a good choice

They ... couldn't even relabel the shopping cart and other stuff? Like, call it "courseload" or something?

When you've finished adding classes, would you know to click the "courseload" button in the top right to complete your enrolment? Probably not.

It's lazy UI but in some ways it's going to be more intuitive, not less, to any student who's ever used an online retailer before.

That's just an example, there are other ways to indicate that they need to click on that button to review all selections and finalize.

Yep, I had never heard of it but recognized it as an Oracle app immediately. Not surprising at all.

Oracle will audit them and find that someone accidentally enabled an extra, unused feature during an install one time. So, there goes another 10 million.

> These things were put together in haste in the 90's and Oracle keeps pushing them and their corporate customers keep these things going long past when the interface becomes utterly unfamiliar.

Actually the history of this app is a little more complicated. In the early 1990s, Citibank decided to establish an outsourced software development group in India to rewrite their banking software from scratch–they established it as a subsidiary company called iFlex. From 2005 onwards, Citibank progressively sold a majority of its stake in iFlex to Oracle, who then renamed the company Oracle Financial Services Software (OFSS). OFSS is listed on the Mumbai stock exchange, but Oracle owns the majority of its stock.

So, rather than being a product which Oracle developed themselves and sold to Citibank, this is actually in some ways the opposite – a product Citibank developed themselves and sold to Oracle.

I used to work for Oracle. I never worked on OFSS software directly, but I did work on an implementation project for some of it. I never saw the internal banking UI, all I ever saw was backend stuff – LDAP, BPML, JVMs, databases, load balancers, firewalls, etc. Indeed this article is the first time I've seen that particular UI in my life. But I can tell from the visual appearance that it is not one of Oracle's more recent UI development frameworks. (I don't know if this is because Citibank is running an older version, or if there are parts of the product still running on a legacy UI framework.)

Oh wow - gotta wonder how this "arms length transaction" got past any internal auditors or technology officers, or if no one was looking. (Then again, did Citibank have a CTO in the 90s?)

Citibank developing in-house software then selling it to Oracle so that they can charge recurring licensing and support fees for eternity is incredibly short-sighted if you're Citibank or Manna from heaven if you're Oracle.

My guess though is the Citibank executives who signed off on this got promoted up and out of the department that has to budget for these licensing fees and support costs, and on both sides they received tidy bonuses for "increasing revenues".

The software fees are likely reduced/free. I've worked at companies that have sold in house software to other companies to make them public. It's usually resulted in lowering in house maintenance costs and the licensing agreements have a sweetheart deal for the original company.

Usually it's a win win, unless you value keeping control (which can be quite valuable)

> Usually it's a win win, unless you value keeping control (which can be quite valuable)

My experience is that it's usually overvalued. An external, focused company will usually do a better job than an internal project. It's emotionally hard to give up control, but successful companies tend to focus on a few things, do them really well (areas of competitive advantage), and outsource everything else.

A best-of-breed external solution almost always beats an in-house solution.

I'd just never do something like this with Oracle. Their business model seems to consist of locking companies in, and then milking them.

Why would the arms length principle be a concern here? It’s generally only a concern in regards to transfer pricing of transactions between entities that have common ownership/control.

> My guess though is the Citibank executives who signed off on this got promoted up and out of the department that has to budget for these licensing fees and support costs, and on both sides they received tidy bonuses for "increasing revenues".

Oracle paid Citibank over US$500 million for this company. There is no way a transaction that big isn't being approved by the CEO and board of directors of both firms. It wouldn't be under the control of the software licensing department. It would be controlled by the business development group in charge of M&As and divestments.

> Citibank developing in-house software then selling it to Oracle so that they can charge recurring licensing and support fees for eternity is incredibly short-sighted if you're Citibank or Manna from heaven if you're Oracle.

I have no internal info on the details of the Citibank-Oracle relationship (my role at Oracle was technical, not the kind of role where I would know that kind of business relationship stuff, and I never worked on the Citibank account either.) But, as @dagmx has already pointed out, I think it is highly likely due to the history of this software that Citibank has some kind of special licensing deal with Oracle which protects them against that sort of thing.

> Oracle paid Citibank over US$500 million for this company.

Maybe they saved the US$500 million they were paid for this software, and they can use it to cover the US$500 million it just cost them.

Even if AaronFriel's assertion is true and they received no special deal, it's a naïve view of corporate governance and SAAS v.s. in-house devleopment to say that (as an example) if their license fee was high enough that they'd start incurring a net loss in 10-15 years that they made the wrong move.

For Citibank that was US$500 million then, which they could either invest directly or return to investors. The opportunity cost of that cash has to be factored in.

It's also an a large ongoing liability to have a sizable in-house development effort when it's not your main business. Departments need to be staffed and run, business plans made etc.

For all of Oracle's flaws they're presumably going to have an incentive to improve the software, and more capital with which to do so from clients other than Citibank. For obvious reasons other banks would be more inclined to buy from Oracle than a direct competitor.

There's also often tax reasons for why it's preferable to buy a service v.s. maintain in-house software, and naïve back of the envelope math usually doesn't account for that.

On the other hand, they also sold one of their core competencies away (managing complex financial transactions) and it just cost them half a billion dollars and who knows how much they've lost since to other errors, let alone lost productivity.

> It's also an a large ongoing liability to have a sizable in-house development effort when it's not your main business. Departments need to be staffed and run, business plans made etc.

This is the same ideology that led to companies outsourcing everything from facility management and cleaning services as the first victims to stuff as critical as IT operations.

Yes, it is a liability and likely also a higher cost (e.g. due to collective wage agreements) to do that stuff yourself. On the other side, you have the large liability of having no direct control over staff and work quality any more - you're entirely at the mercy of your contractors (who are incentivized to find not the best people, but the people willing to be paid the least).

> ...no direct control over...work quality any more...

This is important. Just a few days ago here on HN, there was a good discussion on how deliverables quality on seemingly "mundane" device chargers (in this example, one that was arguably for early Kindles) can vary wildly, while maintaining the same price point, with the sub-standard quality yielding more profits to the seller, and higher-quality units yielding less profits.

This is a impedance mismatch between what the buyer knows and values and what the seller knows and values. With a time slippage between the time the buyer makes a decision and when the buyer's organization figuring out the real ramifications. By the time an organization has outsourced enough to lose competency to not even know the impedance mismatch and time slippage exists or scope it if they're aware of its existence, sometimes they will find might be cheaper to in-source in the first place. My personal rule of thumb going in to evaluate these decisions is if the procedural complexity surrounding such software artifacts rises above a certain level, it is likely better to keep it in-house. Where that level is lays the art of business, it is largely experience-based.

When I saw the screenshot of the app I first thought, "that's Swing". But then I saw that tick mark and I thought, holy moley oculd that be the tick mark from Borland (e.g. OWL)? https://sourceforge.net/p/owlnext/wiki/Replacing_usage_of_BW...

Most probably not Borland. That scrollbar looks like Swing.

Someone on ars said its Delphi. But I don't have a ton of faith in ars commenters.

That green check was in Borland Delphi too.

https://damieng.com/blog/tag/delphi/

Yes, it looks like OWL framework which was bundled with Borland C++. Google Image Search for "borland c++ owl" yields more screenshots in the similar visual style.

That scroll bar is an Oracle scroll bar - you can see it in the GUI installer for the RDBMS (and in all Oracle Java apps). The Borland-like checkmark button can be something Citi-specific or just a developer who liked Borland enough to import the graphic into the app's UI.

When the 3D look became the norm for Windows apps, introduced by Office and, later, by Windows 3.1 (or 3.11), I used the DLL to 3D-fy my Windows app dialogs automagically. I also took some liberties with using Office graphic elements reasoning that a Windows or Office user would know the 3.5" floppy saves things.

Yes, that's Oracle Forms, one of the older versions. 6, I think.

It certainly looks like Oracle Forms 6 (not sure if earlier versions looked similar, though.)

I can confirm it is OF. However versions 7 through 10(maybe 6 as well) all look like that so I can't say which one it is. I used to work in a company using OF for the UI and I can say that while it is horrific to look at, it had its strengths. However it was appropriate 20 years ago when monitors had standard aspect ratio and resolutions. Nowadays with scaling and all kinds of aspect ratio it just doesn't work. Also Java... Ironically for one client our software was interfacing with Flexcube.

However the OF technology is still supported by Oracle and while they themselves are trying to kill it, nobody is willing to migrate their core systems to something new.

Currently rewriting a bunch of stuff in Oracle Forms 6i to .NET 5, definitely looks like Oracle Forms 6

To me it looks like a victim of cost saving and overoptimization.

Somebody requested that feature, and then someone discovered that if you hack around your system and set precise combination of fields, it will do the trick. So man hours were saved, some memo added to docs, and then the knowledge was lost.

And all of that instead of adding separate form for specific case. I've seen it more than once.

Keeping this in mind for the next time someone asks why we need to reinvent UIs every so often, or why technical debt needs to be paid off. Because they tend to cause financial damage, that's why.

Oracle? I didn't realize that from the article. The entire situation makes A LOT more sense now.

I once was deposed in a lawsuit regarding Oracle, and part of it was basically a UI issue. We had a paid customized interface that allowed users to upload basically a blob text or document (word, etc). When we went to UAT, we didn't see it in the UI in the app itself (peoplesoft):

Us: "Okay, we can upload stuff, where do we find it?"

Oracle: "Oh, that wasn't part of the project"

Us: "Yes it was, see spec #<insert #>."

Oracle: "... Well, it's in the database. A DBA can access it, so we met the spec."

Us: "That is not a rational interpretation of '... and users will access it within the application...' "

Oracle: "We're happy to build that customization for another <insert massive $$$ sum>"

This was only one of a few material issue that led to the lawsuit. We probably wouldn't have sued over it alone, which should give you a hint at just how bad some of the other problems were.

As an aside, the $/hour billed for customizations was $600, but the work was also outsourced to India. I asked a PM from Oracle how this was justified, and they gave a vague answer about multiple layers of management needed to oversee the developers. I pointed out that this price tag would allow for 3 managers per developer, with each person in the chain making ~$200k/year, and still allow Oracle $200/hour profit. I got a ¯\_(ツ)_/¯ response. That I have to blame on the organization & not Oracle though. I had the detailed ~500+ page RFP response & implementation plan, which specified 1,000 hours of custom work. But I didn't have the actual contract to see what additional work would cost. The organization either didn't negotiate that in advance (so their fault) or they knew in advance & agreed to it (also their fault)

This sounds like this job was done by consultants (could have been Oracle consultants too) and this is standard behavior across consultants (that doesn't make it right though).

Oracle as a product company (applies to all Enterprise Product companies) will not charge separately for a customization.

I've worked with other consulting groups-- such as on the successor project after the dust settled on this one-- that were not nearly as bad, so I guess sleaziness varies significantly.

In any case, these were from the the actual consulting arm of Oracle inc. And IIRC, our Associate VP for IT said we shouldn't use both Oracle & Oracle Consulting, but that person was overruled by the VP for IT. I the VP was kept around until the lawsuit settled because it might have been taken as a sign or admission of culpability to fire the VP before that, but shortly after the settlement (which we won, incidentally) that person moved on for "other opportunities".

I personally despise Oracle for this. I've seen it time and time again and every enterprise I go into, Oracle is my first target for death. Almost all say "But it's critical" until it isn't.

The problem is partly a lack of choice in off the shelf products. In some industries the choice is basically Oracle, or a 3rd party app that also runs on an Oracle database. The alternative is developing your own system, which really isn't an option for anything but large organizations.

Things are a little better these days where you can often find apps (usually SaaS) that fill a specific need very well, but then you're usually stuck managing a dozen different products and a massively more difficult integration between all of them.

If you're using Oracle to power a home-grown system though, yeah: With a little effort you can swap the backend for something where the vendor won't show up with a surprise multi-$million invoice along with pre-made lawsuit papers if you don't sign it.

Absolutely. In trying to save time business wants an off the shelf product that Oracle claims they have. This isn’t a cost saving move (though I have encountered a naive VP who thought so), it’s strictly to reduce go-to-market time.

What really happens is entirely different. Oracle takes the request, foots the bill for an army of contractors to build it, if they haven’t already, and pass it off as we had it the whole time, just like we said.

Then up charge. Shake down street. Lock in.

I’m sure not all of Oracle operates this way, but that’s been my experience.

Homegrown systems are an easy low-hanging fruit to cost reduction.

Software to tackle the BPaaS on top of an Oracle add-on service, requires a pitch deck.

Oracle takes the request, foots the bill for an army of contractors to build it, if they haven’t already, and pass it off as we had it the whole time, just like we said.

That must be what happened in my case! I didn't realize it was a standard practice. A large part of the lawsuit revolved around functionality the RFP required, Oracle said they had, and that Oracle even showed in what may have been faked screenshots during our review of vendors. Only in our case, the army of developers never actually finished it and Oracle came back to us with another massive bill to complete the project.

I'm not sure I've ever heard someone who's worked with Oracle who didn't have a bad experience. But if you have and you're reading this, just drop a quick comment to say "yeah, they were okay"

I've had nothing but amazing experiences with Oracle. I can basically eat at whatever restaurant I want with our rep at any time, no matter the expense. I can get catered box tickets to any game. I get gift baskets that actually kick ass.

There was a complaining faction in my org, mostly disgruntled hold-outs from before my tenure. Whatever complaint they thought they had, I could bring it to our account manager, and it was pretty easy to see the root of the problem came from personal issues in staff, and insecurity with changing to better technology that they were not experts in.

Now that I've got mostly oracle certified engineers and oracle familiar managers on board I don't hear any complaints. It's a great product, and in all my future roles, transitioning to Oracle is one of the first things I'll do when I get on board.

You’ve been bribed. Everything you just described is unethical in my companies I’ve worked at.

Yeah, I think a number of people in California's Department of Information Technology had similar stories prior to 2010.

"Enterprise Software Consulting" is an accounting field that is at least as deep and deceptive as "Hollywood Blockbuster Accounting." :-)

My school had this system too.

The thing was, when it rolled out (2007), everyone was thrilled with it because it replaced a mainframe that you had to telnet into. That mainframe had hours of operation, i.e. you couldn't look at the course catalog at night because they shut down the mainframe to save power.

Oracle makes a ton of money because what they're competing with is just incredibly terrible.

The last college I attended also shut down their servers from 11pm-7am, so you couldn't make payments, check your balance, view your transcript, drop a course...

This was last year btw.

Was the implementation worth it? Did it works?

From what your describing it seams that what was missing was an API allowing browsers to get back the uploaded files?

Not a browser-based app. It was an extremely customized peoplesoft. And the whole project ended in a lawsuit. We had learned a bit, so we took our time to get our duck in a row, and it was about 4 years before we started fresh with a different vendor.

Was that project worth it? We didn't have much choice. Our old ERP was close to 35 years old, and no longer supported, so we were paying large sums of money just to get annual compliance updates for state & federal requirement. The new system is fine, does what it needs to do, but it's a monolithic ERP system: It does just about everything, but it's mediocre at all of it with most of it's tech perennially 4 to 7 years out of date with the best of breed. So, in the areas where we really need best of breed, we bought that and integrated it into the ERP, which is considered the "system of record". So it's fine, it works, I guess. The real kicker though is that the system we ultimately chose, while not an Oracle product, still required an Oracle DB. Fortunately we don't actually deal with Oracle as a vendor though. Our vendor is basically a VAR.

Are you sure they wouldn't like a new interface if it was actually a real improvement?

I'm reminded of a story[0] about Norwegian doctors refusing to use the new GUI system and stubbornly continuing to use the obsolete text-based system. Were they luddites? No. The old system let them quickly navigate with the keyboard in seconds while talking to the patient. The new system required them to use the mouse, so they had to spend a lot of time focussing on using the GUI, rather than helping the patient.

[0] https://news.ycombinator.com/item?id=10289172

> The old system let them quickly navigate with the keyboard in seconds while talking to the patient.

Encountered a similar issue many years ago at a company trying to sell a new ad booking system to Titan - they had crufty old VT100 based systems, the new one was fancy AJAX HTML whizzbangs. Exactly the same result - they could handle 5 bookings on the old system in the time it took to do one on the new (which was also terrible and ugly besides slow.)

Well, there's "green-screen" VT100 terminal apps, and then there's 90's Vintage Java/oracle-forms UI's and then there's modern applications designed with informed UX approaches.

The terminal apps, at least, have some history behind them and properly trained folks can be very fast with them.

But we're talking here about shitty early Swing-apps (or Oracle forms). If you've ever used one it can be incredibly frustrating. Moreover, people are usually just tossed into a role where they have to use these M-F-ing interfaces with no training. Who makes a table with cells too small to fit their content and where you can't resize the table? Oracle. That's who. And they're paid for it like you wouldn't believe!

>>> Who makes a table with cells too small to fit their content and where you can't resize the table? Oracle<<<

To be fair, lots of software of those years were like this. SAP Ariba was just like this, and so were old Microsoft products

Wife is a pharmacist, can confirm this is true. In school she worked for a chain where everything IT related was done on an awful MSDOS program- she would get calls from other pharmacists asking how to do something and I would hear her walking them through "hit ctrl-F12 then hit tab three times now enter the furble and then return then tab twice more and enter the gurble" and it sounded terrible. But if you had built up the muscle memory, you could do whatever needed to be done while being polite on the phone to an irate doctor who was super pissed that you were questioning their judgement in an effort to save the patient $300. Their new system was an internal web app that didn't do any of that.

Sometimes, a core job of IT is eating all the pain of using the old technology that the users already know so they can go do their job and the company can achieve its objectives.

> The blame has to be shared, however, with the battle-axe personalities of the people that use this stuff. They would like nothing more than to retire while still using the _exact_ same applications and dated interfaces they've used for 20+ years. Literally, these are default "grey-theme" java applications, with all kinds of inconsistent UI quirks and absolutely no shits given for discoverability or even being able to create hyperlinks so that people can share stuff without screenshots.

Here is the thing, the reason these people would want to keep using the software is... The replacement will be broken and they'll spend years with bugfixes and what not to get to the exact same position they're in just now.

Using old UIs isn't fun but if it works it's good enough. This is a case of everyone getting used to the workarounds until no one knew about the workarounds.

Dealing with Java interfaces for Oracle 18 years ago is why I refused to touch Java for my entire career.

This is precisely the story with the reporting software at the company my partner works for.

Everybody who was there when the software was created is long gone. It hasn’t been updated in decades. Nobody really knows how it works, and training is entirely folk wisdom and cargo cult practices handed down over generations.

Yep. We recently retired the Oracle reporting application at my employer. It was called "Discoverer" <-- not just "Discover" but really with an extra "er" at the end. Massive pile of shit that only a few people had the patience (or stubborn desperation) to master.

The people that did know how to use it got constant phone calls (yeah, PHONE CALLS, because that's how these people roll) to do ad-hoc queries for management. I guess I can only fault them so far. This skill put food on their table and kept them employed even though their end product was endless dowdy bar-charts and badly formatted tables.

Now we're using a new product. It's a "BI" tool.

So many corps are doomed to repeat the bad parts of computer application history over and over and over again. In the case of "BI" or "Business Intelligence", it got it's start when middle-management got tired of begging for reports that weren't baked-in by the application vendor years ago. They started with getting permission to directly query the database (a profound coup by their standards). When they found that it was _actually_ possible to get useful information from the database, it looked like "intelligence" to them-- hence the term "business intelligence".

Government uses Oracle based financial software too.

I have a soft spot for Citi because where else can you get 2% back on your credit card? They're not bad, they just like giving money away to their customers...

These 'double checking' things are like a worst case scenario of code reviews.

There's a psychological defect in the human brain called Anchoring. If you are first presented with the wrong answer to a problem, it affects your ability to objectively come up with the 'right' answer. You will not catch all of the garbage I threw at you because I threw it. "Double" checking is a misnomer because it's not really doubling the number of brains.

Some things should work more like nuclear launch codes. I don't approve your action, I perform the exact same action, and if we both did exactly the same thing, then something happens. If we don't, then nothing happens or you get an error.

Might this problem still have happened? Yes. But the first time it happened would probably have been taken more seriously, instead of what we usually do: deflect, deflect, deflect.

The local hospital handles the problem like this:

Doctor 1 gathers evidence and makes a preliminary diagnosis and writes it down. Doctor 1 then presents only the evidence to Doctor 2. Doctor 2 comes up with an independent diagnosis and writes it down. They then see if they match.

Bleeding person with white coat syndrome gets upset because two doctors and a nurse asked them the same questions.

I mean, it's better than cutting off the wrong leg, but boy is it rough.

Selecting which evidence to present leaves this process subject to some of the same errors. The evidence that leads doctor 1 to conclude the correct diagnosis is X is going to be passed along and other evidence may not be, so doctor 2 will diagnose the same. (This doesn't even have to be intentional. If I think I'm right, I'll be naturally inclined to provide evidence to that effect.)

Looking at the UI, anyone would be confused. I feel bad for the employee put into the spotlight. The spotlight should be on whomever designed and approved this software and UI.

Unfortunately because of the court case, instead, people associated are the users.

That UI is depressingly awful, and not having a confirmation screen with a summary of the transaction about to be initiated is so very very bad that it should get people fired.

Pretty much every modern financial app I've used has one of those confirmations - some better than others but all pretty clearly state "you are going to transfer $XXXX.XX to ACCOUNT_YYYY -- click here to approve transfer".

Even venmo's quick prompt to make sure you really mean to send your pet sitter $125.00 before initiating a transfer would have avoided this problem.

The thing is, this is where the difference is between internal and external tools. Your client facing app has to confirm every action because the client isn't expected to have a deep understanding of the process. Bank employees are expected to have that understanding, so I'm not surprised there is no confirmation window. It's like with Linux - certain completely destructive commands have no confirmation before running, because clearly the user knows what they are doing - otherwise why did they type in that command in the first place.

Since firing an employee won't bring the $500,000,000 back, I'm thinking the linux philosophy is probably not the best one to use here.

The linux approach is "what you are doing is not my problem, so go do whatever you want".

For a bank, what their employees do is very much their problem.

> Bank employees are expected to have that understanding, so I'm not surprised there is no confirmation window. It's like with Linux - certain completely destructive commands have no confirmation before running, because clearly the user knows what they are doing

Well, no, it's not like that at all. There IS a confirmation window; it's just not embedded in the software. Three different people -- the flunky, his boss, and his boss's boss -- were all required to confirm the validity of the transaction before making it.

And there was a confirmation window in the software, it just didn't distinguish between some and all of the funds being wired out of the bank.

> Raj then proceeded with the final steps to approve the transfers, which prompted a warning on his computer screen — referred to as a “stop sign” — stating: “Account used is Wire Account and Funds will be sent out of the bank. Do you want to continue?” But “[t]he ‘stop sign’ did not indicate the amount that would be ‘sent out of the bank,’ or whether it constituted an amount equal to the intended interest payment, an amount equal to the outstanding principal on the loan, or a total of both.”

(from the judge's opinion, via Matt Levine and Bloomberg)

Also confirmation is undesirable UX.

In this case it may be the best option you have, but the UX you should be aiming for is always undo.

Humans are always sure this is what they wanted to do right up until they understand the consequences. Then they experience regret. If possible design your software so that regret has a natural response in the interface in the form of an "Undo" option.

A confirmation step doesn't trigger that "understand the consequences" outcome, and so users will mostly be annoyed by the confirmation, and pick "Confirm" even when in fact they'll immediately regret that once they do it. They may even ask you to add another "Confirm" step. You want "Undo".

Notice the law here reflects this preference for Undo. If Citibank wired $500M to me the law says they get to Undo that, because they didn't owe me $500M and so that's just a mistake. They can't undo this because it looks exactly like a legitimate payment to a creditor, and if you could always undo those it opens a real Pandora's box. Normally you can "undo" paying a willing creditor because they'll just lend you the money again. But of course not everybody is a willing creditor, as in this case. Boo hoo for Citibank.

Right, but the desire to undo is incompatible with the desire to have fast transfers. Yet another solution to this problem would be to have transfers take 24 hours during which they can be "un" sent by the sender, but I imagine banks will have a very strong preference towards instant or very very fast transfers.

Not saying the Linux analogy isn't good, but this is more like if entering the flags for `tar` in the wrong order could send you and your immediate family into irreversible ruin forever.

It simply should not be possible, this is about literal tons of money, they shouldn't rely on something this banal.

Simple safeguards can be put in place to prevent this. They don't even need to be very fancy:

Fairly sure a simple trained-human-readable description of what was about to happen (think: this money will go to this acct, this money here and this other money over there) would have saved this people a whole lot of grief.

I saw a great Linux quote recently. I'm paraphrasing, but it goes like:

"Linux gives you enough rope to shoot yourself in the foot. If you didn't think you could shoot yourself with a rope, you should read the man page first."

That said, as Linux has gotten more popular, it's put more safeguards in. For instance, you can no longer "rm -rf / file.txt".

I know someone that works for a (smaller) bank regularly processing loans in the mid six figure range. When I think about all the steps she has to go through around verification on each loan, It's just crazy to me how they're able to accidentally send an amount over 1000x times what she has to deal with.

>ALL THREE PEOPLE INVOLVED IN THIS TASK WERE CONFUSED

I know you didn't mean it this literally but the issue is worse than that. They weren't confused. If they were confused, they would have sought clarification. Instead, they were sure they were doing the right thing.

I did a quick google search and looked at the Oracle documentation for Flexcube. While I'm not 100% sure it is for the same version of software, the documentation appears to discuss this point.

  Overwrite default settlement instruction

  Select the ‘Overwrite default settlement instruction’ check box to confirm
  that the liquidation entries should be posted into the Internal GL account. 
  System posts the entries to the Internal GLs only if you check this check box.
  Otherwise, system posts the entries as per the settlement instructions specified
  for the component of the contract.

It seems to me that the docs say when the user doesn't check the overwrite default settlement instruction checkboxes, the software will xfer the money according to the settlement instructions. Either way, for approving almost $900,000,000 I'd take a second look before clicking the approval button on a subcontractor's work. Heck, I'd even check when approving this transaction for the VP down the hall.

Would be interesting if Citibank could sue Oracle for the confusing design

I'd think they could, but that would bring up the question of how many transactions Citibank had properly run through the software.

I doubt Citi would want to pursue that sort of a suit, since it would lkely highlight they were at fault. Sure the software is probably difficult to use, but if they already knew how to use it, then using it incorrectly is not the fault of the UI.

Reply on news.ycombinator.com