Hacker News Reader

Comment by chadash | Hacker News Reader

Summary for those in EU: The Baltimore police department uses Lotus Notes software for keeping track of cases.

> Officers tap into its databases to track criminal investigations, check arrest data, log ballistic test results and identify “troubled officers,” along with various other uses. There are millions of records and roughly 150 databases built into the system, each designed to address different unit and personnel needs.

They started using LN in 1996. Now, all of their stuff is built on top of it and it's hard to switch. So they are paying $177K for the next two years of software support.

Summary over. Question of mine: is this a particularly high rate for two years of support for an entire police department? I can't imagine there are many specialists out there who know how to support lotus notes, let alone all of the custom data they have layered on top of it. And it seems like it would be a reasonable amount of work to maintain.

View on HN

For an enterprise support contract of that size it's dirt cheap. Try to build a system with 150 databases on a cloud provider or using your own cluster and you'll spend way more than that. VMWare licenses alone would blow that figure away if you rebuilt it in a modern style. If you were crazy enough to spec Oracle for those databases then it would be many times that cost per year.

VMWare and Oracle in the cloud? "Out of the frying pan, and in to the fire."

No different than Lambda and Aurora. Never marry yourself to the vendor.

You’re always married to your infrastructure. That’s like developers writing “repository classes” just in case one day they want to switch from Oracle....

Statistically no one ever switches out their architecture. The risk is to great, the reward is too small, and the number of potential regressions is too high.

How did all of those people who wrote thier website with Angular 1 fare?

Are you going to rebut all of my comments about risk management in this manner (antagonistic, usually vendor lock in, but it's been about other risks as they relate to cloud infrastructure)? Because I work in risk management for a financial utility, this is what I do for a living: ensure developers and other practitioners don't make poor architectural and technology decisions.

One of your own comments [+] exemplifies the consequences of making a poor technology choice ("Amazon has been trying to move off Oracle for 5 years and it’s still expected to take two more. Moving off of an infrastructure is hard......"), and I'm pretty sure the cost of that choice and the resulting migration to remediate is expensive AF.

[+] https://news.ycombinator.com/item?id=17729107

Was it a poor choice at the time? Apple decided to base the ITunes Store on WebObjects in 2001. They certainly didn’t have “vendor lock in” since they owned WebObjects, how does that decision look now in 2018 that they are still running on WebObjects?

You’re always stuck on your architecture and you’re always at risk of a technology becoming obsolete.

Look at all of the companies that are still stuck with an older version of the JVM because they are afraid to upgrade - and thier whole stack is open source.

Unless you personally audited every single line of source code your developers wrote, I guarantee you that your architecture is tied to some assumption that will make it hard to migrate in 5 years.

We audit every single line of code written by our developers. We audit every line of libraries they use. We audit their architecture, their infrastructure, and their plan to move from cloud to bare metal at anytime. Our applications run for decades at a time.

We automate wherever possible, of course, but there are some tasks humans are required for.

It doesn’t hurt for developers here to heed the warnings from professionals who deal with this every day. While the scale is orders of magnitude different, the underlying fundementals are the same.

And you can 100% guarantee that all of your infrastructure dependencies won’t be obsolete in 5 years or any company or open source project won’t be abandoned and that every vendor you depend on will always be acceptable and that you can switch with low friction and no downtime?

Does your company have the luxury of building everything in house?

Yes. And we can if we want to, yes. We’ve taken over open source projects we rely on, but most anyone can do that. We’ve built open source versions of seven figure commercial software just to ensure we have the skills in house to do it.

You don’t need to have thousands of employees or billions of dollars in revenue to accomplish 60-80% of what I’m describing. I’m simply arguing to take measured steps at critical junctions in your software’s lifecycle.

I can tell you with 90% confidence that if you’re working for a financial institution that offers bill pay, your entire bill pay infrastructure is dependent on a third party provider. I know the provider.

If you offer an iOS app, you’re dependent on closed source software. Your project management software would be a beast to migrate, etc.

Are you really suggesting that every company should “take over” every open source project they depend on and build “open source versions of commercial software?”

I feel like I’ve made my thesis clear, and I’m not willing to argue it further. My apologies.

If going Oracle played even a minor part in Amazon's success, then it was a fantastic call, even if now they have a migration task on their hands.

Both of these things can be true:

- we succeeded by betting the farm on a single vendor

- now we need to cut that dependency

I’m no more saying it was a bad idea at the time than Apple basing iTunes and later the App Store on WebObjects. If I’m consulting a startup that needs to move fast and get off the ground, I’m not going to tell them to worry about “vendor lock-in”. I’m going to tell them to use a cloud provider, use whatever “proprietary lock in” technology they need to to help them move fast and get funding and later if they grow big and they see thier needs change, then maybe suggest an alternative when it makes sense to hire a bunch of internal netops people to babysit servers.

But get a developer who knows there way around AWS/Azure and hire a third party company that can do some of the heavy lifting will save a lot money.

Yeah I consider myself one of those people. I think I can go toe to toe with most “AWS Architects” that only know netops especially since most of them don’t know development or devops on AWS.

We're moving a big govt system from Oracle to Postgres. Should never have been in the first place, but you gotta start somewhere.

I was turned off from Oracle over 15 years ago when I couldn't find a price on thier Webster - buying anything from Oracle is like dealing with a car salesman. At least MS gives you upfront pricing. There are some nice features of Sql Server if you're using other peoples money. But it wouldn't be my first choice.

>> Try to build a system with 150 databases

With Domino/Notes, you should encapsulate that word databases in quotes. A Notes "database" can be (and often are) as simple as a single form. I would guess the Baltimore PD has less than 50 very large Notes databases with the remainder being low use and/or simple enough to require little to no maintenance.

Because of that, the 150 database count is probably as misleading as the $176k metric (which is spread over 2y, making it pretty cheap for a skilled Notes consultant).

It took almost 5 years for our organization to move off Lotus Notes and that was a concerted effort with a considerably smaller foot print. We had a handful of full-time Notes developers and administrators on staff and were still paying quite a bit in support from IBM annually.

This is a bargain by comparison.

Lotus Notes is incredibly reliable so if the department is just maintaining the system as is, without upgrades or changes, while developing a new replacement solution then a single administrator is probably more than what's needed.

Our company uses LN. We've been on a five year plan to migrate to Exchange for the last decade.

Exchange can be just the tip of the iceberg. Our organization had a hundred or so custom applications and databases in Notes.

That being said, Exchange migration can be really tricky too depending on your retention policies, legal obligations around data preservation, and other factors. If you're doing onprem installs of Exchange rather than hosted solutions, you also have to worry about disaster recovery.

Say what you want about LN but it's built-in backup, retention, and recovery mechanisms put Exchange and even most Databases to shame.

I wasn't directly involved in our org's migration but from what I understand, the org waited until it had settled most or all outstanding litigation before attempting to migration. When they did, we changed our data retention policy for email from forever to 6 months before starting. This significantly reduced the volume and scope of work. And I think we kept the 6 month policy after migration because it's legally advantageous.

We have our long term retention (SEC compliance) with Smarsh, but everything else is 3 months or less. And even after implementing Exchange, we still have a lot of applications in LN that will be retained until someone decides the cost to re-write them is cheaper than licensing and support for LN.

Amazon has been trying to move off Oracle for 5 years and it’s still expected to take two more.

https://www.cnbc.com/2018/08/01/amazon-plans-to-move-off-ora...

Moving off of an infrastructure is hard.....

It is and vendors certainly don't make it easy to do. Most migration tools are written by the competition and are usually lackluster at best.

As I mentioned, the App Store and the iTunes Store are still on WebObjects. It’s not because of vendor lock in, its always hard to change infrastructure. If ain’t broke, don’t fix it. No company is going to change from AWS to Azure to temporarily save a little money and very few statistically are going to take thier infrastructure in house that were in AWS and buy a lot of colo equipment in multiple data centers and hire a bunch of people to maintain it if it’s not their core business.

Netflix is famous for transitioning from thier own infrastructure to AWS. If you look at thier open source tooling, there are a lot of AWS specific pieces.

According to the company awarded the contract and who built it out in 1996, they said it "runs so smoothly that it isn’t necessary to bring in someone else to help manage it." It goes on to say that “They'd do nothing all day long."

So basically the 177k is an insurance policy in case something goes wrong but it doesn't sound like there is a lot of day to day maintenance and I would hope they also have a feature freeze since its planned to be decommissioned in the near future.

>I can't imagine there are many specialists out there who know how to support lotus notes

There are dozens of us. DOZENS!

Seriously though, it's Domino support ultimately and there are plenty of Domino/Notes specialists working in the enterprise support market today. You don't see them if you're a software house any more than you would expect to see SAP consultants. But there's heaps.

Plus, Domino was from the ground up a multi-master DB replication system which merely happened to have a flagship capability for email - Baltimore PD are probably using Domino/Notes exactly how it's intended to be used rather than being a poorly-thought-out cobbling together of technologies. It's a bit like supporting Access DBs - There's a limited scope to how 'deep' a specialization you need to support it effectively, once it's up and running.

Reply on news.ycombinator.com