Baltimore police will pay consultant $176k to maintain Lotus Notes system | Hacker News Reader

Baltimore police will pay consultant $176k to maintain Lotus Notes system

baltimoresun.com

175 pointslinkmotif229 comments

Two years of support for $176,800? What a bargain! As long as this old system gets the job done it seems like an extremely cheap way to go.

Seriously. This is an epic installation:

> Officers tap into its databases to track criminal investigations, check arrest data, log ballistic test results and identify “troubled officers,” along with various other uses. There are millions of records and roughly 150 databases built into the system, each designed to address different unit and personnel needs.

I also enjoy how baltimoresun.com totally defeats uBlock Origin and brings my computer to its knees. http://archive.is/QZ3q9

... but then the next paragraph paints a different pictuere,

""" It found that the “siloed nature” of the Lotus Notes databases made it difficult for officers to match, verify or search for information. It also found that various systems may contain conflicting information about the same case, or may not reflect the most complete information. Downloading or querying data is made “difficult, if not impossible” because it isn’t inputted in a standard way across databases. """

The contract was awarded to the initial company that built out the holistic platform in 1996 so you've got to wonder how much it has cost them over the last 22yrs.

This kind of legacy stuff makes my stomach turn.

Calling this a "system" or "platform" is probably being very generous. It sounds like a very loosely related glom of Domino databases that grew up organically, probably with a little custom UI sprinkled on top. Notes can be infamous for that kind of thing, worse than Excel even, since it is more powerful...

Most definitely - the systems apparently "talk to each other" throughout the day whatever that may mean in their case.

I once worked for a company where the software that ran thier back end was written in Powerbuilder by contractors in 1999. The two developers who were there had been there for 20 years supporting the system and writing stored procedures.

You might want to double check your setup. I disabled Pi.Hole and reloaded the site with just uBlock Origin running. I did this a couple times with cleared cache, it pulls 22-24 requests and loads in ~1.9s.

> I also enjoy how baltimoresun.com totally defeats uBlock Origin and brings my computer to its knees.

Try blocking sdad.guru. I checked and it's one I've blocked myself, and Google says they're an adblocker-circumvention service.

In Lotus Notes (yes I had the misfortune to use it) "database" is really something more like a "structured file". It's quite a lot different from a SQL database, and 150 of them is not a lot. The system I worked on had 10000s (although admittedly it had a very peculiar design even for Notes).

Some of the less reputable and saltier ad networks intermittently run a malware service that does a DoS on clients running blockers. 'uBlock Origin Extra' is a standalone tool by the same guy that works for me against it.

Exactly, I don't see the problem? That's cheap. I'm sure when they switch off of lotus notes it'll cost several million.

The problem is that Baltimore PD apparently thinks custom enterprise IT is something you should be able to buy at the Apple store. They're going to be in for a really rude awakening one day. Maybe we can send 18F over there to do some pro bono work.

18F?

I think they are talking about this: https://18f.gsa.gov/

I believe they are the team that turned around the Obamacare website in record time after it was on track to crash and burn.

Both 18F and USDS (US Digital Service).

It's not pro bono if your salary is taxpayer-paid.

I think 18F only really works at the Federal levels (please correct me if I'm mistaken). But there might be some Code for American fellowship people who could be embedded with the city.

Bargain? That's practically highway robbery (taking from the developer that is)! A developer making shitty modern react apps can make 180k a year with a bit of effort.

And I have no idea how many lotus maintainers there are left in the world. Definitely not that many.

Not many new ones being minted, but plenty existing ones I am sure :).

>A developer making shitty modern react apps can make 180k a year with a bit of effort.

Living in an area with million dollar homes and 3k rents, sure.

Unless you bought your house 10+ years ago for a third the price. So one can make shitty modern react apps for $180k a year with a $300k home you nearly paid off.

If I could go back in time 10+ years I'd be putting all my money in bitcoin and Netflix

If only $36,000 of those 180k are going to rent, that's not a bad deal.

That's sharing a studio with someone or a 1BR with 3 others. When you actually factor in costs by all metrics you're better off making high 5-figures or low 6-figures in any non-urban area than you are being anything but the top 2-5% of software developers in the Bay Area.

Rents in San Francisco are very high -- high enough, in fact, that this silly, obviously-false hyperbole is unnecessary.

So you're claiming studios are $6000 and 1BR's are $9000? You can get a 1000 sqft for < $3000 within a 1 hour commute of SOMA.

A 1 hour commute costs you 2 hours a day. That's almost like working an extra 7 weeks a year with lower productivity and increased wear and tear on you.

who's debating that?

2 hours is about how long I spend at the gym every day and I'd hate to give that time up.

If you're spending more than ~50 minutes at the gym you're doing something wrong.

What a deal!

When you can make $200k straight out of undergrad, yes, it is.

There was an article floating around earlier this week that goes into the numbers of this: https://priceonomics.com/the-places-in-america-where-all-you...

You have to add income tax to your calculations. 180k in SF is about %60 of that, which is 108k, so exactly 1/3rd.

>! A developer making shitty modern react apps can make 180k a year with a bit of effort.

Only in an economy that runs on monopoly money. Baltimore isn't SV.

Baltimore is in striking distance of the DC area, where salaries aren’t known for being paltry due to federal largess.

Yes there's upward pressure on pay (and the cost of everything that pay buys) from the DC area but the DMV is no SilleyValley in terms of pay and costs.

Pay, definitely not (I live within extended commute distance and have gotten offers for senior dev at $80k and similarly insulting numbers) but it's pretty up there in terms of COL.

Wow, those companies definitely are low-balling (especially for a senior position in the DC area). Were those positions ones that required a security clearance, though?

Federal workers don't get paid particularly well, actually. There's some competition among contractors especially if you have active clearance but nothing approaching SV.

Umm, I know plenty of people in the Baltimore area who are pulling in that kind of money for Rails and Django apps.

I see nothing in the article that suggests this is a single person's full time job.

In that case this is an even bigger steal!

> Computer & Network Consultants’s David Alonge, who is responsible for programming and maintaining all of the Police Department’s Lotus Notes applications, told The Sun that the software is “working wonderfully for the police.” He said it runs so smoothly that it isn’t necessary to bring in someone else to help manage it.

That makes me think it's even more likely not his full time job.

If it's running well, he might never have to touch it. We sell some Notes-based products with some considerable customers (still!) and only get a couple emails a week or month. Annual support and maintenance fees just keep rolling in.

There's a lot to be said about legacy systems that just work.

Or that the Baltimore Police don’t mind it running badly - given some of that department’s troubles, having the odd document vanish might be beneficial to them.

I have every confidence that they're expecting 60-70 hour weeks for that.

Are you familiar with the Baltimore Police Department's RFP and procurement processes? Because companies and governments have paid a lot more for a lot less.

Good, senior, full-stack devs make $90,000-125,000 in flyover states atm. And I do mean good ones. A lot of people are stuck at $80,000ish.

$180,000 for a front end tech lead is unheard of in said states. Architects make maybe $160,000.

150-160... flyover state :)

Baltimore isn't exactly in a flyover state. There's upward salary pressure from all the money shoveled into the DMV area.

hahahaha at Maryland being a flyover state now. With it hosting several of the richest counties in the entire country, and like all of the federal government and contractors that can't fit inside DC city limits.

Just because the customer is in Maryland doesn’t mean the dev has to be.

Hey I'd be happy to get $70,000 but I'm stuck having to take care of family so I have to stay in the middle of bufu egypt. Trying to find remote consulting gigs but they're hard to find!

Most companies will offer to pay for the relocation of you and your family, along with your home, from Egypt to wherever you want to move.

I don't think "most" is a fair statement.

A more accurate statement would be "Many large silicon valley companies will pay for your relocation if you are an exceptional developer for the price."

GP probably doesn't actually live in Egypt, but was probably referencing the expression "Bumfuck, Egypt", which means out in the middle of nowhere.

Nevertheless, his relo won't be paid for unless he's a God-tier, exceptionally desirable developer.

Relocation for new hires isn't rare. What are you talking about?

Contractor could make that, which is what I thought was meant by "with a bit of effort".

The PHP and Ruby developers I worked with ten years ago in DC were making that much, and the good ones were making significantly more. I haven't been involved in state-side web development since then, but I imagine that regional salaries are - significantly - higher at this point.

What do you consider a “flyover” state? From the major cities I know that are not on the west coast, New York City, or DC, $125K is around the top for a “Senior Developer” and $145K is near the top for architect level skills who don’t have any direct reports.

$125K is near the top for a Senior Developer in Charlotte, SC, circa 2014 (flyover? Probably). Probably higher there now, and definitely higher in NYC or DC.

Same thing I’m seeing in Atlanta. I haven’t seen too much of a change since 2014.

$125 seems about mid or upper mid range for Sr Devs based on data from Salary.com. Architects can make over 170K on the top end. Not sure how recent your data is but there's a real crunch for talent up here these days.

It depends on the market. Atlanta for instance is showing $130K median and 144K st the 75%. No that’s not accurate.

But it’s also showing 141K at the 50% percentile and $159K at the 75% for a Software Engineer V. That’s more accurate.

Another data point for different cities....

https://www.matrixres.com/resources/salary-survey/

I believe you answered your own question. When there aren't many individuals left in the world with the associated expertise, and when a younger developer sees no growth potential for learning this expertise, you probably will have to pay more a la supply & demand. That said, this doesn't seem particularly high. My guess is a migration would easily cost seven figures. However, sometimes delaying the problem is really kicking the can down the road to when there are even fewer people with lotus expertise.

if he plays it right, he will have an easy job for life

It's actually not bad, given the way spending usually goes in government in the US. Roughly the cost of one decent engineer's salary in Baltimore (for that two years). I might have expected closer to $500,000 for two years instead.

I was expecting way more. That's less than the salary I would expect them to pay a sysadmin over 2 years.

I was thinking that was low for _one_ year when you consider that it's a contract position, so the consultant is on the hook for self-employment taxes and health insurance and such.

You're on the hook for income tax where I live regardless of whether you're contract or employee. I'm a contractor so my only additional cost is really health insurance (much cheaper in Canada since the basics are covered by the government, like $150/m for a good plan) and Quickbooks.

You are in the US as well. We have FICA (social security, medicaid, Medicare) of which half is paid by the employer and half employee. If you hear "self employment tax", that's typically referring to the fact that you're paying both sides (15.3% over 7.65%).

There are a few more nuances, but that's the gist of it.

Thanks. The state (or most states) tacks some on as well, right? If I was not self-employed my employer/client and I would be paying into Employment Insurance and Canadian Pension plan as well, but I'm able to skip that as I own more than 20% (don't recall the exact percent they require for exemption) of the corp I work for. There is also 13% sales tax but I stack that on top of my hourly. I can claw back some of the sales tax by claiming the tax I paid on business expenses.

Oh yes, I was only covering federal taxes. To my knowledge, the general structure in most states doesn't matter. I don't know of any states where income taxes are partially paid by the employer and partially by the employee (like FICA).

"There are millions of records and roughly 150 databases built into the system, each designed to address different unit and personnel needs." - yeah, good luck replacing that for less than $90,000 a year.

1. Notes -> CSV -> MySQL

2. Hire someone in another country for below minimum wage

3. ...

4. Profit

Exactly how many terrible software project disasters start...

I'm sure when an article like this runs, their IT department is getting bombarded by cold calls from vendors offering to upgrade them.

While the headline seems like a lot of money, it makes me realize how thin the margins are becoming on these tech services.

Full stack is still safe right?

Whilst $176k of IT services wouldn't get you far, it would likely be enough to buy user seats on a SaaS product. Not to mention this software probably sucks and there is a big opportunity cost with detectives wasting time on using dated software. If you spend $2M on software that saves you $3M in payroll on overtime because your detectives are wasting hours per week on case management, then $176k isn't a bargain anymore.

I worked on a project where we were trying to modernize it with a fancy web UX. All the old timer users (people working at the insurance company for 20+ years) were used to working with the old version that was mainframe-based where they'd be opening flat files, and entering data, etc. they were really efficient at it. They knew all the keyboard shortcuts, etc. These weren't educated people, but by rote learned their jobs, well. They hated the new web-based, fancy UX that was built for them. So, I could see how the old-school accountants couple probably be pretty efficient at Lotus....

i wonder whether some of those vendors (may be some one who already has hands deep in that city IT pockets) gave the tip to the reporter and thus it is a "submarine" of an article to force the police department to upgrade and to provide PR cover for it.

Agreed, this doesn't seem overly expensive by any means. But worth noting this is just one developer's cost. Doesn't including licensing/support costs for Domino/Notes itself, hardware to run it on, sysadmins for the servers.

(And I don't know what platform they're using, but it does run on all of IBM's OSes: AIX, i5 (i.e. OS/400), and z/OS. Wouldn't be surprised...)

That developer way undersold themself. They should have asked for a minimum of $200,000 a year. That's a huge system. I am sure they are bound to get lots of late night emails and fires to put out.

Kudos to Baltimore police for good business. The system works, and to upgrade or replace it would cost millions.

Notes and "gets the job done" on the same phrase don't go together

$176k is some tiny percentage of what they'd have to pay IBM for the same thing.

Yeah, that's very cheap for keeping the light turned on a vital system running the police department of a big city.

Summary for those in EU: The Baltimore police department uses Lotus Notes software for keeping track of cases.

They started using LN in 1996. Now, all of their stuff is built on top of it and it's hard to switch. So they are paying $177K for the next two years of software support.

Summary over. Question of mine: is this a particularly high rate for two years of support for an entire police department? I can't imagine there are many specialists out there who know how to support lotus notes, let alone all of the custom data they have layered on top of it. And it seems like it would be a reasonable amount of work to maintain.

For an enterprise support contract of that size it's dirt cheap. Try to build a system with 150 databases on a cloud provider or using your own cluster and you'll spend way more than that. VMWare licenses alone would blow that figure away if you rebuilt it in a modern style. If you were crazy enough to spec Oracle for those databases then it would be many times that cost per year.

VMWare and Oracle in the cloud? "Out of the frying pan, and in to the fire."

No different than Lambda and Aurora. Never marry yourself to the vendor.

You’re always married to your infrastructure. That’s like developers writing “repository classes” just in case one day they want to switch from Oracle....

Statistically no one ever switches out their architecture. The risk is to great, the reward is too small, and the number of potential regressions is too high.

How did all of those people who wrote thier website with Angular 1 fare?

Are you going to rebut all of my comments about risk management in this manner (antagonistic, usually vendor lock in, but it's been about other risks as they relate to cloud infrastructure)? Because I work in risk management for a financial utility, this is what I do for a living: ensure developers and other practitioners don't make poor architectural and technology decisions.

One of your own comments [+] exemplifies the consequences of making a poor technology choice ("Amazon has been trying to move off Oracle for 5 years and it’s still expected to take two more. Moving off of an infrastructure is hard......"), and I'm pretty sure the cost of that choice and the resulting migration to remediate is expensive AF.

[+] https://news.ycombinator.com/item?id=17729107

Was it a poor choice at the time? Apple decided to base the ITunes Store on WebObjects in 2001. They certainly didn’t have “vendor lock in” since they owned WebObjects, how does that decision look now in 2018 that they are still running on WebObjects?

You’re always stuck on your architecture and you’re always at risk of a technology becoming obsolete.

Look at all of the companies that are still stuck with an older version of the JVM because they are afraid to upgrade - and thier whole stack is open source.

Unless you personally audited every single line of source code your developers wrote, I guarantee you that your architecture is tied to some assumption that will make it hard to migrate in 5 years.

We audit every single line of code written by our developers. We audit every line of libraries they use. We audit their architecture, their infrastructure, and their plan to move from cloud to bare metal at anytime. Our applications run for decades at a time.

We automate wherever possible, of course, but there are some tasks humans are required for.

It doesn’t hurt for developers here to heed the warnings from professionals who deal with this every day. While the scale is orders of magnitude different, the underlying fundementals are the same.

And you can 100% guarantee that all of your infrastructure dependencies won’t be obsolete in 5 years or any company or open source project won’t be abandoned and that every vendor you depend on will always be acceptable and that you can switch with low friction and no downtime?

Does your company have the luxury of building everything in house?

Yes. And we can if we want to, yes. We’ve taken over open source projects we rely on, but most anyone can do that. We’ve built open source versions of seven figure commercial software just to ensure we have the skills in house to do it.

You don’t need to have thousands of employees or billions of dollars in revenue to accomplish 60-80% of what I’m describing. I’m simply arguing to take measured steps at critical junctions in your software’s lifecycle.

I can tell you with 90% confidence that if you’re working for a financial institution that offers bill pay, your entire bill pay infrastructure is dependent on a third party provider. I know the provider.

If you offer an iOS app, you’re dependent on closed source software. Your project management software would be a beast to migrate, etc.

Are you really suggesting that every company should “take over” every open source project they depend on and build “open source versions of commercial software?”

I feel like I’ve made my thesis clear, and I’m not willing to argue it further. My apologies.

If going Oracle played even a minor part in Amazon's success, then it was a fantastic call, even if now they have a migration task on their hands.

Both of these things can be true:

- we succeeded by betting the farm on a single vendor

- now we need to cut that dependency

I’m no more saying it was a bad idea at the time than Apple basing iTunes and later the App Store on WebObjects. If I’m consulting a startup that needs to move fast and get off the ground, I’m not going to tell them to worry about “vendor lock-in”. I’m going to tell them to use a cloud provider, use whatever “proprietary lock in” technology they need to to help them move fast and get funding and later if they grow big and they see thier needs change, then maybe suggest an alternative when it makes sense to hire a bunch of internal netops people to babysit servers.

But get a developer who knows there way around AWS/Azure and hire a third party company that can do some of the heavy lifting will save a lot money.

Yeah I consider myself one of those people. I think I can go toe to toe with most “AWS Architects” that only know netops especially since most of them don’t know development or devops on AWS.

We're moving a big govt system from Oracle to Postgres. Should never have been in the first place, but you gotta start somewhere.

I was turned off from Oracle over 15 years ago when I couldn't find a price on thier Webster - buying anything from Oracle is like dealing with a car salesman. At least MS gives you upfront pricing. There are some nice features of Sql Server if you're using other peoples money. But it wouldn't be my first choice.

>> Try to build a system with 150 databases

With Domino/Notes, you should encapsulate that word databases in quotes. A Notes "database" can be (and often are) as simple as a single form. I would guess the Baltimore PD has less than 50 very large Notes databases with the remainder being low use and/or simple enough to require little to no maintenance.

Because of that, the 150 database count is probably as misleading as the $176k metric (which is spread over 2y, making it pretty cheap for a skilled Notes consultant).

According to the company awarded the contract and who built it out in 1996, they said it "runs so smoothly that it isn’t necessary to bring in someone else to help manage it." It goes on to say that “They'd do nothing all day long."

So basically the 177k is an insurance policy in case something goes wrong but it doesn't sound like there is a lot of day to day maintenance and I would hope they also have a feature freeze since its planned to be decommissioned in the near future.

It took almost 5 years for our organization to move off Lotus Notes and that was a concerted effort with a considerably smaller foot print. We had a handful of full-time Notes developers and administrators on staff and were still paying quite a bit in support from IBM annually.

This is a bargain by comparison.

Lotus Notes is incredibly reliable so if the department is just maintaining the system as is, without upgrades or changes, while developing a new replacement solution then a single administrator is probably more than what's needed.

Our company uses LN. We've been on a five year plan to migrate to Exchange for the last decade.

Exchange can be just the tip of the iceberg. Our organization had a hundred or so custom applications and databases in Notes.

That being said, Exchange migration can be really tricky too depending on your retention policies, legal obligations around data preservation, and other factors. If you're doing onprem installs of Exchange rather than hosted solutions, you also have to worry about disaster recovery.

Say what you want about LN but it's built-in backup, retention, and recovery mechanisms put Exchange and even most Databases to shame.

I wasn't directly involved in our org's migration but from what I understand, the org waited until it had settled most or all outstanding litigation before attempting to migration. When they did, we changed our data retention policy for email from forever to 6 months before starting. This significantly reduced the volume and scope of work. And I think we kept the 6 month policy after migration because it's legally advantageous.

We have our long term retention (SEC compliance) with Smarsh, but everything else is 3 months or less. And even after implementing Exchange, we still have a lot of applications in LN that will be retained until someone decides the cost to re-write them is cheaper than licensing and support for LN.

Amazon has been trying to move off Oracle for 5 years and it’s still expected to take two more.

https://www.cnbc.com/2018/08/01/amazon-plans-to-move-off-ora...

Moving off of an infrastructure is hard.....

It is and vendors certainly don't make it easy to do. Most migration tools are written by the competition and are usually lackluster at best.

As I mentioned, the App Store and the iTunes Store are still on WebObjects. It’s not because of vendor lock in, its always hard to change infrastructure. If ain’t broke, don’t fix it. No company is going to change from AWS to Azure to temporarily save a little money and very few statistically are going to take thier infrastructure in house that were in AWS and buy a lot of colo equipment in multiple data centers and hire a bunch of people to maintain it if it’s not their core business.

Netflix is famous for transitioning from thier own infrastructure to AWS. If you look at thier open source tooling, there are a lot of AWS specific pieces.

>I can't imagine there are many specialists out there who know how to support lotus notes

There are dozens of us. DOZENS!

Seriously though, it's Domino support ultimately and there are plenty of Domino/Notes specialists working in the enterprise support market today. You don't see them if you're a software house any more than you would expect to see SAP consultants. But there's heaps.

Plus, Domino was from the ground up a multi-master DB replication system which merely happened to have a flagship capability for email - Baltimore PD are probably using Domino/Notes exactly how it's intended to be used rather than being a poorly-thought-out cobbling together of technologies. It's a bit like supporting Access DBs - There's a limited scope to how 'deep' a specialization you need to support it effectively, once it's up and running.

This article is irresponsibly written, because the two-year term is buried in its middle.

The headline reads "will pay consultant $176,800," which will be parsed as "per year" by, what, 99.9% of readers?

The first sentence is no better: "pay $176,800 to a computer software consultant."

As others have mentioned, this is almost certainly a bargain--I can't imagine any new system being quoted without a price of 'millions.'

Clickbait title. The article itself seems mostly positive, rational, and informative but the poor citizens of Baltimore are getting squeezed for ad dollars instead of being kept properly informed about local events. This is a nonstory and the journalist knows it. Baltimore PD is getting a great deal supporting their legacy IT while they convert to a modern system that will no doubt cost many times more to purchase, integrate and maintain. At just $44/hr the consultant is being underpaid for the area if they're full time 1099.

That's only just over 20% premium to the fulltime UK salary I could get in London which pays a lot less for developers than the USA does.

A average contractor in London would be making about $160k

The contract is for two years. So it's actually a 10% discount.

Well, in three years when the headline is "Baltimore police pay $240M for failed Lotus Notes migration," then I'll click. But sometimes maintaining the legacy system is the lesser of evils, especially in government systems.

We built a custom management system for a state agency here 10 years ago. The state still pays us around $40k/year in associated improvements, changes based on new legislation, reports, etc. Far less than paying $1M-$2M to rebuild the thing, or more. Total cost of ownership for the system is ridiculously low, all things considered. And yeah, it's written on some old crufty PHP, but it still does the job and they use it every day.

This thread is hilarious in how many people apparently have blinkers on about the "find a need and fill it" advice that populates every BizDev-oriented thread here. It makes me wonder how many have any experience in software maintenance.

What comments are you taking about?

The ones implying, "gawd, why does this kind of business even exist?"

Trying to access the site from UK:

Unfortunately, our website is currently unavailable in most European countries. We are engaged on the issue and committed to looking at options that support our full range of digital offerings to the EU market. We continue to identify technical compliance solutions that will provide all readers with our award-winning journalism.

Got same in Austria

I could get in with no problems from the EU.

http://www.cnciweb.com/index.html

They are talking about the Baltimore Sun website.

Use a VPN.

One of the things I do not miss in the slightest from my time working at IBM was their use of Lotus Notes databases all through the processes they used in finance. In hindsight it is a text book example of a tool that gives the installation great freedom in customizing and tailoring the system to their needs, but provides very little in terms of configuration management or evolved system visibility.

What really surprises me though is that IBM doesn't have a tool for extracting the resulting system out of Lotus notes and putting into something else. They could have used it internally to modernize their finance systems, and they could sell such a tool to the consultants who would bid on modernizing the police department's systems.

I think the complaining about the $176K as an expense is silly (its pretty cheap) the complaint that they aren't investing in modernization is more critical.

In terms of ways to change the world with software, these sorts of business process systems are a place where there is always room for improvement it seems.

At some point, didn't IBM move Domino to a DB2 back end? I would think moving data off of an RDBMS like DB2 would be easier than the legacy Domino engine.

I remember there being an Notes-ODBC driver, but I don't remember if it was any good.

Three was a brief moment when this was a thing. But it really did not catch on. They deprecated this feature few years ago (by deprecate I mean the "IBM deprecate" which is ignoring feature in future releases so maybe people will forget about it).

Data is not the problem here - it always was the hakish way most notes app was created. How can any consultant / it dep. keep up when there are hundrets of apps done without any prior design by "Citizen" developers.

Did anyone click the link to gaze on the vendor's beautiful website?

http://www.cnciweb.com/index.html

Where is the "e-mail message going around the globe" gif and the "made in FrontPage '95" banner?

Don't forget "under construction", and the hand exploding out of the page through torn paper.

Or the "Uniracers arrow".

I bet it's super reliable.

"Custom programming is also available in most of the latest languages including Lotus Notes, Access, Visual Basic.NET and C++."

"We had a guy who installed Visual C++ by mistake once"

Still better than Engadget's website.

Hey, if their major selling point is being able to support a system that the police department has been using for 20 years, that website looks just fine.

Honestly it loads fast, looks clean and has all the info I would need from them. I can't complain. Source says it was built using Trellix Site Builder.

I'd rather have sites like this than stuff full of bloated JS frameworks and trackers.

Lmao they have a different font and text color on every single page, what are you talking about? It looks like absolute, pure garbage. I would never want that to be my company website if I was a CEO, even if my company did just support systems that are 20+ years out of date.

Honestly, the more I look at this, this may be the worst website I have seen in quite a few years. I can't believe this is a currently-existing tech company.

That's like refusing to work with a dentist because he drives a beat up old car, or with a mechanic who's got crooked teeth.

What does the ability to create a "modern" website have to do with their ability to maintain a legacy database?

Because to most people appearances matter and that website (or a beat up old car) doesn't speak to them having their proverbial shit together. Maybe that's not your case, but I wager that's true for the average person and therefore to the average consumer.

The HN audience is probably strongly correlated with the minority of people who think appearances doesn't matter as long as it gets the job done.

Please note that I'm not saying one of these outlooks is better than the other. I'm just saying most people care about appearance for better or worse.

Not the prettiest, but nice and fast, and no Javascript. Nice to know I don't need the browser to process an entire framework just to load an h1 tag.

Yeah, 0.3 seconds to render is less time than it usually takes just to load and parse React:

http://www.webpagetest.org/video/compare.php?tests=180809_58...

Still up and working just fine with HN clicking on it all day long. I'll take it!

Check out this Computer Science Professor's website!

https://www-cs-faculty.stanford.edu/~knuth/

And I'm supposed to use this guy's typesetting program?

The profile page - http://www.cnciweb.com/id2.html - indicates support for Windows XP, CDROM, DOS, multimedia, but no mention of Notes. How can we be sure they're qualified?

http://archive.is/QZ3q9

ta Codes

As old as Lotus Notes is, I don't think anything quite like it has come on the market since.

Most organisations seem to stumble around using a combination of unstructured Word documents, file systems and email.

When they find themselves totally at a loss they’ll try and shove the whole mess into a document management system, but a combination of unstructured documents and organisational inertia means it usually does more harm than good.

There’s SharePoint, but that’s best avoided.

Some places, with a bit more technical skill use wikis, and issue tracking systems like Jira, but even they aren’t quite what Notes was in its heyday.

I sometimes wonder why there isn’t a compelling successor to Notes built on CouchDB.

I wonder why you say that SharePoint should be avoided. I get that for small businesses it's a lot of administration to worry about (and ties them to using Microsoft software) - however they can use O365. What features would make a compelling successor that SharePoint doesn't (and what are even alternatives?)

""" The consultant is charged with system upkeep until the department finishes crafting “a 21st century holistic technology platform,” he said. The final technology plan will be delivered to the court on Dec.1."""

crafting? holistic? platform? 21st century?

Hmm, me thinks this "platform" is doomed from the onset if you plan on crafting a holistic platform plan by Dec 1, 2018 after still running a system crafted 22yrs ago.

This "holistic" platform is the one that'll end up costing 20M when the Lotus Notes thing would've worked fine

Holistic at 20m?

Canada tried to bring to the table a holistic payroll system, known as phoenix, which cost them 1b before they decided to scrap it. :)

FWIW, there's nothing "holistic" about Phoenix - it is centralized payroll (same as Regional Pay System it replaces), but the HR is completely distributed amongst departments (running PeopleSoft, SAP, legacy or bespoke apps of various natures and kinds), and Pension, central index, etc are various separate monolithic applications as well. A very large percentage of problems can in principle be traced to the fact it is the very opposite of holistic system (note, I am in no way claiming that consolidating disparate HR systems into one, enforcing data cleanliness, and creating a "holistic" system could be done before heat death of the universe, for sum of money less than totality of our GDP, and resulting in anything else than chaos, confusion and disorder... just clarifying Phoenix was emphatically not an attempt to create such a "holistic" system :)

LOL - It was more tongue in cheek but you are right, Phoenix, wasn't an attempt to create a holistic system.

Usually when I hear projects being spun as "holistic" it means to me that folks haven't done any basic groundwork yet on the requirements and its simply the pie in the sky approach to apps, we want it all in <12mos for half the cost.

The consultant should have added a 25% price increase per year into the contract!

I was thinking that it sounded a bit high but OK for a 1 year contract, but when I read the article turns out it's over 2 years.

At $88k per year that's totally fair.

> I was thinking that it sounded a bit high but OK for a 1 year contract

If you think that's expensive, you really have no idea what other organizations spend on IT projects. Do you have any idea how much the NYPD spends in IT? I once interviewed a Dev Team Leader who worked at the NYPD. Based on the projects he worked on, I can easily say they spend magnitudes more than Baltimore PD, but they also have a much wider range of systems.

BNP Paribas still used Lotus Notes in some departments. And every year it pays a lot of money to COBOL consultants as well.

The highest margin software consultancy I ever saw was specialized in maintaining legacy sw applications, running on setups that were popular in the late 80s and early 90s. A little niche, but extremely high margin.

I work at a company where one group in IT makes AngularJS apps, another writes industry machine learning models, and a few others are using Lotus Notes.

People will use what works

OT - Someone smart once said that Lotus Notes was an email/groupware app wrapped around one heck of a robust sync engine.

As an even more OT comment, I always loved the password entry box for Lotus Notes. When you typed in your password, it would do a simple animation of cycling through hieroglyphic symbols.

What's OT?

[ Please don't dismissively downvote. Google search for OT yields a wide variety of expansions, none of which seem to explain this abbreviation. https://www.google.com/search?q=OT ]

off topic ...

Welcome to the Internet, stranger.

It stands for "off topic"

Helluva deal for the Baltimore police!

Notes and Domino are actually fairly capable from my understanding. We used it until very recently to support hundreds of retail locations, thousands of in house applications, and corporate mail services. IT has been slow long replacing it over a couple of years but it was working well. The email and messaging part is fine, it’s really the calendaring part that was giving us the most grief.

Given the number of FOSS hospital management systems out there, are there no similar attempts to create FOSS software for the common government problem of law enforcement? Some of our Federal agencies might be able to help with this, saving state and local agencies buckets of money over their various bespoke solutions.

If you go with name brand software, you can find someone to support it 20 years later.

I doubt the same is true with most FOSS

It's kind of sad that so much information is stuck in proprietary systems that need to be continually maintained, even if the original vendor of the software has left. With something open usually there'd be a quick and easy way to migrate all the data off of the platform to something newer when necessary.

Are you under the impression that rebuilding the whole thing on any of the popular development options currently available would result in lower maintenance costs? Or a more open system?

Let's say they rebuild it with a Ruby backend (or maybe Go? That's trendy these days), a bunch of React (or vue, or maybe whatever's current for Angular), Postgres (or MariaDB, maybe Oracle's MySQL, maybe even Mongo), maybe throw it on some Linux boxes with Apache. In 20 years that's all going to sound like "We built it on Notes and now pay $7500/month in maintenance and update costs" along with everything that'll happen in between when ECMAScript 10 deprecates a bunch of language features.

Could it be better? Absolutely. Should the current system be kept and extended? Heck no. Is data migration a significant issue? Only in that it's a very complex system with a huge range of different types of structured data. They have programmatic access to the raw data, dumping it out is logistics.

I meant open as in not proprietary. It's a lot easier to migrate an SQL database than a Lotus one, because one is open and people have written code to allow data to be exported.

Actually its not true - notes (or rather domino) data metamodel is really simple (except for rich text maybe). The real problem is lack of schema which ecourages people who dont know better to build horrendous things. And if there is no schema and no documentation then is it really notes fault that migration is sisiphus work?

The same sort of reasoning behind my Java web apps interfacing with a database system that doesn't understand booleans; and needing to interface through RPG programs in order to actually get any business logic done.

As a web developer, it makes me scream sometimes. I get it (somewhat), but I still want to scream.

https://arstechnica.com/information-technology/2018/06/balti...

Some context

I suspect no one 8nderstands the cost of building and maintain gig software past 10 years.

Some problems have been around longer than most developers who want to rebuild every 3 years.

Lotus Notes, Lotus Domino were among the original mobile, offline first business apps.

I have a “it’s not going to help my resume” premium for any technology that doesn’t help me get my next job. I also have a slight “I’m going to learn a technology that will help keep me marketable” discount.

It could be worse. 10 years ago the California judiciary spent over $2 billion for a case management system that never worked.

Am I missing something? Seems like a fairly mundane and common occurrence of hiring a tech worker. How is this newsworthy?

I think a Wire rewatch binge is in order...

I can't wait to see what the ancient systems that need support in 30 years will be.

Isn't it still alive as IBM Notes?

Yeah it is. Some india based IT Sweat Schop (HCL) bought it and they are going to add node.js integration to it to make it look modern :)

If anyone could provide a summary, seems GDPR has really made it hard for some outlets to render text (editors note: this was meant as a light-hearted joke, not ignorance)

  Unfortunately, our website is currently unavailable in most European countries. We are engaged on the issue and committed to looking at options that support our full range of digital offerings to the EU market. We continue to identify technical compliance solutions that will provide all readers with our award-winning journalism.

Or they didn't think it worth it to rewrite their app to selectively allow enabling or disabling of features based on geographic region

Yeah, I said it in jest, have rewritten software myself for the same reason, it is an undertaking.

Are people in those other regions providing revenue? If not, then why should they make their content available in those places?

So their content is available in those places.

GDPR;DR - The Baltimore Police Department has been using Lotus Notes for internal communication and data storage since 1996. It's also been contracting the same IT consulting company since 1996. Some say the system is obsolete and question the decision to renew the support contract, but the IT consultant insists it's working beautifully. The BPD says they will move to a new system, and will roll out a plan in December, but in the meantime they have to maintain what's in place.

Thank you. This was all I was looking for.

That’s not it. GDPR has made it hard for some outlets to make money in order to pay for the content you wish to read.

However, admittedly I run a variety of ad-blockers because newspapers have long abused my bandwidth with grossly inefficient advertising. But the penalties for violating GDPR are so great that it isn’t worth the risk (even if you are doing everything correctly, you still have potential international litigation expenses just to defend.)

Just curious, who should pay for the stories you read on newspaper websites?

Answered another comment with the same, but just in case: I didn't make it clear enough I was poorly executing a joke.

GDPR doesn't mean that you cannot show advertisement. Just means you have to actually be responsible and not abusive as many content providers with ads are.

Whatever. You obviously have never read a regulation and tried to comply without a team of lawyers. Or never run a small business. Or have a team of lawyers (which the nyt does but still can't get it right). Or are in violation and don't know it. Way to kill the internet. Thank you EU.

Please start posting civilly and substantively. We ban accounts that don't.

https://news.ycombinator.com/newsguidelines.html

> If anyone could provide a summary, seems GDPR has really made it hard for some outlets to render text.

If you put it that way and see it that way, it will be impossible for you to understand what businesses like these do beyond just text rendering.

Baltimore Police use a case management system based on Lotus Notes, which isn't particularly well-supported or maintained anymore. They just agreed to a 2-year, $176,800 deal to get support from a consulting firm.

This is a price that we are being asked to be upset about. No, don't take the time to estimate what it would ultimately cost to replace it. Just see $LARGE_NUMBER and be upset.

which isn't particularly well-supported or maintained anymore

What does this mean? Notes appears to have fairly regular updates every 3-6 months.

I think reading between the lines of the article that they aren't paying IBM for software support.

They don't need to, they're paying a third-party for support. I wouldn't be surprised if the contractor has an IBM support account.

IBM really, really does not want anything to do with further developing or supporting Lotus software products. I believe that they gave that whole business over to HCL not long ago.

Rumor has it they've largely stopped using it internally, too.

OK, HCL, but by that token Thinkpads and Deskstars are still being made.

Only for the most current major version. They're almost certainly not running that.

I got the impression that the police department's system might be suffering from some bit rot.

That could easily be an interpolation on my part, though. I tend to just assume that all critical infrastructure built on Notes is poorly-maintained in the same way that I tend to just assume that all critical infrastructure built on Microsoft Access or Excel is poorly maintained. (I'm sure there's some well-tended stuff out there, it's just being kept hidden from me for some reason.)

How ironic for nyt for blocking eu. Maybe eu should get real about onerous and unrealistic regs and stop trying to regulate the world. got to love the dumb cookie notice reg let's go back to using static html and no user interaction

https://news.ycombinator.com/newsguidelines.html, especially the “Eschew flamebait. Don't introduce flamewar topics unless you have something genuinely new to say. Avoid unrelated controversies and generic tangents. ” part. If you want to be corrected about what the GDPR does you could find a thread on that topic.

Of course. It's called the Great Despicable Prohibition of Rendering for a reason. It also applies to absolutely everyone, especially local Baltimore newspapers. The EU cares about local newspapers of foreign cities. It hates them.

If you don't obey, the penalties start at 20 000 000 000 000 000 000 000 €, 400% of your income and your firstborn. They will be evil, disproportionate and diabolical. The supervisory authority will not take into account silly things like "the nature, gravity and duration of the infringement" or "the intentional or negligent character of the infringement". That wouldn't be fun. In particular, there is no chance that you will be issued a reprimand instead of a fine. The authority would do that if its mission were to protect citizens' privacy instead of destroying your business and helping big friendly companies like Facebook and Google, which the EU is well-known to love.

Do not leave your house before you have read every single EU law. If you don't the EU will find a reason to destroy your life. Especially if you live in a weak and unimportant country that is easy to bully around like the United States of America.

It's odd the Baltimore sun even bothers paying attention to GDPR.

Yeah. Are they owned by some company with European operations? If not, they've gotten some really bad advice.

How does one go about getting contracts like these with the government to do trivial tasks for exorbitant amounts of money?

What trivial task? Maintaining a very large legacy system for something as important as a police force sounds like a very difficult task. All this for the price of one cop sounds like a bargain.

I wouldn't say the task is "very difficult" either. The task is probably no more or less difficult than maintaining the inventory system for a store.

The cost and job security comes from the fact that you have to do it in Lotus Notes.

If you screw up and a store is out of X it doesn't matter to anyone but the owners of the store. If you screw up the police databases someone can die, evidence can be lost, all sorts of awful results. It's a much, much, bigger deal.

First of all, if you screw up and say something is in stock when it's out and that thing is critical to someone's life, someone could die. Even with mismanagement of an inventory system.

So let's not descend into "but lives are at stake" hyperbole.

The vast majority of the database are fines and tickets. And it's likely a lot of this stuff is duplicated elsewhere. If the police have a warrant for someone's arrest, that warrant is also on file at the courthouse where it was issued. If they booked a suspect and he went to county jail, the county jail has that information. If the entire evidence log is wiped out, all of the necessary information is written on the bags.

There is a ton of redundancy both intentional and accidental built into law enforcement. No one is dying because you need to restore the database from a back up because Larry deleted the database again.

And even still. Even if the consequences are that much dire, the work is the same. There is nothing special about the data that makes it any more or less difficult to store.

> I wouldn't say the task is "very difficult" either.

I wouldn't say anything, given that we know very, very little about the system in question.

I mean, any difficulties would be accidental difficulties, not essential ones. I mean, according to the article, the system already has consistency and integrity issues. (And no one is dying because of them either.)

The point is that the job is essentially data management.

> The point is that the job is essentially data management.

You might be right, but you could so easily be wrong. You really have no idea what the job entails. The software might be great, it might be a nightmare to maintain/operate.

What I do know is $88K/year for any consultant typically doesn't get you much.

It being a nightmare to maintain/operate would be an accidental difficulty, not a necessary one.

The guy I responded to implied that the fact that it is for the police department made it more difficult. My position is that that has no bearing on the difficulty of the task.

> It being a nightmare to maintain/operate would be an accidental difficulty, not a necessary one.

Who's saying it would be a necessary one? If you inherit an awful system that's time consuming to maintain and operate, you have a few options. You can simply maintain it and bill for that, or you can improve the system. Sometimes it's easy to improve the system, sometimes it's not. It all depends on the circumstances.

I'm sorry. What point were you trying to make?

> The guy I responded to implied that the fact that it is for the police department made it more difficult.

Do you think that's an unreasonable assumption?

Forget whether the systems are actually mission critical or not. The only thing that actually matters when it comes to billing a customer are the terms of the service agreement. I've worked on government contracts where my company was significantly fined if the system did not operate at some minimal operational threshold. The system was definitely not mission critical, but we built it like it was and billed them for it. (We also got a bonus for getting the job done early)

I don't care if he's only working on the parking ticket system. If it's clear from the service agreement that it's going to require a significant portion of his time/energy to service it, the last thing you want to do as a consultant is to underestimate the effort.

I have no idea if this guy is taking advantage of the Baltimore PD or not. Neither do you. What I do know is $176K for two years is not a lot for an organization the size of the Baltimore PD.

> My position is that that has no bearing on the difficulty of the task.

I agree. The terms of the service agreement combined with complexity/maintainability of the system determine how much effort is involved. Again, life/death doesn't dictate the price. The service agreement and expected level of effort do. However, knowing a project is for an organization like the Baltimore PD, it's not crazy to assume they might be more demanding than other customers. State and local governments can be a real pain the ass sometimes.

My position is you're pretending like you know this guy's business. You don't.

> I have no idea if this guy is taking advantage of the Baltimore PD or not. Neither do you.

Never mentioned this. Never said boo one way or another.

Read what I actually wrote and not what you wanted me to write.

I said nothing about whether the cost is justified or not. I said the cost comes from getting someone who can program in Lotus Notes. That's all I said about it.

It is an unreasonable assumption to think that who you're building for makes the actual task more difficult.

> I agree.

Then what's up? You're agreeing with the only thing I'm actually saying. You had one guy saying it was trivial and another saying it was very difficult because it was for the police. I was saying it was neither. That it was equivalent in difficulty to another data management task. If anything people are downplaying the logistics involved in modern retail. Data management is a serious business.

> My position is you're pretending like you know this guy's business. You don't.

And his business isn't dealing with the police department. Because the guy I responded to isn't the guy in the article. So he doesn't know the business either. You are trying to chastise me for making an assumption when you assume I don't have experience, but no one else has claimed any experience in this matter either so their claims are just as invalid as mine according to your standard. They are also pretending to know the business. And so are you.

And by the way accidental and necessary difficulties are covered by Fred Brooks's essay, "No Silver Bullet". Accidental difficulties are difficulties we created. By choice of platform, technology, methodology, etc. So the system being a labyrinthine mess of Lotus Notes code is definitely an accidental difficulty. A necessary difficulty is something about the task itself that's inherently wicked.

Just manage to be the last company standing that stills supports the most obsolete technology possible.

To be clear; I'm not joking in that regards either. I used to do Foxpro work more than a decade ago and apparently it was still on my LinkedIn profile. About a year ago I got a call from a company that was super desperate; I turned them down because it had been a decade and I didn't feel like I was qualified to help them anymore, but they were desperate. Someone made bank off of that.

"While all you high-tech guys are starting complicated businesses that don't make money, City Hall has a contract for $100,000 per year to vacuum the floors once a week and no-one's bidding on it."

Reply on news.ycombinator.com