Hacker News Reader

Comment by kgeist | Hacker News Reader

We use on-premises setups for almost everything (we generally avoid cloud solutions to have full control of our data), sometimes (approximately once a month) it goes down for a few minutes which already feels like a torture because all our processes depend on it, I can't imagine having no access to it for several weeks, all our work would stop to a halt... The office of the guy who administers on-premise servers is literally next door, all it takes is to make a visit to him and everything works again after 5 minutes. Reading horror stories like this (Slack being down, Atlassian being down, no one knows what is happening and when it will end etc.), I wonder why many companies choose cloud solutions for critical business processes. Is it pricing? Ease of use? I can understand why very small companies would choose it, but I don't understand why a medium/large business would choose anything but an on-premises setup.

View on HN

> but I don't understand why a medium/large business would choose anything but an on-premises setup.

Atlassian is in the process of killing the on-premise small/medium business option, already announced an EOL date.

Move to the cloud, buy a 500+ user solution for a much higher price or migrate away are my choices. Of course I use the local database and have local services JIRA/Confluence talk to so it's not really an option to move to the cloud.

I assume lack of competent on-site staff 24/7, having someone else to blame as well as lower costs are why people choose the cloud over on-premise though.

I am biased but I can tell you what works best for mid-large companies: having a solution provider. Basically a partner that hosts and maintains the instance and has enough Atlassian certified people to help you with any question so that you will never have to hire people to just maintain the beasts or tell you about features, tricks or plugins that could solve problem X.

Experienced people hosting and tuning Atlassian products has a greater success rate than someone doing it alone for a large company. Almost every time I’ve migrated an old Atlassian installation under our wing it’s given me shock how users have been made to suffer the loading times and perfs that come from underprovisioning (db or actual machine) and messy configuration. I’m not blaming the former admins but it just happens. Usually end users are happy after we clean the mess up and everything feels snappy.

Disclosure: I’ve worked in this kind of expert role.

It seems like if you are going to pay for a bunch of SaaS seats AND a team of technicians/engineers for make it work, you might as well just do the latter and roll your own solutions...

A lot of these SaaS are just glorified Rails apps with a patina of professional "security" and "reliability", and loads of extra junk that your co will never use.

Trust me, if someone could clone Jira and its functionality they would have done so already. Truth is that if you build one product for 20 years you have a giant lead in features. If all it took was having a Kanban board then Jira would have died years ago.

isn't it one of those "No one ever got fired for buying XXXXXX" type of situation?

Maybe i'm wrong, but the impression I had of Jira is that just like using sharepoint for file storage, the C-level people want it because they were told that's what big enterprise are using. And if it doesn't fit the need of the company and everyone hates it, they just blame the employees or lack of training.

It’s just so flexible that tracking projects and working together is easier with it. Decades of feature requests have made it good for people who want everything made for them or people who want to customise.

I can't see the difference between a "solution provider" that hosts your Jira and just getting Atlassian to do it. What's stopping the solution provider from accidentally running a script that deletes some customer's files and struggling to do a partial backup restore?

I would assume the MSP is running a dedicated instance and can do a full/backup restore just for the user they're supporting.

If it's some multi-tenant solution it's no better.

Correct. There are probably not a lot of MSPs that have so many customers that they need to share that much data, and their customers probably use MSPs for the strict purpose that they don't want to share things with other companies.

Because you can get the best parts of self-hosted and managed services. And on that backup question: self-hosted Atlassian is vastly easier to protect against disasters. The problem these Atlassian guys had arose from multi-tenant architecture. Usually managed service providers will host your stack on individual databases and VMs, and backing up the software is just a matter of taking pg_dumps and rsyncing certain directories (pretty old school) or just taking disk level snapshots.

Many medium-large corporations have their own cloud environments that their IT Ops control. Solution providers can host Atlassian stacks on their own cloud environment where they are not affected by data privacy concerns (it's in their already green-lit cloud providers data center) so they can host it behind a firewall with only VPN access allowed. They can also do all the magic you can usually do with web software like put a frontend proxy in front of it, or use more flexible/legacy authentication methods. Not to mention that for example you could have a Jira Cloud that you would need to integrate with a SCM program. Jira data could be "OK" to live in the cloud but code would be a big no-no. These problems can be solved by having them all live behind the firewall.

A competent managed solution provider also has consultants that can train or instruct on usage. It costs but it is simpler and faster than having to go through the forums or send a support ticket for every small issue to Atlassian itself.

We migrated from Slack to self-hosted Mattermost so we avoid being down. (And I guess money.)

Mattermost is so much worse that the slowness and general issues are not worth it. And in the end it is more down than Slack ever was, because it has performance issues.

I am not sure if it is Mattermost fault or our fault; but my friend from other corporation has similar experience with it. But maybe in general just don't know how to host MM, I donno

We still haven't taken IRC down because it's our backup for when slack goes down.

I swear if IRC just implemented emojis.

we have WhatsApp as Mattermost backup.

Lately we use it more than mattermost :)

Cloud solutions can work well. I've used GitHub, Azure Devops, and BitBucket (another wonderful atlassian product /s) and BitBucket frequently craps out, multiple times a week. We need to rerun builds in TeamCity because BitBucket stops talking to it.

Bitbucket is so slow and unreliable for us we regularly refer to it as Shitbucket. About as useful as a bucket of crap most of the time.

What do you do if your on prem setup lost data? There is an implicit assumption here that on prem is more reliable than cloud. Less downtime, less chances of data loss etc. Obviously it depends on which cloud product we're talking about but I don't think a blanket "my on prem goes down less and when it does go down I can get it back up sooner" is true.

I think for that question we also have to define on-prem just to be clear. To many on-prem means "own cloud subscription".

What do you do if the on-premises guy gets hit by a car and isn't in his office?

There's IIRC 3 or 4 people in their department, they administer the whole building (wifi, security cams, LDAP, etc.), not only the on-premises servers. From what I gathered, our internal systems usually go down due to lack of disk space or some bug in the software which requires merely a reboot, it's not rocket science. Another thing is that our IT department (for internal systems) and the SRE department (for client-facing systems) have 24/7 on-call duty so it's unlikely that no one will respond.

The same thing that the cloud company would do. If there are other people there who share that guy's responsibilities, have them do it. If there aren't, you should have an on-call.

Cloud just outsources that problem to another business. Sure, they have better reasons to actually cover those positions and make sure they have on-calls and backup and a disaster plan, but just because you pay extra money for it doesn't actually make it work better if the company underlying it sucks.

What do you do if your cloud provider is down for 3 weeks?

There are always technical and economical pluses and minuses to any approach - but never underestimate the politics and character flaws that see dubious decisions pushed through by senior management regardless of those rational arguments pondered by the lower orders.

You're assuming every team would have better uptime with in-house solutions

I think many would have worse uptime even with more headcount

I've heard this argument many times before, but is there research into this? I.e. where they would compare uptime of cloud vs. on-premises across a wide range of companies.

I mean, you're going to get biased results, no? Only companies who are confident in self-hosting will self-host it. You won't have any real data about companies who are not confident in self-hosting maintaining their on-premises version of the software.

In our experience, this strongly depends on the services involved, as well as the scale.

For example, for our own service: If you have a hundred or two hundred licenses, you can drop our system on a linux box and usually you have to throw a yum update and one or two service restarts at it every few months and it just works. I honestly wouldn't be surprised if many of our small on-prem solutions have better uptime than the SaaS clusters, or be capped in uptime by some externality, rendering the system downtime irrelevant. If their VMWare cluster is down, our system is down, but no one cares.

This also mirrors a lot of our internal systems. At a small scale, you can just dump chef, jenkins, sonar, nexus, whatever on a linux box and forget about it.

However, this changes with high license counts. We have singular customers in our SaaS offering that are more than 50 - 100x bigger than the small on prem systems. At that point, our SaaS offering is better than anything the customer could to on-prem. I'm confident to say this about all of our customers, except maybe 2.

I find this argument to be totally bs these days.

If anything, a smaller company with smaller footprint and fewer total requirements is going to be more likely to manage a vertical slice of some SAAS product.

The reason things like github go down so often is because they are public/shared resources.

>The reason things like github go down so often is because they are public/shared resources.

Very much this. Managing shared resources at scale is pretty hard. We have a bunch of internal sites made by interns as part of their internships, and, funny enough, those sites have much greater uptime and appear more stable than our own multi-tenant SaaS solution made by seasoned devs.

Reply on news.ycombinator.com