InfluxDB Cloud shuts down in Belgium; some weren't notified before data deletion
community.influxdata.com
community.influxdata.com
They wanted to have the euros flowing in right until the last minute.
1. Flash messages on all user facing consoles. 2. No new resource able to be created for 6 mo this. 3. Emails. 4. Service end date should have been at least a month prior to mandatory shut down. 5. Any people still running workloads in May should have had aggressive contact attempts made to ensure they were aware. 6. The console in the region should have switched to a final backup that can be exported by the user or moved to another region. This should have been available for 30 days.
You don’t do this because it’s fun, you do this because you need to save reputation. If I can’t trust you with business critical data then why would I use you for my critical business?
Also, as someone who works for a large enterprise, if you really believe email is the way to inform them of these changes, well I’d reconsider your beliefs.
> "The region did not get enough usage or growth to make it economically viable to operate, so it became necessary for InfluxData to discontinue service in those regions."
So it's worse than you believe. Yes, the handling is a scandal for all the reasons you say. But they weren't even pushed into this by some regulatory issue; it's pure cost-cutting.
A scream test would have been a better option in addition to the above.
“Yes,” said Arthur, “yes I did. It was on display in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying ‘Beware of the Leopard.”
I hope for at least their sake they took a backup of everyone's DB that could be restored in another region, but based on the fact that they didn't do a scream test, I doubt they thought about this either.
This must have been forced upon by upper management, because there is no way someone along the chain to actually delete data did not suggest a scream test. No way someone didn't say "this is a terrible idea, email is not reliable".
Adding Influx right next to GCP of providers I'm never using. Self-hosting is the way, and use ClickHouse.
> The Scream Test is simple – remove it and wait for the screams. If someone screams, put it back. The Scream Test can be applied to any product, service or capability – particularly when there is poor ownership or understanding of it’s importance.
The CTO seems to have been checked out for a long time (just look at how little developer engagement there is on here) and the CEO seems to have no idea how to run a DBaaS. The fact that nobody else from the company has stepped in to try and defuse this should terrify anyone who has data on InfluxData's cloud.
This is the beginning of the end. It seems like all of the good people have left the company, and being willing to destroy credibility to cut costs is a clear sign that the company is running on fumes.
So, now is the time - find your alternative, whether it's Timescale, QuestDB, VictoriaMetrics, ClickHouse, or just self-hosting.
It's the same "we 'tried'" message they have here. Even worse, this wasn't a regulatory shut-down, this was a lack of demand decision. They had 100% control over the timing and means of the shut-down. They didn't even keep backups! They just deleted everything.
Some highlights from the blog. It reads like a "cover my ass" to the board, rather than fixing problems for customers.
* > Over the years, two of the regions did not get enough demand to justify the continuation of those regional services.
* In other words, they had no external pressure. They just shut this down entirely on their own accord.
* > via our community Slack channel, Support, and forums, we soon realized that our communication did not register with everyone
* In other words, "we didn't look at any metrics or usage data. How could we have possibly known people were still relying on this?"
* That's literally unbelievable. They didn't even keep backups! They deleted those too! Even it the region is going down, I'd expect backups to be maintained for their SLA.
The least nonsense thing I can think of is that they weren't paying their bills. They weren't paying rent, the landlord locked them out and repo'd their servers, or something similar. (perhaps they were inspired by Elon Musk's recent antics?)
If that were the case, they would not disclose that that's what happened. If they disclosed that, all of their other customers would immediately begin migrating their data; not tomorrow, not next week, now.
If there were any excuse they would give it. "We were hacked!" "It was a disgruntled ex-employee!" "The datacenter burned down!" "It's those dirty EU data laws!" etc.
Shutting down the data center and deleting all the data (without migrating) at the same exact time and that was Plan A--nah I don't believe that.
The several things you might mean by self-hosting have their own pros and cons. The right choice is very context-specific, and assuming that it’s always the right choice is wrong. It certainly can be, though.
As for ClickHouse, that mention seems like a throwaway comment, unless you are advocating a boycott of even the open source InfluxDB due to its corporate author’s behavior and view ClickHouse as the closest alternative.
This incident has nothing to do with the comparison of the open source InfluxDB vs the open source ClickHouse, nor would it impugn the viability of InfluxDB hosted by a more responsible data custodian than Influx the company.
And GCP hasn’t done any similar inadequately notified shutdown of service with immediate and irreversible data loss, as far as I know.
(Disclosure: I have worked for Google in the past, including GCP, but not in over 8 years. I’m speaking only for myself here. I’ve never worked for Influx ClickHouse.)
Assume that your users won't see your emails. How do you help them avoid data loss when you shut down a service like this?
One option that I like is to take the service down (hence loudly breaking things that were depending on it) but keep backed up copies of the data for a while longer - ideally a month or more, but maybe just two weeks.
That way users who didn't see your messaging have a chance to get in touch and recover any data they would otherwise lose.
I'm not sure how best to handle the liability issues involved with storing backups of data for a period of time. Presumably the terms and conditions for a service can be designed to support this kind of backup storage "grace period" for these situations.
Stopping service and immediately delete data is just callous.
I think requiring a ticket might be a worthwhile trade-off compared to just adding the button, because that allows you to engage with customers to make sure they can (in a case like this) migrate to a different region of your own service, and the activation energy of sending a ticket means a customer's less likely to click 'Resume' and then forget about it again until it's too late.
Externally you communication: Different announcements each month, final notices at T+5M, System will be deleted at T+6M, data will be lost at that point and so on.
Internally (at least at work) such a timeline is more that at T+6M, we cut access to the systems. Afterwards, systems not accessed for 2-4 weeks are removed periodically and the hard removal is planned for T+9M. Customer support and account managers can manage if systems need to be accessed. If a customer needs the system for a longer time, they can, but then they pay for it. Entirely with all necessary infrastructure, not renting a few licenses on the system.
Call it a bit callous, but this allows our customer support to appear nice and in control. And it leaves the customer happy and relieved that we have left some slack and leeway. But they've been shaken and woken up and can get to migrating.
The biggest challenge here is to stay on it and to not allow customers to become complacent again. This can be done by e.g. limiting the reactivation time to a week or so so they have to get on it.
Agree absolutely wrt complacency, I believe I asked for less than a week because I actively preferred a situation where I had to get on it immediately.
Also you probably can't keep charging customers for that period since you offer a crippled service on purpose.
Brownouts are great for API changes, but not very useful before a full shutdown.
Internally at my company we always do scream tests as part of our EOL process because we know we can't reach everyone, even our own employees.
https://www.microsoft.com/insidetrack/blog/microsoft-uses-a-...
Fun story: my mortgage got sold last year. Not the first time. I got emails from the old mortgage company and the new mortgage company about the sale, but I skimmed them. I got letters via USPS from the old and new mortgage companies, but I mostly ignored those because 95% of what mortgage companies send me via USPS is junk. So I missed the fact that my automatic payments didn't transfer over. The new mortgage company let me get four months in arrears before they finally FedEx'd me something overnight. That got my attention. I was like: you guys should've FedEx'd me this in the first place. For all they knew, I wasn't getting their emails or letters in the first place because nothing had been sent signature required.
If that's the case, these companies/people have no business using cloud services. Fair enough that you might not understand the ramification, in that case you contact support. If you don't see those email... that's on you. We operate out of a number of datacenters, they all communicate via email, giving us one month to three notice regarding service windows. If we fail to plan for an outage because we didn't see an email, that's our problem. I don't know why anyone would expect more from a SaaS company.
For really large customers, I would assume that they have a customer service representative and yes that person should have called. If you're just a small customer (even if you might be big in your own mind) and just have an account that get billed to a company credit card each month, it's a little naive to think you'd get anything more than an email.
We've already seen a number SaaS company just shutting of customers for little to no reason, even AWS has done this. Running things in the cloud is a risk, and it's you job as the operations team to stay on top of things, have backup plan, because you cannot expect cloud vendors to care about some random customer who just signed up using a credit card and a nondescript email. They should, but they don't.
A good rule is: Don't expect a SaaS/cloud company to put in more effort contacting you than you did signing up.
>
> If that's the case, these companies/people have no business using cloud services.
Cloud services are responsible for this. I've signed up to many cloud services where I purposefully unchecked all the newsletter/updates/... notifications.
But I still receive notifications for stuff unrelated to what I use. These emails are full of marketing/PR jargon, where it's unclear whether I'm affected by the change or whether there is even a change!
Cloud services are lazy, don't look at their customer use, spam everybody, and blame their customers when they missed an important update due to noise-fatigue.
This is the main reason why I stopped using SaaS.
The fact is, there are many options from a cooldown, scream test, automated backup for migration/recovery... this organization did none of those things and absolutely deserves to lose massively as a result. This is a DATABASE as a Service... RETENTION should be one of the highest priorities.
For that matter, it would have been better if they auto-migrated in an OFF status, or otherwise backed up... just hitting the DELETE ALL DATA button is wrong. Several of the posters in the thread indicate they received no such emails.
I love the scream test, but the analogy you bring up actually - this seems unfair. The cost of Fedex'ing everyone is astounding (for many businesses).
But I like the concept. Definitely a sort of "shut off the server for like an hour" and then see who yells.
Phone calls for any account that still operating on the location.
100% agreed 3 emails is... hardly anything.
The final price is a combination of all costs incurred + a profit percentage imposed on the consumer.
If all customers would receive fedex mail, then the costs of this fedex would be pushed on them somehow. Probably by making them pay slightly higher interest rates or by introducing some one off handling fee.
> these [communications costs] do not pre-emptively get priced into future products by the sellers
They 100% do get priced in. Whenever you buy a product, you're paying all the costs of that product. When someone originates a mortgage, they're aware of the secondary market for mortgages. If that secondary market is eroded by a significant increase in communications costs, that reduces the willingness of a secondary buyer to bid for your book of mortgages. That erosion reduces your projected profit on originating, so you take a little longer to lower your offered rate to 5.250%, or you charge a bit higher origination fee, or whatever to ensure you maintain a viable business. So long as these fees hit the entire market, the other originators are all making the same calculations.
It seems odd that you [seem to] think that money for these costs would just result in reduced profits for the financial services companies rather than in increased borrowing costs.
I got what you and others are saying but I've been trying - and failing, apparently - to point out that the costs for a mortgage in case of a sale are born by the buyers who are not even the same kind of institutions as the parties that sell them, and so they are in no position to charge the subjects, nor are the sellers going to price the mortgages any higher in the future because the costs aren't born by them.
There is no such thing as 'projected profits' that go into this because mortgages can be sold (and are sold) more than once, the number of times is not known when they are issued first. And it is going to be only a small fraction of the audience that is going to be hard to reach for whatever reason. The presumption that there is some kind of free market mechanism that will ultimately pass those costs back to the original mortgage underwriters is not in any way evidenced by present day mortgage prices. On the sum total of mortgages out there and the - exorbitant - profits they create for the lenders we're talking about such small amounts that it will make zero difference.
A 'significant increase in communications costs' would translate into that being something that is some noticeable percentage of the total yield over the remaining time and it just simply isn't. Typical mortgage rates and amounts utterly dwarf the costs of a one time notification, especially if you don't have to notify everybody like that, and you can try cheaper channels first until you have a hit. Besides, the original mortgage 'service charge' is already a large multiple of the various costs and tends to be mostly pure profit for the initial lender.
It seems odd that you [seem to] think that mortgages are priced such that the mere cost of communications is going to show up in the prices, they are amongst the most profitable financial products.
If it made zero difference, they would Fedex the documents, but they dont.
I expressly don't use automated payments for my mortgage and auto loan(s) as I don't want to have an account miss... I didn't know my mortgage was sold/changed a couple months ago until I went online to change. The same happened with an auto loan a few years back as well.
I wouldn't be surprised if the cost difference is negligible, maybe a couple bucks per mortgage? FedEx bulk discounts can be pretty massive
I didn't realize there was an actual name for this, so I'll add another for the thread that I haven't seen brought up: email is evil in operations (EIEIO, like the nursery rhyme)
I kept sending checks to the old management company for a few months before the problem was discovered. Unfortunately, someone at the old management company apparently had a bit of a gambling habit and petty-thefted that money away, and I never saw it again even though she lost in a lawsuit.
https://abc11.com/nc-hoa-foreclose-sell-house-woman-didnt-kn...
To use an ecosystem analogy, once you expose your software to the world beyond your own dev environment, even internally, you'll eventually find that something colonized it - much like everything on this planet that isn't being actively and regularly scrubbed.
In my own career, I've seen cases of this. For example, once we were tweaking a little embedded database that supported a half-finished feature meant for internal use, and only then we (as in everyone in the dev team) learned that somehow, the QA & deployment support people got wind of it, and were scripting against exposed parts of that DB for a good year. And, it turns out, it wasn't the only part of the software that we thought of as incidental phenotype (or didn't think of at all), and the other team considered stable behavior.
See also the so-called Hyrum's Law: "With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviours of your system will be depended on by somebody."
That sounds the same, but it is not.
The first class is a set of simpler "happy paths" of intended specific behaviors.
The second class is like wargaming. A good way to do this is to replay user requests against your API and see that they return the same results release to release. You may also uncover interesting unintended behavior / conversations to have with users this way.
I would expect:
* Those 3 "email blast" notifications. I'm guessing one of two things happened here:
* You sent them as an "email blast" from a marketing-type email service. These hit email filters because they came from a known spam IP.
* You sent them as a transactional email, but blasted them too quickly and got pegged for spam. Never hit the inbox.
* Ideally, an automated transfer to another region with automated forwarding. It's okay to have poor performance, but it's not okay to go "poof" entirely.
* A soft-delete at the deadline, with 90 to 180 days to finalize migration. If this is costing you dearly, then drive prices up, but don't hard delete data.
Frankly, the last one is the real issue. It's literally unbelievable that a database provider didn't soft-delete. Further, I would expect that you'd be able to migrate these to another region to get customers back up an running.
* Important: migrate to this new feature immediately or you risk missing out!!
Vs
* Important: migrate your data immediately or you risk losing it!!
Xfinity is the worst about this. They’ll send me a so-called service-related email exhorting me to download their app. Same with Capital One and their monthly emails asking me to turn on automated texts.
I ignore most of my vendor's emails because they're simply trying to spam me at this point.
And the shutdown isn't even a month / quarter end, so seems even less like a billing cycle thing.
There's so many more mature ways to do a graceful paid service shutdown. Disable reads first so people get errors & contact support. Then disables writes as well. Somewhere a few weeks later you can consider deleting data.
As others have said, I've worked at megacorps with internal systems that had more mature migrations than this. Honestly I have run internal apps at sub-1000 employee firms where we did various forms of scream tests and soft deletes before MONTHS later, even daring to delete data on disk.
If the data is moving between countries then this is not an option. Your clients may have legal or contractual obligations with respect to data location.
Many people don’t have that option now since their data is dead and gone.
> We notified everyone via email on February 23, April 6 and May 15th. We also offered to help migrate all users. I realize that it's not ideal that we've shut down this system, but we made our best efforts to notify affected users and give them options to move over to other regions.
What other communication methods were attempted beyond just emails? Big, red obnoxious banners and warnings in various UIs? Phone calls?
The fact that it seems as though quite a few customers didn't get your emails, what was the thought process when looking at the workloads that were clearly still active before nuking it from orbit? Or was there no check and it was just assumed that people got the email and migrated?
Of the customers who were in that region, how many actually migrated? Was someone tracking these statistics and regularly reporting them to leadership to adjust tactics if there weren't enough migrations or shutdowns happening?
This screams either gross incompetence or straight up negligence. This is such a solvable problem (as many here have already mentioned various solutions), but I'm honestly just flabbergasted that this is a problem that is even being discussed here right now.
As a DBaaS, the data of your customers should be your number one priority. If its not, y'all need to take a hard look at what the heck your value proposition is.
We weren't impact by this directly, but you can be sure that this is going to be one of the topics for discussion amongst my teams this week. Mostly how we can either move off InfluxDB Cloud or ensure that our DR plans are up to date for the rug being pulled out from under us from you guys in the future.
Depends on where and how this message was added.
It also means that they had no monitoring in place to see how many people migrated.
Edit: They also say that this is reflected on the status page. Here's how their page looks: https://i.imgur.com/xlO4Ik2.png
Yup. It's literally a green status page that no one would give a second glance. That unreadable white on green? Oh. It's a deprecation message. It even has a subscribe link so that people would immediately and completely dismiss it as an ad due to ad/banner blindness.
Edit 2: Someone replied in the thread and added more context for the absolute lack of communication.
I am not a customer, but it’s really annoying me how tech companies repeatedly think sending emails is somehow anything but the absolute minimum, most lazy option.
https://www.v-wiki.net/scream-test-meaning/
Basically, you just turn stuff off, and make people scream, while they can turn it back on. You could have did this a month ago, as a critical warning of impending termination of systems.
You didn't do that.
Instead, you sent a few emails, which itself isn't even a guaranteed delivery. Again, as a CTO, you should know that email is not a guaranteed deliverable.
If you keep customer data after payment period, you're losing money.
So scream test will cost you money.
They decided that this cost is not appropriate payment to compensate for possible reputation loss.
Whether that's a good business decision or not, time will tell.
It sounds like they did turn the system off for paying customers though, why would it be any worse to do a scream test a few days before they pulled the rug out?
If it's just a matter of billing cycles (does everyone's billing cycle end on the same day?) , it seems like they could've handled it better. Just give impacted users a prorated and shortened final month (or even given it out for free for goodwill)
It takes years to build a good reputation and minutes to destroy it.
"We did our best - we have sent 3 emails". I wonder how shitty the product was when sending few emails is their best.
The guy either sounds like a full fleged VC psychopath, or someone very unexperienced.
All you've done is told your customers that their data isn't safe with your service. This was an easily-avoidable "own goal" situation.
And not only Influx's current customers, but also their future customers. I really like Influx for my homelab. But with this attitude, I would be really hesitant for a real world production deployment.
I think the attitude that “you pay us money so you better read every email we send” is at odds with reality.
1) why do you believe a single human being has that email address? As a company of any size I would never assign a human to a vendor email address. Turnover and rogue employee risks are too high for that. Usually these vendor emails are black holes only used to establish the account and recover credentials if needed. Or, it ends up in the hands of a vendor relations person who is more an accountant than engineer. Do you get the emails from GCP and AWS directly in your inbox?
2) because I pay you money I expect the opposite of the relationship you articulated. I’m not here to read your emails. If it’s really important then use my account manager to contact me. Make a phone call. Email is for spam - ESPECIALLY when it’s from a vendor. I view it as “I’m paying you money, vendor, so you need to go out of your way to give me excellent service” not “I’m paying you money so your emails are incredibly important to me I hang on your every dispatch”.
What baffles me is why on earth did you guys delete the data? I get you couldn’t afford to run the region for whatever reason. But you should have retained all the data. Storage isn’t that expensive.
Using billing contacts for this is a mess. The billing contact could be an accounts payable department that will check the invoice against the contract or PO and pay it. It could be an outsourced office that has no idea what a database is. It could be someone who only catches up on email once every few weeks.
What is isn’t in a technical contact who knows what the shutdown of a database means.
- they do soft-deletes before hard
- have robust access control systems and partitioning - so we don't have to give access to everyone in the org to object model with full r/w
- don't instantly nuke the account if a payment goes astray or delayed - try to reach out before to a point of contact before pulling the plug, payment systems can be messy for all sorts of reason, ask before assuming the worst.
- customer managers who can connect couple of times a year which usually benefits the vendor as upsells happens on good % of those connects.
- also small things like training, certification
- Deprecation of service is handled slowly(1 Yr would be expected) and in multiple phases with multiple modes of communication.
Not all companies can move fast to plan and execute a major change in location like this in 4 months, bare minimum you would have to consider
- End customers (your customer's customers) may need to be notified and may need to sign off
- Compliance and GDPR DPA changes - both end customers and internal ones
- DR, BCP concerns have to be planned for , not all GCP regions are equivalent.
- Documentation and certifications like SoC, ISO, PCI, HIPAA etc usually mean ton of paperwork to modify
- SRE/Devops may have to move other services along with telemetry on InfluxDB, may need network whitelisting from their customers, things typically break when moving, need to plan dry runs, rollbacks and so on.
You can definitely do better than shutting down service and deleting data at the same time .
[1] I would do this for internal customers let alone external paying ones
[2] You could have even charged for this to offset any costs, most customers wouldn't have a problem paying if they really needed it.
[3] Not trying to imply InfluxDB is doing these things, or isn't a good vendor, these are some criteria I have come to measure new vendors by.
Hetzner deleted my server just one week after my payment due date. My credit card failed the payment for some reason. I didn’t notice this because I was ill with Covid. They sent me one email (or at least, I received only one email) as a warning. I only realized the server was gone when my services stopped working. I’m not sure if such a short warning time is common practice among hosting companies, or if it’s unique to Hetzner.
After 5 years, they deleted everything 2 weeks after the first payment failure.
Sure it was the cheapest VPS. But still, you don't just delete your customers' data.
I was away from emails and the service during those two weeks. As far as I can tell it might have been some race condition in their payement processing system. They couldn't figure it out. They had no backup. They refused to reinstate the service anew to restore my own backup.
It looks like you had your own backup, which is always a good idea, hopefully you were able to restore your data elsewhere.
Perhaps it's because I pay for several bare metal servers, or because I have a business account with them. Perhaps it's because I pay their invoices by bank transfer manually instead of by credit card. Who knows! You have made me wary of changing to a credit card now, because those do fail from time to time!
What worries me more is Hetzner's reputation for suddenly dropping customers with no warning and no way to retrieve data from the servers. That's always on the back of my mind.
Strange. I've been at least eight days late with a VPS payment at Hetzner (3 euro) and the server is still up.
Also it is smart to have the threshold set to at least 1 cent, because this way you dont ask someone to pay you is supposed to pay you a fraction of a cent due to some rounding error. There are those stories where a company sends you a registered mail, where they ask you to repay a fraction of a cent - what is impossible. Also the cost of the letter (snail mail) made it not worth it. Even if you get an email that is "free" you cant pay 0.0001 cent. I mean you can pay a whole one and then ask to get 0.999 back - the time required by the bookkeeper to process it, then pay it (probably with a fee) is not worth it.
I can empathize with this, but also would expect a good product organization to consider failure modes here and work around them.
Did anyone consider that bob@company.com left months ago, but since autopay still works, no one considered potential problems?
Did anyone consider Bob in accounting is paying the bill but ignores email that doesn’t have “balance due” in it?
… and a million other scenarios that are quite likely and need consideration.
Everyone knows it happens, meanwhile every single company with high turnover is like this (those I've had personal encounters with):
Datadog: our domain has changed, but I cannot change my login. I've changed the email address in my profile, but I'm not sure if my login (which is an email address) is just a name or it may be used as an email address in some context.
Intuit: good luck changing your name
Apple Developer: still addresses the account as Bob No-Longer-Working-Here. It's not very clear how to change that name.
Apple ID: no, you cannot change the email address that had been primary back when you created it. And it better be a valid email address.
Orange: my address has changed twice, they are aware of that, they swore they updated my address everywhere; the invoices are coming with my old old address in their headers despite everything. Good thing they are sending them electronically at least so at least I receive them.
You likely can change the data there if you really need to, but it's very involved.
Someone should tell the IT/CRM drones that sometimes people not only leave the company, but also get incapacitated or die. In their Teletubby universes it doesn't seem to happen to anyone ever.
You've literally just told the world "you shouldn't rely on us for your data. When our business needs to drop you, we will and you might not receive notice."
You're hurting your company's reputation by denigrating your customers like that
Going read-only, waiting 2 weeks, and then deleting. The contact method is peoples’ alerting systems as writes stop working.
Putting a message on your service dashboard indicating the upcoming action. The contact method is exactly what it sounds like, and it’s the only other place you can stick text and know for sure all your customers can access it.
It will probably help customer relations if you don’t hide behind the defense of only having email - there are a few strategies for this that you can use in the future. Best of luck on the road ahead - I know this must be a particularly stressful time.
This is going to sound counterintuitive, like the Birthday Problem or Bayes Rule but at least for me it's true: most of the spam that gets into my inbox is from vendors I have a relationship with. Email isn't always ideal. Did you consider doing any other methods, like turning off writes a day or so before reads were disabled? That would trigger a much more immediate "oh shit" response than an email (unless the subject line is super clear and informative, and the email doesn't go to my span folder)
No, you're legally obliged to keep the service running. They are paying customers and even if you can't reach them through email there are other means of communications. If a business fails on account of your one-sided deleting of the data then you're going to be in for a very hard time, for instance a damage claim for gross negligence and breach of contract. This isn't just going to blow over. The onus for reliable communications is on you and if the channel you've got fails then you seek another one.
If you grow, have a merger, people moving, it is easy for email addresses to no longer be read. Yes there are best practices to prevent this, but most companies I have seen don't do that.
In one company important emails went to the email address (private!) of the founder, who left after M&A.
Explain carefully so their thick minds will understand that NOBODY is lazy enough to quit subscribing to your service just because you added an additional field to your onboarding.
Which most people would be reluctant to provide because everyone hates sales spam, and what else would a SaaS need your phone number for in regular times (impeding data deletion is a good one)? On HN making a phone number for a signup is regularly criticised.
Please make sure you can implement a TCP-type communication with your customers for this kind of critical movement.
We also didn't send any reminders for them to check whether their email address was up to date. No account update reminders. To prevent annoying people with spam.
So other than sending emails and hoping that they read it, there was nothing else we could do.
but there are other ways. you can put a big red popup that can only be dismissed by typing "i agree" when the customer logins, you can put the service into read only mode, even with email you can send daily reminders for the last 30 days with a subject like "your data will be deleted in 21 days", etc there are so many things that could have been done.
Now we have changed it so that the software never turns off even if license has expired (though it will continue to nag an email address). Updates also cannot be installed.
I get probably 100+ emails a day that hit my inbox in some form and occasionally fairly important ones get mixed in with the mostly dross though Gmail does a pretty good job overall.
Had the same issue with amazon though. In the flood of “Information about your order” emails one had some slightly different content (but the same subject): “We haven’t received your entire return. Please contact us in 14 days or we’ll trash it and charge you.”
When I contacted them a month later I was not very pleased.
Why do they need to move mountains so that you can avoid any seriousness about your own operations?
I have far bigger fish to fry than monitoring my inbox for shitty practices.
Deleting their data first and then complaining that your customers don't run a serious enough operation is not the way to keep the best reputation.
> Why do they need to move mountains so that you can avoid any seriousness about your own operations?
You won’t stay in business for long with that attitude.
This kind of attitude will ensure you lose your customers.
Further, they might be going to some alias/group that's not frequently monitored. If a vendor is going to delete all of my data, I expect way more noise than 3 random emails blasts.
Because it's burried among lots of other similar looking emails that's just marketing garbage?
Because the email wasn't delivered?
Because the customer was on vacation and they expect a reply very quickly?
There's plenty of valid reasons to miss an email.
Or suppose an employee did have the email on file and left the company.
Or suppose people assume vendor email are spam because they’re almost always spam.
If you have 20.000 engineers (or even 200) you have a functional service desk and I assure you that no individual engineers email is given as the email for vendors to contact. Even for large contract where you have a preferred contact on each end, there's an escalation path.
Your model is a smart one. It’s smart enough it tells me you’re either a small company or a newer company, or both, or a company with a rarely together vendor management team.
You call three emails (the last of which was almost 2 months ago) "best efforts"?
I had to read your message three times because this is so reality-defying preposterous I just couldn't believe I didn't miss anything. How about warnings on the dashboard? How about an intentional error (or limited service interruption) so that people would log in to their dashboard?
They did have a warning on the dashboard, problem is a lot of people don't check the dashboard because they don't need to, as they just view everything through grafana, etc.
They also had a notice on the status page
You might have literally deleted people's whole businesses, companies, who employ real people, who have families, now need to figure out how to continue. Not least of which, your own. If the company survives until Christmas I will be shocked; no one can trust your company ever again - your core business is storing other people's data, and you deleted it, for many, completely without warning.
I guess people still use Mongo even after finding it doesn't achieve any property of the CAP theorem, maybe some people will keep using a database provider with a track record of intentionally deleting their paying customers' data.
There just aren't enough adjectives for astonishment to adequately describe this situation.
I hope you offer Jay Clifford some support, he's clearly been put in the awful situation of having to explain the decisions of others and deliver the awful news. If I were him, I would be in need of serious mental health support, this is an absolutely awful thing to have responsibility for without any ability to rectify.
This is just regular old incompetence/negligence/greed.
Did you use the same email you use for spam/"marketing" for this notification?
The correct course of action is to shutdown the service and give people time to fetch data, not to erase the data as the first indication of shutdown.
A few emails are not sufficient if the end result is dataloss, a comment in documentation or release notes is not sufficient (the only reference at least one person in the referenced issue found).
truly mind blowing behavior.
Then again, I was only using the free tier, so I guess I got what I paid for.
Not ideal???
You backed up everyone's DB and moved that to another region so they can just restore and change DB endpoints, right?
I don't believe that someone along the chain didn't suggest a scream test or similar. If they did, they must have been ignored.
Trying to assert that you were doing what you thought was right only presents the image that your company is run poorly.
The correct thing to do is to admit that your best efforts were not aligned with best practices, and look into remediation.
Not “well, we tried”
My opinion on best effort: I founded, ran, and sold a SaaS company used by some of the most well known companies in the world. Our "best effort" was a minimum of 12 months notice, with a six month grace period afterwards. Emails weekly. Phone calls at least once a month. Reach out to customer leadership if no response. Then scream test as others suggested.
It is astonishing that you have literally completely ignored one of the primary USPs of your product.
Then recently they made a change to Telegraf that broke all our data collection because they changed the environment variable replacer and their Jsonnet parser broke.
Now this. Shutting down a region without brownouts and only emails is not operationally acceptable.
We’ve moved on from influxDB for a while and only rely on telegraf now.
Also, they did remove clustering in the open source version which was a very poor move from a PR perspective. And in my view, they have never recovered from it - years ago it was Prometheus vs InfluxDB for (non-SaaS) observability metrics, nowadays the only question is which backend for Prometheus to choose.
I sometimes wonder if vendors realize that they put their customers into a buying mode when they do this, when our options are:
- Upgrade to the new version of product X.
- Change to vendor or tooling completely as we're already changing everything.
We might pick another system if we feel like it is more stable.
Either there should have been a breaking change in there or they don't understand symantic versioning
I still remember this. We were ready to standardize on InfluxDB when we got a taste of their business practices.
Deprecation of services is a common occurrence at AWS and many other tech companies. But it's never taken lightly. A mandatory step in this process is analyzing usage logs. We need to ensure customers have transitioned to the alternative. If they haven't, we reach out. We understand why. The idea of simply "nuking" customer data without a viable alternative is unthinkable.
The InfluxDB incident brings to light the ongoing debate around soft vs. hard deletion. It's unacceptable for a hard delete to be the first step in any deprecation process. A clear escalation process is necessary: notify the customer, wait for explicit acknowledgement, disable their APIs for a short period, extend this period if necessary, soft delete for a certain period, notify again, and only then consider a hard delete.
The so-called ["scream test"](https://www.v-wiki.net/scream-test-meaning/) is not a viable strategy for a cloud service provider. Proactive communication and customer engagement are key.
This incident is a wake-up call. It underscores the importance of data durability and effective, respectful customer communication in cloud services and platform teams. Communication is more than three cover-your-ass emails; it's caring about your customers.
The ordering of security and durability very much depends on the needs of the customer.
Some data is vastly more valuable to malicious actors than it is to you, e.g. ephemeral private keys. If lost you can simply replace them, but if (unknowingly) stolen it can be disastrous.
Other data is vastly more valuable to your than to malicious actors, e.g. photos of sentimental events.
Interestingly, this is also how I'd allocate tasks to new admins. Like, sure, I'd rather have my load balancers running, but they are stateless and redeploy in a minute. The amount of damage you can do there in less critical environments is entirely acceptable for teaching experiences. Databases or filestores though? Oh boy. I'd rather have someone shadow for a bit first because those are annoying to fix and will always cause unrecoverable loss, even with everything we do against it. Hourly incremental backups still lose up to 59 minutes of data if things go wrong.
> The InfluxDB incident brings to light the ongoing debate around soft vs. hard deletion. It's unacceptable for a hard delete to be the first step in any deprecation process. A clear escalation process is necessary: notify the customer, wait for explicit acknowledgement, disable their APIs for a short period, extend this period if necessary, soft delete for a certain period, notify again, and only then consider a hard delete.
Agreed. At work, I'm pushing that we have two processes: First, we need a process of deprecating a service and migrating customers to better services. This happens entirely at a product management and development level. Here you need to consider the value provided for the customer, how to provide it differently - better - and how to decide to fire customers if necessary. And afterwards, you need a good controlled process to migrate customers to the new services, ideally supported by customer support or consultants. No one likes change, so at least make their change an improvement and not entirely annoying.
And then, if a system or an environment is not needed anymore, leadership can trigger a second process to actually remove the service. I'm however maintaining that this is a second process which is entirely operational between support, operations and account management. It's their job to validate the system is load-free (I like the electricians term here), or that we're willing to accept dropping that load. And even then, if we just see a bunch of health checks on the systems by customers, you always do a scream test at that point and shut it down for a week, or cut DNS or such. And only then you drop it.
It's very, very careful, I'm aware. But it's happened 3-4 times already that a large customer suddenly was like "Oh no we forgot thingy X and now things are on fire and peeps internally are sharpening knifes for the meeting, do anything!" And you'd be surprised how much goodwill and trust you can get as a vendor by being able to bring back that thing in a few minutes. Even if you have to burn it then to turn up the heat to get them off of that service, since it'll be around forever otherwise.
Incidentally at work we've been evaluating a new hosted observability provider, looks like we can rule out Influx as an option.
I don't see any indication that the person who posted that is associated with InfluxDB. In fact, it doesn't seem like any staff member has posted in that forum in the past week. Up to you if you consider that better or worse.
It could be them not sending notification about it early enough or at all, it could be that note being stuck in spam or something, it could be person complaining not reading the e-mails.
I wouldn't jump to conclusions here
- self-hosted
- provided by a big deep-pocketed cloud infra
- Phone calls
- Scream tests
- Monitor services still in use. Contact these customers individually
- ...
Not a single individual said "Gee, people are still using that DC, should we really destroy it?"
Either this shows Influx is really naive and inexperienced or... they are in deep trouble cash-wise and were working in panic mode to cut costs.
If true, this is concerning. One message getting lost in spam understandable. But three over 6 months would imply they're being blacklisted and/or their mail sender is simply broken.
Do serious companies not have canaries or other checks in place to ensure their notifications are correctly delivered to customers?
For these kind of automated emails getting all emails consistently being delivered to everyone is really hard, almost impossible.
The problem here isn't really that emails weren't being delivered, it's that they seem to have tried only one method to contact people, didn't check how successful that was (e.g. by seeing how many customers were still on those regions), and seemingly never tried anything else (such as notifications on the dashboard, a temporary brown-out to alert people, etc.) – "we tried one way to contact you and that didn't work, so we just deleted your service sucks to be you lol kthxfuckitybye"
A financial service I use was recently purchased by another. The company has been aggressive in keeping me on the loop about what is upcoming. Maybe six months before the actual event a heads up. Again at two months. Then at one month. Then every week, along with countdowns to the deadline. "Are you ready? This is really happening. Here are relevant docs on how to ensure your transition goes smoothly."
Even with a canary, you have no guarantee that a particular customer will receive your email. In this case, maximum CYA is a certified letter sent return receipt requested. There are services out there to mass mail those.
Plus some enterprises change their mail policy out of nowhere. I have a lot of subscribers from enterprise companies, but sometimes every mail delivered to some company comes back with something like "Your address is not on our whitelist. We only accept approved emails." and every subscriber is hard bounced off the list and that's that. (They aren't paying me, though, so less of a big deal.)
We had even tried out Influx a few different times. It was always ok, but never quite good enough. Now with this, I think, this seals the deal on me ever considering Influx either as a product or as a service.
Our team maintains an internal CRM. When we plan to delete data or deprecate features, what we usually do (beyond sending emails):
- hide features/data from the UI without actually deleting them - if no one complains, after a few weeks, proceed with the removal
- for critical data, make sure there are backups, store them for about a month - if no one requests them, delete them
First time: we chose Timescale over Influx (and a few other competitors). I really liked Timescale. This was ~2.5 years ago and obviously Timescale was much less mature than today.
We also sent data to AWS Timestream. We're heavy AWS users and got to try out the product. I found it ok, but it was expensive even for us (Disney+).
Second time: team migrated from Influx to Clickhouse for server and network metrics. Services had been relying on that data being correct and timely in order to route traffic, and well...there were issues. The simplest solution was simply to replace Influx with a product more suited to handling high volumes of metrics...yeah, ironic.
I've been bitten by the old Influx and had to migrate to something we could trust... (Influx basically tacitly admitted that the original architecture was pretty poo and they've since swapped out the engine (twice?), but it smells a bit like mongodb trying to reinvent itself and distance itself from all the early web-scale claims, so I'm kinda skeptical).
So I've rolled our own with MySQL+tokudb, but that's not a good choice for a new system as tokudb is disappearing. When I tried to migrate to MyRocks we discovered the newer kids like MyRocks don't really work nearly so well for specifically this kind of use-case.
Something I haven't personally tried, but have heard rave reviews of, is Timescale. Its a special storage engine for Postgres and it has a lot of nice features like auto-maintained rollups. And they have lots of deep technical blogposts that I find myself agreeing with, so it must be good! :D
I guess influx perf and efficiency is really depending a lot on your data shape then :)
Our experience was that performance dropped off a cliff if you had too much data, too much tagset cardinality, or else your query was too broad. And when it failed, it lost data.
In fact, it lost data generally. When we were replacing it we dual-ran an ACID DB version (which, with tokudb, was fast enough to keep up (although we didn't index every tag column)). So we did a diff and discovered just small random holes in the influx data that we'd never noticed before.
We had other considerations when we went mysql, as in, we were already using it. If shopping for a standalone solution to start a new project on, I'm thinking Timescale is the go-to these days?
I agree with GP about storage efficiency, which is superb. Query performance is good as long as a single query doesn’t deal with more than ~dozens of series. And $deity help you if you want to do hourly roll-ups of all series for a short time range, as RAM usage is wildly unpredictable. Storage is optimized for long reads of a single series, not for short reads of many series (but in fairness, you have to choose one or the other, that’s just the physics of the thing).
If I were starting from scratch, I’d probably pick Timescale. Or maybe DuckDB… I wonder if it would work for our use case.
I think I went properly mad while trying to troubleshoot this. The same query sometimes pulls 5GB, sometimes 20GB, sometimes 50GB and sometimes OOMs at 200GB memory pulled beyond base load of the system. And there's no query planner, no execution log, no metrics to help you. And most documentation or threads about it can be summarized as "well sucks to be you, eh? Maybe less data would be an option I guess"
We don't do that anymore and just roll up a very small select number of metrics.
And yeah, we've committed to Postgres as our main DB 2 years ago or so, and currently time is clearing up to start work with TimescaleDB. Zabbix is supposed to be great with it.
However, I do feel that they are trying to really do the right thing with the new 3.0 architecture, addressing the deficiencies (most importantly performance and full-fledged SQL) while keeping the stuff that works (InfluxQL for simple and legacy queries). Also leveraging open-source projects and contributing to their upstream is a plus. Thus I’m hoping for them to succeed delivering on that promise.
However, once your storage layer is parquet and your query layer is SQL, well... DuckDB is also basically parquet+SQL, and it won't be long before there's a nice Postgres wire protocol adapter in front of it. What's the advantage of continuing to use InfluxDB if you don't need clustering or HA?
[1] https://valyala.medium.com/insert-benchmarks-with-inch-influ...
[1] https://valyala.medium.com/promql-tutorial-for-beginners-9ab...
This situation doesn't seem like it is specifically them doing anything wrong, if indeed they did send out multiple notifications over 6 months. It sounds like it's caught many people by surprise though, which makes one wonder if there was a problem with their announcements of this change. Definitely seems like it could have been handled better than "deletion with no recovery" though, some sort of "shutdown, wait a week or month, delete" would have been better.
EDIT: Also, does the company have a Sales team? Why is that not a top line item for representative to contact those who have active service in the region to emphasize the shutdown? Or was the Sales team similarly in the dark about the messaging?
I literally get more notification for DNS expirations (and those are scheduled and predictable).
TLDR: “We don’t plan to delete your data again in the forseeable future”
Full quote: “You can sign up for a new account here InfluxDB Cloud using a different region. We want to assure you that there are no more scheduled shutdowns planned. Therefore, once you have created the new account and begin writing to it, we do not foresee any data loss going forward”
No more information was provided in the mail.
Three emails clearly wasn't enough right? Now their name is in the dumps, customers are pissed and my only exposure to influxdb is a negative one.
I hope other saas guys learn this very expensive lesson.
Hope they didn’t have any big corp customers impacted. Some big corps would very easily use that to cancel contracts and void out payments. Then let the lawyers deal with it
We have a netflow analyzer with more than 350 b2b customers (regional ISPs) and we use to run InfluxDB as TSDB. A few things were bugging us though:
1) influxdb is being rewrited for the third time in less than 5 years;
2) v2 in hindsight was actually a downgrade from v1 in regards of performance (e.g: drop shard mechanism);
3) will they finally solve cardinality in v3? that was a major issue to be solved by v2...
I was just not confident sticking with influxDB. Thankfully another OSS project really surprised me in terms of performance and reliability, which is questdb.
Now we migrated more than 100 of our base and hopefully we will get all migrations done by the end of the year.
For my use case, there's one feature left to be add which is the inet type, in order to store IP addresses more efficiently.
What value proposition were they providing that is so great that you were willing to risk your critical data by putting it into their service when they have done nothing to prove that they are worthy of your trust? what kind of thought process leads to this kind of decisions? someone, enlighten me. I really want to understand.
>Unfortunately @RoyalBlock at this time it does look like the data is lost for Sydney. We are still looking into it but I am 90% sure this is sadly the case.
https://community.influxdata.com/t/getting-weird-results-fro...
>Data recovery is still in process. If you where part of the Belgium customer you will receive the last 90 days of your data. Support will email you directly once this process is ready.
Better than nothing for the Belgium tenants, I suppose.
Find your way into that Slack workspace via https://www.influxdata.com/blog/introducing-our-new-influxda... and you'll discover some interesting things:
* That the only reason being given in that thread for the shutdown is "The region did not get enough usage or growth to make it economically viable to operate, so it became necessary for InfluxData to discontinue service in those regions.". i.e. there's no regulatory issue here like other answers speculated - just pure cost-cutting.
* That on July 5th, a couple of hours before they started shutting everything down (based on the shutdown timeline at https://status.influxdata.com/), that same "Developer Advocate" announced that they were suspending their live "office hours" sessions for July.
* Multiple people are asking for help after finding that they can't connect and getting ABSOLUTELY NOTHING in the way of support from the company. It's literally falling to _other users_ to tell them that all their data is gone.
* One person chiming in, Matthew Allen, DID have a colleague who saw the notification email, but notes that...
- it was a pain for him to migrate has data to a different region due to InfluxDB Cloud's rate limiting, but that he did it anyway
- ... but that the documented migration process doesn't seem to have worked properly anyway (given that some of his data points have ended up as nulls)
- plus on top of all of THAT, even after doing everything he was supposed to do, he still can't log into influx cloud after the migration because when he logs in he gets automatically redirected to the no-longer-existing cluster and hits an error screen
What a clusterfuck. Shame on everyone from Influx who had a hand in this - the CTO Paul Dix who's turned up here on Hacker News to blame his customers for Influx's negligence, Jay Clifford the Developer Advocate for a spectacular failure of developer advocacy, and anyone else on the team who was close to this and didn't push for brownouts, blog posts, a mention in the newsletter, retention of data for some window after the final shutdown date, or any of the other obvious measures that could've made this less of a catastrophe. The multiple people noting that they were able to receive the company's newsletter but did not receive the notification that it was about to delete all their data tell me everything I need to know about this company and its priorities. I will never willingly do business with them again (although we used them at a startup I worked at once) and will advocate against any time I hear a colleague suggest using them.
I know ap-southeast tends to be pricier, I'm assuming Belgium was the same.
https://www.concurrencylabs.com/blog/choose-your-aws-region-...
Out-of-band data, like emails, is bound to be ignored by some.
I'd incorporate warnings in the application-specific responses; you can also return a different response code to make sure many clients do not blindly ignore warnings. The HTTP standard includes a 207 Multi-Status response code, which is mostly used with WebDAV.
It feels like that could be a good practice, or not, depending on the laws in question.
They might not have been able to do it automatically as the region name were hardcoded in the hostname.