Oracle Cloud is unreachable
ocistatus.oraclecloud.com
ocistatus.oraclecloud.com
> This incident affects: Network Infrastructure (South Korea North (Chuncheon), India South (Hyderabad), Australia Southeast (Melbourne), India West (Mumbai), Japan Central (Osaka), South Korea Central (Seoul), Japan East (Tokyo), Australia East (Sydney), Canada Southeast (Montreal), Canada Southeast (Toronto), Netherlands Northwest (Amsterdam), Germany Central (Frankfurt), Switzerland North (Zurich), UAE East (Dubai), Saudi Arabia West (Jeddah), Chile Central (Santiago), Brazil East (Sao Paulo), UK West (Cardiff), UK South (London), US East (Ashburn), US West (Phoenix), US West (San Jose)) and
> Console (South Korea North (Chuncheon), India South (Hyderabad), Australia Southeast (Melbourne), ap-mumbai-1, Japan Central (Osaka), ap-seoul-1, Australia East (Sydney), ap-tokyo-1, Canada Southeast (Montreal), ca-toronto-1 region, Netherlands Northwest (Amsterdam), eu-frankfurt-1 region, eu-zurich-1 region, UAE East (Dubai), Saudi Arabia West (Jeddah), Chile Central (Santiago), sa-saopaulo-1, UK West (Cardiff), uk-london-1 region, us-ashburn-1 region, us-phoenix-1 region, US West (San Jose)).
So basically the entirety of Oracle Cloud is unreachable. Oops.
Edit: If you were wondering if Oracle really has that many regions, it's actually the same list repeated twice for "Networking" and "Console". Which means both the data and the control plane seem to have fallen off the face of the earth.
The main reason we won't consider using Oracle Cloud is simply because Oracle has already demonstrated their sheer contempt for their own customers through their on-prem database licensing program, so I dread to think what they would do to enforce their IP rights or ToS with their cloud customers.
Another reason we disregarded them (and GCP[1]) is because they simply aren't _big and successful enough_ to give us the confidence they'll keep on running into the forseeable future and with the kind of reliability we expect - and this outage incident demonstrates that they clearly aren't. I know that Azure and AWS have both had significant outage incidents in the past, but both have been rock-solid stable in recent years. For this to happen at Oracle suggests something is institutionally wrong: if they can't get 99.999% uptime on their own worldwide Cloud operations then I cannot trust them with our own SaaS platform because we promise 99.999% to our own customers.
[1]I'm wary about GCP because Google hasn't issued any kind of promissory statement that GCP won't end up on the [Google Graveyard](https://killedbygoogle.com/)
Isn't that literally what an SLA is, a (contractually) guaranteed level of service for some interval of time?
Well, IBM did kinda cheat: rather than go with Microsoft's approach of making the platform backwards-compatible (e.g. by not fixing bugs that programs depend on, or introducing shims, or redirecting/intercepting calls from known older programs), IBM introduced some pretty hefty OS virtualization technology to their platforms decades before x86 virtualisation was a thing - which I imagine makes back-compat much easier.
P.S.: I still remember when I told my dad I had this cool new thing, VMWare now. There were a few other "yeah, we had that on the mainframe X many decades ago" get-off-my-lawn moments like that :)
Java isn't Oracle's though. It was created by Sun (released in 1996) and has had a community driven process for language changes since 1998 (jcp.org). Oracle only bought it and openJDK is a thing.
The terms of use for GCP does say any product that is shut down will give at least a one-year notice before it's shut down[1]. For any given product within GCP that has been shut down the lead time has been greater than a year as far as I'm aware.
The fact that Google is calling out GCP separately on its earning calls is actually somewhat of a commitment to trying to grow that business. Very few products within Google are called out separately, so that was a fairly big change.
[1] https://cloud.google.com/terms/
> (d) Discontinuation of Services. Google will notify Customer at least 12 months before discontinuing any Service (or associated material functionality) unless Google replaces such discontinued Service or functionality with a materially similar Service or functionality.
Or am I missing something?
That's not a given.
At least in the early AWS days, internal sites had special configurations.
Source: worked there.
But in recent years, they run entirely on AWS. They even have solutions architects to help them build things in a resilient way. They also spent years ripping out Oracle and replacing it with RDS MySQL, PostgreSQL and Aurora.
It is true that Amazon runs on AWS.
Source: worked there.
...then they go and buy more of some the most expensive office space in the world per-sqft.
Source: I interviewed there 3 times and every time left me with a dystopic impression of the company. To add insult to injury their recruiter only ballparked me only slightly above half of my then-TC.
How is that a negative? Seems like a more efficient way to operate a company.
> Seems like a more efficient way to operate a company.
I do actually agree with you - it's just a matter of framing the company's practices.
https://news.ycombinator.com/item?id=21258127
That's what I was getting at. There's no way AWS is going anywhere if the entire store is running on it.
true or a team is slowly migrating so one day true
> Google does not run on GCP.
Ive heard other googlers say "GCP runs on google" as a somewhat tongue-in-cheek reference to the internal architecture
Think about components you rely on for your service. How long would it take you to stand up or migrate to an alternative. Consider that you've also got all your other business to do too. You can't spent a year just spinning your wheels from a business perspective, while you're migrating stuff between services because your upstream provider decided to stop building something. You've still got to keep growing. If you're lucky, you've got staffing for meeting your business growth needs, but odds are that's barely true.
On top of having to migrate, what are your replacement's performance characteristics? Are you replacing it with your own solution instead, and if so how do you operate it, what happens under load, how do you ensure durability? etc. etc. lots and lots of questions and you've got scant 12 months to figure out what you're doing and how to get off it _on top of your existing business needs_.
This criticism has been brought up so many times, and yet routinely we still see GCP saying "but we promise to give you a whole year's notice!" as if that's a good thing.
(disclaimer: I work for another cloud provider)
Also, I'm guessing larger companies could push for certain longer deprecation policies as part of their contracts on certain features. I have no insights here, just assuming from what other people on this thread are saying.
(Googler, opinions are my own)
That’s because the company is basically designed like a synergistic portfolio where even money-losing businesses have a purpose in the big picture.
And that’s where Google fails. It’s very well known how ruthless Google can be at killing business that don’t produce tangible results. The Alphabet re-org clearly showcases that they are playing a game of optimizing the allocation of their assets. That mentality bleeds down.
Unfortunately, you can’t run a Cloud Business like that. To Google’s merit, it seems that they are not running their cloud unit with that mentality (I don’t think I have heard about GCP services that shut down). But it’s clear there’s a generalized feeling in the industry that Google can at any point become impatient with GCP, leaving many stranded.
For Google to be a distant third in the Cloud game must be pretty hard. This is something that plays to their strengths and yet they are hardly closing the gap.
The big difference is that the companies that are winning (Amazon and Microsoft) are willing to lose in some areas to win in others. They excel at that. Google on the other hand seems to have a hard time with that shit...
Case in point; AWS SimpleDB. SimpleDB hasn't been available on the AWS console for years (since 2015?) but is still accessible through the SDKs and API calls. They don't promote it or even update it (AFAIK) but it's still there and you can still use it. Amazing!
As long as they don't visibly break it, that provides a reputation that is hard to beat without something much more than one year notice as a binding commitment.
But for whatever reason, when Bezos brags about how making failed products is a core part of the company culture because it shows you’re tajing risks, HN shrugs. When MS kills off billion dollar consumer bets after only a couple of years, HN just goes “lol, why would anyone have used Mixer”.
It is a clear double standard that seems to have happened since memes are self-reinforcing.
But Google has an absolutely dreadful image to drop products and features at the drop of a hat.
Take Microsoft as a counter example, which went through absolute crazy length to provide backward compatibility for their products.
Google just doesn't have that credibility on long term commitments to bank your entire enterprise on its products.
I think this is one of the reasons MS Teams has been so successful, even when there are better apps out there that provide all the same functionality and more.
A year is crazy short if your job is to architect an ideal solution, but insanely long if your job is to simply migrate your employer onto a roughly equivalent solution. If a cloud customer isn't continuously testing a migration they're naive.
General best practice is to have a current best setup using all the first-party services on cloud-provider #1 using all their hosted services, and a fallback on cp#2 that self-hosts the services from cp#1 and uses their built-in services where better. It's what you should already be doing for benchmarking and testing but for a fallback if needed.
And once you've got the fallback planned you're free to move the active service around, almost at a whim. Even for a fortune 50 with petabytes of data - if your data isn't already hosted everywhere you're just begging to be wiped out with a simple account problem.
I do love me a well-formed customer query.
Yeah, it's supposed to work. I get paid more it if does. But in exceptional circumstances, it won't. And in more exceptional circumstances your alerts will fail too.
As an admin what do you do with the 90% of your time that isn't actively fixing something if not planning for how to fix the next thing? We did this years before cloud meant anything other than prepare for floods.
Not every cloud customer has a dedicated admin team, some even don't have any dedicated developer teams. They simply wanted a website done, contracted someone and they uploaded some HTML to a cloud instance.
Sure, if you're a large company, it makes sense to have redundancy. If you're running a small company with some esoteric webshop to serve AFK customers, it makes less sense.
Ideally we keep stuff running, but I don't want to be the kind of person who just tells you that we've got it. You need to know your airbag might fail so that you take your seatbelt seriously.
So, the message is implicitly there.
Like, "it's insane to grow legs just to live, if you can just crawl like a worm".
If you can't afford a recovery plan then what you can't actually afford is the service that needs the recovery plan that you can't afford to develop and test.
All costs, including of switching away if it fails, have to be considered as the sticker price.
this is a dumb point of view, that only cloud providers (and their employers) can advocate.
if i'm a company, being "on the cloud" doesn't automatically makes me money. my business makes me money. if every two years i have to waste a year (or even six months) re-architecting then "the cloud" is costing me people money on top of the infrastructure money.
Most enterprises won't do this, either, but it's not because of recurring rearchitecting costs.
Yeah. Do you think I disagree? It's not for everyone.
> this is a dumb point of view, that only cloud providers (and their employers) can advocate.
No, it's the truth. Stuff fails.
And if you think that this is the company's messaging you're totally wrong.
You see, there's strength in the truth. It doesn't matter if drunk drivers shouldn't hit you, that's why you check both ways before crossing the street, and you'd be naive not to. (In this analogy, drunk drivers are outages...)
Most cloud customers are naive, as the stream of major enterprise breaches caused by S3 buckets without security settings that have been default for many years demonstrates.
As for running your own servers, that too can fail meaning you still need a migration strategy (even just to new hardware) and you need to be testing it constantly.
And no, the benefit of a cloud provider isn't that normal stuff is easy or cheap but that otherwise impossible stuff can be attempted.
That's what DR is for. We have a main bare metal site and a secondary site. Throw a couple of spare servers / switches / PDUs / Hard Drives / whatever in that space too.
Cloud options need to be better and more effective than that.
> And no, the benefit of a cloud provider isn't that normal stuff is easy or cheap but that otherwise impossible stuff can be attempted.
A virtual server is a virtual server. A container is a container. The only thing the cloud offers me is the ability to change my CapEx spends into OpEx spends. Otherwise I have to hope that the vendor won't do me dirty, and will leave me in a stable, workable place 3+ years from now.
The bare metal colo operations will. Long track record of stability at everywhere I've been. Barring act of god or otherwise unusual circumstances I know my tier 4 colo will be there next year, and the year after. Will GCP be around?
That protects you against localized outages but not design failures or systematic outages or incompatibilities in new versions of your stack.
And it takes time to keep your DR plan up to date, patch the VMs, etc, and test it. Almost like this migration plan I'm talking about.
> The only thing the cloud offers me is the ability to change my CapEx spends into OpEx spends.
Ehh, not really. You can setup load-balancer pools larger than your entire colo, or use a globe-spanning backbone to create datasets that auto-replicate worldwide. And which are usually much easier than setting these services up yourself, let alone building the multiple zones. If the cloud is just a big colo to you then you probably shouldn't use the cloud. It's frightfully expensive.
Architect at a F500 here.
It would take us easily 2 years to roll out a solution, get migrated to it, and have it implemented, tested, and signed off. Probably another year (or more) to work out all of the kinks or handle the stuff that wasn't in the initial build (e.g. phase 2 expansions, etc.).
There is a reason we do 5 year TCOs, and it's not just cuz of the depreciation schedules. Also the reason we lean to things with 5-, or 10-year maintenance cycles. 1-year notice is not enough, full-stop.
Not talking about rolling our own solution here either, this is COTS being deployed in a 99.999% environment in multiple countries.
https://www.oracle.com/cloud/free/
Could be worth checking out.
Fine for a personal hobby web server to try out nodejs on?
(Not `dd`. Something actually useful.)
Well, while it certainly looks (and feels) a bit like a cliché, I find myself recommending AWS for such a purpose. Common issues will be a single StackOverflow answer away. The platform (infra, byzantine pricing model) is at least well documented. The company has a reputation for making expensive honest accidents go away :).
This being said, exploration and _actually having to dive into something_ can provide insight and understanding that just doesn't get mentally registered when on the mainstream path (because it all just works).
And as another commenter said, you get 10TB of traffic. Which is hands-down insane. At 450Mbps too. (Boggles) (AWS is 1GB/mo, albeit at 1Gbit.)
And free 10TB of traffic...
Because I heard horror stories about Oracle suddenly discovering that you did something that qualifies for extra-special-charge.
And I am not a major corporation that can easily pay major bill to another corporation.
("requires credit card" part is understandable to me, but I see no reason to trust Oracle given what I know about their approach)
It’s impossible to breakout of the free tier and get charged for something.
> It’s impossible to breakout of the free tier and get charged for something.
Impossible in which sense? If they got my credit card info they can charge - and possibly justify it by convoluted rules.
And "Oracle promised something" is sadly not very credible.
Honestly the biggest issue with Google is that they need to dramatically improve their open source database offerings, including 3rd party vendors.
RDS/Aurora eats their proverbial lunch. Azure at least bought Citus to improve their PG offering.
The GCP database offerings feel like something that was put there to check the box, not to be competitive.
EDIT: The non-Google database offerings.
On the contrary, I'd argue that BigTable, Spanner and Firestore (for Firebase) are all tentpole products that draw people in to GCP in the first place, and they stick around because GCP offers other services that probably do save the customer time and resources and are willing to use them because they've already overcome the emotional hurdle of trusting a cloud provider with their operations.
Then they're losing a lot of potential customers.
Assuming it fits your needs ofc.
What's worse about Google Cloud SQL?
Azure SQL Database has a zone redundant mode, but not a zonal mode. This means the network performance will always be terrible, because the database will be in an unknown zone, so you can't put your app tier in the same zone.
Conversely, App Service has only a zonal configuration or nothing. No zone-redundant option! You'd think this "app+db" PaaS combination would be well designed and tested, but you'd be wrong...
The Azure SQL network performance has two modes: insecure and acceptable OR secure and horrendously slow. To get a private 10.0.0.0/8 address they tunnel(!) through a VPN gateway they manage and then proxy(!) the traffic again. I saw latencies as high as 13 milliseconds for "SELECT 1" within a region, which is just absurd.
PS: This is for the Business Critical elastic pool tier, which guarantees 1-2ms I/O latency, but you can't actually get transaction latencies this low because of the networking issues.
There's lots of little annoyances too, like: you can't copy databases across subscriptions. You can't create databases in an elastic pool in a different resource group. You can only have one "Azure AD administrator", because SQL Server doesn't really support IAM. (You can use a group, but not two groups.) Similarly, non-admin access is granted via SQL only, not Resource Manager, which makes it nigh impossible to automate some builds via ARM templates. The built-in firewall is a disaster: IPv4 only and no support for service tags. The SQL Elastic Jobs resource is incomplete and seems useless. Similarly, SQL Sync is a toy and was broken when I tried it.
Overall, it feels very immature and not suitable for any real enterprise use case. Small Wordpress sites? Sure. Infrequently used apps that don't need performance? Probably okay. Maybe.
I suggest you discuss the requirements with your team and then compare both offerings.
5 years is a lot of room for change. I remember Gmail having to restore a bunch of email from tape a decade or so ago.
---
As an aside, the advice that
> If you are having the issue *stopping* or restarting an instance, try adding and then removing an IP address to the Acess control list. After that you should be able to stop the instance.
was particularly curious to read though. Huh.
Fwiw, the amount of money that Google is pouring into GCP,
including their own transoceanic fiber makes me believe
they are committed to it more than “other” endeavors.
I've got a great big bottle of schadenfreude waiting for the day Goog announce that GCP was just somebody's 20% project and they're moving to another team - complete with an "Our fantastic journey" blog post and a new entry on all the google graveyard websites...
Individual services under GCP though, who knows.
Google has lost a lot of trust and goodwill, but GCP isn't Google Reader. A sudden GCP EOL would be cataclysmic :)
https://www.theinformation.com/articles/google-brass-set-202...
https://www.theinformation.com/articles/google-brass-set-202...
We could either do it fast and have revenue impacting events, or slow and at higher labor and opportunity costs.
AT&T? Standard Oil?
Oh...
Would you avoid AWS because of Fire Phone?
Because if it didn't exist, well, it does now. It's sitting on one of those existentially-depressing parking pages. And ironically has a hilariously broken (go and see) HTTPS cert.
Indeed I did. It showed me some generic buy-this-domain website with no substantial/relevant info (actual products killed by the Amazon mega corporation).
Nevertheless I presented my comment as a concept of `a website that lists the products killed by Amazon`, which could have been located in https://productskilledbyamazon.com, for all I know. It would be more interesting to find out if such website actually exists.
EDIT: A quick internet search does not net me any results. Which is why I'm skeptical it actually exists.
Sad reality if your business depended on that retired product though.
Snapchat paying them $400 MILLION per year takes care of this issue. That's 1 company. A few jaw dropping others have popped up in S-1 filings.
And even if the entire GCP might never disappear, Google is famous for shutting things down so I could imagine people get upset when GCP is closing down individual features people are still using.
Google is famous for shutting down consumer services. It very rarely shuts down major business facing services.
Companies also use Oracle Cloud PaaS to build custom enterprise applications
So basically any business that makes or sells anything that Amazon may possibly make or sell in the future doesn't want to touch AWS.
Do you imagine the fallout if it came out ( and how would in not, thousands of people would have passed through, one disgruntled employee is enough) ? AWS would lose most customers within a few years, Amazon.com as well ( disgusted buyers and sellers), there'd be lawsuit everywhere. I really don't think anyone in Amazon is stupid enough to risk a trillion dollar business for some potential competitive advantage on whatever sold on Amazon.com
What makes you think it would be different with AWS?
[1] https://www.wsj.com/articles/amazon-scooped-up-data-from-its...
It's more Oracle software as SaaS, some basic IaaS (more like a hosting provider than a cloud) and hosted Oracle databases (which are just Oracle running Exadata for you).
So it's big companies struggling to get off old Oracle (and acquisitions like PeopleSoft) stuff and it's gross.
I don't know of anyone using it as "their cloud", but more of another thing to mix in with AWS/Azure/GCP to get rid of on-prem or colo facilities. It's more like IBM Cloud: A way to hang on to customers, sell services (human services like consulting and support), and keep milking those old products for big companies.
A stand-alone Oracle server isn't a reliable solution because patching = outage. So Oracle providing managed Exadata is a win for some (again, usually big) businesses.
Oracle got a lot of companies hooked on proprietary DB features and rewriting can be painful or require spinning up a dev team for something in "maintenance mode". Even Amazon just fully got off Oracle a couple years go IIRC.
I know one successful PaaS which has stayed, which to me says a lot about the quality of IBM Cloud
Order of magnitude cheaper and way more predictable costs than other cloud providers.
https://www.lastweekinaws.com/blog/why-zoom-chose-oracle-clo...
Working flawlessly with ssh, out of the box. It is FAR simpler to manage than AWS in my opinion.
There is "no catch".
The parent post of the parent post.
/s
1. You're valuable enough of a marketing point that they're willing to give you insane discounts to reel you in (see Zoom)
2. You're locked in enough on other Oracle technologies and they've raised your license fees but coincidentally it comes with a big bundle discount if you also use Oracle cloud, or maybe you just failed an audit and Oracle said the problem will go away if you buy some cloud services.
3. For really small scale (like some dude on r/selfhosted scale, not even startup scale), they offer a free tier that doesn't have a time limit
Cloud native can go to either extreme cost wise, and it's very dependent on good engineering (which, unfortunately, is the exception not the norm).
Outbound traffic from US/Europe is $0.0085 USD per GB. That's roughly an order of magnitude cheaper. Also, 10 TB free will be enough for many small use cases. The billing based on source instead of destination also makes the cost a bit more predictable.
That's just list price, of course, and it's still well above e.g. Hetzner, but it's a reasonable price.
We had a database growing by around 200GB of relevant data per month. While I'm not a fan of Oracle having worked there long ago, that database was unbelievably fast. They moved to MySQL years later and it required a much much bigger tin after being heavily optimized.
In the end of the day this is THE cheapest cloud available if you just want few VMs with database and under 10TB of traffic monthly.
Personal opinion: it is much easier to manage than AWS.
Every single AWS post-event report (https://aws.amazon.com/premiumsupport/technology/pes/) speaks to _a_ region. They design their regions to operate independently.
A single fault taking down what sounds like _all_ of Oracle Cloud is... unexpected.
I had a PoC with DBA's from my company and Oracle to see how that works for simple Oracle DB use cases and some Exadata stuff. At the time (2-3 years ago) most of that "rewrite" was just a lot of scripts on their side that reconfigured regular Oracle DB. Same work my DBA's did. They even allowed us to modify managed DB's configuration, but apparently it meant we'll loose support for that and would have to start managing it by ourselves again. It was confusing at best and their team was sometimes as surprised as we were.
Still, it's better then "classic" they offered. Smallest Weblogic setup (we wanted two servers) that was possible to create using a wizard was around 70GB of RAM (AFAIR) and multiple servers. It just spawned full VM with managed Oracle DB for Weblogic configuration, another one for DR, two WLS servers, a loadbalancer. Take into account that the smallest VM was 4 or 8GB of ram and your budget is gone right away. Not really for small scale setups.
I wonder how it looks like from perspective of people more fluent in Oracle products and OCI cloud, currently.
The final objective is “real” SaaS / IaaS though. Making stuff cloud-first for years was the only thing that mattered internally, a massive undertaking that took ages and might not even be done yet for some products. It burned their partner ecosystem almost to the ground, anybody who was in infrastructure basically had to gtfo or be a dumb reseller; and a lot of on-prem customers moved to other products (because Oracle basically stopped updating on-prem, or dropped crippled bundle-patches years after deploying new features on cloud, sending a clear message that on-prem was dead. Faced with a migration to what were often significantly-different cloud versions, a lot of customers opted for a migration to better products from the competition...).
First one I saw (they call it "classic", I guess) was a piece of scripts joined with glue and clothes drying string. Now the main product is okeyish, but last time I checked it had so large hardware (size of VM's and so on) as a minimum that it was out of my budget.
It may be cool for people who want managed Oracle products, but it just solves problems created by Oracle's own licensing and architecture design.
[0] https://www.oracle.com/cloud/free/?source=:ow:o:s:nav:062520...
I'm running some mysql and sqlite databases locally for my apps apart from Oracle DB. All for free.
I would imagine DNS plays quite a key role in the internal operations of literally everything in a public cloud but I could be wrong.
Now make a friend do the same and back your sites up to each other. Decentralized Internet, it works!
If it's disused, it's probably got components that are fairly late in their service life, and it's probably polls enough that it's not energy efficient by current standards, either.
It's the cheapest acquisition cost, though, since that's already paid for.
All I was saying, though, is that you can have dedicated hardware at almost zero costs and without having to use a party like Oracle or Amazon.
But then I just spent 20 minutes trying to sign up. Every time with my credit card being rejected due to "wrong billing address". Never had this happen before to me, I tried pretty much every permutation of my address. Sigh, I guess Oracle will always be Oracle.