Microsoft: Cloud services demand up, prioritization rules in place
zdnet.com
zdnet.com
First of all, I'm curious to what proportions this is driven primarily by remote office work (O365), videoconferencing streaming, recreational video streaming (does Disney+ run on Azure?), or what.
But second... I'm fascinated by the concept of prioritization rules in place rather than simply raising prices. I wonder if it "looks bad" to raise prices, or if the vast majority of customers already have locked-in prices contractually so that raising prices has little effect. But I'd always thought that with AWS's spot pricing and so forth, that auction-style dynamic pricing was a core feature of clouds.
https://consumer.findlaw.com/consumer-transactions/price-gou...
For example, Georgia:
> Selling items or services determined by the Governor during a declared state of emergency to be necessary for public safety at a higher cost than they were immediately prior to the declaration.
Kansas:
> For any supplier of a "necessary property or service" to "profiteer from a disaster" by charging 25% or more than the pre-disaster price for such goods/services.
Louisiana:
> Selling goods/services during a declared state of emergency (within the designated emergency area) in excess of the ordinary price range immediately before the declaration.
Mississippi:
> Selling goods and services at above the prices normally charged during a declared state of emergency (or what was charged immediately preceding the declaration).
etc.
Also consider that raising prices would itself be a form of prioritization, just prioritizing ability to pay over need.
But, when there is a non-artificial supply and demand issue, I struggle with the derivative effects of these policies.
My concern is that rather than simply evaluating on the consumers ability to pay (and backwards-looking at prior statements instead of increased go-forward costs), these policies add only one additional criteria: ability to threaten legal issues. Thus, it’s only effect is to prioritize government agencies alongside wealthy clients. And that’s just governments using force to get better treatment. Which also feels unethical.
But maybe there’s a happier and healthier read of the situation.
Within a crisis, yes. But this also did-incentivises overprovisioning, i.e. building slack into the system.
The more profitable (and ecological) way to run a cloud provider is to provision only for the capacity that you need, and then pack your machines with low-priority tasks with lower guarantees where you can simply start load-shedding in a crisis.
There are thousands of businesses that will look at EC2 spot pricing and simply not schedule their jobs above a certain price point, and I’m sure all the cloud providers have internal workloads running at lower priority (e.g. transcoding YouTube meme compilations) that will run at reduced capacity. Personally I think the whole thing is kind of elegant, even if there are a lot of rough edges in practice.
(If the shortage is long-term, you'll usually want to implement a rationing system to address the shortages, but in the short term this kind of fiat can work.)
If the supplier is only passing on their own increased costs then that's at least fair in principle. Similarly, marginal increases in price for marginal increases in use also seems fair when there are demand surges.
Government has to smooth time.
Your feelings aside it is probably illegal as this is an emergency, and MSFT's PR would take a major, major beating.
So MS is probably hiring by the thousands trying to meet demand. You raise prices when a hurricane or a pandemic hits, if you don't want to be around for a long time.
The normal market logic that high prices encourage extra production doesn't apply during an emergency, when the demand is panic-induced and momentary, and supply changes won't take effect until after the emergency is over.
The actual solution that makes sure everyone gets what they need and theres no hoarding is rationing: limit the number that one person can buy. And that's exactly what most grocery stores are doing.
I'm not convinced the US government, let alone any western government, has the plan, manpower or infrastructure in place to actually take over distribution of essentials in fair quantities.
There is no supply problem for consumer products. The shortages you see at grocery stores are exclusively because of panic-buying and hoarding, the shelves are stocked right back up to full overnight.
buying 1 pack of TP? $2. 2 packs? $5. 3 packs? $12. etc.
The people that truly need 1 or 2 can afford those. The people that want to buy 10 or 20 at a time will pay a huge premium, enough such that it won't really even be easy for them to turn a profit trying to resell to the people who got there 10 minutes after them.
Money doesn't have much relation to opportunity cost in the presence of enormous wealth inequality. An unemployed person willing to pay their entire bank account of $100 for some good clearly wants it more than a work-from-home tech worker willing to pay the $600 they made yesterday.
The same applies to businesses using cloud services. A struggling local business has a ludicrously low cap on what they can pay for Teams compared to Walmart or something.
If I as the merchant believe the amount of money someone is willing to pay is equivalent to the opportunity cost they're willing to incur, I have to assume the person offering $600 wants it more.
When a hoarder has a 5 years stockpile of TP and others have none, that's obviously inefficiently allocated in a time of need. Given that we've seen that the is market unable to handle panic buying and hoarding in (perceived) emergencies, strict allocation of certain necessities during an emergency really does seem like the easiest way to equitably handle the situation. More creative solutions may exist, like volume purchase surcharges, as seen in a Danish store [1], but by and large, "limit 1 per customer" seems entirely reasonable. (For whatever value of 1 is appropriate for a particular product.)
[1] https://nypost.com/2020/03/21/danish-store-instills-pricing-...
As I’ve said several times. I think it’s perfectly reasonable to ration certain staples, and essential services. Everything else should just be priced to supply and demand. It’s not reasonable to call something price gouging if there are actually shortages.
(2) As I often see when people make the case you're making, you are making unwarranted assumptions about the rationality of buyers. If people are fleeing a hurricane and the gas station raises the price sharply because of the demand, what can happen (and we've seen it happen, in fact) is that it encourages hoarding: it is now a precious commodity and those who can afford to buy it do so, denying it to others.
(3) As I also often see when people make the case you're making, you're also inserting a Big Bad Government Guy into the scenario, your "rationing authority," which people making my case aren't necessarily suggesting at all: we're suggesting that it is rational for suppliers to choose to ration goods in order to be able to meet the demand of a greater number of customers, rather than to meet only the demand of shoppers who can buy as much of [thing in short supply] as possible. Again this is what we actually see in practice, at this very moment: stores in my local area limiting the number of certain items you're able to buy at one time are not doing so because they have been ordered by the California Politburo to do so, they are doing so because on balance they can serve a greater number of customers, rather than turning away literally thousands of desperate people who couldn't "allocate their resources" as well as the guy with five Ultra Transparent Titanium Master Cards.
(2) raising prices will always reduce demand, outside of the very specific circumstances where demand is inelastic. Supply and demand is an inescapable law of economics, and subverting it will always result in failures in the supply chain. The scenario you described will also not increase hoarding, it will most certainly decrease it (whether arbitrage trading crops up is a different issue, but one that is completely unrelated to hoarding).
(3) I used the term rationing authority because it was a neutral descriptor. Anybody who acts as an authority on allocation rations is going to be creating the same inefficiencies (though you’d expect the parts of the supply chain closest to the consumers to be the least inefficient)
Most of what you’ve said has demonstrated serious misunderstandings of how the basic principles like supply and demand and opportunity cost works. Your argument that raising prices increases demand should really just be dismissed off the bat.
If that were true, then hoarding wouldn't happen in the first place. You're obviously wrong here, and there's abundant evidence that consumers don't always make the best choices for themselves. Try scaling back your claims to something less absolute.
An individual will always, every single time, without fail, know what they need better than a 3rd party who nominates themselves to make decisions on behalf of others. If you remove market forces from the situation, and establish an authority to decide who gets rationed what, it will always impact the efficiency of those resource allocation decisions.
This is completely false. Most individuals in a panic buying situation never know what they really need compared to a calm, educated and knowledgeable 3rd party. This crisis has also proved your assertion wrong - witness the mass buying of perishable foods and toilet paper.
Food that was thrown out in a weeks time into garbage and food that was denied to others because of panic buyers.
It hasn’t at all. For starters, non-perishable food was absolutely the first to vanish off the shelves. But that aside, buying too much has no detriment at all to the purchaser. The risk that they may not consume all of it is so incredibly minimal compared to what they’re actually worried about. This is only possible because prices are artificially constrained. Most people don’t have to face a considerable opportunity cost when buy way more food than they need.
The phrase “panic buying” is really just a way to confound discussion about what’s really happening. There is a perfectly legitimate increase in demand, and a perfectly legitimate decrease in supply. If you wanted to stop people from hoarding, allowing market forces to respond to those changes would certainly shut it down. Any centrally enforces rationing policy is going to be far less efficient. The distinction between essential products and services, and non essential ones is also very real. You may choose that for any particular category of product or service, that the benefits in providing stability through rationing outweigh the costly efficiency trade offs. However that doesn’t obviously apply to every category of product or service under the sun.
The trade offs in efficiency are two fold. Firstly, the rationing authority has no effective way to gauge the actual needs of any person or organisation. So they will unavoidably end up allocating too much to some and not enough to others. Secondly, the market will typically respond to increased prices by increasing production. By keeping prices artificially low, you completely remove that incentive, and if the costs of production have risen (as they have), then you end up trying to manage the shortage by applying downward pressure on supply.
Finally, the issue of people not being able to afford the basic necessities of life is completely separate to that of supply and demand. There always have been and always will be people in that boat, regardless of whether there’s a global crisis taking place or not. In developed countries those people receive increased government assistance. I’ve never heard anybody seriously suggest that we should instead engage in mass price fixing to reduce the price of everything, rather than simply providing that assistance.
Who is responsible for that decision? Do they become a bottleneck? Might they make mistakes? Are they susceptible to corruption? If so, it may be more wise to distribute that decision-making... via the market system.
As for who is responsible for that decision, it's a role the government generally fills. Can it be a bottleneck? Yes. But there are extreme bottlenecks now with hospitals already out of, or about to run out of, PPE and ventilators. Might there be mistakes? Sure, but again there are mistakes right now too, with market forces favoring those who can pay, have inside connections, etc., instead of those who need the resources. In short, the detriments you cite are a reality of any system, while their presence during a crisis can be mitigated at least a little, and help critical need be filled at least a little bit easier.
Don't let the desire for a "perfect" allocation system blind you to one that is merely "better" than what we currently have now with panic buying, hoarding, price gouging, etc.
Same with the exceptions to shelter-in-place for the suppliers of essential service providers.
I similarly have months of supply of hand sanitizer (one big bottle, not hoarding).
The drawbacks of these laws don’t feel hypothetical to me right now.
Also, high prices sends information to producers that riskily ramping up or shifting production can be worth it. Might not be worth it at a lower price point.
Heartwarming international solidarity and the power of the market in action? Or capitalists selling the rope that will hang them?
There is already plenty of TP in existence in the country for everyone, it’s just poorly distributed because the gov has removed the incentive for the TP-haves to distribute it better towards the TP-have-nots.
Or, more to the point of the article, there is no incentive for someone with a useless side project to give up their Azure instance to someone delivering food or medicine.
When I went to the grocery store a week ago, there were a lot of things, just not toilet paper. If the laws against price gouging are causing a problem, why is it so selective?
Microsoft seems to be doing the same, sensible, thing that grocery stores are doing in some cases - try to ration stuff per customer.
People are talking about how this dynamic is playing out on a bigger scale, as states compete for resources in the absence of the federal government coordinating.
> the gov has removed the incentive for the TP-haves to distribute it better towards the TP-have-nots.
Really?
> I have an entire Costco package of toilet paper that will last me months while some poor bastard near me is probably wiping his ass with his hand (or worse...
This is not a reason to you? Some people might not appreciate the tone of this but, you’re seriously lacking morals if it’s not.
Or try freecycle.
It's not difficult once you realize that if you actually care about waste, then you don't have to profit.
I plan to risk technically violating the "shelter at home" guidelines to bring a 6-ounce bottle of hand sanitizer to a friend in a different city next weekend, assuming my order actually arrives. (It's a little crazy that the stuff is virtually currency at this point.)
> factories can't be retooled overnight, and people die in the meantime.
Of course the masks aren't hitting shelves the next day, perhaps two weeks before meaningful production increases. What a pointless statement -- the crisis won't be over in two weeks, and this same principle would have held two weeks ago, or eight. And then you would blame people dying on the entrepreneurs retooling to create masks? Another pointless statement if that is not your intent.
Were you trying to see how many HN rules you could violate in a single post?
They should be in time to save a lot of lives.
Even the few people who complained that they wouldn't use it because there wasn't a phone number to call for telecons were overcome when we told them to just install the app and join the conference that way.
For internal work, it's actually been a lifesaver. Webex and other solutions have failed to hold up under the strain while Teams has more or less just kept marching on.
I have a feeling as this goes on for the next few months Teams users will not go back to the fractured fragile mess of other apps we tried to get along with before hand.
Hell, there's even a halfway decent Kanban built in.
For me Teams has been a major downgrade from Slack
Unread teams groups are bold and same with chats that have new messages.
Calls likewise get a dot and so on.
Show me a training plan (outline is fine), case study or blog or anything outlining how to train my 50 non-technical coworkers in the use of IRC. They need image support and audio chat and all the other modern chat features users demand. Keep in mind there are a lot of chat options and users will walk if what you want to use is rejected by them.
IRC is not and has not been a even remotely viable option for anyone but nerds and hobbyists for many years now. i would be very happy to see any evidence to the contrary.
I get all the activity I should be alerted to in my alerts button too so document conversation, hidden teams, etc all show up there too.
Teams is somewhat good, good tools to create wikis and integrate third party solution like Jira and office integration. One selling point to me is that "it just works" even with non technical users.
Teams has video conferencing, but also hooks into SharePoint for shared documents.
They bought ScreenHero and tried to integrate it. They screwed it up so bad that the ScreenHero founder quit after 4 years and has gone and rewrote it and called it Screen: https://news.ycombinator.com/item?id=22676040
Teams does it all OKish, although the channel based messaging is probably the worst part.
It's sort of wrong to look at Teams as a Slack replacement. Well, it is for who just want chat, and don't care about channel messaging.
It's pretty easy for Teams too.
I don't particularly like Teams, nor do I like Slack, but there is nothing feature-wise, today, that would make me choose Slack over Teams. Teams also plays much better with OneDrive and Outlook, obviously.
I think Slack is now a goner. They will be acquired in 5 years tops.
Of course now everyone realizes it's a big sham and neither Amazon nor Microsoft are stupid enough to spend a trillion capex on building the datacenters, fiber lines and so on you need to handle a temporary 700% pandemic situation. In a year from now, they don't want that stuff on their quarterly report just as companies going "cloud only" don't want it.
If people weren't on the cloud, you'd see them scrambling for physical hardware, which also isn't exactly easy to come by on extremely short notice when the world's logistic chains are under stress as well.
Due to there being a 250 participant limit we now switched to live events, and there also has been no issue with that.
Some cursory web searching suggests that at least as of 2017, BAMTECH / Disney were using AWS as their public cloud provider. There's enough inertia here that you'd imagine that this is still the case.
Just if Microsoft moved internal usage to underused data centers somewhere on earth they could create more capacity for NA, EU, Asia.
One of the upsides of cloud is that it’s so much easier to meet demand, even as a provider.
I don’t have access to component prices, but I guess they are buying like crazy (aws and gcp too I guess).
https://www.wsj.com/articles/store-shelves-stripped-of-lapto...
While I have no inside information, I have to imagine that these same shortages have affected data center hardware too, and that it's much more difficult/expensive to add capacity at the moment compared to, say, last December.
Do you realise that it's potentially illegal to increase prices based on market demand during a crisis?
Azure could service VM creation requests in the order of their creation, but "take a number" doesn't work as well. The opportunity cost of what else you could be doing with the time spent waiting in line is the "price" that allows the market to clear.
It's important to keep in mind that a higher price is unlikely to lead to much additional capacity due to the nature of this shock. Companies have to do business with their customers for many years after this is over. "Sorry, we're overloaded with demand due to this crisis" sounds a lot better than "Pay us ten times as much during the crisis or go f* yourself." Microsoft isn't in business to allocate resources according to a simplistic strategy pulled from an intro econ course, they're in business to make money over a period of many years.
Although you're right, Microsoft may have correctly assessed that its customer relationships will turn out better if it leaves them completely hosed vs. hosed with an unpleasant escape hatch.
It's still an interesting question: what kinds of allocation systems are best when for whatever reason market pricing is off the table? There are certainly better and worse approaches within the space of lotteries, queues, etc.
If different people need vastly different amounts of something, either they are able to judge, in which case prices work, or they are not, in which case some independent means of allocation has to be used.
I don't think waiting in line is a good method, but that doesn't mean unregulated prices always are either.
But it's analogous to toilet paper. If you are limited to one package per trip through the checkout line, yes, you can consider that to be allocating by willingness to stand in line. But the intent is simply to have a (not terribly strict) quota, and not to have you go through many times.
So it isn't as fundamentally screwed up as proof-of-work. Nobody is supposed to actually treat it that way.
Given the sudden need and the circumstances behind it putting prices up might not decrease demand enough to make any difference so while the system will be more profitable it won't be any less congested and now you have people paying more but feeling like they are getting less because they had to wait an hour before they could start that little herd of VMs they usually have available immediately on demand.
This could drive people off onto other services, or at least make previously entrenched users look up and scan the horizon.
This is happening in real time: https://www.cnbc.com/2020/03/29/coronavirus-mall-owner-taubm...
Service providers shouldn’t discriminate between one bit and another - no deals that prefer some providers or traffic over others.
It usually refers to ISPs, but I don’t see why it doesn’t apply to cloud providers too.
Also lots of .gov runs on azure.
The Microsoft blog post states: "We have seen a 775 percent increase of our cloud services in regions that have enforced social distancing or shelter in place orders."
So it's not an overall increase, just in certain regions?
Also, everywhere's going to have enforced S.D. or S.I.P. sooner or later.
2. I actually think Microsoft has much less runway. From what I understand, AWS has more modern infrastructure and backend, and they shuffle resources easily around, between services, and I think they have much more in reserve. Microsoft has concentrated much more on the sheer number of regions.
3. Azure has a strange way of handling quotas, if you ask me. Up until now, once you provision a VM, it is deducted from a quota and stays like that as long as it exists. It has never been an issue to actually power it on (unlike AWS), once you have it. It's not billed, but we always thought it stays like that. Since last week, you can see failures not only when provisioning VM's (even within your quota) but also when starting them. Nevertheless, I also think a lot of users had larger quotas allocated then they actually use. So they just started creating more VMs or other resources (because they could), and the thing came crashing. I think that's just poor planning on Microsoft's side.
But the thing I'm mostly pissed of is the status page. VM's are failing left, right and center and everything is nice and green on the status page. Once you open a ticket, they send you an incident-in-progress report.
Status pages parroting lies in service of marketing should incur more liability for companies than they do. How does a society discourage vendors from doing things like this?
Personal favorite: Tic Tacs are 94.5% sugar yet Tic Tacs can be marketed as a "sugar free" food.
"The Nutrition Facts for Tic Tac® mints state that there are 0 grams of sugar per serving. Does this mean that they are sugar free?
Tic Tac® mints do contain sugar as listed in the ingredient statement. However, since the amount of sugar per serving (1 mint) is less than 0.5 grams, FDA labeling requirements permit the Nutrition Facts to state that there are 0 grams of sugar per serving."
Yes, this. Where I work, WFH had an outright ban until this all hit the fan. Now, anyone who can do so has a mandate to WFH. We use Citrix to access some internal resources, though it's really inefficient: If you want to access a file on network drive, say an excel file, you need to use Citrix to launch an Excel instance, open the file, and deal with it there. You can't easily get it to your local computer, so this adds to the overhead necessary to use the system, which has been strained of late (and it's not like citrix lacks the ability to transfer files between remote & local, they've just disable mapping of local drives during a session) Luckily I have a server-class workstation under my desk at work for heavy data-crunching, and setup my own VPN, which works much better. At least unless/until they turn power off in building not being used.
There are three kinds of lies... lies, damned lies, and cloud service status pages.
If it’s after deployment that would be concerning
Since we use Azure a lot through the API, there is a difference in the first case. Usually it would fail on the first API call, it would just tell you this or that went wrong. Now the process fails after it has been started.
https://www.theverge.com/2020/3/19/21186452/microsoft-teams-...
This like every day is Black Friday for Amazon. It's rough, but not something they have never dealt with before.
AWS has been rock solid for us in the 3 regions we operate in.
So, a pretty big deal then: https://www.theverge.com/2018/7/16/17577654/amazon-prime-day...
I don't use video since I feel weird doing it. Also I don't want to do my hair and change from pyjamas.
Nobody gets fired for buying Microsoft..er IBM. Inertia too.
Or maybe that's not something providers reveal, so let me as the sys-admins of this thread: For on-premise resources, what is your % margin of excess capacity reserved?
Because of that I always recommend that a project reserve instances to ensure capacity exists when you need it. It's also cheaper, but that risk is important to mitigate.
We (rsync.net) run a sort-of just-in-time deployment model wherein as our storage arrays approach 80% capacity, we completely remove-and-replace the oldest storage array at a location[1].
The 80% number is related to performance penalties that become extant when you fill up zpools too full. In the old days, that is the threshold beyond which we would see significant performance degradation. ZFS behaves better now and you can avoid a fair amount of that degradation by having read and write cache devices (L2ARC and SLOG) but we still hang onto that 80% number ...
When we do such a replacement, we are often replacing a 2-3 year old storage array with a brand new one, which means that the drive capacity of the refreshed system is about double what it was before - so we have a relative glut of space for a time. This allows us to onboard larger (50-100 TB) clients with no lead-time.
FWIW, we have seen a relative increase in signups since lockdowns began. My current theory is that a lot of people are sitting at home with a lot of free time and thinking about risks ... and some of them think about backups.
[1] Denver, San Diego, Fremont, Zurich, and Hong Kong.
While it technically doesn't matter where you run, there are a lot of choices that have a different answer depending on if you ask your vendor or if you do your own checks and research.
Scaling purely because you needed to scale up would be a different story, and then you're expect overall service demands to go up that steep. This seems more like an unpreparedness peak to me.
Disc: Googler
There's a different HN post about it: https://www.nasdaq.com/articles/the-sec-really-wants-investo...
Yet the stock price has the exact same changes as the rest of the entire market.
Their cloud demand is up x7.. it’s not priced in. Amazon same thing, although less so imho
Their current p/e is 26. Remove cash and it’s nearing 20.
I read that a lot of hedge funds running heavily leveraged, marginally profitable strategies have been deleveraging. That's a pretty rational response. Some of the movement is emotional, but some is driven by more rational concerns.
https://www.csee.umbc.edu/courses/471/papers/turing.pdf
Or, we change the kind of economy that we have. We're seeing right now how what parts are actually "essential"; arguably, the rest is only necessary for growth. If growth stops being the goal, the the economy as it was doesn't have to continue.
We're already seeing some rumblings of a stark dichotomy, what with some people advocating for a return to work and acceptance of the risk, contrasting with the backlash that their dollar is not more important than someone's life.
In a Japanese subway, someone wrote: “We can't return to normal, because the normal that we had was precisely the problem.”
It doesn't entirely matter if they're right or wrong objectively; if enough people subscribe to this view, it becomes what we live through.
I encourage you to play with the epidemic calculator. This isn't something you "just" take the risk for. And keep in mind that the deaths in the calculator aren't including deaths due to denied hospital beds for more typical things like heart attacks and strokes.
Is 10-30 million deaths in the US this year an acceptable trade-off to keep the all-so-important economy going? Or can we scale back the economy and distribute the wealth so that only food can be afforded while we build up hospital resources, work on a vaccine and save lives?
"Phew! Now that its priced in, let us focus on putting our $2 trillion to work"
> In a March 28 blog post, officials said that demand for its new Windows Virtual Desktop usage has grown by more than three times. They also said government use of public Power BI for sharing COVID-19 dashboads is up 42 percent in a week. (As is the case with Microsoft's overall cloud services figure, we don't have a base number for WVD and Power BI from which to calculate these percentages.)
Seems to be a cherry picked statistic, not an overall increase in utilization.
Microsoft saying “Ahh! Crisis! Here’s the Windows Server codebase!” wouldn’t help, since you’d have to still find a host and time to make sure your code works on this upstream variant without all of the licensed DLLs that makes Windows work.
Matrix offers free decentralized video/voice communication and there are lots of self-hosted alternatives that often provide better features than things like Teams. With NextCloud you basically get a free office solution that just needs to be deployed. With Kubernetes you can quickly deploy a Nextcloud cluster. These services should be free and open source because the goal should be helping build a better world, not building better profits through backroom deals with tax dollars.
But yeah, it would be more helpful in these trying times if the NHS had the sourcecode to BizSpark, DirectX, and Office Graph backend. :|
[1] https://www.microsoft.com/en-us/microsoft-365/blog/2020/03/0...
[2] https://www.windowscentral.com/microsoft-teams-available-nhs...
[3] https://www.zdnet.com/article/microsoft-teams-usage-up-by-12...
What's happening now is unprecedented. Without public cloud providers, most businesses would not be able to run this giant work-from-home experiment. If they were hosted on-prem then they couldn't scale either.
It's not difficult to bake my own bread, but I don't.