I don't think anyone on hackernews would be surprised by this. The scale they have to operate at is absolutely insane. To scale flawlessly on day 1 is a big ask of any development team.
Video is hard to get right, with quirks across many combinations of platform/device/OS/browser version/video card/playback resolution/format features, and it consumes around 90% of total internet traffic. Netflix has spent the last decade building and tuning a truly world-class content delivery system (big props to all the Netflix HN readers). There's no way Disney would've been able to match that prior to go-live.
The logical conclusion, of course, is that Disney should've just bought Netflix, and it wouldn't be too surprising to see that happen now that their internal attempt has proven that pixie dust doesn't go quite far enough to magically remediate the limitations of physics. Gotta get Netflix-esque content delivery appliances colocated alongside ISP hardware for that type of thing.
Why buy Netflix when you could get a similarly qualified team and infrastructure by buying Hulu? Oh wait... Disney already owns Hulu... Whoops.
What about ESPN. ESPN has a really good streaming product. Oh wait... Disney already owns that one too...
Disney didn't need to buy Netflix, they just needed to use the resources they already owned!
Then I had 6 months of pretty normal hours working on maintenance, then I went back on another project and they had me working 60 - 80 hours (with some 40 and some 100 sprinkled in) for 18 months straight before we finished the project and I bailed.
Great resume builder but be prepared to put in hours.
Is this work culture the norm in the US? I audibly said "wtf" just now, sitting at my work desk.
If you want to make that fat US developer salary be ready to put in the work. Presumably Disney has the money to throw at you via salary... but they're going to get their money's worth.
It's a bad way to run a company, and if this is how they plan to run Disney+, it's going to be a disaster.
Compared to any big 4 or FAANG though? You can work at MS and pull 40 hour weeks or AMZN/FB with 60. Pays better, looks better.
As I posted above, they own BAMTech, one of the most well respected companies in the streaming infrastructure space.
Building for scale is not hard for good engineers, building for rapid, reliable scale from day one is hard. No one knows how to do it. Building for scale is always about patching holes while you grow slowly.
Even Google doesn’t do big band released like Disney tried to do. They keep things in beta, start off with a limited number of invitations until they work all of the kinks out.
Netflix didn’t become popular overnight. They had time to scale out slowly and on different devices.
Um.. have you ever worked at Disney? It's a great company but having worked there I wouldn't describe it that way.
>> From .. the engineers at Pixar and LucasFilms.
Two visionary companies, staffed by many ex-Disney employees and bought by after their technical innovations were completely mature.
Disney is an amazing company that I respect but having worked there I would say that modern Disney is about brands and family entertainment and they figure out technology when they have to.
How is that different than any other company? When you do otherwise, you end up with Google that just throws a lot of stuff up against the wall and comparatively little sticks or the pre-Jobs 2.0 Apple with the “Advance Technology Group” that couldn’t ship a product if their life depended on it.
PS: I have always disliked how they operate, but my sister’s working there making movies. So, I try to keep an open mind.
All of the big tech companies with hardware sales and grunt work do the same.
YouTube is the only platform that can compare to Netflix on reliability. I'd throw Twitch up there too but their scale is much smaller.
It is possible they just had a massive surge of traffic.
https://storage.googleapis.com/titlemax-media/1c8ace8f-every...
Just checked the numbers: Netflix ~700M, Hulu ~28M, ESPN depending on the source ~6M.
I got two free AppleTV 4K’s by taking advantage of a promotion from DirectTv Now, the difference in performance between them - with an Apple A10x chip - and the built in Roku TV is noticeable.
But if you have a smart TV that’s not a Roku, it’s probably even worse.
Also I bought a very expensive TV, and while I didn't dissect it I don't think there are any cheap parts in it - and the reviews all agree it had a top-of-the-line processor at the time and that was only a year ago. I mean it even has glass bezels for the OLED panel.
I don't think anything you say applies to my situation at all. I think Hulu just has some shitty software.
I'm glad it works for you, but it's pretty awful for me.
Does Hulu work on your normal computers (e.g. not a network-related issue)? Everything updated?
I don't know if this works or not, but you might want to try and plug it into a Mac and launch Console to get a sense of what's going on....
I was having issues this morning but just with home and category pages. The “featured” links to content worked fine, but the index type pages I’m assuming were just getting hammered.
Disney+ operates on top of an expanded MLBAM (BAMTech Media [1]) platform, which has been powering all of ESPNs OTT, Playstatiom Vue, and ran HBO’s OTT offering for a while [2] (Game of Thrones). Disney also opted only to launch in the US, Canada and Netherlands today. So overall, this is still a bit of a botched launch, and the expectation absolutely should have been that they would handle the volume.
[1] https://en.wikipedia.org/wiki/BAMTech
[2] https://www.fastcompany.com/90373706/hbos-tech-is-prepping-a...
[1] https://www.theverge.com/2019/9/12/20862167/free-disney-plus...
I hoped they would've had the first few layers pretty scalable and load tested, but either they had orders of magnitude more load than they ever planned for, or they didn't actually load test to millions of users (more likely IMO).
Having said that, there are a lot of reasonably mature solutions in the market to address many of these points, and it doesn't appear that was the issue yesterday.
I'm not going to pay for every streaming service, but these two have so much great content for such a reasonable price it's impossible to say no.
Spotify does that and it's a complete deal breaker.
I want to log in to all my home devices _once_, with a single account. Not continuously log out and log in and try to type complex passwords with a remote control. I mean, I can't even play different songs on different Google Homes or different TVs with Spotify, unless I sign in with different account on each - and then each device is stuck with a specific set of playlists. Horrible.
Netflix's system has worked out brilliantly for us - we have half a dozen TVs, and any member of the family can watch anything they want on any TV with their own preferences and profiles with minimum hassle.
Again, the use case is different.
What you describe: If you are an individual, ability to login same profile on many devices is sufficient (both netflix & spotify do this).
I'm talking about family use-case with multiple devices.
With Netflix, my wife, myself, our kids, anybody, can click a profile button on any of our devices (TV, tablets, phones, etc) and simply continue watching what they were watching, get their recommendations, etc. Crucially, they can watch different stuff at the same time on living room TV and basement TV, with no hassle.
I have opened support cases and there is currently no way I'm aware of to do that with Spotify. I have Google Homes in all the rooms, and I want to seemlessly play different stuff in different places. To do that, I have to create as many accounts as there are devices (that's a LOT!), AND each device is now locked in to their playlist/song/etc.
The underlying difference of paradigm is: 1. Netflix - single account with multiple soft profiles. You can watch as many streams on same account as you paid for, hassle free
2. Spotify - multiple accounts loosely linked solely for purpose of billing. Each account can not stream multiple places. So if you want to stream something on three devices in parallel, you have to a) Create three accounts; b) log in separately with those accounts in each device. What when I have dozen devices? (4-6 Google Homes, 6+ tablets and phones)? It is now insane to do anything.
Basically we now exist as isolated islands and have given up on listening to music in kitchen and family room at the same time. This is awful.
But anyway, I only have one screen, so it's still the case that Netflix charges considerably more just to receive UHD content. As price discrimination goes, that's a pretty scummy way to do it. They've shot and produced this content in UHD, yet they give a much lower quality version to their base customers. I know they have every right to do that, and it might make business sense for them (since people with UHD TVs probably have more disposable income), but it has always felt scummy to me.
Almost every other video service globally does, and of those you mentioned Prime's pricing is mostly based on shipping, Apple TV+ at this point is effectively a PR campaign and not a proper service, and Disney+ is running at a massive loss leader to build share.
Netflix isn't charging a sustainable amount of money as it is. You should probably think of it less as charging more for UHD as offering a crazy discount for lower content quality.
You could just watch it in regular quality - it's not that bad. Either you feel UHD is worth the extra $3/month (honestly a pittance compared to the price of a UHD TV) or you do not.
Netflix's biggest problem is that they've spent a decade educating customers to expect their service at below cost with an unrealistically low price point funded by debt.
They need to hope the sentiment above isn't too common.
It's possible Disney will get the kinks sorted out and start rolling a good-enough CDN on their own, making the benefit in acquiring Netflix minimal. It'd also make it much cheaper to acquire if they could wait a few quarters for the damage to start showing up on Netflix's balance sheet. But the longer they wait, the longer they deprive themselves of Netflix's best-in-class content delivery network, which already has near-universal penetration.
It's ultimately a question of whether Disney sees a bigger risk in the expenditures necessary to buy up Netflix or the chance that technical issues will continue to haunt the Disney+ ramp-up, leading to a loss in consumer confidence and subsequent retreat back to third-party streaming platforms.
One thing is certain: Disney+ is an existential threat to Netflix, and Netflix's only hope is that the implementation will continue to flop hard enough that consumers lose the appetite for it immediately.
They bought BAMTech years ago. The company that built MLB@Bat and saved HBO Go/Now when their in house tech couldn’t cut it.
I don't think it will be anywhere close to that long. Disney already has plenty of experience in streaming video (BAMtech / Hulu).
Video is really hard to get right, a Hollywood-style big bang launch on day 1 is difficult to pull off for any software, let alone video.
I feel for the engineers working there, I wonder what the meetings were like last week, did the engineers know they were not prepared for this?
Anecdotal, but let's be honest:no matter the problems, Disney can hire talent it needs. Question is whether the corporate culture there can not end up being shitty environment for engineers.
I'm not sure some day one capacity issues would justify the enormous expensive of buying NF tbh.
If we're playing this game, Apple should buy Disney. YTD, Apple's market cap has grown by more than the entire market cap of Disney.
Isn't Disney projecting to get like 10-20% of subscribers compared to that of Netflix in five years? Why are people talking like Disney streaming is such a big thing?
I'm sure they'll continue to license something to third-party streaming platforms, but they'll do their darnedest to make sure that if consumers are only going to choose one streaming service, it'll be one they control and operate.
It's nothing against Netflix itself -- they're one of the tech companies I admire the most. But they're stuck depending on relatively-short-term license agreements to offer most of the content that people are willing to pay for.
Whether or not the consumer is willing to pay for multiple streaming services, the media companies are going to be much less willing to make the content available outside of their own little streamer. This means rough seas ahead for Netflix.
I can't think of a launch of this scale that has gone smoothly. Modern console releases, AAA games with online multiplayer, Pokemon Go, healthcare.gov... all cases where organizations literally have billions in cash. It does not matter how much money you have or how great your engineering team is - you can't go scale 0 to 100 on launch day.
Ticket sales platforms for major sports events and concerts don't try to do it - they plan for it by having customers queue.
Did Disney not attempt this?
I imagine the gigantic Arctic cold front passing over the East Coast right now created a much larger demand than they were expecting on a non-holiday Tuesday morning while school is (supposed to be) in session.
Well that's the problem isn't it. They chose a scale they couldn't handle ergo, they fucked up.
A more sane approach would be to start by launching in smaller countries and then slowly scale up.
The tension between Marketing and Engineering makes another victim.
I believe Google specifically fight against this by mandating every project to have an insane scalable architecture from the get go. You can’t be Google and fail at that when you sell your infra to corporations promising that won’t be a problem. Someone at Disney+ probably made a calculated bet that if they can solve the problem soon, none will care about this in a few weeks.
The point of launching on a Tuesday morning was to test out the system with a small audience ahead of Thanksgiving, or even the weekend, when they would properly expect to be slammed. But due to the weather, they're getting slammed by all the people stuck at home with nothing else to do.
Simulating the real world is hard.
Additionally, I've hired a candidate that had an offer in hand from Disney+ and we were easily able to outbid them, (I was at a non-FAANG company at the time)
It usually doesn't take too long for employees to be disabused of their childhood hopes and dreams, but it's often a very painful process, and there are always some diehards who never seem to be able to let go. I lived less than a mile from the Magic Kingdom for about four years and saw this process play out many times.
Most people burn out and learn that maybe working on a boring business backend isn't that bad after all, but there's always a huge line of wide-eyed candidates eager to backfill. Companies that fit this profile never learn to treat employees well because, frankly, they don't have to. The same thing happens in games.
Scaling is hard. For everyone. Testing that scaling works is harder.
Yes it does.
That doesn't make it easy. But it definitely makes it easier. To say otherwise is to argue that experience isn't useful.
"All happy families are alike; each unhappy family is unhappy in its own way." -- Leo Tolstoy, Anna Karenina
Billions of dollars don't give you good technical management and people with expertise to build such things. They only give an organization where there is no pressure to be good at anything and, you know, actually compete, there is only pressure to make sure they don't have to compete, that's how they got their billions after all. And it's not like there is a definitive guide or something that you can follow to build reliable systems without experts.
I know if I had to build this I would throw money at the issue by making an outrageous offer to someone that has built this before (from maybe Netflix?) to come and do it for me.
As an aside I have worked on a project like this before on the networking side for the Olympics around 8 years ago. Heck, back in the .bomb I worked at a startup national ISP were doing QoS for video delivery was part of what we were doing. Never worked quite right.
- App is incredibly slow and takes ages to load. Worst on Kindle Fire and older devices (Netflix/Prime video work fine)
- sometimes a download film will just stop working and crash the device when you try to open it. Only way to fix is to delete the whole app and reinstall. Deleting only the download does not work.
- downloads are huge regardless of device/connection, appears to be no intelligent bandwidth usage.
- logs you out on every single update of the app or operating system (pain on Apple TV where it auto updates every minor version)
- logging back in requires you to go to their website rather than be able to do it on the device.
- never remembers where you are in a film properly
- interface on Apple TV is impossible to use (partly Apple TV controllers fault though)
Yet I still subscribe, because the content itself is great.
It is difficult to built software at scale without having all those users to provide for the load.
They could start with small number of people and get more by invitation. This could have been done before all scaling effort.