I think a better question is "how" is all of that data stored? One large Netapp? JBOD's all over the world? And how does that data make it into our browsers so quickly?
You can serve data straight out of a database like BigTable as long as it's done carefully (to keep latency low). And the dataset can be easily replicated around the world in different datacenters to reduce round trip times.
I used to work on Google Earth. It's a really cool project. The serving infrastructure is pretty simple though. It's basically just a simple frontend server in front of BigTables (or was). The real magic happens in the data processing pipeline that produces the datasets, and in the client side rendering code. It's not like web search where the serving infrastructure is hugely complex and very much secret sauce.
The thing that impresses me the most in recent times about Earth is how the Maps team were able to write an Earth client in Javascript that starts way faster than the desktop client does. The traditional client is a large C++ app that has always struggled with startup time, I think due to a combination of code size and having lots of static constructors that slow down the critical path. And the bulk of the app was written when Keyhole was an independent company selling into government and industry, so startup time didn't matter much.
Also, the image processing pipeline is really amazing, because it blends together lots of disparate imagery sets from many different providers and cameras, with different lighting and colour calibrations, whilst still producing something usable. In the early days of Earth you could easily see the seams where different types of images had been stitched together. And if you zoomed you could clearly see the thin strips from the satellite passes. These days you still can see them if you look around, but there is a lot of very clever colour balancing and reblending being done to make the whole thing a lot more seamless and beautiful.
Before you downvote, please comment – that’s a lot more helpful to discussion
________________________
> In the early days of Earth you could easily see the seams where different types of images had been stitched together. And if you zoomed you could clearly see the thin strips from the satellite passes. These days you still can see them if you look around, but there is a lot of very clever colour balancing and reblending being done to make the whole thing a lot more seamless and beautiful.
Well, I don’t really have to look around: http://i.imgur.com/fhMPHVm.png
Wherever I look around my home town, it’s blurry, stitched together, with bad color balancing.
Try taking a look at the areas where it touches the open sea, it’s even worse.
In addition to the imagery taking half an hour to load even anything that’s not totally blurry (which can’t be my connection as all competitors are instant), and the missing 3D in a city of 300k, the imagery is (a) not updated since 2005, (b) has constant edges and stripes everywhere, (c) is – due to not being updated – missing several districts already.
I’ve complained about this on several occasions before to Google employees who promised to fix it, for example here [1].
I have to be honest, I’m more than disappointed with Google Maps and Earth.
Even here maps imagery (linked below in comparison) is more up to date, has higher resolution, and is blended a lot better.
Google, as always, seems to only (or mostly) care about the US. There, even most small towns have perfectly clear 3D imagery from the current year.
It’s just another punch in the face in a long row of "US-only!". The Internet. Connecting the ~~world~~ US.
________________________
[1] https://www.reddit.com/r/legaladvice/comments/3rer0e/google_...
(a) it’s not just my experience – it’s an issue that affects many people – most places outside of the US or large population centers look like this
(b) after complaining with hundreds of other people, some of whom have complained for a literal decade trying to improve the situation, could you be calm?
Especially when people then say "look how awesome this is", and you yourself know how bad it really is?
[1] http://www.amazon.com/Site-Reliability-Engineering-Productio...
Also, your browser won't use 4GB of RAM if that's all you have.
By the way, I'm writing this on a core 2 duo, which is my main computer. So trust having 4 cores already looks like overkill to me :)
4/8 cores is useful if one does a lot of things with VMs, but above that it's mostly either a server or an auction.
Speak for yourself. Anybody doing 3D, audio work (64 channels all loaded with 2-3 UAD plugins running native), and lots of other things will notice.
Even consumers. Advanced 3D gaming of 2030, virtual reality interactive porn, etc. Even something as humble as local speech recognition and AI processing for real time assistant.
The thing is whether we can make those cpus/tech -- not whether we can use their power when we make them.
We use up so many natural resources in such process that it's not worth it if it is not practically useful.
> Speak for yourself.
I really don't like this imperative style. You're responding to a stranger who has not said nothing offensive in his comment.
> Anybody doing 3D, audio work (64 channels all loaded with 2-3 UAD plugins running native), and lots of other things will notice.
I said "a personal machine" in my comment. A machine which is used to browse the web, take notes, watch movies, write documents, etc. 3D and high-performance audio is not every-day, personal computer use, it's a specialised case.
> Advanced 3D gaming [...] virtual reality interactive porn [...] local speech recognition and AI processing [...]
I recall saying useful.
You're projecting your needs on those of a complete stranger. That's going to get some folks hot under the T-shirt.
> I recall saying useful.
You know what "advanced 3D gaming" also means? It means being able to broaden one's horizons via virtual tourism (and if you think I'm reaching, go use a Vive) without paying tens of thousands of dollars t go be there in person. You know what local speech recognition and AI processing means? It means being able to do all that cool shit Google does without exposing yourself to risk.
These might not be things that are useful to you. They are useful to others. You're projecting, and you're kind of being a jerk about it, and that's why you got the response you did.
Oh, and while we're at it: 3D unambiguously and without question is "every-day personal computer use," and higher-end audio would be if people actually had the hardware for it. Everyone plays video games now. You might blurf from your very, very high horse that that's not useful, but literally tens of millions of people derive value from that right now and that number's only going to go up.
Please don't be That Guy.
Consumption of 3D and high-quality audio usually requires a fraction of the computing power needed for their production. Their consumption is in the scope of personal use, while their production is in the scope of specialised use.
You have in mind 3D movies. We're talking about 3D content -- and spoken about things like 3D games, virtual reality (and VR porn), virtual tourism, interactive stuff etc.
This kind of 3D content is rendered on the client and in real time, and with high CPU (well, GPU) needs, down to absolutely demanding the more realistic you make it.
We already have 128 core cpus -- it's not something out of science fiction. It's just that we don't have them for the average consumer.
Besides, one could say the same about today's 4 and 8 cores back in 2000.
>I really don't like this imperative style. You're responding to a stranger who has not said nothing offensive in his comment.
And "speak for yourself" is offensive, how? The message was while this might hold true to you it's not the general case.
>I said "a personal machine" in my comment. A machine which is used to browse the web, take notes, watch movies, write documents, etc. 3D and high-performance audio is not every-day, personal computer use, it's a specialised case.
And yet it's done on general purpose, personal machines.
And not just by high end studio engineers either (although most recording studios I've been to also use consumer PCs, nothing special there either, usually just more expensive cases and fans to be silent): millions of people recording themselves and their band do DAW/audio work that uses high end audio CPUs, and can leverage more power.
Even more millions do video editing, and soon every consumer camera will have 4K on it (already the most popular phones do).
This is not the era of Boomers and Gen X, where such things were uncommon. Video, Photography and Music as hobbies have exploded in an era when everybody can self-publish on YouTube, Bandcamp etc, and younger kids are growing up with those things all around them.
And all those people do it on their personal machines. Not some special workstation, and not at work.
So to define use of computers as "web, take notes, watch movies, write documents" is too 1999.
And while the more expensive CPU/GPU wise stuff (video, audio work) are not as widespread as passive consumption, I'll give you that (but nowhere near to fringe activities), the argument breaks with stuff like 3D games -- which a large majority under 30 play regularly, and a huge chunk of under 20 religiously update CPUs and game consoles to get the highest and greatest.
>> Advanced 3D gaming [...] virtual reality interactive porn [...] local speech recognition and AI processing [...] >I recall saying useful
What one doesn't have a use for doesn't make it useless. I mentioned some very popular stuff -- speech recognition and AI assistants like Cortana and co are used by hundreds of millions, 3D gaming is done by billions.
You're general response is close to the "no true scotchman". When one replies with stuff we could use that CPU power for, some are not really "personal", the others are not really "useful", etc.
Modern AAA games are over 30 GB installed. Applications in general are taking up more and more space locally.
An hour of uncompressed 4k video is approximately 1.72 TB. Photos have also grown in size as resolution has gone up.
I'm sure there are other industries as well. I know personally between my various hobbies, my SSDs are always full and I have several 4TB drives for storage.
>>> 4096*2304*3*60*3600/1e9
6115.295232
Resolution: 4096*2304
Colors: RGB (3 bytes)
Frame rate: 60
Seconds: 3600 (one hour)
>>> 4096*2304*2*25*3600/1e9
1698.69312
HuffYUV, for example, if you're going to be targeting a 4:2:0 output stream; if you're going for 4:2:2 or even 4:4:4 for some super HD stuff you might use different options.
Kidding aside, you can assume that in 10 years storage will be about 1000 times cheaper then it is now, so 3PB of storage would be around the cost of 3TB drive today, around 100$.
Data storage today certainly isn't 1000 times cheaper than it was a decade ago, I'm not sure it's even 100 times cheaper.
For some reason you seem to believe that storage prices will remain the same while we double capacity every 12 months, that isn't even close to reality.
(Even if you've misapplied Moore's Law to flash storage, that predicts doubling every 2 years not every year.)
The thing is, we've lived through the "low hanging fruit" era of the progress curve in IT, and now we're in the diminishing returns stage (increasingly more difficult effort -- e.g. cpu smaller process size issues -- to get ever less interesting improvements).
Of course anybody having lived the best and rapidly increasing part of the curve (roughly 1950 to 2005 or so) would think this will go on forever, or that it's some "inevitable" "natural" characteristic of technology to always improve rapidly (which mostly means they're either too optimistic or bad at math).
For some areas of tech it's even worse: not only we've reached increasing difficulty in creating machines, factories etc to churn it out, but we've also reached the limits of physical laws involved. So we either need to find totally different processes, or prove Einstein and/or Heisenberg wrong to overcome this (and sure, the first if easier than the latter, but still nowhere as easy as simply creating a new generation of the same process).
We're only just beginning to be able to react quickly enough to requests for different input (VR), so the historical workaround has been increasing the amount of data instantaneously available through things like HDTV. You're never fully consuming the bandwidth provided at any instant but you can increase your awareness by constantly looking at different parts of the screen.
So again we're fundamentally limited by the speed of light and how quickly data can be retrieved. If we can get better at local caching and query prediction (streaming in data right before it's requested), then we could theoretically reduce massively the amount of site-local storage required.
So for supply/demand reasons, I think software combined with HID improvements (better prediction through better sensors) has a significant potential to essentially put the storage industry into maintenance mode.
[1] https://www.newscientist.com/article/dn9633-calculating-the-...
I remember my dad bringing home our family's first modern PC, an original iMac. It replaced some ancient thing with a monochrome display that we kept in the basement. (I wish they still had it, whatever it was—would be fun to play around with). We thought it was like something out of science fiction–you could use Netscape to do a Lycos search! It played DVDs! It came with two: a demo disk and A Bug's Life. The DVDs (4.7 GB) had a larger capacity than the hard disk! (4 GB)
Today, my MacBook Pro has 512 GB of storage. 8 doublings in 16 years (1998-2014) and that's with a change in form factor from desktop to laptop, and in storage medium from hard disk to SSD.
Sure, my first PC had a 40MB hard disk (a generation ago it would be a 360k floppy).
But we also saw the same great advances (even bigger?) in CPU power, and yet Moore's law is on its last legs too.