Google Distributed Cloud air-gapped appliance | Hacker News Reader

Google Distributed Cloud air-gapped appliance

cloud.google.com

195 pointsradeeyate108 comments

Need:

  - photo/video
  - root of trust definition (TPM? OpenTitan?)
  - firmware and OS description
  - specs

There's an edge device family from AWS, with specs and photos, https://aws.amazon.com/blogs/aws/introducing-aws-snowcone-sm...

> AWS Snow Family of physical edge computing, edge storage, and data transfer devices for rugged or disconnected environments.. can be used in a variety of environments including desktops, data centers, messenger bags, vehicles, and in conjunction with drones.. enclosure is both tamper-evident and tamper-resistant, and also uses a Trusted Platform Module (TPM) designed to ensure both security and full chain-of-custody for your data. The device encrypts data at rest and in transit using keys that are managed by AWS Key Management Service (AWS KMS) and are never stored on the device.. use Snowcone for data migration, content distribution, tactical edge computing, healthcare IoT, industrial IoT, transportation, logistics, and autonomous vehicle use cases.

AWS Snowball hardware, https://youtube.com/watch?v=BIx9bbe58K8

GDC video of users and control panels, no hardware, https://youtube.com/watch?v=i5fCfgNaPE0

With hardware expertise from servers, OpenCompute, Project Ara, Chromebooks, Pixels and TPUs, hopefully this appliance is more than a PC OEM whitebox.

> The device encrypts data... using keys that are... never stored on the device..

Incredible!

Not really. They can just use a public key to encrypt ephemeral symmetric keys. The private key is stored inside AWS and is never exposed to the device.

I guess that makes sense. So it's like a ballot box you can drop data into but only Amazon can pull it back out.

Still a bit misleading. The public key is on the device (which is fine) and that's the key it uses to encrypt the data (if we consider the symmetric key as just a performance optimization)

It's telling that all the hyperscalers keep taking runs at this, driven by the DoD. I'm not sure anyone is making money at it though.

Azure tried with Stack Hub (private airgapped cloud), Stack Edge (various options, including ruggedized, gpu enabled, battery powered, rackable). The JEDI contract didnt amount to much so I dont know if this range has a future.

AWS have tried with outposts and the snow family. Seems to be doing ok in the commercial space.

Now google.

They all seem to have some weird genesis as data transfer gateways (looks like a local network share, but really sends data back to S3 or some other cloud store), and they all seem to have weird compromises that the disconnected nature forces upon them. For example you need to connect the box to the cloud at least once every 30 days to have it sync to the mother ship, or whatever.

I wish them well on this but I doubt it will be much more than a tickbox for government contracts and won't see much live deployment.

If google came out with a range of box designs that fit in a backpack or a VPX chassis, could be spared and replaced in the field by any vendor in the defense industrial space, could run disconnected for 120 days or more without degrading, could be operated by an 18 year old under duress in a combat environment (no one is following a manual at that point, it needs to be "turn the key and oress the big red button" simple) and could seamlessly upgrade/clean themselves up when reconnected to the cloud back at base, they'd certainly have my attention. Oh and given the geo situation, maybe made from components that have alternatives not made in Taiwan.

With Google, I have a feeling it largely evolved from two other things [that weren't the DoD]. Google never really tried to sell to Defense until JEDI, and didn't really have a compelling solution at the time to the need for FedRAMP High requirements.

However, what Google did have was a business need to offer GCP in mainland China, and a partnership with Tencent to do so. Additionally, after Thomas Kurian joined, Google also had a willingness to partner with KSA as a tangential part of the overall NEOM investment, but with the hyperscaler providing a dedicated region in Saudia Arabia and in exchange potentially getting heavy commercial workloads from Aramco and other KSA entities. Google already had Sovereign Cloud experience, having built out a data center in Germany (that, among other things, SAP uses for internal development), so it wasn't a huge leap to go from the hoops they had to jump through to offer this combo of stuff all in one package:

* Interconnect partnerships (Oracle Bare Metal, Tencent)

* Integrated management console (Tencent, Anthos)

* Sovereign Cloud services (US Gov't, European governments)

Beyond all this, Google has been offering CDN appliances for ages, and space in local POPs for 3rd parties (like Netflix) to install their edge appliances, so it's not like there were any skills gaps on the networking side, either.

The real question will be whether the hyerscalers will be able to viably sell these sorts of appliances vs their potential customers just running their own data centers and virtual data centers.

It’s almost like the classic balance of storage, network, processor has a fourth leg: information encumbrances, or data sovereignty of various sorts.

I'm in the funny position of having to test products in the cloud at a company that is really skittish about putting any of its own stuff in the cloud. I looked at AWS Outposts, but determined that it likely wouldn't pass muster with our policies. It was also really expensive, and it was only available for rent, not purchase.

I then did an experiment with the Azure Stack Development Kit. It was limited to a weird ghetto of outdated VM images, and had to be rebooted every few weeks. I did not proceed with Stack Hub.

If GCP wasn't a distant third place, I might give this thing a try, but it's probably really expensive just for testing.

The long term solution is going to be chipping away at our policies, but I was disappointed that I couldn't find a usable on-prem cloud solution.

What about oxide's stuff?

Testing correctness and compatibility with particular vendors was the priority: block storage, O/S images, APIs, etc. A third-party solution like LocalStack was on my radar, because it aims to be compatible with AWS. My understanding of Oxide is that it's its own thing.

heheh Vic and I did one in a backpack, worked good :)

I’m over here wondering why we even need wifi in an appliance… my favorite appliances have just a mechanical knob.

> They all seem to have some weird genesis as data transfer gateways (looks like a local network share, but really sends data back to S3 or some other cloud store), and they all seem to have weird compromises…

Yes, but Id say they _should_ be data transfer gateways and instead we ended up with _faux disconnected ec2_ racks. From what Ive seen the bulk of outpost usage is really close to the data gateway; “Im an onprem/hybrid customer, give me an EC2 vpc endpoint in my existing DC so I can move data and run some workloads mixed mode.”

But instead outposts morphed in to “How do we fit EC2/EBS/S3/RDS in to an on prem rack that their cloud team can manage with our existing APIs.” Thats very much a different product and (IMO) a harder, bordering on unsolvable, problem. Getting down to many thousands of single RU appliances is only making it more impossible and disconnected from the AWS “mainline” infra.

Id be REALLY interested to see a version of the outpost product that was a single 1-2RU whitebox router with a handfull of 100/400 gb ports and APIs to setup the tunneling/encap/VPC connections for each port. Give them local “vpc endpoints” termination and “direct” connectivity in to their VPC resources back in the region. But that looks a lot more like a DX kind of product than outposts, I think

Though other use cases for the appliance are given, it seems primarily designed for military applications?

It's designed to military standards and to be as individually transportable as other military communications equipment:

> Department of Defense (DoD) Impact Level 5 (IL5) accreditation

> rugged and portable design that meets stringent accreditation requirements like MIL-STD-810H

> The appliance can be conveniently transported in a rugged case

> Weighing approximately 100lbs, it's human-portable, making it easy to transport and deploy in various locations.

> disaster zones, remote research stations, or long-haul trucking operations

Military operations are all three of these.

Its design enables the offline self-hosting of cloud surveillance tools:

> Google Distributed Cloud air-gapped appliance is designed to operate without any connectivity to Google Cloud or the public internet. The appliance remains fully functional in disconnected environments

> built-in AI solutions from the Google Distributed Cloud air-gapped appliance like translation, speech, and optical character recognition

What about facial recognition?

The "smart border security system" is coming. This feels like it would be a "perfect" part of that looming disaster.

Truly puzzling why Google is doing these things that do not scale. Their DNA historically has been doing things for billions of users, not 10 companies that might ever pay for this. Google is a technology company through and through, they have a great engineering talent, and they can keep shifting paradigm in many areas, especially in cloud. Yet, the short-term profit motive of the rot economy is taking another tech giant hostage.

Kurian = enterprise IT = high-margin low-scale customized solutions. In theory the long tail of the market is just as lucrative as the big head.

depends if you define long tail as customer count vs contract size

That's what happens when you take your most productive/creative minds, thrown them in the trash, and replace them with greedy MBA drones.

Drones, indeed. Now your government's murderbots can be powered by Google Gemini.

One of the more interesting things was the MBAs don't run engineering, it was fascinating seeing how quickly the tide can go out on management quality, especially when you're growing 20% every year -- took maybe 4 years to form a new extremely agreeable layer over significantly worse quality than the one 2 layers above it. Kiss up, kick down.

I have no idea what you’re talking about in practice. It felt like MBAs or less competent perspectives abounded when I was in cloud.

You realise that the idea that developers who work at google are more intelligent than average is the product of the work of marketing graduates who work at google?

I have no experience in this space, but I suspect supplying the US Air Force with this equipment may have a number of indirect benefits.

This seems pretty adjacent to their existing cloud business not requiring major new investments and is likely a requirement to do bigger deals with customers.

They invested in a dead end AI technology. They, like all the other players in the space, are trying madly to recoup their original investments. It turns out "chat bot" is not a viable product on any level whatsoever.

Google Cloud has an totally different customer base, strategy and internal culture from the rest of Google.

I wonder about the weight details and its fundamentals:

"The device weighs about 100 lbs (~45.3 kg) and can be carried by two people. The device is not operational while it is moved from one location to the next. It might be moved on and off vehicles and might be subject to rougher treatment than in a data center. While the device is running, it might be in an uncontrolled environment subject to more temperature variations and dust than a data center, such as a tent or a repurposed building." [1]

[1] https://cloud.google.com/distributed-cloud/hosted/docs/lates...

This seems like GCP's version of AWS Outposts Servers

https://aws.amazon.com/outposts/servers/

Does Azure have a similar option?

Their hardware is called Azure Stack Edge:

https://azure.microsoft.com/en-us/products/azure-stack/edge/

Is the box painted yellow? That's all I want to know, and if you are old enough you will get the Google Search Appliance reference.

We had a blue mini appliance!

Teardowns previously:

https://rothgar.medium.com/google-mini-search-appliance-tear... | http://1n73r.net/2012/12/11/google-mini-search-appliance-tea...

https://www.anandtech.com/show/1781/3

It's for military applications so it's quite obviously green.

In this particular case, no, they're unbranded HP boxes, though some that have been deployed have GCP logos on the racks themselves.

How lame, Google used to be fun.

parts of it is still is, you're just focusing on non fun parts

It's always fun trying to guess which product Google will sunset next month

> unbranded HP boxes

HPEnterprise (Compaq-derived servers) or HPInc (desktops/laptops)?

The post seems to really be vague around the obvious and most likely majority defense use cases this would be deployed for. It instead tries to emphasize all the other potential uses and mentions defense only as the final one with a generic quote from the air force.

I think it’s very likely that’s due to historical Googler outrage against working with defense organizations.

It looks like this is an evolution of an offering they've had for some time:

https://cloud.google.com/distributed-cloud#modern-experience...

This reminds me when Sun Microsystems launched a data center inside a shipping container that you could literally buy and have it deployed anywhere. Great for on-premise "cloud" computing (before cloud was a thing) or in war zones if that was your thing.

So basically a local server. Guess we're on trailing edge of "move everything to cloud" now, slowly eeking back into having more local infrastructure again.

It's not a local server! It's a hyperconverged system, as explained on the product page [0].

Which is, I assume, a very fancy expression for a local server.

[0] https://cloud.google.com/distributed-cloud/hosted/docs/lates...

> GDC air-gapped appliance consists of a chassis that holds three blades and a switch. Customers must provide their own laptop to use as an admin workstation for installing the software and performing upgrades.

It's borderline criminal that they don't include a picture of this thing. Let's see this thing!

Does anyone care about this except DoD?

The sad reality is probably not.

I personally would prefer organizations to own their hardware as in the early age of internet. It was meant to be decentralized. However in the last 2 decades centralization has prevailed.

I think it is sad because look at the CrowdStrike incident earlier this week. Or outages in AWS, cloudflare etc. These are examples why decentralization would give people/organizations power and control.

This mentality of making it “someone else’s problem” with outsourcing is a fairy tale. In the end your business is at risk. Let alone the overhead and inefficiencies.

Perhaps another analogy: if one eats out every day and never learnt how to cook a meal themselves. When the situation presents itself there is no cook around. One would probably starve or resort to simple food sources like whole fruits.

Sounds very military… and yet I see no mention of a suite of remotely-armable tamper sensors built into the hardware, that would enable automatic scorched-earth wiping of sensitive data if their guys manage to shoot your guys and march into your DC.

(Or more importantly, if this thing is just sitting there in a remote unmanned outpost, and their guys find it. If you have no humans to implement a scorched-earth policy, the infra needs to be capable of doing it itself.)

I find this especially strange, as tamper-responsiveness is usually a headline feature following the words “mil-spec ruggedized server.” (See e.g. this thing: https://privatemachines.com/)

This is to let the military use AI to help kill people.

“Don’t be evil” is dead.

> This is to let the military use AI to help kill people.

So are your tax dollars, and some portion of any money you spend or any productive engagement you have with the economy wherever you live on this planet.

This is not a convincing argument for not engaging in voluntary trade with the morally bankrupt.

It is, however, a pretty good argument for the moral basis for tax minimization and avoidance.

My tax dollars are used to bomb the middle east and there is absolutely nothing I can do about it. Voting is useless.

Donate to humanitarian aid organizations to offset your tax bomb dollars

Almost universally those funds are stolen in the name of administrative overhead

Useful for a truly never-connected 'island' (meaning it never needs to speak to the outside world).

However, even some of the use cases they cite rarely exist on a never-connected island, e.g. industrial automation and transportation.

So, to be broadly applicable, it needs to be secure by design for connected use cases as well, even if those connections are considered to be ephemeral (e.g. remote management, periodic telemetry, metadata sharing, etc.).

Curious about open source licenses: this was a big problem for the Google search appliance IIRC

I spent tons of time with Google Search Appliance (at least 100 hours reverse-engineering it) it was just a CentOS machine with a daemon called Babysitter (which was just a loop restarting services), and a C++ binary called gws (Google Web Server).

Fun fact, if you ran gws without its config files you would see the real front end for Google Search, News, etc.

Web configuration interface was in Java, writing some XML templates if I remember well.

So taking all of that, besides a very boring OS there was "nothing" or very little amount of open-source they were using.

It was more all homemade (except the OS).

Fun fact: There was a secret hardcoded password in clear (but only for physical access).

EDIT: Password was different for each instance, not the same as I thought.

Well that's fun. I was the TL of the GSA platform team and you are mostly spot on. You are missing the whole crawling/indexing & security parts though. the GWS on the GSA was, tbh, one of the simplest component.

Each GSA had a set of unique BIOS/root password generated during bootstrap though.

I edited the message, sorry for that mistake, I had assumed it was the same everywhere.

It was great to see how it was engineered, some parts were truly remarkable, my main interest was to learn about the ranking algorithm (not for SEO purposes, but because I thought it was fun and interesting).

We would have been in love 15 years ago when there was the GSA, sadly, our paths have separated :D

https://commons.erau.edu/cgi/viewcontent.cgi?article=1153&co...

How do you know to which libraries the c++ binary was statically linked to?

What was the problem specifically?

has anyone done an analysis on how much big tech revenue comes from the DoD, particularly as its changed over time?

Would never consider this after getting rug pulled when Google abandoned the search appliance. That was fun.

The post announces a physical (i presume) appliance and it’s just a wall of text and not a single photo. Mkay…

Yeah I just wanted to see a picture of it

I couldn't find any specs in the docs either. Welcome to enterprise.

There's a bit complexity there as the system is designed to be modular based on requirements. GPUs? Raw RAM? DC or AC? All different compoents that you can swap in/out.

I would assume there's some kind of catalog or configuration guide y'all could publish but maybe not.

No, the obfuscation helps with setting "enterprise" prices.

I mean this is literally their 'AI, but for TLAs' product. I'm kinda shocked there's a public announcement at all.

Why does it weigh 100lbs?

> Previously, organizations with mission-critical workloads lacked access to important cloud and AI capabilities when in demanding edge environments, including those that present unique challenges and requirements.

I'm sorry, what???

It's interesting watching silicon valley buzzwords mix with DOD speak.

The Hooli/Pied Piper box - https://silicon-valley.fandom.com/wiki/The_box

Given the discontinuance of the Google search appliance, I would be reluctant to consider any Google hardware. I’d likely use something like PCF on owned hardware for the scenarios they describe.

When would someone reach for PCF now instead of K8s?

Given what Broadcom did to almost all of VMWare's products, I'd be a lot more worried about PCF than really anything else.

There’s an opensource version of CloudFoundry. https://github.com/cloudfoundry/cf-deployment

I mean, the search appliance was discontinued after 17 years. Not sure it's that bad...

And they supported their last search appliance over 10 years and provided a transition path toward cloud-based alternative. This is probably better than usual industry cases and I'm pretty sure Google wouldn't get this bad reputation if they adopted this case as their own product longevity standard.

The customer is the DoD. They know how to negotiate contracts to cover this scenario and have the clout to negotiate such and enforce it.

And they’ll likely have a “this need to be supported for 50 years” provision too

Well, Google already distribute hardware to various ISP - it's called GGC (https://www.gstatic.com/isp/docs/ggc-installation.pdf?sjid=5...).

We (GSA) & GGC used to source our hardware from the same supplier (Dell).

Only part of the GGC fleet are Dell machines (that pdf lists Dell, HP, and Equus). Paraphrasing one of the leads from some years back: "Single-vendor is not a vendor strategy."

Between improved negotiating position and resilience to vendor-specific firmware bugs / vulnerabilities, the additional maintenance cost associated with supporting two or more platforms pays for itself very quickly.

In this particular case, they're the air-gapped product is singly dependent on HPE servers, mostly for compliance reasons. Same reason on why it uses Palo Alto firewalls.

Though in the case of the GGC nodes, having multiple vendors was mostly a negotiating component. If we could go to HO and order 3000 servers and have them running, Dell loses a large amount of negotiating power.

Being honest though, working with Dell was significantly better than working with HP or (especially) Equus.

Former Google Employee, on GGC.

HP!=HPE for over 8 years now

This is very different. This is like if Anthos married a Toughbook. There are very real, very sticky use cases for this appliance.

The Google Search Appliance was available from 2002 to 2019, which is a pretty decent run for a piece of IT hardware. Especially given that the average office environment looked quite different in 2002: the GSA was designed for indexing intranets (remember those?) and did not require any Internet connectivity at all.

>the GSA was designed for indexing intranets (remember those?)

Yet the problem of being able to find things still exists. That my "intranet" consists now of a bunch of cloud services accessible to the internet makes no functional difference.

If it's accessible to the Internet, Google can make a private index for you with Cloud Search and you don't need a physical appliance.

https://workspace.google.com/intl/en_au/products/cloud-searc...

That doesn't read like it can login to anything other than Google services. Those internet intranets typically sit behind some kind of authentication.

Does Cloud Search support third-party data?

Yes, Cloud Search includes connectors to third-party data sources, such as Salesforce, SAP and more than 100 others.

One of my previous jobs had this appliance back >10-15 years ago, and honestly I'm yet to come across anything which assists with internal content discovery quite as well. I really miss it! (Side note, Confluence search is awful)

We[0] have made this. Fully self-hosted. Confluence search is a particularly consistent pain point.

[0] atolio.com

I was hoping for a picture of a box with sundar's signature on it.

Ha, very good.

Richard, we're making the box.

It'll be the next iteration, Sundar's signature edition.

I'm glad they've finally learned to appreciate the conjoined triangles of success.

Let me tell you a story ... in 1999, Google was a little startup, just like we are. And when they started bringing in chefs and masseuses, we thought, "They're nuts!"

But, they were attracting the best possible people, and they were able to create the best product, and now they're worth over $400 billion.

And ... do you know the name of that company?

"Erm, ... Google"

(gets me every time!)

I don't get it. :-(

It's a Silicon Valley HBO series reference

"Bzzt, wrong answer! It's Alphabet!"

Ladies and gentlemen, the Gavin Belson Signature Box!

Feels like something that will almost certainly be sunset in <2 years

Reply on news.ycombinator.com