liquidgecka

> I've been using GitHub for other projects and for the life of me I can't see a single area where its better.

Triggering github actions manually is way, way cleaner. Also the pipeline configuration feels cleaner to me bit that might be personal preference more than anything else.. Otherwise I agree. =)

That meme started in early 2007 I believe. I started in 2006 and was in ZRH by 2008 and it was around long before I made that move.

There is confusion here because Greenland is not part of the EU directly (they were, they left) https://en.wikipedia.org/wiki/Greenland_and_the_European_Uni... Its citizens are members of the EU but its territory is not. Greenland is part of NATO though, and has a trade alliance with the EU so its territorial status is very complicated.

Yeah.. I have a five letter email that's a common first and last name @ gmail.com. I second everything you said. Getting report hits every few days are useless given how few sites do any kind of validation. :-/

My former company would send out rewards as a thank you to employees. It was basically a “click here to receive your free gift!” email. I kept telling the security team that this was a TERRIBLE president but it continued none the less. The first time I got one I didn’t open it for ages, even after confirming the company was real. It was only after like the 5th nagging email that I asked security about it and they confirmed that it was in fact a real thing the company was using. I got a roomba, a nice outdoor chair, and some sweet headphones. =)

yea I totally missed the CDU. I thought this was a project I had talked with a hardware person about a few years ago where there was no intermediate transfer and when I read the article I completely missed the section between the images. Rack level water cooling is interesting and I am sure they are doing some really cool bits on it but it’s not as revolutionary as a zero transfer system that I had thought they were talking through. I updated the comment to call out my error and reduce my excitement. =/

[I am still annoyed at how many people are dismissive of Google’s datacenter work simply because “severs have been water cooled before” which completely misses the point of datacenter level cooling. I also learned that AWS is doing this already, along with some elements of OVH] =)

> > CDUs exchange heat between coolant liquid and the facility-level water supply.

Oh interesting I missed that when I went through in the first pass. (I think I space bared to pass the image and managed to skip the entire paragraph in between the two images so that’s on me.

I was running off an informal discussion I had with a hardware ops person several years ago where he mentioned a push to unify cooling and eliminate thermal transfer points since they were one of the major elements of inefficiency in modern cooling solutions. By missing that as I browsed through it I think I leaned too heavily on my assumptions without realizing it!

Also, not all chips can be liquid cooled so there will always be an element of air cooling so the fans and stuff are still there for the “everything else” cases and I doubt anybody will really eliminate that effectively. The comment you quoted was mostly directed towards the idea that Cray-1 had liquid cooling, it did, but it transferred to air outside of the server which was an extremely common model for most older mainframe setups. It was rare for the heat to be kept liquid along the whole path.

It has not been true for a LONG time. That was part of Google early “compute unit” strategy that involved things like sealed containers and such. Turns out that’s not super efficient or useful because you leave large swaths of hardware idle.

In my day we had software that would “drain” a machine and release it to hardware ops to swap the hardware on. This could be a drive, memory, CPU or a motherboard. If it was even slightly complicated they would ship it to Mountain View for diagnostic and repair. But every machine was expected to be cycled to get it working as fast as possible.

We did a disk upgrade on a whole datacenter that involved switching from 1TB to 2TB disks or something like that (I am dating myself) and total downtime was so important they hired temporary workers to work nights to get the swap done as quickly as possible. If I remember correctly that was part of the “holy cow gmail is out of space!” chaos though, so added urgency.

> The part that is new is not having an air-interface in the middle of the cycle.

I wasn’t clear when I was writing but this was the point I was trying to make. Heat from the chip is transferred in the same medium all the way from the chip to the exterior chiller without intermediate transfers to a new medium.

[I am not a current Google Employee so my understanding of this is based on externally written articles and “leap of faith” guestimation]

Yes. A supply and return line along with power. Though if I had to guess how its setup this would be done with some super slick “it just works” kind of mount that lets them just slide the case in and lock it in place. When I was there almost all hardware replacement was made downright trivial so it could just be more or less slide in place and walk away.

[bri3d pointed out that I missed an element of this. There is a transfer between rack level and machine level coolant which makes this far less novel than I had initially understood. See their direct comment to this]

I posed this further down in a reply-to-a-reply but I should call it out a little closer to the top: The innovation here is not “we are using water for cooling”. The innovation here is that they are direct cooling the servers with chillers that are outside of the facility. Most mainframes will use water cooling to get the heat from the core out to the edges where traditional where it can be picked up by the typical heatsink/cooling fans. Even home PCs do this by moving the heat to a reservoir that can be more effectively cooled.

What Google is doing is using the huge chillers that would normally be cooling the air in the facility to cool water which is directly pumped into every server. The return water is then cooled in the chiller tower. This eliminates ANY air based transfer besides the chiller tower. This is one being done a server or a rack.. its being done on the whole data center all at once.

I am super curious how they handle things like chiller maintenance or pump failures. I am sure they have redundancy but the system for that has to be super impressive because it can’t be offline long before you experience hardware failure!

[Edit: It was pointed out in another comment that AWS is doing this as well and honestly their pictures make it way clearer what is happening: https://www.aboutamazon.com/news/aws/aws-liquid-cooling-data...]

As somebody that worked on Google data centers after coming from a high performance computing world I can categorically say that Google is not “re-learning” old technology. In the early days (when I was there) they focused heavily on moving from thinking of computers to thinking of compute units. This is where containers and self contained data centers came from. This was actually a joke inside of Google because it failed but was copied by all the other vendors for years after Google had given up on it. They then moved to stop thinking about cooling as something that happens within a server case to something that happens to a whole facility. This was the first major leap forward where they moved from cooling the facility and pushing conditioned air in to cooling the air immediately behind the server.

Liquid cooling at Google scale is different than mainframes as well. Mainframes needed to move heat from the core out to the edges of the server where traditional data center cooling would transfer it away to be conditioned. Google liquid cooling is moving the heat completely outside of the building while it’s still liquid. That’s never been done before as far as I am aware. Not at this scale at least.

… isn’t that basically what CentOS did in the early days?

Its worth calling out that abstractions can kill you in unexpected ways with go.

Anytime you use an interface it forces a heap allocation, even if the object is only used read only and within the same scope. That includes calls to things like fmt.Printf() so doing a for loop that prints the value of i forces the integer backing i to be heap allocated, along with every other value that you printed. So if you helpfully make every api in your library use an interface you are forcing the callers to use heap allocations for every single operation.

I wrote a tool to handle micro blobs specifically because we were being heavily rate limited by S3 for both writes and reads. We got about 3k/s per bucket before S3 rate limiting started kicking in hard.

Granted we also used said tool to bundle objects together in a way that required sezo state to track so that we could fetch them as needed cheaply and efficiently so it wasn't a pure S3 issue.

My brother used to own <our uncommon family name>.com and wrote on it a bunch. Eventually he bailed out and let it expire. It turned into a porn site for a few years and now its for sale for like $2k from some predatory reseller.

If they try make sure you assert that you do not consent to searches, and would like to be on your way.. Then when they try to hold you ask them if the detention is inline with `Rodriguez v. United States` which specifically forbids cops from delaying a driver so that they can get dogs to the scene.

I dealt with the Twitter office move stuff and there was a real honest to goodness push to get is to love to an office in South San Francisco so we could avaint the payroll tax and have parking. Had it not been for the tax break I suspect they would have left SF completely.

My last five or so takeout runs just failed with no reasoning why. When I try to save to google drive it just fills up the 2TB storage allocation I have without ever making actual files, or worse, making dozens of oddly names files of incorrect sizes.

I filed like five support requests and never got any actual replies.

My next step is going to be moving off google for most things, but amespecially for photo storage (80% of my usage) which I was relying on google takeout for a backup solution.

As a former Googler i have pretty much given up on google being useful these days. :-/

yep.. And it would split uploads across dozens of accounts with parity so that if any account was disabled it could re-create the data from what was in the other accounts. (think bittorrent using imap uploaded content in gmail)

libgmail was the least of our problems. There was a Polish software team that wrote a bittorrent layer on top of Gmail. That thing was a pain in the butt as they constantly improved it to get around abuse filters and such. Plus it had parity bits so if we killed accounts it would just re-replicate the data to new accounts.. That software was devilish and impressive at the exact same time. =)

Pre-AI we had a system that watched user patterns and would identify possibly suspect patterns that were outside of the norm. We also had system that would content-id the images and attachments to see what was going uploaded in a general way. Given enough suspicion then the account would be opened to look for abusive patterns.

There is absolutely no promise on any cloud hosted services that a human will not ever see your data. However, at Google it was made very, very, VERY clear that if we had to scan somebody's personal email for any reason then discussion of the contents outside of legally mandated, or required for work ways would lead to immediate termination and possible lawsuit for any damages to reputation incurred.

While fixing user accounts, or dealing with delivery of content I saw epic piles of personal email. Besides the ones full of CASM or other abusive material I couldn't say that I ever remembered the contents 30 minutes later. Its like a checker at a grocery store. They don't care about whatever embarrassing tings your buying and won't remember you 10 minutes later. =)

I was apparently not watching this well enough, sorry for the delayed response.

Deliver was because we ran the SMTP and queuing infrastructure at the time. We started as Gmail SRE, then split out some of the delivery and abuse services into its own team (SAD), then SAD got SRE, hence SAD-SRE =)

And Gmail and Google groups, and Google drive, and Gchat, on and on. The data you store doesn't even have to be public. With Gmail they would distribute credentials to log in and read attachments that they uploaded via imap.

(I am a former Google SAD-SRE [Spam, Abuse, Delivery])

I just migrated my blog and popular posts from medium. I noticed a ton of issues as well. What's interesting is that a metric ton of traffic to domain is directed at articles unrelated to me. Somehow medium was serving a ton of content on my domain which is a huge issue for me.

In 2010-2011 time frame the mongo team sponsored an effort inside of Twitter to replace MySQL with mongo. There was a group of us that worked to migrate the tweet store to Cassandra that were able to talk with leadership and get that initiative killed. Turns out migrating HIGHLY structured data into mongo was never a good idea, especially at that scale.

I couldn't think of a better title here but Twitter's use of /<username> causes problem when they launch features, or have other static content. I was there when we first discovered that @Flash's profile have become a 403 page randomly so I figured I would document that story a bit. There is still even still users in this broken state today. =)

This is a more formal version of a comment I made here a year or so ago. I wanted to write it up and put it out there because I still get comments on this over a decade later. Lots of people assume that it shows how silly Twitter was but honestly it was a product of the restrictions we had on our environment, and the time we had to solve things quickly vs the right way.

I do love telling the story though. It is very much "startup engineering" at its finest. =)

Most startups I deal with these days don't even have offices anymore, let alone network ports. =)

But yea.. same deal with wifi. Its amazing how often the wifi password is posted on something visible. In fact I have found the password in so many public images which means somebody on the street could just connect to the wifi network fomr the street.

... At the end of the the deal with had two mac mini's with auto fail over configured via health checking. Thank goodness we got rid of the service provider that refused to let us pair with the network. Once that happened the networking team could just do normal peering with a standard router.

But yea, in the early days that mac sat on my desk. It only got moved when I pointed out the issue to our new security team and their jaw hit the floor. =)