liquidgecka

I saw the title and instantly thought of the "Load bearing mac mini" at Twitter.

In our server closet there was a mac mini sitting on another rack mounted server and plugged directly into a switch. IT found it, asked around and nobody knew what it was, so they unpugged it. Immediately the whole of engineering and support were basically offline.

Despite the thing looking suspicious as possible, I had set this thing up as an employee a year before. We were not allowed direct network access to our hosted prod network so as a "stop gap" I setup a SSH tunnel that listened on the mini's IP. At first we used this for access to the support web interface so it could be taken off the internet. At the time my request for a server was rejected. One by one more things got added to the list of things proxied over the device, eventually including basically all internal pages, git access, and about a dozen other random services. I finally got it moved into the server room, but not to real hardware. Once we built a DC we got peered access and the mini finally died.

That's a pretty awful take on engineering culture. I grew up in a city that is one of the most remote in the US and it creates a massive engineering pipeline. It started with civil engineering but moved into ICs, utility power, trains, on and on. Those companies helped build an engineering college which in turn trained engineers.. etc.

None of those companies have had issues getting talent. Not all good engineers want to live in mega urban areas and infact it was quite easy pulling talent away with the promise of a back yard and skiing fifteen minutes away if said talent had kids. Especially when the salary goes 2x as far.

Thats driving on what is known as the "curtilage," the space on a private property that is effectively "partially public". FedEx can drive up a driveway, and use your walkway to deliver a package. That is okay. FedEx can not jump your fence and enter your backyard to leave your package somewhere safe. The backyard is off limits. As such, an irrigation access road is NOT within the curtliage of any property and thus is more of a "backyard" to those not permitted. There are signs and gates which state that yet FedEx directed its drivers down that road several times and they complied.

The people that use it (legally) are the irrigation folks. They use it every day and the property owners can not block access. The private property owners are all commercial developers that have held onto the empty land for a few decades and seemingly have forgotten that they own it given that they never show up in the first place. I can't think of anybody else that would want that to be on maps at all.

I thought that too, but the irrigation district locked a gate when they finished up in October and yet it is still popping up. So if they are using phone traversals then its all foot traffic and that is more terrifying than it being detected via satellite images!

Our local highway district actually makes this data available for free so I checked, its not in there. I also called them and asked about it and the lawyer seemed annoyed that google was adding county easements as public roads too since they alone should have access. So its not coming from the county at least. Its also marked roads where farm tracks are that are not in any way associated with public roads or access. I suspect that there is something looking at the photos taken by aircraft and thinking "that looks like a road!" or something because nothing else would make sense. Why mark something as a road that is just a space between two crops that is driven by tractors?

Also worth noting, its not on OOSM or Bing.. Only Google. So the source is something that only Google is using.

Its illegal to block an irrigation road without prior approval and people straight up ignore signs, especially when the authoritative source says "turn there NOW!" We know this because fedex trucks have traveled down this "road" multiple times despite it being marked as private in several places along the way.

Also, Google is currently being sued for a situation exactly like this so we will know soon enough if that defense works: https://arstechnica.com/tech-policy/2023/09/lawsuit-says-man...

Google lets you edit (or more accurately, suggest edit) maps. It just appears to get overwritten by some process after a few days. I have fixed so many things only to have them snap back to what they were before. Its annoying and leads to situations where door dashers end up coming down an irrigation road, nearly crashing into the canal because its "faster" than using the road that goes around the field. I am shocked Google doesn't have MORE lawsuits on its hands.

Google keeps adding the irrigation access road next to me to maps. This is both dangerous as its totally unprotected and a car could easily drive into the canal, and its private, only the property owners and irrigation district are allowed to use it. I have reported it something like 10 times already and every tie it will be removed for a day or two and then instantly pop back into the map. I don't get how Google even thinks this is a road, or what data it is using to discover it. Its nowhere on any map that I know of!

Ahh yes.. replace the SPOF with a distributed SPOF /s :-)

Fun story time!

Boise State University, and the University of Idaho are two schools at opposite ends of the state of Idaho. UIdaho in the north is close to Spokane, and almost all of its connectivity comes from Seattle. Boise is closer to Salt Lake so most of its connectivity comes via Portland or Salt Lake City. The middle of the state between the two schools is mountains and very, very little large scale connectivity at all, except there was a small line way bad in the day because the UofIdaho had remote classrooms in the southern part of the state. Sometime in the late 90's a network engineer from BSU and one from UofI realized that they both had switches and routing kit in the same building so they ran an ethernet cable between them.

The effect was catastrophic. It turns out that both networks happily started announcing BPG to each other, which in turn announced the connection to the internet as a whole. Suddenly there was a very short jump between networks in Seattle and networks in Salt Lake City. That poor little t1 (iirc) was absolutely getting saturated. But, interestingly only in one direction. See Boise announced the route, but Idaho didn't so the traffic was effectively only failing in one direction.

Needless to say the cable as disconnected and years later when I worked at the UofIdaho it was still well known that the two networks shouldn't ever be connected again! (Which was ironic because I was working on a program to setup I2 at both universities)

Woz still holds a special place in my heart. He was the first to sign my laptop at Twitter (right on the Apple logo even). Talking with him was super interesting even though he was mostly a sales figure for hardware storage. He was super curious about the problem Twitter was facing at the time.

Oh lordy this so much this. As a holder of a beloved 5 character gmail address that is a common first and last name let me tell you that the misdirected email I receive is insane and countless. Services with an reputable unsubscribe get unsubbed and then I moved on. Services with no way to unsubscribe get marked spam each and every single time. Services that do that are effectively dead to me because if they ever do send a legit email gmail will automatically flag and remove them for me and I can't help but believe (with some insider knowledge) that this also impacts deliverability to other gmail accounts.

What's interesting is that blue cruise in my car specifically says "hands off" when it's active. Though it does also say "be ready to take over at any time" and yells at you if you look away.

I didn't realize that Ford was ahead of Tesla in that department!

You still have to build an office building you wouldn't need, run the AC and heater, clean it, etc.

I have all three of these and absolutely none of them contain (or will contain at any point) a three phase motor. All three of these use variable speed motors in my house. I suspect even more will moving forward as well. So honestly three phase is pointless as I just need "power" to drive the variable speed controller system. My hottub has fixed speed 220 pumps but its old. I suspect newer systems would be moving to variable speed as well.

I think your pretty spot on that most people do not need 3 phase, even in the limited cases you mentioned. =)

We had hardware for time sync at Google 15 years ago actually, thats not a new thing. Actually.. we had time sync hardware (via GPS) in the early Twitter datacenters as well until it became clear that was impossible to support without getting roof access. =)

Honestly I would just prefer an interface that can be implemented (not this interface.. one like it that is better thought out).

  type Iterator interface{} {
    // Returns the next object and true, or nil and false if iteration has ended.
    func NextIteration() (interface{}, bool)
  }

Even better (in my opinion) if it also adds a finalizer function to help clean up resources used during iteration and some kind of error handling.. but that might be asking too much given my track record of getting golang improvements approved. I still want else on for statements dang it! =)

That kind of suffers if you want to be able to cheaply cancel the iteration.

Lets say the logic is:

  for item := range list {
      if condition(item) {
          break
      }
  }

That is nice and readable, but if the thing your iterating is returning items in a batch then this gets far, far uglier. Its possible to end up with one loop that fetches x items at a time, and another that iterates over those items.. etc. But if you switch that to channels you maintain the readability and everything is nice! .. Except, you need a way to cancel the iteration, and it can't be closing the channel or the writer will panic.

  items, cancelFunc := apiCall()
  for item := range items {
      if condition(items) {
          cancelFunc()
          for _ := range items { }
          break
      }
  }
  close(items)

Now, this is even worse if you need some kind of finalization of the goroutine that spawns to write to the given channel. Now you have a wait group, or a second channel.. etc. If its possible to error in the middle of iteration then things get complicated as well.

Because at some point you will need to deprecate ciphers and when you do you don't want old clients to explode. The domain is the way you version connection requirements so you can support old clients with crappy ssl options without screwing up the security of new clients.

> We were second in the world in COVID deaths as a percentage of the population.

I am not trying to change your argument in anyway but that specific statement is not correct or its using different numbers than direct COVID deaths. The US is 15th. Peru was #1, and EU members Hungary (#3), Croatia (#6), Slovakia (#10), Greece (#13) and Romania (#14) are all ahead of us.

(Sorry I am pedantic about covid numbers) =)

> While the nine justices agreed that the Sacketts should prevail, they divided 5-to-4 as to how far to go in limiting the EPA's authority.

https://www.npr.org/2023/05/25/1178150234/supreme-court-epa-...

You use the term "This radicalized illegitimate SCOTUS" but you realize that the court was unanimous that the EPA had overstepped its authority under the law right?

I worked in the qualification department for a large memory company around y2k. We ran thousands of computers with samples of all our memory so we could spot issues early. One of our best tests was running Quake in a pre-recorded speed run mode. It could run through in about a minute or so. It torched the CPU and memory and was great at seeing if a system was going to have issues. =)

> just stop mowing

Oof.. this must be regional. Around here if I stop mowing I would have a yard full of tumble weed, goat heads, and white top. All of which are invasive species. The best I can do with native plants that will live without much intervention is sage brush and compacted soil which is neither pleasant to look at or pleasant to live in. =/

When Twitter was still in managed hosting and basically out of space to put new equipment we started an emergency "build a datacenter" project of which I was initially one of the only FTE's allocated too it. My plan was simple: I don't have bodies to throw at problems (we didn't even have data center techs and our facility was still under construction) so my plan was to make something that could effectively "self deploy" a functioning data center for us.

I locked myself in a really hot closet behind the cafeteria with several pieces of our soon to be purchased hardware and started coding. The whole thing ended up amazing and I am still proud of it to this day. =)

Some of my primary goals where simple: No manually edited configs, ever. If you swap a mother board there should be no mac address tracking, no serial number editing, etc. If we can detect it automatically we did and updated the machine database automatically. Hardware techs could simply walk out, swap a drive, and walk away and we would track that.

So we wrote a tool called vex that would export all the machines various bits and bobs. Anything we could access, serial numbers, mac addresses, etc. So once the machine was running we knew where it was, what it was doing, etc. The biggest challenge was getting it running in the first place. The plan was to use DHCP option 82 which includes port information when forwarding DHCP requests to an upstream server. Thus our dhcp server (written by an awesome coworker in like two days) was 100% stateless. It knew the IP range for the datacenter, the ip range for the rack, and the port that the request came from and could respond with a static IP every time. Swap the mother board and it wouldn't matter because the machine would still be plugged into the same port on the switch. Each machine would then request a tftp boot image from the server listed in the dhcp reply. This server was smart, it would look in the machine database to see what state the machine was in and where it needed to go "next" so to speak.

Step 1) Machine has never booted before. It would go into an identify utility that would allow the installer to verify that each machine was plugged into the right ports. The tech could watch the ipmi identify lights on the machine as it activated each in the rack in order and if any where out of order network cables needed fixed. From this point forward we knew that port 3 was the 3rd machine in the rack on the prod and ipmi networks. Once identified the machine would automatically reboot.

Step 2) The machine would run memtest86 to ensure that memory was valid. Sadly I didn't have time to make it automatically update with results and reboot so this step required manual verification when I left. A engineer would check IPMI and mark the test successful.

Step 3) The machine would run stressapptest to push the CPU and memory to the extreme. This allowed us to ensure that they would not overheat and that the rack could pull the load needed to get all the machines to 100%. Once done it would mark itself successful and reboot.

Step 4) The machine would run fio to ensure that every attached disk was in good state and fully functional. Again, it would then reboot automatically.

Step 5) The machine would install our default operating system assigning hostnames and such via parameters from the database.

Lastly once the machine booted into its "real os" for the first time it would setup puppet, install all the back packages, look into the machine database for its assignment and then install all the various bits ad bobs required for it to do its selected function (hadoop, web server, dhcp server, puppet server, etc)

Everything was driven by the machine database, and you could mark a machine as needing burn in and it would repeat the testing steps. Super useful if you suspect a machine is bad in some way but don't have time to debug. Once running vex would export all the details of the machine as well so we could track failure rates, hardware events, etc.

Best part is that the lab ran the same software as the datacenter so we could always test the various bits of this process to see what worked and what didn't. In fact I added several optional steps to do things like get performance profiles and build all kinds of cool graphs from tests on demo hardware.

I ended up bootstrapping three datacenters and all it required was an "Ark" which was just a single server with puppet, dhcp, tftp and audubon on it ready to setup all the other servers in the facility. During our initial power up we had a DHCP server doing something like 100-500 requests per second sustained over the course of like a half an hour (until the memory tests started)

Sorry, not sure why that memory jumped out at me but it seems you did something alike and it reminded me of how much fun it was to be involved in mine. =)

Also, totally unrelated Twitter story (because I am nostalgic lately and you reminded me of it)

Early in my time there we had a major, unexpected load spike in the middle of the day. People where tweeting like crazy, breaking records for volume, etc. The system was groaning under strain but it mostly held. Turns out Michael Jackson had died. We sustained 452 (or maybe 457, it was roughly 450-something) tweets a second. this quickly became 1 "MJ" worth of tweets. We informally used this to define load the entire time I was there. Within a few months we got to a point where we were never BELOW 1 MJ, within a year I think we had peaked at double digit MJs and sustained several even in the lowest periods. Before I left we had hit 1 MJ in photo tweets, etc.

Around the time I left we did something like 300 MJ's per second, and I was only there 3 years.

I think what your describing was actually a problem that I introduced in our bind configuration back when we used DNS for service discovery. Its not exactly what your describing but I can totally see how it would appear that way externally.

So initially we used 2 bind servers with authoritative zones serving using round robin. This worked fairly well as the load on each server was high enough to keep the round robin flowing and the connections from ruby would "stick" for 30 seconds anyway. We had a very large pool of front ends so its not like it mattered. Eventually we had to put a local bind cache on each server, but this introduced another super annoying bug. For some reason, the interaction between the authoritative server, and the caching server would end up causing the served record set to be ordered the same. Normally the authoritative server would serve A, B, C for the first query, then B, C, A for the second, etc. When the cache in the middle for some reason it would reset al queries to A, B, C after the refresh happened. So effectively every 30 seconds it would reset to A, B, C and start round robin again. Since the front ends would connect and stay sticky via connection reuse for a while this meant that A got like 50% of the load, B got 33%, and C got 17%. I am guessing you latched on to this by seeing that one of the servers was horribly underutilized in comparison and reproduced the math accidentally. =)

The fix for this was the massively disgusting, but super effective "make every single machine a DNS zone secondary". Rather than having a simple record cache we just fetched the whole zone if it changed. This actually made DNS quite a bit faster as every machine had a local copy of the zone.. Once that happened distribution returned to normal (20/20/20/20/20) fix for all of our internal services which used DNS for discovery, including snowflake.

Yea.. "auto increment" in mysql. it SUCKED in so many ways, primary of which was that it required a single mysql database to be the primary for all tweet writes. Every tweet just incremented a counter by 1. As we scaled up that server got overloaded rapidly and became a huge single point of failure. Snowflake was introduced to solve several problems at once. =)

I am pretty sure that snowflake didn't use a mod of a signed int32. It used a service discovery pool as part of finagle (and prior to that dns iirc). The server used a very simple method internally to convert time into a integer (that was 52 bits because of javascript). In fact it was completely open source: https://blog.twitter.com/engineering/en_us/a/2010/announcing...

The integer generation was pretty simple, there was a fixed id of each server, and unless I a mistaken we have 5 servers per datacenter. Each id was basically <time><offset><id> where time was a millisecond timer, offset was the number of ids generated in that same millisecond by the same server, and id was the machines unique identifier. When we first talked about this process I thought that offset was going to roll, every id would increment it by one. This was changed to resetting it every millisecond specifically so that it would obscure tweet volumes.

At the time I remember reading a LOT of articles estimating tweet volume and most of them were way, way off. I don't know that we ever really put effort into correcting them though. =)

* - Does not account for changes in the system post 2012.