stlava

My colleague worked on Lumina which solves a related problem of tracking compute cost with discounts and spot pricing: https://github.com/Nextdoor/lumina

Those who hire a crisis management team rarely win in the court of public opinion. BAM just continues to dig a hole when they could instead facilitate fixing the situation regardless of if they think they are a party to this or not.

At the end of the day there has to be a tradeoff between ease of use and performance. Having spent a lot of time optimizing high throughput services in go, it always felt like I was fighting the language. And that's because I was... sure they could add arenas but that just feels like what it is, a patch over the fact you're working alongside a GC.

The movie seems like a fluff piece when you find out what has transpired at DeepMind subsequently with slowing down publishing material to “selling out to product” which the founder was hell bent against in the documentary.

I had some experience with RTK and sensor fusion about 13 years ago on a college project. At the time the only people that were using RTK in real world applications were tractor companies because it kind of matters that you're precise when seed drilling. I'm not sure how far RTK has come but it had pit falls back then like base station drift when the set of satellites it saw changed due to them going over the horizon.

Tractors don't go _that_ fast but I'm not sure I'd rely on it in an actual car for anything but slightly better GPS.

My worry is we're going to have a generation of engineers that have not built up the necessary critical thinking/pattern matching skills needed to weigh tradeoffs and bring in context to ask the right questions and interpret the answers.

Sure we can segment this into code generation models and code review models but are engineers really going to want to be questioned by a code review tool on "what are you trying to do?" or are they just going to merge and pull the slot lever again?

I’m impressed they managed to make an app that makes me not want to use it or my phone to take photos! After I used it for about 5 mins I resolved to dust off my older DSLR and use it instead.

I feel that if you need an LLM to help pivot between existing data it just means the operability tool has gaps in user functionality. This is by far my biggest gripe with DataDog today. All the data is there but going from database query to front end traces should be easy but is not.

Sure we can use an LLM but I can for now click around faster (if those breadcrumbs exist) than it can reason.

Also the LLM would only point to a direction and I’m still going to have to use the UI to confirm.

We've discussed getting backup power periodically since moving (most of our neighbors have something). After the big bomb cyclone that hit the NPW this became a priority and I did a very similar tradeoff matrix.

Here were some of my consideration points against a generator (I had a list for batteries too):

1. the previous owners electrified the house so having a tank just for a generator didn't make sense

2. per 1, a small tank wouldn't last very long and if we're out of power for multiple days gas delivery is unlikely to be happening anyway.

3. tank + generator didn't have a practical placement location for us

4. smaller portable generators didn't make sense from a maintenance perspective since they didn't auto test.

5. what happens if it fails a self test right before a storm?

6. we get intermittent power cuts / fluctuations / outages throughout the year and the generator + ATS wouldn't protect sensitive electronics well

Edit: ATS + batteries can play nice together then I might look at doing a small portable as aux backup

> In the year 2030, no one will remember Kubernetes.

I highly doubt that. Maybe there will be an evolution to k8s but fundamentally it solves a whole host of challenges around defining the environment an application runs in.

Nice! I'm one of the authors of pg-bifrost which is in the same space. Have you thought about / have solved sharding consumption across multiple slots / multi consumers to increase throughput? This is on my radar but not something I've investigated yet.

The issue we've ran into is some team at work decides to re-write an entire table and things get backed up until they stop updating rows.

Sounds like the author is missing an interface to abstract the DB backend.

My team manages a handful of clusters at work and I wrote on an internal redis client proxy (it's on my todo list to opensource). A few things I tell other teams to set them up for success (we use Elasticache):

- Connection pooling / pipelining and circuit breaking is a must at scale. The clients are a lot better than they used to be but it's important developers understand the behavior of the client library they are using. Someone suggested using Envoy as sidecar proxy, I personally wouldn't after our experience with it with redis but it's an easy option. - Avoid changing the cluster topology if the CPU load is over 40%. This is primarily in case of unplanned failures during a change. - If something goes wrong shed load application side as quick as possible because Redis won't recover if it's being hammered. You'll need to either have feature flags of be able to scale down your application. - Having replicas won't protect you from data loss so don't treat it as a source of truth. Also, don't rely on consistency in clustered mode. - Remember Redis is single threaded so an 8xl isn't going to be super useful with all those unused cores.

Things we have alarms on by default: - Engine utilization - Anomalies in replication lag - Network throughput (relative to throughput of the underlying EC2 instance) - Bytes used for cache - Swap usage (this is the oh shit alarm)

Right, so basically taxi cabs but personalized ads. GG on marginally progressing the wheel.

I wrote and operate a high throughput go service with somewhat tight memory requirements. It became impracticable to try tuning the memory footprint to avoid OOMs while balancing gc pauses with just the GOGC flag. I'm happy to say that I started using the soft memory limit shortly it landed and it's been a game changer for us.

If data storage isn't your startup's job then I would negotiate heavily on the AWS contract.

I doubt the city will see a penny from this. There's nothing in place to do an audit to figure out who should be taxed.

That didn't take long...

Is the problem that leetcode questions were used or that coding was a hard requirement for a devops position? I do screeners for my team and being able to write code is a hard requirement for us. If you can't reason through and write the equivalent of fizz-buzz it's a problem.

There was an NPR clip a month or so ago that likened homeowners to a cartel.

The post is good but just scratches the surface on running Kinesis Streams / Lambda at scale. Here are a few additional things I found while running Kinesis as a data ingestion pipeline:

- Only write logs out that matter. Searching logs in cloudwatch is already a major PITA. Half the time I just scan the logs manually because search never returns. Also, the fewer println statements you have the quicker your function will be.

- Lambda is cheap, reporting function metrics to cloudwatch from a lambda is not. Be very careful about using this. - Having metrics from within your lambda is very helpful. We keep track of spout lag (delta of when event got to kineis and when it was read by the lambda), source lag (delta of when the event was emitted and when it was read by the lambda), number of events processed (were any dropped due to validation errors?).

- Avoid using the kinesis auto scaler tool. In theory it's a great idea but in practice we found that scaling a stream with 60+ shards causes issues with api limits. (maybe this is fixed now...)

- Have plenty of disk space on whatever is emitting logs. You don't want to run into the scenario where you can't push logs to kinesis (eg throttling) and they start filling up your disks.

- Keep in mind that you have to balance our emitters, lambda, and your downstream targets. You don't want too few / too many shards. You don't want to have 100 lambda instances hitting a service with 10 events each invocation.

- Lambda deployment tools are still young but find one that works for you. All of them have tradeoffs in how they are configured and how they deploy.

There are some good tidbits in the Q&A section from my re:Invent talk [1]. Also, for anyone wanting to use lambda but not wanting to re-invent checkout Bender [2]. Note I'm the author.

[1] https://www.youtube.com/watch?v=AaRawf9vcZ4 [2] https://github.com/Nextdoor/bender

edit: formatting

I just listened to it last night. It was an eye opening episode. You could see how easy it is for desperate people to fall victim to the brokers.

I run a fairly high scale pipeline on lambda. The post touched briefly on logging but wanted to chime in with a few helpful points:

- Format all logs in json. This allows you to use the json path filtering option on CW.

- Add function version & tag to the log lines. CW doesn't allow you to filter based on version.

- Don't expect CW log querying to work all the time.

- Output runtime stats to CW metrics.

edit: formatting