hasyimibhar

Having worked at enterprise with 100+ engineers across multiple teams with complex reporting lines, I definitely agree with this. There's a lot of nuance behind decisions that docs simply can't capture. I mean in theory you can probably write every considerations that led you to make a certain decision, e.g.:

- I'm writing this service even though team X has built the same thing, because my team lead doesn't trust team X since the last time we depended on their service 3 years ago, they had a major downtime that screwed us up big time

- This service is using AWS Lambda simply because I think it's cool, despite the fact that the company has a dedicated team running k8s stack with argocd, argo rollouts, KEDA, etc for the entire company

- Service Y is written in this particular way because it's a service that is shared with another team that came from a company that was acquired, and they wouldn't use it unless we write it this particular way, and making the top execs happy is more important than dealing with a small tech debt (this is probably true)

But no one is going to write these in their RFC. But Sarah knows.

AFAIK that's what Multigres[0] and Neki[1] are trying to solve.

[0] https://multigres.com/ [1] https://neki.dev/

Looks cool, I've been waiting for someone to build this since dbt and SQLMesh acquisition. It would be great to have model versioning and support for ClickHouse SQL.

Slightly off-topic but Datalog related: is there a way for me to query data in ClickHouse using Datalog without any ETL? I want to do fraud analysis, and I've been reading about how Datalog is a lot better at these kind of use cases than SQL.

Ok then it's definitely the author's fault for clicking "Always Allow". I don't even trust my agent to run grep without approval, let alone curl.

I'm not familiar with Cursor, does it allow the agent to have access to run "curl -X POST" with no approval, i.e. a popup will show up asking you to approve/deny/always approve? AFAIK with Claude Code, this can only happen if you use something like "--dangerously-skip-permissions". I have never used this, I manually approve all commands my agent runs. Pretty insane that people are giving agents to do whatever it wants and trusting the guardrails will work 100% of the time.

I've experienced something like this at work but with data warehouse instead, and it happened multiple times (to be fair, data engineering is still fairly new where I'm from).

One example was an engineer wanted to build an API that accepts large CSV (GBs of credit reports) to extract some data and perform some aggregations. He was in the process of discussing with SREs on the best way to process the huge CSV file without using k8s stateful set, and the solution he was about to build was basically writing to S3 and having a worker asynchronously load and process the CSV in chunks, then finally writing the aggregation to db.

I stepped in and told him he was about to build a data warehouse. :P

I'm not sure why you are giving your agents write access to query your OLTP database, let alone write to it. The pattern that I use at work is:

- Read access through OLAP, not OLTP. You just need to setup a near real-time replication between your OLTP and OLAP.

- Write access through API, just like your application. You can add fancy things like approval layer, e.g. you agent cannot "ban_user(id)", but it can "request_to_ban_user(id)", and the action only happens once you approve it.

> Why

Based on my experience with Claude, it's pretty damn good at doing data analysis, if given the right curated data models. You still need to eyeball the generated SQL to make sure it makes sense.

> and how?

1. Replicate your Postgres into Snowflake/Databricks/ClickHouse/etc, or directly to Iceberg and hook it up to Snowflake/Databricks/ClickHouse/etc.

2. Give your agent read access to query it.

3. Build dimensional models (facts and dimensions tables) from the raw data. You can ask LLM for help here, Claude is pretty good at designing data models in my experience.

4. Start asking your agent questions about your data.

Keep steps 3-4 as a tight feedback loop. Every time your agent hallucinates or struggle to answer your questions, improve the model.

Side note: I'm currently building a platform that does all 3 (though you still need to do 2 yourself), you just need Postgres + 1 command to set it up: https://polynya.dev/

It's displayed Cloudflare Workflows dashboard: https://developers.cloudflare.com/workflows/build/visualizer...

Typically it's not pure random, instead they randomize the ordering of the 7 pieces, otherwise called the 7-bag system. Some games use different bag size, e.g. 35-bag system used in TGM3. AFAIK with 7-bag, a good Tetris player can play pretty much indefinitely, assuming that the speed of the falling pieces never increase, and that they can pause the game to take rests.

I used to play Nullpomino (an open source tetris game) for hours. It has a mode called marathon, where the goal is to play as long as possible. It's normal to see good players surviving for hours, even with increasing level. Here's an example of someone playing for 1h, 90% at level 20 (at level 20, pieces appear immediately at the bottom): https://www.youtube.com/watch?v=rHXxKywAQME

I wonder if it's possible to combine statecharts with durable execution engines like Temporal, DBOS, Restate, etc. At work we use Cloudflare Workflows for managing onboarding and payment workflows. It generates flowchart diagram that is useful for quickly reasoning about what the workflow does, which I guess is what statecharts is trying to achieve.

Ursa has been around for a while (I first heard about it last year), but it stood out to me among the many diskless Kafka implementations because it actually treats Iceberg as first class, i.e. Iceberg is the storage. Other implementations like AutoMQ, RedPanda, and WarpStream treats Iceberg as tiered storage, i.e. you write to Kafka, Kafka writes to disk and ack, then Kafka will asynchronously write to Iceberg in the background. However it also means that Ursa cannot be a drop-in replacement of Kafka, especially if you need milliseconds end-to-end latency (from producer to consumer), since cloud storage + Iceberg adds a lot of latency.

But from my experience, I'd argue 90% of use cases of Kafka don't need sub-100ms latency. In my previous company (fintech company), pretty much 99% use cases are for asynchronous processing that can tolerate seconds of latency. We just use Kafka because it's there, there's a team that operates it for everyone so the ugly parts are hidden from everyone else, and the cost is pretty much "subsidized" (the cost is attributed to the team that operates it).

We use MCP at work. In my team of about 6 people, everyone has Claude access, but about half of us are non-engineers. I built an MCP over our backend and Clickhouse, and setup a Claude Project with instruction (I'm assuming this count as skill?). The instruction is mostly for enriching the analytics data that we have, e.g. hinting Claude to prefer certain datasets for certain questions.

This allows the non-engineers (and also engineers) to use Claude Desktop to do day-to-day operations (e.g. ban user X for fraud) and analytics (e.g. how much revenue we made past 7 days? Any fraud patterns?). The MCP helps to add audit, authorization, and approval layer (certain ops action like banning user will require approval).

Mostly lack of sleep and having to sacrifice a lot of weekends to work on the assignments. I took OMSCS with the goal of learning as much as possible since I did not have a traditional CS degree, so I took all the hard classes in the Computing Systems track (DC, Compilers, ISL:BE, IHPC, etc).

CS7210 (DC) is the hardest class I took. We had to write a correct Paxos implementation, then use it to build a distributed sharded KV store. I remember having to spend 10+ hours on a single day during weekend. It was worth it though, learned a lot about distributed consensus, and how difficult it is to get right (there were test failures that were fixed after hours of debug, and the fix was literally changing the order of some code lol).

I completed OMSCS with 2 kids (both preschool), taking 1 class per semester from Fall 2019 to Spring 2023. It was possible thanks to having full remote job and a very supportive wife. I learned a lot, but I probably wouldn't do it again, it was extremely unhealthy, especially for certain classes like Distributed Computing (CS7210).