jashmatthews

If it works for you it’s great but the RDS EBS limitations are real enough… AWS built Aurora and for RDS a new 3 node topology using local SSDs for writes and EBS only for the data directly.

Most companies can afford not to give a shit until they hit SOC2 or GDPR compliance and then suddenly orphaned data is a giant liability.

Does vacuum not release free pages at the end of an index file in the same way it does for the heap?

> hybrid OLAP+OLTP .... in Postgres this pattern requires careful planning to avoid falling afoul of max_standby_streaming_delay for example

This is a really gnarly problem at scale I've rarely seen anyone else bring up. Either you use max_standby_streaming_delay and queries that conflict with replication cause replication to lag or you use hot_standby_feedback and long running queries on the OLAP replica cause problems on the primary.

Logical Decoding on a replica in also needs hot standby feedback which is a giant PITA for your ETL replica.

Make friends with lawyers.

Purely vibes but as a Kiwi I feel like Number 8 Wire mentality has been dead for at least 20 years now.

Symbols have been GCed since CRuby 2.2 https://bugs.ruby-lang.org/issues/9634

Can we demonstrate them doing that? Absolutely.

Will they fail to do it in practice once they poison their own context hallucinating libraries or functions that don’t exist? Absolutely.

That’s the tricky part of working with agents.

Hallucinations are now plausibly wrong which is in some ways harder to deal with. GPT4.1 still generates Rust with imaginary crates and says “your tests passed, we can now move on” to a completely failed test run.

The USA hasn’t managed to completely impose their idea of intellectual property on everyone yet. Some countries you can’t sign away authorship even if you can commercial rights.

Yeah the older Pilot Sport tyres wore out quickly. I had PS4 before and now PS5 and they are wearing at something like half to a third as fast? Very happy. Only slightly less grip in cold weather.

My understanding is the way Aurora DSQL distributes data widely makes bulk writes extremely slow/expensive. So no COPY, INSERT with >3k rows, TRUNCATE etc

Hey Lev!

I've been looking into PgDog for sharding a 40TB Postgres database atm vs building something ourselves. This could be a good opportunity to collaborate because what we need is something more like Vitess for PostgreSQL. The scatter gather stuff is great but what we really need is config management via something like etcd, shard splitting, best-effort transactions for doing schema changes across all shards etc.

Almost totally unrelated but have you had good success using pg_query.rs to re-write queries? Maybe I misunderstood how pg_query.rs works but re-writing an AST seems like a nightmare with how the AST types don't really support mutability or deep cloning. I ended up using the sqlparser crate which supports mutability via Visitors. I have a side project I'm chipping away at to build online schema change for PG using shadow tables and logical replication ala gh-ost.

Jake

Exactly that. It's an arms race between companies that offer a large number of residential IPs as proxies and companies that run unauthenticated web services trying not to die from denial of service.

https://brightdata.com/

It’s simple high throughput queries that often bite you with the PG query planner.

AWS claims 10x lower latency but I haven't personally checked.

While technologically cool the app is bullshit. 90% accurate isn't accurate enough for the job and it's effectively making shit up.

Are you eating a 10% calorie deficit or a 10% calorie surplus? Cal AI can't tell you.

Not possible to know accurately enough from a picture. Potentially ever.

If you round up great engineering orgs that ship impactful stuff more of them don't use JIRA than do. Linear, Basecamp, Asana, Monday etc.

My experience is by the time an org gets hundreds of priorities and can't effectively delegate to sub orgs they're already fucked and there's no point working there if you want to do anything meaningful.

Intelligence is kinda unnecessary at this point. A Tiktok VP already admitted to a UK parliament select committee that they used to harshly moderate anything the Chinese government found sensitive

Q90 - John Nicolson: It may happen elsewhere, and I can tell you what your official TikTok response was to this leak. You did not deny that these were instructions. In fact, you confirmed that these were instructions, but what you said was that the company had changed its policy in May 2019. Previously, you instructed your moderators to take down videos critical of China, specifically talking about incidents in Tiananmen Square, separatism in Tibet, all straight out of the Chinese Communist Party playbook. You confirmed that is what your moderators did, but your defence was that you had changed your policy in May 2019.

Theo Bertram: It is highly regrettable that that is what it was, but it is not our policy today, nor has it been for a long time.

https://committees.parliament.uk/writtenevidence/13247/html/

When humans learn and copy too closely we call that plagiarism. If an LLM does it how should we deal with that?

Clustered tables and sequential keys have their own downsides though like lock contention on the "last" page.

There are a bunch that offer strong consistency e.g. Cloud Spanner and DynamoDB.

It look like a decline but these are relative numbers right? The whole industry exploded in size over this time period.

Competitive pressure on SaaS companies and startups is much higher now and while there are tons of people are building client wrappers and other much more exciting stuff in Ruby none of its getting extracted, polished and open-sourced because everyone is entirely justifiably freaking out about profitability.

Best to just forget about it and listen on a connection not using pooling.

https://jpcamara.com/2023/04/12/pgbouncer-is-useful.html#lis...

Yeah the ULID/UUIDs which can be be partitioned by time in this way are AWESOME for these use cases.

It's far from trivial. Autoanalyze doesn't work on partitioned tables, only on the partitions themselves. Partitioning a busy job queue table is a nightmare in itself.

By the time you need append-only job statuses it's better to move to a dedicated queue. Append-only statuses help but they also make the polling query a lot more expensive.

Deleting older rows is a nightmare at scale. It leaves holes in the earlier parts of the table and nerfs half the advantage of using append-only in the first place. You end up paying 8kb page IO costs for a single job.

Dedicated queues have constant time operations for enqueue and dequeue which don't blow up at random times.

Do you mean like the consumer for the first phase enqueues a job for the second phase?

Transactional Outbox solves this. You use a table like in the first example but instead of actually doing the ElasticSearch update the Outbox table is piped into the dedicated queue.