rastignack

Yes for this particular use case it might help (no per-application aggregated temp file limit though). Also my grid workloads generate thousands of threads running queries with lots of small updates (job queues, calculation results..). I wish there was a way for postgresql to isolate those connections to a limited set of cores (or number of cores).

To avoid completely saturating the server.

Right now we advise using s3 and storing results in files, and asynchronously loading them with batch processes.

Interesting. It would need some work to handle all the workloads I’ve faced where you have two applications with different priorities (ie oltp or grid workloads and analytics). In those cases you want to make sure that BI users do not interrupt the transaction processing by assigning them a set of cores, a priority, and work mem limits for example. Looks doable without major changes.

I would be interested about a more detailed architecture overview of the io scheduler (like this: https://www.scylladb.com/2021/04/06/scyllas-new-io-scheduler...) and the thread scheduler.

PostgreSQL has historically been bad at managing the noisy neighbor problem, but with thread pools, and io priorities, it can be solved.

Has this been tackled here ?

And why it should be done upstream with a thread pool model and an internal scheduler as MS SQL server does.

The thing is you are going to be comoditized on this too. And it’s going to be faster to migrate off your database with AI. You are going to need to have a plan.

I hope it could be used in the future to export pg_dump formated exports to s3.

One would be able to trigger maintenance jobs via simple lambda functions whose duration is capped.

And if you write to a bytes buffer?

Just monitor it and you’re done. I’ve delivered and maintained hundreds of pg instances and never faced this issue. There is so much literature about it that at some point no one even slightly skilled will face it.

I have tens of clusters to maintain. Quite an advertisement for ECS!

Is there now a way to avoid double buffering and use direct IO in postgresql ?

Has anybody seriously benchmarked this ?

I don’t think io uring would make a difference with this setting but I’m curious, as it’s the default for oracle and sybase.

Remember to turn off autoplay on Twitter.

How about acknowledging you’ve been too sharp with words, apologizing, and attempting to move forward ?

I know other people in the kernel do the same mistake as you frequently do on mailing lists. But two wrongs do not make a right.

Some distributions do release hotfixes before they reach mainline kernel. If you expect end users to compile Linus’ kernel head, why couldn’t they compile your branch though ? They get timely fixes.

Maybe then you should have considered this file system as truly experimental and expected your end users to make frequent backups. And advertise it as such. You could also have some kind of dkms bleeding edge module for your users to test fixes before they reach the kernel.

This way you wouldn’t be so preoccupied about getting code as fast as possible in the kernel.

Maybe the FS was upstreamed too soon, causing your development velocity and the high expectations you have for your end users to be at odd with the well entrenched workflows of Linux maintainers.

In any case as a Linux user, I want to thank you for your work and for your code which taught me a lot.

I hope it didn’t take too much of a toll on you.

Let’s hope that with the recent stabilization, the maintenance will be easier.

Mr Torvalds has already said “we’re done”.

And Mr Overstreet’s behavior is still unprofessional.

I think this thread is really insightful to teach highly skilled individuals that people skills are as important as technical proficiency.

I have been this arrogant and abrasive dev. It’s bad and learning how to behave, mainly through parenting, was a net positive for my career.

People, please read and learn from this thread.

One does not justify a wrong by another wrong. It’s a classic sophism.

I do not live in the US but did see some truly shocking and illegal behaviors at gay prides and drag themed events.

So I’m just waiting to get more information.

So my point is to trust the law. I don’t see how controversial it can be.

Nowadays a lot of people script in python

I’ll finish it this summer, hopefully.

I don’t think channels and goroutines suck. For example their usage in golang’s ssh server implementation is eminently readable.

And performant enough.

Engineers ship with them, and do not care if there’s a purer system elsewhere.

I wrote a simple ssh server in zig to learn the language in my spare time.

The new design makes the event loop / io much easier to reason about. Thanks Andy

Would the sync.Pool trick mentionned here: https://hypermode.com/blog/introducing-ristretto-high-perf-g... help ? It’s lossy but might be a good compromise.

Austin Clements at it again…

In rust with some C code, ok. How is the DER format parsed for example ?

Regarding crypto operations, I know as of now for rust projects assembly is a must to have constant time guarantees.

Maybe there could be a way with intrinsics and a constant-time marker, similar to unsafe, to use pure rust.

In the meantime I think there still is too much C code.

It’s a great step in the good direction by the way.

Is not rustls a mix of c++, assembly and rust ?

I think it’s not a good indication of the success of the language.

At $WORK, we are within the range of 2 billion rows per day on one of our apps. We do have beefy hardware and ultra fast SSD storage though.

It’s not about religion. PostgreSQL devs do make money right ? Despite RDS.

It’s good to know it. I live in a heavily regulated workplace and our data usage is constantly monitored.

Good to know a totally offline tool is being considered.

Thanks for the great tool BTW.

It is important to decorelate compute and Storage.

I think people should be aware that the Core Storage work is not beeing done in the open anymore. The process of letting the open source core grow useless while selling proprietary addons is fine. I just have a problem with people calling this open source.

Some prefer using open source. Are they leeches ?

Regarding scaling open source projects, did Linux scale ?

ClickHouse too. Their sharedmergetree is not open source at all. It makes ClickHouse OSS obsolete design-wise. Shame.