Another related aspect is that the VM approach can allow JITing only some of the opcodes, which makes it more feasible and maintainable.
6,287 karma · joined April 14, 2015
andres [at] hn dot anarazel dot de
I've experimented with lowering into x86 "manually" as well, and it did provide gains, but the portability aspects are concerning. And for much of the desirable speedups you ime want more optimization than trivial lowering would give you.
Either way, without the tree walk -> VM move, it'd have been much harder to lower into x86 directly. And you'd still need a fallback for other platforms.
You can e.g. something like storing the session "application user" in a configuration variable (SET myapp.rls_user =...). But if the user can influence the SQL and that's part of the threat model, you need to do more, because that could be changed by further SQL.
Another solution is to just have a session level temp table indicating the current application user.
The paper documents that disabling SMT does not entirely mitigate the problem (In 9.1). They briefly mention trying instructions to avoid the microarchitectural leaks, but don't go into more detail than mentioning verw isn't sufficient.
They state that a switch to/from SGX, with SMT disabled, doesn't prevent the attacks. See 8.1. That's not the same as a cross-vm switch, but it's certainly interesting that the attempts at flushing microarchitectural state when exiting SGX don't provide protection.
"It is possible, however, for two burstable performance EC2 instances to run sequentially (not simultaneously) on the same core. It is also possible for physical memory pages to be reused, remapped, and swapped in and out as virtual memory pages. However, even burstable instances never share the same core at the same time, and virtual memory pages are never shared across instances. "
I only started reading the paper - so I very well might be wrong here - but it doesn't look to me like you need victim/attacker to be scheduled simultaneously on two SMT threads, but that a single core sequentially executing victim / attacker code would be vulnerable. It's possible that the cross-customer "context switch" is larger than the the vulnerable window, but I'd not want to bet on it.
/proc/slabinfo shows some drastic differences:
slabtop -o|head -n 15
with 100k threads running:
Active / Total Objects (% used) : 10134567 / 10280497 (98.6%)
Active / Total Slabs (% used) : 337756 / 337756 (100.0%)
Active / Total Caches (% used) : 132 / 175 (75.4%)
Active / Total Size (% used) : 3598333.95K / 3674904.09K (97.9%)
Minimum / Average / Maximum Object : 0.01K / 0.36K / 11.81K
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
3419013 3418579 99% 0.10K 87667 39 350668K buffer_head
1485372 1484947 99% 0.19K 35366 42 282928K dentry
635299 635038 99% 0.68K 13517 47 432544K proc_inode_cache
575792 575708 99% 1.14K 20564 28 658048K ext4_inode_cache
514794 514703 99% 0.04K 5047 102 20188K extent_status
503140 502198 99% 0.18K 11435 44 91480K vm_area_struct
501738 500514 99% 0.04K 4919 102 19676K vma_lock
381304 378068 99% 0.57K 13618 28 217888K radix_tree_node
Active / Total Objects (% used) : 7235070 / 7678039 (94.2%)
Active / Total Slabs (% used) : 187813 / 187813 (100.0%)
Active / Total Caches (% used) : 132 / 175 (75.4%)
Active / Total Size (% used) : 1751469.45K / 1851083.31K (94.6%)
Minimum / Average / Maximum Object : 0.01K / 0.24K / 11.81K
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
3419013 3418579 99% 0.10K 87667 39 350668K buffer_head
898296 887716 98% 0.19K 21388 42 171104K dentry
575792 575708 99% 1.14K 20564 28 658048K ext4_inode_cache
514794 514703 99% 0.04K 5047 102 20188K extent_status
381304 376987 98% 0.57K 13618 28 217888K radix_tree_node
164352 164352 100% 0.06K 2568 64 10272K kmalloc-rcl-64
157360 53925 34% 0.07K 2810 56 11240K vmap_area
139638 109093 78% 0.04K 1369 102 5476K vma_lockFWIW I've run into malloc self dreadlocks due to rare signals plenty of time :(. In production workloads.
I also quite doubt that std::mutex, and even more so std::condition_variable are guaranteed to be signal safe.
> Out of curiosity, is that something you ever want/hope to achieve in PostgreSQL? Many high-performance systems use this model, but switching a synchronous system in plain C to it sounds uncomfortably exciting, both in terms of the delta and the additional complexity of maintaining the result. To me it seems like a much riskier change than the process->thread one discussed here that Tom Lane already stated will be a disaster.
Depends on how you define it. In a lot of scenarios you can avoid blocking by scheduling IO in a smart way - and I think we can quite far towards that for a lot of workloads and the wins are substantial. But that obviously cannot alone guarantee that you never block.
I think we can get quite far avoiding blocking, but I don't think we're going to a complete asynchronous model in the foreseeable future. But it seems more feasible to incrementally make common blocking locations support asynchronicity. E.g. when a query scans multiple partitions, switch to processing a different partition while waiting for IO.
> Is having a large number of rings (as when supporting a large number of incoming connections) practical? I'm thinking of each ring being a significant reserved block of RAM, but maybe in this scenario that's not really true. A smallish ring for a smallish number of IOs for the query is enough.
It depends on the kernel version etc. The amount of memory isn't huge but initially it was affected by RLIMIT_MEMLOCK... That's one reason why the AIO patchset has a smaller number of io_uring "instances" than the allowed connections. The other reason is that we need to be able to complete IOs that other backends started (otherwise there would be deadlocks), which in turn requires having the file descriptor for each ring available in all processes... Which wouldn't be fun with a high max_connections.
> Speaking of large number of incoming connections, would/could the process->thread be a step toward having a thread per active query rather than per (potentially idle) connection?
Yes. Moving to threads really mainly would be to make subsequent improvements more realistic...
> That'd be a nice operational improvement if it meant folks no longer needed a pooler [1] to get decent performance. All else being equal, fewer moving parts is more pleasant...
You'd likely often still want a pooler on the "application server" side, to avoid TCP / SSL connection establishment overhead. But that can be a quite simple implementation.
That's not needed to benefit from io_uring
> for you to give it a bunch of IO to do at once (batches and chains), and your threads to do other work in the meantime.
You can see substantial gains even if you just submit multiple IOs at once, and then block waiting for any of them to complete. The cost of blocking on IO is amortized to some degree over multiple IOs. Of course it's even better to not block at all...
> If your code is instead designed to be synchronous and thus can only do one IO at a time and needs a syscall to block on it, I don't think there's much if any benefit in using io_uring.
We/I have done the work to issue multiple IOs at a time as part of the patchset introducing AIO support (with among others, an io_uring backend). There's definitely more to do, particularly around index scans, but ...
There's some minor details that are easier with threads in that context, but on the whole it doesn't make much of a difference.
There certainly are times OSs get in the way. But it's hard enough to write a good database, we don't need to maintain a third of an OS in addition.
> Yet rewrites are always easier and more sexy. At first.
Moving to threads would not at all be a rewrite.
I think the post upthread references an out-of-core extension we don't control, which in turn depends on many external libraries it doesn't control either...