wtarreau

Agreed. I think they purposely looked for a reputably difficult task for a benchmark between their models, without high expectations beyond that.

In my opinion it's the opposite. This type of associations is welcome, and they are fine to promote free software and help people, but they are exactly like neighborhood associations: they're mostly local, relying on volunteers with limited time who become a new dependency for people who were not using these services. That's fine for limited use case but it doesn't scale at all, and causes a huge duplication of efforts (organization, software creation -- several of them reinvented stuff that already exists, advertising etc). And associations rarely if ever merge, because most often association creators have a very clear view of what they're seeking (often idealist) and are rarely willing to compromise it and accept to adopt another association's mostly similar but not exactly identical goals (it often works very similarly to political parties). GAFAMs would never exist under this model.

The problem is that such services which proudly run on low budget, volunteers and recycled hardware, cannot be relied on by companies without risking to enter legal trouble in case of major incident, so it means that a higher-grade service is still needed, with a dedicate funding, and we're facing fragmentation. We must not reproduce the scheme of cloudwatt either. Too much money injected into a wet dream that was only used to spend lots of money in consultants coming here just to confirm their presence and get their check.

What is needed instead is to sponsor the development of such activities by a few (2-3) well-established competing companies, so as to avoid the regular risk of monoculture that diverges from what users expect, and help them reach the point where their offerings can compete with GAFAM's for both end users and enterprise. The contract should be clear that services must rely on open formats, make it possible for leaving users to retrieve all their data, that software developed under such funding must be opensource, though technology acquisition is fine, and that these offerings must become self-sustaining at one point (i.e. a mix of free+paid services). The EU funders should have enough shares of these activities so that their permission is required for business acquisition and that they can restrict it to EU-based companies, so that such companies can still grow and seek public funding.

What we need is a few durable big players, not 10000 incompatible associations each with their own software suite, that no enterprise can trust over the long term and that cannot resist a trivial DDoS by lack of a robust infrastructure, and who are not organized enough to run full-stack security audits to make sure that user data are properly protected. These ones are only fine for friends and family but that's not what we're missing the most (the proof is that they already exist).

No, it needs first to encourage local investment. Companies who seek investors or who get sold do not do it by pleasure, but as a last resort before dying. And in the EU you don't get any offer to save a company that has a limited commercial activity. Many companies die every day by lack of money. It turns our that US etc are willing to take much more risk and invest sufficient money to transform such fragile companies into durable ones. And in the case of software it's great, because software is sold all over the world, and the income serves to hire more local people, so in the end it's a way to really develop EU sales to the rest of the world. It would clearly be better if the investors were EU-based, but at least it's better than nothing that some investors are willing to risk their money on such companies.

Just doesn't work for me, it says "here's the combined image" after ~10s but shows nothing at all. Maybe already victim of its success ?

Blaming the audience makes sense because after all, they're the ones not getting the message right and not asking the presenter to explain it better. But it remains the presenter's failure to catch their attention better and try to deliver a clear message.

Every time I had a presentation, I tried to analyze the failures (including listening to me when it was recorded, a really painful experience). Certain mistakes such as like having slides on a white background that makes attendees look at the screen and read instead of watching the presenter and listening to him can be devastating. Just because attendees are naturally attracted by light. It's not the audience's fault, it's the presenter's fault (and to some extents the tools in use). A good exercise is to stop slides from time to time during the presentation (i.e. switch to a black one), you'll be amazed how much you suddenly catch the attention, you feel like you're at a theater. It even manages to catch attention of those who were looking at their smartphones because the light in the room suddenly changes.

Also another difficulty which is specific to English native speakers is that many of them initially underestimate the difficulties of the audience to catch certain expressions (with some people it's very hard to distinguish "can" from "can't" for example, which complicates the understanding), or idiomatic ones, or references to local culture, because such things are part of their daily vocabulary. Of course, after a few public talk, when they get questions at the end proving there were misunderstandings, they realize that speaking slower, articulating a bit more and avoiding such references does help with non-native listeners. Conversely, when you present in a language that is not yours, you stick to very simple vocabulary using longer sentences to assemble words that try to form a non-ambiguous meaning. It can probably sound boring for native speakers but the message probably reaches the audience better.

In any case, it definitely always is the presenter's failure when a message is poorly delivered and their responsibility to try to improve this, however difficult this is. It's just important never to give up.

Thanks for the pointer, it looks particularly interesting. I'm not good with the terminology and it always takes me a while to figure which properties we're talking about starting from a name. But the reported times in the article look pretty good and certainly interesting to consider. One difficulty I'm facing with ebtrees and strings is that I need the position of the first difference, that strcmp() doesn't return to you, and that if you reimplement yourself in multi-byte matches at once (I did it already), it will quickly upset valgrind and sanitizers for reading out of bounds. That makes such functions annoying to adopt in various projects as it requires more customizations. So I kept the hand-crafted one-byte-at-a-time comparison, but figured that other approaches based on just a comparison (strcmp, like used in rbtrees) could actually be a win for this reason.

The problem is not the performance of the low-level crypto code IMHO, but how it interfaces with the rest, which is where you're crossing a myriad of locks (and atomic ops for newer versions) that cost a lot as soon as you're interested in using more than one CPU core :-/

Just to be clear, we don't care at all about performance of 1.0. The tests resulting in the pretty telling graphs were done in 1.3 only, as that's what users care about.

Absolutely. Sometimes when using OpenSSL in performance tests, you notice that performances vary significantly just by switching to a different memory allocator, which is totally scary.

I hadn't seen the conversation above, thanks for the pointer. It's surrealistic. I don't see how having to support multiple file formats requires to invest so many allocations. In the worst case you open the file (1 malloc and occasionally a few realloc) and you try to parse it into a struct using a few different decoders. I hope they're not allocating one byte at a time when reading a file...

Not to mention the catastrophic security that comes with these systems. On a local ubuntu, I've had exactly 4 different versions of the sudo binary. One in the host OS and 3 in different snaps (some were the same but there were a total of 4 different). If they had a reason to be different, it's likely for bug fixes, but not all of them were updated, meaning that even after my main OS was updated, there were still 3 bogus binaries exposed to users and waiting for an exploit to happen. I find that this is the most shocking aspect of these systems (and I'm really not happy with the disrespect of my storage, like you mention).

But it's the same for other long sessions such as slow downloads and git clones. Sites concerned by the number of source ports are not those dealing with just favicon.ico and bullet.png, but mainly those dealing with long transfers.

Also there's a cascade effect on large sites, where as long as your servers respond fast, everything's OK. Suddenly a database experiences a hiccup, everything saturates, and once you enter the situation where the LB has all of its ports in use, it can take a while to recover because of connect() getting much slower (I already observed delays up to 50ms!). At this point there's no hope to recover in a sane time, because excess connections are not even served by the servers, they're in the accept queue in the system, so they keep a port busy, slowing down connect() which means more even connections are needed for other incoming requests. If the LB is not properly sized and tuned, you'd rather just kill it to get rid of all the connections at once, wait a second or two for the RST storm to calm down and start again.

H2 can avoid that, at the expense of other issues I mentioned in another response above (i.e. don't multiplex too much to the servers, 5-10 streams max, to avoid the risk of inter-client HoL). But H2 also comes with higher xfer costs than H1 for large objects due to framing.

Yep. Actually H1/H2/H3 do have the same problem (remember the good old days when everyone was trying to pipeline over H1?), except that H1 generally comes with multiple connections and H3 currently goes over QUIC and it's QUIC that addresses HoL by letting streams progress independently.

If you transfer large objects, H2 on the backend will increase transfer costs (due to framing). If you deal with many moderate or small objects however, H2 can improve the CPU usage for both the LB and the backend server because they will have less expensive parsing and will be able to merge multiple messages in a single packet over the wire, thus reducing the number of syscalls. Normally it's just a matter of enabling H2 on both and you can run some tests. Be careful not to mix too many clients over a backend connection if you don't want slow client to limit the other ones' xfer speed or even cause head of line blocking, though! By typically supporting ~10 streams per backend connection does improve things quite a bit over H1 for regular sites.

It's amazing how people having visibly never dealt with high loads can instantly become vehement against those reporting a real issue.

The case where ports are quickly exhausted is with long connections, typically WebSocket. And with properly tuned servers, reaching the 64k ports limit per server comes very quickly. I've seen several times the case where admins had to add multiple IP addresses to their servers just to hack around the limit, declaring each of them in the LB as if they were distinct servers. Also, even if Linux is now smart enough to try to pick a random port that's valid for your tuple, once your ports are exhausted, the connect() system call can cost quite a lot because it performs multiple tries until finding one that works. That's precisely what IP_BIND_ADDRESS_NO_PORT improves, by letting the port being chosen at the last moment.

H2 allows to work around all this more elegantly by simply multiplexing multiple client streams into a single connection. And that's very welcome with WebSocket since usually each stream has little traffic. The network also sees much less packets since you can merge many small messages into a single packet. So there are cases where it's better.

Another often overlooked point is that cancelling a download over H1 means breaking the connection. Over H2 you keep the connection opened since you simply send an RST_STREAM frame for that stream in the connection. The difference is important on the frontend when clients abort downloads multiple times per browsing session (you save the TLS setup again), but it can also make a difference on the backend, because quite often an aborted transfer on the front will also abort an H1 connection on the back, and then that's much less fun for your backend servers.

I suspect it might feel indecent to tell others you suffer when you're both free and rich, and it's difficult for them to figure what's wrong with you.

Instead, people in such position should probably go out and join associations which distribute food to those who need it. At least they'll see that they're doing something good to improve others' condition and would probably feel better.

What you're describing is for TCP. On TCP you can perform a write(64kB) and see the stack send it into 1460 segments. On UDP if you write(64kB) you'll get a single 64kB packet composed of 45 fragments. Needless to say, it suffices that any of them is lost in a buffer somewhere for the whole packet never being received and all of them having to be retransmitted by the application layer.

GSO on UDP allows the application to send a large chunk of data, indicating the MTU to be applied, and lets the kernel pass it down the stack as-is, until the lowest layer that can split it (network stack, driver or hardware). In this case they will make packets, not fragments. On the wire there will really be independent datagrams with different IP IDs. In this case, if any of them is lost, the other ones are still received and the application can focus on retransmitting only the missing one(s). In terms of route lookups, it's as efficient as fragmentation (since there's a single lookup) but it will ensure that what is sent over the wire is usable all along the chain, at a much lower cost than it would be to make the application send all of them individually.

> > There's still the problem of sending to multiple destinations: OK sendmmsg() can send multiple datagrams, but for a given socket.

> Hmm? sendmsg takes the destination address in the `struct msghdr` structure, and sendmmsg takes an array of those structures.

But that's still pointless on a connected socket. And if you're not using connected sockets, you're performing destination lookups for each and every datagram you're trying to send. It also means you're running with small buffers by default (the 212kB default buffers per socket are shared with all your destinations, no longer per destination). Thus normally you want to use connected socket when dealing with UDP in environments having performance requirements.

Nowadays the vast majority of CVEs have nothing to do with security, they're just Curriculum Vitae Enhancers, i.e. a student finding that "with my discovery, if A, B, C and D were granted, I could possibly gain some privileges", despite A/B/C/D being mutually exclusive. That's every days job for any security people to sort out that garbage. So what the kernel does is not worse at all.

There's still the problem of sending to multiple destinations: OK sendmmsg() can send multiple datagrams, but for a given socket. When you have small windows (thank you cubic), you'll just send a few datagrams this way and don't save much.

The default UDP buffers of 212kB are indeed a big problem for every client at the moment. You can optimize your server as you want, all your clients will experience losses if they pause for half a millisecond to redraw a tab or update an image, just because the UDP buffers can only store so few packets. That's among the things that must urgently change if we want UDP so start to work well on end-user devices.

Something that nobody seems to be talking about here is the congestion control algorithm, which is the problem here. Cubic doesn't like losses. At all. In the kernel, pacing is implemented to minimise losses, allowing Cubic to work acceptably for TCP, but if the network is slightly lossy, the perfs are terrible anyway. QUIC strongly recommends to implement pacing but it's less easy to implement accurately in userland when you have to cross a whole chain than at the queue level in the kernel.

Most QUIC implementations use different variations around the protocol to make it behave significantly better, such as preserving the last metrics when facing a loss so that in case it was only a reorder, they can be restored, etc. The article should have compared different server-side implementations, with different settings. We're used to see a ratio of 1:20 in some transatlantic tests.

And testing a BBR-enabled QUIC implementation shows tremendous gains compared to TCP with Cubic. Ratios of 1:10 are not uncommon with moderate latency (100ms) and losses (1-3%).

At least what QUIC is enlightening is that if TCP has worked so poorly for a very long time (remember that the reason for QUIC was that it was impossible to fix TCP everywhere), it's in large part due to congestion control algorithms, and that since they were implemented in kernel by people carefully reading an academic paper that never considers reality but only in-lab measurements, such algorithms behave pretty poorly in front of the real internet where jitter, reordering, losses, duplicates etc are normal. QUIC allowed many developers to put their fingers in the algos, adjust some thresholds and mechanisms and we're seeing stuff improve fast (it could have improved faster if OpenSSL didn't decide to play against QUIC a few years ago by cowardly refusing to implement the API everyone needed, and imposing to rely on locally-built SSL libs to use QUIC). I'm pretty sure that within 2-3 years, we'll see some of the QUIC improvements ported to TCP, just because QUIC is a great playground to experiment with these algos that for 4 decades had been the reserved territory of just a few people who denied the net as it is and worked for the net how they dreamed it.

Look at this for example, it summarizes it all: https://huitema.wordpress.com/2019/11/11/implementing-cubic-...

Not surprised. These animals are fascinating. We're not even sure we have caught everything from their language; maybe it's not just sound-based, and the way they shake their trump and ears or they dance counts a lot as well. I've long wondered if some of them developed some religions or even if they've overcome this need, reserving it to humans only.

Another study already showed that some monkeys have vocal words to designate a tiger and an eagle and use that to make all the group go up or down in the tree depending where the threat comes from. Elephants being more complex animals also living in groups are quite likely to have an even more elaborated language.

I ran some tests on phi-3 and mistral-7b and it's not very hard to teach them to use tools, even though they were not designed for this. It turns out these models obey their instructions quite well and when you explain them that if they need to look up data on the net or to perform a calculation, they must formulate this demand with a specific syntax, they do a pretty good job. You just have to enable reverse-prompting so that the evaluation stops after their demand, your tools do the job (or you simulate it manually) and their task continues.

There has never been any CONTINUATION frame issues in the first place. Only a bunch of other implementations did it a strange way resulting in abnormal memory usage, but CONTINUATION frames are handled exactly how the spec meant them to be handled, i.e. concatenated on the fly as they are read. You can even try to send many frames of 1 byte if you want, there's no issue (aside the small overhead of parsing extra frames of course).

Regardless, given that it's being used and abused these days to waste server resources (CPU and network bandwidth in this case, memory for some other servers), they're now counted as glitches and you have an option to automatically close a connection above a certain number of glitches, and you can even track them in a stick-table that will allow you to block a source IP and share that info with all other LBs. I.e. the abuse of CONTINUATION (and all other classical H2 abuses since this one was known for 8+ years) can actually be used as a marker for a bad source that leads it to being blocked early.

There are other well-known attack vectors on H2, such as constantly sending SETTINGS frames to change the initial window size, whose cost can be proportional to the number of established streams, and, of course, the fast creation of streams that are immediately reset. All of this is properly counted as glitches and will be handled likewise.

I'm aware of a few large sites which managed to lower their CPU usage with this, by getting rid of this annoying background noise.

Also it's worth keeping in mind that when someone has enough resources to use your CPU this way, the impact would be roughly 100 times higher if they used the same bandwidth to force make you generate TLS handshakes! Thus, H2 attacks, while annoying, are a just a minor detail.

I honestly don't know, all I know is that we've had demands from users at very high loads because the logs are more compact and their parsing is more efficient. And once you have JSON output encoding, it's not much work to produce another encoding. Maybe other ones will follow by the way, we'll see.

UDP through a proxy is total non-sense. There isn't any single UDP-based service that is cleanly proxyable. Not just one. Most of them rely on the source IP address itself (plus port) or announce it in the protocol, then require either to spoof the source address and reconfigure all servers to route responses through the proxy, or the proxy to mangle the contents (not even always possible). There's no notion or indication of beginning nor end of connection, packet direction, nor number of packets expected in each direction. Sometimes assigned ports on both sides are not even compatible anymore with some protocols. And it's even funnier when some users want to proxy some IPv6 traffic to an IPv4 server and they don't even understand that it's not possible to perform transparent proxying in this case. UDP has always been for light stateless stuff on direct connections, and a proxy is everything but that.

In short, only services that support being NATted by a firewall will be transportable over a proxy (sometimes with some efforts), and will always be better handled using LVS that does that natively, with less overhead, less resources and less configuration. I've yet to see a single valid case for generic UDP proxying.

The only cases that make sense are service proxies (syslog proxy, DNS proxy etc), which I'm perfectly fine with. For example our syslog proxy can listen to UDP and forward to UDP, TCP and/or stdout, message by message. But for now I'll continue to firmly oppose to the supporting generic UDP in haproxy and I will continue to tell requesters that their demand is stupid and proves a very poor understanding of networking basics.

In the end there's a nice tradeoff: internally at HAProxyTech some of my coworkers have implemented generic proxying in the enterprise edition for all those whom we don't want to say their demand is stupid, because they're willing to pay to avoid using their brain. They're happy with that (at least I hope so), and we don't have to pollute the core with code trying to plug uncovered areas for people complaining on an issue tracker that something does not work due to their uneducated architectural choices. So in the end everyone wins.

I totally agree, that's what I had to do with my patchbot that evaluates haproxy patches to be backported ( https://github.com/haproxy/haproxy/tree/master/dev/patchbot/ ). Originally it would just provide a verdict and justify it and it worked extremely poorly, often with a justification that directly contradicted the verdict. I swapped that by asking the analysis and the final verdict and now the success rate is totally amazing (particularly with mistral that remains unbeatable at this task by obeying extremely well to instructions).

In other words the ones that have long been contesting the use of CONTINUATION due to the risk of DoS 10 years ago. Just read any of the long threads there to get an idea, it's always about how to avoid the nasty CONTINUATION: https://lists.w3.org/Archives/Public/ietf-http-wg/2014JulSep...

If at least it had been accepted to forbid it after a non-full HEADERS frame it would have been more robust but it was perceived that the encoding job itself could have been harder (byte boundaries in compressors etc).

BTW I find it funny how we "rediscover" the same stuff every 10 years. Recently it was the well-known RESET_STREAM flood, now the CONTINUATION, soon it will probably be DATA frames of length zero, then single-byte WINDOW_UPDATES, then INITIAL_WINDOW SETTINGS that cost a lot of CPU, etc. The world is just circling in this security circus, provided it's possible to assign a name and possibly a logo to a known problem...

Of course. Depending on the machine and on the network, a single modern machine can do even up to a few millions RPS. This is routinely used in benchmarking tools.

Here with the "attack", it's simply exploiting the ability of HTTP/2 to compress requests and reduce them to just a few bytes, meaning that within a few kilobytes of data you can easily have hundreds of requests. Again this is not new and was already being discussed in 2012 about SPDY's use of zlib to compress requests.

The extra stuff that seems to have made this attack "new" for such service providers is that attackers took care of closing their requests so as not to have to wait for a response and be able to fill the wire with a flow of request. Again this has been known from the inception of HTTP/2 and routinely met by those dealing with proxies which timeout and send headers followed by rst_stream.

Here it makes noise because new records were broken, and likely because the stacks in place were not properly designed so they omitted to check for the real number of streams and only focused on the protocol validity...

Actually even the diagrams are wrong because they focus on a single connection to explain the problem, carefully omitting the fact that a client can easily open many connections to do the same again. I agree it's mostly marketing and press-releases.