robertgraham

This isn't true.

One of the things I'm famous for is creating "BlackICE" 15 years ago, an intrusion-detection technology that we shipped as a variety of products, such as a personal firewall, gigabit IDS, and inline protection (i.e. intrusion-prevention-system or IPS).

The distinguishing feature of this technology is that we wrote "protocol-decodes" for everything. This made the product faster, able to catch more things, yet producing fewer false-positives. It's a vastly better technique than Snort-style pattern-matching.

Yet, it was an uphill battle convincing the market of this. That's because people are stupid and don't understand how things work well enough to appreciate the difference. All they know is that they download a public exploit, run it, and if the IDS catches it, then the IDS is good.

Today, there are lots of commercial products that do things the right way, with protocol decodes. There is also the open-source "Bro" project which does things the right way. All these products can catch my heartleech tool -- it can't evade tools doing things the right way.

Even Snort often does things the right way, but only when people like me prod them. They are going to add an SSL decode (I predict).

So the upshot is this: I really are about the difference between protocol-analysis and pattern-matching in IDS technology, and as long as people like you aren't smart enough to understand the difference, I'm going to keep releasing exploits to demonstrate it. None of my exploits evade properly written IDS -- only IDS that takes shortcuts.

I see what you mean; there are two things that most interest me.

The first is "wow, did I just see them in action?".

The second is that journalists are consulting the wrong "experts" in situations like this. They think "cryptographers" are the experts in these Snowden leaks, but the real experts for most stories are incident responders, pentesters, reverse engineers, and even simple IT engineers.

As for being wrong, I'm sure if I could reveal more details, people might be able to debunk me. Sadly, I can't.

Actually, it's not just Huawei, it's pretty much all everything. Pretty much every router, telcom switch, storage system, etc. sold outside the United States comes with a support contract whereby the vendor's engineers can connect and manage the device.

Indeed, there's a recent legal case of a company selling stuff to Iran. The company said they weren't responsible, because it was resold by intermediaries. Yet, their support engineers were connecting in to manage the box.

The lede was really "here's what we saw", at least to the extent that we can reveal anything being bound by customer confidentiality agreements (which, frankly, isn't much, which kinda sucks for the reasder).

There was no "tone" to the post. It's not pro or anti NSA. Something was in the news. I saw something related to that. So I reported it.

That's my theory. Any monitoring of outgoing information would just see a typical attachment to a hotmail address.

I can't reveal the exact SQL query because that's customer private information.

However, it had both a subject and a timeframe that were peculiar. Googling the subject revealed news stories about it -- making it clear this was something the U.S. was interested in, but which would be no particular interest to anybody else.

Extremely common.

It's the norm today that companies have firewall/VPN holes allowing support engineers from other companies to have access to their networks, to manage things as simple as the HVAC system, or things as complex as their entire routing infrastructure.

Throughout the world, most Huawei routers come with such support contracts.

It was an internal system. We noticed with 'netstat' that it had a connection to an outside system. 'who' told us it was the account setup for Huawei remote support, and the IP address told us indeed that it was from a Huawei network.

The SQL query took 15 minutes to run. We saw it using 'ps'.

We then kept dumping their '.bash_history'.

No. It was a username/password assigned to Huawei tech support.

I think that's FOSS compatible.

I'm not using % on the result of rand().

I'm using the "Feistal network" construction that is at the heart of the data encryption standard, replacing binary operations like 'xor' with the "addition plus modulus" operation.

My found function sucks, and I only do 3 rounds, so there's probably some issues there. But, if I were to fix those issues, then there should be no more detectable bias than in the original DES cipher.

First of all, we avoid well-know "darknet" monitors that generate a lot of abuse reports.

Second of all, we response personally to each abuse report and offer to include them in our "exclude" file so that we won't scan them ever again. Though, of course, I prefer they add us to a "whitelist" file: not opening their firewall, but adding us a file that ignore logging.

It's 3 minutes per port. In the real world, you'll want to scan for many ports at a time. If scanning for all ports, it'd take 108 days at this rate.

My scanner doesn't try to brute-force the logins. It's just grabbing the banner.

"They" are me.

What's up with this belief that anybody you haven't heard of is a member of some shadowy organization? I'm a well known security researcher, I give several presentations a year at cybersec conferences, and my blog gets regularly link to from news.ycombinator.com.

Also, the post above excludes the first part of that: "We are happy to add your IP addresses to our blacklist so we won't ever scan you again".

My point was to talk about the wrong problems.

Networking is actually a "right" problem, and should be nearly embarrassingly parallel since two cores and process two unrelated packets at the same time. But, if you look at network stacks on open-source projects, you see a lot of fail. That's the point of my post, as a reference to point to why that spinlock in your networking code is a bad idea, and why it's probably better to replace it with an atomic operation or a lock-free alternative.

My desktop is 6 hyperthreaded cores (12 "cores" total).

It's a general principle of why mobile phone CPUs aren't putting a lot more cores in their devices, and why they didn't go the Atom route of hyperhreading: code on cellphones fail to take advantage of the additional cores.

Yea, some engineers created a ground-up rewrite of Snort called "Suricata" that was multi-threaded, and therefore faster than Snort, which is only single-threaded. Suricata then failed to show any benchmark where they exceeded Snort's speed, and they fail to mention that Snort works fine on multi-core.

It's one of those things "everyone knows multi-threaded is better than single-threaded", but everyone's wrong.

They are NOT non-existent, they happen a lot in contention. That's why code fails to scale: as you add CPUs, contentions happen a lot more often, and the number of syscalls shoot through the roof.

The entire point of the post was talk about "lock-free" algorithms where two cores can make forward progress without either having to wait or spin.

What systems have mutexes that aren't built like futexes?

When there is high-contention for a resource, it's better that one thread do it and access it contention-free, rather than make multiple threads content for it.

Even so-called "lock-free" synchronization has locks, they are just very short (30 clock cycles). Therefore, you still want to avoid contention if you can figure out a way to do it.

I didn't really go into enough detail in my example, but pulling packets off the network is a good example. You can have one thread do it, and therefore need no contention. Then you can setup multiple single-producer/single-consumer ring-buffers to forward those packets to worker threads to complete the processing of the packet. Thus, you essentially get rid of all the atomic/lock-free contention you would otherwise have.

(a) What part of "In the Linux pthread_mutex_t, when code stops and waits, it does a system call to return back to the kernel" do you not understand? That's how "futex" works: when it fails to get a lock, it must stop and wait, and therefore does a system call. When it doesn't have to wait because there is no contention, then it doesn't make a system call.

(b) The entire point is that for really scalable network applications, you don't want the high-priority thread to suspend for things like disk IO. Really, you read all that and expected the thread to use blocking IO instead of asynchronous IO???

(c) The point wasn't that Snort's multi-process design is better, only that it's acceptable. It's a single-threaded design written in the 1990s that has lots of quirks that make it hard to convert to multi-threaded operation. The point is that you can still get it to multi-core without having to make it multi-threaded.

(d) How is "multi-core doesn't mean multi-threaded"??? Snort is a multi-core app today and isn't multi-threaded.

(e) I keep repeating the claim because my code written in user mode scales better than the code of people who disagree. My code scales to 20-million connections, 20-gbps, 20-million packets/second. What does your code scale to?

(f) "Real-time" is different than the "network" apps I'm talking about. The entire idea is "control plane" vs. "data plane" separation. Control operations that need real-time guarantees are very different than high-throughput data operations.

(g) Multi-threaded techniques that try to interleave multiple threads on a single core suck for high network throughput. Just ask Apache

My blog post was quite clear that I'm talking about futexes, and then when it fails to get a lock that it goes into the kernel. Seriously, that's how everyone did it from Solaris to Windows before Linux "invented" the concept AND it's been in Linux for a decade. When somebody says "mutex", you have to assume they already mean "futex".