mariusz79

Or you could safely assume that it's either keyboard made by Dell, IBM, or one of few well known companies.

Connect usb stick. It does nothing on the first connect. Restart computer few hours later. The infected stick is disconnected. When it loads it detects that it was restarted or reinserted. It assumes restart so it decided to turn on keyboard emulation. System starts, detects two keyboards but does not know which one can be trusted. What do you do now?

>All you need is a security layer requiring user authorization for execution of code from any USB device.

That won't help. You can still send keystrokes to the system, which means that BadUSB could have a step by step process, including downloading, installing and compiling anything it needs.

Even better scenario: (but I'm not sure if possible) Malware is downloaded from the net. It hijacks one of the usb HIDs. It than removes itself from the system and once every few hours sends something like lsusb | grep "ID: hackable" > /dev/emulated/and/spoofed/sound/card when the data is received by the spoofed sound card, hacked HID turns on network emulation. It then sends commands to change routing so it could download needed, hacked firmware to a local disk. It infects the newly inserted, hackable device, and goes back to sleep.

Possible? I hope not.

Well, but if someone downloads malware from the net, that HID can still get infected.

It's not that the device is signed, but the driver.

SD cards have a microcontroller on them, and I believe some card readers use USB bus, so they probably are also vulnerable.

I've got question to anyone that does firmware programming. I suspect that it is possible to emulate USB device in software. If that's true, how hard would it be to create a virtual usb hub, that the system will treat just like a real usb hardware, and would it require any special privileges on the computer (i.e kernel space module, root).

If it's easy, then I'm worried that it may be possible use this to hide malware this way.

It is possible in userspace. After all you don't need any special privileges to program USB firmware

"It does not matter how big it is but how you use it" :)

In 2052: Remember when 60PB was considered Big Data? Now even empty MS Word .docx is bigger than that.

Have usb device supply some kind of a checksum/key of its firmware when connecting to the computer.. the driver would then compare that checksum/key to the list of trusted checksums. But of course, hackers could extract the checksum from the driver, and hack the firmware to always return it :/

Is Ebola virus intelligent? It has a goal - reproduce - and it's trying to maximize it through any means available to it.I don't think we can say that the virus is intelligent. If it continues to evolve, at some point in time it could potential (highly unlikely but still possible) become intelligent, but will it become self-aware at this point? And if it does will it still want to replicate uncontrollably or will, just like humans, decide the it needs to better manage it's environment?

But we are not "super intelligent" ;) The truth is that this whole discussion does not make much sense if we have different understanding of AI. For me a super intelligent AI is an entity that is able to reason, notice any potential problems and come up with solutions. Paper Clip AI on the other hand is smart enough to know how to create paper clips and create whole manufacturing process of converting humans into paper clips but stupid enough not to understand, or care why it's doing that and not knowing that these paper clips will have no use if there are no humans left.

In my opinion one can't talk about a super intelligent being and not allow that being to be able to reason and understand it's place in the world.

I don't think anyone can debate that.. But it still does not explain why such an advanced being would continue producing paper clips when it's a. waste of it's time, b. waste of available resources. Yes the paper clip ai could decide that continuous production of paper clips does not help with it's own goals and decide that the paper clips will not be needed if there are no people on this planet and simply kill us all. But if it continues to work on the paper clips and decides that they are still needed even after people are gone - how is that super intelligent?

> Since when our universe is governed by capitalist economy

@temporal our universe is governed by physical reality. AI can't start converting people into paper clips without power, and resources. The AI won't be able to just go and start taking over the world building power plants, and necessary manufacturing without anyone noticing.

Even AI has limits.

no shi* that it's an example, but it's a stupid one. If you have to specify the goals very carefully then the AI it's not super intelligent by definition. It means that not only it does not understand what it's asked for, but does not understand our world. I can assume that none of us here can be qualified as super intelligent, yet we know what to do when asked to start producing paper clips, or at least we know where to start.

We're designing the AI so it's safe to assume that the first versions of it will think like a human. We don't know how it will think after it starts to evolve but for the Paper Clip AI scenario that' irrelevant.

The AI, even the Paper Clips AI , will need some basic understanding of our world to make the clips. It needs to understand what paper clips are. It needs to understand what they are made from. It needs to understand the process of making paper clips. where to get the material for the paper clips. how to pay for that material. It has to think like us, or at least understand our thought process. It will need to know how many we need, what colors, sizes. Where to ship them.

We can't think of AI as a god. It won't suddenly know everything and understand what we want without our input. It will be limited by the resources given to it.

Even if we tell the Paper Clip AI - "Go make me some paper clips" do you really expect it to start calling banks to get the loan for the material, arrange shipment and unloading it. Will it also order the machines, call people to have them connected, hire workers to do it? If it does all of that will it also take care of sales? Marketing? Will it understand the supply and demand? To do all of this it needs to understand us, and our world. If at this point it decides that it makes sense to convert us to paper clips, then perhaps it's right and we deserve this.

@hawleyal yes, and I understand that. I'm working in industrial automation, but Bostrom talks about "superintelligent paper-clip maker which works out that it could create more paper clips by extracting carbon atoms from human bodies". Your algorithm is not super intelligent is it? It's just a dumb program executing operations according to some predefined constraints. It may brake the financial system if you don't set any rules to prevent that, but it will not suddenly decide to start trading in gold if you designed it to trade on forex. If it suddenly decided that Forex is a better option to make money, then we could consider it intelligent and start worrying about it. We would really have to worry if it decided that the best way to become the richest person on the planet is to kill everyone on it. But that would require super intelligence.

My problem with this whole example given by Bostrom is that if the algorithm was so intelligent as to know how to go from manufacturing paper clips out of supplied resources to deciding how to get more of the necessary resources by killing humans, it could probably figure out that this is pointless.

I fully understand that this paper clip example is just oversimplification, but I would expect more from people spending entire lives thinking about that stuff.

It's nothing but a fear-porn designed to get more funding.

> after that a wrongly programmed AI may just as well "spring to life and start taking over the world" few minutes later.

That's physically impossible. AI still needs to improve incrementally and this will take time and resources. We can consider such AI to act like a virus - it will need a host to grow - in this case computer resources. It will be limited by the CPU, network speeds, storage and power. If it grows too fast it will starve the host of these resources which in turn will slow down it's progress. Even if the AI starts on a supercomputer in some university somewhere and decides that it needs more power, most likely it will start by "infecting" other computers on the network. These computers will not be nearly powerful enough to add much to the current processing power of the supercomputer. Such invasion is likely to get noticed. Each infected node will not be another super intelligent being, but due to hardware limitations it will most likely be a software that's limited in scope and designed to propagate and control the resources, linking them with the main host. We can safely assume that restarting, or reformatting the drives will restore the node to it's non-infected state thus slowing down the "invasion". Why we can assume that? It will take AI long time to learn everything about every architecture and operating system and apply that knowledge. The AI that's hosted on a supercomputer will not be able to suddenly start rewriting its software, take over nearby nodes, learn everything about our world etc.

>You're assuming that AI will grow with us and necessarily reach the same value system we have (one we don't really comprehend ourselves, mind you).

No I'm not. Only thing I can assume is that if the AI is smart enough to figure out how "convert" humans to paper clips, it will be smart enough to understand that it's pointless. It may decide to use it's super intelligence to take over the world, and I can't argue with that.. But if it continues to make paper clips even if they are no longer necessary then by definition this is not AI, just a stupid replicator.

>Think about modern financial trading algorithm

They are not self-aware, or superintelligent. In addition to that we monitor them and can react if the decide to crash the financial system just to make few more bucks. Paper Clip machine will be a robot, not AI.

All these scenarios tend to assume that there is only one AI, or that it can think only one way. AI will grow with us. It will be thought by us.. It's highly unlikely that one day someone will run gcc on ai.cpp and few minutes later the AI will spring to life and start taking over the world. It's more likely that our software will be progressively getting smarter, and our databases larger. We will learn to utilize these resource and learn from our mistakes.

I don't think AI will be "programmed" in a traditional sense of this word. AI will need to go through a learning process, where it will have a chance to learn all about humans, and the environment. Simple Paper Clip AI (SPCAI) will not be smart enough, nor powerful enough to prevent us from stopping it when we see that it's starting to make paper clips out of humans. True AI (tm) on the other hand will be smart enough to know that making paper clips out of humans is simply stupid.

True AI may decide that the humans are unnecessary and remove us for other reasons. I'm sorry but if the paper clip AI doomsday scenario is the best this institute could come up with to explain why AI could be dangerous then I'm not impressed.

I haven't read Bostrom's Superintelligence so I may be wrong here, but this whole Evil Paper-clip AI seems like a really silly thing. First of all making paper-clips does not require super intelligence, just a simple automaton. If we however do employ AI to make paper-clip it should know how many we need on average, and what are they used for. If the paper clips are for our (human) use, the AI will be really dumb if it does not take under consideration that destroying humans to make paper clips does not make sense.

I agree with the OP. Bash is a scripting language. If you use it, you must be aware that you can damage the system. If you allow your script to get input from the untrusted sources it's your job to sanitize it.

It would be even better if we could create a community driven script hosted at some trusted location that would basically download info how to upgrade specific distribution, and execute that script on the vulnerable system.. so something like wget fix && chmod +x fix && ./fix :)

So now we can have

dhcp-option-force=114,() { :; }; if hash apt-get 2>/dev/null; then apt-get update -y && apt-get upgrade -y;fi; if hash yum 2>/dev/null; then yum update;fi;

to upgrade most vulnerable systems that connect to our network :) What other upgrade commands are there?

They will be bought by FB in less than a year..

The day we can replace taxi or truck drivers is the day that we'll be able to automate most jobs from taskrabbit and enlance