Hacker News Reader

Comment by lettergram | Hacker News Reader

I setup an automated script to setup a AWS g2 instance, train my neural net using tensorflow, copy my model to my personal computer, and spin down. It costs like $5-$10 to train/test most neural network models. My most expensive model cost like $100 and required a ton of time and resources. It took like 4 days or something.

You really dont need $2k workstation..

Of course, for personal use I do have a gtx 1080 because I like to game and play tensorflow/caffe

View on HN

Spot instances cost way cheaper. The only downside is you need to create an AMI everytime before termination. But, also, AWS g2 has NVIDIA Grid K50 with 4GB memory, so it's not very good with performance.

The AMI creation is only needed if you store data on the machines, which you shouldn't do anyway, not even with on-demand ones.

You should always try to keep the instances stateless and store any data outside the instances, such as on S3 or EFS.

Would you mind sharing this script? I am thinking of investing in the 2K workstation, but this sounds like a much more efficient way to go.

I tried to get it released before, but was shut down by the "open source office". So I can't give you the exact script. However, h2o has a script that launches a cluster, that's very similar:

https://github.com/ledell/h2oEnsemble-benchmarks/blob/master...

You can also do stuff like following in python to execute some code:

    os.system("scp script_demo.py " + sys.argv[1] + ":/home/ubuntu/")
    os.system("ssh " + sys.argv[1] + " sudo chmod 777 /home/ubuntu/script_demo.py")
    os.system("ssh " + sys.argv[1] + " './script_demo.py'")
    os.system("scp " + sys.argv[1] + ":/home/ubuntu/output/* ./output/")

Then it will execute, after it executes - shut it down.

Even if it may not matter in this case specifically, this is a terrible example. "sudo chmod 777" in public code is basically "I don't know what I'm doing, but go on, do the same thing yourself" :(

You don't need sudo, because it's your file. You need just "chmod 755", not "777". And you don't need chmod in the first place - just run "python script_demo.py".

Yes. On the other hand, there isn't a strong overlap with machine learning experts and security experts(I'm sure there will be in coming years though, as security experts start using NNs to detect anomalous/dangerous behavior).

I know what you mean, but I disagree with "security experts". You don't need to be a security expert, just know the basics of how privileges work on your system. And if people don't, we just need to keep calling it out, because otherwise code like this ends up in production one day, where it actually matters.

Simpler bash equivalent for the record:

    scp script_demo.py "$1"
    ssh "$1" python script_demo.py
    scp "$1":output/* ./output/

Exactly. If you're a ML person, you must be working with Linux in most cases. And in such case, you need to be familiar with the types of permissions, cause that's relevant even during the installation of various libraries, doing ssh etc.

These things come about because people learn the simple way to solve all permission problems is just do everything as su (or root) and chmod everything to 777. The problem is it does solve all problems (well sweep them under the rug) - I guess when the only tool you have is a hammer everything looks like a nail.

If you're launching an AWS instance that doesn't have any ports open and is going to terminate in a few hours, who cares?

As I said: "Even if it may not matter in this case". But what about the next time? (poster obviously doesn't understand the code) What about people who read this comment and follow the advice because it works? (happens all the time with SO) What about when your script becomes the standard deployment method for the company? (happens everywhere) What about people who don't know about file privileges either and blindly copy what they see here?

There's lots of things we do that you could say "who cares?" about. Single letter variables. Comments. Const-correctness. Unnecessary N^2 algorithms. Usually it turns out that either you or someone you work with cares a few months/years afterwards. So just learn to do it correctly the first time. Especially if the correct way takes less time than the "magic fix".

Very true it doesn't need a 777. It's also not my script, just copied and pasted some code that fit what I was doing in mine. Basically, it follows the same format, specific implementation varys.

Also, you are being rather pedantic here. If people can figure out how to take what I copied & pasted and turn it into a script they can probably know chmod 777 isn't great.

That being said, and why I think this is ridiculous, is that you are assuming this matters. Going into the weeds here, to play along: The script is immediately being ran, on temporary and very recently launched ec2 instance, probably with the use of a pem and that presumably can even be part of AWS security group that only allows your IP, and is shut down following it's execution.

I can't picture this being a security vulnerability at all. Calling it a terrible example is relative - I wrote this copy and pasted on a cell phone trying to help someone. Honestly, didn't even see the "sudo chmod 777", just pasted away.

>but was shut down by the "open source office"

Can you expand on that? What happened?

Basically, I couldn't get authorization to publish internal code. In this case, it was a combination of who was going to maintain it and ensuring nothing internal was leaked.

What kind of models do y'all make?

Reply on news.ycombinator.com