Ask HN: $8k on AWS in August for you, any interesting computations? | Hacker News Reader

Ask HN: $8k on AWS in August for you, any interesting computations?

124 pointsadhambadr87 comments

for reasons too long to write here I have around $8k of credit on AWS from my last failed venture that will expire end of this august. Id hate to let those credits go in thin air and I don't have the time to build the stuff I imagined myself building to use them, so I thought Id give it to the HN community if they come up with a really cool 'demanding' computational task to use this kind of server power.

Comment on how you'd like to use it and if you impress us they're yours.

p.s id be super happy if somehow the computational power is used to build/crawl some kind of data that would directly/indirectly help anyone for free. Second to that would be if hackers in the comments come up with an idea and do it together.

View on HN

Give it to Watsi, a non-profit that is making extensive use of AWS. It would probably equate to donating the entire sum to them.

Bayes Impact is another YC nonprofit that may be interested!

Watsi sounds like an awesome organisation, but would they need such computing power within a month ? (the credit is only usable in August), if then can you connect me with them ?

I am doing my thesis (in the general area of video classification) as an electrical engineer in a Greek University and I am currently running some experiments that try to learn a video representation from local features (kernel codebook encoding) using an SVM variant and stochastic gradient descent.

I could use a couple of c4.8xlarge instances to run the experiments trying out learning parameters. I am currently "competing" for time in the lab's cluster so I end up running the experiments in whatever computer I find available.

Anyway, thanks for the opportunity, it is really generous of you.

You know it pains me to see you ask for resources. Everyday at work, my team uses a couple r3.4x large and c4.8x large for trivial tasks. My team does not realize how valuable a resource they are accessing(and wasting) everyday. I wish I can donate those to you, but company policy forbids me to.

Thanks for the thought!

I hope I didn't sound ungrateful... There are resources in the university but they come with overhead and not as much freedom as a VM on which you can do anything.

That said our cluster is about as big as two c4.8xlarge vms.

FWIW, you can have 2 cc2.8xlarge spot instances for about $0.50/hour. $12/day isn't nothing, but it's pretty affordable. Stick your data on an EBS instance attached to the spot so you keep your data if the spot instance gets nuked and you have a functioning cluster for around 13% of the on-demand price.

Your thesis sounds interesting. I'm working with a company doing some research in a potentially similar area. They might be able to spare some time on their cluster - drop me an email if interested and I'll see if I can set something up.

I'm pretty sure what you're trying to do is not allowed under Section 1.3 of the AWS Promo Credit agreement. Personally, I would be wary accepting "free" credit that I may end up being on the hook for.

https://aws.amazon.com/awscredits/

IANAL but I don't see that a violation. As long as he isn't selling or renting it out for a price then he is fine. I have a feeling that provision is there to make sure people don't try to sell credits, screw over the buyer, then the buyer tries to complain to Amazon.

Steam has a similar stipulation - and a lot of people are pissed about it. Basically you can't buy or sell steam accounts - and if you are caught Valve will suspend the account. I personally think it's completely stupid and if actually challenged would not hold up in court (I think the first sale doctrine fits perfectly because if you sell your steam account then the games are no longer accessible to you). But again - I get why, they don't want to put up with scammers or stupid people who try to sell an account and disappear before they send the credentials.

> You may not sell, license, rent, or otherwise transfer Promotional Credit. Promotional Credit may be applied only to your AWS account, and may not be applied to any other account.

Seems pretty clear to me...

> You may not sell, license, rent

He's giving it away, no money changing hands.

> or otherwise transfer Promotional credit

Is what he has "promotional" credit? He says he has credit... is there a difference? If there is, then that quote doesn't apply. If they are the same thing, and it's used on his account?

If there is no "sale" or "transfer", then yes... it is clear: He's fine to use it as he wishes.

Yes, if he bought $8,000 in AWS credits then he can use it as he wants, but I doubt this kind soul is trying to just throw out $8k.

This is very very likely promotional credit. They hand it out like candy.

Edit: Okay, I can see that if his approach is to use it on his own account, but for someone else's purposes, then it's probably fine. I don't think they even have any system to transfer AWS credits

Edit2: Then again I would be wary of publicly announcing that I'm actively trying to use my AWS credits for other people even though I'm not technically "transferring" them...

Be careful the AWS police will get you.

-yup its a promotional credit (started at 10k), i didnt really buy it -nope I can't transfer it -yes my approach was simply to give the person/people access to my account -Its my fucking server and I can run whatever script I see fit .. but ok realistically maybe its not the best idea to give it to a commercial startup/project that has a legal entity to be sued. but I don't think its really that big of a deal to run some crawlers

If he does the computations and simply gives the results as a donation is that transferring?

I would argue no - what would be the difference between him spinning up VMs and handing off the credentials and hosting a public web service on it? The VMs are still his responsibility - he is just allowing other people to use them.

https://www.youtube.com/watch?v=xLnTWxpTQt4

If you are Bay Area local and would like to support soaring pilots community (Glider, Hang Glider, Paraglider), I'm maintaining local soaring forecast and run relatively computationally expensive NOAA WRF model. It is free, no ads, I've been maintaining it for ~7-8 years and for last couple of years it runs at about $700/year using spot/auction Amazon pricing. Available at http://norcalsoaring.org (my e-mail is HN user id @gmail)

edit: after reading a link describing Watsi (current top comment), I would support donating credits to Watsi as well.

Amazon was generous enough to give you the credit trying to help your company, and in general is extremely helpful to the startup community as a whole with AWS credits. Maybe you should just give it back to them so they continue to be so kind in the future.

(On the other hand, the main reason AWS gives out these credits is to get startups hooked on AWS, then charge borderline outrageous prices once the credit is done for, profiting billions) :)

AWS is constantly reducing prices, as their costs go down. I wouldn't accuse them of profiteering, at least any more than any public company (with the requirement of a fiduciary responsibility to shareholders).

They're not a non-profit. They don't always have the best price/performance ratio, but they do offer scaling options that few competitors can match.

Disclaimer: I am a former Amazon employee, though I sold my stock. Regretting it now, but what can you do.

(former AWS employee) You can't give the credits back or transfer them to another account (at least not in a short period of time). Best to spend them.

There's just no infrastructure to support that scenario.

Not spending the credits is like giving them back to Amazon

If you don't use the credits, they expire in two years. How is that not giving the credits back to Amazon?

If the credits ended up being donated to a non-profit org, and Amazon can't overlook it, then Amazon is just being a prick about it. Hopefully, I am jumping the gun a bit here..

I donate to http://www.worldcommunitygrid.org/ when possible. Right now I'm donating my monthly $100 MSDN Azure credit. I would do more through AWS but I don't have the budget to do so at the moment.

You could spin up a large number of ArchiveTeam warrior images

Are they a 501(c)(3)? If the OP's marginal combined tax rate is, say 50%, then donating $8k in credits would be like $4k in their pocket.

EDIT: Of course OP should talk to an accountant first. . . .

The Internet Archive is indeed a 501(c)3, and this is a great idea. It also fits very well with the request in the PS.

The Internet Archive is a different entity than ArchiveTeam. I'm skeptical that running AT instances (which scrape websites into archives which then get hosted by IA) counts as a donation towards the IA.

From archiveteam.org:

    Archive Team is in no way affiliated with the fine folks at ARCHIVE.ORG

Granted, but you could contact archive.org and ask how $8k of AWK credit could be donated and used productively. I'm certain they'd have a valuable use for it.

id rather spent my time writing infinitely nested while loops then talk to my accountant about one more topic

If I get this correctly, this would help make a backup of archive.org. It would download a big amount of data and then store it for... one month, until the instances expire? That seems rather wasteful of bandwidth and effort, or am I missing the point?

That's a separate AT project. The Warrior downloads data from various websites (usually ones that look like they're about to go down) and uploads the data to a central AT server. It's distributed to get around rate-limiting and banning. Data is not stored long-term on the Warrior.

Or buy some 3-year reserved instances and run the iabak scripts http://archiveteam.org/index.php?title=INTERNETARCHIVE.BAK/g...

Actually I don't remember how paying for storage works with reserved instances... if you're going to get stuck with a recurring bill that you can't pay with the credits, then never mind.

If someone wants you to do webscraping, even for a good cause, I wouldn't waste my time if I were you. Most AWS IPs have been long-since blacklisted by the most interesting sites (e.g. craigslist).

TOR ?

I work on deep learning methods to analyze electronic medical records. Institution I work for (high ranking university in the US) has some outdated GPUs I can use to crunch some numbers, but I cannot use really use them to experiment to learn new things or use them for my own projects. $50-100 credit would help me a lot to rent a GPU instance for many hours and learn a few new tricks. I could start with applying Convolutional Neural Nets to play with this: https://www.kaggle.com/c/grasp-and-lift-eeg-detection .

I'm working on a tool[0] using LSA [1] to mine the emails from Hacking Team so that people could use it to get more relevant results compared to what wikileaks has available (a lot of stuff to sift through to not know how each message might be related to ones queries outside of just mentioning the word in an email).

Right now I have to break up the term-message matrices by person to do before doing partial eigenvalue decomposition, to generate inverse(sigma) * transpose(u) and inverse(sigma) * transpose(v) and the lower dimensional space representation of each message for each message, but it would be cool to not have to do that if I had more computing power available (a friend let me use his 12 thread/ 6 core machine which has helped a lot while building things).

Ideally it should it hosted somewhere eventually, because the project it self might be a bit complicated/tedious for most people to set up themselves, that would allow people to search it using these indexes as well as enabling (independent) journalists to be able sift through everything in an arguably better way.

[0] https://github.com/cinquemb/hackedteam-email-index-mining

[1] https://en.wikipedia.org/wiki/Latent_semantic_analysis#Deriv...

That's very generous.

Not sure if this matches your criteria, but I've been wanting to run some extensive benchmark tests on the most multi-core EC2 instance that I can get for this Elixir project: https://github.com/a115/exmatrix

I probably don't need more than a couple of days of uptime to complete this, and the results will of course be made public, with proper credit for your support. jdimov at a115 co uk

You are so aweosome for doing this. I built GoReturnMe.com single handedly with my own php framework Tatanka. It needs an update but we could use some credit :) I love building bands/musicians/artists websites. And I host them at no charge on AWS, credit would help there :) I also an an engineer for label insight.com :) we could use some credit too.

Bands/Artists: http://www.deathbyicon.com http://www.deannadevore.com http://www.theivorysband.com

GoReturnMe: http://www.goreturnme.com LabelInsight: http://www.labelinsight.com

You're awesome!

For my Master's Project (UC Berkeley), I'm working on sports analytics and furthering in-game sports prediction. We are building off some initial work out of MIT that you can find out more about here: http://www.sloansportsconference.com/?p=6137

To start, the goal is to perform real-time MLB pitch prediction. To facilitate this we will be building a crawler to curate MLB data from http://gd2.mlb.com/components/game/mlb/ as well as testing some Machine Learning algorithms. We'd like to use the credits to facilitate this and potentially put together some crawlers for other sports data (NBA, NHL, etc.) before the end of the month!

How about building an Einstein@Home lab with some of those credits? Help discover some neutron stars.

Or Folding at Home for protein folding

That would be a waste since this would contribute very little to F@H. I'm the lead dev at F@H.

I thought F@H was processing dependent?

Wow, 20,000 GPU cores for 20+ days wouldn't help much?

g2.8xlarge is 4 GPUs and costs 0.6945/hr on spot instance. You could contribute 120 GPUs for 2 weeks, which would be pretty insignificant: http://fah-web.stanford.edu/cgi-bin/main.py?qtype=osstats2

Can you explain in more detail why the contribution wouldn't be proportional to the value of the credits?

I'm doing text (both word and character) models based on Karpathy's char-rnn using Amazon's gpu instances. It's not very expensive, but I would like to do hyperparameter optimization which takes a lot more computing power. Specifically I'm trying to see if it can be useful for auto-completion or sentence rephrasing.

We run the largest independent app store in Canada and are fiercely committed to being indie. We store all of our images, icons and Android OBB files on Amazon S3 and sometimes our bills get totally unmanageable.

Having a least a portion of this credit would give us room to maneuver so we can polish up our Android and Blackberry 10 apps and with the money we save it would allow us to scale, rather than just get by.

you can check out our app store http://apps.goodereader.com

I can send you some WPA handshakes for you to decrypt them. ;P

Immediate idea is that scraping current complete product listing from amazon, ebay, alibaba and saving into database for public use. I am not sure one month is enough or those sites have anti-robots policy.

Another is crawling and indexing the internet to make another privacy-respected or specialized search engine(how to continue the effect one month later is an issue).

If you're a FOI supporter, I'm one of the founders of http://lbry.io a startup launching this month that could absolutely put those to good use. We're creating a fully decentralized market for information, with an emphasis on consumer-friendly names.

op here: to clear up I can't really 'donate' or transfer 8k or any amount to your account. I will just give you ssh access to the machine(s) you need. The 8k will also be gone if we don't use them before end of august, so whatever it is you wanna build it needs to be written and executed before that date

Do you plan to follow up when you've made a decision?

op: i planned to, but up till now everyone is either telling me to donate the transfer to an ongoing project (ignoring the fact they only get to use it in August) or stuff people want 'to start' building or want me to start doing. Still no solid lucrative suggestions, do you have a favourite suggestion here ?

yeah, my favorite's the "fuzz testing python" one. Mine. ;)

It's ready to rock and roll right this minute.

security FTW.

Can you spend the credits on reserved instances? If so, they shouldn't have to be used by end of august.

In general, no:

https://aws.amazon.com/awscredits/ (Section 1.1)

I'd like to use Riot's official API (https://developer.riotgames.com/) to scrape a BUNCH of game data, specifically "timeline" data, and see which players do things the best.

Their rate limiting is laughable compared to the amount of data they produce (even on prod apps) so good luck :/

Step 1: I would use a chunk of it for Alexa web services and ranking very large lists of small business and non-profit org URL's from open data sets, and use it to identify sites whose traffic far exceeds their reasonably predicted popularity. Step 2: ____ Step 3: Profit

I'd use it to make progress on some unsolved conjecture in graph theory (for science!). I don't have a specific one in mind right this second, but there are certainly a few where massive computer searches can help.

I am working on a project to mathematically model the elimination of tuberculosis in California. We will need extensive use of AWS... would really appreciate any donation of time.

-Alex alexgoodell@gmail.com

I am working on a real-time collaborative notebook for code and mathematics. We are trying to upgrade to a small ec2 from a micro tier. Some AWS credit would be helpful for us to try this out!

www.escherpad.com

I should like to learn to use AWS without risking being overcharged (a guy went from 200$ to 50000$ next month). You could make a web like try AWS anonymously without risk.

But that is a risky endeavor.

I've been fuzz testing python modules on some whimpy local computers, I'd love to give AWS a try!

I hope to find security problems w/CPython which would likely help the world at large.

Is it just me or has this post been made before here on HN?

I would probably try to use it for some publicity. Prove if NSA can do some stuff, like breaking SSL certificate.

Or you could recover perhaps 30% of your money by mining coins using spot instances.

Other option is to donate CPU time to Folding@Home (cancer research) or other similar project.

And finally my open-source project could use some stress testing :-). I have concurrent database engine. One of the way to catch concurrency issues, is to randomly insert delays around code, and hope it will trigger some rare race condition. I think single run will take a few years of CPU time :-)

Contact any computational chemist. They will use that overnight. Make sure to get documentation for a tax write off.

Amazon and other vendors must not drop fixed pricing and availability for AWS. Unpredictable pricing or resource availability will be a major barrier to cloud computing adoption by enterprises. http://www.besanttechnologies.com/training-courses/web-desig...

how about donating it to some sort of intern/mentorship program? i have a friend attempting to start one with gov't support and is in the trial'ing phase of it now at his company. he may find it useful. I could put you two in contact

If you are into baseball and advanced sabermetric stats, there's a bunch of c3.8XL and g2.8XL instances out there cranking through data to train various neural networks. The results can't be published to the public but I'd be happy to add you to the distribution list of results and future projections.

Hmm? First I've heard of it, and I work in professional baseball and have been active in the SABR community. Can you add me? mike.anon@hotmail.com

Unfortunately it's been caught up in the classic proprietary closed-source vs public source political debates... Pretty much zero chance anything we do gets added to the public domain.

How could I get on that list? I would love to be involved.

Edit: email -> dustin@xdlbx.io

Ethereum GPU mining?

I would use it to find 1,000,000,000 digits of Tau.

Donate them to a cracking team at DefCon

I just joined AWS to move some domains over from another host, specifically one site I'm trying to turn into a startup. I set up RDS thinking that "pay what you use" would mean I would have no bill since the site has no traffic yet. Turns out spinning up an idle "production" RDS instance costs $800/mo which I just found out when receiving my first month's bill. I emailed support but no answer yet. I still plan to use AWS and offsetting this bullshit with credits would be nice.

Reply on news.ycombinator.com