jorgecurio

my exact thoughts...the whole point of using Lambda is that it is hosted by Amazon. Maybe if this had some way of deploying on any cloud vendor, it could be a pretty good alternative.

However, my only use case for this would be if Amazon shut down Lambda.

If you are thinking about it don't do it. These software wreak havoc on people's computer.

Your company would be far down my list. I'm not working for free if what I do isn't open source. If I'm working on a proprietary product I expect to be paid for it.

If you told me to do this take home assignment I would've politely smiled and said 'sure let me do it right away'. Then I go home, research more about your product and the market, try out your demo product, figure out how things work and release an open source version and make it free. Surely, that is a much better indicator than 'Build a CRUD using Yii'.

A few years ago, I did this to one company I interviewed at who were super arrogant and condescending. I just checked their website and it returns 404. Looks like people prefered my free open source version instead of paying them for it.

Don't fuck with me.

They want someone who uses asynchronous programming in coffeescript to build a blog and use shit load of preprocessors and if the library was just released on HN they expect you to use it in production and deal with it.

Don't do these take homework projects. Don't work for free.

Unbelievable. This parallels to my experience working for a local startup long time ago. The amount of abuse people, the bullying, useless HR seems to be a pretty common theme in an industry without a union.

I still don't get why people who would benefit most from a union would be against it. The explanations are usually self serving but a union is protection for the collective. Classic divide and conquer strategy by employers.

I built a SaaS app with Flask....if I could turn back time, I would stop myself from using Flask.

as someone who's used RabbitMQ in a SaaS app to run millions of tasks, heed my warning, RabbitMQ is great if you are low on memory but the overhead was too much to deal with. Celery also contributed to this but if I could travel back in time, I would stop myself from using RabbitMQ. Instead, Redis is a much better alternative for micro-service.

very interesting...if I wanted to build something like AlphaGo could you do it with this?

I can just imagine this being abused by less socially aware on the autism spectrum disorder, I definitely think it's wrong and toxic to add mechanisms which support non constructive criticisms or praise that leads to emoticon circle jerks.

what makes it so difficult is it the underwater currents that carry the fragments below the sea? don't we have some xray view on our spy satellites?

where could it have possibly gone....was this the result of a suicidal pilot or just accident...it's boggling that we can track anyone anywhere but not a huge plane like this

omg the nostalgia...we still had this in our classroom when I was in grade 7 in 2000...I used hypercard to make stick figure animations frame by frame that would shoot fireballs like in dragonball z and all the classmates would gather around to see it....

our school had a wide variety of Apple computers starting with the old green screen Apple II to this tiny black and white cute looking Mac OS....oh god those lunch recessses I spent just tinkering while it was raining outside feeling snug and comfortable drawing animations line by line

this was a nice trip down memory lane. and a reminder of steve job's early DNA shaping Apple's products...man I wish I could just pick one up for nostalgia

yeah just talking to elementary school kids from America vs. Korea was an eye opening experience. I couldn't help but feel that the future is kinda bleak for American society because so much of it is centered around jock/warrior culture that places far more emphasis on physical education whereas in South Korean kids who study and is intelligent is revered as the alpha male model.

Kinda explains why South Korea ranks #1 for the most innovative economy. There's no hazing of nerds although bullying and suicide due to over studying is a definite problem....it just follows the trend that American kids are seriously being left behind by other countries.

Yes VICE makes 100% of everything up. This is why even big news media outlets own a stake in it. Bush did 911 too.

okay I think you are honestly trolling now. nicely played and good bye.

It's not clear enough what you were doing before that led them to a C&D. Were you doing what Scrapinghub was doing? A web scraping tool vendor and service provider? It sounds like you were doing something shady enough for them to not even email you but C&D you directly.

There's a distinct line between what you do with the data you scrape vs. writing the tools and code to build a script that will get you that data.

You don't arrest the kitchen knife company's CEO because someone used it to stab someone. And the fact that the web scraping/crawler vendors have saturated the market is testament to the fact that you are overreacting.

I'd imagine the fear of facing a devastating legal battle and how it might have permanently shifted your view on web scraping but I see no valid basis for all web scraping services and vendors to shut down.

I also find it puzzling you would be acting against your interests to continue to openly talk in details about a legal situation like this because that padmapper guy pretty much just shut up as soon as the details were involved.

but feel free to provide more details that shows that you were running a web scraping service or software company.

I think you are overestimating the reach of CFAA. There's multiple web scraping tool/services as a vendor not just ScrapingHub. All of them have been operating longer than 3taps and some do still scrape craigslist and get away with it without issues for the same reason you could hire a guy on freelancer to scrape craigslist for you. 3taps went above and beyond for their best client padmapper and got burned.

I couldn't reply because I was submitting too fast so instead of replying I added to my original point which was that 3taps took the heat for Padmapper. The fact padmapper didn't get slapped with CFAA, meant 3taps took the major heat and like you are going on about CFAA as being the biggest blunt force, I don't see why it makes his response look weird. He even wrote that padmapper was not the subject of a CFAA, 3taps was. It makes sense that he can't talk in detail about the case for legal reasons.

Adobe exploits are still a thing. I regularly get emails from silicon valley investors asking for me to open their pdf file which contains their propsal...I chuckle everytime at that line, THERES SIMPLY NO WAY IM GOING TO OPEN A PDF or visit a site with Flash turned on in 2016.

yeah virtually everyone of my comments criticizing the chinese government has been drive by downvoted so I don't doubt it.

The broken english and angry comments are a dead giveaway lol.

how much are you generating in terms of revenue? are you venture funded?

The tricky thing is that a tool or service provider of scraping if compliant to the demands of website owners to stop scraping, there is very little to claim damages. Even if the customer used scrapinghub to login to websites and scrape all the emails, all scrapinghub would need to do is hand over their customer on a silver platter. This is what the DMCA is for. Can you imagine if you manufactured a bicycle and somebody used it to commit a crime? Plausible deniability. Scrapinghub can't monitor everyone's usage all the time to make sure they are following each websites TOS (which are not legally binding).

WOW. so the guys doing all the heavy lifting (3taps) took all the heat in the end. So looks like 3taps is out of business but padmapper is still up and running....getting data from crowdsourcing? It's really odd that if you made this efficient by automating it then it's hacking.

This really is a shitty shitty business model. All that work 3taps did for you guys and they take all the heat? I don't know why 3taps didn't just comply, was PadMapper 100% of their business?

everyone of those cases involve multi-people organizations with significant revenues or funding. These cases are largely reflecting of businesses forcefully shutting down other innovators by claiming some bullshit like CFAA. CFAA should really only apply to people who are doing SQL injections and other penetration. Vast majority of people scraping data do not fall under this category of malevolent behavior, although the dumbassery of people who wants to scrape linkedin for 30 bucks on freelancer ruin it for everyone.

I'm not sure Scrapinghub is funded externally as I couldn't find anything on their valuation or revenues so I assume that they are bootsrapped.

I do not discount any of what you wrote but a lot of it are imagined dangers, scrapinghub would be immune to such cases unless they sided with their customers like 3taps did. 3taps did not stop scraping for their client Padmapper. I don't think scrapinghub is willing to put their neck out for someone paying $20/month to scrape craigslist. In fact, those are the shittiest segments of this market imo, the bottom feeders who demand excessively unrealistic expectations from scraping as a blackbox magical world that will solve their problems.

> luxury of simply enjoying organic growth and funding expansion by re-investing profits into the company.

That's a GROSS over generalization. It definitely is not rosy, not when your competitors are VC funded and releasing products for free. You can't compete with free and VCs are realizing that maybe there's no money in free products.

that makes zero sense

I'm wondering about this but how realistic do you expect someone to pay when it's already free? I would imagine only large enterprise users so essentially you are supporting free users by charging enterprise users that may pay you for support.

Horrible industry imho when you have to give away things for free just to be competitive. I just don't get why people would expect software to be free.

yeah I actually thought about this and what I came up was like a circular cage with the person strapped in. It's digitally controlled unlike noisy hydraulics I don't know how but maybe lots of dry ice and superconductor they use for hoverboard.

Then you could generate inertia and g-forces by creating a centrifugal effect. For instance, to simulate a negative g maneuver in an airplane, you'd flip the user upside down and then rapidly spin the occupant towards the direction that creates the g-force due to centrifugal effect. Since you are constantly spinning at a very fast rate of speed, and wearing a helmet and headset you are unaware of this fact.

I'm not quite sure how safe this setup would be (spinning people for hours and hours at very high speeds)

> Most web scraping is illegal in the United States.

Got a source on that? Google scrapes all the time. This is how they index all the pages it discovers.

The only real scenario I recall is 3tap vs craigslist but they just kept scraping craigslist even after they banned their IP addresses with multiple proxies.

Then there are airplane ticket websites scraping each other and getting into hot waters.

Having said that it's not a clear cut definition as you'd like to put it. CFAA ruling was only because craigslist felt directly threatened by Padmapper which relied on 3taps.

interesting, how would a self-hosted open source version make money tho in order to support itself and continue to upgrade?

Is this even a realistic business model? Seems like this is what Scrapy is doing and what Import.io is doing. Make the tool free in order to get free marketing and then charge people willing to pay money to extract data.

Meanwhile I see Mozenda charging like 5 cents for each page extracted, do you think this is a fair model or does it not matter?