Hacker News Reader

wrathon Ask HN: What do people use to prevent crawlers?

When websites get a ton of traffic the concern is that the algorithm that find fake data will not be accurate and start blocking paying customers. So it's a fine line between blocking paying customers and fake data. What these algorithms do instead of blocking is to throw captcha's so if the traffic is really human, the captcha can be solved. The bigger problem is that there's a good chance that humans who are thrown a captcha will leave to go buy somewhere else (either because they are lazy, the captcha is hard, etc...).

Solutions like CloudFare and Distill have sophisticated algorithm to balance out fake and real data but even they are not close to being perfect.

wrathon Ask HN: What do people use to prevent crawlers?

I can't provide details on any innovations we've done with sites like google, but in general if you want to crawl google you'll want to get "many, many" IP addresses. I've heard of people using services like 2captcha.com but the best way is to obfuscate who you are.

If you can hit Google 60 times per minute per IP before getting blocked and you need to crawl them 1000 times per minute, you need 17 IPs per hour. Randomize headers to look like real people coming from schools, office buildings, etc... Lots of work but possible.

wrathon Ask HN: What do people use to prevent crawlers?

Competitive intelligence.

Crawling thousands of website, mashing up the data to analyze competitiveness between them, and selling it back.

For example, cost of flights. Different websites provide different prices for the same flight. The technology crawls all the prices, combines the data, then resells it back to the websites. Everyone knows everyones prices, keeps competition high, lower prices for consumers.

wrathon Ask HN: What do people use to prevent crawlers?

I don't think I'm being hypocritical. I have no issues if people crawl my site, I even identify who they are and give them access to a private API. I do not generate any income from that website though. I provide a service because I love doing it and I cover all the costs. Bots do increase my cost so I choose to limit their activity. Crawl me, but do so using my rules.

One of the alternatives is charging for my service but bots are not my users problem, they are mine.

wrathon Ask HN: What do people use to prevent crawlers?

I've built crawlers that retrieve billions of web pages every month. We had a whole team working modifying the crawlers to resolve website changes, to reverse engineer ajax requests and solve complex problems like captcha solvers. Bottom line, if someone wants to crawl your website they will.

What you can do, however, is make it hard so that the vast majority of developers can't do it (e.g. My tech crawl billions of pages, but there was a whole team dedicated to keeping it going). If you have money to spend, there's Distill Networks or Incapsula that have good solutions. They block PhantomJS and browsers that use Selenium to navigate websites, as well as rate limit the bots.

What I found really affective that some websites do is tarpit bots. That is, slowly increase the number of seconds it takes to return the http request. So after a certain amount of request to your site it takes 30+ seconds for the bot to get the HTML back. The downside is that your web servers need to accept many more incoming connections but the benefit is you'll throttle the bots to an acceptable level.

I currently run a website that gets crawled a lot, deadheat.ca. I've written a simple algorithm that tarpits bots. I also throw a captcha every now and then when I see an IP address hits too often over a span of a few minutes. The website is not super popular and, in my case, it's pretty simple to differentiate between a human or bot.

Hope this helps...

wrathon Ask HN: Who is hiring? (April 2017)

360pi, Ottawa Ontario, Canada Company Overview: 360pi helps top retailers compete and win in an era when shoppers expect and demand complete price transparency. Our customer base of "brick & mortar", e-commerce, and multichannel retailers accounts for over $US100 billion in annual retail sales and includes Ace Hardware, Best Buy Canada, build.com, TrueValue, and Guitar Center, among others.

We are looking senior and junior developers to help us write crawlers and products that will reshape the retail industry. If you are interested, see the links below. You can email any questions at dominic@360pi.com

Apply here: http://360pi.applytojob.com/apply/rJ6rlG5osz/Software-Develo...

wrathon Reddit is still in turmoil

I think it's that's a very good rate of growth for a new service. The scaling challenges alone are not obvious and slowly taking up market share is the safer thing to do. If I was an investor seeing these growth rates, I'd be really happy as long as it keeps up and only plateaus when the goal is reached.

wrathon Ask HN: First time pitch in front of a big company. What do they want to hear?

I think @nunobrito covered most of what I wanted to say.

Without knowing your product it's hard to give you the best advice, but here are things that worked for me in the past:

* Use real data that they can relate to. For example, if you're software does case management for non-profits. Find out who their customers are and create an "fake" example of how your software would work to manage a case for that company, from start to finish. This is partly for the 3 minute presentation, but more so for the barrage of questions you'll get asked afterwards once they like what they see. It's always easier to sell something when you find a solution to a pain your customer has right now, with one of his customers!

* Use only the terminology that they use and understand. Similar to #2, use the language that your customer speaks. Know your user and which language and acronyms they are using on a daily basis. I've seen so many times young startups use scientific terms or executive mumbo jumbo that the audience didn't understand.

* The 3 minutes is only the beginning. Whether you succeed or fail, it's not the end. Before, during, and after, identify the lead of the group that will be using your software and try to identify the decision maker. The person who signs the check if often different then the person evaluating your software. This is a sales cycle and they are both important for you to convince. Get a meeting with them, figure out what makes them tick, what their budget is, when will they be in a position to evaluate your software, etc..

Hope this helps a little, good luck!

wrathon Ask HN: Boss/Founder keeping my departure a secret. What to do?

I would politely insist that an email be sent out by the founder about your departure before the end of day. That way you get a chance of talking to your co-workers and explain the situation in your own words. To be clear, always be polite and never talk disparagingly about the company and co-founder, but the fact that you can't have a proper goodbye is a little strange. I've learned that it's always better to rip off the band-aid as quickly as possible while agreeing on an acceptable and honest message with the employee who is leaving. Better late than never I guess, so I would ask for this first thing this morning.

If it happens, send an email at the end of the day thanking everyone for the time you had with them, the knowledge you gain and the fun you had. Give your personal email and phone number in case there are any questions and/or someone wants to keep in touch with you.

wrathon Show HN: CloudScrape – Cloud-based web scraping platform

As far as I know, shader.io sells individual proxies. Luminati and Nohodo are exit nodes priced by the amount of bandwidth that you use.

wrathon Show HN: CloudScrape – Cloud-based web scraping platform

Sorry, my bad... it's shader.io

wrathon Show HN: CloudScrape – Cloud-based web scraping platform

If these guys are serious about their business they are using proxies to obfuscate who they are. There are many services now that are claiming to have a ton of IP addresses Luminati, Shadio.io and Nohodo are just a few examples.

wrathon Ask HN: Who is hiring? (September 2015)

360pi Full-time onsite in Ottawa, Canada. http://360pi.com

360pi is a company organizing the worlds product data for retailers, brands and consumers. World class dev team, leading edge challenges in cloud, scaling, data and AI. We’re looking for infrastructure/ops developers, QA and data gurus.

Apply here.. http://360pi.com/careers/

wrathon RegExr: A website for interactive regex prototyping with syntax highlighting

I love and miss RegexBuddy! If only they had a Mac/Linux version :( I've had thoughts about switching back to Windows while developing complex regular expressions.

wrathon Ask HN: Should I consider a startup based on scraped data?

I've built several businesses that either relied in-part to scrapping/indexing websites or solely relied on scrapping/index websites. I We never achieved the success of Google but we did get large enough to be noticed by some sites (Amazon for example). We did face legal issues but of a different kind. There were a few bugs early on that made us hit websites too much and we did receive a couple of cease and desist letters. We fixed our problem and explained the situation to the site owner and everything was resolved.

The only "fair use" type issue that we encountered was using logos from websites. E.g. Displaying the logos of the websites we indexes on our site. Once again, nothing serious came of it. I believe our marketing department removed the logos and put text instead.

Personally, I wouldn't worry about these issues until it becomes a problem. When it becomes a problem it means you're on to something and you're disruptive enough to get some attention. It's a good problem to have IMO.

wrathon Ask HN: Who is hiring? (April 2015)

I apologize if I offended anyone and/or if I said anything improper; it was not my intention. I wrote this quickly during the March "Who's hiring" thread and copy/pasted it today. The role that was linked last month is a very early entry, out of college, role. 99% of our applicants for these roles are under 30 so I can only guess it was a subconscious thing on my part to put "young" in the description.

Thanks for holding me accountable!

wrathon Ask HN: Who is hiring? (April 2015)

360pi, Ottawa Ontario, Canada

Company Overview: 360pi helps top retailers compete and win in an era when shoppers expect and demand complete price transparency. Our customer base of "brick & mortar", e-commerce, and multichannel retailers accounts for over $US100 billion in annual retail sales and includes Ace Hardware, Best Buy Canada, build.com, TrueValue, and Guitar Center, among others.

We are looking for young and talented developers to help us write crawlers and products that will reshape the retail industry. If you are interested, see the links below. You can email any questions at dominic@360pi.com

http://360pi.theresumator.com/apply/

wrathon Ask HN: Who is hiring? (March 2015)

360pi, Ottawa Ontario, Canada

http://360pi.theresumator.com/apply/D1tgGu/Technical-Service... http://360pi.theresumator.com/apply/4g5vIa/Devops-Specialist...

wrathon Ask HN: Sentiment Analysis – how to handle biased word list lengths?

1. You can try using bi-grams or even tri-grams to make you word list a little more precise.

2. Create a validation by manually identifying each review as positive or negative. Each time you modify your algorithm run it through your validation set and note the results in a spreadsheet. If you don't do that, you'll never know if and how you've improved the results. The bigger the validation set the better. Similarly, you can use part of your validation set as a training set into a classifier.

3. Find a scale that works to bias your score. For example, I would try to bias your negative score using a log scale. The fewer negative words you have the more they are worth, the more you have the less they are worth.

wrathon Pricenoia shuts down

Amazon has made it much harder to scrape their site in recent months. They now give captchas to bots that try and access their site to often by the same IP address. Unless you have a very large pool of low cost proxies, crawling millions of prices on a daily basis is challenging. I think this is what he's referring to when he says """and has lately made it very difficult to get the data in other ways."""

wrathon Wikipedia refuses to delete photo taken by monkey

http://www.blogcdn.com/travel.aol.co.uk/media/2013/01/323ng1...

My understanding is that this image was captured by a camera trigger that was setup National Geographic photographer (Sorry, I'm unsure the name). This has been a highly published photo by National Geographic; does this mean that the tiger should be owed all the royalties and not the photographer?

wrathon Ask HN: Do you cold call?

A company that I started in the early 2000s used cold calling to generate leads. It was very affective to generate solid qualified leads. We were lucky that we go an experience cold caller that trained other younger sales people. Having someone (either an agency or a person in-house) helping you is key to your success.

IMO, a good cold caller will work very hard to get to the right person, if that person exist. They will be committed and relentless to generating the lead for their sales team. They will work on their script to a point where they have answers for most types of questions. They will use the tools that they are provided (e.g. salesforce) to ensure to keep a log of what they've done and their todo list. And they will be in constant contact with the sales team to ensure that all information is provided back to them in a structured manner.

If you think that cold callers are waisting your time, simply try and tell them that you are not interested or point them to a different person who may be interested (if that person exists). There's no point for a caller to place a call to someone who has no interest in buying their offering, that's just a waste of time. Cold callers are "not" telemarketers.

wrathon Lessons learned from my failed startup after 2 years, 300 users and 0 revenue

Good article but I would look at this "failure" from a glass half full perspective. You "won" because you've learned valuable lessons you can take to your next idea. I've had many products that have not gained many users in their respective marketplaces but I learned from each and everyone of them. All these experiences has brought me where I am today (CTO of a 45+ employee company). No failures in my past as far as i'm concerned; just lots of self teaching (that you can't get in school).

>> ""An Indian company expressed desire to buy something from me other than the thing I was building, so I should have built that instead.""

I may be the minority but I agree with him but on one condition. If this Indian company wanted to pay a small monthly subscription fee for your product I would never have agreed developing "their" ideas. I would have taken their feedback and put in the big pile with all the other feedback I gathered up. But I would have pitched this Indian company a different story, I would have pitched them a professional services contract instead of a product. I did something similar in the past and it worked out very well because in a business money is king. With no money you can't do the things you need to do, like attend conferences to sell your idea, buying adwords, hiring solid developers, paying yourself a salary so you can devote your time to the idea.

In my case the customer was willing to pay ~$10k a month to get what he wanted. We built it for him while building our own product. Once we got big enough and could sustain ourselves without our original customer, I gave the customer away. The developer who maintained the project was interest in taking on the project himself. We came up with a 6 month transition plan, including lots of product/project management help, office space, etc. It was a win/win situation at the end.

Doing this is not for everyone though. There are many days I cursed this customers for taking up the majority of our resources. We had to be very good at differentiating between their requirements and the markets requirements. We weren't perfect at it but it worked out in the end.

wrathon Citibank India wants credit card, bank account numbers to stop marketing emails

This is a phishing attack waiting to happen! I never worked at a bank but I'm assuming (maybe I shouldn't) that there are a few people working there that know a thing or two about security. I doubt that any person who claims to be a "security expert" would have let this go by, but I always seemed to be proven wrong. Take for example TDBank in Canada who has a 80's password policy:

Passwords must:

- be 5 to 8 characters in length

- not contain spaces or special characters (e.g. #, &, @)

Poor customers if TD ever gets their password database stolen.

wrathon Ask HN: Was your college degree a necessity to get a job as a developer?

First, when you read "Bachelors in CS or Equivalent!?", the equivalent part usually means "work experience". So if you don't have a CS degree you need the equivalent work experience.

I'm a non-college grad and I think my career up to now has been pretty successful. It was hard for me, much harder than it should have been if I had a CS degree. I had to work 2 or 3 times harder than anyone else just to keep up at times. I found that it wasn't the course material that I lacked since I read pretty much the same books as them. What took me a long time to realize is that all my colleagues were using the same terminologies and most importantly were "thinking" the same way. The majority of them had either a CS or a physics degree of some kind.

Now that I'm in a position of hiring many developers I'm not against hiring individuals without degrees, but it definitely helps. I've hired two developers without degrees. Both are super bright people but are struggle to keep up as I did.

All that said, IMO it's who you know not what you know, no matter of your background. You can get a job in a "big company" without a CS degree if you have the skills and you know the right people. I did it by attending a bunch of meet-ups to find people with the same interests as me. I made friends that were working for these "big companies" who got me interviews, etc...

I would stay in college if I were you. I didn't go but should have.

wrathon Amazon said to launch Pantry to take on Costco, Sam's

"Warehouse club members tend to be higher income households with kids — the type of shoppers that have huge lifetime value to retailers."

Assuming that this is true, I think this is a brilliant, long-tail, idea from Amazon. If the kids get accustomed to seeing Amazon boxes full of stuff arrive every week, when they move out on their own they probably will also buy from Amazon. Similar to computer manufacturers (e.g. Apple, Microsoft) giving educational discounts to schools/students. It's hard to know if this implementation will work but I would start worrying if I were Costco/Sam's. Amazon has shown that they do find innovative approaches to markets. If this approach doesn't work they'll come out that will later on.

wrathon Ask HN: Who is hiring? (November 2013)

360pi - Ottawa, Ontario, Canada

Company Overview: 60pi helps top retailers compete and win in an era when shoppers expect and demand complete price transparency. Our customer base of "brick & mortar", e-commerce, and multichannel retailers accounts for over $US100 billion in annual retail sales and includes Ace Hardware, Best Buy Canada, build.com, Crutchfield, Overstock, and REI among others.

We are looking for many development positions, including: QA Automation Developer, Application Developer, Data Quality Manager, Data Workflow Developer, Implementation Developer

You can apply here: http://www.360pi.com/about-us/careers.aspx, email your resume or questions to careers@360pi.com and/or dominic@360pi.com

wrathon Ask HN: Who is hiring? (September 2013)

360pi (Ottawa, Canada)

For those who are interested in big data, working with the top retailers, crawling the web and the latest technologies.

Some of the open positions:

- Lead Development Representative - User Experience Designer - Webkit Crawler Developer - Data Import Developer - Implementation Developer - QA Automation Developer

Message me if you're interested or apply through http://www.360pi.com/about-us/careers.aspx

wrathon Why I'll be a solo founder next time

In my experience the bottom line is that you need good people around you in order to succeed. It doesn't matter if that's a co-founder, a CEO, developers, etc.. If you don't have good people around you helping you in your journey of success you won't have any.

For example, in my first startup I had a very good co-founder but we were both very young and inexperienced. We didn't hire the right people to surround us and help us in our journey. As a consequence we had a great run but were out maneuvered by other companies. In the startup I'm doing now, I started with great co-founders and we then proceeded to hire more great people to surround us. We haven't completed our journey yet but the driving force in the company now is not only me (the other co-founder left to pursue other interests) but many others that share my passion and goals.

Take your time and look for people that share your vision and passion. These people are out there it just takes time. And remember, no one can do it by themselves.

If you never read it, by a copy of Good To Great (http://www.amazon.com/Good-Great-Companies-Leap-Others/dp/00...). There's a whole chapter devoted to how great companies need great entourages.

wrathon Show HN: My product lets you catalog your book collection online

Great job! Not many people at your age can create their own products.

Many people here post un-productive comments and are very harsh but there's usually some truth being them. My advice to you is to take all the advice you "want" and build upon that. It takes a long time in order for you to get it "right" so keep changing things until it sticks.

Have you tried using Amazon as your search engine for ISBNs? From experience it is very accurate and complete. You can scrape using a URL like this: http://www.amazon.com/s/ref=nb_sb_noss?field-keywords=034551...