Ask HN: Work to search for if I enjoy web scraping and browser automation | Hacker News Reader

Ask HN: Work to search for if I enjoy web scraping and browser automation

5 pointslatexr9 comments

I’ve always worked freelance. Mostly as a graphic and web designer, including coding the front-end. I’m mostly self-taught as a programmer, skilled in Bash, HTML, and CSS, and competent in JS and Ruby.

For a while now I’ve started to notice that what I really enjoy is the web scraping and browser automation I do for myself. I tend to use Ruby for the task, relying mostly on Nokogiri and Watir. I do it to automate repetitive tasks on websites or to build personal RSS/JSON feeds of websites that do not provide them (or are limited in what they provide). I don’t feel tied to those technologies. I enjoy them but feel confident I can learn another language/library enough to get the job done, if need be.

1. Having never worked professionally as a programmer, and having no formal training, what kind of job or contract/freelance work should I be searching for?

2. What’s the job title of someone that does this type of work?

3. Are there any specific skills I should be cultivating? For example, if most available work for these skills is in testing (is it?), should I be focusing on learning a testing framework?

4. Are there any specific technologies you’d recommend I learn? Why?

5. Any other advice?

What you are describing as browser automation is quite similar to automated QA and end user testing.

Many QA departments need people who can automate their manual interaction step process to ensure that a service is behaving correctly.

It’s costly to manually check all the user stories that an application supports, and automating this can be a career path for an employee or a contractor.

My recommendation would be to get your foot in the door at a company that has a large manual QA process and is looking for someone to write for them automated tests, or convert their manual steps into tests.

There are 2 fields - Web scraping is done to get data from the web, automation is done to ensure websites keep functioning as planned.

1. For web scraping, you may get consulting jobs. For automation testing, search for job title of 'QA Automation Tester'.

2. Consultant / Tester / Analyst (Depending on seniority till you become Testing Manager)

3. I think there is a much larger market for testing. You can start by learning selenium. There are many testing frameworks, but it comes down to the company - which programming language and framework they use.

4. You can think of learning appium for mobile (android and iOS) automation.

5. Maybe you can try to search for work in Ruby for starts.

Hi there, a sector in which web scraping is quite important is real estate. I have written an open source scraper for that sector (using Nokogiri and Ruby on Rails):

https://github.com/RealEstateWebTools/property_web_scraper

Take a look and let me know what you think. I have a few ideas about how to monetize. Get in touch if this is something you would like to work with me on ( ed at propertywebbuilder.com )

I have no clue about the legality of what these guys are doing but they are interesting - https://www.aggdata.com

Might give you some ideas about markets, pricing etc

We need someone with those skills right now. Drop me an email. moura at oko dot ai

4. Javascript, Puppeteer

Hey - reach out to me, contact in profile

create a simple web presence site

then create a portfolio

then start blogging.

create a free tool people in your target market can use

reach out to other web dev/phone shops and mention what you specialize in and see if they come across this work and want to refer it to you or resell it, etc.

create social media profiles linking back to your site.

Reply on news.ycombinator.com