Quick but powerful research for AI agents with data scrapping and selenium | Hacker News Reader

Quick but powerful research for AI agents with data scrapping and selenium

8 pointsalexvomwald6 comments

got curious about what software businesses actually use and how much revenue they might be pulling in, so I built a scraper to dig into it. Using Google Maps listings, reviews, and booking systems, I mapped out an industry’s landscape.

I scraped businesses across the U.S., analyzed reviews to estimate customer volume and revenue, and tracked booking software by checking subdomains and site elements. Connecting the dots between businesses and their software providers, I estimated MRR, ARR, and potential exit multiples.

This gave a solid market overview, showing which booking software is most common, how tech-savvy businesses are, and rough revenue estimates based on customer activity.

I put everything—scraper code, SQL tables, and sample data—on GitHub if you want to check it out or improve it.

GitHub Repo: github.com/alexdelbosque/data-crawler-smbs

Has anyone else done something like this? How would you make it better? Let’s swap ideas!

I did many projects with web scrapping, that is in the past without AI :)

always felt nice to gather data but rarely could i do something useful with them or some sort of a good product. So today i dont think data is the problem, rather usefulness, PMF and marketing

TLDR: it is all about what you make with the data.

Definitely. Data won't create the opportunity if you don't know what to do with the data or it won't answer the right questions if don't know what to ask. I think generally the accessibility to gather, process, and make decisions with data has just become so low and it's almost a given to do such analyses.

At the same time, many "business" people I talk to, don't know how to do these analyses. And many technical people, while they could easily do such analyses, don't think of data from that business perspective. |

It’s a lot of data, but what’s the business use case for it? What’s the most compelling problem it can solve?

In this case, I wanted to map out all companies within an industry for which I am building a solution (a voice and chatbot agent to automate all front-desk tasks).

This meant getting a list of companies and estimating their size to see which ones are the best to target first (e.g., multiple locations, consistent revenue).

I also wanted to get an overview of which software all these companies are using since the solution I am building has to integrate with their existing solutions. So besides knowing the big players in the market, this helped me to map out (for the most part) which companies are using which systems) to then narrow down on which ones make sense the most to integrate first based on their market penetration and availability of APIs.

Then I'm scraping the data from the actual booking site (which uses one of the identified software), to get their menu of treatments, descriptions, prices, and durations. With this, I'm building demos that create a simple showcase of how the agent could look to help them automate their phone calls, chat messages, and social media interaction.

The thing missing would be the integration with their booking systems and some additional personalization of their agent, but I'm planning on using these demos to make the pitch feel more real easy, in an efficient. way.

So it helps in any case where you want to get a more detailed overview of an industry (relying on google maps data) and if you're trying to identify which booking, ordering, etc. software they're using.

is it possible you pull this off but for Chinese, European or GCC market? I feel there is complete gold mine out of the US which people might be ignoring.

It works for any country with Google Maps, so China won't work but besides that, you can get the data in many countries. Some countries are not that active in reviews in Google so data might be more limited, but I've done similar stuff in the past for Mexico and Brazil and it worked great.

Reply on news.ycombinator.com