Hacker News Reader

Comment by jhow15 | Hacker News Reader

Hey everyone, Josh here the creator. As a developer, I’m always looking out for an emerging market trend on which to bootstrap my own SaaS product. This is since market awareness and timing are often so critical, on top of execution skill of course!

So I built Trennd. Under the hood it continually monitors the web for interesting keywords & topics, classifies them using Google Trends data and packages everything as a neat web app where other people can contribute too.

The app itself is built with the Next.js React framework along with Express, Bootstrap and MongoDB. Next.js was new to me, but made sense since it comes with so much out of the box, including server side rendering.

Let me know if you have any questions, and any feedback is much appreciated!

View on HN

These results aren't filtered in some way? I find it suspicious that almost all of the top ~100 results are tech related.

> I have scripts that are crawling the web in places like twitter, reddit and facebook to generate keyword ideas. So the trends that it detects is biased towards the corners of the web that I am monitoring. Which skew heavily tech!

You said you are monitoring interesting keywords & topics but how do you know which keywords & topics to monitor?

Also, if you are choosing the keywords then wouldn't the data be inaccurate for predicting trends as trends occur randomly?

I have scripts that are crawling the web in places like twitter, reddit and facebook to generate keyword ideas.

So the trends that it detects is biased towards the corners of the web that I am monitoring. Which skew heavily tech!

This is a great service. It would be great for you to take suggestions on other locations to crawl.

That's a brilliant idea, will add somewhere for people to make suggestions. Thank you!

Hi Josh, Very interesting stuff. Can I ask what is the math behind defining the trend? How did you calculate the most trendy words, etc?

I'm working on something similar for a internal tool, so I'm curious to see what your solution is.

Thanks!

Of course - I have scripts constantly monitoring the web for potentially interesting keywords. I then check google trends data for that topic/keyword and analyze the gradient of the line of best fit over different timeframes. If the gradient is higher than a certain (arbitrary) amount, then I class it as a trend!

Is the gradient just a linear least squares fit?

Not the author. Seems like it, it doesn't look like Google Trend gives other variables, so just simple 2 var regression I'd guess

Nice work Josh.

It can probably be used for lots of things: startups, as you mentioned, but also journalism ideas, investing ideas, job hunting and many more.

Thank you! Yeah totally agree, I hadn't thought of job hunting though!!

Pretty neat, I see the most granular time scale currently supported is a 3-month window -- are there any plans to increase granularity towards a (near) real-time system? I get your crawler might be rate-limited at that scale and you would likely have to rethink the entire system architecture, but I think real-time data would open up a whole new dimension of possibilities for detecting/predicting trends perhaps even before public awareness catches up. I'd imagine there would be parties interested in paying for that kind of capability if that's the direction you're planning on taking this idea

I'd love to go as granular as possible, at least 30 days. Real-time would be ideal, but as you note - Google Trends rate limiting soon becomes a hurdle especially when you potentially multiply by different geographies.

What could be a really neat addition is having trends sorted by countries

Great idea, thank you!

Awesome I enjoyed making some trend charts, and the results are fun (i.e I added DevOps, Erlang, Elixir). I think there is a risk it could conceal too many errors and biases for it to be useful for trend analysis. Using services like Twitter and Reddit might introduce biases into your data skewing trends. The challenge is using data and not understanding the intent (is it positive, negative or neutral/sarcastic potentially negative/positive). Google Adwords is good at this. I applaud your effort; I hope you keep going but its a tough nut to crack. I use markets like Google Adwords Tools to do this at the moment, have you looked at it? I think more info around time, intent and volume.

I don’t have much feedback, except to say I think this is a really good, useful concept.

Thanks so much!

Hi, it would be nice if the keywords had short explanations. I had to find out what an affinity designer and an enneagram was. Bubble tea was also quite new.

OT but...

> built with the Next.js React framework

Wait wait wait... do you mean to say there's now a framework for a framework? Javascriptland continues to confound.

I wouldn't call React a framework, it's more like a library.

Hey Josh, very interesting concept. Thanks for building it!

I tried to use the "technology" category, but the results are not what I expected. Perhaps it would be more useful if I could narrow it down to more specific things, e.g. programming languages, or databases.

Keep up the good work!

Hi Josh - it would be incredible to have this filtered by location, and I think you can get that from Google Trends.

Absolutely agree! How specific/granular would be the most valuable to you - at the city level perhaps?

Yes - City would be great for people doing stuff around activities, events and local places. But Country would be more powerful for e-commerce I suspect (my usecase).

would be interesting to also to see the opposite, retreating trends

It seems to already have that (the "gradient" arrow)

Reply on news.ycombinator.com