Show HN: I built a service to discover rapidly growing Google search topics | Hacker News Reader

Show HN: I built a service to discover rapidly growing Google search topics

trennd.co

446 pointsjhow15116 comments

Hey everyone, Josh here the creator. As a developer, I’m always looking out for an emerging market trend on which to bootstrap my own SaaS product. This is since market awareness and timing are often so critical, on top of execution skill of course!

So I built Trennd. Under the hood it continually monitors the web for interesting keywords & topics, classifies them using Google Trends data and packages everything as a neat web app where other people can contribute too.

The app itself is built with the Next.js React framework along with Express, Bootstrap and MongoDB. Next.js was new to me, but made sense since it comes with so much out of the box, including server side rendering.

Let me know if you have any questions, and any feedback is much appreciated!

You said you are monitoring interesting keywords & topics but how do you know which keywords & topics to monitor?

Also, if you are choosing the keywords then wouldn't the data be inaccurate for predicting trends as trends occur randomly?

I have scripts that are crawling the web in places like twitter, reddit and facebook to generate keyword ideas.

So the trends that it detects is biased towards the corners of the web that I am monitoring. Which skew heavily tech!

This is a great service. It would be great for you to take suggestions on other locations to crawl.

That's a brilliant idea, will add somewhere for people to make suggestions. Thank you!

What could be a really neat addition is having trends sorted by countries

Great idea, thank you!

Nice work Josh.

It can probably be used for lots of things: startups, as you mentioned, but also journalism ideas, investing ideas, job hunting and many more.

Thank you! Yeah totally agree, I hadn't thought of job hunting though!!

Hi, it would be nice if the keywords had short explanations. I had to find out what an affinity designer and an enneagram was. Bubble tea was also quite new.

Hi Josh, Very interesting stuff. Can I ask what is the math behind defining the trend? How did you calculate the most trendy words, etc?

I'm working on something similar for a internal tool, so I'm curious to see what your solution is.

Thanks!

Of course - I have scripts constantly monitoring the web for potentially interesting keywords. I then check google trends data for that topic/keyword and analyze the gradient of the line of best fit over different timeframes. If the gradient is higher than a certain (arbitrary) amount, then I class it as a trend!

Is the gradient just a linear least squares fit?

Not the author. Seems like it, it doesn't look like Google Trend gives other variables, so just simple 2 var regression I'd guess

I don’t have much feedback, except to say I think this is a really good, useful concept.

Thanks so much!

Hey Josh, very interesting concept. Thanks for building it!

I tried to use the "technology" category, but the results are not what I expected. Perhaps it would be more useful if I could narrow it down to more specific things, e.g. programming languages, or databases.

Keep up the good work!

would be interesting to also to see the opposite, retreating trends

It seems to already have that (the "gradient" arrow)

These results aren't filtered in some way? I find it suspicious that almost all of the top ~100 results are tech related.

> I have scripts that are crawling the web in places like twitter, reddit and facebook to generate keyword ideas. So the trends that it detects is biased towards the corners of the web that I am monitoring. Which skew heavily tech!

OT but...

> built with the Next.js React framework

Wait wait wait... do you mean to say there's now a framework for a framework? Javascriptland continues to confound.

I wouldn't call React a framework, it's more like a library.

Awesome I enjoyed making some trend charts, and the results are fun (i.e I added DevOps, Erlang, Elixir). I think there is a risk it could conceal too many errors and biases for it to be useful for trend analysis. Using services like Twitter and Reddit might introduce biases into your data skewing trends. The challenge is using data and not understanding the intent (is it positive, negative or neutral/sarcastic potentially negative/positive). Google Adwords is good at this. I applaud your effort; I hope you keep going but its a tough nut to crack. I use markets like Google Adwords Tools to do this at the moment, have you looked at it? I think more info around time, intent and volume.

Pretty neat, I see the most granular time scale currently supported is a 3-month window -- are there any plans to increase granularity towards a (near) real-time system? I get your crawler might be rate-limited at that scale and you would likely have to rethink the entire system architecture, but I think real-time data would open up a whole new dimension of possibilities for detecting/predicting trends perhaps even before public awareness catches up. I'd imagine there would be parties interested in paying for that kind of capability if that's the direction you're planning on taking this idea

I'd love to go as granular as possible, at least 30 days. Real-time would be ideal, but as you note - Google Trends rate limiting soon becomes a hurdle especially when you potentially multiply by different geographies.

Hi Josh - it would be incredible to have this filtered by location, and I think you can get that from Google Trends.

Absolutely agree! How specific/granular would be the most valuable to you - at the city level perhaps?

Yes - City would be great for people doing stuff around activities, events and local places. But Country would be more powerful for e-commerce I suspect (my usecase).

This definately has a market, but i'm not sure SaaS is it. More typical eCommerce products have a "flavor of the day" like effect, where an article is written in some niche community, and everyone in that community rushes out to buy the thing. Then a few months later, the craze is over. If you're Amazon, you can spot these trends easily (because they have everyone's sales data from the marketplace). If you're a small eCommerce company, unless you just happen to have that product on hand, and someone just happened to have found you, you may never even realize the trend existed. Being alerted that there is a sudden demand for a product, might give you a headstart on marketing and might help you negotiate a deal with a supplier before the supplier realizes what they have.

I'm a moderator of one of the bigger Facebook advertising groups that exist. This caught my eye instantly as the e-commerce and affiliate crowd are going to be all over it. They love being able to catch trends relatively early. This product is ready made built for flavor of the moment marketers.

That's awesome to hear!

If I were to build Trennd towards these e-commerce guys, what would you improve/add/change to make them love it?

For sure. So shorter term. They're going to want to move faster than 3 months. Maybe like a past 30 day trend if possible? I would say also if there's a way you could dig into Alibaba and see if there's a bunch of new items of the same type getting listed recently that would be extremely interesting for them as well.

Last 30 days - got it!

That's an interesting angle regarding Alibaba. The hard part might be automatically grouping similar items, but definitely can see the huge value in this.

When I was doing affiliate marketing a decade ago, what I did manually was watch informercials and jot down the names of all those products. Then manually run searches in Google Trends to see which ones were spiking, and create content for those products (ie reviews, guides, etc). Here's an example of a product that really took off after they did an informercial in 2006: https://trends.google.com/trends/explore?date=all&q=almighty... . I created a page for this product and was ranked #1 instantly. Because nobody else was writing about it.

I stopped doing affiliate marketing so don't know what people do now, but just throwing you some ideas.

Per the discussion above, this seems to be a pretty spot-on ID of one of these "flavors of the day"...!

not sure i agree so much because its not showing things that are just starting to trend, which is where the real money is at... being in front of the ball rather then behind/beside it. best i can tell this service just shows you the search volume history from about 1100 select terms that anyone can already lookup using googles trend tools.

id be much more interested to see terms each day whose search volume has increased 10x or something... that would be very useful for people trying to get in front of trends.

Appreciate your insight - really trying to figure out who I should be building this for, so I know what features and direction to focus on. I was thinking eCommerce entrepreneurs could be it, but then a lot of other people potentially find it valuable too. I want to see who keeps coming back to Trennd in the future, the power users, and ask them why/what they're using it for!

Thanks Josh, I sent you an email I think you answered most of the stuff here though just came back to the comments now. Would still be great to have a chat sometime

The "flavor of the day" comment really caught my attention, been thinking of a startup between those lines. I think with people having hundreds of decisions to make everyday from a million choices, it would help to answer those questions for the users without them having to spend time deciding.

I mean its great to have review sites, but what should I have for lunch today? what is trending just give me the answer I am happy to pay $5 a month you decide for me. What programming language is trending?

Will look into it at some point, happy to have a chat with anyone who's interested my email is in my profile

Very cool! As a front-end dev it's interesting (but I guess also not surprising) to see React and Vue with very steep curves.

I wanted to try adding Svelte, so I signed up by email, but that doesn't seem to be working. Then I tried Sign In With Twitter but, wow, it requires an awful lot of permissions! Among them... Follow new people, Update your profile, and Post Tweets for you. I'd be a lot happier signing in with Twitter if it were limited to read-only abilities.

yea, or just signing in with an email and password. not sure why everyone thinks they need to have perms to my other services just to user theirs.

Yeah, I’m not crazy about this trend toward emailing magic login links. I’ve got a password manager...

This is similar in some ways to https://meetglimpse.com/

The key value-add, in my mind, from that product, besides the trend, is the editorial content they include in the email. Of course, that's just me as a layperson.

I'd probably dupe their pricing model and approach: i.e., a limited number of weekly/monthly trends, subscribers get more.

Definitely - the glimpse newsletter looks good and is the same concept of surfacing new Google trends. I didn't like being limited to just a few trends per month everyone else sees via newsletter though.

I was looking for an interactive app with 1000 trends I could dig through. But as you say, then the editorial content/insight for each trend is key value-add. That's part of the reason I've built in for users to be able to add their own trends, insights and comments. If we can crowdsource it, then to some extent we don't have to trade-off quantity and quality.

Is there a plan to sell the data as an API?

I am currently working with a client who is exploring the options for building out a similar system.

Selling the data as an API is definitely a potential. I was originally thinking to allow CSV downloads of the data. Would an API be more useful for your client's use case?

You seem to have done most of the hard work, collecting the data. I'm not sure they'd care what format the data was in. Providing it could be expanded to cover the topics their clients need.

Yeah absolutely - topic coverage can be expanded. Shoot me an email and we can chat more: josh at trennd dot co

Sounds great, I'll ping you an email sometime next week!

And/or sell the service where you BYO keyword list and this ranks them. Keyword lists can themselves be scraped from forums and websites in the marked the client is interested. The scraping could also be an add on service.

This would lock in nicely with the keyword research that ahrefs, moz, etc. offer. And other various colour hat seo software.

How can there be a big spike in the 'cattle' trend - https://trennd.co/trend/cattle - but no corresponding change on Google Trends - https://trends.google.com/trends/explore?date=today%205-y&q=... ?

The spike for cattle appears to be in Google trends for me: https://trends.google.com/trends/explore?date=today%205-y&q=...

Looks like if you go back a few years you can see a bump.

There seems to be different classifications of Cattle - search term, Animal and Cattle.

No idea what that means, but it suggests why you're not seeing the same data.

Is there a Pro version where I can get alerts and manage my keywords etc and have a dashboard?

It's interesting to see that topics like 'fentanyl', 'opiod' have done up.

Also 'criminal defenses', 'wrongful death claim' have gone up a lot as well - Perhaps they are related to spikes in searches for plastic surgery topics.

Also it's weird why almost all the topics on the front page are related to software development. It's like there is some kind of conspiracy to turn the world's population into software developers.

Nice idea. For very short time spans, I noticed some "trends" seem to be seasonal trends ("aperol", "lawn mower"...). It might be interesting to compare with the trend for the same search the year before at the same period, to be able to identify those cases (for intance, I cannot see from the graphs whether Aperol is more or less trendy this summer compared to last summer)

Where's the data coming from? I assume the trends are coming from Google Trends but how are topics found?

The examples on the front page looks surprisingly good, which makes me suspect they were hand-picked. Kind of beats the purpose of the app if I'm just shown trends the author and me already know about, since we seem to inhabit the same subcultures.

Yes, I do filter out any chaff, since the scripts I have crawling the web to generate keyword ideas aren't perfect!

Right now they're monitoring the obvious techie havens, but I'm looking to expand their coverage to be more comprehensive.

When building a system that surfaces interesting trends, you have to be careful about curating which historic examples you show. People will craft their understanding of predictive power based on those examples... but if the examples aren't representative (i.e. randomly sampled) of the algorithm, then you're selling people rainbows and unicorns.

Ok, anyway cool app! I actually found some cool trends via this.

What about also crawling data from AdSense? That way you also get the search numbers, not just the relative graph.

You see these numbers as part of planning for a campaign. There is also an API for it.

Many years ago I built a service on appengine to hit the auto complete api endpoint for every single and 2 letter prefix, as well as things like "how do I" or "what is the".

It ran for a few months before blowing up the free quota, but it was pretty cool while it lasted.. Seeing certain things bubble up and then disappear. Like right now 'lion king' is the first suggestion for 'l', but I doubt it was a month ago.

So... whats up with: https://trennd.co/trend/craft-beer

A peak every April and a dip every December? This seems like the result of some smoothing algorithm needing ~4 months of data and resetting every year.

This one even more consistent: https://trennd.co/trend/airbnb

People search for more craft beer before Christmas (because they're looking for a unique gift) and less for AirBnB (because they get together with their families for the holidays).

Seasonal patterns in data about human behavior are the norm rather than the exception. That includes search trends. There are also monthly, weekly and daily cycles, as well as some that have not quite yearly period, e.g. Ramadan.

Cool idea, but all charts are empty. Hugged by Hacker News?

Same issue here, on mobile - firefoxpreview for Android - if that helps

Yeah - just got hugged!

Upgraded my mongodb atlas cluster to resolve the issue, apologies.

I wonder if it would not be more interesting to see the unique search queries instead ie. the least popular search queries.

(In the spirit of "YouTube videos that have almost zero previous views" [1] / astronaut.io)

[1] https://news.ycombinator.com/item?id=20432772

I'd also like to see downward trends, not just growing trend. From the last page or two, I see some downward trend, like "blog", "facebook", and "gmail". I don't know if it means the decline of popularity, but it is definitely interesting to see the declining graph.

Brilliant stuff. Fascinating to see how seasonal some of the searches are – "rosé" and "Aperol" spikes in July and December (southern/northern hemisphere summers?), "carbs" and "low-carbohydrate diet" peaks in January (when people want to change their habits?).

This is pretty brilliant. I'd hook up something to identify trends with companies that can potentially be invested in. Could be useful from a publicly traded stand point (Good ole stock picking) or even VC investing if the product was rapidly growing and not yet heavily invested in.

Very minor bug - if a user has an emoji in their Twitter name the default blank avatar doesn't quite work correctly - https://trennd.co/user/5d31d11ce6f3ce02e53269c7

Ahah - thanks for the heads up!

Is there some open source version of something like this? I would be interested in monitoring some non-tech topics and it would be useful to have something with most of the basic features in place to do so, even if it's a library

`Gmail` appears to be trending down but exhibits local maxima right around Spring and Fall equinox: https://trennd.co/trend/gmail

Now why would that be...

I guess the most logical conclusion is that every time they make a sacrifice under the full moon during the equinox, they get rewarded with more interest in Gmail.

OT, the chart for remote recruitment looks very cyclic -> https://trennd.co/trend/remote-recruitment

any idea where this is coming from?

As a quick feedback, currently the front page has no indication what use signing up is.

Signing up allows you to add and upvote trends yourself.

But duly noted! I will make that clearer.

I figured out the adding trends after clicking on one.

Are the trends on the frontpage sorted by upvotes?

They're sorted by gradient. But you can sort by upvotes too.

The "Insight" section needs more attention, Suposedly is for subject matter experts yet in this example we see uninformed biased opinions.

Examples

1) https://trennd.co/trend/ketogenic-diet

2) https://trennd.co/trend/plant-based-diet

This is awesome. I'd love to have the ability to view topics based on category. IE. see the biggest trends in entertainment or the biggest trends in sports players, or the biggest trends in startup brands.

There seems to be a filter on the top to do just that.

https://trennd.co/?categories=%255B%2522sport%2522%255D

There are quite broad category filters you can use right now, like "sport" or "food".

Definitely looking to classify with more precise categories in the future like sports players etc as you mention

Very cool, but this is essentially not "discovery". There is a heavy skew toward tech - but niche tech (Notion for example) specifically. It would be cool to get a broader, less bias perspective.

If possible, I'd like ti integrate with the real monthly search numbers for each keyword, I can't get it anywhere, only see a trend score, could u explain a bit more on this?

That's very neat and interesting; I guess the two questions are - how do you generate keywords to search for ? - how do you filter out noise (or perhaps you don't filter it) ?

additional question, maybe related to the question on noise : what is the Y axis? number of queries I suppose but are they normalized?

Yeah they're normalized relative to a scale of 100, as it's Google trends data. In the future I want to include actual search volume numbers as a line on the chart too though. Just need to figure out where to get that data first!

I have scripts that are crawling the web in places like twitter, reddit and facebook to generate keyword ideas.

Right now, I'm filtering out the noise and crud manually! So it's a combination of automation and curation by hand.

This could have massive potential with music industry A&R's looking to capitalize on viral songs. That's where virtually all recording contract money is going nowadays.

Looks good, but I miss units on the axis. On mobile, it's missing from the start page. The units for the y-axis on the detail view is missing (as far as I can see).

https://trennd.co/trend/suicide-1

interesting: why spike? why Science category?

What is your Algo for calculating Trends? I mean how do you say that something is trending? It is most popular? Emerging? Z-score?

It's probably built on Google Trends

Are topics coming from twitter trending? Also, can you add filter of industry-wise topics? Good work btw :)

Curious to know how Your filters are defined (spike, potential, hyper) Thanks, i’ll follow and maybe use !

Right now spike/potential/hyper classifications are my subjective opinion.. which I hate because that varies depending on the day I'm having and my specific biases.

Figuring out how to classify them mathematically is next on the to-do list. For example a spike would have extreme outliers in the middle and hyper would be over a certain exponent.

It would be nice if it could discriminate between trends that are still in exponential growth from the ones that have passed the inflexion point.

Very nice.

Are these all public sources?

Have you considered backtesting it to see if you can get a trading signal out of it?

What's the time scale on the x axis? Are these trending over hours, days, months?

you can see timescale on top in a dropdown menu

The default is 5 years, but there is a dropdown select where you can choose from 3 months to 15 years.

Looks like it can be used to discover new form of investments

Web server down? Anyone else getting it to load?

Should be back up now! Apologies - had to upgrade my mongodb atlas cluster, it got hugged.

Very cool! Thank you for sharing -- and for creating it! Have you considered taking a step further in the "open" direction and making it OSS?

Thank you - appreciate it!!

I actually hadn't considered OSS, which part of it specifically would be most interesting to see?

The keyword generation part, for me

This is great!

This is really cool.

Great stuff. Thanks

Reply on news.ycombinator.com