Show HN: Search your Twitter history with Algolia | Hacker News Reader

Show HN: Search your Twitter history with Algolia

github.com

171 pointstransitivebs38 comments

The idea for this product came last night from Guillermo Rauch tweeting that he wished there was a service to search your personal Twitter history (tweets, likes, retweets, filters, etc) using an Algolia-type search UI (super fast search as you type). A whole bunch of people echoed that they'd pay for this type of product.

So I decided to build it overnight and open source it.

The majority of the functionality is handled by Algolia and Saasify which is the only reason this was possible to build so quickly.

Would love to hear people's thoughts - thanks!

Hey your UI is Pretty cool what front end framework did you use to build it

The base webapp template uses https://saasify.sh (which uses CRA and antd under the hood).

The core functionality when logged in is written in React using Chakra UI.

What does Saasify do? I browsed their website (https://www.saasify.net) and don't quite get it - how did they help you?

They are using https://saasify.sh/.

Twitter only stores your last 3200 likes.

What a bummer. I like a lot of very funny or informative stuff on twitter and the tweets often pop into my head even years later. Sometimes I wanna find that tweet (eg to share) and I can’t.

I started using pinboard.in, which archives all my likes, but the search suuuucks.

So, I started building my own archive tool, but the search part is still hard.

I have looked into annoy, perkeep, tf-idf, Doc2Vec, and various other tools, but nothing really works as well as I had hoped.

Your project definitely gave me some ideas! Thanks.

Wow, I did not know that. This is such a no brainer paid feature for a premium Twitter plan: storage & search.

I suspect they probably keep all likes, but do not expose them.

Another premium service that searches your Twitter like is https://www.favourites.io/, but it's too costly for me. I might pay $10 for a yearly subscription, but not 29 GBP/year, which is ~10% of my monthly rent.

If they do keep all the data, you should be able to get it via GDPR

Yeah, you can access all the data of your own account, but third-party apps accessing your data via OAuth is another issue.

In the case of Twitter's API, third-party apps can access all of your history if you upgrade to a paid partnership plan.

> I started using pinboard.in, which archives all my likes, but the search suuuucks.

What's also annoying is that I don't think there's a way to exclude from_twitter bookmarks in Pinboard when you search. So now that I enabled this feature a while ago everything is cluttered with tweets and Pinboard become useless for me.

Something like a -from_twitter search would be helpful but there's only a "+" search right now.

You can access all your favorites (and tweets and DMs) by downloading your twitter archive.

You can use Twitter Archive Eraser to browse the whole archive, search by keywords, regular expressions, mentions, tweet type, media etc.

Twitter Archive Eraser allows you to bulk delete thousands of tweets at once also.

https://martani.github.io/Twitter-Archive-Eraser/

My https://github.com/dogsheep/twitter-to-sqlite tool has a solution for that: you can export your entire Twitter account as a zip, load more than 3,200 liked tweet IDs from that archive, then backfull their full inflate versions by fetching them from the API.

Documentation for doing that here: https://github.com/dogsheep/twitter-to-sqlite#importing-data...

You should put a license in that repo. Witbout a license, the default copyright laws apply, meaning that you retain all rights to your source code and no one may reproduce, distribute, or create derivative works from your work. If you're creating an open source project, it is strongly encouraged to include an open source license.

The readme clearly states the MIT license at the bottom. Are you referring to adding a separate license.md file?

Just added a license file.

Thanks!

Seems this was posted primarily to market Saasify, so I'll take the bait: pretty cool that this was set up so fast. You've built a neat thing, Travis :)

Hahaha you got me :) Thanks, Steven!

But hey, Saasify looks neat!

I'm interested for sure

I'm very happy to see a tool like this that can be self-hosted.

Back around 2010 there was a company called Topsy[0] who was supposed to be the "Google of Social Media". It would index Twitter tweets and Google+ posts. The idea was you could read discussions over the years and find trends that could be used for research, analytics or marketing.

Sadly, it was bought up by Apple who shut it down a year or so later. There is still a giant hole left around search for Social Media, with each of them being their own walled garden.

[0] https://en.wikipedia.org/wiki/Topsy_Labs

Gnip was also working in a similar space before being acquired by Twitter in 2014.

https://en.wikipedia.org/wiki/Gnip

I use the standard twitter search a lot to find some of my own tweets. Mostly a "from:@h43z term" does the job for me. Though it's not instant twitter provides many useful filters you may want to check out (https://twitter.com/h43z/status/1234614509400215554)

Will definitely look into adding support for some of these common filters.

Thanks for the heads up!

Nice job. Had a similar idea after seeing the same Tweet, but based around importing the downloaded data archive, as I’m working with that myself now. Open source, so I might see if I can weave it in when I get a chance, if nobody else does it first.

Yeah, I'm definitely down to collaborate -- feel free to reach out via email and I'm happy to share any profits if you add features to the open source repo.

Does anyone know a tool to download all tweets from an account in JSON format? No need to include likes or RTs. In fct, better if I can avoid those.

You could make a tool pretty easily using Twitter's api:

Get started here: https://developer.twitter.com/en/docs/tweets/curate-a-collec...

Unfortunately, Twitter limits the number of historical tweets they allow access to from their API (3200 iirc). You have to pay for a partner account in order to have full API access.

If it's your own account the API will let you fetch everything.

If it's someone else's account the API limits you to around 3,200 tweets.

I built https://github.com/dogsheep/twitter-to-sqlite tool for archiving my own tweets. I've used it to pull tweets from other interesting accounts too.

Can I use this to search other peoples tweet history?

This is a feature that we'll be adding shortly.

This is a neat feature! Kudos How about allowing to add other's twitter handle and searching their TL?

Yeah, I really like this idea.

Given that the initial version was built in 8 hours, I'm sure there are a lot of features that I'll be adding over the next few weeks :)

I'm thinking this would make sense as a paid feature?

If you want to monitor keywords on Twitter and not just in your timeline, you might be interested to check the iOS app Clatters : https://apps.apple.com/us/app/clatters/id1480930237 Full disclosure : I'm the developer

That being said, I am really impressed by this new service. Creating such a solution in just a couple of hours/days is astonishing.

Thanks, Timac.

You can read more about how I built Twitter Instant Search here: https://www.indiehackers.com/post/launched-a-solid-saas-in-8...

Strange to think that building an actual web spider to make an independent index of public tweets is probably illegal these days.

This is basically a frontend to Twitter’s own datastore via the API.

Yeah; it's just strange that this sort of thing is necessary given the resources that Twitter has available.

from:my_username my query

Twitter dot com slash search

Reply on news.ycombinator.com