Arimbr

Good feedback. Indeed, it is not clear on the UI. The model concatenates the two, so not sure if changing the label "Message" to "Text" to mimic Reddit UI for text posts would be enough... Or rather just have a single "Text" textarea as for the model is the same.

Weird! I just tried "best time tracking app for iOS?" and got "iOSProgramming | 45.0 % swift | 18.0% ProtonMail | 14.0% ios | 4.0% apple | 3.0% iphone | 3.0% FlutterDev | 2.0% freebies | 1.0% jailbreak_ | 1.0% jailbreak | 1.0%"

For the second case: "I'm looking for a time tracking app. Any recommendations?" and got "GearVR | 43.0% TTVreborn | 14.0% WearOS | 4.0% OculusGo | 3.0% IPTV | 3.0% ApksApps | 2.0% animepiracy | 2.0% RabbitReddit | 2.0% NetflixViaVPN | 1.0% androidapps | 1.0%"

Which is a bit better! But still not perfect ;)

Thanks so much for your detailed feedback. There is definitely something going on with the UK thing... Need to dig deeper!

You are right!

Thanks man! Looking forward to working on V2 :)

Thanks man! Right, unfortunately it was not trained on r/SiberianCats. It was trained on what Reddit said are the most popular 4k subreddits.

To make it more useful, I need to collect more data from less popular subreddits :)

lol

Unfortunately, I trained only on what Reddit told me are the most popular 4k subreddits :) It was not trained on r/DevelEire :/

So, I think i need to train it on more subreddits to make it more useful. Thanks for sharing!

Cool man! Thanks for sharing :) I wasn't familiar with the chunking approach. I will read more!

Regarding BERT, it indeed may perform better if fine tuned correctly. For a baseline fastText is great because it is super fast and runs on a CPU. It costed me 24$ to run a 24h autotune on a 16 CPU core machine. Also, fastText is great out of the box as it also builds word vectors for subwords, which helps with typos and specific terms that may otherwise be out of vocabulary.

I am betting that fine tuning BERT will cost me at least x10 more. But I this project is a chance to try it out :) Looking forward to v2!

Luckily, with Valohai, I get access to GPU credits for open source projects!

thanks! That helps a lot, although I am not familiar with that area of knowledge.

Indeed, I got some ML metrics on a test split that gives me an idea of its accuracy :) But it's just an estimation, so indeed I am looking out for feedback to know its real performance so I can debug bad cases and fix those with more data or a better model.

The test performance on subreddit r/hearthstone is 0.21 f1-score, which is not great. And looking at the confusion matrix for r/hearthstone it gets often confused with:

r/BobsTavern r/CompetitiveHS r/customhearthstone r/Blizzard

If you are curious, I uploaded the metrics (precision, recall, f1-score) and confusion matrix on the test dataset on a Google Spreadsheet.

https://docs.google.com/spreadsheets/d/1NBY1o85ZiNpcm4tcYhKk...

The sheet 'confusion_matrix_gt2' can be used to find similar subreddits.

Great idea! Will add that :)

Right! I used the official Reddit API. I created an APP, got the credentials for the API. Then used the Python library PRAW to consume the API. https://praw.readthedocs.io/en/latest/

It took me 36 hours to collect the 4M posts. Reddit API returns results in batches of 100 results, and then sleeps for 2 seconds.

You can find some more details on how it was built here https://blog.valohai.com/machine-learning-pipeline-classifyi...

I can publish on Github the repository that runs two commands to collect the data if your are interested.

;)

Interesting! I didn't know about that subreddit. Which text did you try? :)