Comment by mittermayr | Hacker News Reader

Comment by mittermayr | Hacker News Reader

Parent Full thread

Great, great points. Here are my responses: 1) Yes, that seems to be the only way out right now. But my core features (most popular followers / most valuable followers) depend on a full set and will not be accurate. Growth data works fine though. The main reason why my service got traction were the most popular followers / most valuable followers metrics though. So that sucks. But seems to be my only resort right now.

2) I can get IDs faster, and then theoretically use my own DB to see if I need details for that ID (i.e. follower details) from Twitter or have them stored. Problem is: I am duplicating Twitter's database, also, I had this before and the database grew to immense size and kept crashing all the time despite efforts to avoid it. Also: Celebrities have a large distribution of followers, so unlikely to save much time by seeing who's details I already got elsewhere.

3. Yes, although Twitter announced their quadrants recently, of services that will be supported by them, one of them is social analytics. This is what I do. So that should be good. But you have a point, I can't touch any money for up to 1 year in case Twitter shuts it down, I want to pay back the remaining unused time to my users. So it's frozen money.

Focusing on 1): if "most popular follower" means "follower with the most followers", you could try one of three strategies.

First, just check which of the the top 100 (1000) most-followed users follow the celebrity - this will likely eliminate the heaviest hitters immediately.

For accounts with normal amount of followers, do as you already do.

For the in-betweeners,

    Pick your celebrity.
    For a random 1% (.1%) of his/her followers:
        Retrieve list of people followed by this follower
        For each user followed by a follower of the celebrity:
            Increase number of people following this user by 1
    For each user in the top 1% (10%, .1%) of the above table of users:
        Retrieve number of followers
    Report user with highest number of followers from the above

This is, of course, based on the idea that often-followed users who follow a celebrity will also have many followers among the followers of the celebrity. Results will become more accurate as you poll more followers, of course.

(I don't use Twitter or their API, and the above may be completely wrong.)

see, this is what I came for. I believe there's room for optimization within the limits I have to adhere to right now. your approach seems like a stab at it and I'll think this through again in a bit if it makes sense. but you're hitting the nail on the head in terms of where my problem is, i can't change what twitter does, i can display partial results but it's suboptimal in many ways, so how can i make these partial results the best possible quality? this is where you answer seems to make sense - thanks. i'll think about this one for sure.

Digging further in 3), if your app is proven to be in the 'good' quadrant, have you ever considered contacting Twitter directly and requesting a higher API limit rate.

I can imagine they are ok with that if your app provides value to their ecosystem.

well i constantly mention the issue through their bug reports and discussions, they acknowledge it and promised to talk to the team about re-evaluating the way things are handled now, but I'd assume this means it's not a priority.

i tried signing up through one of their partner thing forms but as expected, no response.

With the right infrastructure and tech, the "mirror" DB shouldn't crash. And also keep in mind how many people follow a lot of celebrities… common point!

i agree, it totally was my failure to comprehend database architecture to a point where I was able to keep it running reliably.

but as said in other comments, i'm not sure twitter is happy with someone duplicating their entire user database over time.

Reply on news.ycombinator.com