uhh... yes it requires a lookup. There is no problem with that since the lookup is not centralized but local. How... read about terracotta. No I dont work for the company just use their open source product.
There is nothing quadratic here. Your are able to scale linearly with the number of messages received in the system. There is a fixed cost per message received which is exactly what you want. By the way, this is exactly how the internet mail system works and it is why it is so scalable.
I dont understand. I am saying that Terracotta, which is an open source product, handles the in memory table and synching stuff I was talking about and that you asked about. Its like you dont have to write a C compiler and you dont need to write linux because someone has done the heavy lifting for us. So as far as that specific issue my point was just that the solution is available in an excellent well tested product that lots of folks are using. By the way, I am actually working on exactly these sets of issues -- not to say there are not bigger minds than mine working on high velocity apps, but it is what I spend most of my time on.
This is true. Though I can say that if one were building twitter and you didnt either have a design like the one I suggested, or a design with some kind of distributed caching then you are going to have exactly the kinds of scaling problems they have had. In other words if you use tradition techniques for something like twitter you are screwed. However you are absolutely right that (as I said in my piece) they may already, for example have thought of all of this and either implemented or dismissed it because they have a better design, and I have no way of knowing.
true, but it makes me curious what brilliance someone bring to the table when they have nothing specific to say other than you are wrong. Also given that much of what I am talking about is running code, I at least know that it solves a major class of problems. If it doesnt apply to twitter it should be pretty easy to explain why.
Well ok. I dont have to call them database problems, but I object to someone saying they are not database problems. I agree it is a bit of semantics and I would not have brought it up if he hadnt. Although I have to say I consider this work (and the good questions you ask) to be in the database domain.
Regarding when shard split how you maintain the table, the easy answer is Terracotta. It is amazing for this and handles all that kind of stuff totally in the background. If you didnt have terracotta it would be a harder problem indeed. It handles locking and all of that magic for you. I highly recommend it. Regarding how you handle queue requests while the split is going on, what happens is that while the split is happening, the original shard continues to respond, but all new updates are sent to the old shard and the new one that is being created in the split, so during a split there is addition messaging going on. Once its done, the original shards range of users is cut back, but actually it is still storing all the old data. At that point you can delete the data that has been moved to the new shard - though this is not necessary.
Yes, I am defining database as "anything that stores data" It does not have to be SQL for me to consider it a database problem. And in fact my solution is well above the sql level.
Actually, this is the beauty of the design: All of the writes happen in parallel. So there if you write to 2000 recipients, but all of the writes happen in parallel you have no bottleneck. And any time a given machine's load gets too heavy, you do a shard split. You really dont care how much writing you do as long as on every machine you write to is not being hit too hard. So if a given server has lets say 10,000 users on it. And each one of those users receives 10 updates an hour. Then it would mean handling, lets say 100,000 writes in an hour, or 27 writes a second. If this was the threshold for this machine (which would be low) you then split the shard so that it then is only handling 13 or 14 writes per second.
I considered this, but given that the messages are small and the primary issue is scaling not storage this, it does not seem to me is a good solution. The entirety of everything ever written on twitter is probably not big enough to worry about storage size. The issue is performance. By the way, this architecture is essentially how email works. All your inbound messages are stored in one place separate from outbound messages from the sender. This allows for infinite horizontal scaling.
With all due respect andrew why dont you share your technical analysis of what exactly I am saying that is wrong here. Btw, have you ever actually worked on any database scaling problems? Just checking.
Uh... yes it is. All scaling problems are database problems. ALL of them. They can be solved by different strategies such as caching, distributed design, etc. But at the end of the day the problem is always that you cant get to the data fast enough. That is a database problem.
Its fascinating funny how you are so smart you can't even deign to share your wisdom on this. If I am wrong, tell me where. Given that I have spent the last several years studying/developing distributed databases if I just don't know what I am doing I'd certainly like someone as smart as you to tell me.
Actually, the two companies I mentioned - Kontiki and Red Swoosh have amazing QOS. Bit Torrent sucks for that, but it is an engineering issue, not an architectural or conceptual one. Akamai actually acquired red swoosh, and is integrating it into its platform as, I think, a defensive move. But clearly, the kings of the CDN world understand that a widely deployed well engineered p2p network will have an impact.
1. This is not about replacing torrents. There wouldn't be much economic impact in that since that stuff is already stolen and in any case torrents are not on any CDN network. The point is that sites like youtube, hulu, and any other flash based video services (almost everything) will be able to operate at radically lower costs.
2. DRM has nothing to do with this protocol. You can implement DRM if you want. Its a separate layer. And many of the key content providers will want protection over their streams.
3. I dont know why it is "Non-Commercial" and "GPL infected" I don't know what that means. I would suggest that torrents are non commercial since there are no businesses offering their content via torrents and making money. That is, I believe, the definition of "commercial".
4. The issue of browsers needing to be open is a real one, though that only effects the level of effectiveness (i.e. cost savings). Most services deliver the most popular content most frequently. So at any given time, someone who's browser is open likely has all of or a part of a file that you want.
Indeed even the best search engines are only going to be able to answer that question if it is stated somewhere. Hakia can do that. However no search engine is going to be able to figure out the answer by, for example adding up all the times he appeared in a given year. So if someone wrote somewhere "john mcain was a guest on the daily show seven times in 2008" hakia should be able to give you an answer.
Excellent piece. The idea of the expensive hobbies is actually something that I had not thought of and is certainly another very interesting vector for offering services people will pay for.
actually the scaling is absolutely an interesting problem. Its the reason they keep going down. because messages are not point to point it is a much harder problem than instant messaging. I have spent a fair amount of time considering the issue in detail because a friend of mine developed a caching system targeted at microblogging. It is non-trivial, and cannot be done with RSS feeds.
I dont think it says anything about things people pay for changing the world or even that payment means something will change the world. In fact it says "Or if its free, at least figuring out how to do some stuff that has the potential to really make a difference". In other words if you are not going to make money at least do something important. Clearly linux would qualify.
actually no I am not ignorning his point. Its just not statistically meaningful. The truth is that of the people that click on ads, they are midwestern housewives. But there are statistically not enough of them to make many services that rely on them profitable. TV and print are broadly successful at attaining users attention. The web is not - at least for display ads.
actually, as the commenter below indicated, according to jackob nielsen's research, I am absolutely right.
http://www.useit.com/alertbox/banner-blindness.html
Summary:
Users rarely look at display advertisements on websites. Of the four design elements that do attract a few ad fixations, one is unethical and reduces the value of advertising networks.
The unethical here is tricking people into thinking popups are system dialog boxes.
The economics of the situation are clear. 100,000 magazine readers is a business. 100,000 web readers is lunch money.
Life is about more than just business. And being bold isn't, at least in my context, about working harder. In fact it sounds like, for you, work is a default. I am talking about exploring things in life that you might not otherwise explore. Doing things you might otherwise dismiss. They do not need to be big things. Its not about who can have a heart attack sooner.
Yes, but the question is whether any YC companies have the capacity to be the next google. I think it is a good question because it really is saying "if you can't find the gems why expect VCs to be able to". Its a people in glass houses throwing stones argument. There may be good counter arguments but it is a very legit question.
Well, as I said, there is nothing that cant be expressed in an RDBMS - at least at first. But let me give an example of the kind of use cases we see.
First, imagine having a database that allows one to freely create record types. One might have standard data types like contacts, events, emails, checks, expense reports, etc.
These record types are nodes. Now imagine being able to connect these nodes using any type of edge you like. For example a contact might be connected to an event as an "invitee". Thats how the edge would be labeled. Now the relational folks will say that that is a relationship that could be predicted. But at some point, some new type of record is created. And you as a user want to connect that record to existing records. For example you have added a "shoe" record type to keep track of all of your shoes. You then decide you want shoes to be connected to events so that you can map what shoes you wore to what events. You don't want to modify your schema. You don't want to add a new mapping table, you just want to connect the record. And you want to be able to query the graph for all the things of any type that are connected to that record. More importantly, you want the end user to be able to decide that it would be useful to connect shoes to events since no self respecting programmer is ever going to design such a system.
This is the type of flexibility that you need in a web application that will evolve over time. But the minute you want to connect that new record type to the existing object, you either have to modify your schema, or you have created a database that is highly flexible via totally generalized mapping tables, but is not optimized for these kinds of structures. For example just creating a giant mapping table to connect objects will work in an RDBMS but it is not at all optimized and will fall over at scale. Since we are building something that will handle awesome scale, using an RDBMS in this way was a non-starter. Philosophically, we probably have more in common with Google BigTable than with an RDBMS.
I am the author of this article, and I must say, though I am not a regular reader of HN, I do find it refreshing that there are smart people here arguing real merits in an intelligent respectful way. I am sure it is not always the case, but it is really fun to read everyones perspective.
I dont think graph databases are for everything, but I do think that they will end up providing a much better abstraction for the kinds of apps we tend to write on the web. I do think an RDBMS is better for an accounting system for example. Oh, and my examples were not designed to actually be great real world examples, but I have a lot of less technical people reading my blog and so my goal was to provide examples that could be expressed succinctly. That said, there is no static example that cannot be expressed in a relational database. The problem is that relational databases (at least the ones that are available to us) are not at all fluid and flexible.
The team at OSAF was really quite good. They werent seasoned open source developers, they were serious developers. Some were world class. But it was hamstrung by a series of bad decisions and some bad management. I also think having infinite money is a horrible thing for encouraging delivery.