Why I can't have conversations using Twitter
antirez.com
antirez.com
If you do get roped into a conversation where you need more tweets to explain, here's a few simple guidelines:
- Don't split sentences over tweets. People will only read one part.
- If you really must, use "(cont.)" on both tweets (end of the first and start of the second), so it's obvious there was another part. I know that reduces your character limit, but it's worth it to be less ambiguous
- Make each continued tweet a reply to its predecessor. You can reply to your own tweets. It facilitates following the conversation on most clients.
- If it's more than 2 tweets, consider using twitlonger. In general, that should be a last resort, though.
Without trying to make it sound high-handed, there's something of an art to smooth twitter conversations. Terseness is the key. I find them rewarding because, when done right, they have an enormous density of information in a short time.
In any case, using some form of video calling is too intrusive. I can type this reply, do something else for 10 minutes, then check back. But then I was always one for text mediums rather than video or audio calls :)
Look! Even that sentence was longer than a tweet! I don't know how people do it.
(That's exactly 140 characters for you :) )
— Blaise Pascal (sometimes wrongly attributed to Mark Twain)
To me, this is exactly the kind of nuance that is lost by training yourself to limit ideas to 140 characters.
And for the record: I don't train myself to limit my thoughts, I train myself to use a form for expressing them appropriate to the medium I'm doing it in. Expressing something shorter and in steps is not inferior to long-winded explanations. If you must blame something for a fine nuance I failed to grasp, blame the fact that English is not my first language. I doubt I would have caught it in Dutch either, but at least that could be a valid reason.
e.g. blunt, rough.
Unfortunately for you, both connotation and denotation were working against you.
Particularly in technical writing, you've got to get your words nailed down correctly. This is why there are things like the RFC defining what 'shall', 'should', and 'may' mean (intended for other RFCs, but it's also used elsewhere)[1], or the development of Simplified Technical English[2]. The latter was developed in aerospace engineering, where minor misunderstandings can literally cost lives.
Subtle differences between words are important, and they allow us to communicate more complex ideas. Twitter may have it's place, but any conversation involving nuance really isn't suited for it. A high-quality conversation includes caveats and careful framing, which can't really be fit into 140 characters. At least, in my opinion.
[1]https://www.ietf.org/rfc/rfc2119.txt [2]http://en.wikipedia.org/wiki/Simplified_Technical_English
Hmmm. I'm not sure how to respond to the suggestion that people who can't fully express their thoughts in around 300 characters are thinking too far ... or the subtly implied pejorative of 'too-far-thinking'.
Aside: I note you used 1386 characters to describe the benefits of terseness.
And he specifically chose a medium which was not twitter to describe it.
Aside: this conversation would have taken a very different form if we had had it on twitter. The medium defines the form. So I don't mind being verbose here :)
The idea of iteration through (a) thought I think has been explained to me before, but I must have suppressed it. I can see why it's necessary within the context and medium of twitter ... however I'm unconvinced (duh :) that it's a useful constraint to apply to a communication medium. I don't expect I'm unique in voicing this opinion.
The idea of developing a thought with another party, or as you also describe it, 'collective thinking', is genuinely anathema to me. I'm happy to have a discussion that involves me adjusting my thinking, and, of course, a discussion that results in the other party(/ies) adjusting their thinking. Actually, I expect that essence of conciliation and compromise is what you're describing. However, the idea of avoiding 'long term debate' really does unnerve me.
Perhaps I've been debating for too many years with people that I really wouldn't want to share a collective thought with.
The advantage of developing a thought in tandem with somebody is that the compromise happens more naturally. You rarely need to 'concede' a point or prove somebody wrong, since you can influence the way the point is made in the first place, pointing out flaws in their reasoning or facts as they happen. Conversely, the other can do that for your reasoning. The result is that, rather than adjust your thinking in a few big steps, you adjust it in minor ways with every sentence. It takes time and practice to get used to this, but it can be very productive and enlightening.
On the other hand, I would never avoid long-form debate either. Both have advantages and disadvantages, just as both have mediums where they work and mediums where they don't. Twitter isn't conducive to classical debate, just as hackernews isn't conducive to the iterative thought development style.
It's true that it's easy for people to interject and annoy, and it happens more and more the bigger your celebrity status. However, unlike the interview example, it's also easy to ignore them. Twitter's reply helped a lot for that. Replying to anybody is optional, you don't even have to read their tweets if you don't want to. It's not perfect by any means, I don't think any platform will ever be. But it isn't bad.
The only portion of OPs comment I can find to "describe the benefits of terseness" would be the last paragraph. If I shorten that paragraph to the last two sentences, it fits in a tweet.
If X or Y --> means that both are possible, right? If you then suggest that it's always and only X, then the or construct is misleading. I was picking up on the fact that there will be some Y, and that worried me.
But, to be absolutely clear, you are completely correct in saying that in some cases X (or inappropriate medium selection) is going to apply. (I wonder whether people who need more than 2 tweets to express their thoughts are capable of self-identifying which category they fit into. If not them, then who?)
Going beyond that I'd suggest that twitter is an appropriate communication medium far less frequently than people evidently believe it is.
I don't believe shortening things to fit into tweets is a healthy aspiration. I ran a character count solely to demonstrate how many tweets would have been required to get that entire message across, and most of chton's message was a guide on how to communicate in a medium that enforces brevity. An undeniably useful guide if you happen to be embracing that medium, mind.
~125 characters, including this and "125 characters"
- Write a post explaining your views, and tweet the link to it.
Very true. It's like blaming a radio for not having pictures to go with an advertisement. Wrong tool. Echo chambers aren't good for in depth conversations.
There is the same problem on Google+ where there is not 140 characters limit. Any discussions with Linus or any other "celebrity" in it and the comments thread will be full of : "+1", "M. Linus you are so smart, you are my hero", "Well said, ripe him a new one" etc... Which drown any constructed argument and render any constructed debate impossible.
Most of the current discussion platforms are way behind what was standard in Usenet or in your average Usenet client.
No threaded discussions, no filtering (not for the one who comments and doesn't want to read those +1 post and most often also not for the OP), not even blocking of people you don't want to read.
Mailing lists can be ok with proper client but it's a step back. I found some subreddits useful (with RES) but even then it's hard to track new replies in threads that you are interested in.
HN has quality discussions but these are just about some links and it's hard to get back to some thread to see new replies (and preferably prioritize those from people who I value).
I too miss Usenet.
But this doesn't solve the issue of being notified of replies to subthreads. Reddit threads up to 100 messages tops are manageable that way but anything larger isn't.
Compare that to usenet threads were you could easily handle thousands of messages in one threads, this is really a shame.
But one design element of twitter might make it easier to stumble into the realm of people who will wildly disagree with a poster's opinion.
Just replying to some tweet you can not know what kind of domain you are entering. With usenet groups the name has always been a strong indicator what people you will encounter.
If you post "Ruby sucks and is dead slow, Perl rulez!!1eleven" into comp.lang.ruby, you shouldn't be surprised if you get flamed. With twitter accounts, this is not that obvious.
Of course, this is a core design element of twitter that it is concerned with accounts and not thematical groups and both have their advantages and disadvantages but I think this is one of the reason that you can much more easily and unintentionally anger the wrong people on twitter.
"You have clearly not understood how the math works or why tail latencies matter in dist sys. I think we're done here."
Was the bluntness really necessary?
Amusingly this has now happened in another thread in these comments.
The meaning of a post has been overshadowed by a conversation on how the author used word A when they really should have used word B, and then they were berated for trying to explain why they thought word A was acceptable in the context.
https://twitter.com/jmhodges/status/527189222546227201
He's saying he's quoted out of context now: https://twitter.com/jmhodges/status/527480854314905600
Reading the full conversation I don't know how that's the case.
Anyway, it doesn't change the point antirez is making on his post.
I rarely see this used in situations that aren't passive aggressive attempts to get followers to pile onto someone you disagree with. Totally ruins any chance of good debate.
(Whoops, accidentally put a dot in front of your name! So sorry.)
And the longer it is, the more likely I am to be scanning quickly, actually. On the other hand, shorter messages and less context result in things simply being omitted, even if you do read carefully.
Maybe Twitter is worse than average, but it's pretty bad with any medium.
Long ago I learned that if you want to have "viral" success, even on a more informed site like HN, avoid making more than one statement or claim in a piece.
"Modern Java application servers offer surprisingly high performance" (followed by details of performance improvements, etc), for instance, is okay: everyone who agrees will forward and agree and post, while those who might not agree or care will remain ambivalent and disengaged.
Change that up slightly, though, and conversationally add "...which might allow you to deploy on fewer servers than a similar solution built in Ruby" and you'll instantly gain armies of very motivated detractors who will go forth to denounce everything you've ever said, declare everything you've written stupid and misinformed, etc. Not based on objective disagreements, but based on a knee-jerk opposition that, much as with politics, can't be on a point or issue (e.g. "agree to disagree"), but has to encompass everything.
People who might agree or yield value skip the piece after seeing endless criticisms and complaints.
The crux being that it's very hard to gain agreement, easy to gain enemies. If you state ten things, people who disagree with but one of them will often fall in the latter camp.
Not that this changes how I personally write -- where I'll load pieces with things that people can disagree with -- but I see it constantly as stories tread the bottom of the front page, the detractors who find that one tiny thing they disagree with and blow it up to be all-encompassing, and it gets flagged to oblivion.
I keep hearing how it's great to keep up to date with whatever you're interested in - but I just can't see it.
(Of course I also have high-volume sites in my feed reader, but that's more for "news" / skimming though, not for keeping in touch.)
EDIT: Both, Twitter and RSS, have their own terms and concepts which need some kind of introduction for most, especially non-tech people. For Twitter its hashtag, TL, DM, etc for RSS its stuff like Feed, Reader, etc
RSS and Atom allow for people to tell the people on the other islands about what they wrote but it's a one-way message. For an interaction to occur someone has to go over to the private island.
With blogs the entire process of publishing, reading and responding is privatized and fragmented.
Twitter is a centralized location that unites publishing, reading and interaction in one faux-public location. There is utility in these kinds of social networks. The issue is that they are not public places.
Twitter controls how the information is written and read. Twitter optimizes for their own private interests that have nothing to do with intelligent discourse. In the early days, Twitter was an open platform with a large number of competing clients. Early users and developers invented the core interactions that we see now see on Twitter. Hashtags and retweets were not invented by Twitter.
Twitter has since put their API on lock down. They alone have the right and ability to make clients. As a publicly traded company they're now required by law to create products that increase shareholder profits. The evolution of the platform is out of the hands of anyone but the managers of a private company.
I hope it doesn't come as a surprise that intelligent public discourse is not as profitable as the cheap and simple thrills that seem to be the core of social interactions on the Internet these days.
Networking, sharing links, tidbits of insight, short messages, and Q&As all work really well. I've learned quite a bit from the curated articles and reports tweeted by those I follow. And if you follow the right people and engage appropriately, Twitter can open up a lot of opportunities. But anything beyond that is a waste of time, and can border damaging.
In fact, for all the good Twitter has facilitated (protests, revolutions, etc), I think it's having some negative impact. It is by far the most popular platform where individuals of all backgrounds, beliefs, and opinions congregate and interact, but it's built in a way where it is absolutely not conducive to meaningful debate, and so when the huge opinionated masses clash, the bite-sized arguments volleyed by either side are taken without context and nuance, further igniting and polarizing people. I'm seeing this happen with issues like feminism and Islam, where someone will tweet a very distilled version of a larger and more thorough opinion, it will be taken at face-value, someone will retweet it with a snarky comment, it snowballs into a food fight with enraged people retweeting/replying, and the original tweeter trying to add context but not being able to keep up with the reactionary domino effect. And because Twitter has almost become the official sounding board for many people, their tweets and reactions to those tweets contribute to their public image, reputation, and online presence, all damaged by death threats, accusations of sexism, bigotry, racism, etc, often thrown around unwarranted by people taking things out of context and who feel antagonized or polarized because of the way Twitter is structured.
And that's why I avoid tweeting about religion, politics, and databases.
Personally I get that they've lived through some pretty gruesome use cases and wars, but jesus, glass houses and stones, guys.
> "You have clearly not understood how the math works or why tail latencies matter in dist sys. I think we're done here."
If, when you write something, it sounds like it's coming straight out of The Simpsons' "Comic Book Guy", you should probably rethink it unless you really mean to be a bit of a jerk.
In a strictly time-ordered tweet view, you'd see one or another of antirez's tweets, be be curious, and go back into his & other respondents timelines to see the full conversation, w/ whatever context was originally there.
But how Twitter works right now is you get what appears to be a full conversation neatly tied together, but it's actually been selectively edited by an algorithm maximizing (I'm educated guessing) engagement, which ends up basically being an automated filter to create misunderstandings and controversy.
Now if I can just kick my HN habit..
Good tweeting requires almost Orwellian control over writing.
It needs to be extremely terse and explicit.
You need to be the kind of person that will write a tweet, read it and if it could be misunderstood in any way delete it, before rewriting it several times.
You need to do this until you are able to fit each inalienable component fact into 140 characters.
This kind of writer is often careful to form only one idea per tweet, considers using mathematical notation if it requires less characters, and will finally consider placing a [...] on the end of sentences which leave their argument hanging. Though when you have to do a lot of these things, it's often because you have not fully analysed the concept to a point where you understand it well enough to say simply.
Additionally, I make sure each tweet is a reply to its predecessor. Here's an example: https://twitter.com/epaga/status/510316379833393152
I mean I understand that's how twitter works and probably part of what made it successful, but it baffles me that it's seemingly becoming a "serious" way of communications. Politicians are expected to have a twitter account, serious news organisations often quote twits directly in the body of their articles etc...
I don't understand why so many people chose to de-facto standardize on such a poor medium for communication, especially since you have a billion alternatives without such limitations.
However, I do get what you're trying to do re: identify whether the problem is related to outliers. But in general, comparing to median has a lot more validity. Median isn't as sensitive to skew, and will be closer to the peak. For your purposes it probably wouldn't be a lot different but you wouldn't have pushed the "it's wrong" button.
All said, though, even comparing 50th (median) to 99th is pretty coarse. I'd probably be looking somewhere closer to 75th percentile for a comparison. Basically, you'd want to guess what percent might reasonably be affected by performance spikes and compare from there.
I don't want to use Postgres. I want to use Redis. Postgres IS NOT the solution for a lot of things. Yeah, feel free to call me stupid for not knowing every nook and cranny of PGSQL. I don't care
"I love to argue, but this is just a futile exercise."
Very wise words
Too bad 99% of Redis users use it as a glorified cache, because it's more than that.
But for most of what I do - smallish bootstrapped things - I want to build on Postgres, because I know I can take it a lot of different directions, as needs be. Unless I'm 100% sure that it's a bad fit for SQL and a great fit for Redis or something else, I'm going to consider Redis as a later optimization, not something to build on from the start.
Like you write though, I mostly view Postgres and Redis as Apples and Oranges that could probably work together quite well in the right circumstances. Most of my ire is directed towards Mysql :-)
I'm trying for the life of me to think of queries I could make that are non-CRUD. Maybe I am just stuck in my current tools to think flexibly about it.
However the DB2 database had materialized views that were crucial to the business.
Porting those over was non-trivial. Oracle returned a different value for null fields than DB2 (empty strings for null strings vs actual null in one case) did so various portions of the code would fail seemingly randomly.
Some of the constraints didn't map quite as cleanly as the others.
And we haven't even started on the performance tuning and how your schema affects them.
Sure you may be able to port all or most of your schema and queries over to another database but you'll have a lot of work to do after that verifying that nothing is broken and your performance is still acceptable.
SQL is not even close to a guarantee of portability.
Trying to load more than one record from a a single row is supported in some, but not all ORMs as well.
Aggregate queries, especially more advanced ones than just min, max, sum, avg, stddev, &c, are not always supported by ORMs either, or you need to use expressions or extend parts of the ORM. It's just a mess sometimes and I'd rather query and then load.
That said, PostgreSQL is pretty fantastic software too.
Well, for a lot more than Redis.
You are correct, but nobody chooses databases by count-of-good-use-cases. You choose databases by applicability against the use cases you have in front of you and are likely to have in front of you.
AFAIK, the arguing with Antirez with Redis has been going on for weeks now (and it hasn't been necessarily been unfounded). The exact issue the Stripe guys had, had to do with a replicated Redis setup.
EDIT: actually I am talking about the second problem mentioned in their post here: https://stripe.com/blog/game-day-exercises-at-stripe (latency spikes). The first problem (data loss) is related to replication.
It might be a good news headline spreading medium but not for conversations.
Seriously, I don't even try to have arguments anymore. A sane and relatively productive one is nearly impossible with some people. I don't know if it's the brevity of speech or the even shorter attention span, but the Online Disinhibition Effect seems to be an order of magnitude greater when it comes to certain topics.
I absolutely think that you should only use twitter to post links to blog posts and mailing lists emails. That way, people who really care will read, others will not, and they likely won't comment without reading.
Complex thoughts don't often fit in a tweet.
My response was, if you want to develop products together, let's do that. Second, Twitter probably has 50 product people who have all sat around and discussed her solution in detail and come to the conclusion that, for some reason we won't understand but that is probably better for Twitter, they shouldn't implement it. Most likely it has something to do with more views.
Then I welcomed her to the shitty reality that once a company gets the market share, they stop caring about a better experience for users and more about the bottom line OR maybe they're just stupid. After all, Twitter didn't have a single backup for something like the first year or two of existence.
As for their engineering problems at the beginning yes it was bad but then again they grew massively very quickly and they needed to come up with novel solutions at huge scale.
Things like the location of rockets associated with sarin attacks in Syria[0] placed squarely on Mezzeh military airbase in Damascus. The rockets at the time were thought to be Iranian Falaq but more evidence showed them to be part of their own distinct program.
The discussion did not stay entirely on Twitter but was the main venue for it. Without being able to rope millions of random peers you may not know you will converse with later into one IRC channel, Twitter works for wide broadcast fast paced discussion.
Blocking people helps sure, but I think many people's problem seems to be a perceived lack of interesting peers.
It's a genius example of design thinking.
It is a genius example of design thinking, I'll grant you, but twitter's priority as a business was to build a database for analysis and that was the design constraint which resulted in a 140 character limit to messages.
It is a communication platform, but only in as much as it fulfills its main purpose of generating datasets.
This is the first time I hear about this. Do you have any source or did you make it up yourself?
Twitter's original function was group SMS messaging among friends, the 140 char limit was the 160 character SMS limit + an 18 character username.
Google+ = Conversations
Averages tend to be a notoriously bad indicator on their own because they don't describe anything about the distribution. This is obviously not a normal distribution, and even if it was, you need to report some kind of variance measurement as well. Otherwise it is hard to say anything about the shape of the distribution, let alone make tests for statistical significance.
This was actually a continuation of a conversation that has been going on for several days, starting with an issue even more serious than 99th percentile latency - data loss due to a broken replica-repair strategy. Some very clueful people have pointed antirez at useful literature on the topic, and made specific suggestions to avoid the problem, only to be met with silence or excuses for why throwing away data was actually OK. Is it any wonder that they're frustrated with him, and ready to interpret an ambiguous statement like "the 99% percentile is bad" in a more negative way than he intended? For him to cherry-pick one exchange and present only one side only continues his own pattern of making constructive conversation almost impossible.
It's not a Twitter problem. It's a people problem.
Everyone likes being right + feeling important. Everyone is lazy. No one wants to be called out on it.
If you are forced to keep it <140 char then no one can fault you for being lazy with your replies and critiques. Of course people are going to jump in and offer their brilliant opinions. Think about the low-cost high they can get.
When I say something is a "Twitter problem," I mean that these patterns are less prevalent on other platforms. I mean that Twitter has created a scaffold for communication that encourages certain bad behaviors.
One example: If I were to pluck the middle sentence from your response and criticize you for saying that "Everyone is lazy" here on HN, I'd be down voted into oblivion because I am obviously being a jackass. Twitter's structure can make it very hard to see when someone is being misquoted or their views misrepresented. It's the Fox News Soundbite version of online discussions.
The problem is desire for control of the message. People who want that sort of control should just issue press releases. People who try to use the Twitter megaphone to promote their ideas, their projects, or themselves have to understand that others are doing exactly the same thing and sometimes the messages will conflict. The community into which antirez dropped this particular comment is one full of people running their own data-storage projects, academics promoting their own ideas, and others with more abstract (but no less passionate) beliefs about things like data protection or 99th percentile latency. I'm part of that community, and I've certainly had to endure pot shots against me or my project because of my presence on Twitter. It's part of the territory - just as it is on sites like this, or has been since forever on Usenet and BBSes and all the way back to the first town square. Among a thousand competing voices, yours might not be heard or understood perfectly.
That's the thing: Redis has no read repair strategy. Redis has a "be an exact copy of your master" strategy. It's not a secret. It's the exact design.
The complaints are like yelling at Linus when you rm -rf / your entire machine. Sure, it sucks, but it's a repercussion if your own actions, not a fault in the system. If you don't want to rm -rf / your machine, go use an OS designed for babies (cough ubuntu cough).
(Plus, there are already Redis improvements (designed within a day or two of the original problem being reported) to provide workarounds to users who _do_ want to run that exact use case. The answer to problems is solutions—not complaining and blaming endlessly.)
"the 99% percentile is bad" in a more negative way than he intended?
Text. It's only text. You can't read the intonation and people want to read absolutes. People want to read anger. Always anger. Always confrontation. It's possible the author of the text didn't mean to insult your mother. Breathe. It'll be okay.
The entire goal of the Internet is to get ALL THE ATTENTION YOURSELF. If anybody hates you online, it's because you got attention and they didn't. Nobody hates insignificant people. So, often times people with lower profiles/attention in conversations will try to increase their attention profile by arguing/hating the high-attention people.
If people hate you, you've already won.
Your analogy to "rm -rf /" is invalid, because that's well defined, well documented, and well known behavior. That's not true of Redis's autonomous and non-deterministic response to a failure (not to a user action). No spec or doc precluded choosing a different master and preserving data instead of discarding it. In the absence of such explicit guidance, preserving data should always be the default. How can it be user error when the user did nothing? Redis did the wrong thing because something was missed in its implementation, not because of any rational or deliberate choice.
The choice of master is a static configuration set by the user.
Redis itself has no failover or promotion ability. There's an additional thing called Sentinel that can failover and promote individual Redis instances, but it is designed to recover complete instance failures (without immediately restarting), so a quick restart means no failover happens [an improvement to the "quick restart" scenario is showing up soon].
Redis made a really dumb choice of which node should be master,
(see previously; master is static, defined by the user)
pointed out well known and fairly simple solutions to the selection problem
(redis doesn't select things)
Also, this issue showed up last week. Last week. People are making it sound like this issue has been ignored for years. Nobody ran into this (and reported it) until recently. This use case is already being adapted into SOP Redis capabilities soon.
Try running into a big problem with any other DB and getting both attention and a concrete fix within two weeks. For free. The entire progress of the project has paused to address these immediate user issues.
because that's well defined, well documented, and well known behavior.
The Redis behavior is: always be a copy of a statically configured master. When the master has an empty dataset, all the replicas replicate an empty dataset. Pretty simple. :)
No spec or doc precluded choosing a different master and preserving data instead of discarding it.
Yup, specs and documentation did exactly that. Redis has no failover capability on its own.
preserving data should always be the default.
Ooops, you just re-invented the Mac trashcan.
How can it be user error when the user did nothing?
The user disabled persistence, enabled replication, restarted the process with zero data, then the replication recovered and stayed in sync with the newly zero-data master.
something was missed in its implementation, not because of any rational or deliberate choice.
nopers. more a lack of thinking it through from the user's point of view. an exact copy of nothing ends up being nothing.
Hasn't it been? How does leaving that latent in the system for years make things better? I rather think it reflects on an inability to reason about failure modes (including user failure modes), and deal with them proactively instead of after data was lost.
The users intentionally configured their options and the system responded exactly as it should have, given what it was asked to do.
Redis lets you have slaves which mirror the master. Hundreds of thousands of redis installations use this pattern to provide read scaling and offline master-loss persistence, and in the normal case, this works great. I myself have implemented systems with hundreds of redis instances which have gracefully survived the loss of the primary.
In this particular instance, the user turned off persistence, didn't understand the ramifications, and then brought the master back up with an empty database after a hard kill without thinking things through.
Fortunately, the user was savvy enough to have kept backups off the slaves, as is the usual pattern, and so was able to continue service.
This is not a normal pattern and goes against the general practice.
Does that help?
* Default to preserving already-replicated data, provide "clean start" as an option.
* Default to throwing away data, maybe-someday implement an option to use data that's already present in the system.
Blaming the user won't prevent another user from making the same mistake with the same result. Saner defaults, and an implementation to support them, will. Who's going to complain that you saved too much of their data?
About pointing to relevant papers: things like pointing to timestamped replication paper, which is a CP system for replicated state machines, in response to the fact that Redis does not support slaves when the master disk persistence is turned off, is actually just another instance of why it is not possible to have decent tech conversations on Twitter.
The problem experienced at Stripe is a result of Redis replication documented behavior, regardless of using Sentinel or not. Redis replication is a very simple system where replicas will try to exactly mimic the master, and if the link breaks, will try to connect with it again and again forever. There is no builtin HA, nor failover or alike.
So before considering failover (and yet IMHO pointing to timestamped replication is not very informative even in this context, since in distributed systems the details matter, and you can't just give a random reference to a completely different system which happens to have just superficially similar issues to fix), there were different useful observations to do.
Like: Hey @antirez, what about supporting master-slaves setups where the master can be configured without persistence at all, and yet when it reboots, it will not be considered viable for reconnections? Which is the fundamental problem: stopping the basic Redis replication behavior from working as it works, with slaves that always want to replicate the current master, which is not ok if we want a system supporing the master restarted without persistence (which wipes the dataset on restart).
However the whole problem with that is that there are a lot of people like you that will regard linking to a paper as a great way to help, and as a very smart thing, while there are other that are instead trying to work to really make stuff better. Before commenting you should make the effort to understand exactly the problem domain and its subtleness: exact details or what you say is not relevant in a discussion which is all about details.
As for your false dichotomy between reading the literature and getting stuff done, or your "make the effort to understand" ad hominem - go to hell. I've been doing this longer than you, I've been doing it better than you, and I've been writing about it as well. You make the effort to understand the problem before you shoot yourself in the foot yet again.
In Redis if you don't use any HA system like Sentinel, the map is fixed, it is an old-style replication system where there is the master IP address written in the configuration file.
Since the system is not supposed to lose the data on restarts, this is fine, but as soon as you want to support a different mode of operation with persistence-less masters, this must be modified, being Redis used with HA or not.
Now if you want to put Sentinel in the mix, the problem with this setup is not that the returning master gets promoted with a broken data set, but that if the restart is fast enough, the failure detection of Sentinel is not triggered at all, so the configuration remains the same. Just what is still, and was previous of the reboot, the current master of the system, restarted with a wiped data set.
In distributed systems this is the process not acting as specified, since Redis processes must reload their dataset on restart, otherwise they break everything, per design.
Now if it is a good idea to change this design: in the future yes, but so far we had not diskless replication, so for the replica to synchronize to write on disk was, anyway, needed, so why turn off persistence, and why to support it if the disk is needed anyway?
See? Arguments instead of random blabling and we can construct a reality or a model we both agree about. Then we can debate about what we think is right or not.
"Old style" doesn't explain it. Even if the original master has unconditional priority when it returns, that does not preclude it gathering whatever data might still exist from the others. I've been working on replication systems since '92, and I can't recall seeing any that would make such a poor choice. Can you point to any, or is this really a "new style" idea?
"we can construct a reality or a model we both agree about."
I will concede that the behavior might be compliant with how the system was specified. I'm not 100% convinced yet, but at least - now - you've made a credible case for that.
"Then we can debate about what we think is right or not."
Not. The data-preserving mechanisms (e.g. view/epoch IDs) are so easy to implement in this case that leaving them out is unjustifiable. You even seem to be coming around to that view yourself when you say "in the future yes" but apparently you can't bring yourself to admit that it was always the right choice.
It absolutely does preclude that because it's not how the system works. You've described a multi-master system while comparing it against a replication-only system. Apples and racecars.
leaving them out is unjustifiable.
Feel free to submit a pull request fixing any and all deficiencies you've found. :)
No, I haven't, unless you'd say it was already a multi-master system because it allows non-masters to continue without the master being present. What I'm suggesting is just a master being smart enough to recover its own state from where it had been replicated before. How is that even controversial? In what possible use case is it preferable to discard readily available data without an explicit user request to do so?
"Feel free to submit a pull request fixing any and all deficiencies you've found."
Give me some reason to believe it won't be torpedoed by the next bad decision or failure of diligence that comes to light, and I might. Acknowledging that this needs to be fixed would help.
It's up to you to form your opinion, but here are the facts:
1) Before diskless replication: even a master with persistence turned down, had to persist on disk, in order to support slaves.
2) Because of "1", it looked like futile to support this model of operations.
3) "1" is no longer true, I merged the diskless replication stuff just this morning.
Still I don't think you can form a fully informed idea unless you consider this: if you have persistence turned on in a setup which uses replication, like in most deployments using replication, you absolutely want the old behavior of Redis, of slaves reconnecting and replicating again on master restarts.
So when I say, in the future this could change, it is just as an opt-in option in order to support this new use case, not to say, the old behavior was crazy.