I can't imagine what such a small team must be going through with a major DDOS - wish them well in their efforts!
"It all scales in all directions with a properly thought through architecture" sounds dangerously like, "Programming isn't that hard if you just do it right."
That's not a tautology. In fact, it's actually worth pointing out, especially to junior engineers who get frustrated by how hard everything is, that it actually doesn't need to be that hard if you, well, do it right. Obviously that's not productive feedback without actually helping them be better, but it's far from a tautology.
For anyone wondering, a tautology is a statement that is logically true by construction, rather than contingently true because of the way the world is. For example, "Programming isn't that hard if it's easy" would be a tautology. Constructing a counterexample by changing programming to something else shows that this was not a tautology to begin with: "Sending a man to the moon isn't that hard if you just do it right," which is obviously false, because even if you do it right that's objectively difficult.
Programming is hard, but we make it much harder than it has to be by doing it spectacularly wrong in many ways, both individually and collectively.
A logical tautology is, "A statement that is true by necessity or by virtue of its logical form."
A linguistic tautology is, "A phrase or expression in which the same thing is said twice in different words."
In formal debating, for example, you can call someone out for either type of tautology.
If you say "All bachelors are unmarried", this is true both because of the meaning of the words, and because of the logical structure implied by the words.
In either case, if you state a tautology, you state something which is true in all possible worlds, given the definitions of the words, at least. Someone can then call you out for stating a tautology, which is to state something that is vacuously true, that is, you've made a statement about the nature of reasoning itself, in any possible world, but you haven't said anything at all about the world we're actually in. So you're wasting your breath even though what you say is unassailably true.
(Note that in mathematics, the tautologies are precisely the theorems with their premises or axioms! So this is by no means always useless.)
The problem with calling out tautologies in common life, however, is that the danger of identifying a false tautology is very high. When someone says "Either X happened, or it didn't." you may be tempted to say "tautology!" but in fact they are probably making some kind of oblique point or highlighting a flaw in someone else's argument, etc. In other words, tautologies may be vacuous as statements about the world within a logical framework, but as speech acts in the real world, they always come with a motivation and that can usually be expressed in a non-tautologous way. For example, "A or not A" can be expanded charitably to "A or not A, and this is relevant to the topic at hand", which is not a tautology anymore.
In this case, if you do something right, it's not extra hard, which is kind of a tautology. But there's a point in saying it, which is that it doesn't have to be that hard... if you do it right. And that's not true of everything or in every possible world, hence not a vacuous statement.
But this boils down to If you build systems using a high level of skill and foresight, it's easy to do.
This is of course not a tautology, but a contradiction. I agree that inexperienced developers can, as it were, 'make life hard for themselves', but that's (trivially) due to their inexperience. I don't think there's a silver bullet for inexperience.
Over-engineering is bad, as is under-engineering. Fuzzy principles like 'YAGNI' can't be applied without skilled discernment, which means experience.
> Programming is hard, but we make it much harder than it has to be by doing it spectacularly wrong in many ways, both individually and collectively.
I think I agree with this, but it depends on specifics. What sorts of things are you thinking of?
The point being made here is not necessarily a flippant 'git gud'. Instead, it is a statement that problems are tractable, and that getting some things right up-front can have good pay-offs down the road.
In other words, don't give up and try to figure out what is good and bad practice.
Yes.
> This is of course not a tautology, but a contradiction.
It's not quite a contradiction! If you bill $1 for changing the bolt, and $9,999 for knowing which bolt to replace, this shows that the work is easy, but the experience required to make the work easy is not easy. If the master can draw it in seven strokes, but you don't see the seventy thousand strokes they did before, it looks easy, and in fact it is easy, for the master but not for the novice.
> I agree that inexperienced developers can, as it were, 'make life hard for themselves', but that's (trivially) due to their inexperience. I don't think there's a silver bullet for inexperience.
That's right. However, we can also make life hard for each other, and there are some solutions for that that are better than doing nothing.
> Over-engineering is bad, as is under-engineering. Fuzzy principles like 'YAGNI' can't be applied without skilled discernment, which means experience.
Yes. This is why we have code reviews, design reviews, pair programming, and so on, but these aren't silver bullets either and there is no silver bullet, but if these things lead to increased awareness of why and not just what and how, then we can accelerate the process of acquiring that discernment. As Dijkstra said, if it goes to the grave with you and you didn't pass it on, you didn't really do your job as a senior engineer (paraphrasing).
>> Programming is hard, but we make it much harder than it has to be by doing it spectacularly wrong in many ways, both individually and collectively.
> I think I agree with this, but it depends on specifics. What sorts of things are you thinking of?
Using the wrong tool for the job. Using too many tools for the job. Using tools that do not afford mastery, because they are too complex for anything built on top of them to be comprehensible.
This is all quite abstract. A specific example: we (the JavaScript community) had a good thing in that JS was a small, human-scale, useful and commercially valuable language with applications beyond its initial environment on the web. We got excited and built the npm package archive, and filled it up, and now we have unmaintainable, incomprehensible piles of trash piled upon trash that can't possibly be used as a foundation for anything reliable, performant, or maintainable. This is unfortunate. What's even more unfortunate is that this piled-up-trash approach still has momentum, still lets people get useful work done, and still has some value to the community. So we keep using it, and even keep piling more on. It takes considerable effort to step back from all this, take a collective mulligan, and start over with a principle of taking things away to make things better, rather than adding more hacks to hide existing hacks.
This is one example of many, but I mention this one because I was there when JS was simpler and better and I watched as we made it markedly worse. I would have recommended JS as a first language to beginners when node.js and npm were new, and I did, but I cannot now recommend them in good faith, because they have become antagonistic to quality and to mastery of the craft.
If it takes years to be able to do it well, it's not easy.
> So we keep using it, and even keep piling more on. It takes considerable effort to step back from all this, take a collective mulligan, and start over with a principle of taking things away to make things better, rather than adding more hacks to hide existing hacks.
True, but it can be done. The community moved away from Bower, for instance.
> Bower
Yes, finally! In another 40 years most of the trash we're creating now will also be gone, probably replaced by more unless we find some discipline.
Is this like saying, programming isn't hard if you choose easy enough problems to solve? Or should we ask for a link to see a demo of an AGI implementation?
I guess math is not hard either if you're "doing is right", as long as it's all arithmetic...
>>That's not a tautology.
I would agree tautology is not the best description, probably fallacy would do fine.
No, this is saying that things don't have to be as hard as we make them. You don't need more than a hundred people to run a top-ten website, and that shouldn't be surprising. It is surprising only because we are so good at making things overcomplicated.
No.
1. Your comment makes it sound like Wikipedia is just, or mostly, serving read-only content, which is far from true. Yes, static read-only content is significantly easier to serve than dynamic, editable one, but Wikipedia is the latter.
2. Claiming it's easy to build something at this scale is "isn't all that hard" just makes me think you've never done anything similar. It reminds me of devs saying they could re-build MS Office over a weekend. It's just ignorant of the software's actual complexity.
I'm not associated with Wikimedia in any way, but have worked on large-scale software projects before, and things are quite different from, say, websites only serving 100k monthly active users.
What, in your opinion, would be the work needed to go from a 100k monthly active user site to a wikipedia scale site - that would be comparable to rebuilding MS office?
The saying usually uses Facebook or Twitter.
I thought it was a good, but snarky, point. Especially given my browsing sped up after I installed extensions that turn all that crap off.
Wikipedia does not need to be globally consistent like Betfair does and the ratio of writes to reads is nothing like 10%, I'd guess at one write per million reads or less. There are several pretty obvious ways to architect a site like Wikipedia for effectively unlimited scalability. The main trick is that it doesn't matter if a page is slightly stale and you can queue edits in the backend for quite some time (many seconds) without severely harming end users. Given those constraints it really isn't rocket science given the plethora of amazing tools we have to hand.
What I'm NOT saying is that I could build it in a weekend. It would clearly require a few teams of skilled engineers to put it all together and, crucially, operate it. My initial comment was in the context of Wikipedia having 100 engineers, and I think it's reasonable to say that a team that size is easily capable of such a feat.
https://wikimediafoundation.org/role/staff-contractors/ has the names of 379 employees. I believe (perhaps astonishingly) that is all - engineers and non-engineers combined. Their engineers spread across departments, but judging by the 141 instances of the string 'engineer' in that page, I'd be surprised if the number exceeds 200.
Though it is worth bearing in mind that everything's open source, and there's a hefty community component. So there's a more vaguely specified number of people who might provide patches, and individual wikis are mainly run by volunteers.
I worked there for four years and I miss it every day.
Sorry but now I'm curious, why did you leave?
But, I'd consider Wikipedia traffic to skew heavily towards anonymous read-only, with very few logged-in write traffic.
This allows for tons of caching opportunities: Varnish, Memcache, etc. And these techniques are well known.
I suspect latency in the minutes for cache updates would be unaceptable to wikipedia users
Additionally if vandals know their vandalism will stay for 30 min, they are much more likely to do it, which is a vicious cycle
Also, the white house PR team is actively watching and editing political figures articles. They will sort it out too.
Next "power users" as others put it are not a single set of editors. It's more of a social network with multiple levels of trust. The idea of a wiki is that all users have write access, even if those changes are moderated to have different levels of latency.
Of course there are ways to engineer the system, but at that point one is, well, engineering a system. And WMF is doing so on a shoestring compared to other comparable levels of traffic.
Is WMF creating new paradigms of computing? Probably not. But they are doing a good job, IMHO.
Their expenses have doubled in less than 5 years...
Even if their ratio Expenses / Assets has now decreased compared to 3-5 years ago (but stalls now), it means that their goal of financial independence is still very far away and they still rely heavily on a huge amount of donations.
But that essay is clearly pure hyperbole. The expenses aren’t exponential, they’ve been roughly linear for a decade. Notice how the word exponential was removed in the second version. The graph is showing increasing savings along with increasing growth, and the expenses appear to have slowed slightly in the last five years compared to the five prior years. It’s completely failing to demonstrate the stated claim of runaway spending, the numbers practically prove the opposite.
Plus it’s not outlining what the money is used for, so there’s no concept of efficiency here, no reason to doubt that increased service came with increased expenses. There’s zero meat in this argument.
Whatever; last year’s total expenses seems very small to me compared to web sites of similar size; there are startups smaller than Wikipedia’s team that have raised more money than Wikipedia’s yearly expenses without managing to deliver anything. Wikipedia’s value to the world is currently larger than it’s expenses, IMO, and I think it’s impressive what this non-profit has done.
I’ve worked with a couple of engineers who are now on Wikipedia’s SRE team. They’re good engineers, but not elite by any means. Not “10x” developers or wizards in castles or whatever. Good solid engineers who I would work with again and fight to hire. But they’re not savants or even the top 10% of folks I’ve worked with. Solid mid to sr level engineers I’d be happy to hand a project off to with ambiguous goals and little oversight, and I’d expect them to get a team of 4 or so other engineers to be more productive.
These are the engineers who meet the job requirements for SRE positions.
No offense to you nor your team, but to me, as a consumer, reddit's product doesn't appear nowhere near as polished as Wikimedia's projects.
Also, with the caveat that I don’t know enough about the implementation details of the product at Reddit: I’d argue that Reddit’s workload is more write heavy that Wikipedia’s workload, which makes caching and scaling a bit harder for Reddit, relatively speaking.
April 9, 2012 "Instagram was acquired by Facebook today for $1 billion in cash and stock. It only has 13 employees and a handful of investors. ... Meet 11 of the lucky employees and 9 investors behind Instagram. ... Two other employees were hired during South by Southwest last month and their information wasn't available for this story. "
https://www.businessinsider.com/instagram-employees-and-inve...
Not only that. They do all this with amazing openness. Their records of incidents and deployments, who's in charge of what, rotation schedules are all public and shared in MediaWiki (although they're not that well organized). I can trace this back to circa 2005. Maybe this could be the largest knowledge base of devops that is public.
cf. https://wikitech.wikimedia.org/wiki/Category:Incident_docume... https://wikitech.wikimedia.org/wiki/Deployments/Archive/2019...