DuckDuckGo was down
duckduckgo.com
duckduckgo.com
"Mondays"
Corporate accounts payable, Nina speaking. Just a moment!
...for 10 hours: https://youtu.be/oFwkL1WgD_0
It perhaps is related to devops or the search index cache servers is broken
Hopefully at least two people reviewed the commit before it was put into production. Someone set up the post-launch smoke-tests/api-test-suits. Someone built the CI/CD-system. Etc etc.
My point is, it is rarely a single individuals fault when something at this scale goes down.
I could not agree more with this statement.
When discussing success to an outside group (giving a presentation or something) or higher up (especially here!) you always use "we". Even if it is a section that only you worked on or you did most of the work. You can often pick out good managers by just seeing if they talk like this, if their focus is around what they individually did or what their team accomplished.
My point is, it is rarely a single individual's success when things work.
There's a balance and I do think we tend to focus on the individual as a society. We can only climb mountains by standing on the shoulders of giants. One person gets to the peak and we should praise them for that, but we shouldn't ignore their foundation either (which I think currently happens). E.G. Kepler was an essential part of science and needs to be recognized, but if it wasn't for the work that Tycho Brahe did, Kepler's achievements couldn't exist. The "single person" mentality ignores the importance of the foundational work that needed to be done and frequently causes many to feel that they are not achieving simply because they are working in these roles, which are essential.
This is a form of illusion. A cover up for what is essentially in reality a mistake made not only by the team but by the individual as well. To mask part of the truth and lay only the blame on the team is an effective form of not hurting someones feelings but also an effective form of avoiding the full reality.
Do people not see how illogical it is to blame the team but only praise individuals? The precedence being set here is that: People succeed as individuals but fail as teams.
The harder ideal to strive for is that both the team and the individual take the blame, but it's harder because people are prideful stubborn and easily hurt.
If you make a mistake step up and admit your mistake. Don't hide in the corner and expect the team to change all their processes to account for your mistake. Yes the team should do this, but yes you should stand up as an individual and do things yourself as well.
There are times when you must blame an individual. Let's say a team member repeatedly pushes bad and buggy code to production. Is it the teams fault or the individuals fault when such actions are repeated? Does it lay on the team or the team member to make sure buggy/bad code doesn't go into production?
The line is blurry here and I feel it is ultimately the wrong stance to say absolutely good teams don't blame individuals. Good teams and good people take responsibility so there is no need to dish out blame.
What happens when the person who spilled coffee all over the on premise servers doesn't take the blame? The team sees this and decides to blame nobody. Is this a good team? No.
The good team member volunteers and states publicly that the fault is his own. The good team agrees with this stance and also says that the fault is with the team as well and both the individual and the team take steps so that this mistake cannot happen again.
It isn't about eliminating recurring problems, it's having the highest chance of reduction. If someone continually performs poorly then thats management's responsibility to replace them.
I never said play the blame game. I never said the team must not take the blame. Please don't put words into my mouth.
I literally said it's the teams fault and the individuals fault. Both parties must take responsibility.
Take it this way, if a single team member repeatedly makes mistakes then is it the teams job to put methods in place just for that team member? Or is it the teams job to help that team member as an individual?
The blame game is a stupid game so don't play a blame game. Take responsibility both as a team AND as an individual. The team taking the blame exclusively is AVOIDING individual responsibility. It is not a sign of a healthy team.
I am essentially saying the solution is not so clear cut. The team can't always take the blame just to "avoid a blame game." The reality of the universe is that problems aren't always team level problems, that problems at the individual level exist as well.
To exclusively avoid addressing problems at the individual level is a stupid and delusional endeavor and you also win the stupid prize for ignoring reality.
>If someone continually performs poorly then thats management's responsibility to replace them.
This is called blaming an individual then firing him. It entirely contradicts your main argument. I didn't even go there yet, I advocated blaming the individual than helping him improve as a team. You immediately cut his head off and let management do the dirty work.
If your system can't tolerate one person's mistake, it's not robust.
None of this means you can't praise a whole team (you should! The original statement just said you CAN praise individuals), or discipline team members who regularly show poor judgement.
But the parent is right that it still sucks to be the persons whose work was the proximate cause of an outage, even if nobody is going to blame/punish you for it. It doesn't make for a day where you feel good when you get home from work.
* You probably got a review on your change before it was submitted, so you weren't the only one to think it was an ok change to make
* If you "revert first, ask questions later", you might also be the first one to fix it
* Hopefully the team recognizes that there are higher-order causes that caused your submit to break things than your own personal skill
EDIT: scratch that. DDG is back in business again. I would love to read a post-mortem.
https://help.duckduckgo.com/results/sources/
"These include hundreds of vertical sources delivering niche Instant Answers, DuckDuckBot (our crawler) and crowd-sourced sites (like Wikipedia, stored in our answer indexes). We also of course have more traditional links in the search results, which we also source from multiple partners, though most commonly from Bing (and none from Google)."
Edit: strike that, only for some entries. Must be some cache?
I was quite confused why my query for a well-known brand name was returning nothing. Bang queries still work.
- It could be an issue with DNS servers - Certificates that got expired - Latest push to production that got wrong - DDoS attack
[EDIT] removed the metrics due to downvotes! check below for traffic information and how much is lost..
"Google suffered an outage late on Friday night, though you might not have noticed -- it only went down for five minutes. During that brief window, Internet traffic around the world dropped by a massive 40 per cent, according to Web analytics firm GoSquared, Sky News Online reports. "
Bing also: https://www.pcworld.com/article/183770/bings_30_minute_outag...
"Microsoft's Bing search engine was offline for about 30 minutes on Thursday evening."
Qwant also: https://blog.qwant.com/qwants-unprecedented-service-failure-...
"Qwant’s infrastructure broke down around 12.00 UTC, and the service could only be restablished hours later, around midnight"
[1]: https://duckduckgo.com/traffic [2]: https://twitter.com/duckduckgo
Edit, and now it's back up for me, so ~15-20min of downtime from my perspective