Rantenki

This is hardly a new problem though. Thieves used to be able to pull the wires from under the steering column and start a car.

For the more serious attacks mentioned, the automakers are essentially saying that they build extremely vulnerable systems and are afraid to disclose that fact.

I suspect we're in violent agreement though, that the correct outcome is to fix the vulnerabilities AND to document the repair methods.

That doesn't return your money to you like the credit card though. The irreversibility 'feature' of most cryptocurrencies makes it really hard to recover funds stolen through fraud... I guess that is a desirable feature for anybody performing fraud though.

>They could just have operators translate the configs automatically on the fly

How do they test that across the infinite number of permutations of configurations and deployments of K8S in the field though? It'll work for some people on the happy path, but it's really hard to maintain over time. Worse, it'll break randomly at some point in the future that is hard to predict, instead of at some publicly announced point in time where the breaking change is deployed (how it happened this time).

OMFG so much this.

I worked at a large company that deployed it's own Kubernetes stack, on a VERY large number of physical hosts. The theory was that the K8S would simplify our devops story enough that we could iterate quickly and scale linearly.

In reality, the K8S team ended up being literally 10x larger than the team building the application we were deploying on it. In addition, K8S introduced entirely new categories of failure mode (ahem; CNI updates/restarts/bedsh*tting, operator/custom resource failures, and tons of other ego driven footguns).

The worst part? The application itself ran fine on a single dev workstation, but also on any random assortment of VMs. Just pass the consul details as environment variables. I am not saying everybody on K8S is in the same boat, but I think that far more people are planning on becoming a unicorn cloud service than have any hope of becoming a unicorn cloud service.

TL;DR: If your hosting solution requires more maintenance than the application itself, you made a boo-boo.

It's worth mentioning that Jason (the author) has dedicated his life to the preservation of information as the "containing" technologies are shifting like sand beneath our collective feet.

He's thought about this stuff for a very long time, in great depth, and is very much worth listening to.

I recently was working on highly connected autonomous public transportation. Can confirm that connectivity is only increasing, and that cloud connectivity is going to be a hard requirement in the future.

As for security, while the automotive manufacturers are aware of dangers, and standards like ISO21434 https://www.iso.org/standard/70918.html are driving them towards better security engineering practices, there is a huge gulf between process/compliance/governance, and truly secure engineering practices as a core part of engineering culture.

Many/most of the people working on the on-board systems come from an embedded development background that doesn't include pervasive connectivity. That's a problem. They might check the boxes to appear secure, but they aren't necessarily invested in the process.

My prediction; there _is_ going to be a severe and widespread automotive security event which cannot be ignored, before that culture shifts.

Any corporation which fails to extract more value from it's employees than it pays them doesn't stay in business for long.

But yes, you are absolutely correct, and the value added by a productive developer is probably vastly larger than the value added by the rest of the rank and file. That's why developer centric startups can grow so large, so quickly.

Still need the rank and file to take care of pesky things like sales, payroll, governance/compliance, etc...

Conservatives: "ChatGPT has gone woke!"

Also conservatives: "Let's trick this chatbot into repeating racist/misogynist hard-right talking points"

ref: https://www.cbsnews.com/news/microsoft-shuts-down-ai-chatbot...

Complaining that the chatbot won't fabricate a false narrative decrying drag-queen book readings strikes me as disingenuous, because that feature was probably added to combat people who were already tricking the chatbot into fabricating false narratives around the LGBT+ community. They literally had to try and do that to find out it wouldn't do it, and then they're acting incensed because it won't.

I guess they're pissed because they have to construct their own crappy editorials for their Parler accounts.

You underestimate the power of the Dunning-Kruger effect.

That 1000 random people will build it, get it on the launchpad, and press the button. That's not the problem. It's getting to orbit (and back) that would be highly unlikely.

Breathing through your mouth for hours at a time _while_ doing that ^. Also not good. Your gums get dried out. Lots of endurance athletes take the "long in the tooth" look a few steps further than usual.

https://www.ucl.ac.uk/news/2018/jun/high-levels-oral-disease...

It's worth mentioning that the world could use a few more grug-brain developers: https://grugbrain.dev/

I forget the actual source of this quote that I am about to mangle, but:

    Bitcoin/Blockchain was designed specifically to circumvent the legal frameworks used
    to regulate financial transactions. It follows then that any non-fraudulent use of
    these technologies is the exception, not the norm.

Blockchain currencies are designed to avoid being regulated. I am not saying that all of the current regulations are _good_, but irreversible "anonymous" transactions are always going to be attractive to fraudsters.

Chesterton's Fence:

There exists in such a case a certain institution or law; let us say, for the sake of simplicity, a fence or gate erected across a road. The more modern type of reformer goes gaily up to it and says, “I don’t see the use of this; let us clear it away.” To which the more intelligent type of reformer will do well to answer: “If you don’t see the use of it, I certainly won’t let you clear it away. Go away and think. Then, when you can come back and tell me that you do see the use of it, I may allow you to destroy it.”

Not to be too inflammatory, but it's always amazing to me how people will ignore a threat as long as possible, then pretend it just appeared once they are forced to acknowledge it.

Not a perfect XKCD match, but pretty close: https://xkcd.com/743

Great; phrenology except the method is an inscrutably complex AI/ML algorithm which will end up hidden behind IP and trade-secret barriers, so it's harder to call bullshit on.

And the people in control of the training corpus get to decide who to arbitrarily add into which disorder category, either on purpose or through incompetence.

What could go wrong?!

This is true IFF you get to the point where you have to split up.

I know we're all hot and bothered about getting our apps to scale up to be the next unicorn, but most apps never need to scale past the limit of a single very high-performance database. For most people, this single huge DB is sufficient.

Also, for many (maybe even most) applications, designated outages for maintenance are not only acceptable, but industry standard. Banks have had, and continue to have designated outages all the time, usually on weekends when the impact is reduced.

Sure, what I just wrote is bad advice for mega-scale SaaS offerings with millions of concurrent users, but most of us aren't building those, as much as we would like to pretend that we are.

I will say that TWO of those servers, with some form of synchronous replication, and point in time snapshots, are probably a better choice, but that's hair-splitting.

(and I am a dyed in the wool microservices, scale-out Amazon WS fanboi).

Aircraft spruce sells tubing in a variety of lengths, and ship to Canada. They're a good source for tubes that don't fit normal framebuilding supply inventories. They ship to Canada directly:

https://www.aircraftspruce.ca/

I've built a bike frame with their 4130 tubes.

I just read the post, and I don't see what's worth criticizing there? Sounds like he's suffering from depression and isolation, and navel-gazing on his situation in life. Has the post been edited or something? I'll admit, I am super tired while reading this, so maybe I skimmed over some troublesome metaphor that I would have caught if I had been more alert.

I am asking seriously; what did you criticize?

Grug state Chesterton's fence very easy understand way.

That's a configuration problem, and not hard to correct. Tell him to go into the audio settings, then "advanced audio settings", and add the "sync offset" in milliseconds required to bring the audio and video into sync.

Also, it's likely that his camera is mostly at fault; OBS with my Logitech camera doesn't have any significant lag. OBS itself has very little lag.

Some additional items to keep in mind:

- Those developers in India are 180 degrees out of phase time-zone wise. There is effectively zero overlap in working hours from the west coast of the US to India. Makes it hard to collaborate.

- You get what you pay for. Skilled developers in India move to the US (or some other country where they have better opportunities). I've worked with some _fantastic_ developers from India, but they've mostly relocated.

- Quantity != Quality, although quantity does have a quality all it's own.

Calling it .5 QPS to downplay the severity is willfully ignoring the complaint in the article. It's not just a query, it's a full Git clone of the entire repo with it's history. That's a huge difference.

Our ECS containers all run continuously processing Kafka messages. Dev has variable throughput, but it's hundreds of messages per second at least. The containers all use between 256MB and 2048MB commits (they're stream processors, so they don't hammer the memory like a stateful monolithic service would).

My point was just that it doesn't have to cost over $1k on Amazon to run a combination of services which apparently CAN run in a private lab for $5/mo, however that is calculated. I can't run my full stack even without load (due to memory commit) on my 32GB, 3 year old Xeon workstation, and that would cost a lot more than $5/mo at any provider I am aware of.

Absolutely. My setup is, as stated, a dev instance. We have a substantially larger prod setup, but I can't get into the details about production scale on a public forum. That said, even the dev instance is moving a substantial amount of data; nothing like 800GB/day, but within an order of magnitude.

And yes, bandwidth costs can be brutal, especially if you try to build a hybrid solution that uses heavy services that live outside of AWS.

That said, the above comment didn't get into enough detail to get an accurate read on their bandwidth or CPU/Memory/Storage utilization, but if they can run it in a lab for $5/mo (which I doubt, since that barely covers power), then it can't be much, so I think it's fair to assume it's at least back-of-the-napkin similar scale.

I have a personal dev cloud on AWS that has:

* AWS MSK (Kafka server) with redundant instances across 3 AZs

* Aurora Posgres instance with redundancy across 3 AZs

* Multiple Lambdas for incoming messages from MQTT to Kafka

* IoT/MQTT and about a dozen iot devices sending telemetry continuously (every second)

* 20+ services with nearly 40 tasks, running on ECS

* All the supporting route53, certificates, VPCs etc. for the above

* A bunch of miscellaneous stuff I've forgotten that's used by other developers.

* All deployed via TF

_THAT_ all costs well under a grand. You're doing it wrong.

Update: My instance was recovered over this weekend (Apr 16). We've verified that integrations (webhooks, jira integrations) are working, and no data was lost that we can see (nobody works overnight, so our last change was at the end of the working day, which matched our recollection).

Thank you for responding here. Even a single successful restoration, anecdotal as it is, makes me feel a lot better about the situation. I was honestly wondering if Atlassian was just pushing the date out to soften the eventual backlash when they had to announce data loss.

Hey Mike; Not dumping on you personally, but the RTO claims to be 6 hours. I can understand that being a target, but we're at 32X that RTO target, with a communicated target date of another 12 or so days IIRC. That's literally two orders of magnitude longer than the RTO. I don't think any rational person would take that document seriously at this point.

I'll also ask (since nobody else has answered, I may as well ask you as well):

1. Are the customers actually being restored from backups (and additionally, by a standard process)?

2. Will the recovery also include our integrations, API keys, configuration and customization?

I've repeatedly asked Atlassian if:

1. They can confirm that they have backups of our data (about a thousand stories, substantial confluence, opsgenie history, and three service desks).

2. Will our integrations, configuration, and customizations also be recovered, or will we need to rebuild those once our data is recovered?

I have received no response, and no human is even willing to acknowledge those questions. The service desk staff ignore them as if I never asked. Repeatedly.

Also, I've been asking around, and haven't been able to find a single story from somebody that can confirm that they were down, who has had their data recovered.

As someone who is impacted, this is obviously immensely frustrating.

Worse, outside of "we have rebuilt functionality for over 35% of the users", I haven't seen any reports from the people who have ostensibly been recovered.

Next, their published RTO is 6 hours, so obviously they must have done something that completely demolished their ability to use their standard recovery methods: https://www.atlassian.com/trust/security/data-management

Finally, there have been some hints that this is related to the decommissioning of a plugin product Atlassian recently acquired (Insight asset management) which is only really useful to large organizations. I suspect that the "0.18% impacted" number is relative to ALL users of Atlassian, including free/limited accounts, and that the percentage of large/serious organizations who are impacted (and who would have a use for an asset management product), is much higher.