“In roughly two hours, 1647 devices are about to be wiped”
infosec.exchange
infosec.exchange
I see that life as enterprise service desk hasn’t changed much. “Nobody tells me nuthin!”
Shout out to the ever under appreciated service desk folks out there.
"Why didn't you tell us you hired something new?"
"They work here for 2 weeks now"
"Tauch' deinen Füller nie in Firmentinte."
That's a very important advice!
Poetic
Chinese lesson two: really don't call them "not a thing".
¹: Also it's still my favourite, uh, thing that the word for a "thing" is, literally, an "east-west".
A "thing" or "stuff" in Chinese is 东西 (dōngxi). That's literally "east-west" if you pick the individual characters apart. That's what the footnote refers to.
Calling somebody 不是个东西 (bùshì ge dōngxī) means something along the lines of being good for nothing, i.e. an insult. Translating it literally, it would be calling somebody "not a thing".
Saying someone is irrelevant is likely an insult, also in English.
Not the corp in question but here is another one: https://www.thewealthmosaic.com/vendors/geissbuhler-weber-pa...
You may have heard of some of them, ServiceNow is one, SAP is another, some small companies like that.
It turns out there is not a technological solution to a management problem.
I can imagine the brute-force approach:
Log every intra-company communication, and if some communication was meant to go to department X, Y, Z, and it only went to X, Y, then a flag would be immediately raised to department Z's attention and whoever sent it.
Of course the personnel in department Z might review it but ignore it anyways, but at least now there's a paper trail of who's at fault.
An Exchange system already gets you 80% of the way there if you force all on-the-record communications via email.
That's the problem right there. One of my clients had a large IT organization of over 1000+ employees, with strict change control rules, and procedures that were tracked in SN. Every time there was a change management meeting everyone who could possibly be effected would get an email from Service Now notifying them of the upcoming changes.
Pretty much engineer ignored those emails, because there was so much going on in the org you'd get dozens of emails in a week and most of them you'd only be tangentially effected by, meanwhile you had your work to do.
So the problem isn't getting the notifications out it's getting people to pay attention to them.
Individual preferences do vary, one ignores 90%, another 95%, another 100%. And the one who's ignoring 100% of them will likely eventually make a mistake that otherwise wouldn't have happened.
But it will be fairly straightforward to resolve, after all there's an extensive paper trail as the chain of custody seems clear. Assuming the "change management meeting" emails were the approved means of communication.
IMO, one of the lessons that came out of Chernobyl is that it absolutely is a problem for the organization. Exposing people to too many "alarms" that are constantly going off will cause people to start ignoring them.
Part of good design is figuring out which things are truly important, and how to communicate that to the people who are supposed to be paying attention.
The emails mentioned by the parent don't sound like alarms. Because an alarm is usually for 'drop everything and focus on this' situations.
The equivalent in email terms would be a receiving an email with a subject in ALL CAPS bolded and underlined.
Or in general intra-company communication terms, a phone call from your boss without any pleasantries and a serious voice.
If someone makes the wrong decisions because they start ignoring signals then don't promote them or give them important coordinating responsibilities. Those who are capable of filtering out a larger fraction of noise do exist.
Of course there will always be folks whose preference is to read near 0% of their emails, but that doesn't imply organizations must be designed around them.
This is simply wishful thinking. Outliers certainly exist, but the idea that there are sufficient number of them that you can just ignore human nature is a path to disaster. You'd have to somehow accurately measure not just who is opening these noisey e-mails, but what they are retaining from them, and measure it over a large period of time, knowing that the vast majority or going to fail. It's far cheaper and more reliable to fix your noisey system than to try to outwit human nature.
Can you describe this 'cheaper and more reliable fix'?
It's really weird that you think you can't decide who is responsible for dealing with a type of notification ahead of time without "endless politicking and horse-trading", but think that blasting everyone with every notification, then attempting to sort out responsibility after something has gone wrong will somehow not cause "endless politicking".
Again, this is something many businesses do already. They don't blast the whole company when a bank account balance is low, when a server's disk is full, etc, notifications are targeted to appropriate groups. The specifics about who gets what is going to vary based on your organization. I can't give you hard and fast rules that will work for every organization, but that doesn't mean it's somehow impossible or not worth doing.
It's hard to tell if your joking.
The reason for implementing restrictive organizational procedures in the first place IS because nobody behaves like a 'rational human being' in such an environment for any significant duration.
When __everything__ is highly important and #urgent#, nothing is important and urgent.
In a sufficiently large and complex organization there will always be some percentage, neither 0% nor 100%, of actually 'highly important and #urgent#' hidden among the pretenders.
The point of any such tracking system is to discover, via positive or negative selection, what is actual instead of what various flawed humans pretend.
That... doesn't sound like a great approach to me.
If no one is keeping track of who is doing well and who to replace, then fixing that first is more important.
:-)
That (probably) means that the system for dealing with planning maintenances (well, usually, "approving them") needs to have a sufficiently good understanding of what humans care about what changes.
At a previous job, the planned change tracking system was REALLY good at tracking what specific compute facility was going to be impacted by any specific change taking place in that facility. And had a really good way of allowing you to filter for "only places I have stuff running" (and I think, even some breakdown of general change types as well).
It was, however, not easy to get notification of "there will be maintenance on submarine cable C, taking it off-line for 4 hours" or "there will be maintenance at cable station CS, taking cables C1, C2, and C3 down for 3h". And as one of the things "we" (the team i worked in then) was doing was world-wide low latency replication of data, we did actually care that cable C was going to be down. But, the only way we could find out was "read all upcoming changes" and stick them in the team calendar.
Was it good? Eh, it worked. Was it the best process I've seen? Probably ,yes.
So what? How would that change anything? Service Desk would send strongly worded emails to... somebody? The Organizational Overmind was already getting warning messages. Daily. There's already a trail of emails and status reports. It wasn't a knowledge issue. It was a "there's no effective direction" problem.
Which is why I consider it a 'partial-solution' as technology cannot do everything by itself yet.
So you'd select "System Z Update Access", write a one liner of "I need update system Z to do my job supporting The Alpha Process", your manager would approve it (providing they agreed that your role did involve supporting the Alpha Process) and then someone from the System Z team would also approve it (providing they agreed that you needed that access to support that process), and then a few minutes later you'd be automatically added to the right role.
Traceable, secure, and fairly painless. Required a lot of setup for all of the roles and automation though, I believe.
Human problems require human solutions. Tooling problems require tooling solutions.
Been my favorite saying at my small business for years now when people propose technological solutions to HR issues. That isn't gonna cut it.
Or did just some clueless start-up beg to get sued into oblivion?
I am making some assumptions when discussing this topic, namely:
1. The CEO and executive leadership are actually competent, have at least average middle aged adult levels of patience, and don't have their daggers out for each other.
2. There exist more than one method of communication between each layer in the hierarchy.
3. The organization is engaged in normal business, not in the middle of an M&A deal, outside investigation, or bankruptcy proceedings.
In reality, most large organizations do have at least a semi-competent leadership team most of the time. Unlike what is commonly supposed by junior level staff expressing their frustrations.
Everyone on HN can read the comment chain and see where you joined in, and who said what... so the claims are difficult to take seriously.
Even if you focused on my alleged lack of qualifications to comment it would have not been as self-discrediting.
Though I don't get riled up by oblique insults, that might not be the case for others, especially for a topic that might be linked to negative emotions in the reader-base.
My sentence on 'junior level staff' wasn't directed towards you, it was a reflection on why it can be difficult to see the grand scheme of things when the environment might restrict that.
Nice Hot Fuzz reference.
This is one of the primary reasons why I am totally done with Tech
The distance between users and builders is so excessively far apart now and the levels of abstraction for actually building things that are robust is just not even a consideration in software-centric design. Literally everything you have built after IDK 2000(?) will have exactly this issue. Just hope you're not at a scale that crushes people.
Software is the language of alienation and increasingly becoming unethical, as these systems are becoming increasingly impactful on the most vulnerable with no buttresses or supports preventing this kind of malfeasance.
It's not trivial. These are people's livelihoods at stake.
> This is one of the primary reasons why I am totally done with Tech
Is it that much better in other fields?
You are only safe when you don't have bosses and all responsibility and power rests on yourself.
This is from 2018 and shows where you can actually have some rights as a worker. It's not hopeless: https://www.ituc-csi.org/IMG/pdf/ituc-global-rights-index-20...
My point was that tech probably isn't all that different from other professions where there are layers of management on top.
The parent comment made it seem like "tech" was worse than "other," which I'm not sure is true.
https://www.smart-jokes.org/programmer-evolution.html
Most working programmers are in the middle near “seasoned professional.” They spend all their time thinking about how to manage complexity when they should be thinking about how to avoid it.
Eventually it might stop being such a cash cow - maybe thanks to endless numbers of teenagers hyping Fortnite.
I expected something basic that implements the login flow and some of the more important calls. Instead I found something that could be described as an C# architecture example application. If I wanted to learn C#, I think I could learn a lot about application patterns from it. MVVM, how to separate the concerns into separate "packages" for reusability and the likes. What I can't really learn from it (without already being a C# pro, I work in other environments) is how to use the API.
Companies are realizing that the IoT node structure is not just a one-shot project that ships and is never maintained. It's part of the revenue stream. So the need for embedded Linux developers is increasing and the experience base out there isn't great. Again, having an RPi in your desk drawer doesn't make you an embedded Linux dev.
> So the need for embedded Linux developers is increasing and the experience base out there isn't great. Again, having an RPi in your desk drawer doesn't make you an embedded Linux dev.
Maybe things have changed, but in the past there hasn't been much financial incentive for devs to build up those skills beyond the "RPi in the desk drawer" (zing!) level. "Work hard, make a lot less than a web dev, be less hirable in the future because of your niche skills." I'm speaking as someone who actually did look into what training myself on some of Wind River's products would cost, about eight years ago, and looked into what the jobs paid.
edit: I can see by looking at a grandparent comment that we're probably in agreement
Then again I refuse to make FPGAs for HF trading firms, and apparently you can make bank there if you can put up with their crap.
I worked for over 10 years doing Linux Kernel driver implementation. At Intel.
When I quit last year my base was $120k (150 TC). I couldn’t afford a house in the HCOLA I’m in. Remote wasn’t an option. For me a house is a prerequisite to starting a family.
So I left to get housing. Either making more or going remote and moving to a less expensive area.
Embedded is really fun for me, but it just can’t keep up in times of such inflation- especially WRT housing.
It's so satisfying to use 10,000x less resources to do something like full text search or handle 1,000 tps.
If anything they are part of the 25% of the company who just got a few extra paid vacation days.
2. The people impacted are employees who now have a few days chilling. I would love to have this problem as an employee.
This doesn't really have that much to do with software. As with many other things, software can make it easier to have this kind of crap, but it is not the cause and not a prerequisite.
Case in point: Franz Kafka wrote The Trial 30 years before ENIAC.
A complete absence of tool selection, migration, or any preparation whatsoever, because the new tool couldn't be funded from leadership, so the horrible ship kept belching forward. Until it was shut down. End result was seventeen people sitting on their thumbs for years.. and a totally fragged publication environment that never recovered, the product of which will - at best - be waived on future contracts at some incredible cost. At worst, it will be yet another barrier to the already marginal business.
This was a management problem. They had a plan to retire obsolete devices, but no plan to replace them.
Cool and now I bet they have a new line on their procedures to ensure continuity of service (so that the next management can fumble this in new and creative ways)
I don't get the impression that this a tech-specific issue.
But nobody wants all that additional spend, so close to year end. Departments bickering over who’s responsibility it was, who’s budget it came out of, and so on. So everyone dug their heels in, and we continued to shout “iceberg!” from the sidelines."
I've seen this play out multiple times over the years. What's the solution?
The correct strategy here is to go on the offensive. Make friends with your manager’s manager’s manager. Just go full on social bribery mode. Invite them for Christmas dinner. Even if they decline, it will make them remember you, and next time, ask them to a barbecue or a picnic instead, and they’ll say yes. If you can’t throw that high, shoot for your manager’s manager. Once you’re in, get the dagger in your manager’s back, but only once you’ve made them a pariah, take their role, and repeat until you retire wealthy. Remember to take every opportunity to accrue political capital.
Eventually you will be in a position to fire people who make poor decisions, but you won’t, because the salary is good, and retirement is only a few years off.
You literally cannot prevent this behaviour in any human-operated organisation of any scale beyond a fistful of people - you can only co-opt it.
Even in my program of just north of 100 people broken infrastructure follows that same pattern. Something moderately breaks, the devs complain, they are sympathetically told to make do. Rinse and repeat until the devs realize their complaints never get addressed and stop complaining past a quick email. Then something REALLY breaks with no workaround, devs mention it's been heading this way for a while, and management, all aghast, exclaims "well why didn't you say so earlier if this was such a problem?". It's to the point where we (the devs) have started keeping locally archived email records just for the "told you so". Which of course makes us no friends because we point the blame where it belongs, so we're officially covered but our complaints get listened to even less. And the infrastructure is fixed just enough to limp along until the next catastrophic explosion.
You need someone who gives a shit with the power to crack the whip. In short, you need to give someone enough power that they can potentially abuse it, something modern business is allergic to.
There is very little reason for an employee to care about preventing a failure they will not be directly held accountable for. I don't care if we lose clients. I don't care if we get hacked.
My life is not impacted one way or another by whether the divisions I work in succeed or fail, unless they utterly fail.
So if this Intune thing landed on my desk, I would do nothing about it. Give me some incentive to care and I would.
You are trying to do a good job which might lead to a promotion. You see the rules and operations of the organisation as something to work within.
The people who ignore you are trying to get promoted. They see the rules and operations of the organisation as a irritating backdrop to their personal goals. The job could be anything. Ascent is all that matters.
They will get promoted. You will retire one day, your nerves fried.
The incentives are all wrong, and playing the social metagame rather than playing the game by the rules always results in an advantage, and thus this behaviour is inherently embedded in any human structure.
The solution isn’t “better management” - it’s in fundamental societal change, which ain’t coming any time soon.
Put an impartial AI in charge and it will make the right decisions — and people will riot over the injustice. Ultimately, you’d have to go machine the whole way and the humans can all go bake bread and have wars over that or something.
Within a minute the deptartment head forwarded my request to the manager who was ignoring me and told him to take care of it. It was promptly. I got the email chain when it was done, there was some comment from the manager who ignored me to the facilites person who ignored me, that it "should never have been allowed to escalate.."
I don't miss big companies sometimes.
Don't want the issue to escalate? Then maybe do the job you're paid to do
So much for escalation. I no longer work there (I left; I wasn't fired).
Cringe.
Apparently most managers can’t even manage properly. Most of the time this “ship” word is tantamount to stolen valor in the workplace.
In dysfunctional organizations, the management structure exists to rein in true leaders. In a healthy organization, leaders are recognized and supported by management, whether they're in management positions themselves or not.
Like, with these 2,000 people locked out of devices? Assuming they're contractors? It probably won't affect the bottom line at all. A minor nuisance, monetarily. Easy to route around; just work people harder until the problem is solved, or take a minor write-off and play Tetris with the books to make it look like you actually made money rather than lost it.
But if you want to have a company where people are happy to work, you absolutely must have non-asshole leadership.
If they don't do that, the guy just does whatever he want, often leading horrible outcomes. It's only working because there are enough people who are passionate enough for the project and want it to succeed. Such people would leave if that was just a software company.
Also, no company I've ever worked for makes their budget after asking people what they need. Instead, the Finance team just copy-pastes last year's budget, usually with a little haircut off the top for "efficiency." So let's say you run a team that depends on a handful of servers. You bought the current ones five years ago, and it cost $500k. Now it's time to replace them. But since you didn't buy $500k in servers last year, it's not in the budget for this year, so now your entirely reasonable request is being scrutinized as "unplanned spend". Several times I've seen teams try to build a plan for the upcoming year, only to find that there's nobody to give it to, because the "budgeting process" is done in about a week's time, usually two or three weeks after the fiscal year starts and with commensurate effort.
Seen some years ago. A mail was then sent from CEO to CxO along the lines "it seems there's something hiding under a rock there, please check it out". The guy who talked about the thing was a recognized expert in his own domain, while the CEO was on a kind of "thumbs-up tour". The manoeuver had been briefly discussed with the expert's hierarchical chain and pitched as an opportunity for action rather than "those guys don't do their jobs".
A small shitstorm followed in middle management, at the end the problem was quickly solved, deemed "not that important in the end", and since no one was at fault and no one innocent, everyone quickly went quiet again.
In this case, the smart thing for every person who could have done something was to take no action, as they would get no credit for preventing a problem but would take a budget or resource hit for doing so.
I am a fire fighter, not a fire marshall. Fire fighters are heros. The fire marshall is a pest.
Archive those CYA emails and enjoy the popcorn as you watch management sink on the Titanic.
You kinda have to admire their commitment to the cause."
I want to know what their org chart looks like.
It was linking to the Microsoft one, but I think the Oracle one is more relevant.
What means "we don't specify it" in this sentence?
FYI for those non-corporate readers, if there's an actual compliance department that means that the cost of non-compliance is really, really high. That either means financial or government/DoD.
> The cost should be billed to the department with the users that were affected.
It doesn't really matter. These are arbitrary slicings and dicings of one big monolithic blob anyway, internally connected by interdependent causal links.
It's a "simple" management decision how compliant the whole operation gets to be, also weighing the cost of compliance versus consequences of non-compliance (from simple things like "have to massage the scope during the next audit" to "might impact a potential lawsuit" to regulator fines us to personal criminal responsibility for the CEO), similarly it's a management decision how much spending each org/department can do on getting compliant without requesting budget for it, etc, etc.
In this case the priorities are clear and it just happens that there is this big discontinuity (crisis!) that will probably look like a bad overall trade off, something that should have been prevented.
Will this even result in some change to the whole decision-making hierarchy? Unlikely. If this hierarchy was able to completely impervious to all the input from below (emails, calls, meetings, 1 year grace period, etc) then it likely does not matter that these devices will get wiped.
They knew it was coming. They were willing to fix it. They spent weeks exchanging emails on how to setup a meeting to solve the problem. The problem eventually solved itself.
Had a pretty similar experience with management early in my career that was wide-opening on how incompetent every single manager was. Became a manager myself with the intention of avoiding that. I could not. Changed career path.
It took me two jobs as a manager to realise that, at least in software development, a manager’s job is to pretend. To make uninformed decisions and lead the team without understanding anything of what is happening. You also spend your time negotiating with upper levels that want everything without even thinking about the implications (I’m not talking about costs or time, I really had meeting with really high levels managers who asked me, straight in the eyes, to make "a solution with all the advantages and without the disadvantages" and they were very proud of their line).
I learned that very high level management meeting are dumb and boring, that those people don’t even have the slightest clue what they are talking about and spend hours discussing micromanagement discussion (I attended a very high-level meeting where I replaced my n+1 and they litteraly spent one hour discussing who should send an email to X to ask him to send an email to Y. I took notes of that one because I feared nobody would believe me).
But I also reached the conclusion that managers are necessary. I even had a very good one who told me after one week: "I’m a manager, I have no idea how you are doing your thing. My job is to set a goal with you then your job is to ask me every time I could help with your job. Also, I’m here to insulate you from the administrative shit".
I tried to become a manager like that. I also lived by the credo: "If anything fails in my team, it’s my fault, I will not put the fault to individuals in my team".
I learned that this work only with very good teams and independant individuals. Some people need to be taken by the hand and a good manager will offer psychological help. But this only work if the layer above is also working that way. I ended fighting with my N+1 because they absolutely wanted to fire someone from my team.
Needless to say, a CEO and friend told me I was not a good manager. I would never become one if I didn’t change the way I was looking at things.
So, in conclusion : there are good managers. But they do not last long. They either quit or becomes bad managers which is the only way to climb in the hierarchy : lick upper levels asses and tell them that any problem is because of the individuals in your team. If you do that properly, you will never stay long enough in a team to have any impact anyway. Don’t try to deliver. Pretend you do it by saying it in a powerpoint. And tell your developers that everything is due yesterday.
They rather you be the bad manager that plays the hierarchy game?
For him, it was positive (and putting pressure is a way to make employee perform so they are happy with their own work. Pressure can be done in a good way like "motivation").
For me, selling is mostly lying and putting "good" pressure is hypocrisy. The strange part is that I realized that he was right. People love him. I’m just not a good fit for any hierarchical structure.
I best liked Peter Drucker's description of a manager's job: "to make individual strengths productive, and weaknesses irrelevant". It's less about the specific tech, and more about the core practices that really haven't changed too much over 10-15 years. And the higher up you go, it's more about the business and less about the tech, and those practices change even less often.
One of the things you realise when you get closer to management is that those policies shouldn't be taken too seriously if they contradict common sense.
Most people are only concerned with their own little corporate corner and doing the least effort that keeps them in paychecks. Trying to follow the spirit of a rule rather than the letter, or pushing for change to improve things overall, is never appreciated.
Longest outage i've ever seen in my life.
But at the end of the day, they did the job they were paid to do and were clear about the looming impact. It's not their job to also wipe their clients' metaphorical bottoms when they were ignored.
> For anyone wondering why we don’t just lift the compliance restrictions, we don’t specify it. Their Compliance department does
after a year-long grace period:
> The machines came to end of life about 12 months ago, and the company being a multi-billion dollar operation managed to eke out another year of manufacturer support. Mostly symbolic as they’re not exactly going to release custom firmware for a handful of devices. They then put a set-in-stone tombstone date on support. 12pm today.
This was imposed internally by the company's compliance and legal departments, TFAA is the executioner but the execution would be contractually mandated:
> They require, and have specified, a zero-tolerance for device non-compliance.
This means an unapproved batched and drawn-out phaseout would be a breach of contract.
You could brownout or kill a few a few days before the real issue, though, potentially.
What one could try is to call the CEO directly. Or maybe try the legal backdoor: contact the general counsel, tell them that the contract says such-and-such, that you think the contract is well written and you intend to do what it says, but that the organization should be aware that it may cause a problem. If legal doesn’t know how to get the CEO’s attention, then something is very wrong.
> 4 meetings, 124 emails, and two phone calls a day for the last 14 days have warned them of this.
There's only so much you can do.
> If legal doesn’t know how to get the CEO’s attention, then something is very wrong.
From the thread legal (and / or compliance) is the setter of the issue, and was well aware that it would cause issues (for a minority), but they were not in charge of resolution. And from downthread posts, they likely extensively documented their warnings:
> oh I’m absolutely backing the horse with the 3 miles of email threads proving this
And methinks legal and compliance had very much planned for the issue coming to a head, because they were getting fed up with being blow off, and having to shoulder the legal or regulatory risk.
A year to make fixes and nothing was done?
Legal is like, "We warned you every way we possibly could have."
"two phone calls a day for 14 days" is far from insufficient notice, to say nothing of the rest.
There's value in face to face, but not when it's a wide reaching announcement like this.
You can always start early.
People here blaming management but discussing how to do things at the last day. You don't
You give them a new device before locking out their old one
Do it for 17 weeks before the out of support deadline? The users start screaming, we still have 17 weeks left before the deadline, how dare you disable us early! And they get enabled again.
Do it for 17 weeks after the deadline? The compliance people start screaming, you have 1600 devices out of compliance, we need them shut down now!
https://github.blog/changelog/2021-04-19-sunsetting-api-auth...
I don't know if that concept would work in this case, compliance is it's own beast, but I love that idea in general.
> Alright, things are moving faster now, so I might condense-toot to avoid pollution.
But I guess the chronically online need something to complain about.
During much of that time, "who paid for what" was a big issue. The thread alludes to the issue: IT says "you need to buy new hardware every X years", department already has less than no budget, has no budget for new PCs and perceives no need for new PCs for workers that could get away with much less than they're using, now.
It was a funny little game that was played because IT would get dinged in their compliance metrics if staff was out of date (and staff hated old hardware/blamed IT), but management would get dinged for spending too much and have little incentive to buy new hardware until the last second. Meanwhile, C-Level executives on both sides get to say "your problem". The difference, here, is that someone gave IT a pretty large sledge-hammer and permission to use it in order to force departments to push for more budget. In our case (and I'm sure others), a bit (a lot?) of non-compliance was normal.
Personally, I think the take that "IT should own the budget" isn't as great as it sounds. It solves one problem: distributing the payment among budgets creates a "shared responsibility" that ultimately becomes "pass the buck". It also happens low enough from C-Levels that "they don't have to think about it."
Having IT own the budget solves this because at least one C-Level is going to have to account for a large enough expense that it's likely to be a little better planned for.
It won't always be better planned for ... depending on the company or manager, it won't often be better planned for. Unfortunately, the consequence of this poor planning only extends to the IT budget. Since compliance is non-negotiable, the largest line-item on the budget -- IT's staff -- is the next hit. In the former model, "making the budget deficit up" is naturally spread throughout the company, in this model, it all hits IT.
The way this is communicated to the users and what actions they had available to them makes all the difference here.
A week later, the customer notified us that they were still seeing some traffic on the old URL, but all they could give us was the IP address it was coming from. Unfortunately this IP address belonged to a server that hosts a lot of our smaller applications, so it didn't really help locate the offender. So I just added a firewall rule to block access to the IP address of the customer's old server, and sure enough I heard the scream 15 minutes later. Removed the rule to get that application back up and running, got it updated to the new URL, and all was good.
A few months go by, I decide to check again. Still hasn’t been fixed, emailed again, acknowledged again. On and on and on. About a year went by for them to finally implement this fix which should take all of 10 minutes, I mean at the very least all you have to do is introduce some entropy into the gps coordinates of the user. Hopefully I am the only one that found it.
It’s pretty astonishing how much people just don’t care even the C suite.
IT admins can set a policy like "must have 6 number PIN for login", if it does not then it is out of compliance.
This can mean nothing at all but if the company wants to act on it, it can.
But they will absolutely blame the dog.
"Ow! Why did this happen?"
well don’t do $that then
Me: If we do A, B will happen
Client: Do A anyway, we'll deal with B later.
(time passes)
Client: OH MY GOD B HAS HAPPENED
So in the end, this is just one piss poor managed division abusing another piss poor managed division. Who gets the heat? Probably the lowest level people.
Why "wipe" them? That seems unnecessarily punative.
You can see the "don't give a shit I work with a predatory organization" oozing from everywhere.
The security guy is trying to claim that they've sent out many many notices, but really this is just an excuse to abuse other people in a machiavellian abusive organization.
"Service Desk is now aware that everyone else except them was aware, and now IT is absolutely incandescent." Whoops, missed an email and a meeting in there bucko.
And it's the SECOND company where this was "implemented" or "specced"? This sounds like someone checked a box or compliance or ass-covering upper management slid this under the table, but all the people it ACTUALLY AFFECTS didn't get any input or opinion on the matter. And when push came to shove over funding it that person had probably moved on to bigger and better things.
So since you get to do it, you seem to be gleefully doing it. Great job.
There was something in the thread about the devices coming out of support by manufacturer, which was already extended by a year.
There are better ways to handle this, when sending the messages out. If the deadline for compliance was 31 Jan, then when sending comms out say the deadline is 31 Oct, and machines would be wiped after that. Then start wiping them, 10% of machines on 1st November, another 10% on 8th November, etc.
There is not necessarily any Schadenfreude in watching and reporting it. No one really needs to be taking pleasure, it's just hard to not pay attention to a train crash occuring in slow motion.
It's very natural to want to talk about something this stupid/bad. Rubber necking is extremely human.
Now everyone in the affected chain gets a black mark on their "permanent records" and gets exposed at a time when likely layoffs are coming.
What I don't hear is "why can't they upgrade, and how can we help them upgrade", it's WE TOLD YOU, NOW YOU SUFFER.
Here's the kicker: it's 1600 devices. Ok, so they've been told for 13 months to do this. Well, let's do some math. That's 260 working days. Oh look, about 1600 working hours. So if you guys had simply upgraded a device an hour over the last year, this wouldn't have been the problem. Yes, that's not fair, but neither is what the person doing.
Security is the military arm of compliance. Finger pointing at compliance is a bit mendacious. Saying LOL it's not my fault, it was compliance. NOW WATCH ME DROP THE HAMMER BOOM.
I mean, I guess the guy is saying LOL I'm outsourced and not even in the company HAHAHA. Still, eff this guy for taking a bit of glee in this.
That is not really how I read it; the devices got an year extension because people had already failed to refresh them within the standard cycle. From the sounds of it these are typical workstations etc, their support cycles are very predictable and if you bought some crappy ones without predictable lifecycle that is on you too. That extension should have been wakeup call, the process had already failed then.
"If you had simply spent nearly three quarters of a million dollars of your own money, done 40 weeks of volunteer overtime on top of your normal job, without any purchase approval, without the authority to do that, and no guarantee of seeing that money back, this wouldn't have been a problem, so fuck you"
is a terrible take all around.
> "What I don't hear is "why can't they upgrade, and how can we help them upgrade""
We know why they can't upgrade, because the departments responsible for purchasing the upgrades won't agree to spend the money. This isn't something which can be helped by more technical input.
> "Finger pointing at compliance is a bit mendacious"
"mendacious: not telling the truth; lying." - nope, wrong. Legal and Compliance say it must be done and you must do it, and have the authority to do that. Pointing fingers at them is honest and appropriate, that is where the instruction is coming from (legal) and the reason why the instruction exists (compliance with internal or external regulations).
Clearly the compliance team (or whoever is implementing it) has failed in its communication
It's more that security teams tend to have uncooperative, aggressive, authoritatian, and punative dispositions. I think ye old security industry had its roots in three letter government agencies which are used to conformance, policy hammers, and enemies of the state.
But when you add that to an organization already rife with infighting, dissatisfaction, and frustration, it will just lead to more resentment and your employees become your enemies.
The biggest security threats these days aren't leet hackers exploiting 0days, or even the county password inspector conning his way in. It's overworked angry pissed off employees leaving the door open. It's like Princess Leia said: the tighter you squeeze, the more people you lose.
If compliance and legal say to wipe the laptops, and everyone with a budget was aware of it for a year, it's not reasonable to put the disaster on whoever was in charge of implementing policy.
This is not a Petrov situation, you're not saving the world by going out of your way to be the person that will defy Compliance today, just because the policy is really dumb.
The people locked out would be shortsighted to blame the random security guy. They joined a big company with a very strict compliance machine, not a startup where you move fast and break things, then ask legal for forgiveness.
Big organizations are dysfunctional, news at 11. Don't blame a random IC for executing policy after considerable warnings. If communication is so thoroughly broken internally, and no one wants to take responsability for necessary spending, it's not the job of some random security guy to fix that internal dysfunction.
Potentially leaving company confidential material on non-compliant devices is not something Compliance department would want to allow
Don't fuck with compliance I guess?
You're proposing to remove the devices from the cloud management service, probably including admin lockouts and anti-malware software, but leaving all the data on them?
I'm not sure I could tell if this was truth or fiction.
The depiction is so close to IT / compliance-office revenge fantasies, so fiction seems plausible.
I concluded one of the last year deals in 8 months total from the first Mail I sent. Fortunately, 1 day before deadline (31st Dec) there were 4 different departments heads (each at least 2 level above my rank but still below C-level) involved with extended working hours on 30th December…… Ha ha.
So when the next renewal comes up, I am gonna kick-start the procedure 12 months in advance. :D
For my mental peace.
1. BYOD is a terrible idea. Work and home devices should be separate, even at the cost of multiple devices.
2. More than emails, they need push notifications and background wallpaper changes on the machine with specific instructions before doing anything terrible. Soft power configuration changes can do much more without potentially destroying value.
3. It's much easier to lock out users from their account with a message about mandatory remediation steps to regain access.
4. Issue newer machines on a lifecycle with an easy return program. When people get new machines, they'll almost always cough up the old ones. This solves the issue and maintains fleet health and reliability.
5. Consider replacing laptops with VDI and Chromebooks where possible.
You could have your own work device at home, separate from "personal use" items. The difference is that you can upgrade & reclaim hardware without having to bring it in to IT.
Where I work, which is a much lower-stakes environment, talking about our customers' issues or choices in public like this is a huge no-no. I'd get fired if someone found me out. Especially since if the customer is large, and the decisions have anything to do with my company's revenue, it could be considered MNPI.
All the management there fucked up. Possibly with the exception of Compliance/Legal and IT.
2. We did it.
3. It was done.
Were the meetings, emails, phone calls had the right people in them? Was the escalation up the org chain? Unclear from the tweets.
Now you're telling me!
It is a "backdoor" into corporate computers so that IT can install programs, reboot, install/force updates, run commands, wipe devices etc...
I don't want to "use the Mastodon web application", whatever that means. I just want to read a web page like a normal person.
Sigh Does anyone have the article text handy?
A hammer can be an important part of building breathtaking architecture. It can also be used to gouge somebody's eyes out.
Just like JavaScript.
EDIT: That "web app" was very nice to use.
And server side rendering can be as ugly, it is just less visible. We can be really lucky that there is a universal language on the client side for web browsers. Even if it is often abused, it is very much worth it.
Maybe not the most elegant language, but probably one of the fasted to develop with.
Do you genuinely believe this is still an open question?
I won't dispute the argument that maybe it was a mistake, but to me it seems indisputable the ship has sailed.
I might buy the argument that any "mandatory" websites - government, library, academic - should be operational without Javascript.
But in the casual or entertainment domain, noone is obligated to provide their users an operational Javascript-free website. If you can't read something without Javascript, that's a you problem.
Some websites don't function without JS. I either enable it on a case-by-case basis, or avoid those websites.
the 2000s and early 2010s era internet ("Web 2.0") was richer with Javascript WITHOUT all the gargantuan trackers and auto-playing videos.
It's banned in the rules on the site. No-one wants to hear about your off topic journey of self discovery. Maybe Reddit does?
The entire premise is absurd. What application spends more time on SSR than data fetch, or other? Exactly freaking zero that ive ever operated.
Fwiw, I'd like this site to have fallbacks for non-JS users, but it's a heck of a lot of work to make everything twice and nobody asked yet or contributed a patch for even basic functionality
"Show post content at standard post URLs when JS is disabled instead of just 'enable JavaScript' message, since this is already done for /embed URLs #23153"
Why are we live-tooting (ugh) our client's private disasters? The indiscretion is staggering.
This account casts its author in as bad a light as it does their client. I wouldn't want to work with either.
Indeed. I'm glad they are, because it is fascinating but very odd. I'm sure it's not hard to identify the customer either.
OP seems rather sure of the opposite:
>But I felt I needed to address one particular concern that has been repeatedly raised. That of the identity of the company in question.
>I’m a professional, and I’ve been doing this a long ol’ time. There is no way I’m going to risk the identity of the company, or my reputation, or the potential legal consequences for some interaction on social media.
>So to clarify, enough details of the incident and those involved have been changed to protect their identity and everyone else involved. I am confident that you could work at the company involved and not even be aware this happened, even after reading this Partly due to scale and partly due to managerial secrecy.
If the writing was nasty and exposing specifics, sure, but it very much is not.
Journalists and other outside observers can and should write about corporate incompetence wherever they find it. When you're in a paid position of trust, though, talking about your client's failings is tacky.
I agree that the writing isn't nasty. The specificity is the key. I guess to some people this came across as sufficiently anonymized. To me, it seems like anybody working at this place knows that the auther is talking about their company, which is a problem in and of itself. But it also means we're just one equally-indiscreet reply away from knowing exactly who this is (something like "yeah, I work here and..."). Though I really don't think that even that much additional info is necessary to deanonymize this. Just a hunch; obviously you disagree.
Do you actually truly believe this?
What I do think happened is that they saw how much pain it would cause people, then they looked at some fucking dollar amount and some spreadsheet and said "that's fine". I think that's fucked up.
Don't get me wrong, I don't dislike this person. It's not a "boycott". I'd just prefer not to be their customer, because this story goes a bit over where I'd draw the line of oversharing. I don't imagine that it will every actually come up, and I hope it never does.
Also he made pretty clear that he was anonymzing it to the point it would be incredibly difficult to tell.
I don't think the reporting being 'live' makes it much better or worse, though it probably wastes more of the readers' time.
* Seems I've misunderstood; I was corrected downthread.
20% can probably handle some legacy stuff that requires an old computer. Or a migration.
In addition some can be management, so they wont be making decisions for some time.
Nuh uh, go read the thread.