When Employees Use Software That IT Hasn’t Approved
hbr.org
hbr.org
One somewhat disturbing trend I've seen at some of the largest corporations- cut/outsource IT support staff to near egregiously low levels to "save money". At the same time kick off 7-9 figure ERP/consulting projects that at best provide fractional value to the organization.
Of course there are counterpoints to this. One of Houston's major pipeline operators pulled off a digital transformation and actually ended up with well designed, highly integrated and easily maintained systems. It took about 5-7 years and had a few reboots, but it eventually landed. That brings me to my final point. These projects often have a timeline that is divorced from reality. Whatever time frame you think a major IT project will take. Double it. twice, then add 50% and you are close. It also seems that C level folks are hesitant to hire boutique/small shops that have industry experience and years of experience in favor of big consulting. Nobody every gets fired for hiring Accenture/Deloitte/PwC. What usually happens in the non trivial niches is that these big shops sleeve the boutiques through them to get things done...
I don't really have an issue with CXXs being ignorant about a subject. No one knows everything. What I do have an issue with is when they act like an expert ignoring all the people who are actually experts in a particular area. It'd be like me going into a room full of IT people and saying EBITDA a bunch of times claiming to be an accounting wizard ready to lead a major initiative. It's frustrating but I've learned all I can really do is smile and watch the show.
It's not like that, it's often exactly that.
I'm extremely fortunate in that while my boss sets the goals he never specifies how they should be achieved.
That means I get to implement them as we need them.
I'll never underestimate the value of smart management :).
>It's not like that, it's often exactly that.
And it usually is sold also from within from know-it-all primadonnas who want to climb the ladder. Or at least put something "spectacular" on their CV.
Get promoted/leave for more money after 18mths.
2 years rolls around, everything is on fire but the guy with the matches is nowhere to be found.
Sounds very very familiar to me.
When our sysadmin left, we migrated all our websites from leased hardware servers to cloud hosting and were able to use that head count to hire a developer instead, who has built great new web apps for staff and customers.
I understand the temptation to be cynical, but these really are useful tools. I say embrace change; it's fun.
Recently, I even told him just to tell me what he wants to achieve because his implementations do not make sense and its my job.
At least my boss also let's me do it my way, still annoying.
That doesn't mean they're wrong.
Actual experts get to the root of what they need, and find ways to solve the requirements, even when they seem impossible.
Mandatory response viewing: https://www.youtube.com/watch?v=B7MIJP90biM
Which, of course, involves talking to actual users.
Perhaps not, but it does make it substantially more difficult to be right.
To provide a prospective as someone who works for a consulting firm like the ones you've mentioned... Hiring the "big" firms versus boutiques is a lot about a perception of risk, maintaining partnerships (procurement with new vendors is a nightmare everywhere), and leveraging experiences across other F500. For big implementations, think any ERP, our consulting teams can number 100+. Overkill? Probably, but there are few boutiques that have those kind of resources. And each of those 100 have more experience doing implementation work across a range of companies. It's a bit of a vicious cycle for boutiques where they will ultimately struggle to be competitive in these types of bids without seriously underpricing their services, which ends up meaning fewer resources or a less comprehensive scope.
As a side note, when projects go south, the company CIO isn't getting fired, but I've know many leaders (from the consulting side) getting let go for botching multi million dollar contracts.
Many ERP implementations fail because of wrong assumptions about the business, or inflexibility of the client to modify their business process to fit the ERP. Enter expensive customizations.
These types of projects also affect peoples' jobs and that can bring up fears of being replaced that can quickly derail morale on a project. Successful projects empower the people with hands on the keyboard who know and live those processes, to own and define their future process too. When the consultants or executives are making future business process decisions without that experience, it is risky.
I see the opposite too, they just staff up on tons of IT people thinking they have a resource shortage, and end up with massive departments that deliver just a little as before.
> It also seems that C level folks are hesitant to hire boutique/small shops that have industry experience and years of experience in favor of big consulting.
The reason this makes sense is because they need to work with companies that have enough resources that they can be really inefficient and have enough capital that they can run for long periods of time and not go under. It’s more of an insurance policy, the quality of the work would be better at the smaller shop of course but they likely couldn’t complete it due to bureaucracy.
Why? Because our organization has been quite forward thinking about allowing managers and executives to source the technology they think they to succeed. As this article advocates for, IT was largely consultative rather than dictatorial, and a lot of business units were able to pick what they wanted.
But what this has left us with is dozens of places where customer data was being stored, some of them now past their end of life. No central visibility into customer experience. People getting multiple copies of the same email from different departments using different email platforms. Poor deliverability. Subscriptions on random credit cards that suddenly turn off because the person left and no one knows how to get into the admin account and update the card.
We hired a boutique shop to do the Salesforce implementation; we're not scared of doing that. Unfortunately this time it did not pay off... their performance fell off, to the point that they couldn't even reply to emails on time. As sometimes happens with small firms, they grew too fast and exceeded their ability to operate. We can't wait for them to figure it out... so here we go with a big dog firm. Let's see how that goes.
Maybe I'm lucky in who I work with, but I find the "add a bullet point to the resume" take to be maybe a bit too cynical. Tableau, Salesforce, data lakes, ERP, identity management, and "cloud" infrastructure each seem like useful tools if implemented smartly. (Note that I took out blockchain...)
Which in turn makes it easier to over analyse and identify user data, exactly what GDPR was meant to avoid.
(While many business people used to how things worked are unhappy with the changes, sometimes you really have to bludgeon a fix through the broken incentive structures that plague businesses.)
GDPR "cares" about identifying user data - how else are you going to protect it and control access to it?
As for over analysing it (whatever that means), it really doesn't care too much about that as long as you have explicit meaningful informed permission from the user to do so, protect it properly, and let them control what happens to it (both during and afterwards).
Who's going to run the thing afterwards? Will the bigdogs deliver something that you can maintain, or is that generally against their own interests?
Finally, who's gonna secure all this customer data? Are they taking that on as part of their remit? They rarely do.
Also, the move to cloud is largely driven by said IT teams failure to deliver much, you could a the CRM on-prem, but we know they'll likely take a few years to fail to deliver it.
Having been involved in IT management of NFPs, the low price is a significant draw, and the total lack of internal skills is rarely able to counter this.
If you don't have some sort of architecture function, technical risk management, application management, and data management these projects simply won't deliver value.
I'm NOT an expert but my understanding is that, in terms of complexity and cost, there's a whole tier above Salesforce where you're provisioning servers and installing Oracle or SAP. We didn't need and could not afford that.
And if you're thinking of smaller CRMs like Hubspot, Zoho, Sugar, Apptivo, or building one from scratch, well, we already had many of those. :-) Those are what Salesforce is replacing.
Our IT department is superb on metrics like security and availability. But they don't know Salesforce, and are not the right people to evolve the broader culture associated with data. The org hired a leader with experience doing this sort of thing, and he is building out an internal permanent Salesforce team which will own the thing after implementation is done.
Still, as an IT contractor, better you pay the big bucks for the projects. :p
In my view security shouldn’t be isolated at corporate headquarters but they should be close to the end users so they see what users need to do and help them to balance security with getting things done. They can’t just block stuff without providing alternatives or they will either hurt the business or they will be circumvented.
After about a year the network through that router started to slow down. When we checked why we realized that we had more than 40 wifi clients, as other (non-dev) teams learned about the network through the grapevine and actually the new employees from sales (the neighboring office) were told to connect to that network directly - as the process to connect to the "normal" network was too much of a hassle.
reminds me of a very old bash.org
We can't connect to anything internal from outside, there is VPN access but it seems a step too far to use that all the time.
Try to control everything is just futile. I think hiring and retaining best people is the best countermeasure to this sort of things.
Dropbox/googledrive is a huge security hole that is definitely blocked at most companies I work at.
There is absolutely middle ground if you have the time and resources to get it running smoothly.
It's not practical to sit around for a week or more while you wait for each piece of software to install. No dev will ever get any work done.
I really don't see how you can develop such software without having at least the ability to easily gain administrative permission on the machine.
The only exceptions are some parts of the C++ debugger and the driver development kit.
I'm not sure if your "almost ten years ago" is meant to be hyperbolic, or genuine... I can't even remember why, but I know the project I was on 6 years ago definitely needed visual studio to have admin access, and it was all standard C# app stuff (maybe WPF?)
A dose of humility might be in order.
Corporate IT can admin the box for corporate training PowerPoint gunk. You get another box to run what you have written, and maybe another to run the development environment. Those don't go on IT's network. You can run a private LAN around the office, not connected to the outside world, in which you break things as you please.
This solution is even good enough for people who are intentionally dealing with malware.
This stuff isn't that hard - but those of us doing it see the mad things that people do when they're given blanket, even time bound, admin access. They're the ones dealing with the support calls when then every SQL Server installation has been done differently with no details of what specifically was done. IaC works.
Now iterate that over 1000s of other instances and you see the financial reason why devs need admin.
Unless the VM is somehow sandboxed it's just another box on the same network. So the same reasons for me not being admin on the physical machine (e.g. to not be able to download and run untrusted software because it might spread something on the network) should apply to the VM?
An account inside a VM will only let you play in that VM.
Whereas your account on the host is available and automatically granted access to all machines, fileshares and services on the active directory network. If it got admin rights, then you've got admin pretty much everywhere.
Nonsense. You can have local admin rights that work only on one machine.
That being said, there are indeed restrictions that can and should be set on admin rights. Not that IT would know about it or that it would limit pivoting much.
Why not report your findings to Microsoft and get your bug bounty payout?
And if this is true, wouldn't they also just do that from inside the VM?
Let's assume for the sake of discussion that to do what I need to do I not only need to install the program that requires priveleges, I also need a few of my company network drives mapped, access to some company systems, internet access and so on.
In fact, several VMs.
Also, lots of stuff simply cannot be installed as a regular user, especially stuff that needs unfettered access to network cards or memory.
IIS development - Visual Studio needs Admin to actively debug IIS.
Memory tools like dotMemory.
Dealing with Windows Services.
Shit... dealing with Windows.
Because most of the non-insignificant ones still CAN'T be, under Windows, to this day. So special people get a completely separate account with pseudo-admin rights. I have to enter those credentials several times a day.
Then I spoke to a help desk guy, who said he had to enter his domain admin account password 40 TIMES a day.
What a waste.
No, we don't have an internal FTP site. No, I won't set one up for you. We use Sharefile for distribution so we don't have to do that. Your IT blocks it? Yeah, that's dumb. Go talk to them; it's not my problem. We're not going to do customized delivery channels just because your halfwit CIO decided to block every site with an upload button.
You are very, very fortunate.
The standard big-company IT person I see at client sites is not very bright or will-informed.
This seems to go hand in hand with a much greater obsession over IP. I've seen people actually threaten to start legal fights over whiteboard diagrams of little to no meaning at all. My guess is that outside the tech industry, new ideas are relatively rare so even very simple ideas feel incredibly valuable. This gets generalised to anything employees produce and is why there's no culture of open source development in most traditional industries.
Are they not rare even inside the tech industry?
See how all the big firms file gazillions of patents but actual patent infringement suits between them are quite rare. Patents are seen as a defensive posture: everyone knows everyone violates a million patents so by and large, mutually assured destruction is avoided. In a world where ideas were rare you'd see patents be treated as much more valuable.
1) I would probably still use different terminal color schemes, but being in front of a different physical box might be quite a good thing.
It doesnt introduce that many positives for lots of admin overhead, not just in maintaijg two distinct networks, but also in ensurijg interoperability when needed.
Yes, there are confidential data, but it shouldn't be any real customer data. Right?!? Frankly, given best practices from professional developers, stuff like cryptolockers just aren't an issue (blank the machines). Developers need admin, so building a network for them is actually a lot easier.
But it might be. Whenever you're doing software other than for purely internal use, you have a customer that gives you sensitive data which devs absolutely need access to - like requirements for the software you're building!
It doesn't introduce that many positives for lots of admin overhead
For the IT department sure. But I'm 100% convinced a big reason tech firms exist at all (given that every company uses "tech" in some way, right?) is simply that they know how to manage developers and make them productive. And IT policy is a huge part of that. Sure, the developers might be 1% of a typical non-tech business but they are the 1% that can give you a competitive and productivity edge, so their needs are in many ways more important than other types of employee who may not scale well.
Being a good software developer doesn't mean you're a good network administrator or good at desktop support. And even if you are, a developer is paid more so it's a poor use of their time.
I'm all for devs having admin access on their own machines, there are too many instances where it's needed, and reasonable exemptions from default policies when they conflict with their work. But a "conflict" would be things like anti-malware software making builds fail, not merely making builds 10% slower.
Given developer salaries, a friction like making builds 10% slower is hemorrhaging money. And it gets worse if someone is actually waiting for the software to be delivered, and the amount of money the company gets is correlated to keeping the schedule.
Working in IT doesn't mean you're a good network administrator or good at desktop support either :)
Because I don't want to, and that's not (and shouldn't be) a core part of my job. It's a distraction that will reduce my productivity.
I understand your argument and in principle I agree with it, but in my experience nobody cares all that much about the data on the primary network, so creating a second network that grants devs things like local admin doesn't seem to increase risk by much.
At minimum you should inform your direct manager of the situation so they can address it or accept the consequences that the work that depends on the restricted resource won't get done w/o circumventing company policy.
Cloning a production database full of private customer data for testing? Well, we can't interfere with a practice that gets features shipped and the team doesn't have space on their roadmap to build out synthetic data...
Not letting people do their jobs, or in how fast they can do their job. Employees are a captive audience, and if there was competition they would probably chose something else.
I tend to summarize my experience with IT in large companies and universities like this: IT is evaluated in "how secure things are", or "whether things are not broken". The only sure-fire way to ensure a system is secure and not broken is to make it completely unusable, so that people don't use it. If people don't use a system, they can't break it!
Never mind that it took double as long, and sucked the life out of everyone subjected to the system.
Efficiency!
Now what.
There is also an inversion of responsibility here. It really should be IT's job to 'adapt the tool to the process' instead of externalizing it to their staff. Until they can do it, the staff needs outweigh the lazy 'default no' of IT.
And I mean incompetent-- our network would go do down for hours, every day, and the problem lasted almost a year. It got so bad our "source control" became yelling "Whose got the latest main.cs?!" and walking it over with a USB key.
I realized we needed to disintermediate the IT group, and my boss was supportive. I ended up establishing our own, parallel IT department with a business cable line, some routers, and NAS devices. I outsourced any task to any cloud service (this was over a decade or so ago when that was somewhat unusual) — eventually hiring our own IT person even. People would ask for access to our network because it was more reliable :-)
Then things got vicious. They actively interfered with our group, trying to get us fired (they had some success with this previously). They were able to get our IT guy fired. Things just went on that way for years. They'd find some way to make trouble, and then we'd route around it. Once they even took my computer for two days, denied they had it, then returned it but locked me out of it. I was certain this was part of some scheme to get me fired, so I turned the computer off, zeroed the hard drive, and req'd a MacBook to work on.
They all eventually got fired when someone took note of their gross incompetence. One of their replacements was eventually fired for embezzling, proving that despite thinking we had the worst IT department anyone could imagine-- we really didn't.
Also it's not always so easy to up and quit every time someone looks at you the wrong way.
Deciding to leave doesn't mean put in your notice. It means look elsewhere. Leaving a job without another secured is always precarious absent a nest egg.
To look on the bright side, my boss was amazing, a truly unique individual whom I thought so much of, I asked to be the godfather of my child.
Despite the chaos of the IT team, almost everyone I worked with was great. Everyone I met was really good at something, but I learned a lot from the people around me and had the opportunity to mentor a young engineer who ended up far exceeded me, which was rewarding in itself. I have several life-long friends from that company. People who have been with me through a messy divorce. A particular friend, when my wife left, called or texted every single day for six months-- just out of the goodness of her heart. The text was always the same, "Hey, wanna get some coffee? How are you doing today?" There are beautiful people working in even the most toxic places.
When I wasn't learning about engineering, I was learning about optics, about politics, and about the real world, and once again, I was winning, which was intoxicating.
So why ultimately did I stay? Because it was comfortable being the smartest person in the room. Because it was fun having latitude enough to start my own department? Because it was fun being engaged in this real-life game and winning? Because I'm crazy?
Things got far nastier than I went into. Once the IT department was openly trying to get me fired, things got personal. Really personal. It would have been easier to move on, but, I hate being bullied. Allow me to repeat myself, I HATE being bullied, and that's what this was in the end, organized harassment.
Through a set of unusual circumstances, I ended up befriending the main IT guys wife-- the man who decided he could play with my life for no reason at all, and his wife had a crush on me. So I and slept with her, and I am still sleeping with her four years later. To quote that great boss of mine, "Revenge is a dish best served every day."
Luckily, S3 is not blocked. I set up a bucket and have them upload to the bucket, which I then download on my own computer. Mission accomplished.
Just getting free Windows applications procured and installed on the company laptop takes several layers of approval, emails back and forth, etc. Also the lovely forced password resets every 2 months.
This is true for the banks I've worked with/at. Security systems actively block you from the tools you need to do your job, let alone get the job done in an efficient manner.
Just because you’ve circumvented IT’s blocks doesn’t mean you won’t land yourself in hot water.
Sure, guys. Of all things to block, let's block the most secure one. That'll really improve our security posture.
At this point, I'm continually surprised they haven't superglued the USB ports.
It's possible to get around this by tunneling SSH over other protocols: http://dag.wiee.rs/howto/ssh-http-tunneling/. Bear in mind if you do this in a corporate environment, security will throw the largest book they can find at you.
However developers are very good at presenting it otherwise. Myself, I have run into issues where admin rights were needed, again always because of some poor installer. USB has been blocked as well and you can guess it, cannot do their work.
If they need admin rights all the time then put those particular machines on a protected network and not allow any other business work to occur there.
Unfortunately from a security perspective devs and system admins are probably the highest risk targets since they typically have access to servers and admin rights. At the very least they have source code an attacker could analyze, and likely have access to external services.
The reality is that compliance, security and usability are often in direct conflict that can only be solved to make everyone happy with significant work.
I’ve heard this before, but never with any detail. Can you explain further, or point to a resource? For example, clearly SOX doesn’t say that nobody can have admin rights - because IT does. And I highly doubt that the law says that only departments with IT in the title can have admin access. So what does it really say?
Mostly when you hear "because SOX", that person has never actually read the document.
These things are all based around "controls", which sometimes can be specified locally and some times are set by outside entities.
In my experience (SOX evidence collecting, and SOC control writing and evidence collecting), a well written control is one that covers the bases without being overly prescriptive or ambiguous. In the case of admin rights on computers, it is more useful to have the control be worded as "Only appropriate people who have legit reason to have high levels of access do" and the audit step is confirming and providing evidence that the people who do are documented and approved as having it for specific reasons, and that people who aren't supposed to have it don't have it.
I've run into crappy controls quite a bit. It's easier to push back on these when you're determining the appropriate controls than it is when you're the sucker who has to collect the evidence that doesn't, and won't, exist. Authentication and authorization controls are often some of the worst. A less useful/less meaningful control is one like "all accounts must have passwords and all passwords must be at least 12 characters long and be composed of a mix of alphanumeric and at least two punctuation characters".
The goal is to say "yes, we do this, and here's the proof" without any qualification.
Sorry, none of our accounts have passwords because we disable password authentication and use ssh public keys for authentication with two-factor via Duo. If you say that, you don't satisfy the control as worded (because none of your accounts have passwords, you can see this in the shadow file and sshd has PasswordAuthentication no, and this is difficult to explain to people who are not familiar with ssh, which is, unfortunately, a significant portion of the people who end up being put on audit projects). If you say that they do have passwords, you're lying/misrepresenting, which isn't good for an audit either. If you say you don't but have compensating controls, this doesn't look as good as it could because it is called out with an addendum/explanation and is a potential exception that needs extra consideration.
These controls should be worded more like "all users have their own accounts and accounts are authenticated using secure methods" with sub-controls specific to the environment/company saying things like "password policy is based on NIST suggestions as of YYYY-MM-DD and enforced via <company-policy-compatible enforcement mechanisms>". The point of the controls is to detect, catch, and re-mediate anomalies, it is for this reason that the controls need to be adjusted as standards change and the state of the art moves forward. The specifics and rationales for why something is in place is for policy documents, which means people usually don't understand why a control is worded the way it is and poorly worded controls make for really rough, drawn out audits.
Back in 2015, I previously talked about a specific experience preparing for SOX. https://news.ycombinator.com/item?id=9025958
The truth is that most of what people do in the name of SOX is cargo culting other people's ideas of appropriate controls.
In this case, you could give admin access to whoever you want. You just need reasonable documented controls around it.
One company I worked for had two lans. The admin lan and the engineering lan. Your admin machine would always work. The eng lan could go to hell, and it/management wouldn't know/care.
I think it worked well. It was kind of like having the common areas of the house neat and tidy, but giving the kids creative control over their rooms (and being able to close the doors when things got out of hand or guests came over).
I ended up leaving my last job over this and more stuff like it. They had a url filter in place for instance, that would randomly block access to our network resources. It would never be the same one so every now and again your stuff would start failing and you would lose a few hours debugging until you realized.... GAH! Then have to email and lose the rest of your day waiting for them to fix it.
Never. Again.
Telnet is pretty hard to procure since it's not included by default since Windows 7.
Saying yes and fixing the problem quickly, without analysis, will often UNDERSERVE your company in the long run, because you only fix a specific problem for a specific person, functional group, or division.
You realize you need a kitchen sink, it turns out there are 2 other teams who already created their own semi functioning kitchen sink which they want you to adopt, but doesn't fit in your kitchen, and the CIO is working with Home Depot to create standardized kitchen sinks for the entire company which will be ready in 3 years (realistically 5 years, or maybe never), but your current sink is leaking and is flooding your kitchen now, so you do what you can to fix it.
Basically a principle of asking for forgiveness rather than permission. Does it cause issues? Of course, any solution to today's problem becomes tomorrow's problem. Now there are 3 semi functioning kitchen sinks in your company, but at least they are functioning
You are adding redundancy and overhead elsewhere. Now you have 30 people at your company playing account admin, managing passwords and permissions to different platforms. "Oh but it only takes me a couple minute a day" x 30 = a part time admin job.
People tend to ignore succession too. What happens if trains hit people, or they land poorly skydiving, or they move on to another job. Oh they signed up for that critical service with a personal gmail account?
Im not even advocating kitchen sinks in my post, and neither was the article. The article was pretty explicit that there are the two types of projects big and small, and to let small projects be agile, but still keep them visible and sanctioned. They dont need to be shadow projects. You are creating a false dilemma. Its not "either it goes into the kitchen sink or its shadow IT." The machine may very well recognize "hey two other teams are working on variants of this, can you all get together and share notes. We would also like copies of everything you have all learned for when we re-implement this in three years." A good IT rule might be "sure you can sign up for your SaSS service and manage it yourself, but if it supports SSO, we are implementing SSO or you dont get to sign up."
If you need a stop gap temporary project to last you until the ERP is implemented, good management will recognize that and support it as part of a continuous improvement cycle. They might buy something damn well knowing that they already have a plan started to decommission it in two-three years.
Agreed. Not saying it's a good solution, just saying it's a solution. Can you get burned by that solution? Yes. Usually is the reason many companies have extremely stringent and inflexible IT policies, because they've been burned hard in the past.
> good management
I feel this is really the critical component that is needed. Management, just like any other skilled labor, like programming, will always have an overabundance of mediocre performers and a shortage of 'good' high performers. Meaning we run into issues described here.
The article we are commenting on is trying to teach management to identify when things need to move through the entire IT project lifecycle, and when to let them mostly self sustain (except obviously nobody should ever sign up for a single thing no matter what it is, if it supports SSO, until the company connects their identity management to it!!)
As a dev, I will get my job done, and if that means breaking company policy, I'll do it. I've used SOCKS proxies to get around company firewalls because the whitelist time is measured in days, and I have minutes. I've used my phone as a hotspot when IT broke enough of the internet to be a bother. I've used SSH tunnels combined with Nginx reverse proxies to get around routing approval processes. I've even built a port forwarding service because IT took too long to approve and implement their own, and it has been in production for years (I don't think IT is aware of it, though I should probably get it all cleaned up at some point).
If management decides security is important, but not important enough to make efficient, employees will work around the limitations.
One day a grey beard took pity on me and installed a Linux VM where I was admin, copied the security certs from the Windows host and I could access all corporate resources at my leisure. Never logged a single IT Helpdesk ticket after that.
Then fullscreen Ubuntu in a VM from then on. Slower but no restrictions. Even better when Workstation supported multiple screens.
We had an outside firm making security decisions and if there were any security issues it would end up being on them. So as long as they did not allow us to release any products and or install any software they could not be held responsible.
I made friends with a lower level contractor who told me off the record to use my judgement on what to install to get the job done, because the security department would never approve anything new unless directly instructed to by the CEO.
Fast forward three months, there was a major security flaw on our website (also built with outsourced labor) which allowed anyone to access private data without a login.
A few of us had reported to the security department that the code running the website was so poorly written that the odds of being insecure were close to 100 percent. We suggested upgrading the website and rewriting the code, and management was on board with this but security department refused to allow us to use any new frameworks since they were not approved. Of course in a matter of a few months the site was hacked and millions were spent as a result.
I quit this job after we were unable to release several products after a year even though we jumped through every hoop we needed to. That department killed all innovation.
But in most software shops, the workers are probably more qualified than the IT department to be making decisions about what applications to use, and what kind of security they need. IT is just there to make things run and fix them when they break. They don't really need to offer guidance.
TechStop is/was great. But Windows users had locked down workstations where IT whitelisted binaries. I assume the approval process sucked about as much as normal.
Many Google employees use desktop Linux which is basically unheard of outside the tech world. That by itself simplifies things quite a bit. Not many people writing viruses posing as screensavers for Google's in house Linux strain. Anyone who cracks that is probably an APT attacker and those require different approaches.
You want to treat people like responsible adults, but they aren't the ones who have to deal with the fallout. Developers know the score for the most part, so full privileges are expected with the caveat, if it all goes bad, we are wiping the machine[1], not doing a recovery.
IT dreads the moment we are called to account for something some user decided they needed to do.
1) most developers understand backup tools and code control - those that don't, well...... with great power comes great responsibility
Thank $deity for the rise of opensource.
Somebody has apparently lost touch with who the customer for IT is.
It's fear of malware.
It's fear of data being locked up in the format used by an employee's personal tool.
Above all, it's fear of being held responsible for any of the above.
It's also a fear of being not needed.
Having the resources to install a CRM for someone this year is a fundamental part of any properly equipped IT department.
Can you call your IT security strong if you are dangling irresistible incentives to break the security?
However it does often seem like IT doesn’t consider SaaS solutions - they always want to build something their selves without doing cost analysis.
If you need rules to force the business to engage with you, you've failed.
Thats the situation the CIO had to respond to. Just because its not part if your role to consider security implications of these SaaS services doesnt mean he's out of line for doing so.
Something like Azure AD, ADFS, or 3rd party (that you assume to trust) like OneLogin. In all cases you would never enter your password into the SaaS service you are redirected to a secure portal controlled by the Auth Service, a token is then issued back to the SaaS service
Further it would be recommended not to use an elevated account and certainly not something like a Domain Admin account for those services
The problem is typically that cookie-cutter solutions don't necessarily map what the leadership requires: either the cost is too high, the knowledge gap is massive (e.g. the tool can do everything, but requires specialized knowledge of an obscure DSL and implementation details only three people in the world have actually mastered...) or the security implications are nontrivial.
To be fair, I do know also people who will always prefer to build their own anyway, because it makes them feel more in control (which they are). It's the CEO's job to rein in these tendencies when necessary, though.
It seems that the whole point of the article is that while IT thinks that it is serving the customer (the business), it actually lost track of what it was needed to: helping it succeed.
var sql = “select * from Customer where firstname = ‘“ + firstname + “‘“;
I was the lead dev at a medium size non tech company, and the hoops I had to go through to get anything done dealing with the “security team” was ridiculous and of course I didn’t have access to production to troubleshoot for awhile.
I had ultimate control of all the code that did go through the process. If I were to do something stupid or purposefully malicious, while I didn’t have access to the environment - my code did.
As far as someone mistakingly installing a “crypto wall”, if a user can download a program that doesn’t require admin access, that program has access to the user’s files. The system can be restored much easier than the user’s data.
If the user has read/write access to the network, so does anything the user run.
A user with local admin access and installing malicious software has a higher risk of propagating everywhere.
A sibling post just use an old example of the ILOVEYOU virus that didn’t require admin access to run or spread.
Somebody clicked on a link, somebody was spear-phished.
And if that happens, and if the user gave up their username and password. The perpetrator has access to everything the user has access to. The perpetrator will probably target a user with the access they desire. You say enforce two factor authentication? That’s also easy to scam out of user - get then to tell you the 2FA code. It was happening to Uber drivers.
If you can’t trust the user not to do something stupid, you can’t trust anything that the user runs not to do something malicious or be tricked into giving up confidential information.
IT policies at large corporations aren't implemented for developers (only). They're implemented for everybody. For every developer, there is a salesperson, admin, manager, or HRBP who will do things they might not fully understand to be "bad".
I came into the industry in the late-90s and still remember the chaos that the ILOVEYOU and Anna Kournikova style viruses caused in corporate offices. Non-technical users didn't know that Windows hid file extensions by default. They didn't think that opening a picture could start a shitstorm the brought the corporate network to its knees. Fun times.
- It spread by reading the person’s contact information which doesn’t require administrator access.
- It also corrupted the user’s files and didn’t require administrator access for that either.
It's really easy to secure / fix / support a system by making it nonfunctional and redefining that as success.
Lol! I'm stealing that.
IT's role is generally to support the organization. The organization is its customer. For the most part, it doesn't "work with," but it supports. In any organization, there's a complex network of who is a customer, who is a client, who is a peer, and so on.
There are places I'm not IT's customer, but they're the exception rather than the rule. If IT isn't providing a service I need, then that's a failure of IT. At the end of the day, the fallback is to purchase the same service elsewhere. If IT needs to know about that (e.g. for audits or security), it's fine to have a process for that (I report to IT, IT verifies what it wants to), but if that process becomes an unnecessary roadblock (IT doesn't want to compete for my business, rather than a core security issue), either people will circumvent that process or the business will take a hit.
The customer-provider networks vary on business. In some cases, engineering is the customer of marketing, and in some cases, the other way around. You have companies where marketing decides what to build based on customer conversations, and engineering builds it. In other cases, engineering decides what to build, and marketing sells it. And then you have all sorts of cases in between, from synergistic peer relationships to all sorts of balances where one drives but the other informs.
That doesn't change the gross organizational dysfunction being described in this article.
The primary question is one of purpose: someone in a support role is hired to keep me effective and productive, and evaluated on their ability to do so. I am their customer. If I win, they win.
The goal of IT isn't good systems architecture or innovation -- it's me. Supporting me well often requires good systems architecture and innovation. It also requires compromising those at times to my goals, having clean transition strategies, and similar choices as well. Those decisions are made based on their impact on me.
I absolutely think you are wrong that innovating is derived from your needs, a technology group driving innovation could just as well include obsoleting you. An IT department can bring new business ideas to a leadership team, and arguably their leader is a part of that leadership team, at least until every c-suite person is tech savvy enough to obsolete the CIO position.
All of those things are there to support the business, not the other way around.
BTW -- if IT's goal is really to protect the business, then you should find & discover the ways people are getting around your fences, because the first thing that a malicious actor is going to do is find & hop those same exact fences. These people finding security holes should be lauded as whitehats finding your mistakes, not people to be punished for not following rules.
Yes but often if that whitehat reported it and they closed those "holes" you wouldn't be able to get work done, because you can be sure that they wouldn't go the extra mile to create a system where you can do stuff, they'd just close the "holes".
There's a reason they went around the CIO. I suspect if the CIO had met with this person they could have learned and helped.
"The CIO admitted that he had been approached and explained that he had informed the VP that IT already had a project with SAP to deliver what the VP needed. “Yes, but that won’t be ready for me to use for three years, and I need something today,” retorted the VP."
The manager had a valid and valuable reason ("increased revenue $1M per month") to require a service that CIO and their organization was unable to provide in a reasonable timeframe, but other companies on the market were.
Try explaining to a non-technical person how how a desktop background image isn't a program so it's basically safe to grab from anywhere, while a screen saver is definitely a program and usually unsafe to get from most places, and a word document is some times a program that might eat your computer. Training could involve things like "which of these 5 webbpages would you consider it safe to download and run executable from"?
Having too cumbersome rules around security just means it's ignored or circumvented, increasing risks.
I actually was talking today with a customer during a logical assessment about if I talked about downloading malware in the training. I dedicate an entire section to downloading documents, but I don't really give people the information you're talking about. I tell them how to avoid ever having to download anything, and if they must do it, how to try and do it properly. All of this is ended with the process on how to report incidents because eventually something bad will happen.
As a company you kind of expect this to be solved at a number of layers. Endpoint management should hopefully help resolve this issue. Restricting web access where it makes sense can help. Sec Awareness Training helps keep people aware. Etc, etc, etc. You hope your controls are what save you from incidents, because there is no way you can effectively train your entire company on security topics to a degree that they can make good, security conscious decisions. That said, many of these SAT's are really just checking compliance requirements, because thats the real need. I put my own training together starting with what I know needs to be covered for compliance (pii handling, passwords, acceptable use policy, common threats, security incident response reporting, etc). Anything else that makes it in is purely because I have extra time and I know it to be important.
So I change my DNS settings to use an 8.8.8.8 dns first, and my company dns second. Now I can access both archive.is and sites on the company network. Excellent. But in doing this I circumvented all the DNS filtering, not just for this site. The reasonable thing would have been a warning like a https-style warning "Are you sure you want to continue to this site"? Or a way of whitelisting, perhaps temporarily, a single address. Instead my options were to ask an administrator or disable the whole security feature entirely. (Or connect/disconnect the VPN temporarily every time I needed something blacklisted, but that didn't feel like a good solution).
And training on different file/executable types isn't effective. Many high profile phishing attacks have been carried out using malicious attacks embedded in innocuous files like word documents or PDFs. The only way to actually prevent malicious code from a download is to prevent the download in the first place.
I found myself tired of sitting around. I found a TSR / print spooler that would use RAM and offload the process of printing. This allowed me to keep working. My productivity (as a temp) was higher than many others including the person I was "reporting to" at the company.
They found the print spooler, labeled it "unapproved software", and I was walked out the door.
The funny thing is, a friend at the time (and I didn't realize it) was higher up in the management. He reached out to me on a multi-line BBS that was popular in the area and offered me a full time job a few days later. I was in school and obviously declined.
Working the rest of the summer for a Chemical Engineer in Martinez/Benecia ended up being incredibly more interesting. So it was a net win.
Now, the company I work for, ostensibly a _software_ company, got its ISO certifications, which meant policies and procedures that make developing hard or impossible again.
How does a software business _successfully_ implement stringent access controls while still allowing for efficient software development? I'd like to see/hear what works.
We have to deal with the fallout when they screw something up, there has to be a happy medium somewhere.
Developers don't need admin rights for much of anything in this decade. No need to bother with that.
Common software has to be made available in self-service, so developers can install development tools like notepad++ or visual studio.
Deployment is usually the challenge because you have to store binaries somewhere, copy it to some random servers and finally execute it, each step causing numerous security headaches, so there has to be some approved tooling to handle that.
Please defend this position.
My experience, mostly with Linux-like tools, is that those tools are built with the assumption they are being used by someone who knows what they are doing, and that they have the appropriate level of control of the machine -- they are tools for professionals to build tools.
If you don't have rights to install or execute them, you're done. You can't make any forward progress.
The most maddening part for me is to literally sit around helpless and unable to do any development because you need to wait for your IT support ticket to be looked at. Then having to explain to your manager why work is behind schedule.
However, idle time alone doesn't seem like strong enough reason to open discussion on changing IT policies.
Boss: "We need access to the database of our primary application that you wrote for us so that we can pull the data into this new tool to track progress."
IT: "No. Not only can we not give you access to YOUR data in YOUR application that we wrote on YOUR dime, we will not allow you to have this new application written by someone who isn't in our group. If you wanted something like this, why didn't you just ask us? We would have written this for you."
Boss: "We had a meeting about this over a year and a half ago, and you told me that you didn't even have the time to discuss it further."
IT: "... Well, we're still not going to let you do this."
IT is effectively holding the rest of the company hostage, and the corporate technical debt is becoming epic. So skunkworks solutions will continue to be developed.
That's what the CEO is there to figure out.
Still possible to support but requires a different model e.g. one where IT delivers a new, unconfigured workstation to your new team member and it's up to them to build it. If it breaks, their loss of productivity is their problem and not ITs.
Authority for something (e.g. software selection) must go in conjunction for responsibly for consequences arising. Those things must always move in lockstep to avoid perverse outcomes.
The configuration was so bad that it exposed the company's network to whole wide world. Google contacted the company and after searching high and low IT security managed to track the pc down and take it away. Finance team promised to hire someone with skillset required to run the software in a closed environment. And that's how I ended up getting my job.
It says a lot about the struggles of enabling change in structures with strong silos that (what are probably among the most risk averse type of) people would go to such lengths.
Maybe not for this particular project, but another interpretation of that is "who cares about security if we're making money" which is a very dangerous argument as well.
"Who cares about your pie-in-the-sky infrastructure project, my boss continues to measure our real performance with basic accounting, and is expecting to be able to report on growth each quarter, which I can't help without tools" seems to be a bit closer to the argument posed here, IMHO.
In that story, is the blame on the VP for going ahead instead of getting dialogue started between CEO, VP and CIO? Is it on the CIO for just saying "no" instead of recognizing the need and the value? Is it on the CEO for failing to empower the VP and CIO to get that conversation started themselves?
And then, it's all well and good to worry about the bottom line first, until you're sitting in Equifax's shoes right.
I've seen it said well in another comment on this post, I feel like could be said about more than a handful of orgs:
> The problem is, I write up a proposal identifying the risks associated with the exemption, along with minimum and recommended compensating controls. This then gets discussed among IT Management, where it is usually decided it's too much overhead, and to just deny the request or if the user can scream loud enough, allow it outright and get some director to sign something. The third oft-used response is ignore the problem and hope the user finds their own work around so we can get back to the 13 projects we're somehow expected to complete this quarter.
> ignore the problem and hope the user finds their own work around
> ignore the problem and hope the user finds their own work around
If this is even remotely the story of what happened, you can't really be surprised when the user went off and did their own thing. If they came to you with a specific priority business problem and an expectation of your support to solve it with a sense of necessary due urgency, and your answer is returned in the format of a 5 year plan... I don't think you can really act surprised in fairness when they end-around you and solve the problem somehow else, anyway.
If it means standing on a mountain of chairs for them to do so then I guess there'd have to be shared culpability. So how do we make sure that it never looks attractive to build that mountain of chairs?
I wish I knew more about the "digital customer acquisition program." The story makes it sound like this "VP for a declining line of business" honestly was not going to make it another 3.5 years without some help.
I struggle with this myself, when it seems like we could go ahead and solve a problem for like $80/mo, but instead we're going to study the problem and spend $20-40k out of peoples' salaries on coming up with a recommendation for an even more expensive project that can only be justified as necessary in order to avoid this other, cheaper tool we could have used.
There's obviously some mismatch when on one hand there's a major project with a vendor like SAP in the picture, but on the other hand there are basic needs that aren't being met, to the point where someone is going to set up "shadow-IT" on a personal credit card just to keep the basic business of the company moving in the right direction.
They know when to move that risk control dial in each direction.
CIO probably wins this battle and gets the VP fired, but will be mystified when they're reporting to the CFO or a Chief Digital Officer when it happens 3 more times by the end of the year.
People are still fighting the lessons in The Mythical Man Month; it's gonna be a while.
I suspect that if a business is complex enough to have IT policy, that policy is always being bypassed in some way, at any given time. Somebody is using unofficial software, or using official software in an unofficial way.
The proper way this could have been resolved is by VP utilizing people's skills they've hired. Does this solution look good and will accomplish the task that was prioritized? Excellent! Pass it to IT to evaluate. If the task has specification - excellent, have somebody in IT look for a product that ticks all the boxes and let's choose it together.
If the tables were turned, say the CIO needed to deliver a service and didn’t have a big enough budget, then what?
That story probably never happened anyway. But the essence of the article is very true. I never have been in a corp where IT enforces 100% conformity anyway (apart from medical industry).
Sure, there are actual successful attacks, but that is mostly not the fault of unsanctioned programs.
But there are systems where people should not just start to use any system, because information gets lost on the way. That would include CRM and ERP in my opinion. That a company can exist without a CRM is questionable to begin with and solutions are plentiful. If they did not have anything like that...
If the story were true, it would not be the fault of Chief Input/Output.
It does shock me that the people who've had their whole infrastructure compromised and held to ransom by viruses and the people who've been held over a barrel by suppliers or had vast amounts of money burned by being locked into a dozen vendor contracts for the same service are so silly and hysterical about it when the solution is as simple as "identify when you need to be best in class and stay small everywhere else".
Once you hit a certain scale different dynamics may kick in but even then, outsourcing is an option.
As a freelancer these days, I bring my own laptop and am granted access to stuff for the duration of the project. It's understood and expected of me that I do such things as encrypt disks, use 2FA, and don't use "secret" as the password. Most stuff I access for these projects is SAAS based. I'd probably walk away from projects where that wasn't the case.
Let me guess: most of your recent experience is with relatively new and/or small companies, probably in the tech industry.
No.
> I would expect people knowing how to use a computer and what they need to do their job.
Yes, but their job generally isn't maintaining the computer or it's supporting infrastructure (networks, shared servers, etc.) or, outside of dedicated programmers, programming the computer even as an incidental task. IT exists to do (or coordinate contracting out for) those things, and the last point—restricting programming to dedicated programmers, has pretty much monotonically increased since the 1980s, when incidental programming was both more common than now and frequently projected to become increasingly common (lots of people said all good jobs would require some.)
Trust is always contextualized. I trust my mom to watch my kids but not to remove my appendix. The purpose of technical controls and security policy is to wall off areas where employee capabilities or motivations are too uneven or complex to safely expose them without a risk of loss that's incompatible with the appetite of the business.
Shut it down right now and ask the VP to tender their resignation. Any company doing a 3-year SAP implementation is a very large company. That $1M in additional revenue pales in comparison to the risk introduced by sharing company or personal customer data with a vendor who has not passed the required security auditing. Data is no longer a thing to be thrown around in search of additional revenue and "but I made money" or "I had to because IT is slow" is not a post hoc rationalization for the behavior.
Regardless of the merits of large enterprises acting this way, this is a VP who clearly cannot function within the enhanced risk-controlled environment of one and should find a position with a smaller company where they have more freedom to pursue personal initiatives at the VP-level. Those companies exist. Go find one.
Not at all. I have a client with ~100 employees who are past year 2 of their Salesforce implementation because the director of technology keeps changing priorities and project requirements.
“Cool, we will get back to you two years after you’re out of a jo- I mean, we will have a solution ready in three years.”
Yes. Someone should be tendering their resignation.