Verelox Wiped by Ex-Admin
verelox.com
verelox.com
There was 250+ dedicated servers, 2-3 weeks of restoring week-old backups (thankfully they had these weekly intervals kept offline). Mass exodus of clients.
"Ex-employee" used root keys and a boot zerofill drop and rebooted every server resulting in severe data loss. Their online backup systems were also using these keys and we're not spared.
They said they would have to shut down the company as a result, but ended up securing capital and eventually launching what would become digitalocean.
They said it was highly probable that it was an ex employee and that the FBI was investigating buy nothing was released about it.
Good cautionary tale for segregation of credentials and proper user key management.
> In 2003, Ben and Moisey Uretsky who had founded ServerStack, a managed hosting business, wanted to create a new product which would combine the web hosting and virtual servers. The Uretskys, having surveyed the cloud hosting market felt that most hosting companies were targeting enterprise client leaving the entrepreneurial software developers market underserved. In 2011 the Uretskys founded DigitalOcean, a company which would provide server provisioning and cloud hosting for software developers.
Seems like the better option is keep your admins happy as much as possible.
If that's the only logic that successfully gets through to the boss, it's good logic.
Not necessarily causality, but I'd say there's at least a correlation, and a good enough reason to make office life as bearable as possible.
Mario Savio was a Free Speach Activist and organized a protest to protect the Freedom of Speech at Berkeley around the 60s. In his speech to protestors, he says "there's a time when the operation of the machine becomes so odious... that you can't take part... and you've got to indicate to the people in charge that unless you're free, the machine will be prevented from running at all!" Applied to free speech, this notion of disrupting the functioning of an organization was lauded, because freedom of speech is just that important.
But let's shift to employment. Without employment, it's very hard to survive. And here's a situation where the people in charge has the upper hand in every arena- hiring, pay, work Place behavior... etc. How do we know that the ex-admin wasn't blackmailed by the CEO to come back to work for free to fix something, or future references will be negative? Why are we so quick to side with the employer in this matter when we know nothing of the situation at all? Why do we start calling the employee a felon? He hasn't even been charged yet.
My point is, context is important. Fine, corporations have the power to ruin your life as a deterrent to keep you from acting against their interests, and that's just the way society is. And fine, We're not all rational at every instance of life. The calculus of establishing status quo equilibrium of those two conditions/constraints is hard, but without context to the situation, who are we to decide who's right or wrong? Would you label Mario Savio wrong for protesting and urging protestors to prevent the operation of the college from functioning in the name of preserving Free speech wrong? No, because you've learned the context.
To us, it's not justifiable. Hell, the ex-admin may think so too. But that doesn't mean we should automatically side with the employer and subsequently crucify the ex-admin. We don't have any information yet.
It's called collateral damage. There is no context, outside of fantasy, where the admin could be in the right to do this.
If you treat people bad, they'll treat you bad. If you nuke Russia, Russia will nuke you. It's the Nash Equillibrium where each party is faced with a game and certain situations call for your best move. But your best move should account for what I'll do, and that should be factored into your initial move.
Mutually assured destruction is actually a powerful deterrent. Do we know what the ex-admin's situation was prior to all this? Only then should we pass judgement.
You are basically justifying the 'killing of innocents' here..
The collateral damage would low to non-existent for the users depending on how critical the service was. The direct damage would be to the employer. If the employer was the bad guy (hypothetically), would you still state there's no context or any situation whatsoever where the employee should cause them damage?
If their employer was a James Bond level super-villain about to shortly kill millions you could construct this scenario.
Otherwise there is no scenario where inflicting what you term a "brief disruption" (but what they'd likely describe as an awful, painful and expensive) to thousands of innocent customers is justified. Customer costs could easily be millions, if 1,000 customers each spend over ten hours of worker time dealing with that disruption.
If the employer did something illegal, report them to the authorities. If they wronged you, other employees, or customers, take them to court and publicize their malfeasance.
If you can't do any of those things because what the company did wasn't wrong or illegal, and instead wipe their servers, you are a sad little loser who can't handle the fact that they just didn't like you, and for good reason.
The ex-admin in question probably understood the impending customer harm. But the customers are the veins of a company- without them, they're not a company at all. The ex-admin just played dirty- something that he thought was the only way to win a lopsided battle against a force much greater than he.
In David V Goliath, Goliath was taken down... with a slingshot. A weapon. And he's celebrated as someone over coming a stacked challenge. The customers were the ex-admins slingshot. The difference is that the law doesn't allow this behavior
That same logic would make it unjustified for them to be using such a small, unproven service to begin with. I mean, the first thing I do for mission-critical stuff is to see if the hardware, software, or service has supported long periods of uptime with easy maintenance and security patches. Also, has anything really bad like preventable breaches happened? And how are the servers configured by default?
Further, if you're worried about downtime so much, you have two providers in a setup with replication plus failover. The only people responsible for service going down and data destroyed completely in a world where basic HA is cheap are the customers. They should assume some shit could happen. They should mitigate it if it matters. Those that didn't took the risk willingly. Backups in particular are also really cheap these days.
"If the employer did something illegal, report them to the authorities. "
People have reported all kinds of big companies to the authorities for breaking the law. Goldman Sachs nearly destroyed the financial system. They got criminal immunity + $1 trillion from the government whose Treasury was run by their ex-CEO who profited off that activity. They and most of the rest like them still in business mostly without anyone doing time on the top. What's your next move for punishment if law doesn't care or is receiving bribes (esp Congress)?
"If you can't do any of those things because what the company did wasn't wrong or illegal"
Wrong and illegal are two different things. Slavery was legal but wrong. Locking child workers in buildings that might catch fire to force overtime was horribly wrong but legal. Civil forfeiture... taking an innocent person's money or property w/out charges... is wrong but legal. All kinds of abuse of employees, esp regarding promotions or references, is harmful to all but people on top and legal.
You're clear that no illegal action should be taken in response to a wrong. I'm guessing you oppose the underground railroads that freed slaves since they were illegal. You would have griped about it at best while all the harm continued to those people if working within the legal framework.
Corporations function as de facto titles of nobility in the US, so crossing one is crossing social rank.
I think that's very on point. While HN is a technology site, it's also one for "founders" * who aspire join the elite of that social order, which explains the exaggerated empathy for the interests of shareholders and companies.
* (and those with fantasies of being one)
There is nothing to imply human rights have been violated. If such information is presented im sure people will respond in kind.
> There is nothing to imply human rights have been violated. If such information is presented im sure people will respond in kind.
What are you talking about? There is no "article," only the statement of one involved party. It's right to be skeptical and theorize about what else might have gone on that was omitted from that statement out of self-interest.
The most prominent example of this in recent memory is Peter Thiel sponsoring Hulk Hogan's lawsuit against Gawker. Do you approve of what he did?
I don't know you.
But what I have learned that people who do not care about other peoples property, care very much about their own. Destroy other people business? Go for it! Destroy my car which I could kill someone with? No way, it's mine, I've worked hard for it. Vandalize houses of rich people? Go for it! Steal my iPhone? Hey, where is the police when you need them?
In Berlin people cheer the burning of other peoples BMWs - yes this is a thing. The same people go to court when the police scratches their table tennis table during a raid.
Lol. Citation please.
I'd like to see the justification for forcing thousands of customers into expensive disaster recovery because the company fired a sysadmin, likely for good reason.
You can comfortably make a determination about what is right and wrong. We don't know the facts, but if the claims are true, wiping out not just that company's property, but that of their customers, is a crime.
Now, sometimes a crime is justified, but I don't think it is a rush to judgement to work from a starting point that criminal behaviour is bad until proven otherwise.
Why did this happen? How can we prevent it from happening in the future? These are the questions we need to stress.
In particular, why does an ex-employee still have access to production? I say when something like this happens and heads must roll, they must roll at the top. Fire the CEO. Fire the board. Leave the sysadmin alone.
This is a civil matter. My tax dollars should not pay for a criminal lawsuit. Screw that.
Oh and by the way if you're reading this: please help repeal cfaa.
Put another way, someone at Verelox screwed up and left the door unlocked, but that doesn't mean that the person who walked in broke stuff is in the clear.
Look, someone, without authorization, accessed a former employer's network and maliciously destroyed data. That's a crime. Sure, the CFAA is overly broad and is abused, but this is not one of those cases: this is a textbook example of something that should be prosecuted under the CFAA.
We should not equivocate on cfaa. It is good for nothing. Full and unconditional repeal should be our only demand.
Nope. The affected customers probably have grounds for a civil suit, but no sane prosecutor would think bringing criminal proceedings against Verelox to make sense at all.
The ex-employee who perpetrated this is at fault as well and must bear responsibility for his destructive actions. Having the ability to do something (even due to a lapse in security) does not make that action moral or legal.
http://www.telegraph.co.uk/news/uknews/crime/12090890/Reveal...
What do you expect me to do? Investigate the matter myself? Buy myself an unlicensed firearm and look for the hacker to get it back at gunpoint?
Maybe he became an ex-employee after (and as a result of) wiping production.
I've heard stories where employees wiped production by accident and promptly became ex-employees, followed by their ex-employer trying to put all blame on them. We don't have enough information at this time to determine whether this is what happened at Verelox.
If he did it (intentionally wiping the servers), then he is a criminal and a bad person.
If he is a scapegoat, then he didn't do it, so that's a totally different situation.
I don't see how you can disagree with either of these statements.
And discussion of how he still had access is an unrelated matter.
IIRC, the Soviet Union had a policy rather like this, referred to as something like the Vertical Stroke, where anytime there was a screw-up at a low level, they would fire the screw-up-ee's manager, and manager's manager, and so on, up to a very high level. The practical result was a drastic decrease in innovation and risk-taking. CEOs and others at that level usually aren't close enough to the guys actually doing direct work to supervise them all closely enough to ensure they don't make mistakes. All they can do is create a culture where there's a book of rules, and you don't deviate from the rules ever, for any reason, no matter what. So that's what they do, and that's the resulting culture and economy that you get.
Maybe we shouldn't rush to judge either sysadmins or CEOs, but instead figure out who, if anyone, actually did something malicious, and let everyone else take the lessons they've already learned from what happened.
They can't have it both ways.
So if an aggrieved ex employee does enough damage that the company has insufficient resources to sue, or has enough resources themselves to make that difficult, it's all good?
Individuals and companies should not be vulnerable to attacks like this based on their resources. It's in all our interests to ensure this sort if activity is dealt with severely because we are all vulnerable. Mutual defence in the form of criminal prosecution of offenders is the way they go IMHO.
Let's be practical. What is our end goal? Is it vengeance or is it prevention? If it is the latter, our actions must not be centered on retribution but rather on logic and reasoning. We should ask what can we do to prevent this from happening again? Throwing someone in prison is not the answer IMO.
We may remain open to additional information without presuming that uncivil and illegal vandalism was justified indeed without inventing a narrative from whole cloth as you have done. The logical conclusion is that drawing from your own life experience you identify so strongly with the narrative of the wronged sysadmin that you desire to fit a narrative to sparse facts that has no basis in fact.
We are merely commenting on a story on hacker news. We aren't members of the jury and don't face the same burden or power. I'm down with repealing the cfaa because its badly written, I'm down with figuring out who dropped the ball as far as giving the sysadmin access post firing, but as to the sysadmin himself, burn the witch!
That the actions described in the claims are criminal is not in doubt. Whether those actions actually took place is unproven, as is whether this unnamed ex-administrator performed those actions.
So, I'm not ready to say that this person is a criminal, or that a crime was committed, but I am ready to say that the actions described in the claim are criminal and wrong.
Finally, all of that is applicable to the courts & government determining criminal acts. The courts don't determine what is moral or ethical. Lots of things are wrong that aren't criminal. Saying something particular behaviour wrong really doesn't have anything to do "innocent until proven guilty".
Saying that, "if someone did what the story said they did, it's wrong" is really not saying anyone is guilty of anything.
We do.
How many little shops rely on Verelox - aren't they little guys?
How many end customers depend on services that depend on Verelox - you know someone like you - aren't they the little guys?
The police/prosecuting agency in the country where the company is based I imagine. That appears to be Holland [1]
[1] https://www.ripe.net/membership/indices/data/us.verelox.html
The fact that it was an ex-administrator does suggest foul play as opposed to a mistake compounded with poor backup processes. We shall have to see how the story pans out.
Sigh. I can't believe we're going there on this one. Just so we're clear, you think that maybe the definition of criminal behaviour shouldn't include wiping out systems without the owner's consent?
> And why is criminal behaviour always placed on the little guy when companies can always get the leeway of claiming ignorance?
I don't know what you are talking about. Criminal behaviour is placed on the criminal... and ignorance of the law is not a defense.
Urbanization fundamentally has been engineered to create "wage slaves", who are basically modern sharecroppers (though the system has some extra steps).
Being employed at a particular organization is, in fact, voluntary. You are not obligated to stay at one employer forever.
I enter into employment agreements knowing full well that it's a business transaction. The business and I have agreed to an amount of money (and other benefits) in exchange for my labor.
It's not about the American dream. It's not about "slave labor". It's about you give me money for services I provide. When either of us decides that deal isn't advantageous for us anymore, then we end that agreement.
Stop being edgy.
Incorrect. Throughout history there has been many societies where people would legally sell themselves into slavery (e.g. to pay a debt, under threat of being imprisoned) and even receive a meager salary.
Yet it's still slavery; you can look up the definition in a dictionary.
I think most of HN is in the "baronet" class (say, 2-3x of median income) or higher, and doesn't necessarily have a good perspective on what most people (ie, those at or below median income) deal with.
Per capita income has flatlined for a decade (since 2008), after 60 years of steady growth. But that largely doesn't impact programmers, entrepreneurs, etc. Instead, it impacts the people on the bottom.
Just because the present system works for you doesn't mean it works for everyone.
It was always good to be king.
So, overall, it looks more like a form of economic slavery than some cooperative, voluntary, win-win system. A good hint is that most of the wealth has moved into few hands who usually didn't work their way up into it from the bottom like the people producing it. The people producing it didn't want that arrangement either. They have no control, though, since the elites outspend all of them combined on politicians passing these laws, media shaping perspectives, and public education limiting their intellectual growth while simultaneously reinforcing submission to authority (esp arbitrary requirements) daily for 12+ years. Most of America is born into a new form of slavery that few ever escape.
I suspect that, for many, if not most of us, the loss of the ability to earn income would be an unspeakably grave event. This puts to the lie that we have any real choice about employment, assuming we are one of the vast majority of people who is not financially independent.
"[the illusion of]"Democracy", "Organized religions", "history of our society", esp. the last 300 years or so", 3 of the most profitable lies in todays day and age. Its wrong to say that this "happens to be happening because A, B .. Z", this is "works-as-designed". A world viewed from the top has no nations, only powerful blood-lines, it has not morals only self interest, it has no humanity only contempt of the "human animals" they have inherited and extended control over. It's them, a new species, and us, the soon-to-be transformed, stripped of everything that could challenge them, rest of the world.
Don't say "no one".
Productivity keeps rising. It becomes possible to support non-workers with a smaller and smaller fraction of that. And thorough safety nets help non-lazy people too. They even help people create their own businesses!
I do my job because I love it. I do my job for you because you pay me more than the next guy.
Suppose someone was born out in the forest, and there were no cities, no societies, no technology. Would you call that person a sustenance slave? Would you say that their foraging was involuntary?
That's right, and it's a shame it's been downvoted.
> Suppose someone was born out in the forest, and there were no cities, no societies, no technology. Would you call that person a sustenance slave? Would you say that their foraging was involuntary?
No, they are not a slave because they have no master except nature.
A wage slave has at least one master: his boss and the system that forces him to work for one.
The difference lies in the fundamental difference between natural law and synthetic human systems. Entropy isn't your master, it's a fact of your own existence.
By contrast, the present economic system is a synthetic construct made by other humans to exploit you, designed with largely that purpose, co-opting your biological needs for coercive power, and doesn't necessarily need to happen.
That's what makes one slavery and the other not.
The difference is so obvious, I actually am confused why you think conflating physics with political norms was appropriate.
Most of society (and the wage slave concept) is about human farming.
would you say you are a farmer or livestock?
No one has the perfect job, everyone thinks they should make more, have more benefits, more freedom. But everyone has choices. You don't have to participate in the rat race, you can stop letting material possessions drive your lifestyle decisions.
The choices that are available, however, have an extreme variance from person to person. At the extremely shitty end, Viktor Frankl, while a captive of a Nazi death camp, worked out that when all other choices are taken from a person, the only remaining one is to die with dignity.
But it's a choice, right?
At the other end of the spectrum, choices may include where to vacation this winter or if the maid should be fired.
The bottom line is that the quality and quantity of choices vary so much, depending on luck and pluck, that it's a bit glib to say "everyone has choices", including, I guess, the choice between working a crappy, abusive, mean job or two to make ends meet, or starve.
I said you didn't have to participate in the rat race.
For most of human history, choosing not to work for 3 months meant starvation and death. Hunter-gatherers had to work.
My brother is 50 years old and has delivered pizzas for about the last 20 years. He only works enough to afford to rent a room, own a car, and pay for an annual ski pass, and skiing/volleyball gear. He spends his free time on the mountain and the lake.
He obviously doesn't own many material possessions and his retirement isn't going to be pretty. But that choice was his, he had opportunities to work corporate and make far more money, own more stuff, and maybe raise a family, but he chose to have more freedom and the lifestyle he preferred.
I don't think everyone has the luxury to philosophize about giving up material possessions and exiting the rat race, either. A lot of people are living from paycheck to paycheck (or worse), have dependents, and don't have the prospects to find a new job. Those people can't afford to say "no" to their boss' unreasonable demands.
I wish HNers would get their heads out of their tech arses and face the reality. Most people do _not_ have a choice where they work. Most people cannot afford to quit. Most people don't have jobs where they can increase their skillsets. Most people do not have time to read HN while they enjoy their 10 AM pause in their comfy sofa while working from home. You are, for the most part, amongst the most privileged people in the world. Yet you continue to spew the "employment is voluntary" propanganda because you never had to face actual hardships. Worse, you push people of lower classes even further in the ground, when what you should be doing is elevating them to make a better and fairer society.
Not free speech: punching your boss in the face, burning the building down, vandalizing the office, deleting all the data on the servers and hurting a bunch of people that had nothing to do with your conflict with management
I'm all about workers right and have walked out on many a job because management were raging assholes. I have convictions and I stand by them but conviction and doing the right thing in life often come with sacrifices. I'm currently underemployed because of my convictions but I can sleep at night knowing I'm doing the right thing. The way for us to take back power in the tech industry is to organize, unionize and refuse to work for abusive and exploitative employers. We're the ones with the skills. They need us not vise versa.
BTW, we had a netadmin interview a few months ago. Guy was really smart, aced the technical and group interview. We were really looking forward to hiring him, and only needed to pass a background and reference check. HR told us in no uncertain terms to run the other way. They didn't share what was in his check but it wasn't good.
It was the creepiest, most cynical thing I've ever seen.
Those guys are gonna be RICH.
Denying transport to someone who's got a history of doing anti-social things behind the wheel seems reasonable. But making them required for all vehicles (if I'm understanding your intention correctly) seems likely to have undesirable results.
Seriously, read up on the history of breath-test machines sometime. Everything from difficult chemistry to literal "oops we misplaced a decimal point in the software".
I was checking out the Sunday paper a few days later (this was the 90s), and it turned out that the guy was a fugitive who basically killed his wife and fed her to the fishes.
Pretty freaky stuff.
You flunked your own background check.
Always ask to see the background check details if you're the hiring manager (you should have the rights to see that despite what HR might tell you). Could be just a personal issue an HR employee might have with a former colleague. Or discrimination-based (happens).
I've had my own background checks in the past and no one has shared them with me. Speaking to HR, unless there is a negative item on your record they aren't required to.
o_O
Why would you assume you couldn't?
If you're referring to private reference checks, of the type that would surface "personal issues people might have with formal colleagues", you're not entitled to anything whatsoever.
If you're a hiring manager in an organization where HR handles background checks, you personally as the hiring manager are entitled to nothing. I would venture further that it's inappropriate for HR to provide criminal/credit background check information to hiring managers.
You mean in any kind of practical or legal sense or in your personal opinion?
I see this happen all the time and am not aware of any thing barring this in the U.S.
Think about the two extremes. If a startup of a few people run a background check there is no HR dept so the CEO/hiring manager is seeing it.
On the other hand if the hiring manager is the CEO at an F500 company trying to hire a handpicked rockstar exec and a background check pops there no way that person will be rejected only on the word of HR without understanding what the issue is and the context around it.
Somewhere in the middle of those extremes companies may have policies to address the situation, but I don't see how any blanket statement can be made here.
* Candidates are entitled to copies of background/credit reports used to make adverse decisions against them, under the FCRA.
* Candidates are only entitled to the specific document, collected under the FCRA, that was used to make the adverse decision.
* Reference checks --- calls to previous employers and coworkers --- are not governed by the FCRA or, as far as I know, any other law. Candidates are absolutely not entitled to any information about reference checks, but a candidate that "flunks" a reference check might be told so when they're refused a job. Or they might not.
* No law entitles hiring managers to any documentation whatsoever.
Only the last point I made was one of opinion, but I think it's a widely held and pretty common sense opinion:
It is terribly inappropriate for HR to share background check documentation with hiring managers. Companies that employ background checks should have simple, static rules, like "no previously undisclosed felonies" or "no violations relevant to the job", that are evaluated by HR.
It's already a grave violation of candidate privacy to do these kinds of checks in the first place (which is why they have to collect special permission from candidates to do them at all!). It's negligent to then pass that information around the org chart to help others in the company read tea leaves from them.
My opinion here is informed by experience working with companies that do background checks, but I wouldn't be surprised to learn that there are fucked up companies where HR passes copies of credit and background reports to other employees.
To be clear I don't think a lot of this stuff is right. It drives me nuts to see someone ace technical interviews and then not be hired for some insignificant drug related issue. How do companies not realize this probably hurts them more than the candidate? A lot of tech companies don't do drug tests at all, I'm guessing because they've figured this part out.
True, but in general, it is likely that whatever a background check turns up on someone, such that they're unemployable, is probably available to anyone with a web browser and some time on their hands. The people in the org who want to satisfy their curiosity or sense of self importance as to why someone didn't meet the company's standards can probably find a way to do it.
The reason one really shouldn't what to see this info at all if they can help it (and should have a trustworthy HR office handle this stuff) is that pretty much everything done in the hiring process can be used later on in court. The safest possible position for a hiring manager if the question comes up is "HR indicated the candidate did not pass our background check and so we moved on." And only an idiot wants to answer questions later on about with whom they shared damaging information about a candidate.
If you are trying to hire a talented employee, and HR says no way, you better demand they justify it. It's your job to advocate for people on your team, even when they haven't come on board yet. Otherwise no one will.
It's far to easy for HR to auto-reject because of hard-wired criteria, they have no incentive to challenge their own rules for a good candidate.
I know a kid who once was a chip-runner at a casino. The way the job worked was he took cash/chips from poker players and went to the cage to get them chips/cash. He got paid minimum wage plus tips, and was on the hook if he got shorted. But the plus was he'd get $1 tip for each transaction, sometimes more.
But this kid optimized the whole job. He hustled like crazy, minimized cage trips by carrying 7 or 8 racks at a time, and even took optimal routes through the poker room. Watching him work was inspiring, the kid was so driven.
He was making a ton of money, far more than the other chip-runners, and someone ended up complaining. So the room decided chip-runners should pool all tips together. He quit in a rage the next day and chip-running service in the room became awful, the remaining chip-runners now saunter at their own pace, carry a single rack at a time and minimize their risk by slowly counting everything three times.
A couple years later the kid got busted for selling weed out of his house. He ended up serving a few years in prison. He's going to now fail any HR background check/screen. I'd still hire him in a fucking second, and expect him to rapidly work his way up from any entry level role.
One day one of the feared internal security team who had caught him years ago was in the building and met him and freaked out - our centre mangers politely told him to f off when he demanded that we fire the guy.
Socially Adept Frenchman: 1 Human Resources: 0
As soon as individual managers want to make those decisions for themselves, it becomes much harder to refute claims of discrimination.
"Your honor, my client, the recently released child rapist with an iffy work record, is suing because BigCorp says they normally don't hire felons, but a few years ago BigCorp made an exception for a felon who sold marijuana, got his college degree in prison, then after he was released worked at non-profits helping the poor while getting an advanced degree in computer technology and writing several highly regarded academic papers on improving user interfaces for the disabled. That's not fair!"
The specific facts matter, and HR should be fired if they have blanket polices that ignore them. HR needs to serve the business, not the other way round.
Regardless of what crime they may (or may not) have made; you are looking for the person and what they do. Not what they did.
This is a common problem; we look at the past a bias of the future. Life only works out that way if the person is too unwilling to change; and that again is something you should look for in the hiring process.
Lastly; hiring ex-hackers isn't a bad thing. Caring about a background check when hiring an ethical hacker or someone who turned their life around; only shames them and pushes them back where they came.
So be careful or you only end up criminalizing being a criminal.
Also, he is not a hacker, just an asshole. Having the admin login and password doesn't make you elite. The only weakness he exploited was himself.
What they did; and more importantly have they recovered and not commited another crime of similar circumstances.
P.S. I knew a kid when I was younger who helped an ISP start up in Colorado Springs in the late 90's... he got fired and he hacked them and spent 6months in jail; eventually they re-hired him after he got out of jail only for a repeat offense.
There is a point; he did it twice. The first time they could pass it off as they didn't know; the second time they hired him they were legally liable.
So this gentleman should spend some time in jail for what he did.
Yeah, that. Also, secure backups and compartmentalized systems and data access.
Automated tools may need to delete and list; but heck off site secure backups are an amazing thing.
If a company walks away from this and doesn't take legal action; they should themselves get sued by their customers.
Destroying company property is a crime; wether it's defacing a website or ... it's not your property; you are just hired to maintain it (in one shape or another)
A similar thing happened to a client. Sysadmin logged into GCP and Azure immediately after termination and just deleted everything. He was in the UK, we were in the US. Wasn't worth it to try to get someone to prosecute, and I'm sure we're not listed as references.
Did get people motivated for a multifactor delete bucket for extra backups.
You'd be surprised. I did something somewhat similar and was convicted of a federal felony. No fast food place or retailer would touch me with that record. But ironically I've found plenty of IT work with smaller companies.
> Do they fire you if you're not testing positive for cocaine?
I always have a hard time with those contexts. HR sometimes has the wrong idea of what is unacceptable and what is appropriate. There really isn't a good reason for them not to tell you. The only thing that would make sense is if they shouldn't know it.
So, just because a guy has a bad reference it does not mean that he's necessarily bad.
This is so true. Years ago I was brought into a company specifically to improve their software quality process, but without being aware one of the company owners (of which there was two) was against be being employed for that purpose. I uncovered a lot of incompetence and outright corruption with some employees. The 'good' owner went on stress leave, and then pressure was brought to bear on me, resulting in me quitting on the spot one Monday morning; not my proudest moment, but I couldn't take the pressure any more, and my ally was no-where to be seen. Fast forward a decade, and I interviewed with someone who used to work at the same company; it turns out that my leaving had been framed as 'fired for incompetence', and word had been put around part of the local industry that I was hard to work with, unreliable, bad at my job etc. I laugh about it now, but at the time it really bothered me about the possible damage done to my career and reputation.
I don't know what country you're in so I won't speculate on what's legal or not in your area, but here in the U.S. it's illegal for a prior employer to provide false information to a prospective new employer during a background/reference check. If your former boss tried to blacklist you like that, he'd put his company at risk for a civil suit. And, while it's not illegal to truthfully say that a current or former employee is a bad employee, doing so rides the thin line of opening the company up to a libel lawsuit.
Generally speaking, a company might "dish the dirt" on a former employee if there are criminal charges to back up the claims. Even then, legal and HR will likely frown upon it. Usually, when the new company calls the old company for a reference, the old company will say something like "Yes, $employee worked here from $startDate to $endDate" and refuse to divulge any other information in order to avoid any semblance of libel.
Once again, this is my limited experience in SMB and government settings, I have no experience in mid-to-large businesses, Fortune 500 companies, and Silicon Valley startups. We've all heard stories about managers at such companies going around HR and discussing potential hires at the bar or on the golf course.
Asking previous employers about their experience with an employee, however is not illegal. It's usually not illegal for them to say something negative if it's true, though some businesses are conservative about what they will say out of fear of being sued for slander. Accessing public court records or news stories about criminal cases and using that information for employment purposes is usually not illegal. Asking prospective employees if they've been convicted of serious crimes is usually not illegal.
It does seem to be conventional to include one's photograph, age, sex, marital status, children or lack thereof and place of birth on a CV in Germany. It's not clear to me if it's actually problematic to exclude those, but it's my impression that not following conventions tends to not go over especially well in Germany.
https://web.archive.org/web/20170603212121/https://verelox.c...
I suspect that person will soon have a fair amount of time to decide on what new career to pursue to pay down the fines when they get out...
Until they release specifics if ever, it's hard to know what to make of the status message.
One of the very neat things about rsync.net is that your account is on a ZFS platform and you have snapshots enabled by default and the snapshots are totally immutable.
Which means that if you back up your VPS (or your VPS company) to an rsync.net account and someone owns you and owns your rsync.net credentials, the worst they can do is delete the very latest backup ... the snapshots cannot be altered.
Just saying.
First, rsync.net is "cloud storage for offsite backup" - you can't run a git server[1] (or anything else) there. It's not a VPS or a web host. It's a remote unix (ZFS) filesystem that you can access over SSH.
The other point you are missing is that the ZFS snapshots I refer to are immutable as far as you are concerned. Of course we can remove them[2], and could do so at your (vetted, verified) request. Further, we don't have unlimited disk space so the snapshots rotate out (expire) over time. Every day the 7th one is removed to make room for the new "yesterday" snapshot, and so on.
The point is, an attacker can gain full access to your backups with all of the control you have ever had over them and they can't destroy/delete the snapshots. That would have helped the victim in this story immensely.
[1] You can, however, put git repos there and interact with them, using git, over ssh.
[2] Although it requires root and is an involved, manual process - which is good.
One concern I have is right here:
"your (vetted, verified) request."
What's that mean specifically? There's potential for attacks there. For instance, a request from several email addresses might come from computers the admin controls. Same with some 2FA's. One would have to be careful here. I got a voice idea that just passed through my head that could leverage their smart (or dumb) phones. Also maybe dedicated tokens, apps on their phone or home computer, or something that come from your company. I'm curious what you're already doing, though.
It means that the owner and founder of rsync.net stares at your request and decides how he feels about it. Then he weighs the financial security of his family and the reputation of the business firm that has become (over these last 16 years) his life's work ... and decides if one of the engineers should call you on the phone and vet your request just a little bit more ...
Aligned interests and "skin in the game" ... those are powerful things.
And having switched jobs quite a few times, the next one is always better for you, regardless.
Apparently they did not.
At the end of the day, the people working at the company are the ones who are doing the work, and who have control of the means of production. The ex-admin's bosses probably thought they were the important ones, and that this worker was a replacable cog, but they found out the hard way that this was not the case.
I worked at a Fortune 100 investment bank where this happened. Everyone knew layoffs were coming. One week after layoffs came, a digital "bomb" went off wrecking many servers. So security went through, trying to find evidence (nothing incriminating from what I heard, although they had a strong suspect) and also looking for more bombs. They missed out on finding and defusing one, because another one went off a month later.
The view from the pinnacle, people counting the dividends on the checks that they inherited is that they're the job creators, and everyone else is dispensable. This company just found out that is not the case.
Therefore, to behave with integrity, you must have formulated your own set of values about what is "the right way" to behave.
Every minute of every day, we all have the option to behave with or without integrity in a whole range of ways.
You earn respect by demonstrating behaviours over time where you have taken the interests of others into consideration, generally people consider someone who behaves like this to have "integrity", especially when they continue to behave that way when no-one is looking.
Saying things like "The ex-admin's bosses probably thought they were the important ones" indicates a childish set of values where there is a power struggle between employers and employees ........ of course the "bosses" are the important ones, they act for the business which is an independent legal entity, upon which many people depend for their lives to work effectively. If, as an employee, you feel poorly treated or otherwise dissatisfied, then the right thing to do is leave in a polite and respectful manner, even if you feel you were not treated in that way. Depending on the circumstances, if you were actually treated really badly, then the right thing to do is pursue your complaint through the appropriate legal channels.
Someone important in my life once said to me "the only thing you have is your reputation". Take that reputation, defend it, enhance it, nurture it and earn the respect to grow it. Don't throw it in the garbage by smashing other people (or their business) in a childish tantrum. I admit this is hard to do - I regret many things I have done in my life, but I try to lead a life consistent with my own sets of values that I think are meaningful and I get rid of people from my life who I think don't have integrity, or whose values are different from mine in critically important ways.
I get rid of people from my life who I think don't have integrity, or whose values are different from mine in critically important ways.
On this, however, we agree entirely.
I think if everyone treated others the way that they would like to be treated, then the world would be a better place.
Maybe that is too cynical, but it works well as a rule of thumb.
That belief was created and pushed by religious, political, and business elites who themselves did not give equally to other people or treat them fairly. They always schemed out more for themselves. Your rule is best modified to do onto others as they would actually do onto you to the best of your knowledge. Otherwise, your rule will result in more evil happening overtime as the good people will work within the schemes created by the bad people. That's already happening.
Illustrated nicely in Hawk-Dove game:
https://en.wikipedia.org/wiki/Chicken_(game)
http://www.oocities.org/hawkdovegame/simulation.htm
Even includes retaliators which are relevant here.
The large majority of the population will succumb to "what's normal" vs. "what's for the best" to avoid being casted unfavorably. Your regimented approach to seek out the carved out channels of recourse, dictated by authority, suggests you won't combat for change.
We live in a society, yes, and we have norms to abide to. But when we propagate the notion that everyone should put their heads down for the sake of reputation, then the world will never progress.
Especially when considering the asymmetries involved in interactions between a corporation and an individual, i would even suggest that it is a form a bias people can have which could be considered to be a conformity excuse[0].
[0] http://www.overcomingbias.com/2017/06/conformity-excuses.htm...
The relationship in most companies is entirely asymmetrical. If times are tough, employees are expected to work unpaid overtime, to sacrifice on pay and perks, to accept layoffs. If times are good, shareholders and executives see all the profit. Employees are expected to show absolute loyalty, but are shown not one shred of loyalty in return. Productivity is soaring across the economy, but wages have been stagnant since the 1970s.
Most employers will never truly respect their employees, ever, under any circumstances. Employees aren't people, they're a "human resource", a cog in the corporate machine as interchangeable as any hardware. More so, in fact - a piece of machinery would be hired on a fixed-term lease, but most employees can be dismissed at will.
I don't endorse vandalism, but I think that it's utterly naive to expect that you can earn the respect or loyalty of corporate America. It doesn't matter how honest you are or how hard you work, you'll still be discarded like an oily rag if you're surplus to requirements. You'll still be lowballed on every pay rise while executives and shareholders make record earnings.
> I don't endorse vandalism, but I think that it's utterly naive to expect that you can earn the respect or loyalty of corporate America. It doesn't matter how honest you are or how hard you work, you'll still be discarded like an oily rag if you're surplus to requirements. You'll still be lowballed on every pay rise while executives and shareholders make record earnings.
You are absolutely correct about corporate America, the executives, and the shareholders. The hard thing is that you can earn the respect and loyalty of the other cogs that you work with, which can be difficult to disentangle from that underlying truth.
The solution to this, should it become a regular occurrence, is to make the folks with the keys to the kingdom replaceable.
It's doable, companies just don't do it because most people don't want to destroy their high paying and relatively comfortable careers committing felonies and getting sent to prison because they had to spend a few weeks or months looking for a new job.
I don't think this really indicates that at all. Maliciously inflicting damage on the company when you're fired is very different from being irreplaceable. It makes it risky to replace you, but that's not the same thing.
Sabotaging servers doesn't mean you're unreplacable any more than a terrorist attack means Western culture is depraved.
I'd believe you if the servers simply started falling apart without this person around, but that wasn't the case.
It's not an equal relationship. One side usually has significantly more power than the other.
* The irrational biases of other companies (even if we didn't care about you being a university grad, our commensurately lower wage offering reflected the fact that other companies did).
* If the employee telegraphed an air of needing the job (e.g. people with a debts), they got offered commensurately less.
It's an ugly process, truth be told.
People assume that that is all about skills but it isn't necessarily. An employee with a serious debt problem and a mortgage payment due soon will be more likely to accept a bad deal than an equally skilled employee who doesn't.
As my own company is growing, we fully trust all employees, (limiting only what is essential), but, a dev ops guy if he was so inclined could technically do something like this... It always scares me.
Beyond that, be sure to keep regular backups (and test them), and audit all user actions. (feed the logs into something like Splunk, running on a separate machine)
For really important accounts - we have three people who each know two thirds of the password. It requires two people to then log in and do damage.
For example if the root password was CatDogFish then
Person1: CatDog_
Person2: _DogFish
Person3: Cat_Fish
Two people can then log in and watch what the other person is doing.
Because if not, once you are admin, you can install programs that let you become admin again at will.
But it's not a bad system.
Password rotation becomes necessary, and a little bit riskier, because now you have to deal with accidental lock-outs in a sane, coordinated manner.
You are stuck trusting somebody, no matter what. Its turtles all the way down.
Soon, dear throwaway, you'll be telling me we should live in fear of the locksmiths, for all their key blanks and such fiendish metal files to abrade them with. What if they should file down a butter knife into the shape of my precious bicycle chain's key?
I worked retail to pay for college. Could always tell when a manager was getting the boot; they'd order new cylinders for all the doors. You kind of have to have that plan in place in IT too.
covfefe
That's how I do it, anyway.
We can grant that this can be logistically difficult at certain scales, but it doesn't fall into the "engineering-impossible" bucket until you reach Facebook's size.
I don't know of any server provider (bare metal or cloud) that forces users to allow the company full access to their data (outside of managed providers were you voluntarily give this up , as you're paying them to fully manage your server)
And do backups. And then backups of those backups.
That's probably because when my grandmother died, my boss at QueBIT said "Ok, go home, call me when you can work again - however long that takes." There was never a discussion of PTO/HR policy, just human treatment.
Also remember to test restoring your backups or they don't count.
If the CI/CD is done right, then no DevOps staff has any access to any servers and no one can delete anything except a scripts and AWS configurations.
The whole problem with limiting permissions is that you have to do all the work of deleting files, servers and drives.
If someone is planning a malicious exit, it can be very hard to stop them depending on how "integrated" they are.
If they're sharing them individually with you... Then clearly they're not paying attention
You can't. Not from an admin.
Same as how if you are rooted the only advice is to reinstall. It's simply impossible to reliably undo everything from inside the machine.
If you are a company, reimage the machine, then reinstall everything, and copy the code fresh from known good source control (and hope someone was watching source control that the admin did not check something in).
edit: also, use a bastion host which has the keys on it and don't allow them to be removed / used from laptops directly.
Also, the CA mode of OpenSSH is great. More people should use it. It's like PKI but sane.
This problem is not as simple as you are pretending it is
Seriously though, would Verelox still be running unpatched AMT many weeks after the disclosure of this authentication bug? Or does GP think there are more bugs which Intel hopes to sweep under the rug forever by individually covering each incident? They would spend quite a money on these bribes while AMT bugs can simply be fixed with BIOS updates.
This Intel conspiracy doesn't make sense. It's aliens, folks, I know it.
Rephrase the question -- what idiot customer is going to do business with such a place that allows such a lapse in security to happen?
Intel would basically have to buy the company.
The kind of people that:
- Use Gmail, iCloud, etc. post Snowden
- Buys SSL certificates from Comodo, etc.
- [put other companies here]
What, exactly can be done to secure a company against a malicious systems admin? These are the guys typically with not only the keys to everything but also the knowledge of how it all works.
You say that the company cannot be trusted for "allowing" this to happen.
I know quite alot about this stuff, and for MOST companies, they simply have to trust that the people with the keys to the castle with behave responsibly.
There are ways to design infrastructure such that it is protected from its builders and keepers, but this is very very hard and complex and expensive.
Presumably you work for a company that has taken steps to ensure this will never happen, what are they?
At my one and only sysadmin job, the network was secured from likes of this guy by the senior engineer, a Vietnam vet with trust issues and whose talents weren't only technical.
Nothing says "web of trust" like knowing your boss could show up at your door with a shotgun demanding answers.
Maybe they were but he'd set up time bombs.
Maybe the admin was fed up, knew he wanted to burn things down, wiped everything remotely and then never turned up for work and he became an ex employee in the aftermath.
It's decidedly not trivial to secure your company against a malicious admin who has control of much of your infrastructure.
At best, these companies will be keeping out the riff raff. Fortunately, that will stop majority of attacks since most admins aren't geniuses or spending personal money on attack toolkits with 0-days.
Or maybe they had backups running under their account and when their account was disabled everything failed so they re-enabled it while they sorted out the mess...
Or they had multiple accounts as part of "security" and HR only knows to disable one and didn't find the other one in time.
There's a whole bunch of reasons why shit like this goes wrong. Every time. You'd cry.
(Btw, IMO there is no excuse or justification for any admin or exadmin to ever do this. Among many other issues is the fact he deleted the data/work of individuals who had nothing to do with whatever "problem" he has with Verelox )
There's probably excuses and justifications. I personally wouldn't do it and they're probably wrong for doing it but I don't want to jump to conclusions and moral absolutes so easily.
Nothing is foolproof, but anytime you've got constant network access to every last copy of your data, you're begging to lose it. It's the reason why people who think one copy (redundantly dispersed or not) in AWS S3 is sufficient scares me to death. Is it unlikely Amazon would get hacked and have the entire thing blown up? Sure... but if we go to war with China I wouldn't want to bet my company on it.
I know you meant it as an example, but this sort of extreme attitude towards security is just another footgun.
Why? We went to war with Europe and Asia a few times and businesses kept chugging along here in the states.
Probably not in the immediate aftermath, but someone might decades later, if the company actually does something valuable.
If there isn't a police report, and charges aren't pressed against a malicious individual, then the company may be at fault.
Certainly the employee could willingly and pro-actively step down, out of personal guilt and feelings of shame, but then, one need not qualify with "ex" as simply "employee" will suffice. A criminal incident is worthy of such a clarification, indicating that the incident is a deliberate attack, but human error is not.
What if the admin was a remote worker in a country that doesn't have an extradition treaty with the Netherlands (Verelox hq country)?
> 4 low-latency locations in ISO/IEC 27001
> certified Tier 3+ data centers in United
> States, Netherlands, France and Canada.
If this was a "cyber crime" they could possibly be picked up to stand trial in any cooperating country which would narrow their choices of travel.
However, in that situation they would probably be fine except potentially limited ability to work for Companies who do background checks
Although that's perhaps not the right word. There is contiguous land between e.g. The Netherlands and Russia, so there's no "shore" involved as there would be between e.g. the USA and Russia. I do think the word's meaning is now more "another country", rather than denoting crossing physical land/water boundaries.
Maybe in North Korea, or the US. Relatively unlikely in civilized countries.
I've been following and partaking in various European cybercrime trials for years and unless this guy is a repeat offender or acted in a particularly methodical manner it seems utterly ridiculous to make blanket statements like "this guy will go to jail". He most likely will not.
Compare US incarceration rate of 693 people per 100,000 to 69 people per 100,000 in the Netherlands.
Possibly related?
Or these two events are unrelated. Or the whole deleted prod on day 1 story is made up.
Edit: Make that the 7th based on: https://www.facebook.com/Verelox/posts/1886196381643427?comm...
If I leave $20 on the sidewalk the thief is wrong for stealing it, but it is partially my fault for being stupid enough to leave $20 on the ground.
This can easily happen to anyone -
5PM Friday - $admin and $ceo have a fight
6PM Friday - $admin decides he's had enough with $company and $ceo, and wipes everything
7PM Friday - $admin is fired
Note that I don't know the details and am making assumptions that may be wrong about the case in question, but in general, if you can't deny access quickly to any given account, you really want to fix that. Not just because of rogue ex-employees - what happens when $important_person's account is compromised?
No. It's easy to revoke access to a user. An admin is different - an admin can install whatever he wants to give him backdoor access. Or a timebomb.
Detecting unauthorized software from a rogue privileged user is a different problem with very different mitigations. It is a great topic that I'm personally interested in, given that I'm implementing controls for that, but I wasn't discussing that.
I'd agree that defending against malicious admins is really difficult. We have really little context to go by here, but I think there is important distinction to be made if the malicious actions (planting backdoors or whatnot) were done while the malicious actor was still employed or after their employment was terminated. Proper exit procedures protect against the latter, but generally are not that effective against the former.
> but in the end humans make mistakes
And it is useful for us outsiders to highlight the real mistakes so that we can learn from them, because that is really the biggest value of stories like this for the majority of people who are not directly impacted.
Some posts from Verelox staff towards bottom third of this forum page search for user name Verelox
Otherwise while the vast majority of your staff will be decent people and not cause problems like this, it just takes one angry ex staff member with a grudge to cause problems.
They also need to revise their backup system too. There should rarely if ever be a risk that any data is 'unrecoverable', yet their update says some data will just be impossible to get back.
As for the employee involved... well I hope they like the inevitable lawsuit their selfish, stupid actions will bring them. I don't care what you think of a company you worked for, there's no excuse to destroy their business through actions like this. Also, good luck getting any jobs in the industry after too. Because with this on your track record, no one will touch you with a ten foot bargepole.
So yeah, what a disaster all round.
I'd like to know more, I think...
Are you sure no matter what the company did? What if the CEO threatened the ex-admins family? Or if the ex-admin found child porn on the CEOs computer?
There's a fine line between right and wrong in most situations. The most egregious acts of disobedience can be seen as defiance or foolish. It's not for you to decide- especially when there isn't any context to this whole situation.
This is not a whistleblowing case, it is a case of wanton destruction by a former employee.
> That line of reasoning doesn't set the bar for what's right and wrong.
Dragging in all kinds of stuff that has no bearing on the case doesn't set the bar either.
> Are you sure no matter what the company did?
Yep.
> What if the CEO threatened the ex-admins family? > Or if the ex-admin found child porn on the CEOs computer?
In that case you go to the police and file a report with them. Hurting the company, the employees and customers when your target is the CEO is ineffective and illegal besides.
> There's a fine line between right and wrong in most situations.
No, it's crystal clear that this was wrong in any way you would like to look at it.
> The most egregious acts of disobedience can be seen as defiance or foolish. It's not for you to decide- especially when there isn't any context to this whole situation.
This is a criminal act, pure and simple. If the CEO did anything illegal this guy/girl is an idiot for doing something illegal himself.
But that does not mean that a single individual can put themselves in the judge, jury and executioner role all at once without any kind of oversight, that's at best a misguided case of vigilantism and at worst an act that is disproportionate against innocent bystanders and possibly a far larger crime than whatever happened before.