The absolute worst scenario happened
reddit.com
reddit.com
Everything they list that's an issue seems to revolve around the fact that the employees with domain knowledge for the system have left the company.
Reach out to them and hire them on as contractors. If they left under bad terms because the business was a bunch of dicks, expect to pay 10x market rate. If this is truly "fix this or the business is out of business" - then it shouldn't be a tough decision to make.
That last part may not be necessary though, as customers will jump ship and the company will probably sink anyway.
Depends, they might have a captive market or a customer base that's non technical and doesn't understand how bad this is.
If their systems are like this, chances are their offering isn't anything groundbreaking and the competition already provides a better service for possibly even cheaper, so the customers who are willing to jump ship would've done so long ago and the fact they're still in business suggests they have a complacent customer base that's likely to stick to them even despite this incident.
The same reason this technical debt was left unchecked for ages applies to customers. The task of migrating to another provider will most likely rot forever in a Jira board somewhere and they will keep paying their bill in the meantime.
I know of a hospital which is still refusing to migrate off of a system which is not supported for a decade. Pay more money? 3 people worked on it: Adam had a stroke, Bob retired with enough money, Charlie left the country.
The reality is company execs like to talk about fixing problems with money but rarely are they actually willing to fix a problem with money. In my career I dealt with the exact three companies where execs decided to throw money at the problem bypassing all the fiefdoms and bureaucracies created to avoid rapidly solving problems using money in favor of slow moving, don't step on Jackson the IT manager's toes, approaches. All three solved their problems, destroying the local fiefdoms in a process.
In context of this case - if they don't have money to fix their backups (https://news.ycombinator.com/item?id=26539988) they likely couldn't drop lots of cash without a serious credit line.
A new system isn't going to do them much good when they don't even know who their customers are. I doubt any customer is going to hang around for a year while you rebuild your system from scratch, ignoring the fact that if they're an MSP the customers may not even have their own copy of the data.
All of that is ignoring the likely contractual obligations they're likely under to maintain the data on their systems.
>Also it's a public hospital - for a fraction of $1M they could get people who's only job is to replicate that system running around with pen and paper - they wouldn't have $1M to just drop on a support issue the same day.
Hospital? He said they're an MSP on reddit:
>Yeah we're an awful MSP. I've been trying to find a new job for quite some time.
https://www.reddit.com/r/sysadmin/comments/ma4mwl/the_absolu...
This thread is about a different case.
The ERP system was a nightmare, though, and the company was difficult to work with (key, long term employees). It was a system I inherited, built out of garbage, that I tried to improve. Before I left, I found a very experienced contractor to take over, on-boarded him, and stayed as a support resource for a couple months. Six months later, he's retired and I'm getting panicked phone calls. I valued (and still do) my current job far more, even though I would probably have done better financially in the short term with the alternative.
A few years ago, I heard they migrated to a new ERP at long last.
After 25 years, I'm an expert at making small, efficient software tools to fix local IT workflows. God knows, in the Fortune 250's I've worked at, there are endless opportunities. And, while I've written many solutions, I've seen this political game of thrones thwart my efforts many times. This is how you get the kind of 20-year legacy mess described in the original post: People sacrificing long-term success of the company for short-term success in their careers.
I guess there are 2 kinds of people. Those that think this is just how the world works, and find this to be a perfectly-cromulent way of getting through it, and those, like me, who find it odious. Luckily, I now work for people like me. We do what we can.
He had retired, but took another 12 or 18 month contract flying around the country servicing these things. Apparently they asked, and not wanting to do it he quoted what he thought was an insane "go away and leave me alone" fee, but then they said yes. (I have no idea what that fee was or what multiplier he applied, though it definitely brings to mind patio11's standard advice [1]).
[1] https://www.kalzumeus.com/2006/08/14/you-can-probably-stand-...
If the long-gone ex-employees hated their employer and would be happy to see if go out of business then you're in quite a bit more trouble.
Sorry, I though I was on the Romania subthread.
That's certainly a simple solution.
The problem with simple solutions is that they are near guaranteed to not actually be feasible or realistic.
> The problem with simple solutions is that they are near guaranteed to not actually be feasible or realistic.
This. There is no amount of money anyone sane can be given to stay in Romania. Most recently i’ve read they (hospitals) were tying covid patients to their beds.
Couldn't help but comment on this as some of things i discovered are shocking for an eu member country (or any country for that matter), such as EU leading human trafficking, police corruption, collapsing health care, and the list goes on.
But sure, the simple solution is to give people money, not the constant abuse taking place at work and outside that makes people want to flee that place.
Companies paying more (offering more at least) would begin to slow this domino effect, which raises standards of living and provides more funding for the government to provide services and ultimately pressure for increases in quality of life by the voting public. Everyone wins.
Ultimately not everyone can emigrate - it's an impossibility. The brain drain will continue to be real, so it's incumbent upon these companies hiring to raise their wages and retain talent.
It is really not as easy as go "this is the exchange rate" and simply use that to compare pay in different countries.
Sure, farmers could pay $10/hr, but unless your Doritos were $10 a bag, they’d lose money.
If nobody wants to buy $10 doritos, that means the product can't exist in a market without cheap labor.
"Exploitation" is a relative term, some people would rather be exploited in the countryside of a first-world nation rather than the countryside of their homeland.
It’s all relative.
The Australian national minimum wage is $19.84/hr.
Getting paid about a quarter of minimum wage is very clearly exploitation. There may be other benefits for actually working in Australia, but that's not really relevant here when you're trying to attract native Australians to work for you.
[1]https://www.ebooksread.com/authors-eng/andrew-carnegie/trium...
And it's not just cheap labor, it's labor exploited under pain of deportation.
Edit: that was just based on my general memory of non-perishable food economics. But I googled it and found this (sorry, quora) from someone who would know, a 2 oz bag of potato chips costs 15 cents to produce and bring to (retail) market. Of that, a fraction is potato prices.
https://www.quora.com/How-much-does-a-bag-of-57g-of-Lays-act...
Edit: fix Verizon math.
Same reason I avoid organic whenever possible. We don't need boutique food.
I am telling this as an engineer working in the Canada's oil and gas sector. Since 2014, salaries in our sector have decreased and firms are struggling. "Just raise the price of oil" is not an actionable advice. The price of oil is not ours to set. It's the market's
https://www.lowtechmagazine.com/2020/04/fruit-trenches-culti...
You're absolutely right that perhaps not everything should be grown everywhere, but I strongly believe that a country should have a local source of all the calories and vitamins / minerals they need. In Canada, for example, maybe they encourage farmers to grow greens, brocolli or potatoes instead of citrus for vitamin c, but they absolutely need a source.
Australia does not use this approach, and instead seems to have pushed other countries to eliminate their own tariffs for the TPP.
But imported products don't fly right into the country. There is customs, and denying/tariffing them is a policy that countries are very actively using. So at the end of the day, it's a political decision.
Edit: On the other hand, if you are exporting stuff (oil for Canada), then you can't really do that for the external market. Might explain why the government will let some sectors run with illegals (to remain competitive); the alternative being their ultimate demise.
[0] https://en.wikipedia.org/wiki/Chicken_tax
[1] https://www.npr.org/sections/money/2017/01/25/511663527/epis...
Of course, you might get pilloried for the artificial scarcity, but hey... nobody ever likes the capitalist :)
Actually farmers could set any price if they colluded to manipulate the market. People will buy food at any price and everyday.
[0]https://www.cato.org/policy-analysis/candy-coated-cartel-tim...
USD $5/hour is what I earned as a teenager working the fields by hand (picking corn, beans, and fruit for sale same-day at the farmers small stands throughout the area) in the early 1990s... That's insane. Is that USD $5 or AD?
Now, outdoor work with a high injury rate is a different story.
What I've seen first-hand in some farming communities was closer to $10AUD/hr, but that is still far below our actual minimum wage of $19.84AUD/hr.
The workers were detained, deported, and so on.
After several raids, not so much as a misdemeanor was levied against a single person, nor a fine to the company in general.
It is also difficult because of the two tiers of labor. A farmer will try increasing wages by $1/hour, and still struggle to find enough workers. Wages would probably have to be doubled before new workers—Americans—would start to enter the industry.
I'm assuming you're implying a conversation about China and their manufacturing capabilities. But worst case scenario short of nuclear war, assuming US-China burns bridges and somehow for some reason now US is fully unable to import Chinese manufactured goods; how does this eliminate the US and all other countries from investing in an area that has now become more profitable? You could argue that prices would increase, but then again you aren't owed cheap manufacturing, and subsidies and tariffs are paid by either the taxpayer or the consumer (which tends to have a knock-off effect on the rest of economy, anyway). There was never a free lunch.
That's easier said than done. The problem is that in order to reform immigration you have to do one of two things:
1. Start treating people fairly, with the result being higher food prices, which will make some of your constituents very unhhappy
2. Codify the current class structure into law so that you can continue to exploit cheap labor legally. The optics on that are really bad.
Exploiting people off the books is the path of least resistance for everyone, including the workers (manifestly so, or they wouldn't risk life and limb to be here).
This is a political talking point that has been repeated so often that people believe it is true.
During the recession of 2008, there were stores on local TV news in many places showing unemployed Americans waiting in long lines for the opportunity do do those jobs that the politicians kept saying no American wanted to do for those wages. Meatpacking plants, especially, had more than enough Americans to choose from who were willing to work for low wages.
Because these things happened in "flyover" states, they were only very rarely shown on television on the coasts, and so the meme stuck.
A similar oft-repeated lie is that American farmers are all a bunch of Republican hillbillies who hate brown immigrants.
Around 2010, I attended a conference in Seattle for farmers in the Pacific Northwest. The main topic of discussion was how to increase legal immigration from Mexico so that they could have more farmworkers to pick the cherries and apples and mint and whatnot. These people were talking about goals of making people from Mexico and Latin America U.S. citizens in staggering numbers.
The more you actually go places and do things and meet people and talk to them, the more you realize that the words coming out of the political organizations on the coasts are mostly for their own benefit, and do not reflect reality.
That and Republicans have grown way more hostile to immigration since 2010, so it's an interesting anecdote but not sure how well it translates across the US, where polling shows farmers are overwhelmingly Republican who tend also tend to hate immigration legal or not.
source?
When you say it's an "educated" guess, educated how? It only seems to reflect the usual bigotry that comes out of the coasts.
polling shows farmers are overwhelmingly Republican who tend also tend to hate immigration legal or not
If you're going to make a statement that sweeping, you really should back it up with credible sources.
I lived first 28 years of my life in the rural south and now live in the midwest.
My entire family and most of my friends growing up are the people you claim I'm bigoted against...
"If you're going to make a statement that sweeping, you really should back it up with credible sources."
The fact that farmers are generally far more conservative and republican than the median voter or average American isn't exactly some shocking new insight.
You seem to have never spent much time around these folks or have done much research on them if this comes as a surprise to you.
https://www.washingtonpost.com/news/wonk/wp/2015/06/03/why-y...
Are these just bigoted facts "out of the coasts" too:?
Your second point about farmers trying to increase legal immigration in 2010 clearly indicates most Americans won't regularly do this work for the wages being offered. Why else would farmers be looking to hire immigrant workers just 2 years after locals were lining up to get these jobs?
While it would be nice if there was a way to get enough immigrants working legally to fill those jobs that doesn't seem to have happened. Short of another large economic downturn I don't see Americans working those jobs either. That leaves people who are working in the US illegally to fill the gap.
> Around 2010, I attended a conference in Seattle for farmers in the Pacific Northwest. The main topic of discussion was how to increase legal immigration from Mexico so that they could have more farmworkers to pick the cherries and apples and mint and whatnot. These people were talking about goals of making people from Mexico and Latin America U.S. citizens in staggering numbers.
Farmers are, for the most part, single issue voters: Whoever has the best plan to support agriculture and small businesses gets their vote.
How do you determine from TV that the people lining up for meatpacking jobs were not undocumented immigrants?
Like, it is generally accepted there aren't any in "flyover states" or something?
I never really thought of it until this moment, but you know how within 100 miles of any border, your constitutional rights are somewhat compromised so DHS can operate? That ought to mean a pretty substantial incentive for people who are undocumented to migrate inward. So I would bet they do.
Companies aren't going to say "I guess we're inefficient and should just die." They're going to fight to survive and thrive, which is what you're seeing.
Then again, some companies ARE just cheap. You never know.
They said the standard is 30 minutes per room per day. I figure at a very generous $50 per hour (100k / year) it should run $25 to have my room taken care of. Now that doesnt cover the shared costs - doing laundry, cleaning the pool, front desk, etc... but my room was something like $150 per night. How the fuck can they not find people to clean rooms?
Answer: they're too cheap to pay a decent wage. But why? My suspicion is that all these places are actually leveraged far too much and paying rent/interest/dividends so much they can barely operate.
I suspect low interest rates and other incentives to "stimulate" economies are actually causing these problems.
sorry for pun. But that's not always a solution. Sometimes employer is greedy. Sometimes his business just does not allow to pay enough salary.
You'll continually be amazed at the human tendency to double down instead of admitting a mistake.
I would anticipate a mix of: the people who know where the right bodies are buried, the right incentive (yes, that includes money), but also people to look over the entire program, a designer to make sure it is designed correctly, an organized person to make sure it is properly documented, and someone to make sure it is adhered to in the future.
Asking because, while, yes, throwing money at the problem might be a solution, I think the obvious bigger issue is how does one allow themselves to get into this situation?
Or rather, how does one avoid getting into this situation. Because it seems to me that the software industry of late has become so mired in esoteric layers of abstraction that this feels less like an outlier scenario for a mismanaged company but more like an inevitable one for many otherwise well managed companies in the industry.
Documentation?
you handle this by having and exercising a disaster recovery plan and having (and listening to) competent people that would spot this issue in a jiffy.
Significant documentation is also not something easily written in slack time. With any non-trivial project good documentation is a full time effort. It also needs to be updated as the project evolves lest it become out of date and incongruous with the behavior of the system.
There's dreams of in-line documentation but that's only going to cover individual methods and classes and not necessarily whole modules or major subsystems of a project. It's also not going to necessarily cover in-line documentation from other parts of a project in different languages e.g. comments in config files.
There's lots of technical ways to help documentation but there's no replacement for documentation being a high level project goal with resources allotted to it.
For example, in code: good naming, simple structure, unit tests, and useful comments all can count. A set of easy-to-understand unit tests that cover real scenarios is much better than even the greatest documentation ever written.
A fully scripted build and ideally also deployment (CI/CD pipeline) beats a checklist with tens of items that have to manually followed.
An issue system with good descriptions of changes (including why and what), reproduction steps, expected outcomes, links to related issues, etc also can go a long way.
All of this stuff has the side effect of making day-to-day work simpler, overall, while actually just being a part of that work.
1. Naming things is hard. It's a bit of a joke but good descriptive yet usable naming is hard. It can get harder after refactoring or adding in features you didn't expect the last time you were in the module.
2. Project structures start out simple. Then you need to add some feature or fix some major issue with a due date of last week. As a project gets bigger the simple structure of yesterday might not serve the needs of today. Resources for refactoring are usually somewhere around the priority level of "Good Documentation".
3. Comments need maintenance like explicit documentation. Tight schedules lead to "# TODO add documentation here". Then you get the assholes that believe comments or in-line documentation is something you put in check in notes.
4. A "unit test" that covers a "real scenario" is not a damned unit test. It's an integration test (or however you label it) and it's not likely something your CI/CD system can run on the regular. If a real scenario is touching a production-load DB or something it's a big deal to run the test. Tests like that need resources to run and non-trivial resources to write. You're not just making sure a method returns a float or throws an error correctly.
I'm not saying your suggestions are bad or shouldn't be used. The issue is management rarely sees any value with documentation, tests, or even code cleanliness. They're rarely incentivized to care about those things so they don't. They get their bonus on feature checkboxes or hitting a deadline. At the end of a project they'll schedule a "documentation" sprint or some dumbass thing that accomplished little useful output. It's worse when you've got a "move fast break things" chanting moron doing the scheduling.
The specializations common on a team even 20 years ago was much closure to the "Surgical Team" model from the mythical man month than what we have today.
At a lower level, it's allowing individuals to become knowledge silos (or having a bus factor [1] of 1); lack of documentation; not having+testing disaster recovery plans; neglecting systems, code bases, and platforms; letting technical debt accrue endlessly (or worse: not even realize that's what you're doing).
How to avoid? Make sure there's a strong, competent technology leader with a vested interest in the company succeeding, and empower them to make the changes they need to.
What should that person be doing? Making sure all the stuff I just said doesn't happen.
I guess any company where leadership got there without progress on the technical side. Law firm partners get there after being lawyers. Consultant partners get there after consulting.
But if you are the CEO of an IT company without ever having to code, then you may never know the things you must take care to prevent a washout.
IMO, the problem is technical debt. Most likely, in the majority of these cases, the engineers working on the system knew perfectly well that it was a mess, was very difficult to learn how to maintain, and many parts were dangerous to modify. Most likely they told management multiple times that this was a problem and what the consequences of it would be. They would have refused to budget resources for improving it, so everyone either saw the writing on the wall and quit, or gutted it out and learned how to manage the system eventually. Maybe the system limps along for a while longer, but it's only getting worse and the knowledge needed to maintain it more siloed and specialized. Eventually you lose a key person, something breaks that nobody left can figure out, or maybe the whole thing just implodes like the main story here.
To avoid it, you must understand the pattern, hire good technical people, and give them the resources needed to improve the system in an orderly fashion.
The relevant part of the article:
> To anyone outside of the ballpoint pen manufacturing world, it might seem hard to understand what, exactly, is so surprising about this development. China already produces 38 billion ballpoint pens a year, according to China Daily, which is about 80 percent of all ballpoint pens in the world. That's a lot of pens, but there was a catch: China had long been unable to produce a high-quality version of the most important part of the pen, its tip.
> The tip of a ballpoint pen is what makes it a ballpoint pen. At the tip, a freely rotating ball is held in a small socket which connects it to an ink reservoir that allows the pen to write or draw lines. Manufacturing a ballpoint pen tip that can write comfortably for a long period of time requires high-precision machinery and precisely thin steel, but for years China was unable to match those crafted by foreign companies.
Mentorship and documentation.
You need to set aside time and resources for these two things, and make contributing to them worth something in the performance review.
If all you do is count JIRA tickets closed, you'll forever end up in this situation.
Usually, since admitting that they were stupid to let all the knowledge of the system walk out the door requires more emotional and intellectual flexibility than managers who let this sort of thing happen possess, the biggest obstacle isn't even the money... it's getting the managers to admit they're idiots.
> The company that built this ERP solution went bankrupt 6 years ago, and we don't have the source code. It uses PostgreSQL and MySQL, and also has a built-in key-value database for which none of us has any credentials. We don't have the source code for this software (it's built in C and delivered to us as compiled packages..).
Sounds like finding their old employees wouldn't help. They need to find old employees of the company that wrote the system and hope they kept a copy of the source code.
Which from a legal perspective might be fraught with peril, at least in theory. In practice, if the company folded then it'd depend entirely on the litigiousness of the current owner of the copyright, which could range from "nobody" to "some holding company consisting almost entirely of lawyers chomping at the bit to eat you for dinner".
Anything but pride, I guess.
Years ago I left a rather awful company - I left on great terms with my boss, though I hated the company. I was the last person who knew how to release X product. While I documented everything, after I left, the wiki lost its mind, backups apparently weren't a thing, and the director of IT moved to Bermuda.
Fast forward 6 months.. a contractual obligation existed to release the software. They still hadn't successfully built (yup autoconf, make and friends), let alone know where to build it. I got the call you always dream about, 30 days prior to the release being due.
My boss asked me to do him a favour - I reminded him that I quit due to a lack of promotion and raises. He asked what I want, and I asked for 125K, which clearly he scoffed at. Then I reminded him that it was a 10M deal they'd signed, because we had an all hands. The CEO got involved and screamed at me for 'torpedoing the company'. I walked out and calmly informed them both along the way the price was now 250K, non negotiable.
I ignored every single mail (and legal letter) sent me way. With 10 days left, they agreed. I had the company deposit a cheque in escrow with lawyers of my choosing. We signed contracts, and I did my thing. Four days of work. It was well worth it.
Dougie, if you're reading this I hope your cringing.
That an overworked employee who had already left the company would stick up for himself? Or that a company would go to such efforts to try to manipulate a former employee?
I realize my tone doesn't hide my feelings, but even if we disagree I'd still be curious to hear more.
Walking away from a toxic situation is not "psychopathic." It's common sense and if we all did it more then the psychopaths who expect us to stick around through it all would finally learn how to treat other people with respect. It's a job, not a marriage.
Sadly, I've seen this. Not first hand, but it happened to a friend and I have no reason to doubt his version of the story.
"We tried to hire back an employee to build something that we didn't train or try and figure out ourselves and a deadline forced us to pay a large lump sum to that employee".
No matter how you slice it, that's really shitty planning on the businesses side. They should have spent the time and money learning how to maintain their shit. By not doing it, they paid a huge price.
Luckily for him they decided it's easier to just pay the fee.
Because here's the thing, even with how weak US labor laws are, one thing they don't allow is for a company to come back and sue your for incompetence. So long as they didn't go out of their way to sabotage things, they are free and clear.
I think in general all of my comments on this thread have been horribly misunderstood - I was just trying to say that if the company was scummy, instead of paying him the 250k fee, they probably could have sued him instead. I'm not saying that should have done, I'm not saying it would have been the right thing to do, that he deserved it.....literally nothing of the sort. Just pointing out that the company could have made an argument against him in court. As many other people pointed out to me, it would have almost certainly failed. But I don't understand how you can deduct where moral considerations are on my priority list from this?
Especially ex-employees who've been gone 6 months already.
Shouldn't be hard to find a no win no fee ambulance chaser to happily take them on and get costs awarded and damages as well...
- if he can prove the threat, the company will be in an even worse position
- given the rather lacking precedent (as pointed out by the sister comment), there's a good chance the suit would be thrown out pre-court and therefore be rather cheap
- given that the OP seems to be a valued engineer, it is reasonable to assume that he has the f-you-money to fight this out
- even if won, the drawn out battle would take far too long to fix the software in time and, given that they then lost a 10 million contract, they lack the motivation (and maybe money) to continue
- lastly, even if they won after loosing the contract, they'd probably not be able to extract much in damage payments
So it's a rather empty threat.
Every negotiation is about the best alternative the parties have, and it's generally harder in real life than in fantasy because you underestimate the alternatives to a deal.
Applying maximum leverage has a way of creating "out of the box" alternatives. And vindictiveness.
I have no need to judge it if it were true.
This is exactly the free market at work regarding labor. Have a problem with the free market if you like, many of us do but do not accuse this poster of criminal action equivalent to a protection racket.
To clarify, the market would likely not allow this specific work to be sold at this price normally. But the buyer wasn't just buying specific work they were buying specific work to be completed by a specific date. The work in isolation does not have this value but the work completed by a certain day does. Companies use this rationale all the time charging more for expedited services.
It's especially common in construction and manufacturing where costs double or even triple when expedited delivery is required.
The way the story is told the price doubled after and because of the CEO screaming.
The market has nothing to do with that except the good old "The fair price is the one that god paid, uh.” I suppose.
I'd be inclined to believe that if there were some other providers but as it is told working with OP was the only solution. And OP's first offer was 125 000. Two screams later it's 250 000 ? It has nothing to do with economical theory or free market interpretation.
And yelling, made the prise jump closer to market value.
I find this explanation (from top comment) more in line with how the world spins:
> If they left under bad terms because the business was a bunch of dicks, expect to pay 10x market rate. If this is truly "fix this or the business is out of business" - then it shouldn't be a tough decision to make.
So, “being a dick” has more to do with price raising than any free market justification (which I am okay with, just don't try to convince me it's the only thing at play. The offer would have been 250k from the beginning then. Or 500K to get 250K, whatever.).
Any small or independent contractor in any business will eventually have a blacklist, or a list of troublesome customers who get charged extra. Most of my contractors charge me less because they like me (and I pay on time). My HVAC contractor charges a particular customer double just because the customer is a rich entitled asshole; the contractor has decided that it's only worth dealing with the man's unpleasantness if there's more money to be made. Apparently the man has been blacklisted by other HVAC companies, so he might be in a tight spot, but whatever the reason, he ends up paying a lot more money for the same service.
This is a completely market-based mechanism: the troublesome customer has limited options (in this case, maybe only one option), and is thus obviously disadvantaged in negotiating price. Regardless of the myriad motivations involved in price discovery, it all leads to the same place: if the price wasn't worth the service the customer was to receive, they would not have paid it.
Honestly, this isn't mysterious or profound. It's economics 101.
Yes.
> It's economics 101.
Not even that. It's payback, that's all it is. There's no need to call for a more elaborate explanation for what was at play here. Not that I disagree with it. It's humane nature.
> My HVAC contractor charges a particular customer double just because the customer is a rich entitled asshole;
Like you say. No need to even know about market dynamics to (wisely) choose that option. Because the other guy is an asshole.
Second, you speak as if his claims were irrational. The way I see it, however, you would set for 125k (probably much less judging from what you write) while he, the sentimental fool, scooped up double the money. Therefore, your decisions, which you personally claim are rational, would have left you with half the money the "irrational" player would get. Seems to me, the "irrational" is actually the rational one, as he netted more profits. Economy and market are not math games in a vacuum, and players are not formless ideas.
The way I see it you are putting words in my mouth and I am definitely not interested in having a discussion like that.
I never insinuated OP was a sentimental fool or wrong to raise the price. I am calling bullshit on the free market and econ 101 as an explanation of why the price got doubled and call it like it is: it's payback from OP because they were dicks to him first, everything else is pedantry and useless justifications.
edit: if anything, OP could be considered a fool since he obviously gave an under the market quote as his first proposal. Had he started with 250K from the beginning he would have gotten more since there likely would have been screams anyway and he would have risen the price all the same.
And yes. I completely agree with your edit and said as much elsewhere. Their acceptance of 250K showed that the OP under-priced their services at 125K.
And yes. Free markets produce monopolies. Especially when time is important. As I said above, the critical good here was not the delivery but the delivery within a certain time.
This is literally the definition of price discovery in a free market. If the price was not worth the service the company was to receive, they would not have paid it.
The fact that they did pay it clearly indicates that the first price was far below what the "market" would bear.
I think so too, however maybe better play safe I guess, and be content with 250? The negotiation partner seemed a bit irrational to me
Anything distorting those conditions - e.g. there's only one seller, only one buyer, a time constraint, a skill constraint, an information disadvantage, will distort the pricing as well. One trade isn't enough information for a market to exist or to settle on a fair price. I agree there that the market has nothing to do with it.
But I disagree with your implication of "the fair price is the one paid" is somehow unfair in an ethical sense. (I also think OP is in no way obliged to be "fair" in any ethical sense to the company, but that's another matter). The company got a $10M contract for $250k, versus losing the contract, is that unfair? If that really was the pivotal factor between losing the contract and not, the company ought to benefit by spending right up to the point where the contract becomes a money loser, to not lose it. Would it be unfair for the company to get an $8M contract by paying $2M? Or a $2M contract by paying $8M? Is OP obliged to help the company make as much profit on the contract as they can, by charging them less for his services?
This is non-sense. Those conditions specifically define a market and make it distinct from any other. To say a market with any of these constraints is "distorted" is to say all markets are distorted and render the term meaningless.
Also, I never said this one trade signified a market. My point was that this is a consequence of free markets. You can make any single trade "irrelevant" by choosing arbitrary criteria by which to separate it from its market. The company could have certainly hired another engineer in hopes they could complete the work in time while running the risk of blowing the deadline. But it was freely and fairly worth at least 250k to them to guarantee the job could be done correctly and on time.
This kind of thing is such a common part of the free market, that there is a whole aspect of contract law that pertains to ensuring companies don't get into this position. It's why contracting completed work is so important. Leaving things partly done leaves you vulnerable to others eating right up to the tiniest margin of your profit.
This person left the company, and documented everything. What should they do, come in and save the day for $15/hr?
If your company were in the hands of incompetents, but still kept going in spite of the incompetents because it had gotten its foot wedged in the door or whatever, and then fell apart in ways you could only scorn, how would you view that sort of situation?
This is pure slander
The company can avoid this if they only care to maintain the documentation / wiki left behind by said person
When they failed to do even just basic proper operations, it's their own fault.
I'm a business owner and a consultant, and I too charge more for these kind of asshole clients, because of the extra stress.
But nowadays I just try to avoid them, it's just too damaging for your own sanity. Their kind of stupidity, self-entitlement, and ignorance is on a completely different level.
<pedantry>It's libel (written) not slander (oral); "defamation" is the generic noun. </pedantry>
It seems that paying to skywrite (you know, small plane making artificial clouds) could be slander but not libel
Citation? By that light, a DVD containing defamatory speech would be libelous, but that's generally not how courts see it.
From the (highly-influential) U.S. Court of Appeals for the Second Circuit, summarizing New York law (which is quite typical): "Defamation, consisting of the twin torts of libel and slander, is the invasion of the interest in a reputation and good name. Generally, spoken defamatory words are slander; written defamatory words are libel. Libel is a method of defamation expressed in writing or print." [0]
https://scholar.google.com/scholar_case?case=601727562995661...
Coding standards, documentation requirements, meetings and code reviews exist for a reason, too. It's not like companies are defenseless.
This is a gross failure of leadership at the company, and the former employee has every right to negotiate compensation for the inconvenience of having to work for them again.
And you identify with this band of miscreants? That's embarrassing.
You may be doing any of that already, since I don't know anything about you except your one comment.
But the answer to your fear of exploitation is avoid making people hate you/your environment enough to want to protect themselves from it.
That's probably what they deserve, if they come asking for a favor and then start getting abusive.
The company could have apologised, and do a lot more ego stroking before asking a favour which they can pay for and maybe throw in an all expenses paid weekend with the SO as a thank you. Not only will you get goodwill but future problems are going to be significantly cheaper to solve. Even if you have a bad break up - as a business owner you should understand the value of mending bridges. It's cheaper than losing sleep at night and losing your steak dinner to a bad stomach.
He left detailed instructions how to release the software. The company managed to screw up so bad they lost the manual he made sure to write on their internal wiki.
I feel for them but they failed to protect themselves.
If you do not have the money to pa for someone's services you cannot complain. You say you have a company? Have you ever let a client get your products by paying less than their value or nothing at all? Work itself is a product too.
For a while, I made it game to light the various communications on fire in interesting ways, but it got boring.
Did they have a legal reason to think so?
The company could say: we’re going to pay you 125 and you could sue us and spend 60k and two years or just settle for that, after he’s already done the work.
The guy could demand the payment directly to his account and then not deliver, and the firm would be in the same situation.
By depositing the funds in escrow, they’re making sure that the money is there and in the hands of people whose only incentive is to give it to the correct party upon fulfillment of the contract.
The contractor knows that the money has been paid by the company and won't be withheld after the job is done.
The company knows that the contractor doesn't have the money until the job is done.
I have bosses at previous companies that I'd help for free today because they were good to me and we parted on friendly terms. One boss helped me move my house once, and I'm not about to charge him to fix some random issue.
I have other bosses whom I'd require a cleared wire transfer of money before I'd lift a pinky to help them, because they were not nice people and I don't want to associate with them or assist them with anything. If someone in this camp approached me, I'd have a story just like yours afterward.
People like that would much rather spend $2.5M on courts and lawyers than let a worker walk home with 250k, or even 25k. Agreeing to a deal like this is an incredible minefield that may blow up on you even years later.
Thinking you're fucked is unhelpful, because you start talking about resumes and expecting some deus ex machina to appear and resolve the situation (for good or ill).
Also, make sure everyone knows you are going on vacation once you resolve things.
If anyone happens to know of a career path or company that does that sort of work I would be interested to hear about it. Bonus points if the pay is half decent.
However, not only is not everything a true positive, probably only about 0.001% of things are true positives, among a sea of alerts and reports and dashboards across myriad systems. Some coming from your SIEM, some generated by security appliances and products, some from internal employee reports.
An ideal place will have people who continuously work on trying to reduce alert fatigue and false positive noise - but, in practice, at most big companies it's probably like working at a fire station and getting hundreds of dispatch calls per hour, every hour, every day, each about a potential fire at a different residence. And then you drive up and see they just used the stove for a few minutes or a character said the word "fire" in a TV show they were watching.
But you have to urgently show up every time no matter what because, occasionally, the house actually is engulfed in flames and might be on the verge of igniting the whole town.
And for some reason warehouse managers seem to swear by it, so it still gets a decent number of new customers - meaning those customers need implementers. And since it's a giant pile of hacks, the average deployment needs a whole lot of customizations - so more implementation man-hours, and a steady stream of maintenance man-hours. Thus, I get pinged every once in awhile for some long-term contract. Too bad they didn't ping me when I was actually looking for work last year, else I probably would've accepted one for the hell of it. Still tempted to; would be an interesting side job, albeit probably soul crushing.
It is... interesting. I have been doing this over last 4 years. The biggest problem, as I pointed out before, is the disconnect between what companies claim they would do to fix the problems vs. what they would actually do. So ~90% of selling is repeatedly explaining the same things to different people whose jobs are in the eminent danger of being restructured/deprioritized/eliminated if your services are successfully engaged.
<multiple layers down, Jack, head of IT in charge of backups>: I'm uncomfortable sharing this information with the outside party. I will need to get the approvals for <blah blah blah blah>
...
CEO: Yes, tell them to do this.
...
Jack, head of IT in charge of backups: Ok, you can have this access.
Consultant: Great. <comes back ten minutes later> It says I'm not authorized to perform this operation. It is a blocker, could this be fixed so I could continue?
Jack: This was not in the scope of authorization that I have received. I'm uncomfortable providing this level of access without additional authorization.
<repeat>
When you’re in this role your goal and job is to build relationships and help people. That includes Jack. The job is getting done with or without Jacks help. So he can either jump onboard or he can be removed and someone else can be added to give access.
In the end people like Jack benefit nothing from being gatekeepers. Because the third time I have to ask for permissions to be changed - I’m asking for Jack to be removed from the task and someone else to be put in place.
When you are in a firefighting role, your job is to fix the fire, not appease Jack and build relationship with him, the person whose practices created the situation to begin with. Unfortunately, even in those cases the highest level of stakeholder who engaged your firefighting service may not be willing to tell Jack's manager or Jack that he will do what he is told or he will be removed.
Jack's gatekeeping is what keeps Jack employed. The company's ultimate bosses will need to make decision if they want to remove Jack and solve a problem or if they do not care that much that the problem is quickly solved. In the vast majority of cases no one in the company who can fire Jack wants to be seen as a bad guy.
The guy did very creative reporting though :-)
And the problem was not fixed, of course.
I tried to help him as much as I could, but at some point, the thing is so intricate that you basically give up.
If you are an external consultant (with no prior knowledge of the company's systems/processes/hierarchies) brought in to "fight" a particular fire, how do quickly get ramped up to a level of knowledge where you can analyze the root cause of/recommend a fix for a particular "fire"?
To be fair, the number of fires is finite, new ones are just getting started faster than they are put out.
Sure it is fun, but the stakes can be high and the stress gets piled on.
It starts with congratulations all-round. You are a hero! Good job! Nice save! Your boss's boss's boss's boss comes over and thanks you personally. Take the rest of the week off!
Later, its people asking when will you fix it? Why haven't you got this back working yet? Didn'y you fix the same thing last month? (no) We really need you to fix this before 5pm or the TPS reports wont go out, and the management will be pissed.
Failures become normalised. They get reliant on people doing heroics. People forget that the systems are crap and need investment, and start to rely on you being there to fix it, and if things don't get fixed then it is your fault the TPS reports didn't go out, not tech-debt/lack-of-investment/bad-design/whatever.
"If you do things well enough, nobody will think you did anything at all."
>We can't access the backups. Even if we could, we noticed two years ago when trying a restore from the backups, that it doesn't work. Booting the restored server leads to a kernel panic we couldn't figure out. Management said we don't have enough money to fix any of this.
It's often even worse. Many years ago I was responsible for backup a DEC Unix system. Asked for funds to conduct a disaster recover exercise to find out if our backups worked and was turned down even before we had tried to work out how much it would cost.
I often wonder what proportion of backed up files and systems are actually in a state to be restored and have the necessary tools and expertise to do it in a timely manner.
An even worse problem in some places, including the one I worked at, is a tendency for users to assume that backups are kept forever and for the management to totally neglect archiving of important documents such as specifications, drawings, and design calculations of products that have an expected lifetime of over fifty years.
I had to inform several users that I could not restore the file that they had only just noticed was missing because it was last seen more than two years ago and presumably disappeared longer ago than our longest backup cycle which was one year of monthly full backups.
If you can afford 12 months of backups then you can almost certainly afford >=12 years of yearly backups by buying a new set of monthly tapes once a year and taking a set out of rotation and into archive.
(GDPR and earlier laws, etc.)
The best way to delete user data is to encrypt it all with per-user keys so they can be shredded immediately upon request. Backups of user keys themselves can be very short-lived since the keys are static and the lifetime is scoped to GDPR/other laws. Then the encrypted data can be archived indefinitely, so long as encryption keys are never stored in those archives.
Typically, this kind of decisions are made more or less implicitly during long meetings about priorities, often without any minutes. Managers will presumably try to hold (senior) engineers accountable as they had not made clear enough how important that was.
I obviously don't know who is right here.
Up until now, they didn't know what 'backup not working' meant. Now they know.
Not being bitter -- there was a failure of communication from both parts.
While you can and should be the cool head in the room, it's a route to burnout if you end up being the hero who has to keep saving a dysfunctional organization and management from the consequences of its own failures. By all means do your best fix the immediate problem but get the hell out as soon as you can.
The same dysfunction that led to disaster after disaster is likely to lead to very little reward (if any) for actually being the hero, also.
In war room after war room there has always been at least one person who would be categorized as 'disorganized' by their coworkers. When thrown into a stressful situation they either don't change or become more focused, while the 'organized' people crumple like tissue paper, so Mr Disorganized ends up either fixing the problem, keeping everyone on task until it does get fixed, or insisted on some process/tool improvement that made the problem easier to identify or fix.
I've been diagnosed with ADHD and I'm not sure I'd qualify as having that trait. So I don't believe this a general sweeping statement you can make for everyone with that, or if your boss even had ADHD.
But maybe check this out and ask if it rings any bells: https://www.nhs.uk/conditions/attention-deficit-hyperactivit...
I have ADHD and recently started an SRE squad in our company since I tend to be the go-to person to fix any production issues.
I had never really thought of a connection between those two things until now.
All computers except one, which due to a power failure had been offline the whole time.
This computer had the critical DNS information needed to restore the network.
Only problem: it was in Africa, in a country that required Visa which would take weeks to apply for and receive. So someone from the African office had to bring the hard drive to an airport that someone from the London office could travel to and pick up the critical hard drive, then bring it on a plane back to London.
A sweaty and nervous trip, I can imagine.
Whole story here: https://www.wired.com/story/notpetya-cyberattack-ukraine-rus...
0: https://www.zdnet.com/article/ransomware-the-key-lesson-maer...
1: https://portswigger.net/daily-swig/when-the-screens-went-bla...
Things I would do that I haven't seen mentioned yet:
- Grab images from the relevant servers to have current backups before messing with anything.
- Try tcpdump-ing and decoding the custom KV protocol - other databases are complicated, but things like memcached/redis/... have trivial wire protocol. Maybe it's possible to recover some data by listing all entries.
- Run `strings` on the KV binary or `file` on the data to make sure it's not just some open source project with a proprietary interface. With any luck it's just something like bdb that can be accessed externally.
- Hook up a debugger to the KV and try to recover the keys (or maybe just strace to see if they keys are in any files)
- Start tcpdump on the old DNS IPs. They mentioned they have no idea which domains they own anymore - as long as NS on customer domains wasn't trashed as well, they could collect queries coming in and recover a list from there.
- See if the customer domains can be collected from billing data / some email store of notifications.
(as much as I try to design for no disasters in production, the rush of "this is so FUBAR, what insane thing I can do to fix it" for me is an amazing feeling :) )
Maybe it's my (US) American experience, but being unable to quit a job strikes me as a terrible policy. This guy(gal?) is really required to submit 3 months notice or face some sort of repercussions (financial, I assume)?
Does this 3 month period apply in reverse (employer can't fire employees without a full 3 month warning)?
Not that the American system (no cause firing/quitting with no notice) is perfect, but 3 months? Yikes.
Yes (in France at least) it is on both sides, unless both sides agree to shorten it
> face some sort of repercussions (financial, I assume)?
I'm speaking for France. This is a tricky subject, you almost can't get any repercussion. But it can sucks, like the employer can refuse to fire you and suspend your contract, in the mean time, you are not paid but are not allowed to work for someone else.
Of course.
Different notice periods for employers and employees would be outright illegal in most of Europe.
AFAIK more common is "lower notice period for employers is illegal".
See my comment below.
Do you have any sources on the legality of different notice periods in Europe?
There are three types of employment laws herearound:
- Statutes can be changed freely if both agree
- They can be changed only to the employees advantage. For example: minimum vacation days are 20 days. A contract can stipulate 30 days, but not 10, or 15 days
- Some statutes cannot be changed at all.
I can obviously not speak for all of Europe (plus INAL, etc), but would assume that it's comparable in most of Europe, or even more stringent in terms of employee protection. Employee laws here are quite liberal compared to other European countries.
Switzerland, default rules regarding termination (if I recall correctly). * first month of work, 0 second notice * rest of the first year of work, 1 month notice * second year of work, 2 months notice * years after third, 3 months notice
Impossible to terminate: pregnant women, before and also during the legal maternity leave. Common termination is the day upon returning to the office.
Personnel called up for the army or civil defence. People on a fixed contract.
Employer must give notice but may be without reasons. Without reasons often leads to lawsuits, because terminated will claim one of the illegal (discrimination etc) reasons applies. Still cost of a lawsuit is normally not prohibitive.
In case of critical financial issue termination maybe shorter (mass-layoffs).
You can still quit if both parties agree. I've done it before.
The clause is there to avoid this type of scenario where a key employee just says fuck it and leave.
It works both ways. So a company can say fuck it and fire you just because they want to.
It's a fantastic system. It can be exploited, but usually works really well for employees and it gives you a huge safety net you can (usually) rely on.
"Can't", right? Otherwise it sounds like American at-will employment.
The law mandates that the firing notice period cannot be shorter than the resignation notice period.
If they are at the "fuck it" point, what's going to force them to be useful and productive during the forced 3 month notice period?
If you reach that you just put in the notice, take whatever vacations you have and then put in sick leave.
But 3 months notice is used for, so the company can hire a replacement for you, and you can help train him, and transfer knowledge.
A lot of people reading this site work in companies that have huge teams, but most people in IT I know from EU (where I live) are one or two person IT teams. 3 or 4 person team is considered almost medium sized.
In such situations you need yo hand of your task responsibly.
And that the one thing that will get you burned, if you leave your previous employer hanging.
Nothing - beyond professionalism. It is common for knowledge transfer to happen in this time, and any outstanding vacation days are usually taken at this time.
There's a huge cultural difference between US and Europe (rest of world?) when it comes to going to work after serving notice. It appears the employer/employee relationship is more adversarial, so much so that the employee is not trusted/expected to do the right thing after being fired (or after they quit) - the whole "immediately getting escorted out of the building by security" is unheard of in Europe.
Three months is the typical notice period for tech workers and other highly-skilled jobs. One month is the legal minimum in Sweden, and certainly several other countries. Of course it goes both ways, in fact it's even more difficult for the employer to get rid of an employee. An employee doesn't need a reason to resign, an employer needs a valid reason to fire an employee, in addition to observing the notice period.
To complete the picture, it's necessary to say that probation periods exist. Employees may be initially hired on probation (typical for fresh graduates), which has a shorter notice period, and either side may unilaterally decide not to continue past probation. That is, an employer unhappy with a probation employee doesn't need additional legal grounds to say "we're not proceeding with full employment after the probation period is over".
Oh, I agree. But 3 months seems crazy to me. If I have an open position and it isn't filled in ~4-6 weeks, it often gets given to some other team (on the basis that if I really needed that position filled, I'd do so). I can't plan hiring over that timeframe.
And when hiring employees in India, we've had them claim a 2-month resignation period, but they just end up using that time to find some other opportunity on the previous employer's time. It's a real mess.
If you're in a culture (locality + industry) where a position can be filled in 4 weeks, that's one thing. If you're in a culture where every experienced person has 3 months of notice, then nobody would take your inability to fill a position within weeks to mean that you don't really need it.
I have some experience with part-time contracting with US customers. Seeing how quickly people can join or leave there is certainly a culture shock!
We have a bunch of employees in Europe and everyone involved except the very marginal employees sees this situation as strongly negative (and, in fact, an argument to not hire in Europe at all).
I don't need my 3 months now. They might be useful if my employer axes a whole department. They're useful for people with family commitments giving them less mobility.
What you might not realise is that these three months would be the default legal position in case of disagreement. If you want to leave earlier and the employer agrees, you can usually leave earlier, too (with them being off the financial hook).
It wasn't a problem. I was on good terms with the previous company (I'm still getting invited to their anniversary parties and stuff like that), so there was no issue in working something out.
Also, remember that ~6 weeks of vacation per year is common outside the US, so often it boils down to taking your remaining vacation. (Though I think you need to be careful with taxes and other regulations if the contracts overlap.)
Just like they have also ruled that mandatory Jury duty is not in violation of the 13th amendment either. Funny how the system protects and perpetuates itself.
But with respect to your point, I am nearly 100% sure that it is not the case that employers can use financial penalties to enforce such a contract. Anything you are paid as wages is pretty much untouchable after it is paid out, unless it was paid in mistake or you intentionally damaged the company or something. The best the company can do is withhold potential future earnings or bonuses, or take back conditional non-wage payments like bonuses within a certain window of time.
This is part of why golden parachutes exist. Your contract has very generous severance clauses which award you lots of money if you follow a certain procedures for leaving, which involves advance notice and such. Fail to follow that and your paycheck just stops the day you walk off. So there are incentives, but they can't go after your already earned wages, paid or still due to you.
That's pretty much what I meant although, as noted in another comment, I assume there are cases where they would try to claw back relocation or signing bonuses. Also, of course, things like RSU vesting.
>they can't go after your already earned wages, paid or still due to you.
Certainly, and depending on how paid time off accrues, anything earned there--which is one of the drivers with "unlimited" PTO because that's not owed when you leave.
There are federal laws that limit this, typically applied to larger employers making entire business lines redundant (closing a factory, etc). State laws very, some with zero extra protection, some with a bit more, but AFAIK none with European-style rules.
In professional jobs, it's traditional for an employee to give 2+ weeks notice. Most employees will try to stay long enough for a reasonable transition - depending on position/project status, that could be 2 weeks or a month or two.
It's traditional for employers to give 2+ weeks severance. Most white-collar employers have periods that start at 2 weeks and grow with tenure. Including benefits is hit or miss, and a HUGE issue because health insurance/care in the US is so expensive.
I think what you are thinking about is the court almost certainly won't order you to go back to work after you quit.
Of course if you have good reasons to stop working, there are good options. If it's stress or similar mental health issues, you can resign and spend your notice period on sick leave. If your employer assaults you or just stops paying, or otherwise grossly neglects their duties as an employer, you're legally entitled to quit immediately.
If you don't respect your notice period they'll remove your salary over the missed time from that final payment. So in that case they'd remove 3-month salary. Sometimes it goes into the negative, and some companies actually send a bill, but there's basically no consequence for ignoring it. Usually companies don't push the matter.
I’m senior enough that I give at least a month, often more, and this made me realize that I’ve been perhaps overly generous.
If you don't work your notice period, you obviously won't be paid for it, and any reference will state that you broke your contract.
(Unless your employer asks that you don't come into work any more. This is called "gardening leave" in Britain: https://en.wikipedia.org/wiki/Garden_leave , and it's normal in some industries and circumstances.)
You usually don't get stuck with duds because there's a probationary period of (max.) 6 months in the beginning in which these protections are not in place.
Also (after the probationary period) the company needs a legal reason to fire you. If they wanna fire you for incompetence oder insubordination they need to have reprimanded you at least once officially, in written form. Only after getting a 2nd reprimand for the SAME THING can they act on it. Meaning if you told the boss to go fuck themselves once and shown up drunk to work once they wouldn't have legal grounds to fire you and if they did you'd drag their ass to labour court and get some kind of compensation.
Yes. Also if you just refuse to do any work.
Turning up drunk to a meeting is enough to start disciplinary procedures -- HR handle this, and work out how to proceed. (On [1], I think they would have been proceeding through the "Misconduct" section, second chance etc.)
Rotating the backup tapes, then carrying said tapes about 1km to the other building to put in the safe, was part of the job, but he was watched while he did it.
Stopping at a bar along the way was not fine.
Being oblivious to the point that a manager was able to take the bag containing the tapes from the bar back to the office was not fine.
Coming back to the office (drunk) and not realizing what had happened was not fine.
This was gross misconduct, and led to immediate dismissal.
https://www.gov.uk/dismiss-staff/dismissals-on-capability-or...
I haven't worked much on regular job agreements (been contracting or "contracting" most of my life), but generally if you want to quit you go to your boss and say "hey, I wanna quit" and you negotiate when, regardless of the default requirements (which only come into play if either side doesn't agree). There's no business in keeping an unproductive employee around for 3 months.
It's certainly a crushing feeling to be chained to a job. Even worse are these stupid 2 weeks or unimaginable 3 months notice. As everyone says- you'll never get notice you're being fired. It'll be quick like a mob hit.
In Europe you will. Or you’ll be sent home with the respective 3 months in salary anyway, but generally you work your 3 months.
To be fair, I’ve never seen a notice period longer than a month.
"We didn't pay for your relocation to X just so that you could change jobs after a few months."
Two weeks notice is a courtesy.
The much maligned PIP is often a hint that you're going to be let go. Rarely is there a "you're being let go today, pack your desk" unless there is cause (the one time I saw it was because someone was using their desktop support credentials to snoop on the CIO's computer - that one they weren't even given the pack your desk) or the company is in trouble (and in which case, things like the WARN period kick in).
Is it the case?
First of all, you get benefits as a full-time employee that contractors don't get, like healthcare, disability insurance, and paid time off. (This all mostly boils down to money of course.)
Probably more importantly, yes, a company can in general simply tell you your services are no longer needed one morning. However, in reality, absent either company downsizing or a documented performance issue, that is unlikely to happen out of the blue. Whereas not renewing a contract at the end of some project is commonplace.
Employee status vs. independent contractors is actually something of a political football at the moment as contractors include both the stereotypical gig economy workers and freelance writers who want to freelance as well as contractors of other sorts.
Looking at peers who struck out on their own, you can make good money as a contractor, but you typically need to be top of your profession and able to command massive fees. And like any business owner, the pressure to always have too much work usually ruins any chance at using the money for longer, more extravagant vacations.
Most large US companies also have a policy of not giving references beyond answering if the person worked there for the time claimed in the position claimed.
There are a number of benefits to full time employment that good engineers appreciate. Healthcare (in the US), known reliable income (this can be a big factor for financing a house mortgage/car), various other benefits (I get massage credit, dental, vision/glasses, acupuncture, physiotherapy). The biggest for me is just avoiding the stress of feast or famine, managing business/taxes, etc..
Handing in your notice to leave is something the employer cannot reject. Depending on circumstances you might try to resign first. Sometimes you may be "asked to resign"[0] and made a nice offer to encourage you to do so, offer will almost always have an NDA attached to it.
[0] this can be preferable to the company as it can be quieter than sacking you
> This guy(gal?) is really required to submit 3 months notice or face some sort of repercussions (financial, I assume)?
Yes. The worst they can do is take you to court for breach of contract though. That can be expensive and embarrassing for both parties so unless they are whiter than white[1] or bitter enough companies often don't bother.
Potentially a big risk to take though. And in an industry that is hardly free from nepotism, you probably don't want the reputation of being someone who walked that way.
Three months is most common in skilled jobs, one month otherwise. In retail work and other low-skilled professions two weeks is not uncommon, though here people often do just walk away as the hassle of chasing them for breach of contract is massively not worth it.
[1] i.e. there is nothing you can say on public record that looks bad for them going forward
> Does this 3 month period apply in reverse (employer can't fire employees without a full 3 month warning)?
Yes. What usually happens if they want to have you gone without notice and don't want to make you an offer to resign "amicably" is that they put you on gardening leave. You are still employed and paid for the notice period but are told to stay home. They can't stop you looking for new work[2] as long as you don't do it "on their time"[3] but you can't start new work without resigning and them accepting that resignation until the end of notice period without the same risks as if you had just walked out without notice yourself[4].
Gardening leave can also happen when you hand in your notice, though if you want to leave and the company wants you to leave then they'll just agree to waive the notice period.
[2] Some contracts have clauses that say otherwise, those clauses are not legally enforcible. Some even have clauses saying you have to actively look for new work, these too are not legally enforceable. [3] Which is impossible for them to prove unless perhaps they have you check-in regularly while you garden. [4] Though they are likely accept the resignation as at that point they can stop paying you for the notice period, unless you are going to work for a competitor, or they otherwise feel like being dicks about it.
I don't know where the OP is located but he should double-check his contract. The first time I heard of the three month period I was reluctant, until I realised it didn't apply to me.
In theory, the business could sue an ex-employee who just left for damages caused by the breach of contract, but in the real world it would be pretty much impossible to prove that the damages were caused by the employee leaving. In most cases it's simply considered a dick move, and you'll end up with a hole in your CV as you won't ever get a decent reference if you just get up and leave.
> Does this 3 month period apply in reverse (employer can't fire employees without a full 3 month warning)?
Usually yes. And unlike the situation above, employees could probably drag the employer to court for it, unless the employee did something illegal or completely irresponsible during working hours.
So, while we don't have 3 months notice in the US, we certainly have similar ways to get stuck.
Penalty from employee side seems to be monthly wage, so in this case 3 month salary. Ofc, seeing how company is probably doing I wouldn't be too worried of them coming after him...
You are, of course, assuming they have a backup and the backup can actually be restored and that the restore contains the required information.
> Our backups are fucked.
So I assume that they have already tried and they didn't work.
No problem they said - we have backups. They had three tapes.
First tape failed.
Second tape failed.
Third tape worked.
In retrospect that was a useful, if rather stressful, lesson.
> a missing $
Bravo!
I've always had anxiety when running a shell script, this is a perfect example of why
Early in my career, we had a storm that caused an electrical surge. In spite of the power protection, it damaged the tape drive on the main server (unknown to us at the time). The entire rest of the system (CPU, RAM, hard drives (RLL!)) were fine.
Listening to the backups being run, I noticed it sounded a little different. Eventually I determined that the tape drive (old QIC, maybe 40MB capacity?) worked, but it would not switch tracks when reading / writing. So the backup ran, but it just overwrite the same track rather than write to subsequent tracks.
A short backup / restore wouldn't have caught the problem, it was necessary to try to read the whole backup to see. This wasn't caught for a while, so all the backups were mostly useless.
> > Every person who knew how the custom DNS system worked has left the company years ago. Our backups are fucked. Our records are wiped from all domain servers out there.
Sure sounds like the data is lost to me.
A company does not just stop existing or being able to function, deleting DNS records didn't empty the bank account or the ability of the accountants to be directed to pay someone... there are lots of options.
Those people are probably employed somewhere else and would need very significant incentives. More than likely their current employers would take a dim view of them freelancing to help a competitor, so: enough to take a year off or even to retire on. Basically the idea of hiring back former employees as consultants is a non-starter.
I would be looking at the Internet Archive and Whois too figure out some basic domains and what the registrars are and go from there but of course depends on what business they are in and what their domain names were doing.
So, it probably is that bad...
But the data should still be there, and it's a matter of reverse-engineering how to go from "ERP DB" to "DNS zone file".
-- https://old.reddit.com/r/sysadmin/comments/ma4mwl/the_absolu...
"So, we have backups on an encrypted drive, which we can't access (authentication is done via our custom ERP software, which is down as well - in hindsight, a very bad idea). And the backups don't work. We can rebuild the DNS servers from scratch, and the records are probably still safely stored somewhere inone of the 3 different databases our ERP software uses, but we don't have an authentication key or the scripts used to pull the DNS domain / record information from the ERP system anymore. The API's are undocumented, and there's only one key that can access those records."
Elsewhere, they say that two years ago, they found they were unable to restore from backups...
So their business actually failed a couple years ago, and they have only noticed this month.
But the fact that after this event the management did not get it fixed (and requested that IT demo it, three times, with no failures) is an existential failure.
Its Password123. Adding capital letters makes it more secure.
Password@123
Highlights:
1 Bookings where made at individual cinemas
2 The network was a packet radio system that blocked :-) only Transaction could be in flight (Leicester square had two channels)
3 Core of system was Perl running on three windows NT machines which manipulated a shonky screen based system (Pacer Cats)
4 Trying to sell tickets online for Phantom menace system crashed at midnight (when tickets went on sale) we made 1 transaction at 6am
% And no credentials so I had to research how to crack NT - I did consider setting up or fleet of a dozen or so of our suns as a cluster to crack NT, but not sure if the internal security would have liked that (aka secret squirrels and the bit that Bruce Shenier worked for
Its complicated by at the time the total lock on ticket booking systems cinema and theatre by a particular company - who wanted 1/4 mill plus to update the system.
It was an interesting project I had to get cover from a v senior manger to investigate how to break into NT.
1) Are you SURE that everything is actually gone? Is there a logfile, a billing trail, sent emails trail from your email provider, something like that? If it was me I'd be looking for a secondary system (like alerts, logfiles, billing, etc.) that had some of this information stored in it as secondary function, and writing some scripts to parse those things and start rebuilding what can be rebuilt.
2) You said you lost all the emails and SMS of your team. Do you have an HR department? Surely they have this information. Your team is likely going to start trying to send emails to each other about this. Can you dump logs off of your mail server to rebuild at least the naming schema? For phone numbers: can you try an old school phone tree style? Start with somebody you know, and ask them for everybody that THEY know and so on.
3) Was there a staging environment that might have had some of the data you're missing on it?
4) Are you really really sure that everything was wiped out? Is it possible that your ERP system is broken (lost its DB connection or something?) and that caused a cascading failure that makes it LOOK like everything is gone?
5) Can you get at the BIND server somehow? Are there logfiles there?
I'd love to know more specifics about what "wiped out" means in this case. It seems unlikely that it's properly "wiped out" unless something malicious happened, and even then I think you could rebuild a lot of this from sortof "secondary" data.
Think like a hacker, but your system is your own target. How can you steal the data you're looking for from places it isn't really "supposed" to be?
Come to think of it, this could be a fun online challenge. I picture a series of VM images, each of which describes the design of the system and the terrible thing that has gone wrong, and you are challenged to get it fixed and/or get the data out of it.
Sure, now you start thinking of backups.
A data recovery shop could attempt to forensically-reconstruct SMS numbers. Look for logs and other places critical deets might be found. Try your best to get things back and make alternatives processes to piece together what you can. A phone carrier or SMS send provider might have a log of phone numbers. The paycheck company may even have the address of everyone. Send them a $0.01 paycheck with a note on them to call in.
I would ride it out for a paycheck because it's not like it could get worse than going under. Just be sure to get paid.
Of course, that may not help with the bigger problem.
"The company that built this ERP solution went bankrupt 6 years ago, and we don't have the source code. It uses PostgreSQL and MySQL, and also has a built-in key-value database for which none of us has any credentials. We don't have the source code for this software (it's built in C and delivered to us as compiled packages..)."
Obviously we all know that source code alone isn't enough, but it is something to start from.
If the ERP solution was not SaaS and installed on site, the purchasing companies IT should have gotten credentials as part of the contract. Again, normal type stuff when buying software from smaller companies that have somewhat higher risk of going out of business.
New executive starts the job. On his desk are three envelopes, left by his predecessor, with a post it note saying "When shit hits the fan, and you don't know what to, open one of the envelopes."
Yadda yadda, exposition, things are normal and exec is successful for a year or two and then things go pear-shaped. Completely out of options, he opens the envelope, finds a letter which says "Blame me." He blames his predecessor for leaving the state of things in such a poor shape that this misfortunate was unavoidable, everything blows over, his success continues.
Blah blah, narration. Another shitstorm. He opens the second envelope, finds a letter which says "fire all the managers." He promises a significant shake-up, says he has the leadership to turn this around, purges the old guard. Things go well.
Hurly-burly story, and so forth. Catastrophe strikes. Having blamed his predecessor and fired everyone and installed his own managers, he has no one else to turn to for casing the blame. He opens the last envelope, hoping to find salvation.
"Write three letters."
Keeping the people who know how the system works is really the last line of defense.
From, the point where such last person left without any compensating mechanism in place, the clock started ticking
House-of-cards core infrastructure is a sign of negligence and OP should have reconsidered working there a long time ago.
If you aren’t running drills then transparency may not be in everybody’s self interest and someone somewhere is covering up for the fact that they only half know what they’re doing.
I feel like this ability to step back and not take a problem more seriously than necessary can be an asset. In this case, it’s pretty bad, but “at least people aren’t in immediate danger of their life.” Think it to yourself, but probably not best to share with others until after the problem is mitigated ;)
"You need to give us 3 months heads up no matter what, but don't worry, we don't have to give you any if you oversleep for a meeting."
Mind you - if the company is going down the tubes they are highly unlikely to enforce any job contracts.
I didn't break the contract anyway out of some combination of feeling like it was breaking a promise and the worry that Cambridge is a small town.
But this is for when both of them can't find a mutual solution. You can always agree to terminate the contract earlier, if both parties agree.
At least that is how it works in Germany.
It goes both ways of course, my employer is also bound by the 3 month period in case they want to fire me.
There are contractor agencies filling the niche of providing employees on short notice (and usually for a short period), but generally having people begin 3 months later is just something everyone expects and factors into their processes.
The average length of time people spend at a single job is also longer than in the US.
Of course, they would rather it was shorter, but hiring is already a multi-months long process in most cases, so a few more months is inconvenient, but not the end of the world. If you need someone now, you hire contractors.
Also, depending on the country, firing/layoffs can be difficult without cause (and often even with cause) (e.g. you can't layoff someone at will, you might need to prove that you have an economic downturn before downsizing). But this varies way more between countries as far as I know.
But I think it would be better to keep it above board. Present it as a solution, kinda. Like, "we both want the same thing here".
[edited to add : not relevant, they're in Finland: https://www.reddit.com/r/sysadmin/comments/ma4mwl/the_absolu... ]
Source: left voluntarily
Or
If your contract is exclusive they can pay you and prevent you from working for anywhere else (gardening leave).
You can’t (at least in most countries) enforce the contract in such a way that you force someone to work.
But there's some additional factors here. If the employer misses salary payments, then they're in breach of that employment contract, and unless they can't fix the situation really soon, that's grounds for immediate termination.
Also in Finland, salaries earned in a normal employment are backed by the state for up to 3 months of missed payments (the state will then try to collect from the employer). So it's relatively safe for an employee to stick with a sinking company. Just get that CV polished and start warming up the network.
Even if you're not responsible for the problem itself, leaving your employer in the ditch, when there's a major outage - well, it just doesn't look good (barring some scenario where your employer is completely unreasonable - like not paying you).
It's kinda like a firefighter that decides to turn off the hose and go home, in the middle of a raging fire. Who in their right mind would want to hire him/her in the future?
But if they get in a situation where management is yelling at them and demanding they work 20 hours a day until this is fixed somehow and they have no idea how to. Management won't bring in consultants, etc. That ship has already sailed.
If your manager does that, you can say "No". They still can't fire you without paying you your 3 month wages.
The firefighter who recognizes when the building is about to collapse and gets out of there is a good firefighter.
It looks like they could probably sort something out, wouldn't be a bad idea to call up any of the old vets who retired to see if they could come down to the office and contract for a week or so.
Ultimately though a person could not show up for any reason at all (e.g. hit by the proverbial bus) and you need to plan for that. Sounds like a company that just flies by the seat of its pants so they probably have that 3-month clause for that reason.
In that particular case, my experience tells me the guy won't get any credit for finding a solution, I would consider sick leave. Burnout, whatever. Doesn't have any monetary consequences in Europe (at least no harsh ones), and Germany covers up to 6 weeks of salary.
Any way, you can forget about getting goods references. Not that would get them anyway, with an employer like that.
The only actual backup is one that has been tested. With the cost of hardware, buy 2 or have an agreement with your vendor to have a machine ready to receive a backup.
I know of one case where power failed, the "24 hour" generators kicked in, power company says "We'll need 12 hours" and the outage happened 6 hours later when they ran out of oil.
When I was there this didn't go too well and they couldn't get one their data centres online again - failover to their other centres did work though. This was ~20 years ago in the finance sector.
I was in a building when someone inadvertently powered off the wrong equipment, which had been running for several years, and several of the power supplies failed to come back up. It was 1+1 redundant though, so we could quickly shuffle packs around to bring it back up without redundancy. Then, jogging through the building and asking if anyone had spares, we found a field tech in the lunchroom who had a pile of stuff in his van. Whole thing was back to 100% in less than an hour, and we let the beancounters sort out the field spares being used for office equipment.
If that same failure had happened during the overnight maintenance window (when volatile work was supposed to be performed), there certainly wouldn't have been the same resources around.
[1] https://www.reddit.com/r/sysadmin/comments/ma4mwl/the_absolu...
Ransomware scenarios come to mind, where the attackers will look for and actively destroy the backups you have if they are remotely accessible with e.g. AD admin credentials, and only trigger the ransomware when the backups are gone. If you have offsite backups in some cloud system, that helps you against natural disasters but not against malicious activity. The need for backups that are not just offsite but also offline is somewhat recent and very, very many companies do not have them.
I doubt any external factors are to blame.
> All domain information was wiped out and records became null [...] Our records are wiped from all domain servers out there [...] We don't even know what domains we own, the listing was hosted in the ERP which is now busted
I find this to pretty common in many setups where the engineers don't focus on simplicity and removing layers of abstraction. If the workforce is young and inexperienced, over-engineering tends to happen everywhere and you end with situations like this.
I have seen worse in my years in the industry, that's for sure.
Setting up a test cluster using real backups can be good. If you have the crosstalk problem sorted out that is.
1. Applications do not need DNS servers, ever. Hosting platforms do. The way I read this is that it's some odd form of split horizon where one DNS server seems to replicate to larger DNS servers, yet the single server is what carries most of the critical information.
2. State should be transferred from where you gather input to where it becomes actionable by an eventually consistent process. ie: If you store your customer CNAMEs in ERP, then something like webhooks to a service that maintains the state in machine readable form should verify, accept/deny, and do something with that change (like update DNS).
3. Customer systems and core/critical systems should be separate. The fact that critical contact systems for employees were not separate from
4. Have a DR plan and artificially enact it at cadence, doing so even in a sterile environment is better than nothing. Successful DR's in production are best. If the first time you live test your DR strategy is when all is lost, then all is most likely lost and at best you have a lot of hours of work ahead of you.
5. The holy grail is replayable audit frameworks. This could mean capturing diffs but it can be as simple as logging the current state of a given $thing and sending it to an off-local system.
They didn't believe me. But I left. After a few months of haggling I went back at $50 an hour on contract. I did this for a few weeks (about 25 hours a week), then started to not want to pay me. So I quit for good. Unless they made an upfront deposit. Which they wouldn't do.
They ended up spending close to $250,000 on equipment and programmers and still couldn't do what I could. They were duplicating and tripling everything. Wasted so much money. Now I'm going to college for my BS in Computer Science with a specialty in Software Engineering. I'm currently running Windows 10 Pro Insiders Edition DEV and VMware with Kali Linux.
There's so many things wrong with the scenario described, that I can't even begin to talk about it.
My experience is that folks never want to talk about backup or DR, as they are expensive, and the idea is that they should never be used.
No one ever has a problem with insurance premiums, though...
If you don't have a backup of your zone files, a good way to quickly pick up the main domains being used would be to turn on logging for your DNS server and output into a log collector. That way you can quickly build queries on which IPs are asking for which FQDNs and start rebuilding a list of your lost zones and can maybe do some guess-work on which IPs those FQDNs were likely to resolve to.
I would expect irregular or extremely infrequent processes that use specific FQDNs will pop up from time to time as errors/failed processes and should prime IT teams on what to look out for.
https://www.reddit.com/r/sysadmin/comments/ma4mwl/the_absolu...
IT reporting to finance/CFO is always the worst decision.
Choice 2: Put your head down and fix this. Your worth (to another company because you really should leave after you fix this) will increase dramatically.
I had an Exchange Server that had issues, and I didn't have enough hardware to try a backup, I kept telling management... it died, and I was able to recover most, but not all of it.
Shortly thereafter they outsourced IT, and I assume the new crew was able to get the resources to actually fix it.
Because, even there might be cheaper solutions, at least I can be sure to have a huge org in place to make sure stuff is up. And to find ways to get stuff back up and running in case something goes south. I value that peace of mind a lot.
It wouldn't have to get into the weeds too much. I mean really basic stuff like finding documents and following them to restore a backup or release a new version or fix a simple bug injected in the system.
Unclear why HN wastes time on this type of 'story' with no attribution and scant details. (I have expertise in this area let's say).
To start this implies that they have not added a customer or made a change in several years by this statement:
"Every person who knew how the custom DNS system worked has left the company years ago."
This also contradicts:
"and we've been trying to replace it for years"
https://www.reddit.com/r/sysadmin/comments/ma4mwl/the_absolu...
> We tried really hard to make management aware and actually succeeded in that, only for management to follow up and say we don't have enough money to fix it, and they understood the possible outcomes.
But knowing Reddit, I'd give this a solid 50/50 chance of being fake.
Not everyone is AWS. There's a lot of obscure software providers nobody cares when they're down.
What is the "E" in ERP without support?
Honestly there is so much WTF-ness to the whole stack I'm starting to suspect this is a fabrication.