Hacker News Reader

I applaud the guts that it would have taken to do something like that. That one hour meeting uncovered so many issues with the company that everyone probably already knew but didn't want to talk about - bikeshedding; long, pointless meetings; incompetence and apathy; etc.

I sometimes dream of doing something like that when there's endless back and forth about the "right" way to do something (in my experience, no one is actually looking for the right way. It's all just politics and laziness which ends up in a subpar compromise).

How did you convince people to sit through the entire hour once they caught on to what you were doing?

"Guts" sometimes means "Something's gotta give or I'm outta here anyway." Other comments here suggest that was likely the case.

Always be leery of emulating such actions. No matter how much detail they give, you will never know the full context and how well your situation compares to theirs.

That's why it's only a dream ;)

Hang on. Let's assume the story is false. Why would you want to discourage anyone from having the courage of their conviction that they could find a better way to do something their bosses say is impossible?

Someone with real conviction isn't going to be talked out of it by a two paragraph comment by a random internet stranger on HN.

You are reading way to much motive and way too much power into a casual observation about how so-called courage is sometimes anything but.

People who seem "brave" may not feel they are at all brave. They may feel fed up, resigned to their doom, disgusted, defeated, like arguing or cooperating with known liars is utterly pointless, like they are screwed anyway and nothing they do will ever fix it.

Sometimes it's nice to get closure and know you weren't imagining things, it really is as bad as you feared and you can stop wasting your time and energy on a situation that simply will never go anyplace acceptable no matter what you do.

So... maybe I misunderstood your comment. Your advice was to be leery of such actions. Are you saying to be leery because you believe those actions arise from a place of despair more often than from trying to remedy a situation or to be successful?

What would you tell someone who is genuinely trying to communicate a message to their boss's boss, about how things are mismanaged or how they could be done better... assuming that person is not lashing out from despair but actually wants to help the company?

This was in response to your prev post.

I'm glad you answered the question directly with your own opinion. (I don't know why people don't do that more often). I think that is a wise and well thought out piece of advice that should be expanded upon. I did want to get past the layers of what you were assuming this person had experienced or what you assumed naive readers would do if they copied it. One thing I've learned - a reason I used to be a bad writer, and a reason why social media doesn't work well - is that I tend to gloss over four or five assumptions about what the reader understands or knows, and jump straight to a conclusion that can come off as weird or out of step. It does help to pry open the thought process that led to a warning or a sweeping statement... often, the thought process is what people are most interested in.

On this issue, I've never worked anywhere I didn't either start with or quickly open a direct line to speak with the owner of a company. I consider it their right to know what's going on - and I never sought advancement or raises, and more often quit, but I do think there's something to be said for sticking your head up and taking the risk of having it cut off, to prove you're right about something. That derives from a different impulse than the need for popular acclaim, and it's usually dangerous.

Does this make you better off besides feeling great for an afternoon?

At best you put yourself into the middle of a political fight fought levels above your head.

Why put your job on the line for no reward? Who cares if their Oracle queries are slow?

Because the op has pride in what he does. Because he cares more about getting it done than the political faffery.

That's a my take on it, anyway

Right, some people want to do a good job and also get recognized for it. Whereas other people want to use politics to prosper from the work and ideas of others. The latter type don't care whose idea it was, who implemented it and did all the work, as long as they the gamers get the credit and rewards for it.

That's just life right? Yes but the point is for the former type of people it is very frustrating to work with the latter type, because not only you have to do a great job, you also have to put in a lot effort to fight the politicians. It is often better to get the hell out of Dodge rather than stick around with the bandits.

Well said. A good piece of work is its own satisfaction, and a broken piece of code is misery for everyone. People who fix broken things just because they're broken and should be fixed are the lifeblood of any organization. Once they're displaced by people who don't know, understand, or care, the organization is basically a zombie.

But software engineering IS about politics as well! We always think that SE is only about tech and being right and smart... the reality is: one needs to be technically right AND play politics in order to achieve anything within a company. lmilcin was probably technically right, but he didn't play well the politics part of the equation. Result? He had to quit, and he company didn't get a better search solution.

> Because he cares more about getting it done than the political faffery.

I agree, 100%, but this sentence is also the problem with OP's approach. Like it or not, being effective as a software engineer isn't just about getting things done. First, you have to get the right thing done (sounds like OP has this covered, though).

But, perhaps even more importantly, if you want to actually have real impact, you need to make sure the things you've created actually get adopted in the organization. And, that's literally the definition of "political faffery."

There's a very real sense in which anything that involves more than 3 people is going to involve politics in some way, but that's not what I'm talking about here. I'm more referring to the definition in The Devil's Dictionary, by Ambrose Bierce, specifically the first part: "A strife of interests masquerading as a contest of principles. The conduct of public affairs for private advantage."

As a mid-level or senior-level engineer, if you've been at the company for a few months, you should have some relationships that you can use to navigate the political landscape. As a staff or principal engineer, it's practically your job to have these relationships. As a junior, you should be looking to your manager, or higher level engineers on your own team to help you, should you get involved in a political mess, but, the best thing to do is to loop in your manager ASAP and avoid politically sensitive issues to the extent that is practical.

But, the bottom line is really just about power and numbers. If you don't have the right person or people on your side, if you try to bypass or undermine those people, or, sometimes, if a lot of people are really against you, it does not matter how technically great your project/service/process/whatever is. It simply will not get adopted, because you threatened entrenched interests.

If you're unlucky, and either don't have sufficient political capital, or you managed to expend most of it during this whole affair, this is where ignoring politics can ultimately lead to a not completely voluntary trip out the door.

I've seen projects that were brilliantly managed politically, and I've seen ones that weren't. Brilliantly managed projects aren't generally going to be the ones you notice, unless you're specifically looking for them and watching the politics of the whole affair. These projects often just look like projects that are succeeding, to the superficial outside glance. Even if there are technical delays, the fact that there is political buy-in for the project means that if you're championing such a project, you won't suffer for having your name attached to it.

Projects that weren't well managed politically can tend to start off well, but turn into shit shows later. Best case scenario for a project that doesn't achieve buy-in is that it gets shelved in favor of another approach. Worst case, the project is radioactive, and that's where it can lead to an involuntary trip out the door.

TL;DR: You can't care about getting things done to the exclusion of politics if you want to have any actual impact within an organization. Politics can easily make or break the success of a project, and it can be hard to tell if you're stepping on a political landmine unless you know the terrain well.

Interesting story. I did something similar for a bespoke monitoring/helpdesk/knowledge base for a big client in energy trading. However it took me an afternoon and I did get permission: I just installed Oracle text on the existing database and replaced the old search by tags with the Oracle text instance.

Typically in these large orgs, writing an app is only one of the first steps; they also need to be maintained, updated, monitored which might require extra budget, which typically nobody wants to pay for because "everything works".

If you ever end up in a similar situation, you could consider the environment as a part of the challenge before you figure out a solution.

Nevertheless, the 1 hour meeting is a hilarious story, although I personally would avoid burning bridges on the way out: it might hit you in the face in another context when you least expect it....

I like this, but somehow I wonder, was this the right thing to do? I would say it wasn't, because both parties lost: lmilcin had to quit, some colleagues felt ashamed (rightly or not), and at the end Elasticsearch wasn't taken into account as solution.

Now, my cynic side of me also wonders: perhaps Elasticsearch wasn't the answer and that's why the proposal was shot down? We'll never know. What I do know is this: software engineering is as much about humans as it is about tech, so even if you know you are (technically) right, don't forget to address the issue from a human perspective (i.e., convince your colleagues that your solution is right, but don't make fun of them by highlighting how one individual (you) can make the whole project "in just 1h").

> both parties lost: lmilcin had to quit

I think lmilcin won when he got to quit :-)

Was the right thing, I think (based on the info available here). There's an opportunity cost in staying and wasting years of ones life at such a place

(Note that GP wrote: "I was immediately shot down that the project will take huge amount of time and effort to complete and so is completely off the table" -- that was the stated reason for refusing it)

Curious to know your motivation to do this instead of more tactful methods? Were you aware ahead of time how it could come off?

I kinda suspected how it will end up.

Honestly, I knew immediately when I joined that I made a huge mistake. My patience for bullshit is very short and I am extra infuriated by people who just don't care to do a good job or help the company (that is paying their salary) and are completely fine crippling their own project for some strange power games.

I knew that if I was going to do what they (management) wanted of me -- which was to sit quiet and do nothing, I would have to leave anyway.

So I decided to give it a shot and it did not work.

But it sure was fun to watch these people trying to gather some response before they had a chance to reach their back channels to solve the problem another way.

This is why being a solo dev kicks ass, and why it's incredibly painful for me to read the stories of people who care about doing a good job getting crushed by these managerial hierarchies.

First of all, a good CTO would probably fire the lot immediately. But secondly, there's this human dynamic - did you ever have to work on a chem project in high school with 3 randomly assigned classmates? Were you the one who made sure nothing exploded, and then found yourself writing the paper and sharing the grade with them? Sure. Because they knew you would pull the weight. And they would go on to be middle managers just as lazy and work-averse as they were as teenagers. They prey on us... the people who think hard enough to do better. I don't know why you'd ever want to join an organization like that, but it's good you found a booming way out of it.

I much agree with you. I'm curious, does this apply to pair-programming?

I tend to pair up with artists where I handle the code and they command the visuals. I find that highly effective and the conversations around UI drive both people to double down on their work. I haven't participated in paired coding but I'd think it would work in a similar way. Really the smaller the team, the more motivated everyone is to pull their weight, but I've even worked on 5-6-person teams where at least one person declared themselves the "marketing" person was living off the fat of the land.

I think the most productive teams have people with partly shared, partly complementing skill sets. The shared skill must be important communication enabler.

For example, in a development team the shared skill sets could be ability to code (so don't bother with Scrum Masters and Product Owners who can't appreciate coding).

But it does not have to be coding, I have also seen teams which were all proficient in financial markets and used it as their language.

The complementing could be:

- one with long beard who has lots of experience and knows how to and how not to do stuff, can debug the hell out of everything and acts as problem solver of last resort,

- one who knows how to make beautiful graphics and UIs,

- one who knows how to manage infrastructure efficiently and reliably and knows how to do all the tiny pesky things to make stuff happen,

- one who is good at understanding the product side and can be liaison between dev team and clients/management (assuming he understands to use "we" rather than "I"),

- one who is not afraid to dirty their hands and do the absolutely most mundane stuff that nobody else wants to touch with 10ft pole,

and so on.

I think this promotes ownership which is always a good way to get them motivated. On the other hand having people with exactly same skill set (for example due to constant hiring profile) tends to result in people who feel to be exchangeable cogs in the machine and less of ownership. At least this is my subjective feeling.

Thanks for sharing, I've been in a similar emotional position, it's a bunch of bad stories to come before the good one, if there is ever a good one. Sounds like there was!

Hope you were able to find a better fit!

I'm working in a company with 20+ years of the legacy code base. We have a custom search engine fully optimized for our purposes. Decades were spent writing this code. It supports all the newest Linux kernel technologies. The codebase is really good. But every year a new inter tries to show that open source solutions are more reliable and faster. They don't try to write code in 1 hr of course. They spend 10 days on average to show proof of concept that is really fast for benchmarking, but it fails when we try to load testing data. I don't know why people think that nobody checked other solutions before them.

The argument wasn't that ES is wrong choice, but that it will take too much effort to integrate and too long to do so.

Group of managers wanted to choose Oracle based technology for this that would further entrench their team and they argumented they can do it faster and cheaper than the development team.

They have already succeeded getting sizeable part of business logic implemented in PL/SQL supposedly to improve performance.

ES would make all their plans obsolete in a hurry.

Obviously, they were not interested in improving the product but in grabbing power, budget and control of the development process. Also they needed to score some points to show they accomplished something during calendar year.

Of course development team was also slow to deal with problems and overworked with other projects. This only fueled actions of database operations.

So my actions angered both managers of database operations as well as development teams.

> Group of managers wanted to choose Oracle based technology for this that would further entrench their team and they argumented they can do it faster and cheaper than the development team.

That's true. Most Open Source solutions don't work on huge workloads. You can read about Hadoop at Facebook or MySQL. They fully rewrote MySQL engine and partially Hadoop. Oracle solutions work well on huge workloads.

This company is not a software developer. The software department is just another team inside the company. The first goal of any non-software company is to reduce costs for IT infrastructure and development. It's not Google or Facebook where software development is a key factor of company capitalization.

> Obviously, they were not interested in improving the product but in grabbing power, budget and control of the development process. Also they needed to score some points to show they accomplished something during calendar year.

Or just do their job: provides a reliable service for storing data. The main goal of the DB/SysAdmin department is to show 99.99% uptime. The software department could not guarantee this uptime. Of course, if all these software developers are good SRE that's another situation. But I'm sure no one has experience of supporting huge systems for years.

> So my actions angered both managers of database operations as well as development teams.

If you really want to change something it will be better to make a virtual team from both departments and discuss all pitfalls from both teams. Maybe you are solving non-existent problem.

> That's true. Most Open Source solutions don't work on huge workloads. You can read about Hadoop at Facebook or MySQL. They fully rewrote MySQL engine and partially Hadoop. Oracle solutions work well on huge workloads.

Most “huge” workloads are not actually Facebook-huge.

Well done. And I'd counter the naysayers by asserting that this is the only right way to go over the top of entrenched laziness. If it cost you the job, so much the better, because you'll never run short of clients desperately seeking someone who can think outside the box the IT department creates with every response saying "sorry, that's just not possible."

I feel like I just read this somewhere... https://news.ycombinator.com/item?id=28564507

Specially with ElasticSearch there is a huge difference between the time needed to set it up and the time needed to index it, making sure you don't crash your production DB while doing so, maintaining the index as the tiniest incorrect index insertion can mess up the entire index, and finally making sure the nodes in production don't run out of memory or other resources without extensive and property monitoring in production this is a perfect recipe for downtime. So I see their point.

See the game theory on this says it wasn't worth it. Should have just laid low and continued to collect the pay checks while plotting your next move.

I'm willing to bet they're doing better in life now than if they had stayed.

I think what I've proposed is a short term micro-optimization where you can eek out a few more checks while you search for a place that supports your creative thinking

Totally agree: why stop the pay cheques coming in because of pride.

It might seem like an easy situation when nobody's demanding you do a good job, on the contrary, as was in this case you did a good job and got fired because of that.

But if you agree to their rules then you must play by their rules, politics, and if you then don't put your best effort into playing politics you may find yourself on the losing side soon.

Not sure I follow the logic, he got fired for trying. Simple solution is don't try, no?

Ultimately you will lose in either situation, but one solution prolongs the paychecks.

The paychecks can of course be prolonged indefinitely, if you agree to play their game with their rules. And depending on your situation of course it is useful to prolong the paychecks. I'm not saying quit, but consider that there is a cost also associated with staying which is that you can not do your best but have to spend time and energy to learn to "play the game by their rules".

I'd rather collect thousands of dollars while I figure out how to crack the google hiring firewall then worry about the rent.

All you have to do is not push an agenda to improve things your way. Sounds easy..

Looking at this another way if that person wanted to get fired for no cause this might be a good way.

Only if you value monetary compensation over the opportunity to deliver such a sweet dish of passive aggressive justice.

Thats a very expensive opportunity, id rather just argue with people online

The true definition of less talking more doing. I salute you!

I LOVE jhipster for rapid prototyping, their tool is the best in class.

You indexed hundreds of millions of records in under an hour? Doubt.

I have obviously not indexed the records. I just made an application that would index new and modified records and indexing existing ones would have to be performed separately.

Also you need to realize it wouldn't be very wise to run your just written code directly on prod.

I could believe it — I’m pretty sure I’ve done it before using Postgres. Benchmarks appear to confirm this possibility —- 100 million towns indexed in about 35 mins using the standard b tree index, and 20 mins for BRIN on a laptop in 2019: https://blog.crunchydata.com/blog/postgresql-brin-indexes-bi...

Elasticsearch is famous for slow indexing. This is basically because it is trading off slow indexing for fast search later.

And data to be indexed was incredibly complex and rich. So every detail of every car you own, family member information, addresses, phone numbers, emails, identification documents, details of your contracts, and so on.

Neat! I’ve never used Elastic but might have a use for it un the near future. Do you recommend any resources?

If you insert in batches, elastic is quite fast. Also it shards and scales horizontally really well without the client having to know that.

My cluster of 6 instances could index ~20k large, complex docs per second while still serving production query traffic.

I like this story.

You shouldn't. It's a terrible story in burning bridges, losing trust with leadership, and basically optimizing for being RIGHT over actually fixing any problems or enacting change.

If you find yourself considering doing anything of the sort, you should just quit. And you might say what's the difference - he ended up quitting anyway. Yeah, but he also did so by ruining his reputation with a bunch of people who you might say you don't care about their opinion, but they'll talk to other people whose opinion you DO Care about, and they won't tell a charitable version of this incident. They'll just say that you weren't a team player, that you don't know how to collaborate with others, and to not hire you.

What he should have done is spent a week writing a document to senior leadership preparing an executive overview explaining the process he would follow, tools to use, and a detailed breakdown of the estimate of building this system (by someone else mind you) - in a timeframe that he understood the leadership would accept as reasonable - but longer than 1 hour.

Then he should have presented the document to them.

> What he should have done is spent a week writing a document to senior leadership

Except this rarely works.

For starters, senior leadership also plays games only on higher level.

Second, they don't like wild cards they can't control. And even more, they don't like people who skip chain of command because they want to control the information that passes up from their organization.

And this particular company is extra political.

In my team, there were only people who worked there for more than 10 years or less than a year.

There is special type of person that feels well working there -- one that is fine doing nothing and playing cutthroat politics game. Everybody else quits right away.

It was my mistake for not learning this before I joined. I have now a protocol to look for that kind of thing.

What's the protocol?

I mean… OP said his stunt didn’t sit well with people and he had to leave soon after. So if you’re looking for an effective strategy, this ain’t it either.

Either it works or it doesn't. But the role of architect is working with leadership on their level, not being a super hyped-up developer. But cool story nonetheless.

Eh — that leadership is in title and name only.

Anyone who does this is not a dumbshit, those who get in their way probably are.

OP: would love to make your acquaintance, email is checked in profile.

Nope, gotta disagree.

Sure, the ppl you embarrass might talk. But they won't say much, and likely won't have a lot of credibility.

At least this shows a grounding in reality, has the only chance of success (a white paper /proposal was already shown to go nowhere), and supports the (likely few) other people who care about reality & getting things done - ir at least shows them the consequences so they can adjust their own path.

If you are good, in this economy, you'll be in demand. So don't play the coward's game - be bold, speak (& show) truth to power.

I’ll never understand this mentality.

On the contrary, it's a great story where the trickster creates wonderful things but is punished. Prometheus. Maui. Coyote. Rabbit. lmilcin.

Let people do as they please, not everybody has to live with that “team player” mentality you describe.

They could have also just built the PoC app and presented it after the fact.

In my experience, managers aren't usually malicious, they're just risk-averse. They've seen good pitches turn into failed projects. A working PoC counts for a whole lot more than just a plan.

Hierarchies dont like to be turned upside down on the scale of competence. Managers hate when their managing cant be attributed to a success. The lesson here is if you do something out of band, if you want it to work, find a way to fake the participation of the powers at be.

This sucks, but it is the only option when you only improve one facet of something. If on the otherhand you could do the whole business then it is time for you to found.