The shittiest project I ever worked on | Hacker News Reader

The shittiest project I ever worked on

blog.plover.com

501 pointsm0nastic181 comments

Although it's a fun read, it's a classic example of diving into coding without giving a project it's due diligence. There's nothing that's derailed my projects more consistently when I started out as not understanding the user's needs. They'll never tell you what they want, only what they don't want after you deliver something.

I think that's the key difference to an experienced dev/BA. One who can actually sit with the stakeholders and build the system on paper and go through each of the problems as the diagrams connect. What you end up with is the stated requirements (tip) and the unstated assumptions (iceberg).

These types of projects are easily spotted as they're often called "quick" or "easy", which in layman's means no one's really thought about it yet.

Oh yes the "easy" project... we just need...

The real issue is sales...you make the sale because you have to. Then you stick the project on some poor PM/Ba/Dev who will fight with them forever over this stuff.

Maybe I'm just bitter about the situation that I continually get stuck in.

Boy, do I feel with you. But look on the bright side of it, we are not alone, are we? ;-)

Yea there are times when I think gee "I'm so happy to be above the ground and vertical." Then I get an irritating email and well that thought flees.

Amen, brother :) !

"One who can actually sit with the stakeholders and build the system on paper and go through each of the problems as the diagrams connect."

Reminds me of an old saying: "If you don't know how to do it, you don't know how to do it on a computer."

heh true, but what about if you were going to use a neural network to do some sort of pattern recognition? You dont know how its done, but the computer does! :D

now you have two problems... (because the NN will probably fail at the most inopportune moment and you have no idea to fix either the NN or your actual project)

Then well and good, but alas situations where that will work and deliver business value are rare enough you can easily spend a lifetime as a professional programmer without encountering one.

That's not actually how artificial neural networks in pattern recognition tasks really work.

Once you learn and understand the learning algorithms, you will experience this Wizard of Oz moment, you see how the trick's done and it's not quite as impressive as you hoped. This is confirmed by the fact that it exactly matches the limitations of a typical neural net you've established experimentally.

The tasks that are within these limitations, they perform quite well at, but it's not that hard at all to understand how they are performed. In most cases you can basically run the algorithmic equivalent of a trace debugger on the system. In as far as you can compare it with a brain (but don't do that, it's the wrong analogy--compare it with statistical regression instead), it's one you can take apart entirely, reassemble, run partial components of, tweak, and understand exactly how it works to the instruction level. Plus you can do these things automatically for the most common cases.

In short, there is nothing "magic" about neural networks, or any pattern recognition algorithms.

And to get back on the subject, if you intend to work with machine learning algorithms in a professional/consulting/project setting, probably best to first check the scientific literature to see if anything like this has been done, and what the accuracy/false positives/negatives/ROC curve is that you can expect. Don't expect to beat them, either (you may, but planning to do so by any significant amount is asking for trouble--especially if it turns out it just can't be done). Then, I think it would be clever to first build a "fake" system/interface, without the ML component, limited to just the labeled training examples (if you can't acquire these, give up now[0]), see if the system is actually useful from a software engineering, application and usability perspective, before you actually begin (the fun part) tweaking an ML algorithm, only to later turn out, even if it did its job perfectly, it wouldn't be all that useful in the operating environment in the first place.

This is different if you work for a corporation with a huge research budget, of course. Like Microsoft or Google. But even then, I think that for their ML-enabled end products, similar reasoning like I sketched above is used. Of course the same goes for every type of project that depends on an unknown to-be-researched technology.

[0] yeah yeah unsupervised learning. but if you know what you're doing that well, why are you listening to my advice? :)

I have an upcoming project where we'll be implementing a pattern recognition algorithm - think smart flash cards. I've read up some on timed-repetition and a small amount on how neural networks work.

With your experiences, do you have any recommended reading that would help me with the neural network learning curve?

I think this is wrong. Take for instance the task of classifying images. You can train a RBN with backprop (after contrastive divergence alg) to correctly classify images. In the process it has automagically determined properties of the image which allow it to perform the classification. These properties are combinations of pixel elements. So it has in effect determined how to solve a problem without your input. In a similar way, solving a set of simultaneous equations using any of a huge array of mechanical mathematical techniques is also solving a problem which you do not personally know how to solve. you could even consider using a Library of code as solving a problem you do not know how to solve...

"In the process it has automagically determined properties of the image which allow it to perform the classification."

(Emphasis mine.) But that's the point; it may be "auto", but if you understand how NNs work it's not magic. It's not even all that hard to understand (considered broadly), and once you understand how they work it is, for instance, easy to construct cases they fall flat on....

"So it has in effect determined how to solve a problem without your input."

... and it's less "auto" than you think. It figured out how to solve a problem based on your input of sample cases. And there's a certain amount of art involved in selecting and herding your sample cases, so regrettably you can't discard this part, either. Just flinging everything you've got at the NN is not going to produce good results.

If you don't understand NNs, you are unlikely to get good results by just flinging data at them; if you do get good results, it's probably because you have a problem that could have equally well been solved by even simpler techniques. They're really not magic.

Understanding NNs is not the same as understanding "how they do it". You may have a good understanding of how the algorithms work, but after training your moderately sized network to do what it does, it's not very easy to know what _exactly_ it does to solve your problem.

I think that's what chii was trying to point out; not to say that devising them and training them is incomprehensible.

It's classic emergent behavior. While you may understand how the algorithm works, and even be able to step through and see how each neuron affects the whole, that doesn't mean you know why the answer is correctly achieved through all of them combined.

The classic example is facial recognition. Training a neural network for facial recognition will result in lots of neurons contributing a very small part of the whole, and only when all (or most) are involved is the answer correct.

To most people, this (emergent behavior) is "magic".

But it's not emergence. Using Gaussian elimination to solve a gigantic system of equations isn't emergence either, even if the numbers are a bit too many to carry in your head at once. (as a matter of fact, solving systems of linear equations is part of the RBM training algo)

And even then, if it was emergence, doesn't automatically imply we don't understand it or that it's "magic". The famous Boids flocking simulation is a classic example of emergence. It's not very mysterious. Yes large-scale behaviour emerges from simple rules, this is amazing that it happens, but it doesn't hold up a barrier for us to understand, analyze and model this large-scale behaviour. Crystallisation is emergence, again we model it with a bunch of very hard combinatorial math.

But in this case, neural networks are not an example of emergence. They are really built in a fairly straight-forward manner from components that we understand, and the whole performs as the sum of the components, like gears in a big machine.

Test-Driven Sudoku? :)

Paraphrasing: "You don't know how to do something until you can write a program to do it"

The tricky part is often convincing a client that pre-planning is a necessary part if the process and, though it is billable time, will ultimately save money. Many client don't appreciate that programming is not just writing code.

Also you can wind up meeting and planning yourself right out of a gig sometimes - which is fine if the gig wasn't truly necessary. But can be bad for your business if you spend hundreds of hours doing this for no compensation. This is a real situation that happens - you get a client that wants a big project and after many meeting and calls you determine that a Wordpress install will suit them perfectly. Their network guy does the install and for all the money you saved them your reward is that you don't get the gig!

If you're billing time & materials (which might be a good idea for a contract of indefinite scope), you'd still get paid for the time spent doing due diligence. You might even be able to sell a time & materials contract on the basis that it gives you an incentive to point out simpler solutions that require less maintenance even if they involve using competing off-the-shelf products.

Exactly. Good clients will understand "hey, this guy just saved us $150k". Bad clients will think "I just paid a programmer to do nothing!"

Learning to recognize and get rid of those bad clients (or convert them into good clients) is a skill that can take a while to learn.

Along the lines of: "I paid my attorney $10k and I didn't even get to court!"

The trick then is to get clients who know they just want to "have it taken care of", and have much higher billing suppliers (IP lawyers, pharma experts, etc). In that case, they don't really think twice about paying for process in advance of coding. Pharma is a good example.

Being a contract resource for a software company, where management tends to think everything related to software/IT is "easy", is a guaranteed shitty project. They'll second guess and micromanage everything you do. And then when they burn through their OPM, they'll screw you.

I ran a small software consulting business for about 5 years and I agree 100% with you. Business was stressful for the first few years until I narrowed down my client base to only those in the first category. I never did land a huge, corporate client, but I did have several smaller sized companies with a never-ending list of things to do and around X number of dollars budgeted each month.

> Although it's a fun read, it's a classic example of diving into coding without giving a project it's due diligence.

If everybody was so picky the size of the IT industry would diminish by 90%.

and the success rate? and value delivered?

How dare you bring technical excellence as an argument? Don't you know this is about money, not performance? Jeez, get an MBA.

that took a minute to process !

cheers

Kind of agree with you and OP, so let me add to the discussion with the caveat that YMMV etc. Some of these points may overlap with comments by fellow HNers in these threads.

In any organization as a prospective customer, there are three forces at work that matter to a consultant: users, decision makers and finance folks. Ideally, the decision makers appoint a dedicated person (or team) who mediates users and finance guys to create a coherent road map for the system to be built. What tends to happen in practice is a combination of chance, human errors and conflicts in priorities.

To survive, we must understand ourselves as consultants, the terrain and the opposing forces (anything that prevents money getting deposited in the bank).

Consulting is a business => the individual consultant must shoulder quite a lot on himself:

* Advertising himself

* Finding work

* Meeting prospective customers

* Writing proposals

* Closing the deal

* Writing a contract

* Defining the scope of work

* Nailing what is to be done and what is to be avoided

* Constructing components of software

* (In worst cases) baby-sitting customer's devs / ops

* Testing his modules

* Getting money in his account

So, when we talk with customers, there is inherent pressure in closing the deal versus precisely defining everything. Icing on the cake: back of the mind, you might worry about milkman's bills and the birthday gifts to be brought for your kid's friends. Always have a good runway: six months seems optimum. Technically, even three months should suffice, but it is insufficient for the edge case when these three things hold true simultaneously:

* You've to walk away from a customer in the middle of a project, and

* Dry sales pipeline, and

* Unexpected crisis at the personal front: very close friend in need of money for surgery etc.

God be with you.

Now, back to the main track. The key is to first understand the priorities of stakeholders. The contradictions are easier to grasp once we put ourselves in the individual's shoes:

* [Convenience] The users want everything including the kitchen sink,

* [Accountability for the risk] They want it yesterday but won't commit to it on paper since you're in early sales stage,

* [Cost] The finance folks want stability and the most economical price,

* [Availability] The decision maker prefers to spend the least of her time to be spent on this activity, except when it's her pet project.

These roles may be played by different people or the same person.

Again, we don't have to agree, but acknowledge that things are meant to be imperfect. Most people are actually decent and rational. Now, somehow, the consultant must figure out a good path. To begin, he should have a good handle on:

* The industry in which the customer operates, e.g. BFSI, Energy, online marketplaces, food, fashion etc.

* The nature of the market: oligopoly, crowded, emerging et al.

* The level of Govt. regulation: HIPAA? PCI?

* The customer's position: among top tier, mid-size, small firms.

* Customer's customers: helps to track the wind.

* Standard pain areas in the market.

* How customer's competition and partners survive the pain, specifically the differences among the customer's approach and the competitor's ones.

* The financial health of the customer: if possible, try to talk with their existing suppliers in other arenas (e.g. the guy who supplies printer cartridges, the cleaning lady etc.).

Based on that, assuming we've landed a hot lead, say that we want to convert it to money. There will be a lot of conversations over the phone, e-mail, chats, face to face; what is to be left out of those conversations is really context sensitive. The important parts are:

* What problem we're going to solve for all stakeholders?

* How we're going to solve it?

* Communicating our approach and progress on a timely basis.

* Improving Vulcan skills: like money, expression of anger is a great servant but a terrible master.

I leave out the details on the development process, since it depends on the individual.

Here are a few things which worked for me marvellously in certain contexts and doomed in others:

* Setting expectations by explaining the cone of uncertainty in the estimation process. Always be prepared for some customers, who deliberately mislead you that they don't understand the mumbo-jumbo, projecting ignorance as a shield so that they can ruin you on payments. Mostly you should refuse such customers, unless you're in an extreme crisis on multiple fronts.

* Writing use cases for the modules / systems I was supposed to work. Most customers say that they don't want any time spent on "documentation," to which I reply that writing a document is not "typing" but "nailing the thoughts to deliver higher quality work."

* Creating mock ups using sketching tools (even Excel/LibreOffice can be good enough, if one is pressed too much for time) in the absence of UX designers on the team.

* Writing test cases before writing my code. Helps to ease the integration.

* Optimizing lazily: progressively refining the artefact (document / mock up / test / code).

* Knowing the expected value of each task: (amount of money * satisfaction * improvement in reputation) / (e-mail tennis * stakeholder's understanding of requirements * payment latency * time spent)

Finally, reducing it to just one sentence: be nice, believe in yourself and humanity, be multi-dimensional and please heed Guy Kawasaki's advice about reading voraciously and omnivorously.

Edit: formatting.

Commenting to save this for myself (fellow consultant) -- thanks for the detailed list. :)

You're the most welcome, :)

I'm curious about any differences in conulsting approach to data science compared to software products: estimation strategies, what works etc.

Ditto to the first part, commenting to save or myself.

Awesome list - deserves a nice little blog post of its own that I can bookmark.

Thanks for the kind words.

Great list! Mostly works for projects in general, too! Agree that there needs to be a blog post of its own for that.

Really great advice, would you mind if I put it up on pastie (obviously with attribution to you) so people can bookmark it?

Thank you and everyone for the words of encouragement. I've posted it on Medium: https://medium.com/p/23a07cb0d5ea

What stops you from bookmarking it on hn?

https://news.ycombinator.com/item?id=6443135

I rarely comment here, but your excellent post deserves to be recognized. I'm someone who has thought a good deal about starting a consulting company but never managed to convince myself the risk/reward numbers would work for my life situation. Your richly detailed and honest perspective is much appreciated. Thank you.

Your words widened the smile on my face. I agree with you about the risk/reward numbers.

Should you be comfortable, I'll be very glad to keep in touch with you. My e-mail is on the GitHub profile.

Awesome post! One thing I find is really important is to charge 'properly', which at first seems like exurberant fees. My friends think I'm rolling in cash because of my 'rate' but when you outline all of this stuff up here, you can see where it all goes. I don't do much better than when I worked full time, but I have the potential to work harder and make more money. But the extra super hard work doesn't' seem worth losing the life balance (and well, the nice parts of contracting, i.e. TWO WEEK HOLIDAY, even though it really isn't).

Happy to learn that the post resonated with you. I appreciate it.

Once a VP at a customer told me that a diamond is just a piece of coal that shines under pressure.

The quest to become a jewel in the market can be a powerful initial thrust to work as a consultant. Combined with the opportunity to work in a diverse set of technologies and business domains turns to be a pretty compelling force.

Re: compensation, tptacek and patio11 have shared some great thoughts in the past.

The flip side of this is that when stakeholders/customers don't know what they really want or need, you can labor in the "requirements phase" for months. My team spent 900 hours last year just getting a functional spec and high level SRS together for a project that hasn't happened yet. Why hasn't it happened? Because at the end of the 900 hours, we presented our estimate (about 1100hrs) to the customer and he balked at the cost & time. Then my boss bitched at us for spending so much time on requirements in the first place.

<8 months pass>

Now we're looking at commercial solutions in an attempt to not have to build this thing and deal with this department anymore.

I agree with you 100%, but the number of people astute and experienced enough to quickly and deeply understand the problem to be solved (and the quirks of the customer) are few and far between. In my experience, perhaps 5-10% of any large IT organization fall into this category -- everyone knows who the superstars are and their time is always at a premium. Drives me bonkers.

Try to keep your spirit up despite all this, especially your team. Things like that were part of the reason why I quit my last job.

Completely true is the thing with the superstars. I knew one myself, a genius in warehouse management systems. If I had him on a project, it went smoothly as anything. if I didn't I stayed in development hell for months. But seeing this guys workload, he wasn't off any better.

But in retrospect, the general direction the organisation took at my last employer wasn't that bad. Execution lacked, so.

If you billed those 900 hours for doing RFP for the customer, it doesn't sound too bad.

We couldn't bill them -- it was another SG&A department and we (IT) don't bill them for our work.

How can you spend 900 hours without someone taking responsibility for it? After 4, you say 'give me a project nr or talk to my boss.' I'd be pissed too if my staff spend 900 hours on a dud without approval or accountability.

> Although it's a fun read, it's a classic example of diving into coding without giving a project it's due diligence.

The formation of this sentence seems odd to me. Why can't we say it's a fun read because it's a classic example etc.?

For those who don't know about the author of this post. His name is Mark Jason Dominus, the author of one the most awesome programming book that one can ever read.

Higher Order Perl, is available for free download. If you read it you will see some amazing insights into programming techniques most people would have never heard of encountered in MegaCorp jobs. You will also grow a great appreciation for Perl in general and understand how it can be an amazing language of choice for a wide variety of problems.

I know it's probably totally unnecessary but still, here's a link to the free download page of the book [1].

Together with Modern Perl (also free for download [2]) this book completely changed my view of Perl. It helped me see through the misconceptions built (most of the times unintentionally) around this language and made me even switch from Python to Perl.

If you're interested, spend a few minutes browsing through one of these books. I think you'll like what you see.

[1] http://hop.perl.plover.com/book/#PDF

[2] http://onyxneon.com/books/modern_perl/

>Prudential didn't need an affiliate locator application. They needed a static HTML page that told people to call the number.

They needed a competitor.

> In 1995 I quit my regular job as senior web engineer

You had the job title "senior web engineer" when the web was 4 years old. That's pretty cool.

US corporations tend to do that. Instead of a big raise, small raise and "senior". That can even be negotiated in hiring, so you may join as senior for a position just created in the market.

Not only in the US, unfortunately. I've even seen company creating other kind of superlative, when lead, senior, manager or director was taken. What about "International Senior Art Director" (even if your job is basic graphic designer in a 25 persons company) ?

I had a "Senior Software Engineer" title after only about 2-3 years of real-world programming experience, at age 23 or so. I had shipped a couple games as lead by then, so it wasn't entirely unearned.

But I'm sure software had existed a long time before you started. The GP's point was that being a "Senior web engineer," when the web itself has only been around a few years is impressive.

I'd say that it is trivial - if a new topic starts and is rapidly growing, then after 3-4 years almost anybody who started working on it near the beginning would be better than 99% of everyone else (simple 99% of others are newbies in the are), and his/her tasks would include a say in 'how it should be done' and mentoring those others - so, Senior xxx.

I was senior consultant, but for a two man organisation. The other man was the owner. Later on he told me I wasn't up to standards to be "senior" consultant.

He then became the sole member of the business :-)

I started out at a non-technical startup that was pushing a technical tool as its main product. Out of a team of three developers, I became the Lead Developer, despite having barely a year of experience, simply because I was the better programmer and was willing to call management out when they tried to make technical decisions. I was offered a choice of "Senior Developer" and "Lead Developer" and went with the lead role because I didn't feel very senior.

In reality, I was barely a junior dev, and being in charge of a site that's getting millions of views was absolutely terrifying. In some ways, it was one of the best training exercises I could have. It forced me to learn things, and to learn them well. It was also total bullshit. My pay wasn't that great, and although I was working in a startup that was slowly becoming technical, had absolute control over technical decisions, and was able to turn a company that struggled with FTPing to a server into one with automated builds, regular code reviews using a new source control system (Git), and much more, it still highlighted more than anything that I really didn't have the slightest clue what I was doing.

It really couldn't have gone better for me. The company was acquired, and I landed a job at a great agency, and was able to work on enough projects to actually know what I was doing, and to see what good code looks like. I'm a mid-level developer now, with much better pay, but instead of looking back over old code and laughing, I tend to look back over the time where I led the technical side of a startup and laugh.

Congrats! Sounds like quite a ride. :)

Can you believe that when I started, barely anyone used source control? That's what jumped out at me about your story, oddly enough. When I started the Web hadn't been invented, and the only version control packages were CVS and commercial options. CVS sucks, in case you're wondering. So did most of the commercial options.

In my own case I'd been programming and writing games for fun since I was 14, so by the time I took my first job at 22, I'd already been coding pretty regularly for 8 years. Some of that was actually for-pay, though only a few months worth. So between the experience I did have and my oversized ego, I pretty much expected the "Senior" label.

Sorry if it reduces self-esteem, but we (software developers) are not engineers.

I've been brainwashed for many years by my brother, who is a Phd civil engineer, that professions that hijack the term engineer are kidding themselves.

I know it sounds cool, and my current title is 'Senior Software Engineer', but it is all a scam.

I'm not convinced. Care to make a case?

Well it is really just semantics, so there is no right or wrong, and anyone can define what particular words mean to them.

But there are a couple of things that make me personally think software engineering is a misnomer.

Firstly, traditional engineering disciplines are generally pretty black and white. It is applied maths and science. If you build a bridge, you can use physics and maths to pretty much prove that it will handle a particular weight/load and so forth. Software development is rarely like that, and even if you are writing software for an engineer and therefore you could argue that you can prove the calculations the system is generating, it is really the problem domain that is engineering but the way the software is put together is not so easy to prove.

Secondly, I think the term actually does us a disservice. Software development requires intuition, craftsmanship, pragmatism, determination, adaptability, artistry. It also often requires understanding human nature, empathy for users, being able to learn and understand endless problem domains, an eye for the aesthetic. All of these things aren't generally required for traditional engineering disciplines.

The distinctions you are making are mere tendencies. Automotive engineering, for instance, requires everything on your list of software development traits—in spades.

I would say that the essence of engineering is a devotion to verifying that the design works like you think it works. Software development often lacks this devotion, but when it is there it is entirely proper to call it software engineering.

i suppose there aren't very many software engineering fields (and most of them really are just computerized extensions of physical engineering).

That's why i call it software development, and i m a software developer.

There are plenty of different software engineering fields, it's just that it's usually just called "domain knowledge" and nobody outside the field makes the distinction.

One of my internships was at a company that did avionics software. Avionics is about as different from Web 2.0 startups as you can possibly get and stay in the same industry. There are very rigorous traceability metrics that pretty much dictate exactly what can go in the software. Every line of code has to be traced back to a particular paragraph in the design doc, which has to be traced back to a particular numbered requirement. Oftentimes garbage collection or even heap allocation is banned to avoid unbounded pauses in execution time. So the behavior of the code is fully specified under all conditions, and often formally verified. This was the one place I've ever worked where I'd hear "So we should write it in Ada then?"

I've also worked in the financial industry. Little known fact about the financial industry: it's not one industry but many. The code that you'd use to build a quant trading model is very different from the code you'd use to build a retail banking portal. The former actually tends to look a lot like academic research code (i.e. a software engineer would consider it shitty, but it has impressive math and gets the job done).

I work in consumer web now, but at Google. We actually do care about latency, and CPU load, and memory consumption, and developer productivity. And so it actually ends up being a lot like real engineering, where you perform experiments to measure the performance of the building blocks you're using, carefully consider goals and trade-offs, make estimates and back-of-the-envelope feasibility studies before implementing, and then rigorously test that solution.

This is in complete contrast to when I had my own casual gaming startup, where neither performance nor reliability mattered all that much, the only thing that did was the user interface.

I think the big difference is that the civil engineer doesn't actually lay the cement himself, but the software engineer does. This has a lot of ramifications about how each approaches the problems and sees themselves in context.

Some would argue# that the compiler lays the cement.

#have argued, actually

Its rather a case of each engineering profession is a regulated industry with an accreditation body, that certifies you as an engineer. Mechanical, electrical, electronic, chemical, industrial, civil, all have standards organisations, usually different ones in each country.

See IEEE for instance...

Also the engineer term is a protected designation in many countries, just as doctor and laywer is...

I am a programmer, but my diploma says Electrical Engineer. I actually majored in EE because I thought it would help me be a better programmer (and I think it did, but that's a story for another time). I actually did EE work, too, and continue to do so sporadically, but I only tackle tasks where I can't screw up any actual EE's work. I know my fundamentals and have some design experience, but since I spent most of my career doing (admittedly, low-level) programming, I'm no match for someone who spent most of their career doing circuit design.

The whole frame of mind in which I do EE is incredibly different from the one in which I do programming, to the extent that, for much of my early career, there was simply no way I could do both of them efficiently during the same day. The time of context switching was, literally, a good night's sleep. There are marked differences; the ones that spring up immediately are:

1. I spend a lot of time doing actual computation, optimizing my design on paper and attempting to predict interactions that occur due to reality not being quite like its ideal model. Some of the younger engineers frown upon this (mostly because they suck at math and think theory is for bookworms), but it's very productive once you manage to do it right.

2. Even when specifications are complete and respected to the letter, there are still differences between what you specify and what gets built. There are technological variations you must account for.

3. Better yet, you always need to bear in mind the limitations of the manufacturing process for what you design. When I think about how I'll write something, my own ability to code is literally the only limit I have to deal with. There are, of course, hard limits due to the constraints of the platform you work on (e.g. there is some hard limit to how much code you can fit in 16K of flash), but these are of a very different nature compare to manufacturing constraints. Designs must account for the limits of your technological process (e.g. you may not be able to mount some types of components on a PCB), and they must also ensure manufacturing is feasible and scalable (something called DFM -- Design For Manufacturing -- which I have screwed up a couple of times, due to my inexperience).

Overall, there's a huge gap between how much intuition and craftmanship I put in each activity.

I'm not saying this disqualifies programming as an engineering discipline; such differences may also exist between other engineering disciplines I have no idea about.

I, for one, tend to frown upon it being called "software engineering" though, mostly in virtue of these differences. I don't consider the term "software development" demeaning -- in fact, I prefer it, and I always present myself as a programmer and my job title as "software developer", even though I'm legally allowed to call myself an engineer.

The degree of responsibility is vastly different. Engineers are licensed, and can (and often are) sued for professional negligence. Failing to do your job adequately means you'll be stripped of your license and prevented from practicing engineering.

No, only certain fields of engineering are licensed and regulated. Those include Structural, Chemical, Mechanical, etc.

Others are less rigorous, but can still are the practice of engineering. Software Development is one of those.

This is the best answer I've seen (speaking as a control systems engineer who develops industrial software for a living), and a big part of the reason I'm not keen on software developers calling themselves engineers. When your software crashes and causes $XMM worth of damages, injury or death, and you're held personally liable for that in a court of law due to your status as an engineer, then I'd be willing to accept the Software Engineer title.

The title implies a certain amount of rigour and due diligence being applied at all stages of a design, as well as compliance with all relevant standards and regulations. This is audited by certification bodies and explicitly stated by your signature on any document or code you sign off on.

Personally I think this is somewhere that software development could head in the future, but there seems to be too much disagreement on coding best practices to standardise them, and anyway I doubt most people would be willing to pay the cost associated with this, for the majority of software.

It's kinda ironic, but my girlfriend's business school classmates (most of whom are electrical, mechanical, or chemical engineers) seem to think that software engineers are at the top of the engineering totem pole. Something about being the only engineering subdiscipline that's paid what they're worth. If it's a scam, it seems to be working.

As a former chemical engineer, I agree with your gf's classmates. In late 90s, I was working on an advanced chemical manufacturing process control project with a team of software developers. I found out they were getting paid twice as much as me and they were half as productive as me (they knew nothing about manufacturing, chemicals and process control) and I was twice as stressed as them. The best decision I made was to dump chemical industry for technology industry soon after ... Never regretted.

lol, so what were those software engineers being paid for!?

"technology industry"?

Or you could have moved to the petrochemical industry, where you would have been paid more than any software engineer anywhere...

Is that really true? This article [1] lists a top 10% petrochemical engineer's salary as $146,000. A fairly skilled senior software engineer at, say, Google can easily pull in $250-300K in total comp, and a programmer on Wall Street or independent consultant with good domain knowledge in wealthy industries like banking can pull in $300-500K/year. That's not even counting talent acquisitions, which can go for $1M/engineer/year.

[1] http://work.chron.com/salary-petrochemical-engineer-4334.htm...

yeah but those wallst and google jobs are not top 10%.. probably closer to top .5%

@Nostrademons

Ok clearly I am out of touch then. But seriously, how many SE really gets paid 250k+, even at Google? Quants is in my mind the same as the petrochemical industry tough, rediculous salaries so that one I can believe.

Ones with about ~10 years good industry experience, or with several years of good Google experience+equity.

Actually, it was petrochemical company, one of the largest. And no, petrochemical industry doesn't pay much more than software profession.

Comparing with my friends who are still doing ChE jobs, If I was still in petrochemical, I most probably be getting paid little less than what I am being paid now, but working much more harder, longer and more stressed.

Our salaries aren't a scam, per se. Just a reflection of a market equilibrium condition driven by classic rent-seeking dynamics:

  https://en.wikipedia.org/wiki/Rent-seeking

which don't apply as much to lower-demand professional categories.

Which isn't to say we aren't worth it; I'd like to think we are. Just that we often delude ourselves into thinking that our salary differential is linearly correlated with the economic "value" we generate. It isn't.

Or rather: the "value" of an engineer is a lot like the value of an apartment in a high-rent neighborhood: the price holds in the market because, well, you have to have it, and at (almost) any cost in order to do all the other stuff you need to do, like get around the city and be close to your job.

You also have to have access to food and safe drinking water -- even much more than you need a convenient or semi-stylish apartment. The priority for these isn't just a bit higher: it's categorically higher. Yet their base cost (i.e. the cost to obtain them at nominally satisfactory level) is an order of magnitude less than the cost of your rent.

The distinguishing factor between the two classes of goods is, of course, scarcity. That is why your rent is damn high.[1] And that's why an average software developer can earn twice as much as a decent chemical engineer, and an ungodly multiple of what a great public school teacher makes.

[1] Along with the fact that high-demand apartments -- like high-demand engineers -- are also, arguably, Veblen goods in some markets:

   https://en.wikipedia.org/wiki/Veblen_good

Whilst all people who have the title "Software Engineer" are not engineers, it does not mean there are no Software Engineers.

As a software guy who used to work in a factory alongside "actual" mechanical and E&E engineers, I daresay our goals and responsibilities are so similar, that we have every right to call ourselves engineers.

I like the way it works here in Québec. To hold the title of Engineer one has to have an actual engineering degree.

For example I am getting a "Software Engineering" degree. It has all the same math and science classes as other engineering majors.

If I ever want to use my title as engineer I will be held to the same standards as other engineers.

https://en.wikipedia.org/wiki/Software_engineering_professio...

Alas, here in Texas, it's illegal[1] to call yourself a Software Engineer without a PE certification. The decision came out in 1998/99, and you should've seen everyone scramble to change their business cards.

You can't even design an embedded electronics device for a 3rd party (for a company you work for, is ok - industrial exemption) or write software for the same without a PE.

[1] http://sce.uhcl.edu/helm/SWEBOK_IEEE/papers/10%20reprint%205...

I use the title that will make me look the best for the roles I am after. I currently have the title of Software Architect, which is probably about as pompous a title as it gets. To my clients, however, it conveys that I focus on large interconnected systems where the need is less about code and more about system design, and that equals more $ and more projects that fit my skill set. I could care less what a certified architect actually does. Nobody is asking me to design their house, and titles in the software industry are not yet standardized. It is not a "scam" to optimize your title for fit and profit (provided it is legal and is not completely misrepresenting what you do).

In Australia, my five year Bachelor of Software Engineering with First Class Honors was the first Software Engineering degree to be accredited by the Australian Institute of Engineers, of which I am a member.

So, yes, I am an Engineer.

It would only be a scam if anybody but engineers thought there was a real distinction here. But nobody is, and the language is defined by use. This tiresome argument is never raised by "real" engineers, just software engineers who feel like impostors. Relax. You're not scamming anybody.

I find it hard to believe. In 1995 people barely knew what the internet was. I was accessing the internet through Trumpet Winsock on Windows 3.11 and had a 14.4k modem.

The idea that Time Warner already had a sophisticated Web division smells like BS to me. I would accept 1997 or 1998, but not 1995.

It was called Pathfinder, and it launched in 1994.

https://en.wikipedia.org/wiki/Pathfinder_(web_site)

It was not sophisticated by today's standards, but it was one of the first real magazine sites on the web. Here are some screenshots.

http://www.disobey.com/ghostsites/pathfindermuseum/index.sht...

How old is 'big data' thing? Would you be surprised to see a 'senior big data engineer' title these days?

> The idea that Time Warner already had a sophisticated Web division smells like BS to me

the title 'senior' doesn't imply the division was sophisticated. If something smells like BS, doesn't mean it can't exist. There is a lot of BS around.

It's a rank indicator, not about experience. Also, fancy job titles are free employee motivation.

The Emperor has spoken.

"Give me enough medals and I'll win any war"

attributed to Napoleon Bonaparte

You call that sophisticated?

You had the job title "senior web engineer" when the web was 4 years old. That's pretty cool.

Well normally they only hired people with 6+ years of solid web development experience as senior engineers (just like no one today would think of hiring a senior Node.js developer with less than 5 years under the belt). But they decided to give the article poster a break for his people skills and Team Player potential.

I'm surprised nobody, not even the author, has said that the program was useful anyway. But instead of publishing it in the web, it should have been used by the toll-free number operators.

I never cease to be amazed that the more things change, the more they stay the same. If you removed the dates from this story, you'd be hard pressed to fix when this happened.

(Except the static HTML idea is a bit of a give away, no web developer would ever dare suggest a static page in our brave new world of Web X.0)

If a static page is the right answer, why would you not say so? Scales effortlessly for one thing.

I think it was a joke.

Bingo. A surprising number of modern web problems can be solved via static pages - speed, usability, scalability etc.

I remember finally turning down one project after failing to convince the small businessman that his site didn't need an unskippable flash intro with techno music.

Edit: I think that nowadays web 2.0 needs have been replaced with 'BIG Data'. I have a current client whose data set is less than 30 MB, but he insists that it is not just big data, but HUGE data! Which is a shame, As it is an interesting problem he has, but I will probably turn it down too.

If Excel can't open it...

Just say we will load data(phone number) using AJAX in JSON format and everyone will be happy.

Page is static. JSON is a static file with a phone number. And use jQuery to load the JSON file.

My sarcasm detector is showing erratic results from your post. AJAX was 3 years from being invented and John Resig was about 10 years old in 1995.

Perhaps a whooshing sound should be made though, if I didn't get the joke..?!

You missed "If you removed the dates from this story.." in the parent comment.

Ah, thanks!

...into Angular.

Except the most successful political fundraising machine in history that ran on Jekyll [0] :P

[0] http://kylerush.net/blog/meet-the-obama-campaigns-250-millio...

So true. Before leaving to focus on more interesting things I've spent a chunk of my career working developing software services for some big insurers. This tale from 1995 could easily be a story of the challenges we faced in 2012/13.

Data managed in spreadsheets, layer upon layer of disconnected management, no engagement with users, very little willingness to understand software development. More spreadsheets...

I currently work for an insurance company.

Warren Buffet was spot on when he said (after investing in insurance companies): “Invest in businesses that any idiot can run because, sooner or later, one will.”

> These days I would handle this easily; after the first or second iteration I would explain the situation: I had based my estimate on certain expectations of how much work would be required; I had not expected to clean up dirty data in eight different formats; they had the choice of delivering clean data in the same format as before, renegotiating the fee, or finding someone else to do the project.

Great advice for dealing with issues we did not consider in our estimate.

Had he charged hourly instead of a fixed price, would this project have been less shitty?

Edit: Fixed grammar. Thanks b0z0. :)

Props for trying - project have been

An hourly rate has the effect of focusing the client's mind a bit.

I find that fixed bids they try to take me for a ride, time and materials they are afraid to spend time with me to figure out what they actually want.

I have yet to find a customer unwilling to try to doom their own project or a compensation strategy that reliably prevents it - its just tons of work and communication. Unfortunately I'm just an okay coder and most people seem like they actually want a code talk therapist.

Sounds about right.

I worked at Prudential about 10 years ago, as a FTE. Our small division mainly ran on a bunch of custom Access reporting applications. It wasn't quite cutting it, because Access, so it was decided that we would build a portal on the company's intranet. The only problem was that we, as accidental web developers, were not allowed to run development web servers on our dev machines, because they were locked down by corporate. We had to use an extra PC that, by some miracle, had IIS, and develop against that remotely. Good times.

The title reminds me of the shittiest software job/project I ever worked on- short and sweet: Got hired off craigslist. It was all php. Their main competitor was "the spreadsheet". Worked directly next to a cold caller that repeated the same phrase over and over again, fake laugh and all. They were all from the same church group and tried to convert me multiple times. Paid minimum wage.

In addition to being funny in retrospect, it was a good lesson to me to learn that no matter how shitty your current situation, you can always improve it.

oooh... looks like someone has a case of the mondays.

The punch line here is great, but even if the specs necessitated something more than a static page, then it'd still be a hard job.

If the most critical part of a data heavy project is speccing it out, I'd say the next most important part is the data munging process...and sadly, both of these things are the most overlooked.

I'm sure this is the very tough first lesson any new consultant with limited experience would learn. The summary is that the specs are incredibly important, they should be expensive to produce and they should protect you and your client.

tl;dr

lol

I'm sorry.

The grandparent is the author of the post.

Ha! I think the people who downvotted this didn't realize that "mjd" is Mark Jason Dominus aka the guy who wrote the blog post.

Even then, the post adds no value to the discussion, so it deserves downvotes anyway. Its a great blog post, and I upvoted it, but its a bad comment, and I downvoted it.

Sure it does. It says that mjd is on HN, and reading this article's comments at least enough to post this, and that he has a sense of humor.

I'm sure if someone so desired, they could ask a specific question regarding the story either in reply to the root comment or as a root comment and expect it may be seen by the author, and possibly get a reply.

I think the quality of the other responses to the parent indicates exactly how likely that is. A whole subtree full of gray comments.

Maybe I'm just getting old.

Indeed. There's no place for humour on HN.

It looks like mjd is the author of the linked post.

tl;dr - A very long, difficult, and complex calculation multiplied by zero is zero. Or in this case, a very long, difficult, and complex database manipulation culminated in an invariant "Call 1-800-555-5555".

Somehow reminds me of a co-worker wandering into my cube with a dazed look, mumbling "I just ran an 8-hour database job with the wrong data...I just wasted $64,000...".

Yes, I had this impression once.

It's like a very long function that returns nothing and has no side effects.

That's not bad at all, it's sounds more like business as usual. I think it's my regular workday. Btw. Did the customer require extensive documentation, escrow, you to fix their data when they can't get it done (like invalid post area codes linked to wrong addresses, fixing post number / city information based on post area code) etc or looking it up based on street adress or so. Been there, done that. Did you spend several weeks in meetings where they can't decide how their stuff works, and you'll just keep wondering if they'll ever decide what they actually want etc. Of course they're going to completely change that week later etc. They don't have a clue how things should technically work, or even what the actuall business process is. Because they have bought an integration, everything must just automatically work, right? We need to know what females in age range 20-25 have bought during last month. Err, but we don't record customer age or sex? Well, but our management team needs that information. It's sure alarm sign, that they want to know how much "this" project will cost, but nobody really knows what "this" is. Also it's needs to be completed by end of the month. I have declined so many projects, and clearly told customers why I'm not going to do anything for them at all. Unless they accept it as "agile" project, with unlimited budget. Then I'll promise them that I'll personally see that it gets done, but it's going to be expensive. - 15 years of ERP/POS/BI/CRM integration programming & consulting.

I am working on quite a tedious project right now. It involves 10 years old, quite extensive, Visual Basic 6 programs. No source control was used. In our company it is practice to hire interns for 3-month periods to work on production software. A mix of programming styles can be found in this project. Some functions return 0 when they fail. Others return 1. Or -1. Or False. Or "False". I love it!

Count yourself lucky there was any error handling at all! My current job involves a lot of old (and new) visual basic 6 and there are parts of our software which don't work once you remove the magic line:

On Error Resume Next

I cracked open some code the other day to find it riddled with:

    on e goto end

"In 1995 I quit my regular job as senior web engineer for Time-Warner and became a consultant developing interactive content for the World-Wide Web, which was still a pretty new thing at the time."

I am confused the person stated that they were a senior web engineer but quit to work on a new thing the WWW. How can you be a senior web engineer for something brand new?

Just nitpicking...

I believe "interactive content" is the key difference. I.e. non-static pages served via CGI.

This gave me a good laugh, i didn't know what to expect. Just today i had to deal with something at a similar level.

One of the best examples of consultant failure i've ever seen. Clearly an example for

- not enough questions asked - not listened to the customer

Do not ever start building something or proposing a solution if you don't understand the customers problem deeply enough.

i dont think the problem lies with the consultant tho - how could he have figured out the actual business problem, if no body bothered to tell him? He was hired to do a particular task, and that task was pre-determined by some "stakeholders" who dont actually know what they want.

I can highly recommend Flawless Consulting by Peter Block. He covers all these issues and many others. Self awareness is critical to success. If you are inexperienced you need to be able to recognize/acknowledge your inexperience. Then read everything available on the subject and interview every expert you can identify. Clint Eastwood said it best, "A man's got to know his limitations". http://www.youtube.com/watch?v=_VrFV5r8cs0

Fascinating! As somebody who is starting to do freelance work a story like this is not only an interesting read but provides insight on what to avoid and when to speak up as I start freelancing.

hmm yes... until your pipeline slows down.

In general or because of tips I would take from the article? I'm not that concerned about a slow pipeline because at this time, I'm only going to freelance on nights and weekends.

The shittiest project that I worked is just finished where we are not allowed to write any test cases cause we don't have time. I just can't believe that I finished fairly big project without writing a single test case. I hate my company, my role and managers, sales people who negotiate tight deadlines. Over all don't work for any consulting companies out there. They care less of the code quality. They just want money and no ethics.

My next job will be a product based company.

Consider the price you paid to learn a key life lesson which will pay for itself multiple times over. If you learn from the situation, you have not failed.

This is why:

1. You should spend more time designing (away from the computer) so that you find a _problem_ and then come up with a reasonable solution. Instead of putting together a list of features. (see Rich Hickey's talk "Hammock-driven development").

2. Think of the sum you're going to charge for your consulting and then multiply it by 4 and charge that because you have to take risk and other factors into account.

The shittiest project? Hmm...if he got paid, not sure I'd classify it as such. A waste of time because of business getting in the way of potential efficiency that they perceived they wanted? Yes.

Also, let's not forget that this was 1995 and most big businesses weren't really fully aware of the potential of the internet and the disruption on standard business models that was going to ensue...

Sounds like poor contract negotiation more than anything. Fixed quotes are very tricky things to navigate. I simply don't go there. If the client insists then I try to negotiate a fixed budget, then when the budget us running dry they can extend the budget or reduce scope. If they don't agree to that I walk.

So you would take on a project if they kept paying you per hour, even if the end result was useless?

Well to answer that I would need some context. Like :

a) Do I need the cash and is the market quiet at the time? b) Are there some other opportunities that might result? c) Is the hourly rate way higher than I could get somewhere else? d) Is the tedium only short term? e) Am I gaining some significant skills / experience? f) Did I give my word to do the job?

If the answer to any of the above is yes, I may well accept the project or continue on with the project, otherwise I would move on.

I remember when I was working in London, I was on the same contract for 2 years. About 9 months out from the end they extended my contract and we were asked to build a prototype of a system that automated a lot of human elements of the original system we built. We built the prototype, and it was very good, but the funding was pulled and most people left the project. By that stage I was about 4 months from my visa running out, and there was no work to do. So I stayed on and did nothing but about 30 minutes of support work per day and the rest of the time worked on pet projects and amused myself. I felt a little bad about it, as I was earning $120/hr, but I may well have been screwed if I tried to find work with only a few months left on my visa.

Most shitty software is not because of the engineers, but management. Not putting value on good coded and tests.

Interesting read. However, I helped build a web based system for the management of a bowel cancer screening programme. Considering what would be sent back on the testkits and processed into the system... it's the shittiest project I've ever worked on, but not for the same reasons :P

The reason the fee was there was to pay for the 0800 thing.

Some executive rightly saw that paying you to render that useless would free them of that service and the need for the fee (which i bet was not turning a profit)

But, since big companies run on cargo cult... that happened.

Wrong.

In pretty much any business, and extremely so in real estate, referral fees are many orders of magnitude larger than paying for a free phone line.

I'm not working in real estate, but I'd guess that each referral would earn them at least something like $100 or much more - so forcing the "customer" (not really their customer since they don't sell anything to them, but only to the realtors) to call is the correct business decision.

The only WTF question is why they initiated a project that they don't want to use.

If that's the shittiest project you ever worked on then you're a lucky guy. User's will use a software project for all sorts of political purposes that have nothing to do with you and it does nothing but f up the process.

This is how life insurance companies operate on a daily basis. OP had a pretty shitty deal but he should count his lucky stars that he didn't have to interact with actuaries, who would have multiplied the same problems ten-fold.

wow! I am jealous. If that is the shittiest project he ever worked on, he really has not too much to complain about.

Add to this that this was back in 1995. Companies had no clue what the internet was nor what they wanted to do with it, so this kind of clueless behaviour what the customer wanted was pretty much standard for most companies up to at least 1998 - 1999.

The guy has either been tremendously lucky, or he has not worked in too many different projects / companies for the last 18 years....

This got infinitely more readable when I loaded it into Pocket and let Pocket do its formatting on it. Adjustable font size and a more reasonable line width...

I could see why development service team always want to get more and more information about a project. Open it more, then you get better return. Fun reading.

Someone once told me: The client know what he wants, but not what he needs. Success is discovering that before the project ends.

Great post!

Great http://thedailywtf.com/ material.

In that case... TRWTF is that he did not charge them an arm and a leg for all of those planning meetings. If you charge appropriately, only necessary changes get made, and only needed software gets built (most of the time, but you stay paid and happy either way).

Without a pollyfill to support older browsers this is merely a peak into the future.

Ask them what they're praying for, not how to pronounce the Latin.

The shittiest project I ever worked on was a php project that was converted from another language (I don't remember which one). This doesn't sound bad, except they used software to automatically convert it. The PHP code had no comments, minimal white space, and the variables were all hex. My job was to fix bugs.

I worked there for about a week before I quit in frustration.

Well, I once worked on a Web App that was written in PL/SQL, that produced HTML. Every variable was named v1, v378, vT31 (yeah no order), and the whole app was in 3 procedures each 80,000 lines of code, AWFUL AWFUL CODE. Honestly the horrors. Stuff like:

   if condition
      do x
   else
      do x

I worked on it for a month and it felt like years.

That's nothing. I had to fix someone else's program once. It was written in PERL.

How can someone suggest auto convert into another language like that?

In the long run it would be cheaper rebuilding the entire thing.

Every developer would agree and every suit would disagree.

I once worked on a project that was auto converted from VB6 -> C#. Amazingly it compiled and parts of it actually ran.

We still rewrote the entire thing, line by line and then spent months refactoring it. It was our only choice really, so much of the original code was so unclear and all the original developers had left so it was our only hope of keeping the business logic intact.

Joel will have a word with you now

There are some (very few) cases where it makes business sense.

For example, the company I work for has software developed in a legacy language with no support (Forte 4GL).

There are companies that offer Forte to Java migration, it would have been worth it because of better tools and support (even if the code was the same crap as usual :) ).

I've actually been involved in at least three projects to do precisely that. And mostly they worked fine.

In two of the three cases I was converting between "the same language" but different incompatible versions of it and the original code bases had strict style guides. At the end of it everything worked brilliantly, and the code was just as maintainable as it was before the conversion.

In the third case was an full lexer/parser built out to compile VBScript down to JavaScript. The resulting code was not readable or maintainable but it did run and that satisfied the needs of the project.

In my experience is that language conversion projects are gnarly, and depend heavily on style guides and accurate specifications for the languages to produce good output. But these are not doomed enterprises from the start and in the end it's the quality of your convertering code that determines the success of the project.

I have done it for a few files here and there without too much hassle. An entire project sounds like a nightmare though. Also, which language you are converting from/to will have a big impact. VB.Net>C#; no big deal. FoxPro>Erlang; ouch.

Code generation isn't particularly hard as long as your source language or API is well specified. The problems come after.

One shot code generation is usually fine (make it, run it, throw it away).

Similarly, if the target language is never going to be read or edited by a human then you should avoid most issues.

Damn, it reminds me of one of the projects I worked on, a .NET Winforms project with about 2 MLOC.

It was the result of automatically porting a codebase written with a RAD system called Gupta Team Developer / SQLWindows[1]. The original language supported multiple inheritance and the converted code was a huge mess of tangled interfaces trying to reproduce that. The tool generated lots of duplicated code and some monster constructs like a static class with 140.000 lines of code, with all public methods.

I still weep for my colleagues who were left to tame that mess.

[1] http://en.wikipedia.org/wiki/Gupta_Technologies

Reply on news.ycombinator.com