Hacker News Reader

"And we're still hitting up against major unknowns all the time at every level of the practice, big and small."

I was with you until you got to this line. Let's be brutally honest: the limiting factor for most of today's software projects (i.e. CRUD websites and iPhone apps) is not technology. It's developer competency, experience and self-discipline. I'd go so far to say that most working software developers don't even understand the fundamentals of the technologies they're using -- it's why you see people replacing their relational databases for key-value stores, then trying to re-invent database join algorithms from scratch in the application layer. Or why every new generation of "engineers" re-discovers asynchronous programming in a new language (then promptly writes a framework in Blub...because Blub is so much better than last year's Blub.)

Truthfully, it's pretty damned hard to find gigs working on the frontier of technology. Most projects are predictable, tedious slogs to fit together well-understood technologies without shooting yourself in the foot. And yes, requirements shift and clients get fussy, but that's true of any engineering discipline. You learn to pad the schedule to compensate for the shifts. It's part of being a professional.

As far as I can see, commercial software isn't limited by "major unknowns", so much as an industry-wide unwillingness to learn from experience. But hey...that tends to happen in industries when you're considered "old" at 30....

Granted, most engineers are not exactly employing quantum physics to parking lot design. Most people, in general, work on gluing together pre-built parts that they do not understand all of the fundamentals of.

Ah man, this hit slightly hard as someone who just moved from Postgres to MongoDB to avoid database migrations only to realize that I've now lost the power of a many to many relationship.

Wouldn't you look for better migration tooling rather than jumping technology?

As far as I know, the best migration tool for SQLAlchemy is Alembic, and it can't magically handle migrations (which is what I wanted). I jumped technology because it seemed like MongoDB did (and still does) meet my use case better. I'm open to any opinions or advice though.

How would one avoid database migrations by switching databases?

MongoDB is schemaless, so you don't have to do traditional migrations. Unless you're making a joke about me migrating to a different database to avoid migrations, in which case, good joke.

Migrations are about data, not schema. You don't need migrations with a "schemaless" database at the beginning, but as soon as you have data you care about, you're gonna need to ensure some sort of consistency.

Do you think I should move back to Postgres? If so, why? Genuinely looking for well backed technically minded opinions here.

I think you should learn the relational model (not SQL). Understand the problems it was trying to solve and how the relational model solves them. Then figure out how that applies to your app. Database in Depth was helpful for me http://www.amazon.com/Database-Depth-Relational-Theory-Pract... .

Yes, you should move back to Postgres. The reason is you can do many more operations on structured data than on structure-less data. You found that out already with the many-to-many thing. There are plenty of other intangibles that will bite you in the ass later.

An undocumented and unvalidated schema, you mean. A database without a schema is a contradiction in terms.

My apologies, the official documentation referred to MongoDB as "schemaless" so I figured it was a real thing.

You aren't wrong, MongoDB is schemaless in the sense that you don't need to define a schema to use Mongo. But to use any datastore, you'll be defining a schema somewhere - generally, in your app itself.

Depends how you define database. Perhaps it's more precise to say that Mongo is a schemaless key-value store, but in truth it provides many functions that people think of as a "database", and the differences between it and traditional RDBMS seem inessential to what distinguishes a database from other things.

MongoDB guys call a "database migration" something like "adding a column to a table". They'll tell you it's an incredibly complex and time consuming thing to so, AND THAT"S WHY YOU NEED MONGO!

Of course out here in the real world, an "alter table" statement takes no time at all even on a large table, and can be done while the system is running.

Now I am not saying that MongoDB has no uses at all; but I am saying that the people around it don't really have much of a clue about databases in general and aren't in any sort of a position to be choosing technologies.

Could you please explain to me what you dislike about MongoDB?

I downvoted you before I realized that you coming at this from a position of ignorance rather than ego.

Mongo is a way of storing 'stuff'. You don't have to define the form that the stuff takes before you shove it in the database. This makes things easier by lowering the knowledge barrier to getting started, which is a double-edged sword.

What I've gathered is that the tradeoff of not having to deal with schema migrations now is something I'll have to pay for with data integrity down the road. Do you think that's a good summation?

Yes, that's an excellent way of putting it.

Right. Thanks for the help. I'm gonna move back to Postgres.

> As far as I can see, commercial software isn't limited by "major unknowns"

I'm sorry, I needed to get in an afternoon run and actually deleted a paragraph that explained this a little more that I couldn't get just right before posting.

I completely agree with you. There aren't a lot of technical unknowns for most things people are trying to do.

Thing is, I wasn't actually talking about technical challenges with that point, rather, the requirements that are given at the beginning of a project. The requirements for the new Bay Bridge, for example, are pretty darn clear: make a new bridge from Oakland to Yerba Buena Island.

The requirements for making G+ attract Facebook users... errr... uhh...

Or how about the requirements for the next Call of Duty to be fun...

A great many of today's software projects -- especially those discussed here on HN -- aren't straight up engineering problems, they're more design and UX problems. Even for something like the software for the Joint Strike Fighter, you're more into UX than hard-hittin' avionics for a lot of it these days.

And I think that's what makes it hard to predict software development schedules. If we were all building ATM software again and again, we'd be knocking it out of the park. A competitor to Snapchat that is able to sway users over... that might be harder to define up front.

"Real" engineering has the commonality with software engineering that many projects are quite similar to previous projects and are thus fairly predictable. (This is true also of safety factors; I read a good book on it not too long ago[0]).

A bog ordinary 12 story building will probably come in close to schedule and budget because it's a well-understood problem domain and there is a large pool of existing examples to draw data from.

But the Burj Khalifa required solving some genuinely novel problems, of the sort which only turn up at that scale. When a project involves substantial R&D, schedules become much less certain.

If my business is churning out Wordpress brochureware, I can give a pretty good estimate of how long a given job will take and how much it costs.

If on the other hand my customers require me to develop novel, publishable algorithms, then any estimate I give will need to have very wide bands.

[0] http://chester.id.au/2013/07/07/review-to-engineer-is-human-...

I agree with you that the author of this article doesn't know much about how engineering projects really get done, but I have to disagree with your claim that software engineering has only been around for "3-4 decades... maybe". In the 1960s, IBM was building its OS/360 operating system, the massive project that inspired Brooks' famous book The Mythical Man Month. The Multics operating system (which inspired Unix with ideas like a hierarchical file system and an OS written a high level language) was started in 1964. Lisp, Algol and Fortran were all implemented in the 1950s. So software engineering has been around for at least 6 decades.

You are correct, but I would point out that this does not detract from the argument considering the time scale. Software engineering could be 150 years old, and still not have the close to the history of traditional engineering.

Still, you'd think that after three generations of programming, we'd wouldn't have to constantly re-learn such basic lessons as:

- Accumulation of technical debt is bad.

- The latest trendy programming language won't make your hard problem all that much easier to solve.

- A complex system can't be properly integrated and tested in two weeks (as recent news reports have shown).

- The CEO setting an arbitrary deadline doesn't magically alter the laws of physics.

> Accumulation of technical debt is bad.

Is it always? In dollar amounts, I really do not know that it always is. "Don't let the perfect be the enemy of the good" and all. As a developer, I am always going to be the one push for better tests, more efficient systems, etc. but pg's post a month or two ago about scaling without automation (very rough paraphrasing) points to the fact that this does not always hold.

> The latest trendy programming language won't make your hard problem all that much easier to solve.

Maybe not the latest, but I would think that it was way easier to develop a desktop app in C++ than Fortran, develop a website in PHP than C++, develop a web app in Ruby than PHP, etc. Tech definitely improves over time. Jumping on every third bandwagon seems to be worthwhile.

> A complex system can't be properly integrated and tested in two weeks (as recent news reports have shown).

> The CEO setting an arbitrary deadline doesn't magically alter the laws of physics.

These are not engineering problems, but political/administration problems that have been around forever and in every endeavor. I am certain that stone masons building cathedrals dealt with the same sorts of incompetent management, despite a dedication to craftsmanship and engineering.

Technical debt, just like traditional debt is fine, as long as you understand what you are getting yourself into

Mortgages are debt, are they bad?

> Mortgages are debt, are they bad?

Yes. Yes they are. Many people make a huge bet on the housing market with borrowed money, without even realizing what they're doing.

See where I said "as long as you understand what you are getting yourself into"

We don't have three generations of programming, really.

The majority of people involved in making decisions and programming right now haven't been doing this professionally for 30+ years. And for those who have (I'm closing in on 20), the culture that they grew up in was not shaped by people with experience programming.

The examples of "programming" and "lessons" from IBM/etc in the 1960s hardly affected anyone outside of those few small teams with respect to learning lessons re: software development.

IMO, "modern" development didn't really start until the early-mid 90s, when networking really started to take off at the small business and consumer level. This has meant that people who'd never dealt with making software decisions before now are - most of the clients that I have who are in their 30s and 40s have had 'website' duties thrown in their laps, and it's just another item on a checklist.

The lessons and timetables are still new to these people - certainly the mentors and bosses they had 15 years ago never had to deal with this stuff, so they were never exposed to it. The last 15-20 years have been huge in terms of businesses getting crash courses in software experience (whether they wanted it or not). It'll be another 15-30 years (imo) before these maxims are taken to heart by organizations when planning, because they'll have had 2-3 generations of institutional successes and failures to draw on. MOST orgs don't have that yet.

I'm pretty sure that the kind of engineering that designs circuits to do things in hardware that you'd normally do in software can't be that much different than writing software. A title that I find more ridiculous than software engineer is software architect. Architects are partly like engineers, but they also design things to look pretty while serving their function. Making software look pretty on the inside is utterly pointless, because users only see what the software does, not what the design looks like. Designing software is a lot more like engineering than architecture.

"Or the California High Speed rail is projected to be twice the budget they thought, and they've barely broken ground on it?"

A correction: They've broken no ground on it. An AP article today reports that the "California High-Speed Rail Authority is in escrow on just one parcel of the 370 it needs" for the first 30 miles. http://abclocal.go.com/kabc/story?section=news/state&id=9320...

And the entire effort is in trouble: "Sacramento County Superior Court Judge Michael Kenney ruled that California's high-speed rail authority had violated the letter of the 2008 ballot initiative authorizing $10 billion in bonds for the 500-mile train's construction." http://online.wsj.com/news/articles/SB1000142412788732410820...

I actually agree with the parent comment re: engineering but wanted to make sure that folks didn't think ground had been broken already on California's high speed rail plan. I wasn't sure and had to check myself.

Ah, thanks. I thought they had started building something out near Bakersfield, Modesto, or wherever the heck that thing is going on.

If large scale civil engineering projects were not muddled by politics, contracts, and bidding wars, I imagine that wouldn't be the case.

So in an abstract since, and in a world where engineers are the only ones involved in construction, I understand his point.

> If large scale civil engineering projects were not muddled by politics, contracts, and bidding wars, I imagine that wouldn't be the case.

And software development is never involved in any of that?

While I suspect software is more often plagued by problems I do not see that it is another kind of problems than other forms of engineering.

Waterfall software design would also probably work if it was not for changing requirements, limited budgets, optimistic estimate push, etc.

Like the healthcare website?

>So the idea that "Real Engineers" building bridges and railroads and skyscrapers get it right in plans and schedule is total bullshit.

A few examples do not an argument make. You don't think in the "law of long numbers" sense.

If you did, you'd find out that MOST engineering projects do end on time and budget, and even if they miss, they miss far more gracefully and with extremely better predictability of outcome that MOST software projects.

That you can find some counter-arguments doesn't mean much -- what TFA proposes holds in general, and holds much better than the alternative propositions (that engineering is as fucked or more fucked than software). Of course there will be several counter-examples, it doesn't have to be perfect to be near-perfect compared to software schedulling.

>And by the way, those "Engineering" disciplines have only been around for, what, hundreds, if not thousands of years. What they do for a living is mostly explored territory.

That just reinforces the original posts thesis.

The skyscrapers and bridges that are finished on time and on budget are the ones that have been built before. For physical structures, it makes sense to do the same work again, because then you have two bridges or two skyscrapers instead of only one. People can live in two different locations, and they can cross yet another river, which is useful.

For software, if you need another copy, that's a trivial task. It's more on schedule and budget than any skyscraper or bridge could ever hope to be. Once built, we can use it everywhere. The only thing that requires any real effort, is building that very first skyscraper.

I agree with you. My post was only really trying to promote Agile over Waterfall. Never intended for it to be taken so literally. I hope people enjoy reading it regardless.

It certainly makes for good HN discussion fodder :)

FWIW, I've had the "Real Engineering" discussion before -- way more in depth than you illustrated here. My grandparent response above is probably a bit of pent-upness towards those discussions more than anything about your article in particular.

I come from a civil engineering background. The problems you cite are not engineering problems - they're management problems. In the environments I've worked in, heads would have rolled for the over-runs/over-dues you mention. We call these NFE situations.

BTW - thank you for mentioning (indirectly) that civil engineering is the world's oldest profession. ;-)

The author didn't say that every engineering project was perfect — only that he would believe an engineer if they gave him a very specific estimate.