Damn Excel – How the 'most important application' is ruining the world
finance.fortune.cnn.com
finance.fortune.cnn.com
Putting a very powerful (if archaic) tool in the hands of people without the experience or understanding to use it effectively and then passing around the results for editing by other people with similarly (or differently) deficient ability with the program is the problem.
The tool is simply too flexible to be used for many of the things it's doing by many of the people who are using it. This is one of the good reasons that we (application developers) do our best to gather requirements: Excel is usually good at what it does, but it does so many things that understanding what it does that you don't need it to do, or how and why it does things you've accidentally done or need to watch out for, has become as -- if not more -- important as/than understanding what it was you wanted to do in the first place.
So yes... Excel plays a role in some very bad things. But no, it's not Excel's fault.
So don't try to hide and dumb down the programming part. This of course doesn't mean that mistakes wouldn't happen with better tools.
This attitude bothers me somewhat. I'll certainly agree that the combination of Excel's power and ease of use allowing people to model really quite complex calculations and scenarios quickly and easily, and the ease of which they can do it leads to a inappropriate level of "respect" for the gravity of what they are calculating, insufficient testing, double checks, etc, leading to some massive blunders.
But many people seem to go beyond this, asserting that one cannot do such things properly in Excel. I've actually heard one person with an absolutely straight face say that Excel should literally be taken away from engineers and finance folks, and any calculations they need to do should be custom implemented by "professional" developers who will do it "properly".
Thoughts?
In my opinion, people who don't check their work, just don't check their work. Whether they're using Excel or a "real programming language" doesn't matter. You can't blame Excel because people are reckless.
However, in addition to problems caused by its users, there have also been problems resulting from design deficiencies in Excel: automatically and irreversibly converting strings representing gene names into dates [3], displaying some numbers incorrectly [4], and incorrectly evaluating some statistical procedures [5].
[1] http://www.edwardtufte.com/bboard/q-and-a-fetch-msg?msg_id=0... [2] http://www.wired.com/wired/archive/11.09/ppt2.html [3] http://www.biomedcentral.com/1471-2105/5/80 [4] http://www.joelonsoftware.com/items/2007/09/26b.html [5] http://dl.acm.org/citation.cfm?id=635312
Excel is designed to be used for mission critical applications under certain circumstances. And it does a piss-poor job of delineating what those circumstances are, it deserves a lot of criticism for encouraging people to misuse it.
I don't know how anyone could see the last bit about Native Americans and think that the article was actually serious.
Accountants during preparation of these documents often "hide" for a reason - they want to try out how certain numbers work, and want to keep their work-in-progress easily accessible without deleting them. And often corporate secretaries just send out these documents "as ready" without thinking that there's tons of confidential and sometimes very wrong (or very incriminating) numbers "hidden" as work-in-progress.
This happens so often on Quarterly financial statements that it's an embarrassment, frankly. Disclaimer: I worked as a lowly proofreader of financial documents, quarterly and annual reports, etc.
The problem is not Access Control. The problem is people failing to recognize when a document is "ready for public consumption" vs an internal working draft.
The real story of "lacking access control" is when these excel worksheets actually link into MS Access databases hosted inside the firewalls of these companies - and holes were poked in the firewall to let the data out because executives wanted to work on these documents in their lodge in aspen, etc.
I see this all the time - it's really freaky when the excel worksheet I was proofreading start reaching out over the net and into supposedly private networks deep within financial houses. If I run something like little snitch, it would show access attempts over nonstandard ports, etc.
It's scary.
* Interesting to me at least, because I work on things like this. Not contradicting your comment at all.
"Microsoft’s continuing inability to correctly fix errors is discussed. No statistical procedure in Excel should be used until Microsoft documents that the procedure is correct; it is not safe to assume that Microsoft Excel’s statistical procedures give the correct answer. Persons who wish to conduct statistical analyses should use some other package."
On the accuracy of statistical procedures in Microsoft Excel 2007, 2010 http://www.pages.drexel.edu/~bdm25/excel2007.pdf http://homepages.ulb.ac.be/~gmelard/rech/gmelard_csda23.pdf
Excel does a pretty good job in highlighting which cells are selected when you edit a formula, and it does a pretty good job of maintaining the meaning of the formula under sheet transformations. For example, if you inserted a row between rows 8 and 9 then the two formulae would be =SUM(A3:A13) and =SUM(A3:A11) respectively.
This may have been a genuine error, but the two researchers definitely started the process with a goal in mind, and when the results agreed with their goals they didn't bother to check.
What would that goal be? According to Megan Mcardle, Rogoff was a mild proponent of stimulus. For example, he said this in 2012:
"Back in 2008-9, there was a reasonable chance, maybe 20% that we’d end up in another Great Depression. Spending a trillion dollars is nothing to knock that off the table."
http://www.thedailybeast.com/articles/2013/04/17/did-reinhar...
I think we're thinking there is a technological solution to a human problem, and personally I don't think there is or could ever be.
This is a process problem. And by process I mean the process of building complex datasets, validating them for "correctness," and judging the quality/correctness of different data sets.
Unit tests are a massive asset to programmers. I wonder if unit tests (i.e. "sanity checks") would also help in this situation? I mean you would have to force people to write them and monitor them, but once they've been created they pay for themselves by picking up unexpected errors.
However, all of this is fixed when you are using SQL to properly query a database. Why? Because you are forced to write a SQL statement that details exactly what you want done. With Excel it can all be hidden away behind the cells. With SQL, it's out in front, so it's easier to check.
People like to use Excel because they can get an answer quickly without all that "programming". The problems start to arise when you need better tools, but only know Excel. So, in this case, it's not a matter of a craftsman blaming their tools, it's closer to an amateur trying to pretend to be a professional.
Excel is a wonderful spreadsheet. It is a horrible database.
I mean Excel is at its heart a query language. So your logic equally applies to Excel, why not write a massive query in Excel that does all the calculations in one go so you can see the inner workings?
All you're really doing is playing musical chairs with the data. SQL query language for Excel query language, and data moved from tables to worksheets.
As I said earlier, unless there are procedural changes upstream nothing will change. A tool is just a tool. You can use it in a way to minimise mistakes, or not. Humans are the weak point.
Now there are tools that automatically identify common mistakes but neither Excel or SQL/relational database engines are in that class.
The least readable query language ever. Instead of names, everything must be addressed as column,row. Imagine a C program that instead of declaring variables with sane names as needed, simply created a large array of each type and used constant numeric indecies. That's what Excel requires.
With respect do you even know how to use Excel? Because everything you just said is incorrect.
Only one of the tens of heavy users of Excel I have worked with know of this feature on others like it. I am not a heavy use of Excel but my theory is that this is due to a failure of explanation by the software. Excel does not by its design encourage good coding, and most of the users of Excel do not have any programming background so they do not realize that these features should even be there so they wont know to look it up in the manual.
If you plan to use Excel to aid you in making serious money (as countless business do), can't you fork out a thousand extra for an employee who actually knows how to use Excel properly?
You can have named ranges in Excel - in fact, you can have named ranges which are dynamically calculated based on a formula.
SUM(A100:A112)
SELECT SUM(yp.profit)
FROM yearly_profitability AS yp
WHERE yp.year > '2012-12-31'::Datetime
SQL and other programming languages use words to describe the variables. Treating SQL like Excel would be DailyWTF material:
CREATE TABLE sheet1(
a INTEGER,
b TIMESTAMP WITH TIME ZONE
)Which is easier to understand:
=SUM(profits01_2012:profits12_2012)
Or:
SELECT * FROM dassaddasads WHERE date > 1325394000 AND date < 1356930000
Which is easier to understand?!
I'm not saying that Excel can't do it, but a) it sounds like it would take a huge amount of time, so b) it is not practically done in "the real world."
Why would you quit the job over a few minutes of scripting with "WorkSheet.Names.Add"?
http://msdn.microsoft.com/en-us/library/office/ff835300.aspx
The really amazing thing is that the authors have not retracted the study, and, last I heard, they claimed that their analysis was correct in spite of the errors and exclusion of data which did not support their thesis.
Additionally, slow growth causes lower govt revenue causes less incurring of debt to make up the shortfall in tax base.
Debt and growth are both inputs and outputs that are codependent. The degree to which they impact the overall financial picture depends HEAVILY upon the opportunity cost of the servicing of the incurred debt.
For bonus, consider what will happen to heavily indebted countries when interest rates begin to rise. Servicing of debt will become more difficult leading to a decrease in our ability to grow the economy, leading to more difficulty in servicing the debt, and on and on.
http://www.nextnewdeal.net/rortybomb/guest-post-reinhartrogo...
http://krugman.blogs.nytimes.com/2013/04/20/the-good-glitch/
I've often wondered about this aspect of political discourse. It's not rational, it's symbolic. Yet most of the time when a symbolic soundbite takes off, even though it's wrong or misleading per se, there's still some poetic justice to it. That is, it's rare that such a soundbite is completely unfair; usually it turns out to be a metonym for an arguably legitimate criticism. This is a case of that.
HOWEVER, spreadsheets are rather unique in the fact that they hide the very formulas used to generate everything, unless you have a specific cell selected.
So, opposite of computer programs, they're almost impossible to "read through" and verify that everything makes sense. If 99 cells have identical formulas, and 1 in the middle has a slight typo, how on earth are you ever going to see that?
I wonder if there are other similar spreadsheet-like tools that one might use, that make a point of showing everything going on in the spreadsheet, rather than trying to hide it?
If you've got 99 cells with identical formulas, and one without, it's not far fetched for Excel to speculate that there may be a mistake (particularly if they're within X% of being identical). At which point it can provide a simple, non-intrusive hint.
And going further up the wishful thinking tree, into more difficult territory, Excel should be able to grasp what you're attempting to do by understanding, say, the top 10,000 (arbitrary number) tasks commonly performed by people using spreadsheets. That 'understanding' could then tip Excel toward deciding whether there's a problem, and whether it should leave you a nice little illuminated icon at the top right of the program with an exclamation mark.
Microsoft should have insane amounts of data on everything people do with spreadsheets, and they should be able to make it radically more intelligent (without being obnoxious or intrusive - it should be optional and highly passive).
It already does this, actually. Enter some data in two columns, like:
4 1
2 3
5 6
3 2
4 2
2 1
http://www.c-sharpcorner.com/UploadFile/7e39ca/error-check-f...
But that's not to say that it'd be easy to spot the one that is different. This is why excel has a little tooltip that pops up and says 'inconsistent formula' when it finds a cell that doesn't seem to belong.
I think a better idea would be to scrap the datasheet concept completely. Make people use a dataflow programming language, it will give you:
a) versioning of source code with standard tools
b) separation of inputs and process
c) program source that doesn't look like a game of battleship unless who wrote it was disciplined
d) strong typing
But it won't happen, god forbid we are faced with an initially steep learning curve.
If we want to enter 99 identical (or relatively identical) formulas into 99 cells, then we should be able to select all 99 cells, and enter the formula once without any dragging. If we want to change that formula, we should be able to change it once and not need to remember to redrag. We should be able to enter a formula into the column heading and know it'll apply whenever relevant to every cell in a column. That we can't means Excel (and Lotus and OpenOffice) is defective.
What QA is to blame for isn't letting unfinished spreadsheets out the door, but rather for letting users in a company use defective tools like Excel and Lotus and OpenOffice in the first place.
Many of us developers learnt the hard way that visual programming tools from the 1990's don't work, which is why scripting languages became popular. Users in corporations (and the IT staff who authorize what tools that can use) need learn the same lesson.
Agreed. You can do this by creating your own macros, but it's a pain. Considering how often I do this for personal spreadsheets, you'd think that this would have been simplified long ago.
And, this deficiency has been ported to both Numbers and Google spreadsheets.
Double-click the fill-handle (thing you click on to drag) and Excel will fill-down as far as the next empty row.
If we want to change that formula, we should be able to change it once and not need to remember to redrag.
How would it know whether you want to change it in one place for a special case, or update many places?
You can complain about the UI (I certainly do. For example, try inserting a row in an array formula), but the feature is there: http://office.microsoft.com/en-us/excel-help/introducing-arr...
A5 = (C6*C6 * Sheet5:D1 + Sheet6:D1 / B3)
Sure there are; You can add a name for a cell or range of cells (and have been able to since at least Excel 97):
http://excelsemipro.com/wp-content/uploads/2010/09/Named-Con...
Unless you click the button to "show formulas" in Excel, then you can view them all at once.
http://i237.photobucket.com/albums/ff263/dharma-putra/Excel2...
http://www.lytebyte.com/wp-content/uploads/2008/04/view-form...
If only I had used a recipe management package that would have automatically told me that I wouldn't have enough buttermilk to make waffles for 5 kids.
26.5 meters + 56.3 dollars
In Scala one can certainly build this. You could certainly do it in Haskell, though it wouldn't be as nice syntactically.
26.5_meters + 56.3_dollarsWithout the neat inherent aspects, any language with lightweight support for phantom types can achieve some level of this functionality.
For instance, you could define a sum type with only one case taking the type to wrap. Thus you can easily have TransactionId, AccountId, etc. as distinct types, without spending more than a line to declare them.
*Main> 2 meter + (1 meter / second) * 5 second
7.0 m
*Main> gallon `as` (cubic (deci meter))
4.546089999999999 dm⋅dm⋅dm⋅#
Anybody can define new units just by giving a conversion factor and a string to display. These new units immediately work with other units. You can probably define new "dimensions" as well, but that's bound to be more involved.
All the units are represented as empty types, so it should have no overhead at runtime.
So it is very nice syntactically, with the only real problem being the error messages and the fact that it uses a whole bunch of extensions and only supports very new versions of GHC. I think it's pretty cool.
[1] https://www.google.com/search?q=1+gallon+in+cubic+decimeters
As to why it's an imperial gallon, or what an imperial gallon really is, or who uses it--don't ask me. I'm a very metric sort of person :P.
[1]: https://www.google.com/search?q=imperial+gallon+in+cubic+dec...
Excel's real limitations came around scalability of developers (beyond one developer you are in a bad place) and performance that can drop of a cliff beyond a certain size. But it is an astonishingly powerful tool.
Reinhart and Rogoff were important because their idea supported an existing financial theory. If their paper had reached an opposite conclusion, there still would have been austerity.
Now that their paper has been shown to be in error, there will still be austerity. And that will be because the people who benefit from it have the power to impose it. Where they don't have the power, it will not be imposed.
I feel like the same thing applies to Excel. Essentially what gets built with Excel are small programs where all the variables are unnamed, everything passed around is a numerical value and there's almost no capacity to see the entire 'program'.
These always get built because Excel is easier to start with (and you can always pass an Excel spreadsheet around and let other people 'run' it), but it means in the long run with the bigger programs you hit exponential complexity pretty quickly and by then you're invested in Excel, even though R or Pandas would have been more appropriate for the size of the program. Just my two cents (coming from businesses that LOVE excel for completely insane purposes).
Not to mention Excel often ends up having better performance than most home-grown solutions, especially now its core functions are properly multithreaded.
TLDR is that the ResolverONE spreadsheet writes Python code as you manipulate the spreadsheet. A software developer can then organize that Python code, write unit tests for the code and validate that the calculations do what the CFO intended. Once this extra work is done, then the business can have confidence in the results of the calculations. But the user interface, i.e. the spreadsheet that end users see, is the same as it was originally.
This allows mission critical software development discipline to be applied to the development of spreadsheets. I suspect that the existence of a tool like ResolverONE is one of the factors that led the SEC in the USA to require Python source code for new derivatives that are created, so that there is a standard way to express the calculations of the derivative.
Bug tracking with Jira, Bugzilla, Mantis, or Trac? Heck no, Excel is the tool for the job!
I just choked on my soda. Bugzilla packages install in a command line and you can be up and running with a product, a couple of components for categorization, and user accounts 15 minutes later.
The only time I don't take that 15 minutes is if I'm working on a project where I'll NEVER be working directly with anyone else. Even then, it's often nice "just in case" someone else comes along later and wants to collaborate, you need to keep up with issues fixed in various releases, etc.
In particular, any piece of software that needs an MTA in order to run is not going to be pleasant when you are running a one-person project. What, are you going to want to receive email updates from yourself whenever you update a bug that you created on the computer you are using right now?
I wasn't even talking about installation, however. Go to the Landfill if you don't believe me (http://landfill.bugzilla.org/). You want to create a new bug? You first have to create a product, then you have to create components. You have to fill out the entire form when filing a bug, even if you only have one product, one component, and only target one operating system.
Then look at a bug report. Log in to Landfill and look at a bug. Here's one: https://landfill.bugzilla.org/bugzilla-4.4-branch/show_bug.c...
Then think about the cost of running Bugzilla. You are now running a web server. What if you don't have a server? What if you just have a laptop for the road and a desktop at home? How are you going to do database backups?
Tell me: does this beat an Excel spreadsheet for a one-person project?
Edit: I'm not saying Bugzilla is bad software. I'm saying it's like a stick shift. If you put an automatic transmission in a car, you alienate users who prefer manual. If you put a manual transmission in a car, you alienate users who don't know how to use it.
From the same docs you linked to:
> Bugzilla is dependent on the availability of an e-mail system for its user authentication and for other tasks.
> This is not entirely true. It is possible to completely disable email sending, or to have Bugzilla store email messages in a file instead of sending them. However, this is mainly intended for testing, as disabling or diverting email on a production machine would mean that users could miss important events (such as bug changes or the creation of new accounts).
(emphasis mine)
What distro doesn't already have an MTA installed?
Create product: Click the "new product" link then type in "My New Software Project" enter.
Create components: Under your new product, click the "new component" linke then type in "Miscellaneous", press enter. There, you have your product and a catch-all component for all your new issues. Divide into more meaningful components as time goes on.
What if you don't have a server?
Now I see the misunderstanding here. I was referring to professional software projects where I've had to work with other people, some of whom here and there wished to track bugs in spreadsheets. Any such project that I work on has at least one server available. Hell, for small consulting projects I just use one of the servers running in my house to bootstrap things.
You know, you can get a RaspberryPi and plenty of flash storage space for running as a server (with Bugzilla, an MTA, and everything) for projects for less than $50.
If you're referring to small personal projects then sure. Use a text file or a spreadsheet or strings tied to your fingers to remind you of things.
> I was referring to professional software projects where I've had to work with other people
We're having different discussions. It happens. I was pretty careful about choice of words: I said "tiny" and "one-person" rather than "small". For a "small" project I'd use Bugzilla or Trac or whatever makes the grass greener with the least fertilizer.
Not every developer wants to be a sysadmin. Some developers are hobbyists with a single Windows machine—for those developers, Bugzilla administration might be an arduous task which requires skills they don't want to waste graymatter on.
Excel is quite sufficient for many tasks.
All these failures just get magnified when the software is then distributed out in the field or new developers join the team down the road.
Personally, as a developer with other things to do, I'd rather just jot my notes in my bugs.txt file instead of remembering to put them in BugZilla's "Steps to reproduce:" text box.
I don't understand your point about searchability. It's easier to grep through a text file than remember the proper incantation to go in <random bug tracker package's search box>
Now rather than hire a professional or take performance driving courses they try and do the same in high end usage and things go wrong.
It's not the sports car that's at fault by someone driving it at the edge of its performance without the proper skill. If the just drove it around town they'd be fine, but taking a high performance car to the track with no experience, well bad things happen. We've all seen youtube clips of high performance crashes from inexperience ;)
http://office.microsoft.com/en-us/excel-help/correct-common-...
Probably that means such an add-on would not be sellable? However it wouldn't be the first time someone paid lots of money for something that software they owned already did just fine.
There is no other way to detect errors involving somebody using SUM instead of AVG or aggregating data from wrong fields (examples from the article).
Use a table saw. Cut off three fingers instead of wood. Blame the table saw.
Go scuba diving. Return to the surface too fast. Get the bends. Blame the equipment.
Drive a car. Have an accident while texting. Blame the phone.
Go fishing. Be careless. Put a hook through your hand. Blame the fishing rod, the reel and the hook.
Use Excel. Don't make an effort to learn what you are doing. Mindlessly copy and paste. Do not seek out peer review. Cause massive financial losses. Blame Excel.
Boy, do I like the sound of the word "moron".
Excel is sometimes the culprit (Excel 97 bug: http://blogs.office.com/b/microsoft-excel/archive/2007/09/25... ), but usually manual error is to blame.
I wanted to voice my thought anyway... this article prominently features the information that the mainly cited study showing that national debt is bad is no longer valid. Those wanting to fight for higher debt levels are absolutely ecstatic and will up-vote any article that even tangentially mentions the Reinhart/Rogoff failure.
Unfortunately, these things are treated like earthquakes in some countries: it's rare, we don't care. However, bad quality of reports is not rare. But the problem is: companies do not have other ways. They are stuck with the big players (SAP, Oracle, etc.) and they can't understand why their IT stack is crap.
I wish from all this Excel bashing we could have a serious talk about these tools. BEFORE writing any line of code, we must first understand deeply why professionals use Excel in this way and what we can do about it.
Finally, realize that the "cloud" is not considered always "safe". People put in Excel all kind of "secret" data and they are scared of having them on a third party server. Please, consider this too when thinking about a possible solution.
One is a large state school, the other an elite liberal arts school. UMass Amherst has a notoriously leftist econ program, Amherst College your typical neo-liberal.
One interesting thing that came out of that thread was a link to this review article on spreadsheet errors: http://mba.tuck.dartmouth.edu/spreadsheet/product_pubs_files...
This is a big deal. In fact, the ResolverONE spreadsheet was created as a way to reduce the risk of the kind of errors that are discussed at the EUSPRIG conferences.
The popular Microsoft program has been implicated in the financial crisis
Excel is great at providing a mediocre database, presentation template, calculation engine, and model builder, all-in-one. It's even Turing complete (although it makes it hard to write decent code, since that's not the point.) It does a large number of things, and a mediocre job of all. Often, this is enough. It gets the job done for small toy problems, and often that's a start. However, basing billion-dollar decisions based on Excel code? Are you kidding me?
This is exactly why I rail so hard (and without giving, ever) against Corporate injections into software. Corporate America is founded on mediocrity being good enough. Sometimes it is. Sometimes, it's fucking not. When it's not, stop using Excel as a database (it's not one) and fire all your CommodityJavaDevelopers writing SingletonVisitorFactories and start doing things right. Since you don't know how, you have to hire someone like me and give that person lots of autonomy, but it can be done.
I'd prefer to spend most of my time on actual work. I'm getting back into Clojure and rediscovering the joy of coding (which disappeared, with fleeting breaks in it, when I went to Google and had to answer to a middle manager, which just sucks all the life out of what we do). That said, if someone is willing to give me $40,000 to tell him what he's doing wrong, I'll do so.
(In person, I'm much less abrasive, though. If I actually respect someone, and I respect most people, I'm nice.)
Burned Bridges Consulting would work because it's structured to produce the only employment relationship in which telling the truth is possible: one where there is no possibility of continuance.
Imagine if a manager had tons of advice for programmers, ie; 'listen to what management says, build what we want not what you think we want, pay attention when we interrupt your coding!'
If you've already done the whole management thing also and it worked out well then good luck to you.
The only credibility that has any place in anything is the credibility that comes from making others' lives easier-- solving their problems, teaching them, building things they want.
I think I'm poisoned by corporate environments (a startup being the worst) where you have to yell at people for days to get the permissions necessary to do a few hours of real work. I fucking hate that shit. Eventually, people stop yelling at each other about work (because they forget how to work) and are just yelling about the yelling. That's what Corporate America really is: recursive wankery with almost no one really working.
That was fucking low, man.
I'm actually extremely productive when things work right, but when people persist in fuck-uppery it's like a low-grade chronic earthquake-- not exactly dangerous, but jarring and impossible to ignore and eventually exhausting.
By the way, the danger of startups isn't failure. It's that you end up like me: someone who can't tolerate the mediocrity, idiocy, and resistance to creativity that most people bring to their working lives. In most corporate environments, those are affordable background noise. In startups, they're existential risks. Unfortunately, it's hard to unlearn an allergy to fuck-uppery.
I am humble in the sense that I have a keen awareness of my limitations. I don't know what happens after death. I don't know if there is a God. I don't know more than 1.5 natural languages. I'm a mediocre athlete. On most topics, I'm less smart than a person with a passing knowledge of the field, and my opinion is consequently less useful.
I'm arrogant insofar as I've seen through the corporate nonsense and, after having watched people way inferior to me making huge decisions that affect peoples' lives, I feel an urge to step up. I know for a fact that these intellectual children are not the best people to be making such calls, because I am better.
Now, that said, I'm far from the smartest person out there and if someone smarter than me steps up so I don't have to, then that's great! I don't care about being the leader and I'd rather not. I want a competent leader. If that person is someone other than me, then great! But usually the people who are smarter than me shy away from power; they're smart enough to realize that that competition is utterly soul-raping.
The world needs people like me with actual competence and the courage to, at least, try to turn it into something.
I like animated disagreement. My wife and I argue constantly. It's great, because although I have a lot of good ideas, I have plenty of bad ones. Ideas are best tempered with fire. Let's beat the damn thing until we have something good. It's a difficult, chaotic process and it takes a lot of passionate minds. Corporate processes tend toward creative mediocrity by default because no one cares. Well, if no one cares, then why are we having this fucking meeting? Just go off and do it, then.
Ad hominem attacks piss me off because they're just so fucking useless. They add nothing, but they change the discussion to a toxic one quickly.
Maybe he wasn't making an ad hominem attack. Maybe I'm just insecure because so much of my career has been wasted on manager-blessed bullshit and I'm finally waking up. I lost someone last winter, so the value of time has really been impressed on me of late.
"jeremiad"
"screed"
"phillippic"
"tirade"
MOC is a strong advocate for scala and clojure, so I figure he's ok
I get that railing against unsympathetic targets like big corporations makes for entertaining reading, but I don't think that actually qualifies anyone for the job he was describing.
Yeah, you don't. But you let them fail, and then convince the next generation that their elder's messiah is a fraud. Mainly by being very vocal, and using the failures to showcase your point.
In a while, either everybody realise what they're worshiping is just some normal piece of data, or you started another holly war.
Only the 10 or so Google trolls.
But you amassed crazy Karma in just 2 years and I very much agree with your last two blog posts, so I'll give you the benefit of my doubt. Also this comment about Excel is spot-on.
Thanks. I try.
I feel like mediocrity (as a pejorative) is an artifact of a convex performance function where excellence is 10x as potent as a half-assed performance, as opposed to concave work where it's 1.2x. We don't gripe about cab drivers doing "mediocre" work. We ask them to take us somewhere and if they do it, we're usually happy.
[ETA: I realize I didn't answer your question. The reason corporate mediocrity is evil is that it's an artifact of the wrong people making decisions. If you wish to excel, you'll make a real effort to listen to people and get the right ideas into implementation. But a lot of companies tolerate executive bikeshedding because they don't care about excellence.]
Good risk: trusting smart people with their own time. Don't take this risk and you will fucking fail because everyone with an IQ over 125 will sabotage you (or, at least, abandon you for better things.)
Bad risk: not backing up your database, skimping on code correctness in safety-critical code, playing with guns.
People who don't know the difference should not be making decisions that affect other people.
Watch a skilled engineer or accountant use Excel and you'll see it's good for more than "toy problems".
* a database,
* a high-performance computation framework,
* a good programming model for long-lived "code" or multiple-writer projects.
It's also not a bad computational framework, within the bounds of its capabilities.
Corporate America isn't founded on anything in particular. Corporations are founded on what its individual founders think is important.
This implies that if you have a better idea on how to run a corporation, start one and prove it's better.
If a dataset is under 200k or so rows, I'll generally do analysis in Excel. Larger than that, and I'm turning to Python.
Even then, the end result has be be passed to someone who isn't an analyst/developer, so where does it go? You guessed it: into an Excel spreadsheet, formatted nice and neat, and sent up the chain.
If everything works they can claim they did in one month what a team of 10 would do in a year. Bang, instant bonus!
If things doesn't work, they blame the tools and try again.