Steve Yegge Grades His Own Predictions
sites.google.com
sites.google.com
Imagine someone in 1850 making the prediction that "by 1950, everyone will have a horse". Though they say "horse", they really mean "everyone will be able to afford their own personal transport", as a reflection on the changing socioeconmic something something. And they would be right. Sure the horse is mechanical, isn't shaped like a horse, and consumes petrol, but your prediction is correct, even though you didn't know exactly how it would play out.
The same is the of Yegge's XML DB prediction. If he was saying that XML is going to be amazing, then he's wrong. If he's saying the RDBMSes have problems, then he's right. And actually he's saying "RDBMSes suck, so something will replace them, and XML is big enough that it will be XML". So that looks like a 50% score to me.
With Apple, he's basically saying "apple is shiny, and shiny will win". That prediction has been 100% correct. The fact that iPads disrupted the laptop model means he's more correct than if they had just sold more Macbooks. 100% score here, IMO.
But XML database technology actually has nothing to do with XML. The idea is that you store nested data nodes. Imagine you have a mongodb collection, and in it you have documents, and each of those documents would also be a mongodb collection in which you could insert more documents. You can build full graphs like that, and if you allow references to nodes those graphs can be cyclic too. Pretty powerful right?
Now you might say yeah of course, but that's got to be crazy slow! And at the time Yegge made his prediction you would have been right. But then academics came into the scene, they developed data structures and algorithms that allow you to store these recursively nested documents in column stores (think SQL table with 2 columns, a primary key and a piece of data) in such a genius way that you can make whatever query you can think of (recursive, joins, whatever) and execute them almost as fast as any indexed search query.
That magic happens, and is implemented for example in MonetDB/XQuery which sadly got abandoned a few years ago.
I still think if a crazy person sets off to go build a database company (crazy people like that seem to be quite popular in SV these days) they could make a lot of money forking MonetDB/XQuery, and search-replaceing every instance of X with J and every instance of XML with JSON. Crazier still, you can even let MonetDB/XQuery results be formatted as json, even before MongoDB became popular I think.
It's easy to say that in hindsight, but given all of the tension around the Iron Curtain and all of the times the US and USSR came very close indeed to going to war despite how likely total nuclear destruction was, it seems short-sighted to just brush off the idea of that war actually happening if neither side had any nuclear weapons.
History isn't over yet. The risk is probably higher now than at any time since it peaked in the mid-80s.
Ultimately, one naval officer's minority vote[1] prevented a Soviet sub from launching a tactical nuclear weapon at US shipping. Several thousand miles of ocean prevented the Soviet Union from feasibly achieving decisive military victory against the United States. [1]https://en.wikipedia.org/wiki/Vasili_Arkhipov
For their part, the Soviets were very worried about a combined NATO invasion, which seems quite reasonable considering that Germany alone, while also fighting/occupying most of the rest of Europe, managed to get to the suburbs of Moscow, and kill millions of Russians along the way.
Now if the Soviets did manage to invade and conquer Western Europe and maintain control over it, then they might be in a rather better position to consider direct attacks against the US. Which is a good part of the reason why the US invested so much in keeping them out of Western Europe.
It didn't prevent the superpowers from having wars with each other. It just shifted the battle fronts, so now the superpowers fight through proxy wars, and the deaths all happen in distant, "developing" countries.
By that logic, we can 'never know' anything about historical effects, ever, and it's pointless to discuss the consequences of any historical events, including the others you mentioned. But really, the effect of the nuclear bomb in shifting the battlefronts of wars to proxy wars in developing nations under hegemonic control is as clear and as accepted by historians as we can expect.
It may not be quite on par with the "theory" of evolution being all-but-proven fact, but it's in that ballpark.
The smallpox vaccine is a strong contender. Smallpox killed 300m people in the 20th century, more than all wars combined. And we wiped it out.
" In the late 1940s and early 1950s, Leslie Collier, an English microbiologist working at the Lister Institute of Preventive Medicine, developed a method for producing a heat-stable freeze-dried vaccine in powdered form.[50][51] Collier added 0.5% phenol to the vaccine to reduce the number of bacterial contaminants but the key stage was to add 5% peptone to the liquid vaccine before it was dispensed into ampoules. This protected the virus during the freeze drying process. After drying the ampoules were sealed under nitrogen. Like other vaccines, once reconstituted it became ineffective after 1–2 days at ambient temperatures. However, the dried vaccine was 100% effective when reconstituted after 6 months storage at 37 °C (99 °F) allowing it to be transported to, and stored in, remote tropical areas. Collier's method was increasingly used and, with minor modifications, became the standard for vaccine production adopted by the WHO Smallpox Eradication Unit when it initiated its global Smallpox Eradication Campaign in 1967 " -- https://en.wikipedia.org/wiki/Smallpox_vaccine
That speaker was right: don't underestimate the impact of bicycles.
I agree with you about the Apple and XML ones though.
In practice, the ultimate form of today's smartphone was starting to take shape in secret in Cupertino pretty much as he wrote his article, the now ubiquitous design went on sale within 2 and a half years, the App store came the following year and Apple alone had already shipped 30M units before the "at least" 5 years were up. The release of the iPad also took less than six years.
Yegge didn't see significant enough innovations taking place within 5 or so years to make an impact. In fact, all the innovations that lead to over a billion people owning smartphones today took place in that time frame, with only incremental improvements since then.
I think it's pretty generous of Steve to give himself more than half marks for that!
I somehow feel that this would depend on the decision you made.
Like he said, his specificity screwed him (laptops, xml, etc) - He's totally not being too hard on himself. He's being fair and honest in his evaluation.
It would be super easy to predict trends if you are nonspecific: I predict new and shiny will win in 10 years. Bleh that's a non-statement.
Of course someone can. You can do whatever you want too. Predictions are hard and any correlation in 11 years is impressive, as are these. The question raised is not "is this fair or accurate according to my standards" but moreso, "who cares?"
Otherwise known as the Nostradamus principle.
"a new internet community hangout will appear" is not really a prediction. People have been "hanging out" online since BBS's and AOL. Would anyone bet against the same prediction looking from today forward?
The only other one where he gave himself full marks is "someone will make a lot of money by hosting open-source web applications." It's true of course, but PaaS is not specific to open source applications. If you take away the open source part, what he got right was that someone would make money by hosting webapps. And that's been happening in some form since at least the early 2000's when AWS launched.
That probably sounds more critical than I meant it. My hats off to him hitting a lot of them pretty well. But I wouldn't say he is going easy on himself with the scoring.
The way you hosted a website in 2004 was that you signed up with a web hosting provider and either rented a shared webhost for $10/month or paid for a dedicated server for about $60/month. There were little or no provisions for being able to grow your hosting needs at the click of a button. When you read stories of how eg. HotOrNot scaled, it involved a lot of panicked calls to investors, and on-the-phone negotiations with hosting providers where they were like "We need more servers now, we'll make sure we can get you the money within 30 days", often followed by a tech going down to the datacenter and physically installing a new server in the colo cage.
The other revolutionary part of his prediction is "Someone will make a lot of money". ISPs & hosting in 2004 were a very competitive commodity business, filled with overcapacity from the dot-com boom.
He did work at Amazon until 2005, so he likely saw the internal development that resulted in what would become the publicly-available AWS. And some of his other predictions strike me as being heavily influenced by what he saw working at Amazon. Companies like Amazon and Google tend to encounter problems a few years before the rest of the world realizes that those are even concerns. So making predictions about the industry at large based on the cutting edge technologies used inside those companies is actually a pretty reasonable assumption.
For example, according to Wikipedia, Dynamo and BigTable were developed in 2004. So while NoSQL would have seemed revolutionary to almost everyone outside of Amazon and Google in 2004, to people working inside those companies, that prediction was basically predicting that their ideas would work and become very successful. I'd also bet there were a ton of Google engineers that saw Map Reduce before it was made public that could have predicted in 2003 that computationally-heavy tasks would be primarily distributed in a few years. And there's probably a bunch of Intel employees who could have predicted in 2004 that the chips Intel would be selling a decade later would run at the same GHz as their current offering since 2005 was about the time that they switched to adding more cores.
When you're ahead of the curve, you can see what's coming around the corner before the rest of us.
I could quite happily interpret that as "schemaless databases (using the most popular vehicle for representing loosely-structured data) will surpass relational databases in popularity by 2011". I'll take marks off for 2011 being too early, but not for the datatype choice.
It's unfortunate that he went into more detail by describing XPath and current support for XML operations, but I feel that's just justification for the specificity of the datatype choice in the title.
I totally agree with you regarding laptops and apple though. "New and shiny" is hardly a prediction, and the specificity is relevant to the prediction.
I would. If he had said something like "XML is most popular... but really meant for documents not data, so something similar will probably replace XML" then I would have given full marks, even if he didn't specify JSON or BSON explicitly.
(I would give ~50% though, not 20%)
This is exactly why all futurists are full of shit (see Kurzweil) and predicting the future is impossible. Getting some aspects of your prediction right is virtually guaranteed if you generalize when you re-interpret later.
You'd have to make spectacularly bad predictions that are overly detailed and specific NOT to do half decently if we're going to allow post-hoc retrofitting of which broad trends kind of fit the original prediction.
That's just the bubble you're living in talking, as in most of the world, iPad sales have been declining, with the Mac revenue surpassing them again. And besides the iPad, the industry for tablets is basically non-existent. Really, our world is weirder and weirder.
... I'm sure someone will let me know, probably by tying a nice note to a nice red brick and hurling it through my window. You should see the email I get sometimes.
I think you can estimate the maturity of a culture by how easy it is to commit heresy. There is simply no excuse for how people reacted to his writing. It reflects poorly on all of us.
Maybe Alan Kay is right that we (the IT industry) are a pop culture, complete with our own sort of Kanye West.
It must be the combination of interesting and unique ideas with a really entertaining writing style that got me. Maybe it even had something to do with the years he was writing (I was in high-school & college at the time).
I don't really think Yegge is going to get back into it (and I don't blame him), I just hope to find someone else with similar qualities.
I think it was in a Stackoverflow podcast where he said that the hateful feedback he got was the main reason he stopped writing. It sad that we as a community did not get more of that content because of anonymous haters on the internet.
(Pretty similar to Joel Spolsky actually. Both in quality of content and hateful feedback in return.)
Wait, what is this based on? Hacker News postings? There is no way that NoSQL has surpassed RDBMS even among startups.
I'd probably measure by how much programmer time is spent working in each kind of DB.
Which means that you're missing things like caching engines, such as memcached, and other services which are not replacing an RDBMS but filling a different niche, which are widely deployed in industry.
Is that a prevalent opinion? I'm often in #clojure/#clojurescript and found the community quite helpful.
> Unfortunately I've conflated saying Yes to features vs. saying yes to users. Saying yes to features is clearly a way to get a kitchen sink a la C++ or Perl.
I didn't see any instances of him insulting people, though he was hostile in the same way he accused the clojure community of being hostile. But the points he made seemed sound to me. Saying "no" to users can seem hostile to the person you're saying no to. You need users, or you die. So say yes and survive.
Speaking of scala, you could also use it as an example of what happens when you say yes too much. In fact, to me, that's scala's major weakness. Sure, users can use it like a better java, but you aren't gaining as much from doing that as you would if you actually adopted a more functional style of programming. This is exactly what clojure encourages. I wouldn't count promoting this philosophy as any more hostile than any other opinionated language.
It certainly sounds like Clojure is not for Yegge, but that's not a valid reason to call the community hostile and claim that it's a dying language. Especially so because it's the complete opposite of what is true. The community is very helpful and welcoming to newcomers and the usage is growing, not declining.
He said that being hostile to users is the quickest way to kill a language. Now I don't know much about the Clojure language or community, but that post from Yegge was over 4 years old. You say that it's growing and I'll take you at your word, but had it reached critical mass at 2011?
Yegge is a languages guy. He knows some of what it takes to make a language successful, and one of the main things is advertising. Being hostile to users is the worst possible form of advertisement. That was his critique 4 years ago.
Maybe things have changed. Maybe the hostility wasn't as bad as what he thought, I don't know. But the comments in that thread aren't hyperbolic if you're associating them with the state of things today.
I'm not sure on the relative prevalence but I can say that I've read relatively little about what he's calling multi-threaded programming. Mostly just a handful of lock-free data structures from the C++ guys and Rust's take on how they can do it while avoiding data races due to the type system. The alternatives, on the other hand, have been on HN pretty much continuously since HN started up.
In 2004 single core computers were still quite prevalent, and the idea of steady significant performance improvements coming from increasing clock speeds was in its death throes. From memory it was about then that Intel had given up on the P4 Netburst architecture and stepped back to improving the P3 to release the new Core and Core Duo architectures.
Nowadays multicore has been reality for a long time and the number of cores in a CPU has increased a lot, and we're thinking much more about different competing ways of achieving better concurrency. Back then it was just about getting better multicore concurrency at all using bog standard threads.
If you look at some languages / frameworks I see now this: If you are programming now in C# you have keywords like "await" which make background processes much easiert.
When you are programming Android in AndroidStudio you are warned if you try to call a web service in a GUI thread.
15 years ago, when I've started programming commercially (MFC and C++) most of the books said something along "well this is multi-threaded programming, but 'There be dragons'.."
https://plus.google.com/110981030061712822816/posts/KaSKeg4v...
which basically boiled down to an impression drawn from a conference where a speaker warned against using macros. I fully accept Yegge doesn't dig clojure, and perhaps he owes nobody any further explanation, but given his previous visibility as a champion of lisp and all the cool stuff happening in clojure / clojurescript, it just doesn't add up; his critiques are incomplete in that they fail to acknowledge the positives in the language.
Can anyone find a source for this claim? The 2 I found [1][2] claim OS X has about an 8% market share, which is about 40% lower than his claim.
[1] https://en.wikipedia.org/wiki/Usage_share_of_operating_syste... [2] http://appleinsider.com/articles/15/07/09/new-idc-numbers-sh...
"Apple's laptop sales will exceed those of HP/Compaq, IBM, Dell and Gateway combined by 2010"
So, the prediction mentions "laptop sales", but in his grading, he talks about "Apple Macs only have 13% market share", which I would assume means unit market share? Given the profit share vs market share dichotomy seen in smartphones in particular, I'm left wondering if he isn't a bit too harsh on himself here. It seems likely to me that sales-based market share would substantially exeed unit market share for Apple Macs; surely Asymco or some other smart analyst has run the numbers on this?
Number of people is much more relevant and interesting.
7) The mobile/wireless/handheld market is still at least 5 years out. Nope, it was 3 years out. iPhone came out in 2007. Pretty easy to calculate this one. Score: 0.6
Or Nokia Commumicators from 1996 onwards. https://en.wikipedia.org/wiki/Nokia_Communicator
The whole point, which Steve himself covers eloquently in his original post, is not to score the predictions but to talk about the ideas. And if you look at intent behind each prediction, some of those "partially-correct" ones are actually completely wrong.
I mean look at these lines on the prediction about the mobile market being at least five years out: "But the carriers don't know what the hell they're doing, and the content and distribution models aren't worked out yet, and won't be for a long time. A lot of players need to get their collective acts together for the mobile market to take off, and it just hasn't been happening. [...] Don't get me wrong: it will happen someday. But I think it's not going to be any time soon. Betcha!" Then Apple told the carriers what they were doing, and figured out "content and distribution models", and this all happened within three years, and the reason it happened so quickly is because Apple didn't rely on any of the large set of third parties and vested interests which Yegge thought someone was going to have to motivate in order to make progress.
That prediction was apparently 60% correct. It's doublethink.
NBL?
Next Big Language
VR has the potential to become a big and interesting market and one of the leading tech guys in that field is fascinated by a Lisp. He might just make it work.
[note: I've never used Lisp for anything just an obvervation]
Lisp has some really compelling properties as means of expressing a program.
Unfortunately there is a severe impedance mismatch between Lisp and the Algol inspired languages of today's mainstream. Overcoming the mismatch is not difficult but takes effort and so few are willing to expel any real effort; Carmack is certainly not the sort to shy away from some effort.
I would recommend trying out Racket or Common Lisp if you can. Common Lisp really need Emacs so if you are not familiar with the editor I would recommend the former.
The next "culture shock", though I think this has lessened now due to other languages catching up, is the concept of closures, functions as values, and higher order functions.
Lisps also tend to be dynamically type so if that is not familiar to you then it will take some getting used to.
There are other high level concepts such has syntax abstractions (macros) but that can wait.
And Common Lisp has CLOS which is a different take, compared to the likes of C++ and Java, on Object Oriented programming.
As for what to do first; I am not really sure, depends on whether it's Racket or Common Lisp and a lot on what interests you. If you like doing small algorithmic problems such as the Euler Project then start there.
For Common Lisp there is Practical Common Lisp which is always a good primer.
http://www.gigamonkeys.com/book/
And Learn Lisp the Hard Way (I am not familiar with it though)
http://learnlispthehardway.org/
For a Common Lisp implementation I would recommend SBCL if on Linux (your package manager should have it) and CCL on Windows or MacOS.
SBCL: http://www.sbcl.org/
Also use quicklisp for grabbing Common Lisp projects, it's alot easier than doing it manually.
https://www.quicklisp.org/beta/
This might also be useful:
http://eudoxia.me/article/common-lisp-sotu-2015/#specific-pr...
Someone else will have to give info on the Racket ecosystem I am not familiar with that.
Eh, dynamically typed languages like JavaScript, Python, Ruby, PHP and Perl are rather popular.
I think what the Lisp world needs is a huge push from one of the big players. Not just an in-house, small-ish tool, but rather a publicly-facing bet on it, as dramatic as Facebook's bet on javascript with reactjs, or Twitter's move to Scala. After a 15 minutes survey I couldn't find an example of a big, respected company whose tool chain has a substantial Lisp project for other people to contribute to. Sure you can find multiple companies building Lisp solutions, but those rarely open source them and even when they do they are no where near as substantial to warrant massive public attention.
I sincerely think that this the only thing standing between Lisp and universal adoption.
https://plus.google.com/+MichelAlexandreSalim/posts/2vhp2GE9...
Yeah, not really [1].
In the first two months, iPhone sales were slacking so much that Apple suddenly dropped the price by $200 after just two months on the market: http://www.nytimes.com/2007/09/07/technology/07apple.html?_r...
The 3GS was when it exploded but it was already a big success by that point.
What people have to remember is the success of the iPhone is practically unprecedented in terms of adoption. It's incredibly rare to come out and dominate an industry like that when you're a new entrant into the space.
Even if the iPhone had continued it's July-September sales curve it would have been a success. It just got even more-so when the other versions came out.
The Nokia N95 sold 7 million units in a year after launch, pretty much exactly the same numbers as the original IPhone.
Unless you are declaring the N95 a wild crazy success you are talking rose tinted nonsense.
Also - either selling 25% of the U.S. market is not a success to you or you don't understand the market.
If you bought it unlocked (or if you still do) it was the same price. It sucked a little for Early Adopters (like me) but it was completely predictable and wasn't honestly a huge surprise to anyone paying attention.
By the time the price drop happened they had sold over 25% of the smartphones that were bought in the previous quarter. They'd already met their goal of 1 million smartphones by the end of September at that point.
I swear... the rubbish I see over this: the iPhone was a wild success considering that Apple had zero involvement in that business before. It was the kind of success that many companies dream about for new hardware.
Smartphone sales in the US in the first quarter the iPhone was released (July-September) were ~4.2 million units, Apple sold 1.12 million iPhones that quarter. In other words - the iPhone had ~27% of the US Market sales.
That's... not a rounding error. That's an incredible and wild success. The price drop after September helped of course - but they'd already sold over 1 million iPhones by then anyhow.
By the end of the year they had a 5% market share worldwide. Nothing to sniff at for something only 6 months old.
It took the 3G to change the world.
I mean it's like complaining that Aston Martin is a failure in car market because Chevrolet sells more cars.
When the iPhone launched the "smart phone market" didn't exist - there were a couple of other smart phones. It was a incredibly niche category.
Also, I think "Homoiconicity isn't the point" misses the point. Homoiconicity isn't about syntax - indeed, Lisp isn't technically confined to s-expressions, and was originally designed for M-expressions before folks realized that s-expressions were much easier to implement - but rather about the ability to represent a program as ordinary data and to manipulate it as such. Yeah, I can do all sorts of metaprogramming in, say, Ruby or Perl, but you won't be able to go very far before you end up passing strings into `eval`; this is because they're not homoiconic (at least in their current forms). Meanwhile, the ability of languages like Elixir to actually modify a program's syntax tree directly - a.k.a. homoiconicity - makes metaprogramming much easier. The two concepts go hand in hand.
S-expressions happen to be very convenient for defining tree-like structures, like - for example - Lisp programs. They're not strictly required, however.
I liked using PyCharm and WebStorm and I love emacs and I've been using Atom lately, but the convenience of an always running virtual machine with a pretty good text editor is hard to beat.
You say, for example, "I want to run Wordpress, I'll pay Heroku to do it for me."
Pivotal is on track for $100 million in revenue this year, the fastest growing revenue for any open source product in history. Ever. In any category.
Disclaimer: I work in Pivotal Labs, another division in Pivotal.
Steve, what are your next 10 predictions? Obviously, you're at least 10 years ahead of me...
I'm curious as to what this means. In what way is clojure hostile? Is there some reason I should avoid it?
What do you use? Why don't use Lisp anymore?
The early-to-mid 2000s were an exciting time for Lisp. Practical Common Lisp (http://gigamonkeys.com/book/) was coming out; Paul Graham was writing about how Lisp was a secret weapon; reddit was based on it. It seemed like the stars had finally aligned.
But then…nothing happened. Ruby captured a huge amount of the hype cycle, Python captured a huge amount of the pragmatists, and…nothing happened. The Common Lisp community kept on churning out incredible code, building a better mousetrap and waiting for the world to beat a path to its door and…nothing happened. It really is dispiriting to reflect on.
Error Prone == Evil
far more than programmers.
For the record, I was heavily involved with XML years ago and helped create a popular light weight xml-based cms.