rusanu

To put this into context I would recommend reading 'MonetDB/X100: Hyper-Pipelining Query Execution' [0]. Vectorized execution has been sort of an open secret in database industry for quite some time now.

For me, is particularly interesting reading the Spark achievements. I was part of the similar Hive effort (the Stinger initiative [1]) and I contributed some parts of the Hive vectorized execution [2]. I see the same solution that applied to Hive now applies to Spark:

- move to a columnar, highly compressed storage format (Parquet, for Hive it was ORC)

- implement a vectorized execution engine

- code generation instead of plan interpretation. This is particularly interesting for me because for Hive this was discussed then and actually not adopted (ORC and vectorized execution had, justifiably, bigger priority).

Looking at the numbers presented in OP, it looks very nice. Aggregates, Filters, Sort, Scan ('decoding') show big improvement (I would expected these, is exactly what vectorized execution is best at). I like that Hash-Join also shows significant improvement, is obvious their implementation is better than the HIVE-4850 I did, of which I'm not too proud. The SM/SMB join is not affected, no surprise there.

I would like to see a separation of how much of the improvement comes from vectorization vs. how much from code generation. I get the feeling that the way they did it these cannot be separated. I think there is no vectorized plan/operators to compare against the code generation, they implemented both simultaneously. I'm speculating, but I guess the new whole-stage code generation it generates vectorized code, so there is no vectorized execution w/o code generation.

All in all, congrats to the DataBricks team. This will have a big impact.

[0] http://oai.cwi.nl/oai/asset/16497/16497B.pdf [1] http://hortonworks.com/blog/100x-faster-hive/ [2] https://issues.apache.org/jira/browse/HIVE-4160

Can't rly answer since I have never ever tried an audio book, but I can tell that imho there are important visuals every 4-5 pages or so.

I have a question for the HN molecular biology aficionados: Where can I find some good critique of Nick Lane's ideas? He sure convinced me, but I would like to see what the experts say about it.

Oxygen, a previous book from Nick, cover these topics in great detail. The Vital Question deals with events that long predate the cyanobacteria, the Great Oxygenation Event and how we avoided oceans evaporating into space under UV radiation (at least the first chapters of the book, rest of it covers mostly origin of eukaryotic cell).

I have read The Vital Question, after hearing about it from BillG's blog. And it did blew me away. I immediately read also Oxygen, by same author, and I found it equally interesting (it goes in more detail on some topics, but some of the ideas in Oxygen are superseded by his later books).

If you can spare an hour, I recommend this video: https://www.youtube.com/watch?v=UGxAB4Weq0U . Is made by the author and it covers the ideas of life origin exposed in The Vital Question.

Also interesting his paper on the double-agent theory of aging: http://www.nick-lane.net/double-agent%20theory.pdf

video: https://www.youtube.com/watch?v=oMnK_llHoK8

> Disney ceases console game publishing, roughly 300 jobs cut.

You mean Bf109 I think. The P-51 started 3 and then moved to 4. Ditto P-47. FW-190 and F6F stayed at 3 whole time afaik. Tempest started with 4. So it varied. But there is a trend of the early (1939) designs to have 3 and the late (1944) designs to have 4. I guess available power from the engine plays a big role, as engines got more powerful they started fitting 4 blades. Using the extra power on 3 blades would require longer blades (which is problematic), spinning them faster (again, problematic as tips reach sound barrier) or increasing the pitch (I guess there are aerodynamic limits for that).

Case in point: (much of the) $100M donated to schools in Newark NJ evaporated into union contracts: http://www.businessinsider.com/mark-zuckerbergs-failed-100-m... Sure, it benefited the teachers and perhaps many deservingly so, but it failed to make the expected dent into the system.

TFA is not strictly about the fact that there is a tax break, but about a great increase of the break in recent years and some possible shenanigans vis-a-vis some Nevada operations. My reply was driven by the comments here that why breaks exists in the first place, not strictly to the situation presented in the article (of which I understand little to tel you the truth).

This video shows that from times immemorial local authorities tried to attract business: https://www.youtube.com/watch?v=hExE6iLxfEw

States/Cities offer tax breaks to attract/keep the business in the first place. The Pacific NW city wanted to become a tech city and offered incentives (=tax cuts) many years ago for companies like MS to settle here. Successful companies bootstrapped satellite dependent businesses (the myriad small IT shops in area, like SourceDynamics Inc) and attracted in the area subsidiaries from big SV (FB, Google etc). The affluent workforce they foster spends their disposable income in the area and this benefits all other businesses in the area. And both MS and Boeing can decide to move to other locations, taking with them huge swat of tax paying employees and a myriad of satellite smaller businesses that depend on the 'mother ship'.

Here is a simple thought exercise: salaries -> area desirability -> high property taxes -> high budget.

I don't think this is a case of corrupt WA politicians being in the pocket of Big Business Co and writing off tax breaks for kickbacks. I think is a calculated deal in which tough negotiators for both sides reach an agreement benefiting both the state and the business. At 83.4 billion $ budget ([0]) WA state does not seem to get the short end of the stick...

[0] https://en.wikipedia.org/wiki/List_of_U.S._state_budgets

Perhaps one day I will learn how to write HN clickable URLs...

There is a fundamental difference between code deployment and DB upgrades: DB upgrades must preserve (and upgrade) the existing state. Code deployment only needs to ensure that the new code is in place. It can do so by deploying to a new location and then redirecting the entry points, or it can do so by laying over the new code. Either way, the ways ways in which it can go wrong are few (at least compared to DB upgrades). DB upgrades, on the other hand, must take existing state (which can measure GBs and TBs) and transform it into a new state. If you're really unlucky, this may involve a size-of-data transformation (eg. update every row in a big table). I've seen 'upgrades' that lasted weeks. Having witnessed DB upgrades at MS Azure scale (see [0]), and having to code myself several SQL DB upgrade steps (internal system upgrades, the kind done when you install new DB engine, not user upgrades) I can say with confidence that DB upgrades are hard.

What we all end up with are either DB migration steps (a-la Rails migrations), of which I approve see [1], or schema comparison tools (a-la vsdbcmd.exe) of which I'm very skeptical after being burned several times on large DBs (they are as good as the tool, and some tools are better, still I find explicit migrations much better).

As a side note, My startup DBHistory.com is recording all DB schema changes, like a DB flight-recorder of sorts, and one of my future goals is to add capability to generate compensating actions for any DB schema change and thus be able to revert the DB schema to any previous state (by simply rolling back every change, in reverse order). But I must reckon I'm quite far from having such a thing working, and I'm not even considering migrations that modify data, not schema.

    [0] https://medium.com/@rusanu/why-i-have-high-hopes-for-the-quality-of-sql-server-2016-release-6173bc1fbc82#.wp1zt5pn0
    [1] http://rusanu.com/2009/05/15/version-control-and-your-database/

I don't like this because it leads to the misleading assumption that it doesn't matter which value you refer (left or right), since they are equal. But once you consider collations is easy to see the fallacy:

   left.id = 'Something' COLLATE case insensitive
   right.id = 'SOMETHING' COLLATE case insensitive

   SELECT id FROM left JOIN right USING (id);

Will the value be 'Something' or 'SOMETHING'? Better avoid the surprise and be specific...

Thanks, I will

I worked +10 years in Redmond, WA as a microsoftie and came back to Romania in 2012. Since March 2016 I quit Microsoft and focused 100% on my startup, DBHistory.com. I now rent space and work at a local tech incubator www.techhub.com. I wanted to start with this to to put my opinion in perspective. While the article is bubble-gum and is clear PR, the point it makes is a valid one. Compared even to US, today Romania has a very active entrepreneurship mindset. Small businesses are created all the time, some thrive, some go under. Despite the locals skepticism, I see the economy here as effervescent, with a growth rate still north of 3% (for Europe, this is not bad). The overwhelming majority of this entrepreneurship is not in IT.

But this article is specific to IT, and I do not see the Romanian IT anywhere near its potential. This is because the local IT is almost entirely focused on outsourcing, not on original development. Big companies have significant presence here (Electronic Arts, Adobe, Intel, Microsoft each have ~200,300 dev offices here, although not everything is core R&D ). Original Romania IT has a very strong presence in anti-virus (for some reason it seems a lot of local IT youth are experts on exploits...). Many young IT Romanians choose to go abroad (in Microsoft Romanians are the 4th largest foreign community, after Indians/Chinese/Russians).

Since I've returned I've been approached several times by US/Western Europe based entrepreneurs who desired to move development to Romania and open office here, solely for cost reasons. Purely for outsourcing, this is my 2c:

- Romania has significant talent pool

- EU membership (with all tax/export/legislation alignment that entails)

- Good spoken English is pretty much universal, German/French are common

- Political stability (governing parties rotated power several times in past 20 years, peacefully)

- Cost is higher than one would expect. Bucharest/Cluj/Timisoara all have high cost (office) and salaries are higher than some neighbors. If the outsourcing decision is purely cost driven then Ukraine has a better deal for you, with a similar talent pool.

But I would like to see a lot more IT original development coming out from Romania. The article gives some examples, like LiveRail, but overall I see a lot more outsourcing done here than original development. I'm glad projects like cloudhero.io originate here.

Presumably one would catch such a blatant error early in testing. The problem is the odd case, the code that only crashes in a rare state, never seen in test. When the odd state does occur the effect is mild, as only one actor dies as opposed to the entire OS process.

Article is correct, only the HN title has the s

What makes you sure? Central nervous system neurons are incapable of regeneration[0], so is anybody's guess what would happen if they did get replaced. And, long term, "significant" has to be "all".

[0] https://en.wikipedia.org/wiki/Neuroregeneration

Oxygen[0] by Nick Lane makes a pretty compelling argument why this will not happen. With perfect health you can at best achieve ~120 years age. I think this paper A unifying view of ageing and disease:the double-agent theory[2] covers some of the arguments in the book.

To give a simplified view, the big obstacles are a) a technology capable of repairing damaged mythocondrial material in a living organism and b) a technology capable of repairing neural cells w/o replacing them (that is if you want to preserver 'you', not just your body). That is assuming that our current understanding of senescence[1] is even mildly accurate.

[0] http://www.amazon.com/Oxygen-Molecule-World-Popular-Science/... [1] https://en.wikipedia.org/wiki/Senescence [2] http://www.nick-lane.net/double-agent%20theory.pdf

In Romania the law requires all retailers to display the maximum price markup they have. Dutifully every single retailer displays the same notice "This store practices price markups between 0 and 300%" (over 300% notices exists, but are rare).

And for the Cambrian Explosion, although somehow dated now, Gould's Wonderful Life https://en.wikipedia.org/wiki/Wonderful_Life_(book)

The book is excelent. Nick Lane also has many YouTube posted lectures that cover the topic. Eg. Alkaline hydrothermal Vent origin of life[0]

[0] https://www.youtube.com/watch?v=UGxAB4Weq0U

Wikipedia https://en.wikipedia.org/wiki/Cryptanalysis_of_the_Enigma also a lot of details

Enigma: The Battle for the Code[0] is a very good read. It details the Polish achievements, the rush to save the Polish cryptographers out of France as the it was collapsing in 1940, and then goes into details with the many more details that gone into the cracking effort, besides Turing's team. Eg. tt shows the importance of weather report patterns in finding cribs that short-circuited the decryption (known clear text headers) and the importance of the submarine captured Enigma machines that showed the disk settings in the Navy setup (the Navy machine was much harder to crack than the Army ones). Not only a good read, but also very educating for anyone using cryptography today!

[0] http://www.amazon.com/Enigma-Battle-Code-Hugh-Sebag-Montefio...

Reminds me of the 'leafs' from Anathem[0]. for sure none of the current digital storage artifacts will be readable in +500 years, like scrolls. Contiguous upgrade required by digital will always have a cost and only 'certain' items will end up surviving, where the criteria for choosing will vary with fashion, religion, politics. Wonder is archaeologist in 2500 CE will reveal "500 years old file discovered in digital archives" and then, year later, announce the succesful decoding of an original cca. 2016 animated cat GIF...

[0] https://en.wikipedia.org/wiki/Anathem

- Icelandic rocks could have steered Vikings (2011) [0] - 'Viking sunstone' found in shipwreck (2013) [1] - Crystals 'helped Viking sailors' (2007)

[0] http://www.bbc.com/news/science-environment-15523520 [1] http://www.bbc.com/news/world-europe-21693140 [2] http://news.bbc.co.uk/2/hi/europe/6338535.stm

"Consider this: Many commercial transactions now take place with some short-term credit attached — for example, a company that gets a 60-day grace period to pay bills from its suppliers. Would that flip, and suddenly suppliers would prohibit upfront payment and insist that their customers wait 60 days to pay?"

Betteridge's law of headlines should tell us the answer is "No"

TL/DR: "Apple's stock value dropped to below $100 for the first time since October 2014".

If you happen to target Windows platform (I know article does not), don't reinvent the wheel. There are NT Job Objects [1] for this. The 'target' process creates the job with `JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE` set and keeps the job handle open. 'dependend' processes (by default all children spawned by 'target', but you can add more) will all be killed when the 'target' process terminates, no matter how is terminated. Jobs can do more, like cap CPU/IO/Memory for the job.

[1] https://msdn.microsoft.com/en-us/library/windows/desktop/ms6...