daniel-levin

>> The Camry ETCS code was found to have 11,000 global variables. Barr described the code as “spaghetti.” Using the Cyclomatic Complexity metric, 67 functions were rated untestable (meaning they scored more than 50). The throttle angle function scored more than 100 (unmaintainable).

>> Toyota loosely followed the widely adopted MISRA-C coding rules but Barr’s group found 80,000 rule violations. Toyota's own internal standards make use of only 11 MISRA-C rules, and five of those were violated in the actual code. MISRA-C:1998, in effect when the code was originally written, has 93 required and 34 advisory rules. Toyota nailed six of them.

How the ACTUAL FUCK did this happen!? The article makes Toyota's engineering team seem egregiously irresponsible. Is it typical for vehicle control systems to be this complicated? I would love to hear the other side of the story (from Toyota's engineers). Maybe the MISRA-C industry standard practices are ridiculous, out of touch and impractical.

Unless of course the language is designed and optimised for it. For instance, Jane Street use OCaml - in which recursion is a standard language primitive - and they sometimes prove their mission critical code correct.

Your discussion has been hijacked by somebody posting /r/spacedicks type pictures.

Thank you very much for this!

How interesting. I would have thought such a common database operation (querying by ranges) would have been a better solved problem by now.

Also, how come the author of the blog post writes O(k) instead of O(1) for constant time? Is it because 1 is as arbitrary a constant as any or is there some difference that I am not aware of?

Link to the original paper [1]

[1] http://www.vldb.org/pvldb/vol6/p1714-kossmann.pdf

I must respectfully disagree that Haskell's memory footprint is simply 'low'. This is because the memory footprint of a given Haskell program is not at all transparent, and Haskell is notorious for leaking memory in a maddeningly opaque fashion [1, 2, 3, 4]. Space leaks might be relatively straightforward to diagnose and fix for a true domain expert, but I would not want to have to rely on someone having such abstruse knowledge in a production application. It goes without saying that a space leak in a production app is a really, really bad thing.

Although, I suppose one's choice of Haskell is a function of one's own risk/reward profile. Haskell and its failure modes are hard to understand. That induces extra risk that some people (myself included) might be uncomfortable with. That said, I am now enthusiastically following you guys and hope to see the proverbial averages get sorely beaten.

[1] http://neilmitchell.blogspot.com/2013/02/chasing-space-leak-...

[2] http://blog.ezyang.com/2011/05/calling-all-space-leaks/

[3] http://blog.ezyang.com/2011/05/space-leak-zoo/

[4] http://blog.ezyang.com/2011/05/anatomy-of-a-thunk-leak/

Since the original paper is behind a paywall (at least for me), can anyone explain the specifics of what the researchers did to produce this new alloy?

>> Dr Kim and his colleagues have, however, found that a fifth ingredient, nickel, overcomes this problem.

I'd imagine that it didn't take a world-class team of scientists to have come up with the idea of alloying using nickel. There is no way materials scientists and metallurgists hadn't tried this by now, so what did they do differently?

For interest's sake, note that the blur kernel used here is an approximation of the Gaussian [1]. Also, the vImage documentation includes a brief discussion on where the values in these kernels came from [2]

[1] http://en.wikipedia.org/wiki/Gaussian_blur

[2] https://developer.apple.com/library/ios/documentation/Perfor...

Hey Patrick! I'm a huge fan of your work, and really enjoy reading your blog. I have one question though.

AR is HIPAA compliant, which implies that there is (medically) sensitive information hitting your servers. Why is it not an issue for you and your support agents to actually see that data yourselves (as you would when manually fixing CSV errors)? If your seeing this data doesn't violate the letter of HIPAA, surely the ethical impetus behind the act would prevent you from doing so?

As someone who grew up writing code, and is now studying mathematics at a tertiary institution, I was quite surprised to read that parallels between mathematics and software engineering are 'surprising'.

On the contrary, mathematics has formed the basis (no pun intended) of so much software engineering. Take for example the very concept of a function/subroutine/method/ - this comes straight from the world of mathematics (albeit with minor modifications to make it convenient).

The algorithms that do all the heavy lifting in order to facilitate this web browsing experience are all grounded in mathematics - memory management in your kernel, database {everything}, even HTML layout management! The whole of complexity and asymptotic analysis is actually just mathematics.

Many of the pioneers in computer science originated as mathematicians. Alan Turing, John McCarthy and Donald Knuth for example.

It may not seem like it on a daily basis writing CRUD apps in an OO language, but software engineering is inextricably linked to mathematics. Such results are the furthest from surprising!

>> how many intro stats book, of the traditional kind, mention MLE, method of moments, biased vs unbiased estimators, etc...? None that I've seen

Oh - there are quite a few. Here's a small sample (no pun intended):

- Probability and Statistical Inference by Hogg & Tanis (we used this in my stats course)

- Modern Mathematical Statistics with Applications by Devore & Berk

- Probability and Statistics by DeGroot & Schervish

The UCI machine learning repository [1] contains a wealth of data sets intended to be used for machine learning. Many of the data sets have had analyses performed on them that could be considered canonical. For example, the Abalone dataset's [2] associated problem is the prediction of a specimen's age, given its measurements. The problem has been analysed thoroughly; a cursory Google search for "Abalone data set" reveals that plenty of people have considered the problem.

Also Amazon (via AWS) [3] have made it really easy to access public data sets.

I hope this is helpful.

[1] http://archive.ics.uci.edu/ml/

[2] http://archive.ics.uci.edu/ml/datasets/Abalone

[3] http://aws.amazon.com/public-data-sets/

This book interesting because it forgoes the traditional approach of most mathematical statistics books. The preface states that it is done like this in order to avoid the "cookbook" approach taken by many statistics students. This is why it is ironic that "Bayes' Recipe" appears 15 times in this text, and on page 131 there is a five step algorithm for parameter estimation, and my favourite, oft-repeated, never explained recipe - "n > 30, you'll be fine". There is no mention of the CLT, MLE, method of moments estimation, biasedness of estimators, convergence in probability, how sampling distributions arise, or any of the theory of distributions that underpin all of the inferential procedures detailed in the book. I think that excluding these topics actually increases the cookbooky-ness of the text.

It is important that students understand the provenance of the inferential techniques they use so that they don't land up doing bogus science (which hurts the world) by not knowing the failure modes of these techniques. Of course not all students of statistics know the requisite mathematics to understand it all, at the very least put the failure modes into a cookbook form.

For the sake of science please don't ever do any inferential statistics without knowing when the method you're using works and when it breaks, what it is robust to, and what assumptions it makes. Statistics is really easy to break when used naively. The mathematics of statistics is not easy, and often results are highly counter-intuitive.

Eli Bendersky's blog. It gets reposted on HN quite often and with very, very good reason. It's an absolute treasure trove of technicalia. I've spent many hours deep in his articles. On all sorts of cool technical topics, like parsers, debuggers, cool language features, abstract math...

And I'm sure I'm not the only frequenter of HN that loves this stuff.

I suppose the overarching principle here is communication between programmers. If I was the programmer building some system depending on an API with some opaque behaviour I'd get really frustrated: "Why does the connect method just not work sometimes and block??!!?".

It's just considerate to other human beings to let them know (using a suitable means) that an API call has failed (for whatever reason) and quickly. Opaque loop-and-retry-until-we-succeed makes problem diagnosis stupidly difficult. Anything that makes the audience programmer's feedback cycle slower and impedes problem diagnosis is both counter-productive and irritating. Simply communicating "I CAN FAIL AND HERE IS WHY..." is a Really Good Thing.

In my experience (read this with an 'anecdote' filter turned on) teams that communicate everything to the point of superfluity generally work better. This extends to your code, particularly APIs.

What did you do about proofreading bits of your manuscript that you couldn't break apart into randomised, individual units? How did you make sure all your punctuation and grammar was correct? For example, placing commas appropriately in a sentence that only a human can understand. Surely this is also part of the proofreading process?

I wonder if it's possible to design (and build) a digital sundial with arbitrarily many digits (the obvious upper bounds of 'arbitrary' imposed by our physical universe notwithstanding). It's an interesting thought experiment...

edit: spelling

Worked for me, and I live in South Africa. Maybe you left out the country code or got the format wrong?

This sounds very cool. If you're planning on open sourcing any and all of this I'd like to contribute

Of course. They're called expert systems [1]. Yesterday on HN someone posted a link to a book [2] on probabilistic models of cognition which includes a section [3] on medical diagnosis.

[1] http://en.wikipedia.org/wiki/Expert_system [2] https://probmods.org/ [3] https://probmods.org/conditioning.html#example-causal-infere...

The Massachusetts police have privatised part of their operations. And the 'general counsel for the Massachusetts Chiefs of Police Association' has said that they're immune to information requests because they're private corporations. The article positions the purpose of the privatisation as secrecy. This seems like a very Bad Thing, so it's not surprising that the ACLU have stepped in.

But I'm going to play devil's advocate, because I want to know the truth:

I wonder if there are other, possibly more important reasons for forming these private corporations. Maybe the existing police system doesn't function optimally, and provisioning resources (such as trained officers with appropriate equipment for drug busts) is a process that's too slow, or inadequate. Maybe this is a way of detouring the bureaucracy and systemic bullshit that encumbers civil servants whose job is ultimately to keep people safe.

The article says that Tewksbury, MA paid $4600 for membership to NEMLEC. That town has a population of around 28 000. It doesn't make sense for a small town like that to have a police force with a dedicated SWAT team, computer crimes unit and schools incident response team. It also doesn't make sense for a larger jurisdiction to serve the smaller community when it [the smaller community] needs it, and get nothing in return. It seems as though it's a way for police departments to share resources. I'd imagine that if a small town's police department found themselves unable to deal with a time-critical scenario, like a shooter in a school, they'd call in backup pretty damned quickly. Perhaps the quality of response a small town could get through NEMLEC would be better than going through traditional police channels?

To me it seems like the primary purpose of forming these private NPCs is not secrecy. As the article says, government police agencies already do that - "police agencies have broadly interpreted open records laws to allow them to turn down just about every request."

So, if it is actually easy for police agencies to turn down requests in the first place, why go to all the effort to form, finance, and manage a 3200 member [1] corporation?

[1] http://www.nemlec.com/who.htm

edit: clarified a sentence by adding 'through NEMLEC'

These sort of tips are only useful in highly specific cases. It's probably not worth learning them off by heart because of how infrequently you'll end up using them. BUT, understanding how the author(s) came up with these is incredibly powerful. The combinatorial explosion of the composition of Vim features means that it's practically impossible to learn all of the commands.

That said, it's really beneficial to learn how to form these commands yourself. For instance, if you know regexes, a good chunk of the commands presented would be relatively easy to come up with yourself. Is it worth learning regexes, Vim shortcuts (basically a lot of arcane things)? Maybe, maybe not. If you spend (or plan on spending) a huge amount of time manipulating text files it's probably worthwhile. For me at least, it's much more satisfying to do most text manipulations (e.g. remove trailing whitespace) by running a concise command, than doing it manually.

For what it's worth, it's even fun to sit and think of how you can avoid doing some manual task. Vim shortcuts + regexes are a really good way of avoiding silly work.

Seconded. I'd also like to get my hands on it.

Markov Chains are really cool. One of the many applications [0] being that you can 'train' them on a text corpus, and then by repeatedly generating random numbers, create sentences that are (mostly) gramatically sound but otherwise absolute nonsense [1], [2].

[0] http://en.wikipedia.org/wiki/Markov_chain#Applications

[1] http://en.wikipedia.org/wiki/Mark_V_Shaney

[2] http://kingjamesprogramming.tumblr.com/

Have you seen [0]? This is biological CAD. IMHO about as cool as it gets.

[0] http://www.genomecompiler.com/

I love this.

I clicked on the link, and being South African, I've been conditioned into thinking that all sorts of cool services/products from overseas either aren't available here or are prohibitively expensive to import. So, I was delighted when your page said, "Free shipping from Japan, even to South Africa". Thank you.

The article misses a very important thing Microsoft still does very well: makes money [0]. Irrespective of one's opinion on the quality of their products, they still produce software the provides value for real people. They employ a ton of people, all across the world [1]. Microsoft may have lost a decade in the visible consumer segment (mobile phones, tablets), but they still make boring, profitable, enterprise software that helps them on their way to $22B in annual profit. They're not shrinking either, according to their fast facts page [1]. Microsoft still does business very, very well. Even though I personally don't like most of their products, I'm genuinely excited by the change of CEO, because Microsoft has the resources (perhaps not the culture) to build awesome new technology in the next few years.

[0] http://www.nasdaq.com/symbol/msft

[1] https://www.microsoft.com/en-us/news/inside_ms.aspx

> Lenovo said it would settle the transaction with $2 billion in cash and the balance in its own Hong Kong-listed shares.

Now this is the interesting part of the article. This means that IBM will own $300 million in Lenovo stock. Sure, it may dump them for cash, but it now has a stake in Lenovo, the company that 'has overtaken HP and Dell to become the world’s biggest manufacturer of PCs.'

I don't mind using my real name online because it is stupidly generic (Daniel Levin). If someone wanted to identify me, my name wouldn't be of much help to them. This is interesting because my real name is less useful in being identifying than a screen name used on multiple websites

Vim-fireplace with Clojure is really awesome. Check it out [1]. It's written by Tim Pope, who is a very active developer of Vim related tooling. The setup is simpler than Emacs, and works out of the box (cider for Emacs gave me endless headaches [2]).

I recommend it highly because it shortens your code-run-feedback cycle dramatically.

[1] https://github.com/tpope/vim-fireplace [2] https://github.com/clojure-emacs/cider/issues