east2west

I submit computer science departments have enough expertise to operate a graduate data science program. Maybe not every CS department, but enough. It is interesting that the actual stats departments do not encompass all statistical knowledge and expertise. More specialized experts in econ, finance, many fields of social science, not to speak of all engineering fields and computer science and applied math. Data science predates modern statistics; Laplace used least-square to ignore "the God hypothesis;" the function-approximation view taken in many engineering fields is just as powerful as the statistical approach.

I agree with you that NYTimes piece is largely anecdotal but I will give my reasons for not believing a SVP of Amazon PR.

1. Personal attach on a former employee as the head of public relation for a multi-billion corporation from a personal blog seems cowardly. Did he give Olson a chance to respond? Is there extenuating circumstances? How did Amazon carry out the investigation? Who knew what when? He accused NYTimes lapses in journalistic standard; he isn't even pretending there is any standard.

2. His attack on reporters is unconvincing. His rhetorical questions sound conniving. The reporters might not have asked those exact questions but they must have examined the named sources' credibility and employment history. Amazon might have refused to give details about Olson's termination, ironically on employee's privacy concern, and then itself divulges specifics, though I am sure after being cleared by their legal department. Noticed that Carney went into hyperboles here rather than stating any facts. A few sentences describing Amazon's interaction with reporters would go a long way here. But no, nothing.

3. His refutation of other three named sources amounts to verbal parsing. Some parts of the review are good so no parts can be called bad. Written review is good so no verbal review can be abusive. No direct requirement so everything is employee's fault and they deserve everything they get. It is almost offensive to read this part. If this is the best Amazon can come up with, it must be bad.

4. Cheery picking Amazon's interaction with NYTimes' reporter. Select publication of correspondence and one-sided characterization of interaction is sly. Make public all correspondence, with proper permission, so we can get a full picture.

Fundementall Carney is a former reporter and whitehouse spokesman who is now attacking former employees in behalf of a giant corporation with enormous resources on personal blog. He is committing every bad act he accuses NYTimes of and a lot more. It is debatable whether NYTimes followed the ideal journalistic standards but it is clear Carney follows none here.

It is a familiar comment that Ph.D. holders cannot program. I have a Ph.D. and I dare say I can program. I was offered a position by Google recently so I am above some threshold. It is really unfair to paint a broad class of people with such a sweeping brush. I find that people often overestimate efforts required to attain a skill level, probably due to human nature to elevate one's own skills or importance, or they unconsciously conflate familiarity with general intelligence. It is impossible to reinvent collaborative filtering within 45 minutes without any prior exposure if not told any reason behind or constraints under which collaborative filtering was invented. Software engineers ask basic questions from the narrow fields they operate in day in and day out and believe that the interviewees have all requisite knowledge and answers should pop into their heads. On the contrary, basic questions are often fundamental innovations and it is no small feat to come up answers in the first place. It is a hallmark of a real intellectual to probe deeper and learn from that experience and realize insights and reasons for reaching innovative solutions, so that next time he or she will be innovative quicker or better.

It is hard to accurately judge someone in 45 minutes. I made a mistake with fizzbuzz three years ago when interviewing with the hacker school (now called recurse center), so what? It isn't I cannot do it but that I was looking for better ways and not finding it. After reading that there is no shorter way than to enumerate all four possibilities I was able to mentally unlock. It took just a few months of studying example code and discerning coding patterns for different logic control flows and now multiple programmers have commented that I write readable code. Anyone who has worked through real analysis problems can reason through most code well. It is a matter of concentration and time. I firmly believe programmers can make good ML engineers even though I spent years on numerical linear algebra, mathematical programming (optimization), statistics with measure theory and I still feel I have huge gaps in knowledge. So what if someone doesn't know a basic theorem, as long as they can learn enough and apply it to do their job. We take people as they are and help them grow into roles we want. I hope you will give everyone a fair chance.

Since big-O and other asymptotic notations come from calculus I don't think it is a bad idea to know precise definition. For programmers it is questionable how useful they are in everyday job and I would just forgo asking them during interviews, although I have been asked in recent interviews. These asymptotic notations are immensely helpful in finite approximation of continuous functions using infinite series like Taylor series or orthogonal polynomials. Computer scientists borrowed the notations and abused them as if they were the final arbitrator of things when they are not; Knuth noted too much game is being played with them in academic publication. Further attempts at drawing analogies different kinds of convergence and growth functions have not found widespread use.

I guess I am saying don't underestimate the importance of rigor in knowing precise definition of mathematical notations.

I don't use "quick-and-dirty" as a derogatory term, so mostly I mean no error checking for rare occurrences or not ensuring correct values are being passed through. An example would be in Python if I just write MatrixA * MatrixB without any checking. I should check that they are both matrices if that is what I want, and that is what I did for reusable code. For one-off data analysis or computation, no-checking is certainly quick.

I love Julia for its expressiveness but I am somewhat confused about its dynamic typing and its powerful type system. Are we to write mostly Python-like function and rely on multiple dispatch and JIT code generation for speed? Duck typing is one reason I stopped using Python for technical computing. I tried Julia out a few months ago and found that type inference has some holes that make type annotation brittle for function parameters. I finally gave up when I found out functions have no return type in Julia.

Hopefully Julia can work out all the language features and mature into a good technical language. It would also help if there is a guide about the canonical programming style. I prefer static typing but thought using type annotation I could come close to static typing while still retain the freedom of quick-and-dirty code. I believe for the current Julia I was wrong.

Does Java have any way of auto vectorization? On modern CPUs SIMD can speed up numerical computation several times easily. However auto vectorization is hard in general, so compilers don't necessarily do a good job and manually optimized code is common. C++ has TMP and libraries such as Eigen use TMP to provide a nice interface to SIMD optimized code. I don't know whether Java has equivalent of compiler intrinsics for SIMD instructions, without which Java code would be far slower for many numerical computation. I believe C# has plans to add an abstraction for SIMD and Mono can already emit SIMD instructions.

Matrix is not a convenient way to write linear transformation; it is a good way to reason about linear transformation, because among other things matrix product is function composition. Loose definition and incoherent reasoning are not good motivation for anything let alone a mathematical subject. Bringing matrix analysis as an introduction to linear algebra. For example, there are different ways of writing chain rule for matrices with radically different looking formulas; higher derivatives of vector functions are multilinear forms; convergence in multidimensinal space has too many subtleties for beginning students. Solving systems of linear equation is a perfect way of introducing linear algebra. Not only is it vitally important in math and used everywhere, it is also almost half of linear algebra. Gaussian elimination is the elementary school method formulated as an algorithm and, not coincidentally a sequence of linear transformations culminating in LU decomposition because we can represent a sequence of linear transformation as a series of matrix products. Writing Gaussian elimination in partitioned matrix form, which includes regular scalar manipulation as a special case, results in Schur's complement, and taking inverse of that is a good way to deductively prove the matrix inversion lemma, an important formula in numerical analysis and signal processing and usually taught in advanced courses. Nothing I mentioned is inaccessible to an undergraduate. Linear algebra is an unique mathematical subject because it doesn't require much basics to begin and is very visual, unlike calculus which needs a good foundation in analytic geometry and has many techniques what beginning students call tricks, not to mention mental obstacles like limits. Mathematical intuition comes with practice and experience and mathematical maturity, not dumbing down the topic. I agree motivations should be taught along with any mathematical subjects, but they should be mathematical motivations. Applications are important too, but teaching well and thoroughly theory and application at the same time is impossible and undesirable; different people need different applications which in turn emphasize different aspects of theory; it is just too much.

I am considering converting a C++03 math library to C++14 as a side project to learn C++14 and I examined Eigen and Blaze. Eigen's code size seems to be a fraction of Blaze, even though their functionalities are similar. Eigen also has some design documents while Blaze has papers but not much more. It seems I will try my hands on Eigen library for now. It is amazing that a couple of people could do in a few years; Blaze has hundred of thousands of lines of code.

That sounds like standard affairs. I read your interview experience, it is almost funny. Amazon has a body-shop quality to it. Once someone told me he failed an interview, then changed his email address and phone number and got another interview and failed again, and on his third try he finally got an offer. I don't remember whether he accepted the job, probably did. He is a foreigner needing visa sponsorship, so he would be gone after a couple of years.

I have heard repeatedly that potential workers are asked behavioral questions during their interviews about the leadership principles. I wouldn't call that "taken with a grain of salt." Anecdotally their turnover rate is high and a larger proportion of tech workers have to be on call than other large tech companies-- this could be due to the nature of tech, clouds and probably the largest web portable on earth. Different divisions are going to be different, and the article focuses on more extreme cases, true, but Amazon is a "tough" place to work, no question about that, at least compared to Google or Netflix.

If Amazon is not a nice place to work, honest people writing good journalistic reporting on it are going to write negative stories, no way around that. As negative stories, this particular article isn't bad. At least it puts a leadership-principle spin on it.

Amazon does just have stack ranking, the company basks in it.

I have always suspected so since Google still does not open source its MapReduce, but I haven't found any actual evidence directly comparing Hadoop MapReduce and Google's. There could also be technical issues preventing Google from open sourcing its implementation, although I don't know any. Now that we are moving on to more sophisticated models and implementation I am hoping more details can emerge.

Some clarifications. Explicitly computing matrix inversion is never a good idea, unless you require an inverted matrix. For solving the normal equation, indeed for solving any linear equation, use appropriate specific algorithms (variants of Gaussian elimination, QR, SVD, ...). For regression, computing the inverse of covariance matrix is actually faster than QR methods ( plural because there are more than one) at the cost of numerical instability. SVD is the more stable but also more expensive method. These are the standard methods you will find in any linear algebra or statistics textbooks. For regular use they are just fine. It is when you have thousands of variables and tens of thousands of measurements when they fall short. Iterative optimization algorithms can be used, or random matrix algorithms. There is no free lunch, because numerical instability is inherent in these algorithms, but you do have some, more relaxed, guarantee of errors or probability of errors.

The reason I said iterative optimization algorithms can be used for regression because linear regression is a quadratic optimization problem. We have analytic solutions because of special linear algebra properties. It can be said that we have a good handle on quadratic optimization problems, hence the multitude of algorithm choices. General optimization algorithms necessarily carry cost in numerical instability, but it may be a cost worth paying, especially for large data sets.

Finally, I want to echo your sentiment that DO NOT invert. Solve linear equations should be by tailored algorithms. In theory they yield identical results, in practice specific algorithms are much better.

I know you are being rhetorical but even New York Times would agree the Greek government is more corrupt than the German government. Some level of corruption is inevitable in every ruling class; an old Chinese saying is that pure water has no fish: no one will work in a graft-free institution. Your rhetorics is not helpful at the current critical juncture; cooler heads must prevail. It is far more important to consider whether the existing economic, financial, and political arrangement is sustainable and whether Greeks may be better off with a clean slate. A massive European aid program is necessary in any case and the current level of Greek debt is unrealistic. Why not acknowledge the facts and spend money on people to help them get back on their feet and at least reap political good-will, which is sorely lacking right now. Muddling through has not worked so far. If a fourth bailout becomes necessary, it will be a political earthquake with strategic implication for the rest of the world. God, I hope it does not come to pass.

It may be because of your math education but I was introduced to trigonometry in junior high in China. I had, and my classmates had, no trouble understanding them as functions of angles coming from ratios. It is the glue that binds circles and triangles and squares and ... . By high school analytic geometry greatly expanded their scope and use. This is the problem I saw in American high school when I moved to US: shallow introduction to mathematical topics made them vapid and jejune. Ancient Greeks were enthralled by trigonometry, ancient Egyptians built Pyramids and ancient Chinese built great dams with trigonometry. Calling it practically useless and pedagogically only useful as a prep for calculus is going too far.

The list is not just skewed toward bioinformatics; a majority of them is bioinformatics software. I would not call BioPerl or HTSeq high-performance-computing software. Some of the software are written by academic programmers with all attendant baggages. I am not aware of any bioinformatics software written in Fortran, which is not popular in the bioinformatics community. But this is not to say Fortran might not be a good fit. The Broad Institute is rewriting some of its tools in C++ with help from Intel to leverage vector instructions. Who is to say if they had started with Fortran they might not have gotten vector instructions for free earlier.

I totally agree that the programming model of Spark is the right direction. I dream of the day when compiler and OS cooperate to expose a simple interface to distributed memory and an optimal execution-communication system, kind of like Cilk but for clusters.

BTW, thanks for a thought provoking article. You have given me a lot to ponder.

I am trying to get into distributed computing so this article is particularly interesting to me. I may be mistaken so please excuse my naivety if my points are off marks. I thought MPI was mainly geared towards communication-heavy tasks where the underlying network is specialized, for example infiniband or bus between CPUs. One use of MPI is to manage distributed memory tasks between different physical CPUs while threads run on multiple cores of same CPU. Spark, I believe, doesn't handle cases like this well because JVM hides low level details. I have read papers that propose to layer MPI over RDMA rather than expose a flat memory model, which came as a surprise to me but it shows the flexibility of MPI. One thing unclear to me what performance we can expect from MPI when we use commodity network gears, and how it compares to Spark. The article is absolutely correct MPI leaves robustness to user and that is today an oversight.

Modern Hadoop ecosystem is designed for different workload from MPI's. It emphasizes co-localizing date and computation, seamless robustness,and trades off raw power for simple programmingmodels. MapReduce turns out too simple, so Spark implements graph execution, which is nothing new to HPC. As far I know Spark's authors don't believe it is ready for distributed numerical linear algebra yet. But a counterpoint is that I am seeing machine learning libraries using Spark, so perhaps things are improving.

One thing I have learnt today is that MPI isn't gaining popularity. I just have a hard time picturing a JVM language in overall control in HPC where precise control of memory is paramount to performance.

This topic periodically flares and provokes so much emotion it is almost unseemly. The two camps are so diametrically opposed that I think other factors are at play here. Programmers who cannot or don't want to understand some of more advanced/esoteric features of expressive languages like C++ or Rust insist on the virtue of simple languages because it guarantees job opportunities for them. Their slogan is group power; quantity has a quality of its own. Interestingly, communists have similar slogans, who are mainly intellectuals self assured to govern/manage the mass and be handsomely compensated. I never realized the common interest between the top and middle, or how corporate the communism is. The other camp is equally vehement because their livelihood is at stake. If all their time and effort will not make them better/more productive programmers, the upper limit to their compensation is drastically lowered, not to speak of damage to their self-image and statue among peers. They are the liberal individualists who insist that there is no limit to what one individual can achieve or make. There is no need to debate this further. Everybody WILL not change their mind. Go will go far, just look at Java.

T

I heard from my postdoc friends at Stanford a couple of years ago about the pay levels. It could have changed now but I doubt it. Do postdocs at Stanford still pay tuitions? My experience is that UCSF > UC Berkeley > Stanford as far as postdocs' compensation is concerned.

I know many biomedical faculty members in Stanfod pay their postdocs below NIH suggested levels, just because they can.

I would caution people of survivor's bias. Only saints promote people with strong different options and who act on their options. Academia is full of strong egos.

You may not realize this but because you worked hard for your position you believe you are entitled to it and the system that promoted you is near perfect. The reality is not so. I have been to UC Berkeley, where the average faculty is far more interested in gaming the system than making real contribution to science. They orbit around "famous" people, co-author papers with "big wigs," be co-PI with well connected people, because this way is easier than doing real science. There are top people at the top universities, and there are top people at the lower-tier schools, at a lower rate but with a far greater base population. Scientific breakthroughts historically do not favor top rate institutions.

Let's face it. The "top" students that Harvard and Stanford accept are past experts at gaming the system. They are total believer in and are totally committed to climbing the existing social-economic hiearchy. They worship the authorities, are perfectly content in following established lines of research, and pad themselves in the back when they re-search past results to apply to an arcane area, which in all probability ten people are ever going to find interesting and never useful. They are the perfect sheeps pretending to be lions.

Talk about entitlement.

The human genome project certainly was overhypered and had underdelivered on its biomedical benefits. You should look for articles, even in science press, about the 10th anniversary of the human genome project; it was not all roses, to say the least. In fact they were quite frank about it. The main governmental proponent, the current NIH director, acknowledged it as much. HOWEVER, there is no doubt HGP is a resounding scientific success for its technological breakthrough. Genome sequencing is now a basic scientific tool in all branches of biological science. Its impact on medicine will only increase. A lot of the recent crop of newly FDA approved drugs can be traced to genomic insights.

One good thing coming from HGP for the general public and journalists, I have noticed, is that they have more discernment regarding scientists' promises. Trust, but verify. There is a snowball's chance in hell that NIH will see a substantial budget increase in this decade, or the next for that matter.

BTW, that $1000 human genome sequencing claim is not believed by most scientists. Lawyers can argue it, especially Illumina's, but not according to common sense.

What is interesting is obviously personal, and I know you don't mean to denigrate some topics in general, but I want to caution people that characteristic equations and root finding are complex and far from useless. Numerical computation of eigen decomposition starts from finding the roots of the characteristic equations (in altered form), the eigen values, without which there is no computation of SVD. How the behaviors of roots change as coefficients vary are fundamental in control engineering. Newton's method holds up half of numerical optimization. Undergraduates don't have to learn them because others have worked out the details and implemented them in software.

By the way, Cramer's rule is useless for numerical computation, but it is immensely useful in theoretical work. It belongs to the vast body of work dealing with determinants before the rise of linear algebra. Determinant is the only obvious connection to algebra left in an undergraduate's linear algebra course, so I can understand people are turned off by it.

I appreciate what you are trying to do, but I cannot agree with determining the cost of a drug based on potential loss of income/productivity. Should the CEOs therefore get first dips on life saving treatment because they have big impact on his/her employees? Do patients in developed countries have an economic right to better treatment first than people in developing countries? Are some lives worth more than others? I don't think we are ready to go down this route. Incidentally, a very expensive drug is posing these questions, implicitly. But explicit statements like yours are not politically practical or even humane. Ten years ago it was outrageous to hear a nurse to tell a kid's parents to pay up or watch their only child die.

I understand your strong opinion and since I am no sysadmin I have no technical problem with arguments. But I am not sure I totally agree with your characterization of slow pace of change by Apple or the wonderful state of Unix/Linux. Aqua was quite a break from the previous GUI and Apple changed the whole stack at one point from computer architecture to OS to graphic library. I don't know a more radical change than that for a software company. As for Linux graphic environment, I can only say that replacing X-win with Wayland is not evolutionary and it cannot come soon enough. Anyway, hopefully things will quiet down for a while and we can compare and contrast alternatives in the real world.

Fantastic paper. I wonder what "irrelevant theory, questionable conclusions" he had in mind.

>It's a dataframe attached to ranges. This design extends lower classes like IRanges, and all the accessors, setters, etc are consistent. You can manipulate the range part with integer range operations and manipulate the dataframe part as a dataframe. I would check out the IRanges vignette — this is very helpful.

I agree that the accessors, setters of GRange are consistent. I was having trouble picturing how manipulating the range part impacts the dataframe. I think they are like a database table joint but that is not explicitly spelled out. It becomes even less clear what happens to the attached dataframes when merging two GRange intervals. I once had to a symmetric difference of two GRange and it took a few hours of staring at the manual and trying out things at the command prompt.