NyxWulf

Yeah that xkcd came to my mind as well. I'm amazed at the timeless truths he captures in those comics. Creating another version which may be better isn't the hard part. Gaining consensus and getting people to give up the other ones is the hard part.

wc -l is faster however it only works on an uncompressed file. The pipeline in the article was doing much more complicated work on the stream of output. I will often separate my pipeline logic from my "read this data and feed it into the pipeline" logic, since the logic around what to read in and where to write it can often change even when using the same logical pipeline.

As a simple example: cat somefiles.txt | pipeline_script > stored_output.txt

Other times I'm reading gzipped files or remote log files, I don't want that data mixed in with the pipeline logic. If I want to move the output files to a different directory on one set of servers, that may not impact my generalized pipeline.

I work with trillions of lines of log files and there are many ways to scale up pipelines. I wouldn't start optimizing the difference between 1.6s and 1.275s unless it made an economic impact on the problem. How often is it run? Can you process more lines in an economic unit of time with the faster version? If this is a job that runs once an hour or even once a minute how many lines are collected during that time?

Intermixing the grep or wc -l logic into the pipeline logic can have adverse support and maintenance costs and many times saves machine time while spending programmer time. Which one dominates in this scenario?

It's like the old saying...there is more than one way to skin a "cat". :)

This works for me in zsh on OSX

history \ | sed "s/^[0-9 ]//" \ | perl -pe 's/ \| */\n/g' \ | gawk '{counts[$1] += 1} END { for (x in counts) { print counts[x],x}}' OFS="\t" \ | column -t \ | sort -k 1,1nr \ | head -100

Ok not sure how to format this on HN https://gist.github.com/nyxwulf/5608955#file-gistfile1-sh

Observing other channels, or seeing when a channel does fire is what the select structure is for. It is one of the features that makes channels and goroutines very nice in go.

http://golang.org/ref/spec#Select_statements

http://blog.golang.org/2010/09/go-concurrency-patterns-timin...

http://talks.golang.org/2012/concurrency.slide#1

Video for the slides: http://www.youtube.com/watch?v=f6kdp27TYZs

Interesting concept, but the flaw at the heart is the presupposition that lack of progress right now means we are at the ultimate limit of what can be accomplished. Imagine if cavemen learning to paint on walls said, well we haven't improved in a few millennia, so this is probably the most complex thing that can be represented by drawings. Or what about math stopping with Euclid? It was thousands of years later that progress happened.

Technology comes in fits and starts. A lot of new things happened in the 50s and 60s, we are still trying to figure out ways to use and apply them. Just because someone thought about and prototyped something then doesn't mean it's not new when that feature goes mainstream (e.g. garbage collection in Java, channels for concurrency in Go, etc).

For a long time we couldn't break the sound barrier, which is a limit, but not the speed of light limit. Because progress is stalled now doesn't mean there will never be progress in the future.

When I was in high school one of the students committed suicide. There was a lot of discussion and outreach at the school at the time. One of the things they talked about was that once one student commits suicide it greatly increases the chances that others will follow.

My mother is also a social worker at the VA Hospital, and she's had some of her patients opt out so to speak. It does appear to have a very strong pull on other people who are suicidal to choose that path.

So if anyone reading this is having those thoughts, please, please talk to someone professionally. Call the suicide hotline in your area. There are very real things that can be done to help you through this.

It seems like you are confusing predictions about this election with overall model accuracy.

The state by state predictions are the primary method by which you can judge the accuracy of his model. If his model forecasts a state as 60% for one candidate, you can assess that level of accuracy by looking at all 60% predictions.

Let's take some examples from todays date on FiveThirtyEight.

* Florida 54.8% chance of Romney win

* Virginia 67.0% chance of Obama win

* Nevada 67.9% chance of Obama win

* North Carolina 79.6% chance of Romney win

* New Hampshire 80.4% chance of Obama win

* Iowa 80.7% chance of Obama win

* Nevada 88.7% chance of Obama win

Other states like Texas, Utah, Idaho, Wyoming are projected at 100% for Romney and New York, California, Oregon, and Illinois at 100% for Obama.

If any of the 100% states go to the opposite candidate, that is a model problem. If some of the ones specified with extremely high percentages 95% go against his predictions with a high margin of victory, again that is a model problem.

Finally, some of those close races should go against the models prediction.

Let's take the 7 states listed above. 5 of the 7 are projected for Obama and 2 for Romney. However there is only about a 37.68% chance of that exact distribution happening. I break it down as follows:

Obama-Romney * 0-7 0.02%

* 1-7 0.14%

* 2-5 2.26%

* 3-4 12.14%

* 4-3 31.48%

* 5-2 37.68%

* 6-1 16.00%

* 7-0 0.28%

Each of these numbers are probabilistic statements about the likelihood of the overall event occurring based on the probabilities. They are from a single simulation of the 7 state probabilities run 10,000 times. Each of them has it's own distribution, eg 5-2 was 37.68% in the first run, 37.38% in the next, then 38.03%, then 36.85%, etc.

This is a very simple model prediction, but by taking all 50 states into account you can get a very clear assessment of how well his model is actually predicting the outcome of the election. Complicating matters is the time series nature of the predictions.

This type of model is precisely the way you get away from "Fooled by Randomness and all that". His model is clearly articulating the amount of uncertainty in the forecast.

You can get the kindle version now. John Cook may have gotten an advance copy of the book.

The arguments advanced against an "Operations mindset" are mostly strawmen. Operations is primarily about efficiently operating an organization towards a goal. The type of thinking advanced in the article however is more likely to be pushed by someone with no operations management training.

For instance, putting people on multiple projects versus single projects. Multi-tasking and context switching cause known losses in efficiency even under the perfect scenario with no startup time. So if you want to emphasize getting projects completed, multi-tasking must go. This is an operations management mindset, though it runs against much current practice.

The points about efficiency are similarly misguided. You don't focus on worker efficiency, operations points you at throughput and global optimization rather than local optimization.

Those however are about focusing on improvements and operations of the current systems. It is true that innovation is different than operations. In the same way that learning to program efficiently in a language is different than inventing a programming language. They are different kinds of things. Many operations principles however can be successfully applied to innovation. For an excellent read on the topic, read "The Principles of Product Development Flow" by Reinertsen.

To get a better understanding of operations in general, and not the hyperbole advanced in the article a good starting place is "The Goal" by Goldratt.

One big revelation I learned while studying operations is that the things that largely drive developers crazy aren't good management practices. They aren't advanced by operations researchers. They are in fact what I always thought they were, bad management practices.

I use vividchalk, which I was happy to see there on the list. It's amazing how many variations on color schemes there are.

That's the difference between guaranteed capacity and most services. It's very common to use the properties of normal usage patterns and pool the excess capacity. The phone system has done that for ages. Ever try to make a call during an emergency?

Just because their service doesn't cover certain extreme situations does not mean it's worthless. It would worth knowing how much excess capacity they are pooling together, but vendors generally won't release those types of details.

In Utah the year is the only one that changes color, all of the months are white. That's why I was looking at his plates, to see if I could see even the letters on his plate. Granted I'm not a trained observer of registration violations and maybe it was observable to someone who is. Given the distance and the visibility during the night I'm pretty skeptical. It's just a sneaking suspicion I've had - because I know I'd want to use those license plate readers in exactly that why if I was in their place.

I knew it! Roughly a year ago I was driving home from work at night. My registration was expired by a few days, and I drove past an officer in a parking lot. As I was passing him, I knew my plate was expired, so I was looking at his plate to see if I would be able to determine from that distance if the plates were expired. So I was scrutinizing his plates to see how much detail I could see. I have better than 20/20 eyesight, and I knew there was no way I could see that level of detail at the distance he was at. So imagine my surprise when he comes after me immediately and pulls me over.

More out of curiosity than anything I asked him how he knew my plate was expired, his response was "I have really good eyes". I've thought back on that experience over the last year, and I knew the technology for reading a license plate wasn't hard, but I didn't know how widespread the use was. After reading this piece and seeing the picture of those cameras, which are mounted in that area, I'd bet they are using it to give expired registration tickets too.

Even if people don't learn to "properly" understand the Gaussian distribution, basic understanding of the nature of variation and the difference between random events and something that indicates an actual change in an underlying system would be immensely useful to people.

You don't have to know all of the fundamental underlying ideas of any concept before it has practical utility.

Let's take the salesman example. A salesman isn't literally making $150/hour. As a hypothetical the value might be $0 in hour 0, $450 in hour 1 and $0 in hour 2 for an average of $150/hour. Much of sales work is this way, their is preparation time, and sales time. There may be prospecting, etc. So even though they may not make the money during that hour, if you are burning their motivation to prep or prospect you are burning their ability to get through hour 0 and into hour 1.

Part of the reason to use an average is that it gives you a very simple, but generally applicable metric. There are always exceptions and the rate may change frequently or slowly. Those are nitpicks though. The general point is that your time should be valued against the best use of your time. For instance the hotel example only makes sense if the alternative to walking to the convention is working or more time networking to generate revenue. If the alternative is sitting in your hotel room longer watching TV then $160/hour is not the replacement rate.

That's our next step is automating our fail over. We currently use DNS Made Easy and while you can do that with them it's tougher. We are switching to Dyn, and their Layer 2 failure detection will put our failover in that 6 second range.

If you operate a service where uptime is critical, two of everything isn't enough. You have to ensure they will be separated geographically and logically so that when things like this happen, you don't lose your systems.

We operate systems that sit on the pages of the top e-commerce companies in the world. We have 10 separate segments of clusters. Operating in four AZ's in East, three AZ's in West-1 and two AZ's in West-2. When this outage happened, the servers that were impacted in east were removed from our DNS and within 9 minutes the impact of this event on their sites was eliminated.

heh, I was about to, but then I saw your response. Thanks.

I agree with many of the points in the article, however it is conflating two unrelated things. Using simple averages in isolation is almost always a mistake. The real story in a system almost always lies in it's variation and not in the average. There are many ways to look at that, one of them is a histogram, although I tend to prefer time series graphs with moving ranges. One of my favorites is the Xmr chart or any of a number of statistical process control charts. Using those you can get a much better understanding of systems over time.

The transition to Real User metrics is not at all related to using an average. All recording and instrumentation processes will benefit through using more sophisticated and nuanced tools than a simple average. Real user metrics are an important data point, but they lack certain information you can get from external monitoring systems. We use both pingdom and catchpoint, by far my favorite is Catchpoint because I can see things like what ISP is involved with a slow request, what geographic region, etc. I can also get scatter plots and nice statistical graphs around median, geometric mean, 75th, 95th, 99th percentile.

So in short the main points are good, simple averages are misleading. Capturing end user performance data is good. Not using external monitoring though isn't a good idea because there a number of things that you can identify if you have that insight.

For this type of service spamgourmet is my long time favorite.

Create disposable e-mail addresses that forward to your real e-mail address. You get a name stub and then you can create e-mail addresses on the fly. For instance you can have something like "hacker" as your name stub, and they have 20ish domains to use. So it works like this, say you are signing up for a subscription with inc you can then use inc.hacker@recursor.net or inc.hacker@neverbox.com or inc.hacker@spamgourmet.com. Whichever you prefer as the domain. In this case it would use my default burnout rate which is set at 3 when you signup. I bumped mine up some, but you can also specify it. So if you wanted say 10 e-mails before it burned out you could use inc.10.hacker@recursor.net. If you don't do anything else, once you receive 10 e-mails at that mailbox it will burnout and start eating them. You can also go whitelist the domain so if you get an e-mail from inc.com it will come through, but it will use the normal burnout settings for e-mail from anyone else. There are also much more advanced features for when/if someone figures out how this works and tries to spam you like setting watchwords. So if an e-mail doesn't contain the current watchword it will not allow the creation of a new e-mail address, but old ones continue to work.

One thing that is nice is the interface shows you how many e-mails have been received on a given address versus how many have been eaten. For instance when I signed up with inc, they apparently gave my e-mail away and I started getting e-mail on that inbox, so I let it burn out. Now they can no longer reach me. Market Samurai has been incredibly bad, they've had my e-mail address for almost 2 years and I have already received 2500 spam messages on that inbox.

The great thing is this is all free and open source. You can donate, and if you use the service a lot, you probably should. You can also download all the source and set one up yourself. Overall the current setup is not user friendly by design. In the FAQ it specifically discusses they don't want to support end users who can't take the time to read the FAQ. Overall I cannot overstate how great this has been. It solves such a major source of pain and does it in such an elegant way. If you haven't found a service you are happy with, give spamgourmet a try.

Most business people aren't interested in model accuracy as a term. They want something that provides benefit, e.g. cost savings, increased revenue, increased profits, etc.

The sales process of convincing someone they need an accurate model is tough, especially because robust models are time consuming and expensive to build.

If you can come up with a model that shows good results, and people know they need those results, then you can start a company selling either a service or product to get those results. If people don't know they need your results - then you have to educate them, in which case it's a much more difficult business to start.

I don't know many business people with the temperament, understanding, or the pocket book to deal with general research type problems.

I couldn't agree more. Accuracy is a problem, variation is another problem. Dealing with layers in the business who have no math or statistics background but very strong opinions is yet another complication.

These types of conversations aren't uncommon.

Other - "I need you to prove our stuff does X, Y, and Z".

Me - "Ok.."

<time elapses>

Me - "Ok the data shows our stuff does X but Y and Z are just random noise"

Other - "We ran it once before with this other guy and it showed our stuff did X,Y and Z. We've been promising it to our clients for a year. He gave us several examples, but when the clients asked to see the underlying data he couldn't produce it. So we just need you to prove it does X,Y, and Z."

Me - "The data only shows it does X. Y and Z are impacted positively through X, but once you condition on X, Y and Z are not causally affected by our stuff"

Other - "Yeah...well I promised client we would give them a report by {{insert random ridiculous date here}} proving it did X, Y and Z. We are going to lose them if we don't deliver a report saying that"

Me - trying for the 50th time to explain they shouldn't promise a positive result when we've never looked at the data.

There are hundreds of variations on this conversation. Your code is wrong is one variant (which depending on the timeline is hard to dispute). Of course if you take long enough that your code is correct, then you are going to slow. This isn't a science experiment, just make it work is another. Watching someone go slack jawed and start drooling because you accidentally used a math term is always interesting.

I have a whole new perspective of being on the cutting edge. It seems like it mostly means you are on the cutting edge of comments from people who don't know how ridiculously hard what you are doing is.

I agree with Paul's comment. The 250 number feels low to me, but that is applying a specific model. Typically people come with some set of favorite models, and many of them provide that vast majority of the benefit a business needs. Especially when the current model in use is slipshod and busted at best.

As someone in the big data field on the ground (VP of Engineering). Let me give you my thoughts on it.

Your impression about the hype is correct. There are a lot of vendors offering BIG solutions, if you pay them BIGGER money. Where I used to translate the word enterprise to $$, now I translate Big Data to $$$$$$$.

When I'm hiring, I don't go looking for Big Data people, because generally they don't exist. Statistics is a really great general addition to a programmers toolkit. Machine Learning is valuable as well, although in my experience the application is more limited. What this article doesn't mention is a whole host of other skills required.

Modeling, and not just a formal mathematical model, but applying any type of model to your data to get insight. Check out the model-thinking class on coursera.

Exploratory Data Analysis, much different skill than confirmatory statistics.

Design of Experiments, specialized subfield within statistics.

Logistics, how to setup, maintain, and maximally utilize an efficient distributed cluster and build a pipeline getting your data to the cluster, cleaning it, building it into a model, and then extracting insight and delivering that end value.

Those are a couple of the skills at a high level. At a more nuts and bolts level, Hadoop is the defacto standard for Big Data. Learning how to build a data pipeline out of the Linux tool chain is very common in the data science world.

The overall value stream for Big Data is deep and wide. Most companies don't have expertise in much of these, and so at the current time you have to learn them yourself or find a company focused on building a team around it.

If you are just learning this yourself, you'll probably get an academic knowledge. If you want to make yourself valuable in the marketplace, you'll really want to get hands on experience. Knowing a z-score is one thing, building a process to gather data and compute a model against it is a whole different ball game. As the article mentions, if you have nice clean data it's easy to apply a model. If you have messy ugly data from 20 different vendors and 200 clients with various failures, anomalies, and you have to figure out what type of model is helpful, oh and you have a deadline because for 500th time someone promised something impossible to the client, then you have something closer to what Big Data is today.

* grammar edits

It's not that you can solve every problem, but supply and demand are very real. If the wage rates for Big Data get high enough more and more people will try out the field, which will generate a larger supply.

The reasons companies don't just throw out huge salaries though has to do with the demand side. The salaries companies are willing to pay is related to the marginal advantage they can gain from hiring someone with that skillset. If for example a company will gain say 200k per year in total advantage, that would place a hard cap on how much they would be willing to pay in salary.

So if the advantage is very high, companies will pay more. If the supply increases sufficiently wage rates will drop because there is over supply. If the supply doesn't increase enough, wages will increase more - however each company will drop out at it's own value point. This provides the natural limit to where most salaries cap out.

I've seen several articles like this, and there are a number of things to consider.

Logging to ascii means that the standard unix tools work out of the box with your log files. Especially if you use something like a tab delimiter, then you typically don't need to specify the delimiter.

As an upside you are aren't storing the column definition in every single line, which if you are doing large volume traffic definitely matters. For instance we store gigabytes of log files per hour, grossing up that space by a significant margin impacts storage, transit and process times during write (marshallers and custom log formatting). Writes are the hardest to scale, so if I'm going to add scale or extra parsing time, I'd rather handle that in Hadoop where I can throw massive parallel resources at it.

Next you can achieve much of the advantages of json or protocol buffers by having a defined format and a structured release process before someone can change the format. Add fields to the end and don't remove defunct fields. This is the same process you have to use with protocol buffers or conceptually with JSON to have it work.

Overall there are advantages to these other formats, but the articles like this that I've seen gloss over the havoc this creates with a standard linux tool chain. You can process a LOT of data with simple tools like Gawk and bash pipelines. It turns out you can even scale those same processes all the way up to Hadoop streaming.

Can you elaborate on a few things? You say that UUID version 1 was designed for an outdated purpose. What do you mean by that? UUID V1 does sort by time, so I'm not sure where the outdated purpose is relevant to that consideration.

You say you cannot allow the number of bits a UUID spans. Is that a hard constraint, or is that an intuitive desire? in my experience limiting the number of bits is usually to fit on a single machine or in limited memory. However designing a distributed id system is all about scale and minimizing complexity from my point of view.

When you start to scale out to dozens of machines, the weight of these complex methods really starts to become evident.

I resisted UUID's for a long time, but I finally gave in, and you know what? It's one of the best decisions I've ever made. The design simplicity with v1UUID's is amazing. With the simple_uuid ruby gem I can generate 40kish UUID's per machine on a modest ec2 instance. Using NTP our clock drift is in the 10s of microseconds range IIRC. Using Cassandra and Hadoop we don't have to worry as much about fitting into a single machine. The overall improvements have been staggering though. The simplest and most important benefit is none of the services have to read or request information before generating an ID. Now it's generate an id, send the event on it's way, next. I'll leave it as an exercise for the interested reader to map out the synchronization process for generating unique id's without locking across multiple threads and multiple machines.

tl;dr Just use version 1 UUID's and NTP.

I've been through the process of litigating a contract in court. Not fun, but definitely was an exquisite education in the school of hard knocks.

Lawyers throw all kinds of language into a contract, that doesn't mean the intent was there. Vague, unfair, or overly broad language that was not clearly understood by both parties tends to get reduced, removed, or modified in accordance with what both parties reasonably understood to be the intent. So if they tried to get overly sneaky and do something where the basically version says X, but the full clause says we do X,Y,Z, then the courts may very well rule the implied intent was X but Y and Z were not implied and agreed to.

If they understand what they are doing, and plan to strictly abide by the basically section, this is a fantastic idea. If they at some point decide they want to get sneaky and put something in the detailed version that is not covered in the "basically" section, they may well find themselves in a world of trouble if it comes to court.

My experience was a company trying to sue me, despite them breaching the contract. The judge took a dim view of the crazy clauses in there and was pretty sharp with them. I expect the same thing would happen here. A normal user would probably abide by the "basically" section and the courts would probably take that interpretation. Of course YMMV.

I'm not a lawyer, but I have been through the process of litigating a contract in court.

The law does cover this. Contracts are about a meeting of the minds. Do both parties understand and agree to the terms. Things like email, other written communication about the contract or terms are all reviewed at trial. It's amazing how much of the "sneaky" verbage just get thrown out or ignored as unreasonable or not understood.

Initially it was not encrypted and was stored in plain text in the database. Then we started encrypting sensitive information at rest, both the database files and the backups. We also encrypted certain sensitive information with a secondary encryption. The next massive undertaking was to find all of the processes that use the data, how they use it - both internally and externally, and then ensure the data was protected throughout the entire process. The company was still working through that process when I left. They had identified many places that were problematic and were actively working on fixing them. The team I managed was the payment processing and cash management team. There were a number of legacy systems written in access, fox pro, DTS, Excel even. Many of those were the most painful, and frankly the most problematic.

Many people were put in a situation where the requirements for getting something implemented through the IT teams would take months, yet they were still held accountable for getting things done, so they would build an Excel process to get the job done. They would build some fantastic process, and get promoted, then that Excel system would get dumped on either someone in IT or someone replacing them.

I'm getting off track, but looking at it superficially the data was encrypted. If, however, you watch the data through it's entire value stream, there are many places it is vulnerable. Given the state of affairs in most companies and how few of them think in terms of managing value streams, I expect my observations are fairly widely applicable.