1,056 karma · joined January 28, 2011
I think, fundamentally, files are the wrong abstraction for log data (as pointed out by Adam Wiggins of Heroku fame: http://12factor.net/logs), but it is something that we are stuck with.
I agree. There is a strong need for common interface(s), and that's a strong part of the motivation behind Fluentd/Kafka/etc. www.fluentd.org/blog/unified-logging-layer
As someone who works as both a maintainer and community manager of an open source project, I find it naive to think that an open source project can thrive completely free of corporate sponsorship/benevolent dictatorship.
For example, I am hired by Treasure Data, Inc. to work full time on Fluentd (https://fluentd.org), the open source log collector. Treasure Data does use Fluentd, and Fluentd is a core piece of the technology that we offer to our customers. However, my boss explicitly tells me NOT to herd the community to serve our interest. Why? Because we know that we benefit a lot from the Fluentd community both for community contributions and awareness, and it servers our corporate interest to ensure that we stay on good terms with the community.
At the end of the day, open source community members are intelligent. They can smell when companies start exploiting the community to serve their own interest, especially in a way that's not beneficial to the community. So, in that sense, my response to the OP is "let the community decide. if enough people hate on community managers, they will disappear."
That being said, I can agree with the OP that "community manager" might not be the best nomenclature here. But that doesn't mean open source communities do not need the go-to person for ensuring the community's health and longevity. Whether that person is called a community manager, mailing list patrol, event coordinator, conflict resolver, or geek herder is somewhat irrelevant.
I have to agree. I am one of the main people to support Fluentd (an open source log collector https://www.fluentd.org), and a lot of people do forward rsyslog/syslog-ng to Fluentd for further processing/analytics. We've been trying to figure out a way to do the same with journald, and while it does seem possible, it will require a ton more work.
With Facebook's brand and recognition, I am sure this JSX will eclipse the other JSX, which is rather unfortunate.
As someone who has/can use any of the listed software, another benefit of Excel (in addition to its pervasiveness and popularity with "non-technical" people) is its rapid prototyping capability. If your data is small-ish and requires no serious stats calculation (which accounts for a lot of data analysis, especially in the early phase), Excel is super fast to prototype in. You can do basic math, aggregation & filtering and charting all within a few keystrokes.
Another benefit that I see is that as an interface, spreadsheet shows you to show each step of the work. As a programming environment, spreadsheet is just a mess, but I find it easier to track what I am doing in Excel than working inside R/Python REPL and staring at my history or currently defined variables.
This is exactly what we are trying to solve with Fluentd, an open source log collector with the goal of unifying log management (http://www.fluentd.org/blog/unified-logging-layer) Syslog is a solid, "tried and true" tool, but when it comes to extensibility/modularity, it is definitely lacking.
- Linear Algebra Done Right by Sheldon Axler
- Linear Algebra by Steven Levandosky
- Linear Algebra and Multivariable Calculus, and Manifolds by Ted Shifrin (you probably do not need the manifolds part)
Also, as paradoxical as it may sound, CS229 lecture notes themselves. The key here is to verify the details painstakingly. In math, we joke that whenever a book says something is "trivial" or "easy to verify" or "clear to show", it is a substantial exercise in disguise.
The real reason statistics is losing to machine learning is M&M: money and marketing.
Money is a huge factor when kids choose majors in college. Many of them have student loans to pay off, and for many of them, getting a high-paying job post college is a serious consideration. In this regard, statistics is a great major, but computer science is flat-out ridiculous. When big tech companies are offering a Stanford graduate with only a few summers of programming internships under the belt for 150k/year, naturally a lot of kids are lured into computer science.
Then, once they start studying computer science, they discover this thing called machine learning. While I do think there is a difference in emphasis between stats and machine learning, the fundamentals are same, except the nomenclature sounds much cooler in machine learning. Nonparametric inference sounds esoteric and cryptic, but unsupervised learning sounds futuristic and cool.
The biggest problem with both statistics and machine learning education, I would say, is their lack of emphasis on mathematical foundations. When I was in college, CS229 (Introduction to Machine Learning) was touted to be the hardest class at Stanford. Having helped my friends wade through CS229 problems (a lot of which comes down to wading through linear algebra and multi-variable differential calculus), I do not think this is remotely true: CS229 is hard because it attracts students who do not have the requisite mathematical maturity to learn statistics/machine learning in a serious way (I am not even talking about the real-analytic foundation of probability and calculus but rudimentary linear algebra and chain rules).
Also, I do think CS229@Stanford is a great class that brings theory and practice together =)
As for R/Python/Matlab/etc., I will let the zealots argue what's best. To me, they are like statistics and machine learning: similar with different emphasis.
While Carousel itself is considered to be a flop by some (http://techcrunch.com/2014/08/04/why-standalone-apps-are-sup...), I think the overall strategy seems correct, or at least, less wrong: go up the application stack based on their strong user base in the data storage layer.
Yes, I agree. What I tried to convey is the importance of balance between hard work and a degree of fatalism.
You can't solve a lot of things with hard work, but many American elites I know are wired to think otherwise - until they start to fail classes in college or can't advance anywhere near as fast at their jobs. When life hits you hard, accepting things as-is and not trying to work even harder might prove to bring a happier outcome.
>School is so artificial; everyone is solving the same problems to which the teacher already knows the answers.
That actually wasn't my experience. I had a lot of fun in junior high, high school and college, and while there was an element of sycophancy, my education was largely intellectually honest, challenging and rewarding. I would say it was at the first job out of school where I learned the important skill of "just doing it because your superior told you to" to earn credibility and trust.
I only know of Japan and the US, but as someone who went to one of the most prestigious secondary schools in Japan and universities in the US, I have seen well-educated, smart people with "growth mindsets" struggle later in their lives.
1. Regardless of what we say, in many corners of adult life, results are valued over processes. While a superior process has a higher likelihood of yielding a superior result, this is often not the case, and in a perversely Murphy's law-esque manner, it turns out to be false at critical junctures of one's life. And the deeper the growth mindset is ingrained into you, the more disappointed/despaired you find the situation and feel incapacitated and betrayed. Of course, a singular emphasis on results with no consideration for process is equally bad. Most people find their own local optimum between the two extrema, and I don't see how a campaign towards one end of the spectrum is all that meaningful or worthy.
2. This probably sounds terrible, but not everyone is "smart" as measured by academic performance. Certainly effort is a huge part of the equation, but some minds are better wired for academics than others. And the longer you work at it and hence surround yourself with qualified peers, the more apparent it becomes that not everyone is working equally hard. This realization usually does't mesh well with the emphasis on process from one's formative education, and many people become jaded/hopeless. (And of course, even within academic subjects, there are individual variances). While it is important to try, it is also the responsibility of educators (and adults) to see if the child's potential lies somewhere else, or to borrow Mr. Khan's words, to see if the child can be tenacious and gritty about something other than academics.
Would you be interested in leveraging your skills AND learning about distributed systems and databases in a highly concurrent setting? If so, drop me a line (my email is on my profile).
I am pleased to see a company like yours cropping out of YCombinator. Not for nothing but: get some help with your website. It might seem like such a frivolous thing, but having a well-designed website goes a long way (or rather, having a bad web design mars your authority and reputation undeservingly).
Also, I observe that the data science community does not talk about these issues nearly as much as they should either because 1) it doesn't make a very inspirational topic 2) they are shielded from this kind of issue by a data engineering team.
I agree with you here as well. I recently wrote a blog article about this, at least for log data > http://www.fluentd.org/blog/unified-logging-layer
1. Data hygiene issues show up inconsistently, even within the same data source: I experienced this first-hand as a quant trader: I only had half a dozen, well-structured time series data coming from our trading apps and switches. You would think that I can get reasonably clean data every day, or failing that, being able to automate the data cleaning script. Nope! I experienced every data hygiene issue imaginable: from ntpd being broken, local time/UTC inconsistencies, switch firmware acting up, all the way to human errors in the ETL process. Building a reliable data pipeline has so many moving parts that I am fairly pessimistic about any one-stop solution.
2. Then there is a volume issue. If the data is small enough, humans can correct them fairly reliably, especially with some help from automation scripts/software. That said, doing this at scale is a very hard problem. One thing I learned working at a big data company is that many folks use MapReduce as a data cleanser at scale, and for that, MapReduce is a pretty awkward tool.
3. Anecdotal evidence: I have talked to employees at various big data platform/software companies, and though they have a wide ranging opinions on stream processing, Hadoop, Spark, etc., they all agree data cleaning is a huge pain/deal-slowing/-killing nemesis/unsolved problem that their employers semi-solve with increasing Sales Engineer headcounts. If this issue of solving data cleaning at scale was easy, I feel that someone would have come up with a very effective answer by now (and as an industry insider, I should have heard about it).
1. Mirzakhani is the first Iranian 2. Bhargava is the first Canadian 3. Hairer is the first Austrian 4. Avila is the first Brazilian
"This is easy to verify for N = 2: You are just flipping a coin to decide if you swap 1 with 2. For N > 2, you just need to show that each of 1 through N has an equal chance of getting the k-th slot for 1 through N. In the first step, every number has a 1/N probability of getting into the first slot. For all other slots: the number has (1-1/N) chance of getting/staying there, then by induction, all slots are equally likely, hence (1-1/N)*1/(N-1) = 1/N. This completes the proof."
That said, the challenge with online (streaming) data is that you have to constantly estimate the distribution you are sampling from. Someone here might know better =)
Treasure Data is hiring a software engineer/developer advocate for Fluentd, the open source data collector to unify log management.
Description: http://www.treasuredata.com/en/careers/careers-posting-004.p... Fluentd: www.fluentd.org
Yes, you will get to work on open source and get paid. If interested, please email me (it's in my profile)
Also, sometimes "X is down" type of submissions can start conversations interesting in their own right in the comment section.
I added a random URL string to have HN recognize this as a new entry. Perhaps it's a good idea to allow re-posting the same link for certain URLs so that "XYZ is down/having issues" type posts aren't deemed as duplicates?