matan_a

$3B in RSUs for employees. Assume 50 employees and equal share - that's $60M each. Wow.

While there are ways to create private registries [0], having a new packaged way to do it would be very useful to some orgs.

[0] https://npmjs.org/doc/registry.html#Can-I-run-my-own-private...

We're currently using Apache Kafka which is working out great, but does include a broker as a middleman. This is good for us since it's designed to accumulate unconsumed data (or even roll it back) and still provide good performance - rather than having that happen on the producers or consumers.

It would be cool to see some stats on throughput and latency relative to # of producers / consumers and amount of data currently accumulated in the producers (since there is no middleman).

It would be great to combine some public data source about browsers together with what you have like, for example, adding information from WURFL:

http://wurfl.sourceforge.net/help_doc.php

Mind their (new) license tho if you plan to use them specifically.

Seems like a great fit so the user does not have to perform their own cross-referencing.

seems like a good candidate for Kickstarter

i'm curious because right after i registered, i received an email with the subject "Scouts: Please confirm subscription" which goes to a listserv and has a reply to: will -at- vinova.sg.

What gives?

By internet, i think they mean the WWW:

http://en.wikipedia.org/wiki/Les_Horribles_Cernettes

We've been using Pig extensively for our MR queries and we're very happy with it. While i haven't used Cascading, i do like the pipelining approach of Pig more than the SQL-influence method of Hive. It lends itself better for meeting our needs.

Testing is a good point. If you're running Pig through their APIs it is definitely easier to test than command-line running scripts. We've written test code that reads and runs pig scripts through the API using fixed sample data (stored in HDFS for easy access), read the results, and compare it to expected results (also stored in HDFS). Honestly, you don't need too much input data to prove the correctness of the query.

Remember that Pig also supports placeholders in your scripts so you that you can set them in run-time to define input/output paths, etc. This makes testing easier.

Dependencies can also be stored in HDFS which makes it simple to run your scripts w/o the need to distribute jars around.

I wonder if the will succeed where many others have failed.

I would have to go for official Redis HA support. Restart time is painful when your Redis instance is large.

This doesn't actually discuss how Facebook manages and maintains their 100PB of storage, rather just talks a bit about the namenode SPoF issue. Disappointing article for sure.

Just please consider that XPath needs to store the items in a tree structure first (XML needs to be parsed and loaded into memry) before it can be useful. Running that on a large log file would be an interesting performance experiment.

I know of a service which kind of does this (w/o the social network part tho):

www.wearelistening.org

It's nice to see Amazon entering this space as they do have the expertise to keep this up and running as well as provide the scale needed as apps grow.

What i always tho is the ability to run search queries that also involve dynamic grouping (like grouping by random combinations of facets) and providing those aggregated results.

Only thing i've seen that can do this "on the fly" is SenseiDB. CloudSearch/Solr/etc seem to need preprocessing to get this right.

Good stuff. I see you're on Heroku as well which is always a win.

Now if someone could put SenseiDB on the cloud, i'd pay for it...

There are quite a few other performance related points to think about for Solr speed for queries and indexing.

Here are some that come to mind right now that are very useful:

- Be smart about your commit strategy if you're indexing a lot of documents (commitWithin is great). Use batches too.

- Many times, i've seen Solr index documents faster than the database could create them (considering joins, denormalizing, etc). Cache these somewhere so you don't have to recreate the ones that haven't changed.

- Set up and use the Solr caches properly. Think about what you want to warm and when. Take advantage of the Filter Queries and their cache! It will improve performance quite a bit.

- Don't store what you don't need for search. I personally only use Solr to return IDs of the data. I can usually pull that up easily in batch from the DB / KV store. Beats having to reindex data that was just for show anyway...

- Solr (Lucene really) is memory greedy and picky about the GC type. Make sure that you're sorted out in that respect and you'll enjoy good stability and consistent speed.

- Shards are useful for large datasets, but test first. Some query features aren't available in a sharded environment (YMMV).

- Solr is improving quickly and v4 should include some nice cloud functionality (zookeeper ftw).

Curious...what throughput do I get when I use your service? Twilio limits you to 1 SMS / second unless you get a short code (30 SMSs / second) or otherwise use multiple numbers.

Do I get any benefit from you guys regarding this?

This is my rule for Yelp:

1. Listen to bad reviews. 2. Ignore good reviews.

Wow great stuff. I just wanted to say that I appreciate you sharing this with us. Bookmarked :)

Looks like an interesting talk. I'll definitely try to be there. Thanks!

While i did see quite a bit of options for node, i haven't been playing around with ORMs in Node too much. ORMs to me refer to those used with relational databases and I haven't done that yet in Node.js. i've just been using MongoDB (mongoose) and Redis (node_redis) - both of which have good libraries.

Also remember that the very nature of JavaScript might make the classic idea of how an ORM is implemented a little different.

Here's a rehash of my comment on the blog which might cover this point:

From my experience, performance is a tough question to answer because you really have to think of the system as a whole. The weakest link (slowest piece) usually dictates the overall system perceived performance. Having a slow DB isn't going to make your choice of Java or Node.js matter that much...

So assuming that everything else is equal (like architecture, data structure choices, database driver quality, database design and queries, etc) it might be that Java is faster.. but it might be a moot point.

Writing asynchronous code in Node.js is much more convenient than Java and helps in doing more at the same time so that you aren't waiting for all those database queries to run one after the other, for example. Check out the Node.js async module for some cool patterns to use (https://github.com/caolan/async).

In all, I wouldn't make the choice to move to Node.js because it has better performance or not, but the overall holistic view.

Here are some of my suggestions (i've been consulting for many years for large and small clients alike).

First to get it out of the way: Consultant != Contractor. Some people think they mean the same thing. They don't. Contractors vs employees is just a hiring logistic. Consultants are usually contractors, but the scope of work is usually more specific and specialized.

I'd define a consultant as a subject matter expert who helps clients with a problem domain for a fixed interval.

"Being a consultant" to me means usually helping a company with a problem or role that they can't help themselves in. That means you have to be not only good at what you do, but confident in your decisions and suggestions. You are the authority in their point of view - or at least close to one so you'd probably want to find that area of expertise where you feel that level of confidence.

Having said that, reading books might be a good start, but that's definitely not enough to become a highly paid consultant who gets great referal gigs. "Been there, done that" is a good expression to describe where your experience would best come from. Obviously, you'll get better as you consult more, but it would be wise not to improve your skills on the behalf of others checkbooks in a consulting role.

So assuming you're ready to go that route, you need to fill your calender with enough work to keep yourself busy (paid) and your clients happy. I personally don't have a secret formula for that, i just try to do the best work i can. Leaving a good impression tends to make it easy for people to recommend you to others which brings you more opportunities.

I'd start simply by working for someone else (i.e. a consulting firm). They'll make some dollars off your work, but you'll start to get a feel of the process and build a client base.

I hope that is at least a bit helpful.

I've seen first hand that even a large "fortune top 10 places to work" silicon-valley company forces IE6 on their staff. You don't have to look very far.

I did see Chrome installed as well so that might be their solution :)

Thanks for the suggestion, but still not ideal. It performs a global lock on the DB which would block all operations while it computes.

I've been developing using MongoDB for a few months now and the thing that always comes back to me as missing is the need to group a set of actions so that they perform atomically. It's not exactly meaning i need transactions (i don't care about the commit/rollback part), just need the need to say - hey server, do X, Y, and Z as a batch so that another thread won't do something in the middle of that.

The general consensus on this is to structure your data so that it encapsulates your business needs in one document structure (which is atomic on changes), but i find it hard to always conform to in the real world.

So now i have to use zookeeper (memcached also works) to setup global locks on those specific batch update actions. I guess it's a small price to pay right? Right?

Well, Mongo does support queries on embedded objects as described in http://www.mongodb.org/display/DOCS/Dot+Notation+%28Reaching...

It might not be exactly what you were looking for, but it could have the same outcome.

I came across this and i was curious how prevalent this is in the HN community. 45000 is a pretty big number.

How is this news? Most technology is dying slowly to be replaced by the latest and greatest.

Tablets and smartphones will be dying slowly too in a few years to be replaced by some other idea.

Definitely. Performance requirements do justify what you're suggesting. Preprocessing when possible is a great principle to follow.

The point is that you're keeping your original data around is the key part.