petrohi

When semantics are very different and syntax tries to pretend they aren't to create false familiarity this will lead to some very painful brain-splits later on. I can assert from my own experience with alternating the use of = between Erlang and C++.

Glad you liked the blog.

It is interesting to look at virtual machines from the prospective of what they enforce. The JVM was a huge step forward when it allowed for safer memory management. Garbage collection is possible precisely because JVM enforces its memory model. You cannot use pointer arithmetic and arbitrary casts. This helps to ensure the level of ecosystem and make developers more productive. Erlang VM enforces immutability at its core. We can call it safer concurrency management. We can combine hundreds of libraries and still maintain this safety. I would argue that if something is not enforced in the VM, but rather a is "good guideline", the ecosystem descends to the lowest safety that is enforced.

Immutable message passing is an abstraction. In a optimized virtual machine it may well be implemented by leveraging cache coherence. The sender process writes memory block. This places it into its local cache. Then the receiver process reads this memory block. This uses cache coherence to sniff the block directly from the sender's cache. This is awesome way to leverage parts of cache coherence protocol that play well with immutable memory blocks. It can also scale well if sniffing is not a broadcast. Cache coherence can quickly get ugly when a cache line is contested for writes--be this false sharing or a spin lock. My point I guess is that immutability and message passing promotes awesome parts of cache coherence and limits the need for the ugly parts.

There is still great chunk of goodness in the Erlang VM, which is not bound to Erlang language and can be further reused. Much like JVM having things like Scala and JRuby.

One of the huge hurdles for me personally was Erlang's heritage of Prolog syntax. I actually gave up once, and only cleared it second time around. Now I can see certain elegance and succinctness in the syntax.

We are planning to use pgpool2, but only for connection pooling. In our case the subset of postgres databases to execute a distributed query is dependent on the input parameters, current sharding configuration and cached routes. Cached routes are used to handle adding new shards without repartitioning--I will cover this in the second part.

I guess we could use pgpool2 parallel queries, but this would likely require some adjustments in how they are implemented.

Yes, I can see where you're coming from. Message queue in the article is more of an abstract service that provides certain guarantees and less of a specific RabbitMQ implementation. There are examples of message queues implementations on top of database--the resulting tradeoffs can be an interesting discussion.

It is not the number of tables that requires sharding, but the amount of data and associated workload.

While automatic sharding definitely has its place, we wanted to be very explicit in our consistency tradeoffs.

Good explanation on wikipedia:

http://en.wikipedia.org/wiki/Garbage_collection_(computer_sc...

+1 on that.

Yes, type=json|jsonp|xml|csv would work. We would just need an extra parameter for JSONP to specify the name of the callback function.

Consideration for not using HTTP status codes was that they are not working very well with JSONP-type integration. This is when page JavaScript adds "call" script object to DOM and then expects call back when it's loaded. To pass error information in this case we need HTTP server to return 200.

> Your API is pretty strange. Some initial things that pop out on me: > json=true to specify the content type. Ideally, this should be from the accept header, but at the very least it should be possible to only specify one content type. Right now, I can specify json=true&xml=true.

I agree, content type selection currently is very rudimentary and ability to combine parameters could be confusing. We want to distinguish between JSON, JSONP, XML and CSV. We could use text/plain MIME type for CSV, text/xml for XML, but are there standard MIME types for JSON and JSONP?

> Only using GET.

API is currently immutable towards analytics data, so only GET is used.

> Session management on the client. Why would I want to log someone out?

It's less of the session management and more security token management. Log out is to revoke specified security token. We should be more explicit about this.

My experience with nolock is that you may get inconsistent rows with some fields before and some after update. Or even seeing duplicated rows when b-tree is rebalanced. But I never seen single fields being partially updated. Per Microsoft, page latch protects atomicity of a single field update. This is why nolock was extremely useful for insert-only tables and in our database design we had many of them.

In Microsoft SQL Server row/extent/table locks are of transactional semantics and often turned off with nolock option. What really matters for concurrency is page latch, which is per 8K page.

In SQLite readers actually do block writers by default. Writing transactions are committed with lock escalation steps. First shared lock is acquired, then reserved, then pending and finally exclusive. Pending blocks new shared locks and waits until all in-flight shared locks are released. Again this is default rollback journal behavior. As of 3.7 write-ahead log allows readers to be concurrent with writers, but AFAIK is still rarely used.

Certainly. We have the ability to attach custom attributes to events and then segment by them [1]. We envision both backend and frontend ways to attach attributes. With frontend they could be set by JavaScript in the page. In backend we have plugins for popular platforms in works.

[1] http://infunl.com/#split

Target user is professional analyst. Someone who is hired or is consulting the organization to provide "data science" services. We believe that this user becomes increasingly technical with abilities to create workflows with multiple tools and data sources to get insights. We hope that our service will become indispensable tool for such user.

Yes, they indeed provide comprehensive UI to slice and dice the clickpaths. Our goal is to build flexible and fast API to enable creating custom reports and dashboards.

We also envision extensibility with custom functions and datasets so people can do just about anything with their clickpath stream, including mashing it up with their data.

Yes, visual query editor is a good idea, we will try to add it.

We hoped that people will try out examples from the front page to get used to the language. Also writing small tutorial is in plans.

In our opinion, current state of the art practice with analytics APIs is to extract semi-processed data out and do more massaging locally with tools ranging from Excel to SQL databases and Hadoop. This could be dramatically improved by providing new higher level API with SQL-like declarative syntax specifically designed for analytics. This would outsource hard problems of event collection and big data processing to the API provider, while giving analyst enough power to quickly build reports and live dashboards with potential to mashup other data sources. Infunl is building full fledged tracking service, API and query language.

By standalone you mean installed on your servers?

Great that GA recognizes importance of visual representation of flows. Although for sites with loads of dynamic content where URLs include one or more ids you would see picture segmented by content. Unless you instrument your site by features. Project we work on allows for quickly finding patterns of URLs that correspond to features while visually showing the top flows: http://infunl.com/screen/0

Would it be cool if the tool could detect/suggest/visualize such emerging funnels based on aggregation of all clickpaths?

post updated with clarification of replication mechanism and related consistency issues