Hacker News Reader

Comment by pauldix | Hacker News Reader

I'm one of the committers. The project is still early stage. At this point we're looking for feedback on the API, which we're planning on finalizing this month. Would love to hear about anything you'd like changed or added to the API.

View on HN

I wouldn't use this bastardized SQL dialect. SQL comes from relational databases, which comes from relational algebra, which is an exceptionally poor model for time series data. It's going to end up a mess and confuse people. SQL is already a mess by itself.

I would simply use functions and operators over time series or data frame types. Perhaps take a look at the R zoo library for examples of more advanced things people do with time series.

We opted for the bastardized SQL because we thought it would be easier to understand. Of course, that may not be the case which is why we'd like to hear what people think of it. We based our decision on people using our API in Errplane and not understanding some parts of it without a lot of additional explanation. One of our users said about one part of it "oh, that's just like group by".

I'm curious, did you find the SQL dialect readable and understandable?

I personally like the simplicity of the SQL dialect. It's a universal way to query data every programmer is familiar with it. Alternatively using functions and code to query data easily gets very complicated.

I really like SQL - as I was learning SQL, I thought it was terrible, but now after years of using it really makes getting to data easy, I'm very excited to see you choose SQL instead of JSON or something less query like to query...

... but that's sort of a stockholm syndrome; it's because you know SQL and it makes you feel comfortable, and you can think of worse options (like JSON).

But SQL is really a bad option in the moodern world, especially since (I estimate) about 90% of SQL statements are built programmatically; Thus, a query language that is easier to construct from code makes a lot more sense. (And not, that's not JSON - some form of algebraic notation or LISPish notation makes much more sense).

Also, SQL semantics are horrible if you have order involved (as you always do in time series).

I disagree - it's not stockholm syndrome.. for me it was realizing that SQL actually has solved a lot of common and useful ways of expressing the process of getting to the data you want in the order process you want... simply because we write code to generate SQL does not make it bad... I suspect you also think HTML is bad? Being able to say select x,y,z from table where x=1 is very simple and very clear IMO... I'm just giving you my opinion though... you clearly see things differently :D

HTML is indeed bad as a machine generated format -- which is what it is; e.g. <p>, list items and a few other things don't have a close tag, but most things do.

These things (like SQL) make sense if you assume that the input is (a) written manually, and (b) by people who are not expected to do this "professionally". Neither is the case of HTML nor SQL anymore.

(Seriously, SQL was originally marketed to managers with the idea that "it's just plain english so you can do it yourself, and don't need programmers!". You know how well that worked out)

> (Seriously, SQL was originally marketed to managers with the idea that "it's just plain english so you can do it yourself, and don't need programmers!". You know how well that worked out)

Pretty well, actually -- lots of nonprogrammer analysts use SQL for queries, and IME the ones that do consistently are better able to answer questions based on data than the ones that use "friendly" query tools, that inevitably end up being much more limited in practice, and requiring a lot more support from both programmers and DBAs to make the data that is already available accessible through.

Unfortunately, lots of environments prevent direct SQL access to DBs for "security" reasons (as if mutiuser DBMS's didn't have role based access controls as a core feature)

I still think SQL is a needlessly verbose mistake. The same people who can successfully write SQL for queries would have just as easily (or more easily) been able to use some algebraic notation. I am not advocating GUI query builders - I'm advocating a non-natural-language-looking (and hopefully better) language, along the lines of kdb+/q.

If you can properly do inner/outer/cross/asof joins to get to the data you want, the english-like syntax is just a burden - two queries that seem similar in their English more often than not produce completely different results because of SQL's 3-value logic, the way NULLs are joined, and various other things like that.

> I still think SQL is a needlessly verbose mistake. The same people who can successfully write SQL for queries would have just as easily (or more easily) been able to use some algebraic notation.

I don't think that's not true -- there's lots of adults that have anxiety around "maths-like" notations largely as a result of issues with maths education, cultural factors, etc., despite being able to intellectually handle the relevant manipulations -- and lots of those people end up in non-technical business positions that end up having to deal with data. Lots of the non-IT people I've seen using SQL definitely fall into that group, and I don't think they'd be as proficient with a more algebraic syntax like the comprehension syntaxes used in many modern programming languages.

Conversely, the people that can be proficient in those syntaxes almost certainly can be proficient in SQL, though they may complain about its verbosity.

Sure, in a perfect world where the cultural context was different, this wouldn't be necessary. But we don't live in that world.

This is a really strange opinion. SQL is an awesome DSL for interacting with structured, relational data. Got no idea where you get your 90% stat from; seems to me that's a product of your personal ecosystem rather than an objective statement about the universe.

Actually, it's nice to have both. you may have something already that is doing a lot of queries and this way you can swap this database in and not have to change the queries much.

reminds me of epl http://esper.codehaus.org/esper-4.2.0/doc/reference/en/html/...

I'll second what Chubot said that I'm not liking the SQL much.

The interface I'd like would be something close to numpy (or matlab/r, if those are more familiar). Let me do vectorized operations and write functions in code. Let me load a few different timeseries into a dataframe. Most likely the easiest thing to do would just be to either embed numpy into your engine, or create simple wrappers to load data to it.

The numpy style interface might be interesting. The problem is that it would probably kill performance to marshal into a Python process. Unless there's some way to embed numpy?

Supporting custom functions is definitely something we want to do.

Why would it kill performance? All you need to do is this:

    double* values = (double*)malloc(sizeof(double)*num_of_values);

(If you are unfamiliar with numpy, it's just a python wrapper around raw blocks of memory.)

You still have to get the bytes out of the server and into the Python process.

Assuming Python is not running in the same process as InfluxDB, amend my code snippet:

    double* data = shmget(key, sizeof(double)*num_data_pts, whatever_flag);

But directly embedding a Python interpreter is pretty easy in C, so I imagine it should also be fairly easy in Go as well.

(Of course, given that my code snippets are C, you can probably deduce I've never written any Go, so take my comments on how easy it is with a grain of salt.)

Go interop with C is very doable (although I've never tried it). Not sure about shared memory WRT garbage collection though.

A route to this would be providing an FFI for loadable modules that provide custom procedures or functions.

Folks that wanted to could then write Python/Numpy code for these and use Numba+LLVM to compile them.

It could be quite performant and avoid having to marshal data or do IPC, could possibly avoid even copying data in some cases (Numpy/Numba have pretty robust support for structures coming over an FFI, not sure about FFI in Go.)

This looks awesome. Interestingly, I started writing something similar to this myself not too long ago. I was using DynamoDB as the backer though.

I think some more aggregate functions would be useful:

- Count (not listed on the Functions page, but used in the query examples?) - Sum - Standard Deviation - First - Last

This project looks really, really promising, thanks for working on it!

How robust/scalable in your opinion the backend is at this stage? I'm just trying to set my expectations properly when checking it out.

Thanks, Sasha

We're writing the clustered portion of it right now. That won't be available until December, but we'll have performance benchmarks on a variety of configurations.

The single node performance at this point for writes is tens of thousands of points per second if batched, and for reads we haven't optimized yet. Queries that only have to go through a few hundred thousand points should return in < 1s. Of course, those numbers will be highly variable depending on if you're writing a series with 1 column or 30 columns. We'll be adding things to make that better.

For now we're focused on creating a developer friendly API and building out clustering.

That makes total sense. Thanks for the info!

Reply on news.ycombinator.com