Hacker News Reader

Comment by thibaut_barrere | Hacker News Reader

As always, if you know other open-source OLAP or data-processing stuff, I'd love to here from you.

Have a look at knime.

http://www.knime.org/

Check out Mondrian: http://mondrian.pentaho.com/ I used it a couple years back at a startup. It's written in Java and IIRC works (only?) with MySQL. Mondrian took a bit of work to get setup, mostly due to my lack of OLAP knowledge at the time, but once setup, it was pretty nice and fast. I think it uses materialized views for the cube data. I'm actually working on a new project where an OLAP will be nice. Thanks for posting Brewery, I'll definitely give it a spin. OLAP's don't get much love, but are very useful for certain types of problems.

We use mondrian currently. It supports just about every database that is supported by JDBC (we use it with PostgreSQL). Just an FYI.

I will definitely look at it soon, thanks!

One thing I wonder is how easy it would be to integrate this with a MongoDB or Redis backend.

Other useful links I found on my quest:

- https://github.com/rsim/mondrian-olap (jruby olap queries on mondrian)

- http://www.slideshare.net/rsim/multidimensional-data-analysi...

- http://www.amazon.com/Pentaho%C2%AE-Solutions-Intelligence-W...

A few questions:

  * What kind of limits and performance does this implementation have?
  * Will data be fetched from the database for each query?
  * Is it possible to have dimensions with millions of
    values and expect reasonable query times?
  * Looks like it supports advanced topologies and hierarchies.
    How will dimensions with a high carnality affect performance?

I have no idea (I just came across this link and thought I would share).

If you find out, I'd like to know too.

See my post about the projects: they are very young, just little over half-year old - performance was not focus yet. I would definitely have them to be able to handle more data more efficiently, however, goal more on simplicity of use than on ability to process really huge amounts of data (like telco data - background where I come from).

Before I answer your questions (I assume that you are referring to Cubes - OLAP framework), I think it would be good to note, that Cubes has pluggable backends. Currently simple denormalisation-based SQL backend and MongoDB backend are implemented. I want to have them more advanced.

* Will data be fetched from the database for each query?

- currently yes, however we did some experiments with plain HTTP caching of Cubes/Slicer server and it worked pretty nicely for our current needs

* Is it possible to have dimensions with millions of values and expect reasonable query times?

- not tested yet

* Looks like it supports advanced topologies and hierarchies. How will dimensions with a high carnality affect performance?

- right, it supports hierarchies, however same as above: not tested yet for performance

I am open to any commeents/suggestions regarding the framework(s).

Stefan Urbanek, @Stiivi on Twitter (author of Cubes)

Reply on news.ycombinator.com