Internals of Cubes – Lightweight Python OLAP Framework | Hacker News Reader

Internals of Cubes – Lightweight Python OLAP Framework

blog.databrewery.org

46 pointsstefanu11 comments

On a related note. I was involved prototyping olap cubes (Microsoft stack) at my dayjob. HUGE success in terms of value added to the business. Sr Management was used to getting stale data (at least 24 hours old) of things like trading positions, profit/loss, alpha/beta/gamma for portfilios, etc. in Excel pivot table canned reports. We turned that into same-day, build-your-own reports in your web browser.

My group spun it off into a separate project group that took it all the way to effectively-realtime views of the entire company's money. The project rapidly became 100% mission critical and the related display framework absorbed just about every other reporting tech in the entire company. (This was in a 9billion/yr trading company with a few hundred employees)

So, if you're a pythonista looking to make yourself more visibly valuable to the management of your company. Consider taking something like this technology and running with it.

Seconded - I work mostly with < 100 employees companies and with Ruby, but I'll give the same advice.

There is plenty of very useful and fun projects to complete in this area (50% of my yearly income as a freelance comes from data processing, etl, datawarehouse etc).

In addition to that, I've started to write a JavaScript library for the Slicer server (not mentioned in the blog post). I am just starting with JS, so pardon my style. The sources can be found here: https://github.com/Stiivi/cubes.js No examples yet, however the goal is to be able to browse aggregated data directly from JS and transform them into tables/charts.

Concerning backends: I have not much time to write other backends, however, if anyone is interested in helping me with them, just drop me a line. I would like to have at least nicer star-schema browser and perhaps the mongo DB backend.

Very nice. I did a day project a while ago to do something basic. http://www.binarymax.com/cube.html (NOTE: doesnt work in firefox, but is fine in all other browsers).

I'm pretty sure my algorithms would quickly run into stack and memory issues quite quickly, but I havent looked at it in 6 months.

How long have you been studying OLAP structures and algorithms? Its the kind of stuff that is very interesting, but I sometimes have trouble wrapping my head around. I'm curious if you have any favorite reading materials on the subject?

EDIT: also FYI, your link on the blog to the github js repo goes 404 since it has an extra '.' on the end

For a good description of OLAP data structures and algorithms, I'm a fan of _Data Mining_ by Jiawei Han and Micheline Kamber

Thank you.

How long? I was working for around 5 years in (later with) a data warehouse in a mobile telco company, now I am doing "data brewing" as a freelancer. My favourite book is Star Schema The Complete Reference by Christopher Adamson. Very very brief introduction to OLAP and Cubes you can see in my slide deck: http://www.slideshare.net/Stiivi/cubes-7781602

Thanks, fixed the link.

A similar project is Saiku ( http://saikuanalytics.com/ ), for the front-end they use saiku-ui. https://github.com/OSBI/saiku-ui It's one of the most popular niche-sounding areas to be in right now.

Very interesting, thank you. There is a big opportunity for someone to re-imagine the power of Lotus Improv with a more intuitive touch interface for tablets. Perhaps it will be me, but I'd rather it be you than that it not happen.

Data analysis as a service could be a big market. Why do Google Analytics, Mint, my time-tracking service, my to-do list and every other online app I use implement their own, custom, proprietary versions of the table/sort/pivot UI for advanced use-cases? I'd rather they provided clean interfaces for easy things and allowed me to join all their data together into one data cube for serious reporting.

Thank you.

There are couple of analytical services out there, as SaaS, that can connect to various third-party sites, databases or other data sources. However, what is missing is some nice, reusable, simple UI set of elements for pivoting and drilling down. The idea is that apps can use Cubes as module for providing aggregation and drill-down capability. Then you have a front-end module, either within the app or any external, that can connect to the Cubes OLAP API and you can do reporting as you like.

I'm not too much front-end guy, so any help would be appreciated. As mentioned in one of my other comments here, I've started https://github.com/Stiivi/cubes.js . Just proof-of-concept.

That's interesting because I happen to be something of a front-end guy :) I will have a look at it when I have a chance.

Have you seen polychart? They are providing an ad-hoc data visualization layer in javascript.

Reply on news.ycombinator.com