Hacker News Reader

Comment by RivieraKid | Hacker News Reader

What technologies / architecture does Amazon use for business intelligence? I've just done a business intelligence course so I'm interested how do more "technology-centered" companies approach BI, whether it's the same thing I learned in the course (put everything into an integrated relational database if I simplify a lot).

View on HN

Lots and lots of SQL (a very good thing IMO). This can come in the form of Oracle, Redshift, or any of the commonly available RDS databases (probably not SQL Server though). This is augmented with a lot of Big Data stuff, which used to be pretty diverse (Pig, Hive, raw Hadoop, etc.) but is moving very quickly towards a Spark-centric platform. There is occasionally some commercial software like Tableau/Microstrategy. Apart from that, it is a whole lot of homegrown technology.

As far as architecture is concerned, every team is different. At least on the retail side, we tend to center around a large data warehouse that is currently transitioning from oracle to redshift, and with daily log parsing ETL jobs from most services to populate tables in that data warehouse. Downstream of the data warehouse is fair game for pretty much anything.

Would you know why sql server generally isn't used?

Probably because:

* SQL Server is proprietary

* SQL Server licenses are expensive

* SQL Server runs only on Windows (or least used to?)

I chuckled a bit because the GP mentioned Oracle. I'm sure you've heard this one:

Customer: "How much does an Oracle database license cost?" Oracle Rep: "Well, how much do you have?"

That's actually not a joke. Oracle liked to argue you should accept server licences as being a fraction of your budget, so when it went up you'd automatically pay them more.

Amazon actually does allow SQL Server. The poster saying it probably didn't was influenced by the fact that Azure, Microsoft's own cloud solution is an AWS competitor.

To your own points:

SQL Server is as proprietary as Oracle SQL Server is cheaper than Oracle SQL Server is being ported to Linux in 2017 :)

Amazon.com seems to strongly encourage the companies it acquires to use aws. Apparently, woot.com is fixing to rewrite the website in Java or something just so they do not have to use Windows. Apparently, it makes sense to them.

Are you using Re:dash[1]? Seems like a good fit if you're doing lots of SQL and working with many different databases.

[1] http://redash.io/

Do you have somesort of framework for those ETL jobs? How do you handle ETL dependencies/workflow?(=thinking spotify's luigi[1] here)

[1] https://github.com/spotify/luigi

Reply on news.ycombinator.com