Show HN: Real Time Black Friday Dashboard – using Perl, Postgres, Geckoboard

thanksgiving.semantics3.com

46 points·netvarun··6 comments

I like the layout and information overview. However, I'm disappointed clicking on items doesn't give me more information or send me to the relevant external page. It looks good and runs well, I would just like a little more interactivity.

Like the others, I too find the layout and info overview is neat. Good use-case demo of real-time information scraping of the sale.

Also, like a few others, the "Perl" in your title is very interesting for me. As a Perl old timer working with a team that hasn't dabbled in any Perl at all, this would be a wonderful, wonderful hands-on demo of what Perl can and does do.

Is there a detailed technical write up on your blog, on how you did this in Perl? Is the code open-source, so I can show them with examples...etc?

Any more details about how it was built?

Geckoboard part was relatively easy - they have a nice interface to design beautiful looking dashboards.

Main magic was in the data pipeline. We tagged products that are Black Friday specific as seen on the merchant specific "Black Friday" category pages and other aggregators. These sites' pages were checked and the products' prices updated more frequently than our regular pipeline.

Everything ran as part of the Semantics3 infrastructure but at the last step we piped the updated information of the Black Friday specific products to a separate Postgres DB as well and that allowed us to run fancy queries that could answer almost any kind of questions we have about the data.

I'm interested in knowing more about the Perl component. What can you say about that?

Our entire data processing pipeline is actually written in perl (that is several 10s of thousands of lines of code). So is the subcomponent that actually pipes the data to Geckoboard.

Reply on news.ycombinator.com