Show HN: Datastringer, a tool for hacker-journalists | Hacker News Reader

Show HN: Datastringer, a tool for hacker-journalists

bbc-news-labs.github.io

60 points5c2v18 comments

There's another open-source tool named Huginn[1] that does something like this. You write "agents" that perform a given task and then pass the resulting onto other agents that do things like run stats, send emails, etc.

[1]: https://github.com/cantino/huginn

Thank you for the link, I'm going to have a look at that.

Two other tools do similar things too:

- Datawi.re, by Annabel Church [1]

- Stream Tools, by the NYT [2]

[1]: http://datawi.re/ [2]: http://nytlabs.github.io/streamtools/

I got my first reporting job after writing a bot to scrape the SEC for filings by major financial figures.

Data journalism is very sexy! Unfortunately it's still very hard to find that story needle, and increasingly so when you open up the data aperture.

Hey, well done! Got my first after putting together immigration stats from many European countries and making a nice viz out of it.

Hello,

Nice job but why do you require postfix?!

You should be using either sendmail/mailx or a custom user/password SMTPD server (e.g. google's smtpd).

sendmail and mailx require a running SMTP server on your computer, which we chose to be postfix.

To actually send the mails, we use the aptly named 'nodemailer' module, with the 'smtp' transport mechanism, which also relies on a running SMTP server. (btw, one can use a 'sendmail' transport mechanism).

It looks like a meta Google alerts, but it's open source and hackable, which can be really handy. A documentation of how to write custom alerts would be useful, I guess.

Also, you should host a demo somewhere.

Yeah, we thought about a demo. It was kind of hard to picture a way to demo it, given that you can't really see it in action. Any suggestions welcome.

Re: Google Alerts - you're right... to a certain extent. Here, you can mash up datasets, compile them, tweak them, THEN and only then compare them to parameters you set. Alerts++, then.

How about a screenshot of the alerts configuration page, and the screenshot of an alert, in your mail client?

A very large part of the work is configured by the user, and the "alerts configuration page" is not yet functional.

Here is a screenshot of the alerts I received this morning: http://imgur.com/vG0PqDf

The alert was set for London Euston area.

I will add an issue to the repo because, as you see, the styling is not very pretty - and the wording is far from eloquent!

The install procedure is a bit complex, this should just be done with a "npm install -g ..".

Especially the install.sh only works on Linux.

Otherwise very cool idea :-)

Hehe, yes, there's an opened issue about the install process on Mac - we couldn't perform tests over the weekend!

Cron wouldn't work on Windows either, by the way.

Yes, we are still a bit new with Node and the best practices when it comes to installation process. I went for what came in mind immediately, but there's obviously room for improvement. Any thoughts or comments are welcome! Thank you :)

PS: I just created an issue on github to adress your remark: https://github.com/BBC-News-Labs/datastringer/issues/11

Feel free to contribute :)

As a person with a penchant for data, but with limited coding experience, I agree.

I have hard time reconciling:

"Don't know how to code? That's fine."

With:

"The code is available on Github, and that's where you should start. Just clone the repo and run the installation script located at root."

I think you seriously overestimate the layman's abilities here. Is this for programming laymen? Github is one of the last things people learn, for better or worse.

Intriguing, nonetheless. Will follow.

I just created a Terminal.com snapshot with DataStringer ready to use. You may want to give a try at https://terminal.com/tiny/eFW4xKrxZJ

Funny, it talks about being easy, but I had a hard time finding a snippet of the real use.

Or why you shouldn't do the doc last minute! Any suggestions here are very welcome.

To have a look at some snippets, I would recommend going through the wizard, then cracking open one of the examples. The plumbing is a bit overly-complicated, though, I am sure there is a way to make queries in a simpler way.

Reply on news.ycombinator.com