Hacker News Reader

Comment by lucaspiller | Hacker News Reader

One thing this glances over is that you should have something monitoring your production systems to make sure that they are running correctly.

To start with get something to monitor errors/exceptions and email you. To name a few services:

https://airbrake.io/

https://rollbar.com/

https://github.com/errbit/errbit (can be hosted on Heroku for free)

Also make sure that you have accessible logs that log useful information (timestamps, the user making the request, unique request ID). Then use syslog or a SaaS service to aggregate logs from all servers in one place, and keep them for as long as you can.

View on HN

Great point - a small note though, for those of us who don't like using other people's hosted services and have some of our own infrastructure to host - use Zabbix, Nagios, Cacti, Spiceworks, Munin, PRTG, etc. Lots of nice options. I know everyone loves cloud services these days, but I feel absolutely no temptation to put any of my business infrastructure in other people's hands when I don't absolutely have to.

I use a combinations of Zabbix and a very agressive Smokeping mesh deployed with docker (10 pings per minute from all separate DCs all other DCs) to monitor our worldwide nodes and be able to account for routing problems on the backbone rather than our nodes themselves. Very handy tool, particularly if you work with latency sensitive applications. I've been surprised some of the datacenters we use appear not to have access to similar...

I wouldn't describe Nagios a 'nice' option. Every time I have to install or reconfigure it, it opens the wounds afresh (and I only have a relatively small deployment) :)

Go for Sensu, you wont look back and it has terapeutic powers, when it comes to monitoring ;)

Hard to have a discussion about open source monitoring without mentioning OpenNMS. Best monitoring back end by far.

https://getsentry.com/ is another option (my favourite) for exception monitoring.

It's also open-source https://github.com/getsentry/sentry

We use it for GitLab.com and love Sentry.

One of my clients got Graylog up and running pretty quickly.

Logs are only one part of monitoring though, and it's easy to miss the wood for the trees if you're drinking from the logs firehose. Some metrics monitoring is also vital, naturally I recommend http://prometheus.io as I'm one of the core developers :) https://blog.raintank.io/logs-and-metrics-and-graphs-oh-my/ discusses this a bit more.

I've used https://opbeat.com/ for a small side project and been fairly happy - they track exceptions, releases and performance.

Another error tracking system for your list: https://bugsnag.com/

It has been rock solid for us with no event limits and cheap pricing.

(not an employee, just a fan)

Same here, we just started using it and are happy with it!

The ELK stack is also good but will require some config - you'll need to pay for a subscription for alerts however there are free alternatives like Yelp's ElastAlert on Github.

Personally I found the EHK (ES, Heka, Kibana) stack much nicer than ELK. I especially like that each node can run Heka locally and ship directly to ES without having some intermediary server we have to maintain. On top of that, Heka is great for general event streams and can grow quite a bit more into your infrastructure as a general tool than logstash (at least last time I used logstash, which was a few years ago). Heka can also monitor and send you alerts via email ("Hey this server got 3 exceptions in the past 5 minutes!") on certain conditions.

We do use ES as a core part of our infrastructure, so our only real barrier was setting up Heka.

> keep them for as long as you can.

Careful, you can run into Data Protection legal issues with that approach.

Data is a liability as well as an asset.

The operator of Pinboard says that companies should treat personal data like radioactive waste. You should not keep it around, or keep as little as is required/possible, and keep it secure and away from everything else.

It's a great way to think of it.,

Shameless plug, but my co-founder and I are about to launch our powerful logging service this month. One of our features is session-specific logs and the ability to send alerts at user-defined thresholds.

We'd love to get some feedback on our product when we launch to beta, so if you're interested, please sign up for our email list at http://logdebug.com

One suggestion is remove the "Most Popular" banner above Small Business on your pricing. Being as your site is plastered with under construction, it's a turnoff seeing such false advertisement.

Sure, good tip.

Though the pedantic butt-head in me wants to point out that out of all of our paying customers, they're all using that level of service. :) So technically...

So keep it. Ditch the under construction stuff instead if you have customers.

Let us know when it's actually out. This is a big pain of mine right now but frustrating I have to sign up to be notified. That pain might cease to exist when you're actually ready

Your website is still new, I see, so I imagine it will be more explanatory about how your service works. ;)

One thing about the plans that made me cringe was that "SSL" encryption was not available to the Startup plan. I feel like we're entering an era where HTTPS isn't really something that should be offered as a perk, ya know?

I'll vouch for Airbrake. It does an excellent job of identifying patterns and grouping similar errors, tracking resolution across deploys, recording request context to help you figure out exactly what was going on while avoiding spamming you with error messages.

I haven't had much of a need to look at logs since I started using Airbrake three years ago. For the price, it's really hard to beat the value when you compare the price to various log aggregation systems.

A great list of resource - glad I ran into this post! I have rolled my own log scraping and exception monitoring and notification, but some of services do a lot more.

I have been using https://www.pingdom.com for uptime monitoring and some basic perf/latency tracking. It works well and doesn't cost a lot, plus it can hit my services from multiple regions.

Let me start with a disclaimer that I work for Atatus. Front-end monitoring is the other end of the circle, that complements production monitoring. With Atatus that is what we have done. We monitoring the front-end for both errors and performance issues - https://www.atatus.com/

> timestamps, the user making the request, unique request ID […] keep them for as long as you can

That's a sure way to go to jail.

Don't store which IP requested what for more than necessary — in many countries, there are limits if a few months on how long you may store metadata.

Can you elaborate? I find this incredibly hard to believe.

If it's true, you'd think it would be widely-known and publicized in those countries (whichever ones they are) but I can't find any reference to such a policy existing anywhere.

Germany (and probably other countries n the EU as well) considers IP addresses PII, and you have to have consent to store it. Short-term logs might be justifiable as technical necessity (but the lawyers I've heard talk about it didn't want to put a "safe" time frame on that), but certainly not "keep as long as possible".

That's the reason Google Analytics, Piwik and others have an option to blank out parts of the IP address.

Jail is a bit dramatic though (at least here), unless you do something bad with them fines + maybe damages would be the maximum, and it is quite unlikely you'll actually get hit. But if you are ever in a situation where this would come to light (= already in legal trouble), it could make things a lot more complicated, especially if you process data for other companies.

It's commonly known in business. I've heard of it repeatedly, usually in terms of "here's what our retention policy has to look like for European customers."

It falls under personal data about someone. Many (but not all) EU countries consider IP addresses to be personal information, and hence "Store it forever for no real reason" is illegal. You need to have a legitimate, legal, reason to store PI, you need the person's unambiguos freely given consent, etc.

There is currently a court case as to whether dynamic IP addresses from ISPs counts as personal information. But static IP address probably count. And all the other information (user X logged in at time Y and send message to other user Z) would be PI too.

> That's a sure way to go to jail.

Not in (any of) the United States.

It could still be good practice to minimize all data when it doesn't serve a purpose, and I can't think of a reason not to delete the last byte of an IP address (except possibly in an internal setting where all clients are from the same subnet).

Personally, I don't see much of a problem when my IP is stored somewhere since you'd need a reference, but others have apparently thought more about it and want it to be anonymized.

(I've now come up with scenario: If you have access to multiple such sets of data, you could connect accounts from different systems. I. e. when the youporn logs leak, someone could tie the data to your HN account etc.)

> (I've now come up with scenario: If you have access to multiple such sets of data, you could connect accounts from different systems. I. e. when the youporn logs leak, someone could tie the data to your HN account etc.)

Combine that with leaks of a social network where you have real names and IP addresses, and you could actually identify the name, job, address, etc of everyone who watched a specific clip of porn.

That’s very bad.

Metadata. Not even once.

And that's why Safe Harbour was found illegal, and why it's probably illegal for EU companies to use those US services.

Reminds me of that scene in Snatch where Dennis Farina says "yeah, don't go to London."

Don't make any plans to have a physical EU footprint and you can care less what the EU thinks of your internal policies. EU customers can take it or leave it. Their choice. It's not a warm stance to take, but you've probably got a dozen other balls in the air that you're juggling. EU data compliance probably isn't one your worried about as a small startup.

On the other hand, if you comply with EU data policies, your customers might choose your app over another app that doesn’t

(especially european users, and businesses)

Of course. Such are the decisions and tradeoffs that any young startup needs to make. That's why I said that they could take it or leave it.

Too late to edit, but the actual quote was, "Yeah, don't go to England."

syslog and SEC is a good place to start before deciding to ship all your logdata to a 3rd party.

Reply on news.ycombinator.com