Heroku Gets Sweet Logging | Hacker News Reader

Heroku Gets Sweet Logging

blog.heroku.com

88 pointsjkvor24 comments

$ heroku logs --tail

! Realtime tail and filter are not available with basic logging, please upgrade to expanded logging

You need to install the addon and plugin, instructions are at http://docs.heroku.com/logging

I did. That's how I got the original output.

Until the Advanced functionality is rolled out publicly, does anyone have suggestions for logging all requests to something like S3? I was going to write my own this afternoon since I couldn't find anything.

I see the market currently providing two options, neither of which encompass my use case.

1) Exception Logging via existing Heroku addons like Exceptional

2) Tracking via javascript, e.g. Google Analytics

The problem I'm experiencing is that I have successful ajax calls to a Sinatra app that I'd like to log. Running 'heroku logs' will give me the last couple hundred but I'm generating many thousands. The new new Heroku logging functionality will expand that tail to showing me a few thousand queries across my dynos but as I understand it, it won't write out to S3 (nor a file since Heroku is write-only).

I planned on writing a custom logger that will output to S3. If anyone has suggestions in lieu of rolling my own, please advise. Otherwise I can post the code I create on github for others in my situation.

You can use MongoDB Logger which works pretty good https://github.com/peburrows/mongo_db_logger

Although I'm not sure if its still being maintained.

central_logger is the new mongo_db_logger: https://github.com/customink/central_logger

We use it in production on heroku and it works great.

S3 is not the best AWS for the functionality you seek: http://aws.amazon.com/simpledb/usecases_logging/

my favorite: 2010-10-21T10:11:16-07:00 app[web.2]: Company acquired /_party (0.1ms)

Only a partial party?

I guess they would be re-using it multiple times, hopefully...

and it didn't last for very long ;)

That's great. My first attempt at using heroku was frustrated by their not-so-sweet logging. I'll have to try it out again for my next project

http://en.wikipedia.org/wiki/Inner-platform_effect

I understand it seems witty article to link to, but it's not giving the Heroku guys much credit. It seems that they ran into some fundamentally difficult problems while trying to implement a pleasant logging system - I'm inclined to give them the benefit of the doubt for a number of reasons, but I know that anyone can fall into NIH trap.

What, then, would you propose as a solution, given the nature of Heroku's architecture and customer base?

Unix

Doesn't the second paragraph under examples more or less describe the motivation behind NoSQL?

In the database world, developers are sometimes tempted to bypass the RDBMS, for example by storing everything in one big table with two columns labelled key and value

If so, maybe this is not always an anti-pattern.

Prolly not the place to ask this...but I will go ahead anyway...I recently heard that if you have a Rails app that will be accepting files from users (e.g. say images/videos being uploaded) Heroku is not the best host to use.

Anyone have any thoughts/experiences with that? Is that complete rubbish or is there truth to it and why?

AFAIK, if you have a Rails app that will be accepting files from users, Rails is probably not the best thing to use. Ruby likewise, short of using something evented (and I say this as someone thoroughly enjoying working with both Ruby and Rails.) Seeing as you'll be tying up processes rather quickly that way, no?

That aside for the moment, and based on your S3-related comments nested below, I thought I might pop by to mention that I (using Rails on Heroku) have been handling uploads by using direct pre-authorized posting to S3 and it has been very nice to work with.

What I am handling is basically what Basecamp is handling. If Ruby and Rails can manage Basecamp, then it sure can manage what I am doing.

It's not so much the S3 storage I am concerned about. It's more the allowing me to backup everything properly, concern I have.

Regarding backup, I definitely understand the feeling.

If you're using something like Passenger, I believe it does the upload in nginx/apache to a tempfile and then hands it off to Rails.

Thanks for that bit of info. At the moment things like Passenger are obscured from my view/concern, as I'm not managing the servers myself, but should that change I'll definitely keep that tidbit in mind.

I'm guessing they said that because Heroku doesn't let you save files to their system. You have to use something like s3. Other than that, I see no reason why someone would say that.

You can save things to the tmp directory for a little while. The files should last long enough to do some work on them. For example, I've had an app parse an uploaded XML file that gets saved in tmp. This might not be a good practice, but I don't feel like setting up anything special for that app.

I use the paperclip gem and store files on S3 with my Heroku apps and it works great. I believe this approach is very common.

I too am implementing paperclip...but I am wondering about backing up the uploads. How do you backup those uploads? I was thinking that I would store the first copy on the host - then push them to s3 for backup (and maybe 1 other CDN, just to be ultra paranoid).

But, it seems to me, that if you are just passing through Heroku altogether - that removes one point of fault tolerance and redundancy - right?

Reply on news.ycombinator.com