Hacker News Reader

Comment by kelp | Hacker News Reader

I worked at Go2Net when they acquired Dogpile. I was directly involved in migrating it’s systems to our data centers after the acquisition closed.

I don’t really remember the tech stack, but it was something like 10-15 web servers, probably pentium II or III.

One detail I do remember is the original setup had an Apache log rotation script that would try several times to restart Apache and if it didn’t succeed, it would reboot the server. This ran nightly...

Not sure what bug or issue they were working around. But I thought it was quite an aggressive solution!

View on HN

"let's just leave this ugly hack here for tonight and get some logs about the restarts, then delete it tomorrow..."

"oh, we forgot the print. lets retry tonight."

"darn, did not happen. lets leave this part in our infra for now until we trigger the edgecase again."

5 weeks (or days...) later: "damn, why did we do that again? better not touch it"

i love it when i find myself cargo culting code like that and finally have the time (and courage) to remove it :)

Apache still doesn't know how to reopen log files when they're rotated :(

That’s news to me. I’ve been managing Apache hosts for well over a decade and never had an issue. Used apache 1.3, 2.2 and 2.4.

What’s your log rotation code look like?

You have to do a graceful restart after rotation, according to the docs, which then leads to scripts that try a couple times and then reboot. (It's been a while, but I did have an install where graceful restart would work once, but consistently fail on the second rotation).

https://httpd.apache.org/docs/2.4/logs.html#rotation

If you don’t change the inode then no restart should be required. There’s definitely a few tools out there that solve this problem without needing to restart httpd.

How do you rotate without changing the inode?

Sorry, my previous comment was really unhelpful.

What I meant was you should be using pipes and apaches own log rotate code.

Read the “Logging Using Pipes” section at the very end.

https://www.digitalocean.com/community/tutorials/how-to-conf...

(Excuse the DO link, it was just the first result in Duck Duck Go. There’s nothing specific about DO or Ubuntu in this approach)

I’ve used this method on large clusters of very heavily utilised web servers and it works great. No restarts required.

Reply on news.ycombinator.com