How do you guys log application events in such a way that extracting information from it is easy, but still keep the size of the logs manageable?
Application logging has been a solved problem for decades now. syslog or direct-to-disk in a reasonable format, let logrotate do the job it's faithfully done for years and let the gzipped old files get picked up by the offsite backups that you're surely running, and use the standard collection of tools for mining text files: grep, cut, tail, etc.
I'm a little weirded out that "my logs are too big" is still a thing, and that the most common answer to this is "glue even more complexity together".
grep, cut, tail, etc. work quite well if you're working on a single machine or small number of machines.
The "ELK stack" (ElasticSearch, Logstash, Kibana) is a step up in complexity but gives you much more power than command line tools.
There are also some great commercial solutions that abstract away some of that complexity if you don't feel like rolling your own (Scalyr, Splunk, SumoLogic, etc.).
But regardless of the path you take, don't reinvent the wheel!
(Disclosure - I work for a company that provides one of the commercial solutions above - https://scalyr.com)
I am a big proponent of JSON-based, semi-structured logging. Most of log data today can be parsed with reasonable rigor at source, and doing this before shipping data to the backend saves so much future agony.
I recently blogged about this here: http://radar.oreilly.com/2015/04/the-log-the-lifeblood-of-yo...
That kind of fractal complexity -- complexity added to every single layer of software architecture -- is a real pain in smaller environments where there's just one guy responsible for figuring out what the heck just went pear-shaped in the application. (Although, the real world problem I've had more often, as that guy, is nonexistent logs...)
Ideally, I'd like to see both purposes handled -- applications logging semi-structured JSON with a utility that's real-time translating the output to a human readable format, or, preferably, vice-versa, with applications logging unstructured data to a file that's then being munged into JSON (or whatever) for machine consumption.
I'm not such a fan of tools that are intended for humans to mine structured logs as though they were text files, because it's too easy to miss important data.
It's a security pattern to have the logs shipped to another host in production anyway.
There might be a good reason why they don't do it that way and I wouldn't have a clue.
The most commont logging formats I've come across in production environments are:
1.log4j(java) or nlog(.NET)
2.json
3.syslog
Tools that I've used to search ,visualize and analyse log data have been:
1.Elasticsearch, Logstash and Kibana (ELK) stack
2.splunk (commercial)
3.Logscape (commercial)
Changes to the fields representing your data with the database approach is expensive because you are locked in by the schema. The database schema will never fully represent your full understanding of the data. With the tools I've mentioned above you have the option to extract ad-hoc fields at runtime.
Hope this helps.
Seq is great because you can set up your own instance very near to your servers for low-latency/high-bandwidth logging, which really changes the game in terms of what you can feasibly (perf/financially) log. It also has some decent visualization options, and it's got some great integrations, with a decent plugin architecture to create your own real-time log processing code.
Logg.ly has some amazing GUI/search options.
Log.Info("Customer# {customerNumber} completed transaction {transactionId}", customerNumber, transactionId);
Then using the Seq log viewer you can simply click on "transactionId" in the log line and filter by "transactionId = 456" or whatever. It's one of the most exciting advancements I've seen in the .Net logging world.
EDIT: I realized I didn't really answer OPs question regarding space. If you used Serilog, you can set up different sinks to export to, with different options. For example, you could send all your logs to mongodb, and just recent 1 week rolling logs to the Seq server.
Log.Info("customerNum={customerNumber} transactionId={transactionId} state=completed", cN, tID)
Log.Info(LogState.Add("state", "completed").Add("customerId", customerId).Add("transactionId", transactionId));
I guess the version that works best depends on the tool that is consuming the log text.
Right now at work everyone just logs to a single CSV with an inconsistent format and it makes me cringe every time I look at it. It's also really difficult to parse.
Your current way does sound like a headache. If your logging lines are in the standard NLog format, you should be able to drop in serilog without many changes.
Using a system like Ganglia (or the Etsy inspired statsd) is an important idea since the OP's original question included how to limit the size of logged data. These systems provide a natural way to aggregate data.
The question is, what information do you want available within the second about logs from a year ago? Aggregate that, and move the rest.
Just as an example of how awesome Elasticsearch is, you can trivially segment your storage tiers (say, SSD versus HDD) and then easily move older data to other storage, with a single command.
They have a log-specific handler called Logstash, and a dashboard system called Kibana (which is sorta neat but the UI seems a big laggy in my brief experience). Apparently some folks use Logstash/Elasticsearch to record millions and millions of events per day and ES does a great job.
If you want hosted, check out Stackify. I'm totally blown away with the product (no affiliation other than being a new user). You can send log info to them and they'll sort it all out, similar to Splunk, but not ridiculously priced and no dealing with terrible sales teams. But it gets better - they offer all sorts of ways to define app-specific data and metrics, so you can get KPIs and dashboards just adding a line or two of code here and there. It's a lot easier than running your own system, and it looks like it can make ops a ton easier.
Another hosted service is SumoLogic. I only used them for logging, but it seemed to work well enough.
At my current employer, we are currently implementing Splunk. And it's taking forever and they do charge arm and a leg for their offering. I don't mind if a good product costs money but you shouldn't need a consultant on premise just to configure your logging solution.
- Make frequent visits to answers.splunk.com. It has a very active community, and I've frequently been able to type "how do I do X in Splunk" into Google and found multiple answers on Splunk Answers.
- Deployment Server. Make friends with it. In a perfect world, it should hold your configurations for all Indexers, Heavy Forwarders, and Forwarders. If you're having to populate $SPLUNK_HOME/etc/system/local/ yourself, you're doing it wrong.
- Make friends with the "splunk btool config_file_name list --debug" command. That makes it dead simple to know which configuration options a Splunk install is running. Append "| grep -v system/default" on the end of that command to filter out the defaults and you'll more easily see which of your options are being used.
- If you have the cash, attend Splunk Conf and load your schedule up with presentations. It's worth every penny.
Hope that helps.
We use Splunk for logging and I wrote custom app for fraud detection in financial security field, custom events correlation and alerting.
It's very good tool for enterprise data analytics as well as for any custom dashboarding and event processing.
Splunk consultant maybe needed if you have complex enterprise deployment scenario or wish to develop really advanced apps - but configuring logging solution?
It's a point and click exercise.
It was cool software, a bit slow (this was several years back). But with things like Elasticsearch catching up release over release (even if you don't use it directly, other platforms will build on it), Splunk is no longer the totally unique thing they used to be. I can't figure out their $8bn+ market cap with revenue under 500M and costs increasing as they grow.
Apart from that, Stackify is $15/server/mo for the server monitoring and app metrics, IIRC.
I used graphite and now I'm using influxdb, in the other hand kibana+logstash+ES.
With statsd and influxdb you can measure all the events in a database, it's pretty easy and you have statsd libraries in some languages. I measure all the events in my products, from response timings, database queries, logins, sign-ups, calls, all go to statsd.
Logs are good to debug, but if you want to measure all events in your platform, statsd+influxdb+grafana are your best friends, and your manages will be happy with that ;-)
A few weeks ago I gave a talk about this, you can see the slides here + a few examples+ deploy in docker:
http://acalustra.com/statsd-talk-at-python-vigo-meetup.html
Regards ;-)
I have a HAProxy instance I want to log into something like influxdb.
So in Heka set up an input (LogStreamer for example), and use that haproxy decoder script (as SandboxDecoder) with it. After that pass it through any filters you'd like (like StatFilter) and output collected stats with HTTP output + influxdb encoder.
(I just built a log parsing and stats collecting pipeline for our Heroku apps with haproxy + heka + influxdb + grafana. So far happy with the result.)
No :-(, all the software that I use is python, and Kamailio servers. So I didn't have this problem. I can't help in this case.
Sorry
Logstash + graylog / elasticsearch - mostly for monitoring application error logs and easy ad hoc querying and debugging.
statsd+graphite+ nagios/pagerduty - general monitoring/alerting and performance stats
zeromq (in the process of changing now to kafka) + storm and redis for real time events analytics dashboards. We are also writing it to hdfs and running batch jobs over the data for more in depth processing.
We also have a legacy sql server in which we save events / logs which is still maintained so maybe this could help you. Just FYI we analyse more than 500 million records / day and we had to do some optimisations there:
-if the database allows then partition the table by date. -create different tables for different applications and / or different events -1 table / day which is then at the start of the new day getting merged in a different monthly table in a separate read only database. -create daily summary tables which are used for analytics -if you actually need to query all the data then use union on the monthly tables or the summary tables -I want to also say this, I know it's a given but if you have large amounts of data batch and then use bulk inserts..
I suggest you take a couple of steps back and think hard about exactly how you want to access and query the data and think what the best tool for you in the long run is.
If log entries take up too much disk space, switching to a different system will not help; you will have to do something with the data. You can either archive old years (export in some way, compress, put in cold storage) or throw them away, either partially or fully (do you need to keep debug logging around forever?). Using partitions can help here, as it makes it faster to drop older data (http://www.postgresql.org/docs/current/interactive/ddl-parti...)
You also may consider compressing some fields inside the database (did you normalize logType and userAgent or are they strings? Can your database compress descriptions?), but that may affect logging performance (that's a _may_. There's extra work to do, but less data to write)
If, on the other hand, indexes take up too much space or querying gets too slow, consider using a partial index (http://en.m.wikipedia.org/wiki/Partial_index). You won't be able to efficiently query older data, but if you do that only rarely, that may be sufficient.
If you want to use your logs for troubleshooting (e.g. ad-hoc queries to find error messages) or ad-hoc analytics it is ideal. Hundreds of gigabytes can be searched or analyzed in 5-6 seconds per query.
Fluentd can be used to collect log data and send to BigQuery.
Custom NLog renderer which implements SysLog protocol and NLog target which pushes logs to RabbitMQ.
Do you persist sessions in some global cache (We'd use Oracle coherence at my work, but I guess you could use redis/memcache/infinispan?). Side note...does anyone have experience with Infinispan?
I might be biased, though, against all the idiotic uses of session that just tend to wreck the user experience. It's not specific to any type of technology, I suppose. Maybe it's just being stateful, overall. (Looking at my bank which can't handle multiple tabs doing transactions at once, or Lenovo which will silently corrupt the displayed pricing info if you price two machines at once, or the new Azure Portal, which manages to display the wrong data even with a single tab.)
At some point, with a complex enough application, reconstituting all of the required session data and state for each request becomes a massive performance bottleneck, and stateful sessions solve that very nicely. Most app tiers support session failover pretty nicely, and many clients I've worked with have logic where if the session doesn't exist (i.e. if their session's sticky server has crashed or been removed) then reconstitute the state based on a cookie value. Either approach gives you the best of both worlds - the performance of stateful sticky sessions, and the cluster flexibility of stateless.
But with simpler applications, with less session data/state to manage then stateless is probably fine.
But reconstructing session from cookie value would work just as well I suppose, at the tradeoff of having everything you want stored in cookie (vs storing just session ID in cookie and having that ID mapped to data in one of the aforementioned caches or other alternative)?
It's similar in spirit to elasticsearch + logstash + kibana, but more integrated.
Disclaimer: I work on it, so I'm not going say what's better, just giving another pointer.
2) hbase+phoenix: http://phoenix.apache.org/
3) opentsdb: http://opentsdb.net/
• open source solution require a lot of work
• commercial solution get very expensive very quickly
If you can narrow down how much logs you want to keep, then the commercial solutions are amazing, but as you need (or think you need) to keep them longer and longer, they become prohibitevely expensive.
The next time I have to tackle this issue, specifically keeping the log forever, I will give the hadoop stores (HBase, Impala etc...) a try. Hadoop solutions work really well for very large set of write-once only data, which is what logs are.
Just use grep to query recent logs, zgrep if you have to dig a little.
zgrep somepattern /var/log/messages*After a week, it goes out of cache. After a month, we no longer keep multiple copies around. After 3 months, we gather stats from it, and push it to some tar.xz files, which we store. So its out of the database.
We can still do processing runs over it, and do... but it is no longer indexed, so they take longer.
After 3 years, the files are deleted.
The important feature for us is S3 archiving. They'll keep your logs online for a certain period of time, and then copy the old ones to S3. You don't have to get rid of anything, and you're still able to keep costs under control.
It works well for us, nice accessible UI if you need it and a solid database behind it. Also RSS/Email alerts if you need it. We've got thousands of entries in there and even on the old SQL2005 box we use, it seems to work just fine.
Like others here said, extract what you want to keep (unique visitors per day or so) and throw the rest out after a few weeks.
Winston is a node/iojs library though, but I guess you could find something equivalent in any other stack. The Pushbullet part is really useful.
Edit: I run a pretty small site however (http://littlist.no). I don't think I would enable the Pushbullet part if I had several hundred thousand visitors per day.
Why are you keeping all of the logs? Are you doing anything with it?
Are the old logs relevant at all? If your program structure has changed, then anything logged before that point isn't even applicable.
My advice: If what you is working, but only failed because of volume of data, apply a retention policy and delete data older than some point in time.
An example: Nuke all data older than 1 month for starters, and if you find that you really don't use even that much (perhaps you only need 7 days to provide customer support and debug new releases) then be more aggressive and store less.
Blog: http://engineering.indeed.com/blog/2012/11/logrepo-enabling-...
Talk: http://engineering.indeed.com/talks/logrepo-enabling-data-dr...
tl; dr: a human-readable log format that uses a sortable UID and arbitrary types/fields, captured via a log4j syslog-ng adapter, and aggregated to a central server for manual access and processing
We have web server logs going 30 days back, on disk, managed by logrotate. Then we have error logging in Sentry. For user level events, we track in Analytics, but we also have our own database-backed event logging for certain events. Currently this is in the same db as everything else, but we have deliberately factored the tables such that there are no key constraints/joins across these tables and the rest of the schema, which means it should be trivial to shard it out in its own db in time.
For performance metrics we use graphite/stats-d This allows us to log hits/access times for many things, all without state handling code inside the app.
This allows us to get rid of a lot of logs after only a few days. As we're not doing silly things like shipping verbose logs for processing.
However in your usercase this might not be appropriate. As other people have mentioned, truncing the tables and shipping out to cold storage is a good idea if you really need three years of full resolution data.
That's what I would do.
> However, after three and a half years of continued use, the table is now way too large.
Yeah, that's what happens...
There are many ways to handle this issue. The simplest is to start archiving your records ( i.e. dumping your old records into archival tables ).
Do you have access to a DBA or a data team? They should be able to help you out with this if you have special requirements.
With splunk, you just output your logs in this format:
<timestamp> key1=value key2=value key3=value
install splunk agent on your machines, and splunk takes care of everything from there. You can search, filter, graph, create alerts etc...
Splunk indexer allows you to age your logs, and keeps the newer ones in hot buckets for fast access.
Disclaimer: I work at splunk.
Here is a good presentation of Fluentd about it design and general capabilities:
https://www.youtube.com/watch?v=sIVGsQgMHIo
note: it's good to mention that Fluentd have more than 300 plugins to interact with different sources and outputs.
Disclosure: I'm on of the co-founders. We've a couple of other related tools in the pipeline, but the BF remote logger was the first we built, mostly to solve our own need at Mobile Jazz.
I agree with many comments that this isn't ideal, but setting up weekly/monthly partitions might buy you plenty of time to think through and implement an alternative solution.
We use them for all our apps and have not seen any issues so far. It can be a bit tricky to set up, but once the logging works, it's hassle free from then on. Pricing is also very affordable.
Also NewRelic if you want to spend the money (or get it throuhg Amazon/Rackspace for free)
This is very convenient for decently complex querying and analysis at great speeds.
This has the benefit of making logging more or less asynchronous. You still need to handle the logs coming out of this, of course.
If you go this route, use a capped collection. I generally don't care about my old logs anyway.
If I had a lot of logging to do though, I'd use elasticsearch since that's what I run for my main DB. It handles sharding beautifully.
For instance, we use a log file for HTTP access logs but I store all of errors and warnings from MongoDB. However, I clean the log storages every month.
We use NodeJS and MongoDB in www.floatalk.com
Works in cloud. Easy to setup and very scalable.
Free tier: 500 MB/day, 7 day retention
Disclosure: I work there.
As it grows into a seemingly useable feature, I might move it to GA or Mixpanel.
When it gets to be large and stable, then it goes into syslog
At least this is what we're moving to at work.
Very easy to setup.
You never fill table with non capped/infinitly growing records (capped = collections with an upper limit).
You use at best rotating collections (like circular buffer ring). But anyway, if you have success the log flow should always grow more than your number of customers (coupling) thus, it grows more than linearly. So the upper limit will grow too.
Tools have software complexity in retrieving, inserting and deleting. There is not a tool that can be log(n) for all cases and be ACID.
The big data fraud is about letting business handling growing set of datas that are inducing diminishing returns in OPEX.
In software theory the more data, the more resource you need that is a growing function of size of your data. Size that grows more than your customers, and linearly other time.
The more customers you have, the longer you keep them, the more they cost you. It is in terms of business stupid.
Storing ALL your logs is like being an living being that refuses to poo. It is not healthy.
Solutions lies in sampling or reducing datas after an amount of time and scheme like round robin databases.
That's gold.