Think Prometheus, but for logs (not metrics). Simple, efficient, fast log store
parseable.io
parseable.io
But also, running on your laptop isn't really the intended use case.
Wait, am I reading that right? Does this mean by default the docker container is going to store my data in a random place on the internet?
If so that's a crazy bad default.
No bueno.
It is there for demo purpose ig
It also gives you this warning when you run the build
Warning: Parseable server is using default credentials. Setup your credentials with P_USERNAME and P_PASSWORD before storing production logs.
Parseable server is using default object storage backend with public access. Setup your object storage backend with P_S3_URL before storing production logs.
I believe they have good intentions and are just trying to make it easy to try out, but this is just too big a footgun hanging out. Makes it untouchable for me.
As Developers, SRE, DevOps look to replace Elastic from their logging stack - we believe the log storage of future won't be another search and indexing engine in a new language.
Parseable leverages Apache Arrow, Parquet and widely available object storage platforms for efficiency, cost effectiveness and performance.
https://github.com/metrico/qryn
Aside - with so much to choose from wrt logs and metrics - I'd assume "prometheus for logs" implied "pulling" log data (rather than push). While maybe "Victoria metrics for logs" might imply "easier to operate, efficient, fewer moving parts".
I also have seen a trend over the years of companies who adopt AGPL not caring about the open source community around the code. This is not specific to the AGPL, and not every user of it does so, but I see it correlating more often.
Many companies refuse to use AGPL licensed code... some claim their lawyers are stupid and don't know what they are doing... but it can still prevent adoption of AGPL licensed products.
So, it means that I don't get to use awesome looking toys at work because no one is going to pay the lawyer fees to find out what, exactly, we can and can't have those network requests touch
I have been trying to be much more careful in commentary about this because I'm not trying to change the author's license, it's their code, and I'm not trying to get into debates about "well, it only matters if ..." because IANAL and I don't want to assume the risk, because for this situation, being wrong would be very expensive, so I just stick to Apache or MIT tools
My comment wasn't a call to action, it was just crying out loud
I imagine you've seen apenwarr's blog post on logging: https://apenwarr.ca/log/20190216
With that in mind I'd like to read more about the architecture. Specifically:
1. Do you have separate components for ingest and querying that can be scaled independently?
2. How dumb is the log ingest path?
The only use I can think of for historical logs is security auditing. In which case there is only a subset of logs you really need to trace a breach and they don't need to be searchable online -- you can pull them back from cold storage if you need to check for an incident.
Moving and storing logs and making them all instantly searchable is awfully expensive for little benefit.
Edit2: Disclaimer: This opinion does not apply to government regulated services like financial services.
We put a big emphasis on the second S in SaaS and we want a magical 1-email support experience whenever possible. Audit logs help this.
Depends really how long your customers can wait.
They're particularly valuable for production issues. If something has gone wrong and alerts are pinging around, I appreciate the chance to figure out what it was from what the logs are telling me, rather than having to turn the logs on and try to trigger the same problem again deliberately in a production environment!
Edit: just realised who I was replying to. Netflix and Reddit might be very different environments with very different reliability requirements to a lot of other types of software.
As I mentioned in my post, that's the one case where I find value in temporary turning on logging, but having old logs around is awfully costly for the off chance that they help.
> tracing individual resources/requests through a bunch of microservices to better understand some unique situation or behaviour,
That's request tracing, which is different. Just storing a request trace is useful, but that's not what I mean by logging. I'm talking about server application logs.
> identifying whether a particular code path is actually in use and if so how heavily
Again that would be request tracing, not application logging.
> They're particularly valuable for production issues. If something has gone wrong and alerts are pinging around, I appreciate the chance to figure out what it was from what the logs are telling me, rather than having to turn the logs on and try to trigger the same problem again deliberately in a production environment!
If the problem doesn't happen again after you turn on logging, then the incident is over isn't it? The customer is no longer affected. If they are affected, then your logs will keep showing it.
from what ive seen people who rely on logs do not have proper request tracing and/or metrics. its much simpler to dump a bunch of unformatted junk to stdout and throw elasticsearch at it, but not a good experience.
i agree overall, i find logs to be a last resort with really bad signal to noise ratio.
> Having old logs around is awfully costly for the off chance that they help.
Depends massively on the log volume. I've worked at places that generated terrabytes of logs over the course of a few days, and required very large ES clusters to try to keep on top of it. There was definitely value in having those logs, but it only made sense to retain them for a couple of weeks. It's much less costly for low volume services.
> ...request tracing, which is different...
Ye-es, until it's not. If your current setup isn't yet developed or grown enough to have full request tracing, you can get a long way filtering aggregated logs by a correlation ID. I know it's no substitute for a fully feature tracing system, but in situations where you have limited resourcing you need to prioritise it can get you a long way
> If the problem doesn't happen again after you turn on logging, then the incident is over isn't it? The customer is no longer affected.
That massively depends on the impact of the incident, I think. If it's a "There was a problem playing this title, please try again" kind of deal, it's no biggie.
If it's a "my broadcast TV channel is off air" or "my financial system has crashed in an unexpected way trying to reconcile this transaction" type situation, you might be a little more worried - and the customer is likely going to be very upset for quite a while, and some people with very serious faces might come to talk to you about very large fines.
More mundanely, if I get paged about an error rate, the first thing I'm going to do is look at the logs to see what the errors are.
Our logs tell me the state of the application when the error occurred without having to reproduce it. They also tell me how many customers are having the issue, so we can set the severity of the incident at the appropriate level and reach out to those customers proactively if needed.
Sure, we might be able to do this with metrics or tracing instead, but that's often just duplicating information we're already logging. For example, by adding a correlation ID in my log metadata, I suddenly have end-to-end request tracing. It's not as pretty as what New Relic can produce, but it definitely works.
We still did rely on logs to troubleshoot and understand some issues.
- [0]: https://sentry.io
Written in Rust.
GNU Affero General Public License v3.0
There will always be potential for a 'better mousetrap' to take some market share from existing solutions, but are current ones bad enough that a great solution could come in an capture significant market share? Or are today's systems 'good enough' that any new solution would face huge headwinds, even if it has some cool features?
The reason I ask is because I am building a new kind of general-purpose data management system (https://www.Didgets.com) that can handle all kinds of data. It does file data, relational tables, Key-Value stores and other kinds of NoSql data. I did a little proto-typing of it managing some simple logging data and it seemed to work really well (in my opinion). But I would need to do a bit more work to get a good demo going.
I have limited resources, so I would only want to do that if there is a significant demand in the logging space for something new. What is the biggest 'pain point' for people working with log data today?
Oh, I can help you with that one: don't make it require 5 separate Ph.Ds in everythingology to deploy and run the PoS
Then, for an extra $100k, have the new mythical company actually be responsive to bugs reported instead of "well, sucks to be you"
Then better is faster, less annoying, easier to integrate, better UX, etc., even if it sits in the same spot in the workflow.
There is a market, but it's saturated with offerings. There are so many solutions out there right now for metrics and log aggregation.
>There will always be potential for a 'better mousetrap' to take some market share from existing solutions, but are current ones bad enough that a great solution could come in an capture significant market share? Or are today's systems 'good enough' that any new solution would face huge headwinds, even if it has some cool features?
I'm going to take a guess and say the latter. There are a lot good(enough) off-the-shelf and managed services, that another offering is going to have a very hard time breaking through.
>What is the biggest 'pain point' for people working with log data today?
Ingestion, transformation, indexing, querying, visualization - those are all pretty much solved problems and everyone has a good enough solution. For me, it's specialized/niche use-cases and workflow, where the gaps are. For example, is your logging managed service able to be a drop-in HIPAA compliant solution (are individual customer logs physically segregated at the host level? Do you support automatic detection of Personal Health Information? Do you have an audited workflow to remove or deidentify, in batch, the offending logs. Do you support specialized access and dashboards to my customers so that they can see metrics and logs for their site? Do you support air-gapped, enterprise, on-prem deployments?).
1/ I know that Parquet and Arrow are very good for time series (mostly metric names, labels, timestamps and real values), but are they good for strings, which are quite important in logs?
2/ Is DataFusion able to execute an SQL request over a large number of large Parquet files without loading everything in memory at the same time, or can it process a Parquet as it reads it, in a streaming fashion?
Are you forming a company around this and do you plan to monetize it? I would find that information helpful to know whether I should consider it as a candidate for a tool in my day job, or disqualify it for my hobby projects.
If you have to describe your project as "some other project but like this" you're already doing it wrong. Do you think Bing markets its maps by saying "Think Google Maps But Microsoft"?
"Think Google Maps but for the grocery store (not roads)" would be more apt.
> Loki is a horizontally scalable, highly available, multi-tenant log aggregation system inspired by Prometheus.