> it wasn't as simple as it says
mind elaborating? we built loki for some pretty massive scale but I've always tried to make it work at super small scale to. what went wrong?
589 karma · joined April 23, 2015
mind elaborating? we built loki for some pretty massive scale but I've always tried to make it work at super small scale to. what went wrong?
Continuous profiling is a next big thing IMO - easier to get started with than distributed tracing and delivers immediate value.
Do you mind if I ask what isn't super-easy about linking self-hosted loki search queries with SaaS-Prometheus? You should be e.g. able to add a Prometheus data source to your local Grafana (or securely expose your Loki to the internet and add a Loki data source to your Cloud Grafana)
There are a bunch of other reasons why people might choose Mimir; perhaps they have out grown some of the scalability limits, or perhaps they want faster high cardinality queries, or a different take on multi-tenancy.
Do remember Cortex (on which Mimir is based) predates Thanos as a project; Thanos was started to pursue a different architecture and storage concept. Thanos storage was clearly the way forward, so we adopted it. The architectures are still different: Thanos is "edge"-style IMO, Mimir is more centralised. Some people have a preference for one over the other.
It doesn't have to mean the end for Cortex, but others will have to step up to lead the project. We've tried to put other maintainers in place to kick start this.
Thanos is an awesome piece of software, and the Thanos team have done a great job building an vibrant community. I'm a big fan - so much so we used Thanos' storage in Cortex.
Mimir builds on this and makes it even more scalable and performance (with a sharded compactor and query engine). Mimir is multitenant from day 1, whereas this is a relatively new thing in Thanos I believe. Mimir has a slightly different deployment model to Thanos, but honestly even this is converging.
Generally: choosing Thanos is always going to be a good choice, but IMO choosing Mimir is an even better one :-p
Grafana Labs is hiring! Come work on Grafana, Prometheus, Cortex, Loki, Tempo and more - lots of opensource source, with both a SaaS and an enterprise team. We use Golang, JS, Typescript, Kubernetes, Jsonnet, Tanka, CUE and more.
We're growing fast, have lots of happy customers and many exciting projects in the pipeline. We need good engineers to help us take Grafana and Prometheus to the masses.
Not to bash Jaeger though, its more powerful than tempo in that it allows you to search for traces. Tempo is about integrating with Grafana, Loki and Prometheus for finding traces.
The local Cortex storage works pretty well but we have a very high bar for production worthiness. Right now I'd recommend using Bigtable of DynamoDB, and if you're on premise Cassandra. In the future the block storage will allow you to run minio.
Its like looking at the module interdependencies of reasonably large piece of software; of course its going to look complicated.
No! Prometheus is and has been production ready for many years. Cortex is a clustered/horizontally scalable implemention of the Prometheus APIs, and Cortex has just gone production ready. Sorry for the confusion.
The biggest take home here - and the first thing the post mentions - is the a single HA pair of Prometheus servers is enough for 80-90% of people. TLDR you probably don’t need Cortex (or Thanos, etc)...
...unless you run multiple, segregated networks (regions). Then something like Thanos (or Cortex) is useful - not for a the scale argument, but because you need a way to “federate” queries and get that global view. IMO!
(Its something I'm really keen on myself - We use jsonnet internally to version control our dashboards and load them into config maps in our kubernetes clusters)