Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD
serhack.me
serhack.me
We (I work at TimescaleDB) recently announced that multi-node TimescaleDB will be available for free, specifically as a way to keep investing in our community: https://blog.timescale.com/blog/multi-node-petabyte-scale-ti...
Today TimescaleDB outperforms InfluxDB across almost all dimensions (credit goes to our database team!), especially for high-cardinality workloads: https://blog.timescale.com/blog/timescaledb-vs-influxdb-for-...
TimescaleDB also works with Grafana, Prometheus, Telegraf, Kafka, Apache Spark, Tableau, Django, Rails, anything that speaks SQL...
Previously at NGINX I was part of a team that built out a sharded timeseries database using Postgres 9.4. When I left it was ingesting ~2 TB worth of monitoring data a day (so not super large, but not trivial either).
Currently I have built out a data warehouse using Postgres 11 and Citus. Only reason I didn't use TimescaleDB was lack of multi-node support in October of last year.
I sort of view TimescaleDB as the next evolution of this style of Postgres scaling. I think in a year or so I will be very seriously looking at migrating to TimescaleDB, but for now Citus is adequate (with some rough edges) for our needs.
I am biased obviously as a contributor. Having said that I think it's always worth understanding active/passive type replication and the implications and see how other solutions handle this scaling and reliability problem to better understand the underlying challenges that will be faced with instance upgrades, failover and failures in a cluster.
My current use case isn't monitoring, or even time series anymore, but will keep M3DB in mind next time I have to seriously push a time series/monitoring solution.
We were successfully ingesting hundreds of billions of ad serving events per day to it. It is much faster at query speed than any Postgres-based database (for instance, it may scan tens of billions of rows per second on a single node). And it scales to many nodes.
While it is possible to store monitoring data to ClickHouse, it may be non-trivial to set up. So we decided creating VictoriaMetrics [2]. It is built on design ideas from ClickHouse, so it features high performance additionally to ease of setup and operation. This is proved by publicly available case studies [3].
ClickHouse was certainly something we evaluated later on when we were looking at moving to a true columnar storage approach, but like most columnar systems there are trade-offs.
* Partial SQL support.
* No transactions (not ACID).
* Certain workloads are less efficient like updates and deletes, or single key look ups.
None of these are unique to ClickHouse, they are fairly well known trade-offs most columnar stores make to improve write throughput and prioritize high scaling sequential read performance. As I mentioned before, the amount of data we were ingesting never really reached the limits of even Postgres 9.4, so we didn't feel like we had to make those trade-offs...yet.
I would imagine that servicing ad events is several factors larger scale than we were dealing with.
The main issue I've got is how to actually get data into TimescaleDB. We use telegraf right now, but the telegraf Postgres output pull request still hasn't been merged: https://github.com/influxdata/telegraf/pull/3428
Any progress on this?
If you are looking to migrate data, then you might also want to explore this tool: https://www.outfluxdata.com/
[1] https://medium.com/@valyala/promql-tutorial-for-beginners-9a...
It's the main reason we decided to use influx in our small team with simple enough timeseries needs
I eventually spent a while reading about Timescale 1 vs 2, and testing the pg_prometheus[1] adapter and started thinking through integrating its schema to our other needs then realizing it's "sunsetted" and then reading about the new timescale-prometheus[2] adapter and reading through its ongoing design doc[3] with updated schema that I'm less a fan of.
I finally wound up mostly-settling on Timescale although I've put the Prometheus extension question on hold, just pulling in metrics data and outputting with ChartJS and some basic queries got me a lot closer to done for now. Our use case may be a little odd regardless, but I think a timescale-prometheus extension with a some ability to customize how the data is persisted would be quite useful.
[1] https://github.com/timescale/pg_prometheus
[2] https://github.com/timescale/timescale-prometheus
[3] https://docs.google.com/document/d/1e3mAN3eHUpQ2JHDvnmkmn_9r...
Because of that and the small scale required, the choice of table-per-metric would be a tough fit and I think a single table with JSONB and maybe some partial indexes is going to work a lot better for us. It would just be nice if we could somehow code in our schema mapping and use the supported extension, but I get it may be too baked-into the implementation.
Anyway, overall we're quite happy with TimescaleDB!
"Why is it difficult, because you're self managed?"
For serious applications, it doesn't cut it. Trying to do more than a few TB a day is a waste of time outside of enterprise, which ain't cheap.
I plopped VictoriaMetrics in place of Influx for my cases and haven't even had a single hiccup.
[1] https://medium.com/@valyala/insert-benchmarks-with-inch-infl...
[1] https://victoriametrics.github.io/#how-to-send-data-from-inf...
If you are looking for a vendor to host, manage, and provide attentive engineering support, check out my company: https://HostedMetrics.com
We are a company that ingests around 90M datapoints per minute across an engineering org of around 4,000 developers. How do we scale a timeseries solution that requires an upfront schema to be defined? What if a developer wants to add a new dimension to their metrics, would that require us to perform an online table migration? Does using JSONB as a field type allow for all the desirable properties that a first-class column would?
> What if a developer wants to add a new dimension to their metrics, would that require us to perform an online table migration?
Specialized time series databases usually don't need defining any schema upfront - just ingest metrics with new dimensions (labels) whenever you wish. I'm unsure whether this works with TimescaleDB.
I see no reason to use it over VictoriaMetrics.
https://medium.com/@valyala/high-cardinality-tsdb-benchmarks...
https://docs.timescale.com/latest/using-timescaledb/compress...
Are you saying the compression shrinks the data down to 2% on average?
If the compression only makes the data 10 times smaller (I think I'm being generous with that ratio), it's still 5 times larger than the others.
Prior to the compression feature, I had the same complaint. Timescale strongly advocated for using ZFS disk compression if compression was really required. Requiring ZFS disk compression wasn't feasible for me.
https://github.com/timescale/timescale-prometheus/blob/maste...
https://github.com/timescale/timescale-prometheus/blob/maste...
That configuration you cite isn’t for the core TimescaleDB time-series database or internals, but to add a specific configuration and setup optimized for Prometheus data, so that users don’t need to think about it and TimescaleDB can work with the specific Prometheus data model right out of the box.
Those “scripts” automatically setup a deployment that is optimized specifically to have TimescaleDB serve as a remote backend for Prometheus without making users think about anything. Some databases would need to write a lot of specific internal code for this; TimescaleDB can enable this flexibility through configuring proper schemas, indexes, and views instead (in addition to some specific optimizations we've done to support PromQL pushdown optimizations in-database written in Rust).
This setup also isn't something that users think about. Users install our Prometheus stack with a simple helm command.
$ helm install timescale/timescale-observability
The Timescale-Prometheus connector (written in Go) sets this all up automatically and transparently to users. All you are seeing is the result of the system being open-source rather than having various close-source configuration details :)
The project you cite is super-optimized for the Prometheus use-case and data model. TimescaleDB beats InfluxDB on performance even without these optimization. It's also not possible to optimize in this way in most other time-series databases.
These scripts also work hard to give users a UIUX experience that mimicks PromQL in a lot of ways. This isn't necessary for most projects and is a very specialized use-case. That takes up a lot of the 2000 lines you are talking about.
Would you mind sharing the schema used for this comparison? Maybe I missed it in your documentation of use-cases. When implementing dynamic tags in my own model, my tests showed that your approach is very necessary.
Pretty much it's a table with time, value, tags_id. Where the tags table is id, jsonb
That may be true, but: Instead of figuring out how to meet the user's needs you're going to say the user is wrong?
What I consider to be said is that they optimize for extendability ( not just one use-case) and that those 2000 lines do a lot of things.
Including replicating optimizations that are are not easy to achieve out of the box in their and other solutions. But they mostly found a way ;)
You can say you don't want to support that usecase but someone inquired about whether you would. You said no, they're wrong to compare the two. Sounds a lot like "you're holding it wrong."
TimescaleDB is a Postgres extension, which means they have to play PG's rules, which rather implies a complicated mesh of functions, views, and "internal-use-only" tables.
But once they're there, you can pretty much pretend they don't exist (until they break, of course, but this is true for everything in your software stack).
Is your complaint targeting the size of these scripts, or their contents? Because everything in there appears pretty reasonable to me.
TimescaleDB doesn't support this out of the box, the reason being Postgres' limitations on multi column indexes when you have data types like jsonb. What they have to do to work around this is to have a mapping table from dimensions/tags/labels which are stored in a jsonb field to integers which they then use to look the data up in the metrics table.
I would expect that a timeseries database is optimized for this type of lookup without hacks involving two tables.
If you have a data model that is slightly more fixed than Prometheus, you'd create a column per tag and have a foreign-key to the labels table. That's a pretty simple schema which would avoid a whole lot of complexity in here. So if you are developing most apps this isn't a problem and filtering on dimensions/tags/labels isn't a problem. Again, this is only an issue of you have very dynamic user-defined tags, as Prometheus does. In which case you can just use Timescale-Prometheus.
Most of these custom databases (Influx) use techniques similar to TDB's under the hood, in order to persist the information in an read/write optimized way; they just don't "expose" it as clearly to the end-user, because they aren't offering a relational database.
A critical difference between the two, is that custom databases don't benefit from the fine-tuned performance characteristics of Postgres. So even though they're doing the same thing, they'll generally be doing it slower.
Gotcha, this makes sense. To this, I'd pose the question: is the same not true for Influx? (i.e. With IFDB, you should be ready to know its ins and outs to secure and maintain it correctly). I guess I think about choosing a database like buying a house. I want it to be as good in X years as it is today, maybe better.
From this perspective, PG has been around for a long time, and will continue to be around for a long time. When it comes time to make tweaks to your 5 year old database system, it will be easier to find engineers with PG experience than Influx experience. Not to mention all of the security flaws that have been uncovered and solved by the PG community, that will need to be retrodden by InfluxDB etc.
Anyways, it's just an opinion, and it's good to have a diversity of them. FWIW, I think your perspective is totally valid. It's always interesting to find points where reasonable people differ.
I prefer older houses (at least pre-80s). They have problems that are very predictable and minor. If they had major problems, they would show them quite clearly in 2020.
My friends prefer newer properties, but this is an act of faith, that the builders today have built something that won't have severe structural/electrical/plumbing problems in 20 years.
Of course, if you only plan on living some place for 3-5 years, none of this matters; you want to live where you're comfortable.
There are always tradeoffs, but in cases like mine, I find the basis of the platform - PostgreSQL - to provide a very useful set of features.
I don't work for any time series database provider.
(If you want to provide any early feedback, please email me at ajay (at) timescale.com)
For now, the recommended approach is to perform continuous_aggregates on single tables and perform joins, order by, and window when querying the materialized aggregate rather than when materializing.
This often has the added benefit of often making the materialization more general so that a wider range of queries can use it.
Does this mean if you were using it with Prometheus you could get around issues with high cardinality labels?
Influxdb has been a real workhorse. We suffered through some of their early issues, but since then it's been extremely solid. It just runs, is space efficient, and very robust.
We almost went with Prometheus instead of Influx (as I said, early growing pains), but I had just struggled through managing a central inventory and hated it, so I really wanted a push rather than pull architecture. But from my limited playing with it, Prometheus seemed solid.
I just hope they learned their lesson for the v2.0 release...
Then try VictoriaMetrics - it supports both pull and push (including Influx line protocol) [1], it works out of the box and it requires lower amounts of CPU and RAM when working with big number of time series (aka high cardinality) [2].
[1] https://victoriametrics.github.io/#how-to-import-time-series...
[2] https://medium.com/@valyala/insert-benchmarks-with-inch-infl...
No thanks. :/
If they've got a thousand of hosts, the costs of the infrastructure itself must dwarf the salary of any developer by orders of magnitude, the salary of a developer is simply irrelevant when it comes to acquiring software/hardware.
This is true, but my comment was an offhanded way to say that my "salary" (as in, the one on my contracts and the one I "see") is less than a month of Datadog for our number of hosts.
As for the rest of your comment, I wish it was true.
Developer salaries outside of the capitals is quite low in Europe, and even inside the capitals only go to "near double"
So, instead of 12x it becomes 6x developer costs per annum, which is a fair whack of money.
For me to justify spending "3-6" peoples worth of money it had better save "3-6" peoples worth of time.
Well, it does in my experience, especially if you have to handle 2000+ hosts, that's some serious infra there, need serious tooling.
May I ask which country is it?
Some industries need a lot of hardware because they crunch a lot of data but they aren't software companies. Think Computer graphics rendering.
Paying a FTE salary for software is crazy for them. I would love to see a ration of developers/infrastructure per industry/company.
Looking at the pricing page the cost $15/instance so $37,500
Middle-of-the-road developer salary is like $35k in most of Europe, outside of the capitals.
Although it does say: "Volume discounts available (500+ hosts/mo). Contact us." at the bottom, so I guess 500 is a lot.
Spending a fraction of that to monitor that sort of infrastructure is absolutely justified. I can tell you from experience that datadog gives discount even for 100+ hosts, I don't know what they can do for 2500, but if it were me I wouldn't accept anything less than 50% off.
Honestly you need to forgot about your salary, it's irrelevant when it comes to running a company. Imagine a driver in a shipping company deciding to deliver on a scooter rather than a truck because the truck is worth more than his yearly salary.
Also, of course their salary is relevant. The cost of an engineer's time is an important factor to consider when making build vs buy decisions. Usually it's one that argues in favor of "buy", but not always.
Add 10% support fees, EBS storage for the OS, bandwidth fees and it's quite a bit more.
I guess the point I am driving at here is that there's such a thing as "business critical costs" (IE: can we ship our product or not) which is the majority of infra costs we have today, and then there's "optimisation costs".
Usually when we discuss things like optimisation costs its along the lines of: "Will this product save us enough time to justify it's expense". Often, sadly, the answer is no.
Terraform Enterprise is an example of a time where we said: Yes. -- because the API allows us to deploy CI/CD jobs which provision little versions of our infrastructure, saving us many man-days of time in provisioning and testing every year.
As eluded to in the sibling thread, there's almost no way that we can save 3 or more peoples worth of time every year, we're 3 people right now and we have metrics collection, log tracing and alerting already. -- so it's a hard sell to the business types.
Monitoring and logging are business critical. It's an integral part of infrastructure and it is very normal to spend 10% there. It's really not possible to operate stably and efficiently at a large scale like that without a trove of tooling.
Tools usually justify their costs by allowing to optimize the infra and helping to prevent/fix outages, though not all companies care about stability or hardware costs.
And it's not a choice of free vs paid. open source software costs a lot of money too, pairs of large instances to run it don't come cheap, they're probably more than a salary too if the company wants to have any sort of redundancy or geographic distribution.
May I ask what do you have for logging? I guess you must be screaming in horror at the price of elasticsearch/kibana/splunk :D
Zabbix is the weak link here for sure, but the monitoring is quite comprehensive.
At a certain scale, rolling your own monitoring and alerting becomes cost effective again as Datadog begins to charge an arm and a leg. I've seen Datadog bills that could easily pay for 2 full time engineers.
Which means the companies are running thousands of hosts. So they definitely need both datadog and a full team of sysadmin/devops/SRE to handle that infrastructure.
Rolling a custom Prometheus / Grafana / Alertmanager setup is not hard at all, more powerful, and it's much easier to do it right from the start.
Disclaimer: I work at Sumo Logic.
It's meaningless to look at the price of Datadog@5 hosts -- at 500 or 5000, you're paying a completely detached number from the website list price, likely a small fraction.
You'll also experience their habit of launching new features, waiting a while for customers to adopt them, then starting to charge extra for them.
Don't get me wrong, Datadog has great products. But they're also great at extracting money from their customers.
We run Prometheus in production and this hasn't been our experience at all.
A single machine can easily handle hundreds of thousands of time series, performance is good, and maintaining the alerting rules is a shared responsibility for the entire team (as it should be).
If you use it as a store for all time series data generated by your business, and you want to have indefinite or very-long-term storage, managing prometheus does become a challenge. (hence m3, chronosphere, endless other companies and tech built to scale the backend of prometheus).
IMO, this is a misuse of the technology, but a lot of unicorn startups have invested a lot of engineering resources into using it this way. And a lot of new companies are using it this way; hence the "one engineer's FT job".
After that, we needed some more effort to scale out horizontally with Thanos, but again, once it's set up, it maintains itself.
You can be small with Prometheus and grow into needing an FTE for it - w/o having the migration hurdle of moving out-source to in-source
I'm now having better results and experience storing data in ClickHouse (yes, not a timeseries dB).
From time to time I also follow what's coming in InfluxDB 2.0 but I must confess that 16 betas in 8 months are not very promising.
It might just be me.
Don't read too much into that, it's more a result of wanting to get testable releases out early and temporarily redirecting engineering resources to take advantage of opportunities that arose in that timeframe than anything having to do with the code of 2.0 itself.
Our Cloud 2 SaaS offering is already running the 2.0 code in production (albeit with changes to support it being deployed as a massively multi-tenant service)
No, the company recognizes that the success of the SaaS is inevitably tied to the success of the OSS product. The only reason we were able to give the SaaS more focus recently is because we already had a working OSS product in the 1.x line that was meeting the needs of existing customers that we were and are continuing to invest in.
Again, just take it as one person's opinion. I can't even grasp how complicated and challenging is to run what you offer. This is strictly user's opinion.
If you're thinking about using Influx for long term data storage, look elsewhere. The company continuously burns customer goodwill by going against the grain, and bucking the ecosystem by trying to control the entire stack.
I appreciate this sentiment. We've been focused on a building a SaaS version of Influxdata and are committed to a paired open source version of that. The open source version has been lagging as we work on the SaaS side.
However, we are committed to shipping a GA version of the OSS 2.0 stack around the end of Q3 that offers an in-place data migration capability from 1.x OSS.
We've spoken about this publicly in other forums. You can google "influxdays London talks" to hear Paul Dix (CTO/Founder) talk more about our OSS plans.
We continue to support all users, including OSS users, in our public Slack and Discourse, as well as Github. We have not "dropped all DevRel" activity.
There are 100k's of happy InfluxDB users -- untold millions of completely open source Telegraf deployments doing meaningful work for people -- integrated into our products and our competitors' products. We make the vast majority of our codebase public under liberal OSS licenses.
And we'll keep listening and learning how to do more.
... off to see if I can change my internal slack handle to @corporatetool. Unless there's a policy against that ;-)
Just struck a nerve that reminded me of some leaders at a previous gig who made things a little less sensible.
I apologise, and you definitely don't deserve it :-(
He was giving you an honest assessment of what was going on, not sugar-coated or wrapped in corporate speak. The work he described on our SaaS offering was directly tied to what our current and potential customers wanted.
I don't think that's fair given they are still a startup.
Now of course our goal is to help everyone upgrade to 2.0 and beyond, but we know that we made a lot of changes and improvements in 2.0, this isn't a minor upgrade. We will focus first on what we need to do to help 80% of our users upgraded, then the next 80%, and so on, until we've got you all covered.
Meanwhile new users can benefit from starting off on 2.0 as soon as it's available (or get the beta which is already out) and not have to wait.
(Source: I'm the Community Manager for InfluxData)
I ended up moving away from it to TimescaleDB and I’m pretty happy so far.
Its easy to deploy and its x-platform. Not sure what are all the comments here about - maybe for really huge loads which I don't have experience at, I usually use separate influxdb per service.
The author compares with other storage engines comprehensively from time to time.
[1] https://medium.com/@valyala/insert-benchmarks-with-inch-infl... .
EDIT: okay, I get it. You don't need 8GB ram. I was just going by the hardware requirements in their docs: https://docs.influxdata.com/influxdb/v1.8/introduction/insta...
Also, the aesthetics of the UX, you see all the graphs in one single page[1], no additional clicks are required - a quick glance with a slow scroll and you can see if there were any unusual things during the last day/week.
[1] - publicly available example, found by googling - https://ansible.fr/munin/agate/agate/index.html
Check out: https://github.com/influxdata/community-templates/tree/maste...
We have a free tier which allows 2GB of data ingest per day (paid licenses are unlimited) which should be more than enough for capturing logs and flows. The resources needed to run Gravwell basically scale with how much data you put into it, but it's a lot quicker to install and set up than something like Elastic, in our opinion (https://www.gravwell.io/blog/gravwell-installed-in-2-minutes)
Edit: it's currently a bit roll-your-own, but we're really close to releasing Gravwell 4.0 which enables pre-packaged "kits" containing dashboards, queries, etc. for a variety of data types (Netflow, CoreDNS logs, and so on)
> Gravwell is developed and maintained by engineers expert in security and obsessed with high performance. Therefore our codebase is 100% proprietary and does not rely on open source software. We love open source, but we love our customers and their peace of mind a lot more!
does that mean you've even rolled your own webserver? Programming language?
Edit: We've had lots of people assume we use Elastic under the hood, so I wonder if that was just a (poorly-worded) attempt to indicate that our core storage and querying code is custom rather than some existing open-source solution.
Especially when it comes to security, I'd like to see the lowest complexity possible. Harden your software instead of feature-fu around. That would be a good USP (I've got the feeling that no vendor has realized this so far - but customers neither did).
Edit: Sorry, I was misreading your comment. Premium is unlimited...I will look into it, thanks. :)
We at Tenzir are developing VAST for this purpose: https://github.com/tenzir/vast. It's still very early stage, but if you're up for trying something new, a lean and modern C++ architecture, BSD-license open-source style, you may want to give it a spin. The docs are over at https://docs.tenzir.com/vast.
It supports full PCAP, NetFlow, and logs from major security tools. There is CLI and Python bindings. The Apache Arrow bridge offer a high-bandwidth output path into other downstream analytics tools.
[1] https://grafana.com/oss/loki/
[2] I'd argue this sort of thing should be published as a metric anyways so you don't have to pull it out of the logs
As for storage, the default is BoltDB for indexes and local file system for the data, but you can also use popular cloud solutions like DynamoDB, etc. AFAIK BoltDB is automatically installed when you install Loki.
The only possible pain point I see for you is that Loki is tailored for Kubernetes. It is totally possible to use it without running a K8 cluster, but you lose some features.
If you want to graph connections from service to service in real time. I've actually never found anything that was capable of doing that, not even paid software.
If anyone has any Nagios questions, I'd be happy to answer them. I'm a Nagios masochist.
(Also, I can recommend Nagflux[2] to export performance data, metrics, to InfluxDB because noone should have to touch RRD files)
[1]: https://lambdaisland.com/blog/2019-08-07-advice-to-younger-s...
It then rots within your infrastructure, because it resists being configured any way _except_ by hand. I've built two systems for configuration-management of Nagios (at different companies), and it's an unpleasant problem to solve.
Prometheus's metric format and query syntax are cool, but the real star of the design is simply this: you don't have to restart it, or even change files on your Prometheus server, when you add or remove servers from your environment.
I'm surprised no one has named Zabbix. Zabbix is way better. I hadn't the chance to use Zabbix past 4.something but it's worth it.
I've been using Prometheus/grafana and frankly the value I see is it's out of the box adaptability at capturing a mutating data source (example: metrics about ephemeral pods Una kubernetes cluster).
This is an extreme oversimplification. The value is not in "seeing" if something is "up or down", the value is in the modularity of what a "service" can mean in the first place (anything you can script -- and the eco-system of plugins is huge), the fact that you don't have to "see" it (because notifications are extremely modular), the fact that escalations of issues can happen automatically if they are not resolved, and the fact that event-handlers in many cases can help you resolve the issue automatically without even having to raise an alert in the first place.
Nagios is a monitoring tool built with the UNIX philosophy in mind, and it's ingenious in its simplicity: decide state based on script or binary exit codes, relate dependencies between objects to avoid unnecessary troubleshooting, notify if necessary (again, with scripts/binaries) and/or try to resolve if configured. It hooks into a server frame of mind very well if you're a sysadmin.
Sure, if you main use case is "mutating data sources" and collecting metrics, any Nagios flavor won't be for you, because it's not what Nagios is made to do. There's a reason it's extremely popular in large enterprises, because it was created for them. No monitoring solution is for everyone and solves every problem.
This read like a joke. Nagios looks like it's from stone age having files in cgi-bin folder with unnecessary complication to installation and management, unless they made it any better at some point.
- Realtime GUI which works with windows 10 (we have a web site and nagastamon) - aggregation of alerts / alert roll up - sharing filters - summary + description - temporary downtime of alerting - message rate suppression (to stop floods) - filtering of columns/ordering etc. - bulk actions for closing alerts
I've come from using: - HPOV (great but can't handle bursts of alerts) - email (everyone has to have filters, can't handle bursts of alerts that well (runs everyone out of email space!), prone to failure/delays due to email - home grown solution
Other than that I don't quite understand the point of your comment, you say you have "severe reservations" but many of the points you list are available even with Nagios core, and most of them are available in other Nagios variants.
Can you use Nagios to stream metrics exported from your applications binary in real time?
For example, can you use Nagios record each http request processed by your application webserver, tagged with http method, code, latency etc?
https://sourcehut.org/blog/2020-07-03-how-we-monitor-our-ser...
In my previous job I had a very clear use-case for not using Prometheus and did for a while use InfluxDB (it involved devices sending data from behind firewalls across many sites). I found it pretty expensive to scale and it fell over when it ran out of storage, which feels like something that should have been handled automatically considering it was a PaaS offering.
https://github.com/go-pluto/styx
This is how I made the plots in that blog post.
[1] https://victoriametrics.github.io/#how-to-import-time-series...
With Vector and Toshi you can kinda (I am not sure Toshi is as mature as Elastic) use them to replace LogStash and Elastic, the missing piece is Kibana
Right now all of my metrics are sitting in a PVC with a 30d retention period, so we're probably fine but for longer term cold storage the options aren't great unless you want to run a custom Postgres instance with the Timescale plugin or something else more managed.
It's as simple as another Prom instance pointing at the "live" Prom instance. You can filter only the metrics you want to keep for a long amount of time, and downsample if necessary (by just setting a higher scrape interval on the "long term" Prom).
Since this "long term" Prom isn't in the critical path you could skimp on processing resources and just give it a big disk as a cost optimization.
If you are on AWS (or equivalent) the storage there is pretty durable. On-prem you can run HA (two instances with identical config).
[0] https://prometheus.io/docs/prometheus/latest/federation/
https://www.robustperception.io/scaling-and-federating-prome...
https://prometheus.io/docs/prometheus/latest/federation/
You can set up a new "cold storage" Prometheus with a longer retention that scrapes select metrics from your regular 30d Prometheus to store for longer periods of time.
VictoriaMetrics pretty much has everything you'd hope for as a Prometheus long term storage, like direct PromQL support, good performance and ease of installation.
[1] https://medium.com/@valyala/promql-tutorial-for-beginners-9a...
I would add to the guide that you need to be careful about formatting the lines into InfluxDB because where you out the space and commas determines what is indexed or not! Also data types should be specific (ie make sure you are setting integer vs float correctly).
cinst influxdb1 /Service
cinst grafana
start $Env:ChocolateyInstall\lib\grafana\tools\grafana-*\bin\grafana-server.exe
git clone https://github.com/majkinetor/psinflux
import-module ./psinflux
1..10 | % { $x = 10*$_ + (Get-Random 10); Send-Data "test1 value=$x"; sleep 1 }Can anyone comment on Prometheus vs Timescale? What are the tradeoffs? Or would I use Prometheus on top of Timescale?
Repo: https://github.com/timescale/timescale-prometheus Design Doc: https://tsdb.co/prom-design-doc
[1] https://prometheus.io/docs/operating/integrations/#remote-en...
I manage a small homelab infra, but also an enterprise infra at work with >1,000 endpoints to monitor, and I/we use simple shell scripts, text files, and rsync/ssh. We monitor cpu load, network load, disk/io load, all the good stuff basically. The monitor server is just a DO droplet and our collectors require zero overhead.
The specs list and setup costs in time and complexity are steep with a Grafana stack - is there any value besides just the visual? I know they have the ability to do all manner of custom plugins, dashboards, etc, but if you just care about the good stuff (uptime+performance), what does Grafana give you that rsync'ing sar data can't?
PS: we have a graphical parser of the data written using python and matplotlib. very lightweight, and we also get pretty graphs to print and give to upstairs.
- app-specific metrics
- quick and easy way to build a number of graphs searching for correlation (how to slice the data to get results that explain issues)
- log/metrics correlation
- unified way to build alerts
- ad-hoc changes - while you're in the middle of an incident and want to get information that's just slightly different from existing, or filter out some values, or overlay a trend - how long would it take in your custom solution vs grafana?
And finally - grafana exists. Why would I write a custom graph generator from a custom data store if I can setup a collector + influx + grafana in a fraction of that time and get back more?
- Fairly lightweight. Prometheus deals with quite a lot of series without much memory or CPU usage.
- Integration with a lot of applications. Prometheus lets me monitor not only the system, but other applications such as Elastic, Nginx, PostgreSQL, network drivers... Sometimes I need an extra exporter, but they tend to be very light on resources. Also, with mtail (which is again super lightweight) I can convert logs to metrics with simple regexes.
- Number of metrics. For instance, several times I needed to diagnose an outage and I need a metric that I didn't think about, and turns out that the exporter I was using did actually store it, it was just that I didn't include it in the dashboard. As an example, the default node exporter has very detailed I/O metrics, systemd collectors, network metrics... They're quite useful.
- Metric correctness. Prometheus appears to be at least decent at dealing with rate calculations and counter resets. Other monitoring systems are worse and it wasn't weird to find a 20000% CPU usage alert due to a counter reset.
- Alerts. Prometheus can generate alerts with quite a lot of flexibility, and the AlertManager is a pretty nice router for those alerts (e.g., I can receive all alerts in a separate mail, but critical alerts are also sent in a Slack channel).
- Community support. It seems the community is adopting the Prometheus format for exposing metrics, and there are packages for Python, Go and probably more languages. Also, the people who make the exporters tend to also make dashboards, so you almost always have a starting point that you can fine-tune later.
- Ease of setup. It's just YAML files, I have an Ansible role for automation but you can go with just installing one or two packages in clients and adding a line to a configuration file in the Prometheus master node.
- Ease of use. It's incredibly easy to make new graphs and dashboards with Prometheus and Grafana, no matter if they're simple or complex.
For me, the main points that make me use Prometheus (or any other monitoring config above simple scripts) is alerting and the amount of metrics. If you just need to monitor CPU load and simple stats, maybe Prometheus is too much, but it's not that hard to set up anyways.
Thanks for suggestion
SerHack
During proof-of-concept testing, some historical data on disk perhaps wasn't lost per se, but definitely failed to load on restart. I haven't worked hard to replicate this but there are some similar unsolved tickets out there.
Additional traps for new players include customizing --storage.tsdb.retention.time and related parameters.
Alerting with the Prometheus AlertManager is also pretty straight forward and I'm looking at dashboards every day to see if everything is running smoothly or tracking down what's not working well if there are any issues. Grafana dashboards are always the second thing I look at after an alert fires somewhere and it has been invaluable.
It gets complex only when you start monitoring your apps (i.e. using a prometheus client library to generate and export custom app metrics) and create custom grafana dashboards for these metrics. Or if you need to monitor some niche technology without its own existing prometheus exporter. Then yes, you need to read the docs, think about what you need to monitor and how, write code...
In my view grafana shines in app data collection. And what we want a lot of is that (my simplest thing that can possibly is just have a web server accepting metrics ala carbon/graphite)
So one is likely to have grafana already lying around when one does the infra monitoring - I guess your choice is now how to leverage your set up to do app monitoring ?
PromQL support (with extensions) and clustered / HA mode. Great storage efficiency. Plays well for monitoring multiple k8s clusters, works great with Grafana, pretty easily deployed on k8s.
No affiliation, just a happy user.
Maybe they need a PR person.
I see lots of posts complaining about the stack in the post, but all of the alternatives posed don't really explain why.
Amusing that I've never heard of any of this but have heard and used collectd.
It's obviously no where near as common as a lamp stack or anywhere near common at all, so asking why it over something else is answered by "someone made it up so it's better".
I don't get the comparison to LAMP popularity. Insects are more common than cars too. They're different things ¯\_(ツ)_/¯
> If not, you wouldn't run into TICK. It's not common overall, because few environments need to go that far.
That's my point though. The post I replied to acted as though everyone knows of and uses it, but provided no information on what makes it a better choice for such usecases.
Both telegraf and collectd are mature programs with a large base of well-documented plugins. However, telegraf development is more active, and I find its golang codebase much more approachable than the C codebase of collectd. Obviously, YMMV.
In general though, I wouldn't advise adopting the entire TICK stack. Chronograf is inferior to Grafana. Kapacitor has a super-steep learning curve and is being deprecated by InfluxData. TIG (telegraf, influxdb, grafana) can be a good choice. Prometheus plus Grafana is worth evaluating as well; there's arguably more momentum in that ecosystem than in InflxuDB's.
Datadog is too expensive because of the number of hosts we have. So we're thinking of eventually going to a hosted InfluxDB setup.
But we also want to revisit other hosted solutions. Does someone have some experience with using Cloudwatch + Grafana? I've used Cloudwatch many years ago and it was clearly subpar to something like Influx. Is is better nowadays?
If I were to do it again I'd probably use Grafana and Prometheus
When we evaluated multiple options, we found that netdata does a lot of the heavy lifting. What are the advantages of a CollectD-based setup?
¯\_(ツ)_/¯