I've found many issues with providers this way, often before they even knew. It's also helped inform decisions to migrate to alternative providers or services when we are able to measure what the improvements would actually be, rather than relying on hand waving and marketing materials.
This is all pretty easy stuff, but of requires discipline and the resources to invest in instrumenting everything. You need some level of buy in from leadership and it's all the more difficult if you have a toilsome ops or oncall rotation. If you are large enough and can afford it, I recommend empowering at least one reliable engineer to be tasked to solve the problem across the stack.
The real problems are when you're operating a service you don't really own (i.e. a vendor) and there are issues related to how it interacts with something else. The only real solution, aside from getting the thing fixed or abandoning it, is to shim or proxy the dependencies such that you can instrument it as a black box. For example, if your vendor gives you a .jar that you configure to use S3, run a local proxy for S3 as a side car and collect stats there. This is a contrived example, but the concept should be clear. Often you can't even do this, as vendors hardcode stuff like AWS, and forget it if you're using something managed like Databricks or Snowflake.
We have a big vision beyond our functional checks, and would welcome any feedback.
https://www.fsl.cs.sunysb.edu/project-osprof.html
This covers all the examples you gave except CDN latency. You could try polling CDN endpoints from an external client that also gathers histograms.
- performs queries, measurements, log anomalies, etc
- collects the data and examines it
- sends alerts via independent channels when appropriate
For example, you might have an end-to-end query that is a curl call to an authenticated API which returns the free space allocated to a database table. When conducted from a host outside your network, this tests all of:
- reachability of your service
- minimum performance of your service
- availability of your database
You then need to establish what the normal values look like, so that you can send an alert when the range is exceeded in a bad way. Do something like this for everything critical.
You can also do negative checks: run a job that pushes some tiny amount of data to your monitor once a minute. If you haven't received a push in the last 90 seconds, send an alert.
If you host your service on cloud A, use a different cloud to run the performance checks. If you use Gmail for email, use another service to send alerts.
The last thing you want to be doing when an issue arises is debugging your whole stack to find the root cause while your customers sit on their hands angrily.