1. Detail response/time taken for every single endpoint. So I want a histogram and not just an average. I have seen often you get the top X endpoints (for me mostly requests) in tools and top X by global measure. At my work the number of unique requests are a lot and the good response time for them varies a lot from 50ms to 1s easily. A global threshold is useless. There's also the fact that users have different cost for same endpoint but that I don't have a super good solution.
2. Some level of traces for requests that crosses a threshold. That means the agent has to keep the threshold for every endpoint but is that so expensive? I think I saw some ideas (in some commercial product) where they collect data and drop them if the request ends up being fast enough. I think that's a very good approach. So I want specifics when something is slow and against a per endpoint threshold. Perhaps I want all requests slower than 99 percentile, perhaps I keep 20% for > 90 percentile etc.
3. Now the per end point statistics has to be kept for significant period, months not days otherwise how would you see the change. I have things slowed down due to code change, due to usage increase, due to query plan getting whacky. You have different performance due to different level of usage (concurrency). No one can probably afford per second resolution for a year sure but if my 9am spikes are averaged out how would I know that this is an old problem or the spike actually worsened by 20% in the last two month? I think you can get away with reducing resolution a lot if you keep a histogram and not just average. But I don't think anyone optimizes for that. And you also need to keep those important traces for quite a long time.
4. I work in Java and use unstructured log and have been trying to figure out how to parse that reliably for debugging so that we don't spend hours to grep the logs. I just realized recently the most common queries are easily parsable for me users, servers/app-instance, codeLine (java loggers prints the class:line) that narrows things down so much for me that I can afford to just export those and grep if I need to. But most log tools seems like either grep & super structured. Also someone else mentioned cardinality because while the rest can be fine, user is definitely high cardinality so everything might break down there.
I think for my problems, at my scale Million of user not Million concurrent user I think these are nicely solvable given the capabilities the tools today have (even if they are not doing it exactly like that) but is is unscalable at large or this is just not needed if you do 'something` which I/we are not doing?