...then how do you know when third-party upstream services are obeying their contracts to your service, if not by testing how your service interacts with those third-parties?
(I know my answer, but I'm curious to hear yours.)
Of course, both have the same form, you run the system and verifies if the results match the expected. But monitoring is done constantly during the lifetime of your infrastructure, and verifies the entire infrastructure; while tests are done episodically, and verifies your program or a component. Tests also often block some procedures, while monitoring doesn't (but it certainly starts some).
Some people run (subsets) of their tests in production as a form of monitoring. Sometimes monitoring does not pass or fail and is instead qualitative like a dashboard or raw logging, without alerts
I’d say there is a grey area between monitoring and testing, it is more precise to ask if you’re verifying pre production, post production, or both
Edit: That is to say, what distinguishes testing from monitoring isn’t content, but purpose.
End-to-end (which I will fight for being the highest value test site, and it's not close) had no external dependencies.
And a separate test suite that touched external services, split into two components: one that tested our integrations, typically against a remote testbed (if the 3rd party was competent enough to have such a thing), and a second chunk that attempted to see if remote api behavior had changed. Which it does with annoying regularity.
So that gives you a number of options for testing the second one of those. Recording sample traffic and replaying it in the test suite is one approach. Actually running an instance of the service (if it's open-source - there's still value in paying someone to competently run an OSS service) in your test suite is another, as is running some clone of the service (e.g., if you're talking to S3, there are probably a hundred S3 API-compatible clones that are good enough to run in your test suite, even if, again, you are happy to pay Amazon to competently run production).
You also want to pay attention to the first one of those, but that's not a job for your test suite. That's the job for some balance between their test suite, your monitoring or production logging, and your business relationship with them.
That said, I too avoid mocks unless there's a specific good reason to add one.
This is why the classification of the test into e2e, integration, and unit can cause confusion. I like to try to encourage people to avoid bucketing and instead say “this test should be more integration style than it currently is”, “this test should be more isolated than it currently is”. At the end of the day all testing mocks out the user and things like old web browsers or other factors that are a part of the real world system you care about may not be simulated in your test, so the way to get ”real” e2e verification is probably monitoring real users, if you consider that the user is a part of your “system”