Hacker News Reader

Comment by cmckn | Hacker News Reader

I’m a bit confused as to why a “unit test” would ever be hitting a real, external API. I would call that an integration test, E2E test, etc. that probably shouldn’t be run frequently (as part of a normal build, for example).

This API sounds like garbage, don’t get me wrong; and because of that, mocking it’s behavior is going to be almost impossible.

View on HN

I don't think I realized this was a thing. Are other developers mocking API endpoints, to test the code their as they develop and maintain it? It seems like a tremendous amount of work - recreating a service that will make an async call back to your endpoint as a means of testing. I've thought about it, but the fact that it seemed like so much additional work and that I'd be mocking a moving target just made me thing that would be a visit crazy town.

We did this at my previous company. What our app did was get data from a queue, enrich it with the API, then send it off to another queue. They provided test data and a test environment, but it was horrible, changed unpredictably and data frequently went missing. We recorded all endpoints hit during our integration tests, saved them to files and mocked their API (mostly GETs with no state so backed by a simple rest service with a hashmap).

The API team was an internal team to the company and there was absolutely no communication possible with them, so we kinda had no choice.

I avoid writing automated tests that hit an external API - especially one out of my control - because I don't want my CI runs to ever fail because someone else's service wasn't responding. I want CI to be a completely closed box, such that any failures mean there's a bug in my code.

If I want to test external APIs I'll do that in a separate set of integration tests which are run as part of a separate system, not as part of my CI for every code commit to my repo.

I mostly use Python, and the APIs I talk to are mostly accessed via the requests or httpx libraries - both of which have excellent libraries for productive mocking:

- https://requests-mock.readthedocs.io/en/latest/pytest.html

- https://github.com/Colin-b/pytest_httpx

It totally depends on the API and how critical of a code path you're writing tests for. For simple GET/POST calls, it's usually pretty easy to do this and I think you get a lot of value out of it. I've used an in-process HTTP server to mock such calls for a unit test, or you can add a layer to the nginx docker image to have a composable mock of the service.

It depends on a lot of things, but let's say (for concreteness) you are using Java or something similar. I'd write classes to represent the request and response models, call them A and B. Then I'd write a interface/class with a method from A to Future<B>. That method would be mocked everywhere, and most of your code doesn't deal with the API nonsense. (B is the data you want not the link to get the data.)

I may or may not actually test the implementation of that method.

In short: wrap up your external dependency.

The startup costs are high, but once you're in the flow of doing it, it's not so bad.

I usually have a switch in our tests: default is to run against local mocks, but the test data also works against the third party sandbox environment. Periodically we revalidate our mocks against the sandbox responses to make sure the other API hasn't drifted.

CI always runs against local mocks, for speed and reliability.

Depends on the API and how much testing you need. You want to test your code, not the API's availability or correctness.

But it can be as easy as using a fake http library and mocking the responses, or using a httptest server: https://onsi.github.io/gomega/#ghttp-testing-http-clients

If the API is complicated and you have to write your own fake server, that might not make sense for small projects.

The unit under testing could be the API interaction. If I wrap an external API in a client library I would test it’s behavior before then mocking that out when testing other code that uses the library.

I think situations like external API hit a fuzzy line between unit/integration. For testing an API unit I would reach for some way to save, re-play, and re-record those interactions just for sanity sake.

To me an integration test would be multi-step behavior, not just testing a specific request does a specific thing but a chain of requests, or verifying side-effects.

One reason for tests to hit a real external API is if you're using a "record and replay" test framework to capture the interactions so that you can run the tests against the recorded data quickly later. But because the API calls you make change (and the external implementation changes) you need to re-record from time to time.

This strikes a balance where 99% of the time you are making calls that never leave the process for fast testing, but can validate against the real implementation as needed.

This. Unit Test should not connect to external service imo.

And I do not get the gripe with a production rate limit tbh.

Same. I’d be more worried about an API with no rate limit. Say if another customer ships a bug that DoS’s the API on accident.

I would think a user token level limit would prevent this, although I could imagine a case where a bug simultaneous affected all user tokens, but I'd imagine you'd set that app level limit pretty high because otherwise you'd be making life very difficult for legitimate use cases.

Rate limits are of course fine. I think token level limits make more sense, because application level limits force the consumer to either to track a rate window across asynchronous processes or make the calls synchronously. But, I mean, that's fine too. Think it's just like OAuth1a, fine in itself but add enough of these things on top of one another, and you've created a technical hurdle that's just too difficult to leap.

I've seen quite a few people that just call all automated tests "unit tests". I wouldn't read too much into it.

Reply on news.ycombinator.com