> Use a helm chart and just bring your own little database for dev test and e2e tests.
dev, test, and e2e tests should be done against full-size db clones
dev, test, and e2e tests should be done against full-size db clones
Real customer/sensitive data should not exist outside prod (and backup). So generally no, not full-size clones. I'd argue instrumentation in prod should give information on performance - for some tests/development you might need prod-size fake data.
Monitoring prod closely and understanding what could constitute a costly workload/query which would in turn require a temporary test env with similar sized dataset is the correct approach.
Imagine a system that could do this in CI/CD pipelines:
1) first, run all the tests on a tiny DB as usual 2) extract queries 3) run them against full-size DB branch/thin clone (thin provisioning, CoW; PII is not there of course, wiped out for security/compliance) -- auto-guessing parms, that's the trickiest part, but assume it's solved 4) collect all the details about performance, focusing on IO numbers (rows, block read/writes) 5) if some queries are off – say, you forgot LIMIT – post a warning, block the change, do not allow deploying it, letting backend dev fix it.
This would be proactive. And it's becoming possible with modern tools.
that's cute, what is your "full-size"? I don't have 2 days to run a test, and I'm pretty sure every single compliance requirements we are following would get obliterated the second someone hears about us doing that
Although, cannot agree with the former part. If your tests are running 2 days on a full size clone, and it's an OLTP case, what about users, do they suffer from long query duration too? It sounds like it's time to optimize queries and/or redesign test sets (or both).
If it's bad in testing, it will be bad in prod. That's the idea of testing.
Example: how do you check schema changes?