624 karma · joined November 1, 2016
Anyway, big thumbs up for the LemonSlice team, I'm excited to see it progress. I can definitely see products start coming alive with tools like this.
Wanted to add a bit more info about how Pythagora works and the current limitations.
When an API request is being captured, Pythagora saves all database documents used during the request (before and after each db query). When you run the test, first, Pythagora connects to a temporary pythagoraDb database and restores all saved documents. This way, the database state is the same during the test as it was during the capture so the test can run on any environment while NOT changing your local database. Then, Pythagora makes an API request tracking all db queries and checks if the API response and db documents are the same as they were during the capture.
For example, if the request updates the database after the API returns the response, Pythagora checks the database to see if it was updated correctly.
Finally, Pythagora tracks (using istanbul/nyc) lines of code that were triggered during tests, so you know how much of your code is covered by captured tests. So far, I tested Pythagora on open source clones of sites (Reddit, IG, etc.), and some personal projects and I was able to get to 50% of code coverage within 10 minutes and to 90% within 1 hour of playing around.
Here’s a demo video of how Pythagora works - <https://youtu.be/Be9ed-JHuQg>
Tbh, I never had enough time to properly write and maintain tests and I’m hoping to solve that problem with Pythagora so that creating a meaningful code coverage can be done in a few hours instead of a few weeks.
Currently, Pythagora is quite limited and it supports only Node.js apps with Express and Mongoose but if people like it, I'll work on expanding the capabilities.
Anyways, I’m excited to hear what you think.
How do you write integration tests for your API server? Would you consider using Pythagora instead/along with your system?
If not, I'd love to hear what are your concerns and why this wouldn’t work for you?
Any feedback or ideas are welcome.
Yea, agree, I'm also curious why this doesn't exist yet. I thought there might be a tech roadblock but there really isn't any - especially since Node released Async Context. We did a POC for covering Redis and requests to 3rd party APIs and everything looks very doable, just need some time to implement it.
Re PII, interesting question. We were thinking Pythagora will be used in a local/qa environment and not in production so that there won't be a need to anonymise data. However, it's good to have it in mind.
Re alerts, yes, definitely. I think that's a great idea for using Pythagora in a CICD pipeline. Especially since, once the test suite grows, it will be super handy to get alert for failed tests after branches merge.
Re sockets, yes, they are definitely on the roadmap - my previous startup used mostly sockets so hoping we cover them soon and we would LOVE if you want to contribute. If you have any questions or need anything, just let me know, I'd be happy to help you with the contribution.
My friend and I just created Responsiveye (https://responsiveye.com) - the first platform that generates automated end-to-end tests by itself.
Basically, you need to take 3 minutes to set it up and that’s it. Automatically, Responsiveye scans all pages on your site and generates hundreds of automated end-to-end tests that are ready to go. You can trigger them within your CICD pipeline after which all of the tests will run in parallel on our infrastructure so you get results back within minutes.
We're currently in an alpha stage and we're applying for YC this week.
You can also sign up for beta access and any feedback you can give is appreciated a lot.
But yea, makes sense to talk more about how we find these bugs in the video as well.
Explanation video link: https://www.loom.com/share/7c7ed621d030428e979ac6111c8bcb35 reply