Launch HN: Preflight (YC W19) – Better automated testing for web apps
preflight.com
preflight.com
I was the first engineer at ShipBob and ended up hiring over 40 engineers. We had a lot of UI issues after we grew beyond a team of 5 or so. With all the engineers adding new code to the existing code, everything started to break.
That's when we started writing Selenium tests. I and several other engineers wrote 100 Selenium test cases in 2 months. But we were not able to maintain them in 3- 4 weeks because we had other responsibilities, and our Selenium test cases soon went to trash.
The problem with automated UI tests with Selenium or Cypress is that they depend on the internals of the design (e.g. via CSS and XPath selectors), resulting in tests that break frequently because of unrelated changes. When that happens, you either have to fix all the tests and run them again, or replace the broken automated tests with manual testing.
Then I hired 2 QA engineers, they took 5 months to come up with 5% test coverage and we needed more testing so we couldn't add more to it and maintain those tests as well. These issues pushed me towards creating a tool for automated testing that could understand the context of the page—the way humans do, not the way browser engines do—and can adapt to changes based on each steps' context. So if the UI changes completely and the test is looking for a Password Input in a sign-up form or an Orders menu item in the menu, we will find it without any related selectors.
As humans, we understand the page in terms of buttons, menus, headings, borders, and so on. I wanted to mimic that. On top of that we have implemented a way to understand navigations through these different elements with their context, again, as humans do. We call this Context-Awareness. It’s our secret sauce to adapt to the changes and also calculate what your test coverage is. Since we understand the context of each step, it’s pretty easy to see if, say, a given button is in the tests.
Here’s a video that shows the Context Awareness aspect I just mentioned: https://www.loom.com/share/50426244253943e49906dd881e0fb7d7.
You record your tests using our browser extension, and then we run them on a bunch of Windows 10 machines that access your web app over the internet and execute the tests you recorded. We’re working on adding Macs. (Fun fact: Selenium cannot handle more than 40 nodes. The first version of Preflight was depending on Selenium Grid and it stopped working when we wanted to add more than 40 machines to it.)
We charge money based on how many tests you are planning to run. We don't want to limit you based on server time because it's not easy to predict how much server time your tests will take. So everything is based on test results.
Anyone can sign up and start using Preflight in minutes after signing up at https://app.preflight.com/get-started.
Thank you for reading. I'd love to hear your thoughts and help all of you with any of your QA needs!
Just a couple of days ago I started trying checkly [1] and have been quite happy, although I didn't try their browser testing yet. What do you offer over other services like this?
Why won't I sign up to find pricing? Because gaming pricing makes a bad impression and makes me think you will try to obfuscate other critical information once I become a customer and generally be a difficult vendor to work with. Nobody needs to add a difficult vendor, especially in a competitive category.
Edit: there are 6 top-level comments on this post besides yours. 4 of the 6 are looking for pricing.
We don't obfuscate, we care deeply about our clients. I always say this. If Preflight doesn't add value you don't have to pay for it nor use it.
Part of my first impression of your product is the fact that you have something to hide. I want to trust the products I use.
Also, my time is precious. I can't count the number of times I wasted time by looking at a product to find out the pricing makes no sens at all.
If I didn't care, I wouldn't have taken the time to write a comment as if I had a stake in your company beyond just wanting entrepreneurs to succeed. If I had seen this on Twitter, I would have just kept scrolling. But HN has given a lot to me, so I try to help others.
So in that vein, one more constructive criticism: learn to take feedback better. The best customer feedback often comes packaged in vitriol and is much more unpleasant than anything posted here. (Especially from support tickets. People are frequently angry when they write them, but they also provide invaluable insight into your product.) You'll have more success if you gracefully accept all critiques, both warranted and unwarranted, regardless of how gently they are phrased.
Good luck!
I don't care at all about anything else, as soon as the pricing page is hidden means it's not affordable to me.
For us, code-first / dev-first means we directly integrate with popular JS frameworks (like Playwright and Puppeteer), allow you to code any setup / teardown scripts in NodeJs and generally want you to treat your testing and monitoring setup "as code".
So, TLDR is that I would 100% give Preflight a try and see if it matches your workflow. Always choose the best tool for your problem!
P.S. we do help with generating test scripts with our Chrome extension https://github.com/checkly/headless-recorder
Can you run Preflight on-prem or must the AUT be accessible to the Internet?
Regarding the Context Awareness, is there any issue with working with React apps? If I renamed an item in the navbar, or changed the position of an element (thus changing the XPath), would CA be able to identify and workaround that?
Questions aside, it’s a great idea and the app looks great.
We can access to environments behind the VPNs. But it's hassle-free if it's accessible through the internet.
Actually we also implemented Context awareness for SPAs with CSS Module to adapt to the changes. We are agnostic to HTML changes and if we can find the element we will update the selectors after the test is successful.
Thank you very much for the kind words!
I'm bombarded by vendors every day trying to "recapture" a sale, as part of an automated sales pipeline process.
I rarely provide contact details prior to being able to:
1) understand the product, what it does, and whether it will work for me
2) understand the pricing and sla
3) understand the business continuity model
I'd rather have a more difficult solution (selenium) that I know the above three points about than to get sucked into yet another sales pipeline.
Also, no public pricing page is a bit of a turn off, especially when I created an account and still couldn't find any pricing info.
It should be no problem with MetaMask.
Edit: Also listened to everyone and put the pricing on the web page: https://preflight.com/pricing
Here's the pricing for those who are curious:
Starter - 300 tests/month $99 /mo
Startup - 1,000 tests/month $300 /mo
Competitors are cheaper and include more tests.
Which competitors? Where are they running the tests on? Do they adapt to changes and save your valuable time? Or charge small and run tests with selectors like: .wudfj289hf .a9dfh82hf .sadf98h which breaks each time you deploy?
pretty misleading for you to act like that page was already there. you just added it. your lack of transparency is pretty disturbing.
EDIT: wow OP got flagged and deleted their original reply, lol.
cheaper options (just YC, not counting non-YC)
You changed it, I know.
You totally could go on to be a huge company and that's great, but just not the people I want to associate with.
Absolutely, integrating with CI/CD is super easy. We have some integrations already with major CI/CD pipelines. You can find it right here after signing up: https://app.preflight.com/account/integrations
It should be about what you can offer, and I can determine if that fits my needs.
Here is the visual checkpoint (regression): https://www.loom.com/share/b9448ba5e76e410089be6840053a8596
$99/mo 300 tests / month
$300/mo 1000 tests / month
Aside from hiding prices, the sign up process has another annoying dark pattern - you sign up, they prompt you to schedule onboarding call (only option, no other button), then if you click anywhere or wait a few seconds a "Continue to Dashboard" button appears. If you click it you have an "Are you sure?" button instead.
Instant turn off.
We want to make sure we help you. Testing is not easy even though we try everything we can to make it easy.
Most distros offer font packages that can get you the stock Microsoft fonts if you're missing those, although these days I think it would be safest for the server to serve any fonts that aren't on all three platforms by default. On Ubuntu, you can do `apt-get install -y ttf-mscorefonts-installer` to get them.
Even on GitHub -- owned by Microsoft -- Windows runners are twice the cost of Linux runners (macOS is 10x). Given that, the extra unnecessary cost is almost certainly pushed onto the customer.