Ask HN: How to price a service that saves hundreds of engineering hours?

It is an interesting idea, but your approach to pricing seems off:

100/day is not reasonable, at least not for small/medium teams. Trying to do arithmetic on run times and hourly rates to convince people it is reasonable is not only overly simplistic, but actively distracts from the conversation you really want to have - how does your solution solve their problem? People buy when they believe you can solve a problem better than they could without you. So keep the conversation in that space and don't set up pricing that invites nitpicky arguments about hourly costs, etc.

Instead, I'd just price based on your own needs - figure out your actual operating cost per test-minute, tack on a reasonable profit margin and operating costs, and bundle up some packages that have a quota of test-minutes per month in sizes ranging from 20/month to 100/month. Make sure that you are profitable at every package, and see if you get takers at that price point. Then let prices increase naturally when they run out of minutes and need to ask for more.

Speaking from experience, the promise of completing tests in 1 minute or less is very appealing. This is one thing I don't want to compromise.

In the case of the two companies that are helping me to test this service, they both pay a fixed fee of USD 2000 for an agreed maximum parallelization of 50 browsers. This means that up to 50 concurrent tests, I can still deliver the promise of 1 minute tests.

For context, BrowserStack, which they used before, is charging USD 999 for 5 parallel browsers. I believe BrowserStack has this pricing model because they assume long running tests. It is a lot harder to share VMs if tests take 10-15 minutes than if they take 1 minute.

I could have pricing as USD 999 for 50 browser parallelization, and then charge extra if either of their tests take longer than 1 minute to execute. This model would ensure that we don't have one client hogging all of the resources.

Speaking from experience, the promise of completing tests in 1 minute or less is very appealing.

Indeed. There is a qualitative difference between tests completing fast enough that a dev is OK with just waiting a few seconds for the results and tests becoming a "go do something else for a while" background activity. There is also a qualitative difference between tests running fast enough that if something does go wrong the code can often be backed out before it interferes with anyone else and tests that run slowly enough that the CI system can't keep up with merges from the team and changes start piling up that then need sorting out somehow every time a set of tests does fail.

Another thing to consider is that most engineers do not run all integration tests locally before committing them, simply because they take long time, i.e. A typical workflow is: commit changes -> raise a PR -> wait for all tests to complete -> [if tests fails, go back to step 1] This hugely impacts developer ability to switch context, because they need to stash code, switch branches, etc.

If all integration tests could complete in pre-push commit hook (which is reasonable if they take less than a minute), then we would dramatically reduce the context switching involved in day to day engineering.

I agree. That's another reasonable point.

If I were in your position I would absolutely be emphasizing these kinds of qualitative benefits in the marketing even if your pricing model is actually very quantitative in the end.

Anyone who runs the kind of engineering group you're currently aiming at won't have to think very hard to get an idea of the quantitative scale of the benefits they might enjoy as a result. Ideally you want authorising a few (or more than a few) thousand bucks a month to be a quick and obvious decision for them. "We provide a simple tool that enables radically better development processes with likely ROI between 10x and 100x" is a decent elevator pitch to people like VP Engineering and CFO who are going to end up making a purchasing decision.

This is pretty bad advice. Price based off value delivered. If I was a sales person selling this product the first thing I’d want to do is calculate out the value of the time saved. “It pays for itself” is a wonderful proposition - why would I not buy such a product? Not only is it time saved, but it’s more efficient engineering cycles/sprints/etc.

If you know your market, you know your price. It’s very easy to A/B SaaS pricing. Does the proposed pricing sounds too high? To me, yes, but go out there and test it. Worse thing that can happen is nobody buys it.

Agreed. This is awful advice. You need to be pricing based on value not cost. OP has it right. It’s just the question of how do you peg that value properly.

100/day is not reasonable.

If I'm understanding OP correctly then at that level this tool would be saving the customer's engineers 14 minutes of disruptive downtime 100 times per day. If that's correct then $100/day may indeed be unreasonable, but because it's way too low.

I doubt the useful target market for a tool like this is in the $20/month or even $100/month category. What does a series A/B company that runs 100 slow rounds of integration testing per day spend on its cloud hosting every month? That's obviously going to be a much bigger number than they'd pay for a service like this alone but it will give you some idea of the magnitude of numbers that are reasonable to discuss. How much does running those tests slowly cost them not only in direct developer salary but also in lost productivity because of the slow feedback, reduced developer satisfaction leading to people being demotivated or even leaving if it gets too frustrating, etc? What is the extra risk to them of not having new projects ready fast enough to add serious value before the next funding round?

Okay — but running 60 copies of a small EC2 for 1 min is $0.009 to $0.04

Running 100x test suites would be $0.90 to $4, in server costs.

Charging 25-100x the server costs is optimistic, at best.

The only time your costs should have anything to do with how you price something is when your customers know how much they are and they influence how much they'll pay, or setting lower limits so you don't go out of business.

Software and infrastructure companies charge huge multiples of the hosting costs of providing their products and services all the time. The value they add isn't just leasing a VM on your behalf.

Unfortunately, EC2 is not an option if we want to deliver 1 minute promise. I need VMs that I can flash start with all the software needed to run the tests immediately available. The best I was able to achieve is 20 seconds using my current provider. However, they charge in hour increments.

That being said, I am constantly experimenting how to reduce costs. I think the final result is going to be a fairly complicated orchestration of pre-booted machines that are rented by month and those that we spin up ad-hoc to handle demand fluctuation.

EC2 bare metal + running your own VMs would solve the latency issue; but now you’re going to have more complex rental dynamics.

That would be $0.05 per core for an hour… or $0.05 for 60 copies for a minute. (Price for m5.metal on demand; a1.metal is half the price; m6a.metal is around $0.04)

I think the idea is cool — and wish you the best of luck.

I’m just trying to share why as a senior SDE your product doesn’t make sense to me: 100 integration tests a day is a fairly substantial team… and you’re charging a lot for it, at $2-3k per month.

Sounds like you are on Hetzner.

If that's so, you can take sever for around 60$ month there and put some load there - I doubt it gonna dip your pocket for such test.

Would Lambda be an option? Not sure what the stack looks like here, but a quick Google suggests people have gotten Cypress to run under Lambda, at least.

In my experience Lambda can have a pretty slow start when it's cold.

Sure, for an interactive endpoint, but for this a cold start of 200-500ms, or even a second or two for a Docker backed Lambda, sounds very workable.

You could always be pre-pooling VMs once you hit a certain scale.

Y’all need Kubernetes.

I think your pricing math is a tad off. I don't think it's harmful to use that for marketing, like you're saving engineering hours, but when you use it for pricing it falls apart. The reason is that you're selling to engineers, and we all know as CI is running, we aren't blocked. So, it isn't a 1:1 minute saved, although, I wouldn't conflate that with zero saved. In my experience a 1-5 minute test run isn't much different, but 3mins vs 30mins is a huge difference. I wouldn't force an engineering team to chisel 6mins to 2mins for example. I would roadmap how to reduce 30mins to less than 10mins. That being said, I think your equation needs more nuance to sell to an engineering manager or leader.

The only time I felt blocked by CI was at a company whose tests had a 50/50 chance of failing. The tests also took upwards of an hour to run (the engineering culture was a cult of True Believers in a terrible technology stack).

I think having something like this might have actually kept me at that job, at least a bit longer, though we had the same problem with our unit tests, and I don't even recall if Cypress was what we used (probably not, now that I think about it).

Just to use your example, you would have a VERY hard time convincing me to spend $3,000 a month to speed up testing time. Running tests isn't exactly dead time. It gives engineers a break, they can think about other things while running them, get coffee, etc. There is DEFINITELY value there, but not $3k a month worth.

That said, pricing correctly is hard and requires a lot of experimentation. Growth and adoption might be more important at this stage.

From my experience, it depends...

Is this a test suite I run during my development cycle? If so, I get distracted if it takes more than half a minute, open HN, and lose a bunch of time.

Is this a test suite that runs once I open a pull request? It can run an hour or two, I don't mind. I'll move on to the next thing and check tomorrow morning if it's passed. If it failed I start my day by investigating why. I care very little about speed here.

Do you have a market strategy otherwise?

What is your customer base and target market? Both pricing models and numbers will come out very differently for the same underlying technical work depending on if you're aiming for fewer how-much-could-a-banana-cost-biggies (these need service agreements and someone who they can talk to on a first-name basis) or a turnkey system for every startup and corporate using Cypress out there (they just register a credit card and e-mail and start integrating).

While you are considering your margins, you're going by value-based pricing here - you focus less on how much it costs you to build and run and your bottom line and more on how valuable it is to the potential customer. Some told you hundreds of hours were saved.

It sounds like you have the pricing model for a mass-market SaaS with the pricing for the high-end.

One risk with such high value-based pricing is that assuming initial success, some happy customers might get restless after a few weeks/months/years and realize that for the price they're paying you they could actually save money by building and rolling something similar in-house. The customers who are ready to pay those prices are precisely the ones likely to do it, I think. In any case, unless your market is super niche or you have some crazy edge, you will get competitors if you're successful enough. Maybe even straight-up copycats. This is nothing bad at all but it can be good to keep in mind that you may lose customers if they're able to slaughter you on cost-performance.

Also consider that these higher-end clients likely care a lot about availability and stability. If you don't have brand-recognition or a personal reference, it can be hard to convince them to take a shot. Free trials will never be enough for them to be confident either. Initial underpricing (not too much obv) is a way to compensate for that initially.

But again, I think we need more context before we can make a fair assessment of if the price is reasonable or not.

Your pricing sounds ambitious considering something like Autocad, which is a primary tool, costs about $2000/year, and you plan to charge $3000/month?

Since this is essentially a cost-saving thing, I don't think it's realistic to try to charge the cost saved. If I as a customer can save $100/day by using your software, but your software costs $100/day, why bother?

Isn't that $2000 / year per license? Where this is $36,000 / year for the whole company

Good point – it should be phrased as improved productivity. You have to think more broadly.

You are not only saving engineer time. By delivering more in a shorter amount of time, you have compounding effects across the entire organization.

Indeed, you should focus the message to some marginal advantage if possible.

Eg, If I sell an animal feed supplement that increases cow weight by 1%, it means customers can increase output by 1%. If the same supplement can make racehorses 1% faster, that's something entirely different.

At Series A / Series B, they can’t think about serious commitments (and this sounds like one) in increments smaller than $10k. I’d try that and, if you get any pushback, ask what it would have to do to be worth $10k.

(A month. Bill quarterly or yearly at their option; raise prices annually to capture increase in use.)

Remember that their alternative is not “spin up VPSes” it is “staff an engineering rotation to maintain this service” which will cost them minimally a million annually.

That doesn't square with my experiences at startups. A $1k/mo service would have been something a manager could sign off on themselves, and all it took to do that was to get an engineer on the team be really excited about it. On the other hand, a $10K a month would have been a much harder conversation to have, falling outside the discretionary budget of a particular manager, and probably requiring meetings and demos with the head of the department to get approval.

Remember that their alternative is not “spin up VPSes” it is “staff an engineering rotation to maintain this service” which will cost them minimally a million annually.

But the default alternative is "continue making limited use of these integration tests and continue accepting the delays of running them slowly" which has a cost of $0 compared to status quo.

What proportion of series A companies that are trying to scale up and show results quickly so they are still around to raise a series B is going to spend $1M+ on a dedicated team to run a service for this? I'm guessing 0.000%. That means the comparison isn't a very compelling argument for the leaders in that organisation because really you're just arguing that your solution is better than something they obviously weren't going to do anyway.

For context, they were previously using BrowserStack and paying USD 999. As far as I know, they are using both services at the moment in parallel (mostly because of their year long commitment and other services that BrowserStack provides to them that fall out of scope).

We already agreed to discuss the idea of doubling the capacity.

You can price based on eng time saved, but the rejoinder will be that the devs use test time to answer emails, respond in slack/jira, etc. And to some extent that's probably true.

I do have some very strong advice on how not to charge: (1) do not charge based on your costs (customers won't care), and (2) do not charge based on fine-grained usage. The latter is important because if a VPE signs a contract for $3k/mo, he/she doesn't want to sit there worrying that devs run a couple extra tests or add a few tests, etc, and boom: all of a sudden the VPE is going to the cfo to explain $10k in overage charges.

So: price in broad tiers, and price based on value. If you save hundreds of eng hours/mo, charge for the first 20 hours saved. At $150 per, that's $3k/mo. Multiply by 5 to have a functional company (you can't do sales for under $10k; every successful sale -- which will be 20% if you're very good and lucky -- must pay for all failed sales, all marketing, all dev, etc. Oh, and you're doing sales.) So $15k/mo. That's within the approval authority for a vpe with 50 reports.

And accept that bigger companies don't worry about $2k-$3k/mo. So maybe your target should also be smaller shops that can't accept the infra overhead to build this out, but will buy a cheap solution that saves their 1-4 devs a ton of time. Dunno.

edit: and you need a strong cross-tenant security story.

The latter is important because if a VPE signs a contract for $3k/mo, he/she doesn't want to sit there worrying that devs run a couple extra tests or add a few tests, etc, and boom: all of a sudden the VPE is going to the cfo to explain $10k in overage charges.

Counterpoint: I have personally seen companies of the size OP is talking about run into unexpected big jumps in spending on key infrastructure and tooling because they made some architectural change and then suddenly realised it was going to run the tests 10x as often or something. Without exception the response has been twofold. First they did take the immediate financial hit because they couldn't afford to be without a critical system. Second they diverted significant resources to getting the usage levels back under control very quickly so they didn't have to rework that critical infrastructure as well.

If anything, since bringing the costs of some important infrastructure down is exactly the problem OP's tool is solving and its value is directly demonstrable in terms of time and therefore money saved, your general argument should be a positive sign for OP that they probably have a viable business model.

I could buy an argument that tiered pricing might work better for OP than metered if someone showed up with data supporting the theory. But metered pricing seems to be working out pretty well for a lot of popular cloud-hosted dev tools and of course is almost universal among cloud infrastructure providers. People complain a lot about AWS having unpredictable pricing but I heard AWS has still had some success breaking into the cloud hosting market and probably it's making enough money that Amazon might keep it running. To be fair that's partly because although AWS lacks safeguards that would certainly be welcomed by many it does also have a reputation for being very decent about support in the occasional "Oh s@£t" scenarios.

Metered: significant internal and external resources are devoted to understanding aws spend. See the whole industry springing up around this, starting with Duckbill. No vpe is looking to add more to that list of things they have to understand and plan out.

Flat rate or bucketed pricing is common across many dev tools. eg ci/cd: see bamboo, gitlab ci/cd (purchased in blocks), circleci (credits w/ account, more credits sold in blocks). apms like honeycomb.io. etc.

Some of the companies I mentioned were using some of the tools you mentioned. You don't necessarily avoid the price shock with tiered pricing models. It just means there's a step function and if some change does unexpectedly put you over the next threshold that just became some relatively senior people's #1 problem today.

Ask your customers. Propose your current amount and be honest "do you think this is fair?" If they say yes great! If not, ask what they think is reasonable. Set yourself a lower limit to keep it profitable for yourself.

"Sorry I can't price it below X". See how they respond.

I am trying to figure out how to scale this to the point where unit economics makes the most sense.

If I simply transferred current costs on to the end user, it would be too much. It would be USD 0.20 x number of tests x number of hours, e.g. a company that works 9 to 5 and has 100 tests, that is 0.201008=USD 160 per day.

However, changes the more clients you have. If I have 10 clients paying for 100 concurrent test capacity, I wouldn't need 1000 VMs, as they are unlikely to be all running tests at the same time, all the time.

Go with actual costs today. Don't run at a loss "because maybe someday I'll be profitable". What are you charging today?

Charge fairly. If you get more customers and improve efficiency again, be honest, "thanks to some improvements in scalability I'm able to reduce your costs to X".

You'll also build a long-term relationship by being fair and taking advantage of your customers.

Good advice

Why are you assuming that you will need to maintain a static infrastructure?

Aren’t you able to (fairy quickly) autoscale up/down based on the size of some queue of test requests?

Pooling is not an easy problem.

As I have only two clients at the moment who are helping to test this, keeping a hot pool allows me to focus on other feature development while sacrificing some of the costs.

Keeping a hot pool makes it a lot easier to immediately start test runs. Run tests, restart browser, clear sessions, return to the pool.

Right, at your current stage there is pry no need to build some task queue. However, if you anticipate infrastructure costs to grow quickly I think you’ll want to invest some effort in this.

You could even have tiered services. For example, some users may be willing to pay more for more consecutive tests.

This is great advice for a startup trying to figure out pricing!

I think if you find companies who do run integration tests 100 times per day, paying 2-3k per month to save around 500 hours per month does make sense.

You really have to work out your ideal customer, though. And find a way to scale it down - you can potentially use the capacity of more VMs spread across many smaller companies.

Maybe your business model is having a few clients pay you a couple grand a month, maybe it is having a couple hundred paying you 20-50 or so a month.

You do seem to have solved a problem, which is a great first step.

You need to determine which type of customer you’re targeting.

At the high end, some enterprise companies won’t blink at the prices you’re suggesting. However, they will expect a level of customer support and service agreements that are going to make you work hard to deliver what they ask. You’re also going to need to be heavily involved in sales cycles and support, which will start costing you many, many hours just to get the sale.

Are you prepared to handle this and stay responsive? If not, you will have mixed results pushing high priced offerings, especially if they have deal-breaker limitations (like your 30 second test limit) that could saddle the customer with even more work to integrate and use your tool. You have to acknowledge that your service isn’t purely savings for the customer. It’s work to implement and maintain on their end, as well as handle billing and so on. You cannot make sweeping calculations about savings without ignoring the costs it brings to a company to add your tool.

On the other end of the spectrum, you could target the self-serve market with lower priced services. These offerings can get away with a “take it or leave it” type of service where customers are mostly on their own to evaluate, pay for, and self-support the software. If the tool is good, engineers will try to make it work. However, I suspect you’re massively overestimating the way that test times are a blocker for most engineer workflows. Even with long text suites, we don’t just sit and stare at the monitor while we wait. That’s time for responding to communications, checking other things, taking a short break and so on. You could have a hard time selling engineers on the idea that they can do more work and take fewer breaks while paying for the privilege. Choose your marketing angle wisely.

> However, I suspect you’re massively overestimating the way that test times are a blocker for most engineer workflows. Even with long text suites, we don’t just sit and stare at the monitor while we wait. That’s time for responding to communications, checking other things, taking a short break and so on. You could have a hard time selling engineers on the idea that they can do more work and take fewer breaks while paying for the privilege.

Speaking from experience, my productivity is down by like 80% if I know that there is a test I am waiting to complete. I can do other things, but I will constantly switch back and forth to check on the progress. I know from talking with other engineers that they feel similar, esp. when flakky tests are involved.

However, I get what you are staying. I will think how to position this in a more compelling way that cannot be easily challenged.

I agree. It's hard to work when you have a 10-minute test running, either I wait/do busywork, or I start doing something else and then I don't want to interrupt that line of work to context shift back to a failed test.

Just to clarify, 30 second limitation is only in the context of 1 minute test-result guarantee. Tests can technically run indefinitely, though I want to design the business model in such a way that discourages the long individual tests, as it aligns the incentives of platform and the client.

Pricing decisions can be subtle. I like to try and calculate the value I delivered to the customer. Here are a couple of questions you may find helpful.

What are you replacing? What else could they pay for to achieve an equivalent speed up?

What are your enabling? How much time are you freeing up that can be used for other purposes?

What is the impact on time to market / release inter-arrival time?

What is the shadow backlog of tests that are not being run but now could be?

What is the impact on quality: for example time to fix (MTTR) or production problems prevented because of increased testing?

You are welcome to schedule a no cost office hours session to walk around your situation: https://www.skmurphy.com/blog/2016/08/01/skmurphy-office-hou...

The only way to figure this out is to go to your clients and give them a price and see how they react. You want to get to the point where they are angry about the price but are still paying.

This is going to be a lot of trial and error and no amount of analyzing is going to give you a good first answer. So start with something you find reasonable (the pricing you mentioned) and see how it goes with your current clients.

If they are totally ok with that then you’re priced too low and on the next sales cycle change it up. If they all balk and threaten to walk away then adjust your price down, until they begrudgingly pay.

If you want to target the sme's I might suggest just separating out your value, set a few simple tiers and make it easy to plug a cloud backend in (only needs one provider at first). This way your value and cost is direct and obvious and you're not left managing a fleet of vm's. This probably increases the total cost a little as there's less sharing, but a lot of medium org's can manage that internally and it's much easier to convince a manager to spend $xx/$xxx a month rather than $xxxx or more

My biggest concern would be that this is all predicated on the idea that engineers are doing nothing useful (and thus burning time/money) when tests are running, which isn’t usually true.

> and they care a lot about moving fast.

You are thinking too much. Just tell them $5K/month. If they balk re-think your product.

> existing customers are series A and series B companies

With 2 VC funded customers, try getting a referral for cloud credits. If you cant get $10K credits try to find a business-side co-founder.

Finally, would you be interested in some sort of experimental "guaranteed minimum activity" swap contract? No cost to you, but you may need to provide analytics. Email my puppet email which i dont check often enough

> completes all tests (no matter how many you've got) in less than a minute.

> My thinking is to charge 2 cents per a test-minute

Doesn't that reduce to 2c/run?

That aside, my gut reaction was don't charge for time if your selling point is how much time you save.

Charge for #tests, or something even better targetted at what makes it slow usually (I'm not familiar) - you charge some (<100) percentage of the reason tests are expensive and make that go away, and it seems like a sound choice to use your service doesn't it?

test minute = test * minute, i.e. 50 tests = USD 1

However, based on everything I've read in this thread, I am leaning towards max parallelization based model, e.g. maximum 50, maximum 100, etc, and charge extra for tests that take longer than 1 minute.

This allows straightforward sales while providing most of the value to an avg. client.

I understand the meaning of a 'test-minute', but in OP you said however many tests, it ran under a minute. Not a minute per test. So if that's the case then fine.

(But also now there are 'tests that take longer than 1 minute'? It's gone from wow to not really clear what you're offering beyond managed test-running (that's otherwise equivalent to self-hosted) to be honest.)

Edit: do you mean they're parallelised, so they all complete in however long the longest one takes (which in OP is <1m but in comments might be more) and you'll charge for #concurrent jobs?

Yeah, you've got it in the edit.

Using Cypress you are going to have specs and specs will have tests.

You can parallelize Cypress per spec. Meaning that if you had 50 specs and 50 VMs, then your entire test suite will take at most what your longest spec takes.

Most companies are not going to have 50 VMs ready to run tests. However, even if you did, you still have the problem that your tests will run for as long as it takes to run the longest spec, which can be a lot.

I've seen companies try to diminish the latter issue by splitting specs into multiple files. However, that will get you only so far.

So what I do is... I read all your specs, extract individual tests, and spin up VM for each test.

If all your individual tests complete in under a minute, then you will wait at most 1 minute.

However, if there are individual tests that take longer, than yes – you will wait for the longest test to complete.

Cost-plus with a low cap and a “call me” for anybody who exceeds.

The tech sounds great. I'm regards to pricing, people aren't machines; you save them 9 min a day per engineer, doesn't mean you get 9 min / workday extra productivity * engineer. And that's not about laziness or anything, good programmers also need time to think about things abstractly etc. I wish you luck in getting properly compensated though what you did sounds great.

There is a tool from Nx in similar space, where they cache test results based on code paths.

They charge by hours saved, for example they calculate how long it would have taken to run those tests and when they serve the cache instead, they charge money for it.

Not sure if it could work in your case, but if it's saving hundreds of engineering hours, charge based on the saved time (and not on the normal runtime)

Note: when running untrusted user code don't underestimate how much of a liability this is.

It's so hard to be truly sure that there's no compromise path. Particularly when optimized for speed like you're saying.

Security vs performance is often an explicit trade off in these scenarios when it's tempting to reuse processes or volumes or VMs.

1. Why can't you use something like AWS Fargate and spin up the required number of VMs as needed to keep your running costs completely variable? 2. Similar to many other comments here, you should price your service based on value offered to the customer (i.e., how much $ they are likely to save based on some engineering cost model) as opposed to a cost-plus-margin model. 3. Once you have a critical mass of clients and tests running, you can then optimize your Fargate pool with spot instances.

Starting with #1 will give you a floor for your running costs and if you tack your other costs (overheads) and a desired margin on top of that you will have a floor price. Then you just need to figure out the value you are providing (and you can A/B test that with prices higher than the floor price you obtain).

We pay around 1500 per year for 8 cores (~10 tests on puppeteer). If your service offers not only test parallelization but also isolation of tests then that is powerful.

Charging per test runtime in seconds is also a good idea.

As for pricing, look at Browserstack, browserless, Saucelabs. They are well established players.

Why do you need to spin up VMs? You could use a service like fly.io and have these running instantly.

This is an interesting project.

Is your project so technologically advanced that cypress could never get there? If so maybe you can join forces with them or sponsor them.

Your minimum price is what it actually costs you, your maximum price is whatever you can get. You can convince customers to pay your price by demonstrating how much your product will save them and charging a discount on that amount.

Different pricing strategies:

Market Pricing: what are your competitors charging.

Cost plus: What is your investment to date and what do you see ongoing marginal costs as, and what type of pricing do you need to make this worthwhile for you financially?

nearest substitute pricing: what’s the financial savings of your product versus the competitor? Charge a price that captures some of this value (what you’re proposing)

Also be mindful of Cypress possible taking steps to improve their own run times. What will you do to stay ahead and keep your marketshare? include this in your considerations as ongoing r&d.

With Cypress, atleast in what I've heard of (and seen), flaky tests seem to be a larger waste of developer time than the time it takes to run the test suite overall. Does this help improve that?

It sure does. The way Cypress works is that it injects their entire driver (test API) to the browser and then loads the tests themselves, also in the browser. It is a finicky process because they need to rewrite all JS and HTML (https://github.com/cypress-io/cypress/tree/develop/packages/...). It is slow because they have to proxy everything through HTTP 1.1 proxy (https://github.com/cypress-io/cypress/tree/develop/packages/...).

The thing that I built takes a different approach. It parses Cypress tests, interprets the commands that Cypress wants to run, and then uses Chrome DevTool protocol (CDP) to evaluate commands such as navigation, click, etc. If you are familiar with the JavaScript ecosystem, my tool effectively transforms tests to Playwright compatible tests and then uses custom orchestration logic to run them.

I wouldn’t give away too many technical details of how things work.

My experience with flaky tests is that they seem to leave some state lying around, or are impacted by some randomness that wasn't accounted for in the test writing (lists in different order, DB state etc).

So I actually do think this would benefit that, as these tests are run "independantly", however that's just going to hide the problem of your bad/flaky tests, and you'll never solve the underlying problem.

I have the opposite experience. LocalStorage can leak between tests and that has been an issue for years: https://github.com/cypress-io/cypress/issues/2695

What has the reputation of being less flaky then Cypresss? My experience is that it is easier to write a non flaky test in Cypress than in Selenium.

Nothing. The problem is E2E UI tests are always going to be flaky.

That’s why many recommend unit tests or pseudo E2E tests that save a copy of an API response. Unfortunately this advice is rarely followed.

Sauce Labs is a good comparison. They charge for more concurrency, which really strikes directly at the value prop you are discussing - saving time running a large test suite.

We've built a similar thing for playwright tests but on cheaper lambda functions if anyones interested! https://github.com/mentimeter/play-lambda

I would expect to be charged per minute, like I do with my current cloud. (Actually they charge per second)

You might consider migrating to cloud that supports per second billing.

Regarding your scale/customer density problem, you may want to check AWS’s EC2 pricing, which bills by the second rather than the hour.

Have you thought about approaching cypress and selling it for $1-2M dollars (after patenting it first of course)

I like to hack this problem. Maybe once I get bored if there is demand. For now, I enjoy the challenge of figuring out how to make it work and scale.

It's possible you could sell it, and still work on it as an employee/product manager at cypress... So both could be true.

I'm interested in trialing this service. Cypress takes forever. Feel free to reach out.

Drop me an email at ahoy@punkpeye.com

My $0.02: Price high with generous free trial, and promote in niche places (like HN)

Inspired by all of you, I've secured an _amazing_ project name. It is going to be a while before I launch it to the public, but this is exciting.

Sell this to cypress.

apex.sh is a nice example. it's not for everyone, but it's the best fit for those who need it.

maybe give them knobs on the amount of maximum concurrent tasks?