Ah you're right - I never noticed the nuance of choice being a key part of opt in, rather than the participation itself. TIL!
674 karma · joined July 8, 2014
Shameless plug: I've been working on a DD alternative, pricing based on data volume (instead of adding on by hosts or users), Otel native as well so you're never locked in. If you want to check it out: https://hyperdx.io
We've [1] been taking advantage of it - counter-positioning against DD by offering low cost and really straightforward pricing while NR and DD both have chosen non-sensical (to me and many others) pricing models.
Context: Founder building an affordable alternative in the SaaS observability market (hyperdx.io) - I assume your company might have moved when NR introduced new seat-based pricing?
As a vendor building in this space [1] - it definitely is. We're able to onboard teams faster to do side-by-side comparisons because they can simply point their existing Otel telemetry to both us and their existing provider with just a few lines of config. That wasn't possible before otel, and levels the playing field more than before. As otel matures, it'll continue to erode against DD's position.
It also allows us as a company to focus on what users care about (as you mention that's things like dashboarding, search performance, RUM, etc.) as opposed to spending all our time building basic integrations into every platform (though we still do plenty of work to polish places where Otel hasn't). Again, levels the playing field.
it currently supports logs, traces, session replay/RUM, alerts and dashboards - would love to hear what you think :) https://www.hyperdx.io
I also totally agree - e2e tests deserve better tooling to debug, and that alone we've found quite compelling of a product to be working on already, and I appreciate you find it cool as well :D
With regards to picking the right level of testing - I think there's a few km long rabbit hole to go down with that one, and I totally admit I hold strong beliefs that are different from most people in testing today. I agree though - there's always going to be a different preferred tool for different problems. It'd be pretty wild to expect e2e testing to accomplish the bulk of it! That being said, I think if e2e tests were better in a few dimensions, developers might find it useful for more use cases than they do today (some teams we talk to want to do more e2e tests but find the shortfalls as a real barrier to justify the investment).
The reason we're tackling this space is because we don't think this needs to be true of end to end, or even heavier integration tests. They have really awesome qualities such as being able to test close to things users actually care about (which is the most important part of testing at all!), can identify broken pieces across or in between parts of your stack, and are fairly intuitive to write/understand (imo).
But what if we could then build tooling to minimize some of the bad stuff - make tests more stable by providing better debugging, enforcing best practices automatically (imagine we can tell your tests are behaving non deterministically because of X async action, or use X conditional wait instead of a time-based one), and even giving you knobs to trade off between test fidelity and speed (ex. click a button and you can turn on/off network mocking in specific tests so you only exercise a part of your system instead of spinning up/testing the world).
Hopefully if we can nail our vision, you might be saying in the future "why are there so many unit tests? They cover so little surface, don't test how our app fits together at all, and require a lot of maintenance since they tend to test some form of implementation detail or another. We should be doing a lot more integration and e2e tests :)". It'll be a long road until then, but I think developers and users will be much happier the better we can make testing.
If you're open to it! Would love to hear more about the use case and where you think we can help the most in Selenium :) Give me a holler at mike [at] deploysentinel.com :)
Absolutely when we're working with larger teams, we're frequently putting together a custom plan with tailored feature and a custom quote that takes into consideration their volume (both current and projected), term lengths and other misc things. Unfortunately this makes it really hard to describe pricing expectations at scale on a pricing page.
If you think we can help your team, I definitely think we can put something workable together at that volume - feel free to hit me up at mike [at] deploysentinel.com and let's chat. (Or I can shoot you a message to the email in your gh profile)
Common examples of this entails race conditions, your test server might be super snappy on your local machine, but run much slower in CI, and expose race conditions that were unaccounted for. Other times it may be due to subtle difference between environments (service doesn't behave properly in CI due to build step differences or service is initialized differently in CI than locally).
The list of scenarios where local reproduction can be hard is unfortunately quite long, which is the downside of running end to end tests, the surface area they test (and can fail at) is really wide and complex, much like production failures. Enough so that teams find it useful to have proper debugging tools to debug what when wrong when tests fail in CI, like how you'd want proper debugging tools when apps fail in production :)
Of course there's the other side of rerunning failing tests locally, even if they are reproducible. E2E tests and their associated environment can take time to set up locally. Why bother spending 20 minutes checking out the failed commit, searching up internally how to spin up the local e2e environment to reproduce the failure, and then running the test(s) to debug. Instead you can just click through a UI and get everything you need, it's much smoother!
Saw your YC profile as well - I'll shoot you an email as I'd love to get a chance to chat with you more about what you thought about the messaging/product :)
We're focused on code-based tests, which is a lot more flexible in terms of things they can test, and scales better with the complexity of your application and testing needs. (Can have test code that sets up mock data, shortcut parts of your application to speed up tests, work with dynamic data, and be updated alongside your product changes so your tests never break).
We've found that with the advancements in DX for e2e testing libraries today, most developers find it quicker to just write tests themselves in code, rather than having it done in a 3rd-party UI tool. This also gives the bonus that developers are aligned with delivering high quality, well-tested code during their development cycle, as opposed to encouraging a "throw-it-over-the-wall" culture of having someone else test it for them after broken code has landed into the main branch.
Last point, we're not a full testing framework ourselves - we're currently compatible with Cypress, a widely popular e2e testing library, but are definitely keeping an eye on other frameworks such as Playwright as well :)
If it's absolutely a requirement at your workplace, we're happy to partner with you to make it work. We've discussed this internally and there's nothing inherent in our infrastructure that would make it impossible to deploy within an enterprise (the primary needs are the ability to run containers and have an S3-compatible object store).
If you'd want to dive into details I'm happy to email you via the email in your profile, or if you want to give me a ping at mike [at] deploysentinel.com
Typically our customers see the pain around debugability and currently waste dev & CI cycles rerunning tests and seeing their CI as largely ignored, or a productivity bottleneck to pushing code into main/production quickly. On top of that, they usually look for some tablestakes ability to load balance tests for parallelized runners, and some basic reporting to get a health check on their suite now and again.
Teams where their primary concern is around in-depth analytics and graphing reports, we have those on the roadmap, but can't say we're better than Cypress Dashboard yet :) But we think our focus on debugging overall gives teams a better chance of fixing errors, as opposed to reporting on them.
We have a free trial if you want to take it for a spin yourself! A lot of teams start off introducing us in an experimental PR to try out the product, and then merge into their main branch when they see the benefits. I'm also happy to chat in-depth on specifically where we might be able to help if you already have an existing Cypress set up, feel free to ping me at mike [at] deploysentinel.com
Glad to hear from others in the space! Hope to learn more if your team's GTM person jumps on :)
We indeed do approach from a different direction, having been built around the e2e testing use case from the start, we've focused on integrating with existing CI setups without swapping browsers (some teams really love the bundled Electron runner!), as well as knowing exactly what actions are running in your test runner relative to the replay in a likely familiar UI (answering questions like what DOM element did Cypress match for this command? What was the DOM like exactly when Cypress started running that command? Which command executed before this network request?).
Since you're from the team - one thing I've always wondered is how the pricing will work out for CI test runs? From the current public pricing I've found, it looks like it works great for developers recording test runs manually, but is extremely cost prohibitive at any scale if you're running it continually in CI.
If you're open to it - I'd love to chat more on what your experience has been debugging Playwright tests and seeing how we could help there! I'm at mike [at] deploysentinel.com
As for Github - no as well. We just have a Github app integration if you do to comment in on PRs. But we have teams using Gitlab/Gitlab CI as well. We print out our debugger links directly in your CI stdout or also add links into your JUnit report if that's enabled (ex for Jenkins). Though there's plenty of teams that just check our dashboard directly for test results as well! We just want to make it easy to access your test failure when they occur.
Our main product is indeed only for Cypress today, I'm assuming you're using Playwright today? If so I'm wondering if you've had a chance to try out their trace viewer feature as well (https://playwright.dev/docs/trace-viewer). We're itching to build on Playwright as well, but want to do it once we've built out a set of features that can provide a few times improvement over the existing trace viewer. So I'd be curious if you've used the trace viewer, and if you already see our product as having a few legs up on it :)
If you'd be referencing how Cypress compares with Playwright - I'd say the two largest differences is Cypress has a promise-like/chain based syntax for authoring tests versus Playwright allows for async/await syntax. Outside of that, Cypress provides a pretty awesome local developer experience (imo) whereas Playwright has a leg up in flexibility (multi-tab support, browser support, better simulated actions like mouse hovering).
If you're referencing how does our tool compare to what you might get with Playwright out of the box, Playwright does offer a really awesome trace viewer (https://playwright.dev/docs/trace-viewer) and actually had a brief chat with the original PM on Playwright a few weeks ago about it. It does capture a lot of similar debugging information - where we differ today from our debugger is a focus on making these historical runs easily accessible for engineers (don't need to go to your CI run, unzip an artifact, load it up locally, or build your own analytics dashboard for trends) as well as iterating on DX improvements like letting you scroll through/search all your network requests at once, and then jump to the point in time where a network request was made, which isn't possible in the trace viewer.
In the future we're looking into providing deeper information beyond just browser-level telemetry, so a few ideas we've kicked around with our users include capturing Redux/React state (like Redux or React dev tools do locally today), or even being able to sync up backend telemetry relative to what's happening in the browser. (Ex. show me the logs that were printed to stdout in the app server container when my test was clicking on the check out button).
Let us know when you get a chance to play around with it as well! Would love to hear what you think when you get in app.
- The experimental Chrome recorder is only for Puppeteer, so if you like Playwright, you're out of luck. (imo Playwright's ergonomics are way more refined than Puppeteer has and is the way we'd want to be writing tests in the future)
- The Chrome recorder introduces a _ton!_ of boilerplate code that's really bulky to work around if you want to maintain the tests yourself afterwards. I'm not sure what workflow the creators envisioned when they initially created the test generation system.
- You have to save the code to a file with the built-in recorder, you can't just copy and paste it into your editor, which is frustrating to me, especially if you're trying to record 1 new test into an existing file with a suite of tests.
- It lacks a few events you can create using the DeploySentinel recorder such as hover events, capturing screenshots, awaiting/asserting on text.
I think there's a few pros/cons to different tools out there, I've put together a side-by-side comparison here: https://github.com/DeploySentinel/Recorder#alternatives-comp...
The chrome built-in recorder is finally on Chrome stable recently, I'd say you should try it out, and then try out DeploySentinel recorder.
If you love the Chrome built-in one more, I'd love to hear what you think is missing!
I put together the DeploySentinel Recorder because I really wanted a quick browser automation/test-authoring tool that can do 80% of the script authoring for me, with a really smooth developer workflow and output maintainable code.
If you’re wondering how this compares with a few other awesome recording tools out there - I’ve put together a breakdown down on the Readme: https://github.com/DeploySentinel/Recorder#alternatives-comp...
I've gotten some really great feedback from early users so far on the extension. Would love to hear any and all feedback from the HN community! :)
Direct Links:
Chrome Ext: https://chrome.google.com/webstore/detail/deploysentinel-rec... Firefox Add-On: https://addons.mozilla.org/en-US/firefox/addon/deploysentine... Github: https://github.com/DeploySentinel/Recorder
I have a feeling that advice was great 10 years ago, and not so great now.
I think you might be confusing this product with shorteners like bit.ly. The format should actually be more semantic such as `m/roadmap/q2` rather than `bit.ly/Fa2ca`
That being said I agree the long term durability of the business would be a concern (as with any SaaS) for people. But that hasn't stopped the growth of other SaaS cos (and I'm sure there can be workarounds for ensuring usability even if they fold).
[0] https://chrome.google.com/webstore/detail/shortlink/apgeoooc...