tmcneal

Reflect | Account Executive | Philadelphia, PA / Seattle, WA

Reflect (https://reflect.run) is a no-code testing platform for web applications. Developers and QA testers use Reflect to create end-to-end tests for their web apps, and we let them do it 10x-100x faster than traditional code-based tools.

With the ubiquity of web technologies, and with so much of our day-to-day work going digital, it’s more essential than ever to provide software teams with better tools to ship software faster and with higher quality. One critical component to delivering software fast and with high quality is testing. With Reflect, our goal is simple: to be the best platform for end-to-end testing web applications.

We're looking to make our first sales hire (all sales thus far have been founder-led). More information about the role is available here: https://www.workatastartup.com/jobs/41675

Selenium IDE uses a Chrome or Firefox extension to record your actions, but they're rewriting it to be an Electron app. I'm not sure about the status of that rewrite though.

But the benefits we have vs. what you experienced with Selenium IDE is:

- Instead of using an extension to record your actions, there's nothing to install to use Reflect. We spin up an instrumented browser inside a virtual machine and screenshare that with you within our web app. It records your actions and translates those actions into a repeatable test automatically.

- Since we have complete control of the test environment, we can do cool things like make recording file uploads really easy or present a nice workflow for getting coverage of visual regressions via screenshot testing.

- Our recorder is more accurate :)

- For every test you run within Reflect, you get a video synced up to the steps in your test, along with console and network logs from the test run.

- Instead of emitting Selenium code that you then maintain like a normal code-based testing tool, we are completely no-code meaning that even updating a test is completely codeless. So for example if you need to make bigger changes to an existing test, we expose a way to re-record only the portions of the test you want to change, and keep the rest as-is.

If you're willing to try something a bit different, we're working on a no-code approach to test automation that tries to avoid the pitfalls of code-based testing [1]. We're focused on building a E2E testing platform that (1) makes it as easy as possible to create tests and (2) makes tests easy to maintain. Easier said than done but that's our goal.

[1] https://reflect.run

I'm co-founder of an end-to-end testing product, but I don't actually agree that better end-to-end testing coverage means you should create less unit tests. The reason why is because there's a class of things you want to test that are much better served at the unit testing level. Unit tests are always going to be faster to execute and by definition have a much smaller scope, which is helpful in reasoning about what you're testing.

Our product is really focused on making end-to-end tests much easier to create and maintain. Here's an example video I recorded yesterday of automating Monday.com's sign-up flow using our product: https://www.loom.com/share/1336a5bfa9f54f2190269961991802b5 - looking at this I don't really see what unit tests this would replace, rather I think it gives coverage for things that you'd often have no automated coverage for - i.e. the interplay between disparate components along with interacting with email - basically the steps that form a complete workflow from the user's perspective.

We sell a product in the QA space and get to talk to a lot companies about this.* The pattern I see is that companies in the 1-10 employee range tend to have a QA process that comes about organically. You start out with zero customers so a bug doesn't really "cost" anything. As you get customers, the cost of a bug grows to a point where it makes sense to do manual test passes - maybe before a feature release, maybe after every deployment. Usually it's in a Google Doc or Google Sheet, and it's either one person running through the whole thing or a bunch of people divide-and-conquering it.

Eventually QA ends up being owned by either the Engineering team or a standalone QA team. I've seen both work, and both not work. Generally if it's not working for Engineering, it means they're not automating enough or the tests are flaky and they're just ignoring it. If the QA team approach is not working, they're probably overworked... Likely because the engineering org has deeper quality issues they need to fix, but they're trying to make up for it by throwing bodies at it in the QA org.

* We have a no-code test automation product that is used by both QA and Engineers: https://reflect.run

Me too. My co-founder and I both have young children and we were in the last YC batch. We live on the east coast and wouldn't have been able to do it if it wasn't remote.

If anyone is applying and is looking for someone to review their application or has questions about the remote batch, feel free to email me (email is in my profile). I was helped by many people over the past few months and would love to pay it forward.

Edit: And for anyone wondering about it being remote vs. in person - YC was an overwhelmingly positive experience for me and I would encourage you to apply.

Sorry - I just saw this! Some customers on our Enterprise tier have asked about this and we've agreed to provide them with a one-time Selenium export of the tests as a contingency for if we go out of business. Happy to chat about that if you'd like! My email's in my profile

We have the ability to handle MFA logins that use email as the second factor. The way it works is you configure your user account to have an email address that we control (e.g. foo@account-randomchars.email.reflect.run). When the multi-factor challenge comes in, we receive the email, parse it for the challenge code, and fill it in live in your test. We don't have SMS support at the moment but we could take a similar approach there if SMS is used as a second factor. There's a one-time setup here where we set up our system to parse your challenge email.

No support for TOTP unfortunately, but if the SMS or email-based challenge support would work for you feel free to reach out and I can talk to the specifics of the one-time setup - todd at reflect dot run

This looks cool! We use the DevTools protocol for a few things like our mobile emulation and screenshotting. Happy to chat - my email's todd at reflect.run

Thanks for reporting! We'll get the misalignment of the form fixed up for Firefox. The Google sign-in says 'continue to amazoncognito.com' because we're using AWS Cognito as our auth provider, but it looks like there's a way to configure it to be "continue to reflect.run" so we'll look into that!

You're right, there's certainly security implications for magic links. Unfortunately for an auth that incorporates hardware keys, I can't think of how you would test behind that without some sort of workaround, but I may be overlooking something.

Ah - sorry for misunderstanding. No we don't have a view like that, though I can see that being pretty handy. This is great feedback - thank you!

Extremely biased of course :) But here's where I see the advantages of Reflect vs. Cypress:

- Cypress has a really nice local development experience that feels at home to front-end devs. I would describe Reflect as a nice remote recording experience that simulates an end-user. So we're kind of attacking the same problem from a different perspective. You can technically record Reflect tests against your local env with ngrok, but Cypress is certainly a better local testing experience. So advantage Cypress here.

- There are actions like interacting with iframes, drag and drop, links that open a new window, and file uploads that range from difficult to almost impossible to do in Cypress. We support these actions out-of-the-box.

- Similarly if you want to do visual testing in Cypress you'll need to integrate with a third-party visual testing tool like Percy or Applitools. We have visual testing support built-in.

- I've seen folks struggle bridging the gap between using Cypress for local testing and actually getting it set up in their build/deployment pipeline. Since we're a fully managed service it's just a single API call to get Reflect in your CI/CD. We also have built-in scheduling so if you just wanted to run your tests at a fixed schedule you don't need to integrate with anything, which I think is a nice way to ge t going quickly and prove out our tool.

- Because of Cypress's design, it's not so easy to get Cypress tests to run in parallel. This is really important because E2E tests take way longer to run vs. unit and integration tests, and the best lever IMO for reducing this is parallelization. This is also another tripping point for folks getting Cypress set up in CI/CD.

- The final differentiator I'll mention is really the difference between a code-based vs. codeless approach. We're trying to reduce the time it takes to create and maintain tests and we think the way to do that is to basically handle as much as we can automatically on your behalf. Instead of writing code that simulates actions, you just record your actions and we generate the automation for you. So for a flow like adding to cart or user registration, you might only take you a few minutes to set that up in Reflect but it'd be a lot longer to do in Cypress. Certainly as your test suite grows things like reusability become really important, and we support that as well. This also means that non-developers like QA testers w/o dev experience, PMs, designers, and customer support can write and maintain tests.

Thanks! Yep tests are versioned - if you click on the 'History' button when viewing a test you'll see all the executions within your retention period. We don't show a diff view of historical changes, however we do show a diff view when a text change causes your test to fail. So say if your 'Log In' button changed to 'Sign In', we would show you a diff of that text change and give you an option to make 'Sign In' the new baseline used when running future tests.

If the elements you interact with are DOM elements and non-DOM elements within a canvas then we should be able to detect and replicate those actions. The biggest remaining issue might be performance - our VMs don't have a GPU attached so depending on the application it may be slow because WebGL is not running with hardware acceleration.

We don't offer an on-premise version, but we do have customers testing environments not accessible to the public internet. They're doing it by allow-listing a static IP and we configure all traffic to come from that IP for that account. There's other options if a static IP allow-list doesn't work - we've specced an approach for a prospect where we would set up an agent inside their firewall that we use to access their internal environments. This is an approach used by other tools to access secured environments - we haven't done it ourselves yet though.

I would suggest adding the ability to auth via a magic-link in your web app. This would allow your tests to bypass the auth flow entirely by passing an auth token as a request parameter in the first URL of your test. You can pass new auth tokens when you go to run your tests either via the UI or via our API if you have it hooked up to your CI/CD pipeline. More docs on how to do it via the API is here - we call these 'request overrides' in our docs: https://reflect.run/docs/developer-api/documentation/#run-te....

In terms of added security, some options there would be to only enable these "magic links" in staging and only enable this type of auth for a least-privledged user account (e.g. no admin or employee accounts could auth this way).

Totally! If you have data-test* attributes set up, we will use those first when generating selectors for each action. The full list of data-test attributes we use is listed here: https://reflect.run/docs/recording-tests/creating-resilient-.... If we don't find data-test* attributes we'll also look for other attributes that tend to have a good degree of specificity, like alt, rel, schema.org, and aria-* attributes.

We fully support e-mail based logins, but OAuth can be challenging.

Github OAuth for example will issue a 2FA email-based challenge non-deterministically. We handle that by detecting the challenge and filling out the challenge code based on the contents of the email sent by Github. This requires a one-time setup where you add an email address we control to your Github user so that we can read and parse it.

For Google OAuth we can execute all the steps but the two issues there are (1) we run everything in a single window and some web apps don't like that because they assume the oauth flow will happen in a new window, and (2) sometimes Google prompts you to answer a security question and we don't yet support marking test steps as optional.

What our customers have been doing instead is setting up a mechanism to auto-log in the test user using a magic link. Basically sending in a one-time-use auth code to a URL in their app that then authenticate the user. I think some platforms (Firebase?) have built-in support for this.

I'm certainly happy to brainstorm what could work best for you though if you'd like (my email: todd at reflect.run)

False positive failures are a really common issue with existing E2E testing tools, so we try to do a number of things to prevent them in Reflect tests:

- We generate multiple selectors for each element you interact with. So if in the future you change a class or id, we'll fallback to other selectors when running your test. We also only choose a selector if (1) it matches a single element in the DOM, and (2) that element is interactable (e.g. not completely hidden), and (3) if it has text associated with it then we'll only choose it if the text captured at recording time matches the text at run time. This helps prevent us from selecting the wrong element when running a test.

- For React apps, we use heuristics to ignore classes generated by CSS-in-JS classes like Styled Components and EmotionJS. We also have the ability to target elements based on React component name (it requires a one-time setup on your side to enable this)

- For Visual Testing failures (e.g. you've screenshotted an element and now that element has a different UI) we have a simple workflow to 'Accept Changes' and mark the new UI as the new baseline for tests going forward.

Certainly more to do here but this is one of the key problems we're looking to tackle.

Just curious, why do you not allow the interviewee to use jQuery? Do you expect the candidate to know JavaScript's DOM-level APIs without having to look them up?

PMP certifications were established in 1984[1], and while I don't think the certification itself is nearly as abused as "Certified ScrumMaster", PMI and Six-Sigma are definitely touted as panacea, especially in the enterprise.

[1] http://en.wikipedia.org/wiki/Project_Management_Institute

Yep, it can be pretty challenging to deal with: http://sitr.us/2011/07/28/how-mobile-safari-emulates-mouse-e...

I can't imagine you'd come across many folks with the experience you described looking for a sales role at 4 person startup. If a salesperson was employee #4 at a startup, and that startup was successful, I would imagine he or she would pursue a Senior Sales / VP of Sales role at a more established startup vs. a lateral move.

I think one of the points you're getting at here is that there's many different ways to solve a problem in Scala, versus something like Python. It's definitely true, as can be seen when comparing a codebase like Lift (https://github.com/lift/lift) with Play (https://github.com/playframework/playframework). Both are two MVC frameworks written in Scala but have vastly different coding styles.

The company I work for is a Scala shop, and we've found success in codifying our coding style using the Scalastyle linter. The combination of a linter and peer code reviews has been very helpful for us in figuring out which language constructs and libraries we use and which we shy away from.

The closest I've seen an ORM come to this is Squeryl: http://squeryl.org

As another data point, I've found Philly to have a lively and close-knit startup culture, albeit much smaller than NYC and SV. Gabe Weinberg (founder of DuckDuckGo) keeps an on-going list of links related to the Philly startup scene: http://foundedinphilly.com

I'd be curious to hear what ways you think Philly can improve.

To be fair the hacker didn't say the keys were stored on the same server as the credit card numbers, he said they were stored on the web server. It's most likely the database containing the CC numbers resides on a separate set of boxes than the web servers.

We've got our JS unit tests running in Jenkins via a Maven plugin (jasmine-maven-plugin). IMO Maven can be a pain to set up but the end result is less janky than custom shell scripts. Actually I think the same could be said for Jenkins itself. :)

Now that you know that this segment of customers have a higher lifetime value than non-tablet users, how do you intend to capitalize on this knowledge? I understand you can't get too specific on future strategy... but has this data changed how you intend to develop and market your mobile channel in the future?