hassy

This is a great blog post! just taking the opportunity here to comment on this:

> Finally for full scale high fidelity load tests there are relatively few tools out there for browser based load testing.

It exists as of a few months ago and it's fully open source: https://github.com/artilleryio/artillery (I'm the lead dev). You write a Playwright script, then run it in your own AWS account on serverless Fargate and scale it out horizontally as you see fit. Artillery takes care of spinning up and down all of the infra. It will also automatically grab and report Core Web Vitals for you from all those browser sessions, and we just released support for tracing so you can dig into the details of each session if you want to (OpenTelemetry based so works with most vendors- Datadago APM, New Relic etc)

> I'm very biased

Don't write your own load testing tool other than as a fun little exercise. At least not without understanding coordinated omission and thinking about workload modeling (open? closed? hybrid? all of the above?) [1]. Get this wrong and the results produced by your tool will be worthless.

Once you've got that out of the way, don't forget that you'll want a distribution story. It does not matter how efficient your tool might be on a single machine - you'll want to distribute your tests across multiple clients for real-world testing.

"Sure it's easy" you might say, "I know UNIX. Give me pssh and a few VMs on EC2". Well, now you've got 2 problems: aggregating metrics from multiple hosts and merging them accurately (especially those pesky percentiles. Your tool IS reporting percentiles rather than averages already, right?!), and a developer experience problem - no one wants to wrangle infra just to run a load test, how are you going to make it easier?

And, this developer experience problem is much bigger than just sorting out infra... you'll probably want to send the metrics produced by your tool to external observability systems. So now you've got some plugins to write (along with a plugin API). The list goes on.

I'm very biased, but it's 2024. Don't waste your time and just use https://www.artillery.io/

1. https://www.artillery.io/blog/load-testing-workload-models

curl is fantastic. There's also HTTPStat which provides a waterfall visualization on top of curl timings: https://github.com/reorx/httpstat

There's also Skytrace (made by yours truly), which provides timing info as a waterfall visualization inspired by HTTPStat + lots more (syntax highlighting for responses, built-in JMESPath support, command-line assertions and checks etc) - https://github.com/artilleryio/artillery/tree/main/packages/...

The main thing about the advice comes immediately before the bit you’re zooming in on. Want to talk to people who have seen hundreds of companies deal with what you’re dealing? YC is great for that.

Did YC S21 which was an all remote batch. my 2c.

YC is 100% what you make of it. It's not a lean back experience.

I did not meet most of the companies in my batch but I've gotten to know many founders through YC that I would not have otherwise. Founders that have been source of advice and support.

Network of clients - yep don't go into YC expecting to sell to other YC cos. It is easier to get warm intros through the network though.

YC advice, office hours specifically - it's what you make of it too. Expecting a group partner to know your space in great detail is unreasonable but if you recognize that they've seen hundreds of companies with similar problems and make use of that pattern matching, you can get very valuable advice. Some of the advice I did not take and did the opposite and it was the right decision. And some advice that I did not take was exactly right, but I only saw it in retrospect months later.

Fundraising - being a YC company definitely opens doors, and also helps protects you from bad actors who have to think twice before fucking with a YC co. Very valuable for any first-time founder. You have someone to sanity-check everything, terms you're not sure about etc. The bump in valuation is real too.

I'd do YC again in a heartbeat.

Clickable links:

Blog post with demo: https://www.artillery.io/blog/this-load-test-cost-us-how-muc...

Project on GitHub: https://github.com/artilleryio/artillery

This lets you load test with Playwright (very similar to Puppeteer):

https://github.com/artilleryio/artillery-engine-playwright

Yes indeed but wrk2 or Vegeta is still better for this particular use case (unless k6 has support for setting a constant RPS rate, afaik it does not), as otherwise the overhead of establishing a new TCP connection for a single HTTP request will dominate the benchmark.

Yep, k6 suffers from coordinated omission [1] with its default settings.

A tool that can send a request at a constant rate i.e. wrk2 or Vegeta [2] is a much better fit for this type of a performance test.

1. https://www.scylladb.com/2021/04/22/on-coordinated-omission/

2. https://github.com/tsenart/vegeta

Bombardier is cool, but serves a very different use case.

I mentioned elsewhere in the comments that we have a Docker image, and are working on other methods of installing the CLI to alleviate some of these dependency-related concerns.

Getting side tracked here, but there seems to be a common sentiment when it comes to Node.js that it's uniquely insecure. Node.js has indeed had some unfortunate press when it comes to supply-chain security, but every other runtime is susceptible to those attacks (PiPy, Gems, Maven, Rust Crates). Ultimately of course, if you choose to avoid using any software built on top of those stacks, that's your choice.

Artillery specifically is no different to any other Node.js-based project in how large the dependency tree is. VSCode for instance is used by millions of developers has 1.6k dependencies [1].

1. https://github.com/microsoft/vscode/network/dependencies

I haven’t come across Hurl before. It looks cool!

Yes, those dependency trees can be large. Yes, supply chain attacks are a real threat. But Node isn’t that different than Python or Ruby in that regard. How far down the stack do you personally choose to go? I trust you’re familiar with that famous paper published by a certain mr Thompson in the mid-80s?

The world is a big place. There’s a lot of software out there written in Node.js, used happily and productively by millions of developers, many of them in corporate environments.

Given the opinions you expressed elsewhere in the thread here I think it’s clear that this tool is not for you. I hope no one is forcing you to use it.

thanks for trying it out! adding “make timeouts configurable” to the todo list.

you’re right on that YAML-as-JSON thing. If everything is quoted as JSON, those type conversions shouldn’t kick in. Otherwise there’s room for surprises - perhaps we can do something to make those cases more obvious.

“mini” in the sense that it doesn’t do everything that curl does. curl does a whole lot. this tool focuses on more common use cases and makes them friendlier. plus with request waterfalls and assertions it does things curl can’t do.

the number of packages is a bit of a misnomer anyway. Artillery Probe is part of Artillery which does load testing, with multiple protocols, support for multi-step scenarios, publishing to a variety of monitoring systems (Datadog, Prometheus etc) and more. That’s what most of those packages enable.

sure, that’s a valid concern in some environments. fwiw we use Snyk.io for dependency scanning

Yeah so the JSON quoting part is something I’m pretty pleased with. We use a YAML parser (JSON is a subset of YAML) to parse those values, which is what allows for double quotes to be omitted.

Good point on iterating on the query! We already save the body into a temp file, so we can make Probe be able to run queries on a file. Adding it to the todo list. :)

(In my own workflow I use gron a lot for getting an overview of the shape of unfamiliar JSON, super handy tool)

As to an interactive shell… yes, 100%. Kicking ideas around something like that as well!

A few things :)

Syntax highlighting

Request performance waterfalls

Built-in JMESPath queries for JSON (same syntax as in AWS CLI). XML and HTML can be queried too.

Ability to set checks on the response. This one is super handy for quick acceptance tests.

curl is amazing! I love curl. This is not meant to replace it completely. Probe just makes those most common use-cases friendlier, like looking at headers, or inspecting JSON retuned by an API with syntax highlighting, or querying without needing to reach for jq etc.

thank you! we just grab the timings info for the request from the underlying HTTP library, and sprinkle some ASCII art on top. That part was inspired by httpstat [1]

We want to extend those with support for Server-Timing next, and also Core Web Vitals [3] (via Playwright) for web pages.

1. https://github.com/reorx/httpstat

2. https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Se...

3. https://web.dev/vitals/

if you’re on a Mac, Artillery has a Homebrew formula too. We have an official Docker image too. Other ways of installing are on the roadmap too (self contained tarballs and binaries).

As to what it does better than curl/wget. Probe is geared towards interactive use with HTTP APIs. So you get syntax highlighting and pretty printing for JSON, built-in querying, request waterfall visualizations, and ability to set expectations on every response.

HTTP/2 is supported.

clickable url: https://www.artillery.io/blog/swiss-army-knife-for-http-test...

Playwright's built-in codegen tool creates scripts that can be run as load tests with this project :)

https://playwright.dev/docs/cli#generate-code

Yes! that's how we ourselves run it. The project includes a ready-to-go Dockerfile: https://github.com/artilleryio/artillery-engine-playwright#u...

clickable link: https://github.com/artilleryio/artillery-engine-playwright

Artillery (YC S21) | Full-stack Product Engineers | Full-time | Remote (GMT±4)

We're building a modern performance testing platform for DevOps & SRE.

Today, we make it easy to run planet-scale load tests from your own AWS infrastructure. Developers love Artillery because it's modern and comes with batteries included. SRE & Platform teams love Artillery because they can provide a self-service load testing platform to developer teams. Security teams love Artillery because it runs in your own VPC with no data leaving your AWS environment.

We love, love, *love* load testing, and have a lot (like, A LOT) of new ideas to shake up the space, but the Big Vision is much bigger than load testing. The state of testing tools for today's complex production systems is pretty lacklustre. We'd like you to join us to help us build a better future.

$110-$125k + equity, remote anywhere in GMT±4.

More here -> https://www.artillery.io/blog/artillery-hiring-product-engin...

Hello HN! Hassy from Artillery.io here. Load testing complex web apps is no walk in the park. If you've ever had to do it, you know. Traditional load testing tools are designed to work with API endpoints, whereas the concept of pages is a more natural way of thinking when testing web apps. A page may make calls to multiple endpoints, some of which may depend on in-page actions (and even in-page Javascript). Whereas APIs usually have specs (e.g. OpenAPI), pages usually don't. Trying to track it all down in something like Chrome DevTools can take ages. It's a mess.

So we thought, why not try load testing with real browsers instead? Especially if we can reuse existing E2E testing scripts we may already have? (based on Playwright) Playwright gives us an excellent API to run headless Chrome, and a way to record test scripts with "playwright codegen". Couldn't we try running thousands of browsers that way?

Turns out we can. :) That's what this project does. It's super early days, and I'd love any thoughts or feedback!

Member of team artillery.io here, my 2c on the subject. We chose MPLv2 specifically to address licensing concerns. With MPLv2, you can build on top of Artillery, you can build plugins and extensions for it, and integrate it into your systems without worrying about licensing. It's a well-understood license with very clear boundaries between your code and MPLv2-licensed dependencies (e.g. all Hashicorp tools use MPLv2).

That's unfortunately not the case for AGPL. There's a reason a lot of companies have policies banning any AGPL dependencies outright, Google probably being the most prominent example: https://opensource.google/docs/using/agpl-policy

AGPL is designed to be extremely viral and has not been tested in court. The definitions of boundaries between your code and AGPL-licensed code are not well understood. Consider that MongoDB, probably the most popular AGPL-licensed project in use before 2018 when they switched to SSPL, had to explicitly publish their drivers under Apache because communicating with an AGPL dependency over a network was not sufficiently distant enough to prevent infection. https://www.mongodb.com/blog/post/the-agpl

As engineers we need to be aware of licensing implications of code we depend on. I am obviously not a lawyer, and obviously your employer's legal team are the people you should be talking to if you have an absolutely critical dependency on an AGPL-licensed project.

Artillery team member here. "Rather pricey" really depends on what you compare it to. Artillery Core is free. Artillery Pro costs money, but it runs directly in your cloud environment, so it's extremely cost effective compared to hosted SaaS solutions. We designed it for large-volume use, especially in CI/CD pipelines. There are no limits on test minutes, vuser concurrency, number of tests you can run etc.

Can you DIY a solution that will let you instantly scale up from running tests locally to running them on 500 workers in any of 13 different geographical regions? With no servers to manage or maintain whatsoever. Sure you can, but it's not the best use of time for a lot of teams.

Many CI server configurations will rely on running jobs under a role which can only assume other roles. Individual jobs will then assume the role they need.

Something like this infecting a popular Node.js / Python / Ruby package could potentially do a lot of damage.