Hacker News Reader

Comment by smarkov | Hacker News Reader

Please make this the industry standard for testing APIs.

I'm tired of having to look at Postman screenshots sent from QA. I'm tired of having to wait for them to press Send once I've implemented a fix. I know they're tired of waiting for me to do that, too. Hurl is something both the devs and QA can speak and write. It can be automated and a part of CI. It makes communicating expectations straightforward. It can be chucked along with PRs as a starting point for QA. I don't see a reason not to use it wherever possible.

View on HN

I have been using it for API testing manual and in CI of one of our services and it's been very nice. You can basically put a series of http exchanges for a workflow per hurl file and get a nice test suite that also checks directly into git.

For a team using VSCode you can try the vscode-restclient[1]

But really Hurl looks really interesting, being editor agnostic is the best solution for your problem, I agree.

[1] https://github.com/Huachao/vscode-restclient

I use the vscode-restclient and my primary reason is the conversational flows you can build against an API. Does Hurl support this? If so I would absolutely switch. All I would need to complete the experience is a plugin to do highlighting and integration with the Hurl files.

digikata’s comment above suggests it is possible. It would be great to have something that isn’t attached to the editor of choice.

Indeed it does with https://hurl.dev/docs/capturing-response.html

Also for VS Code integration specifically there is https://marketplace.visualstudio.com/items?itemName=JacobPfe...

Hi maintainer here! Thanks a lot for the kind words!

Great project!

A comment I'd have is that it's quite hard to find out who authored this fine piece of software.

Have you tried https://k6.io/ ? (Full disclosure: I'm one of the maintainers.)

It allows you to write load/performance tests in JS, commit them to your repo, easily automate them in CI, send metrics to several backends, use protocols besides HTTP, with a modern CLI, and many more features.

There's also a Postman-to-k6 converter[1]. The conversion might not be perfect, but it will give you a head start.

Note that the k6 philosophy is for developers to write these tests, similarly to how you write unit/integration tests, and to break the classic QA-dev cycle.

I don't want to steal Hurl's thunder, it does look great, but it's limited in features compared to existing peformance testing tools, and I'd personally rather write tests in a programming language, than in a bespoke text format.

[1]: https://github.com/apideck-libraries/postman-to-k6

If you maintain k6, can I log a complaint?

I feel like I used to be able to run the CLI with just an URL and it would issue GET requests to it -- without needing a .js script to run

Am I crazy, did this never exist? Please bring back/add this feature

I think you're mistaking k6 for another tool. k6 has only supported writing tests in JS. There are several other tools that are better suited for this single URL use case: ab, wrk, even Hurl. I doubt this will ever be supported in k6, since the focus is on scripting test logic, not firing off requests to a single URL (though this is certainly possible with a simple script).

> but it's limited in features compared to existing peformance testing tools, and I'd personally rather write tests in a programming language, than in a bespoke text format.

I’d hardly call hurl’s config bespoke, it basically amounts to HTTP codes and some header stuff, which basically everyone is familiar with.

From personal experience a couple of weeks ago I needed some http tests. I got a CI-ready Hurl config working on some endpoints for work sorted within like, 5 minutes of downloading it. In comparison I opened the page for K6, saw the “write tests using JS” and basically noped out immediately,

That's fair, and k6 is not for everyone. JS was chosen as it's a popular language with a large web ecosystem, but the tool itself is written in Go.

That said...

> I got a CI-ready Hurl config working on some endpoints for work sorted within like, 5 minutes of downloading it.

I'm obviously biased, but I'd argue that k6 is equally easy to get started with. Take a look at the docs[1].

> I’d hardly call hurl’s config bespoke

But it is. It has a unique syntax, and no other tool uses it, AFAIK. I'm not saying it's not simple or easy to understand, but it's not as expressive as a programming language. Which is fine if your tests don't require any logic, but from experience, in order to simulate accurate real-world traffic, which you probably want when writing a load/performance test, you do need some logic, different scheduling options, a way to process and analyze the metrics, etc.

I don't want to turn this into a versus battle, as there is a place for tools such as Hurl, ab, and wrk to coexist with tools like k6 and Locust. I just wanted to mention it as an option for anyone who hasn't heard about it, as any personal biases aside, I do think it's a great tool.

[1]: https://k6.io/docs/

> Take a look at the docs

Yeah, I did they start off with JS shenanigans, including importing packages, which immediately makes me concerned I’m going to have to deal with JS packaging dumpster-fire. There’s then one line of CLI args, followed by even more JS boilerplate.

> in order to simulate accurate real-world traffic, which you probably want when writing a load/performance test

I think the mismatch here is that Hurl is closer to being a http testing tool, not a load-testing tool, so most of these features are unnecessary.

> It has a unique syntax, and no other tool uses it, AFAIK.

Taking the example from the front page:

GET https://example.org

HTTP 200 [Captures] csrf_token: xpath "string(//meta[@name='_csrf_token']/@content)"

Character for character, most of this is bog-standard http verbs, a url, and an html path string. The Jetbrain HTTP tool also sports very similar interface, and I believe there’s a VSCode plugin somewhere that does a similar thing.

I personally like the approach of defining things via an OpenAPI and then using Dredd to validate the spec against itself.

Even for tools that generate the spec from source code, it is usually still possible for user error to define the metadata for an endpoint incorrectly. Dredd catches that.

I also think that validating an API against it's OpenAPI schema is a great methodology. You should checkout schemathesis, it's fantastic for doing that.

https://github.com/schemathesis/schemathesis

There’s also Step CI: https://stepci.com

(I’m one of the authors)

Hurl is brilliant though

I was just looking at this. We might use it at my company, but i was wondering, is this funded? Or is this just an open source side project for you guys.

Self-funded atm.

Not that I necessarily think it's best to stay with Postman, but have you looked at newman, which is the CLI runner for postman configs? We had postman as a test suite for something (which is more an API than an app), and I got tired of having to deal with setting up extra steps to test and of exporting the postman config to save in the repo, so I put newman on the test system and just run against the config directly in the test environment and check the output.

I don't necessarily recommend editing the postman config json directly to set up new tests as it's a PITA, but it's generally what I do so I don't need to keep importing and exporting it with Postman.

A tool designed for working with on the shell is likely better than what I'm doing with newman (since the config is not the most accessible), but it also meant I didn't need to rewrite a bunch of existing tests and verify they actually did the same thing.

The problem Newman/postman have is the same for every GUI based testing application. They almost always produce non human readable config files. Making any kind of code review of such changes at best extremely painful and at worst impossible.

IMO any testing tool that does not save it's test classes in a human readable format is DoA.

The bigger problem here is that you have QA, not the lack of tooling.

The bigger problem here is that you have devs, devs make bugs.

if you don't have devs there are no bugs, problem solved.

If devs have to do QA themselves, many issues magically disappear.

Honestly that sounds like making pilots build aircraft engines.

these are different disciplines that deserve to be done well.

Maybe I am biased because I spent the last 10 years in gamedev, or maybe this is another push to make devs do basically everything tech related: but if a developer tells me a feature is done I always look to QA for a nod.

That nod rarely comes, the feature is not done, the developer merely got it to work on their machine.

Then your developer should learn that "done" means more than "works on their machine".

The difference between devs and dedicated QA people is that devs know the dark corners of the implementation. They know the edge cases and scenarios that they struggled with getting right. That's where most testing focus has to be. QA doesn't know any of that. They can play through some scenarios that are the expected ones from the spec, perhaps have some hunch of what could be the tricky cases, but they don't actually know.

I love seeing all those downvotes for my GP comment. That's all the people working at companies where stuff that devs feel is below them is dumped on QA people, which is the main reason my company is well-known for shipping top quality (without having a QA dept) while our competitors struggle with quality (despite having a QA dept). That's all the evidence I need to back up my claim.

How big is your company and product?

Anual revenue is in the ballpark of about $1bn-$5bn.

If this happens to be a reference to Microsoft, their famous laying off of qa people has been regarded as a disaster by everyone outside Ms, with windows update quality dropping off a cliff.

It wasn't such a reference, no.

And I believe it's hard to do this kind of thing as an afterthought. If your whole engineering culture is built around having a QA dept then just firing that dept is obviously going to have disastrous consequences.

The devs should produce passing tests, not spend hours clicking around in their app like a QA person.

never underestimate the ingenuity of a good QA person. "app freezes while triple-clicking About button while changing wifi network when storage is 89% full and screen reader is enabled"

it's the same with good security folk. sure you can pretend you'll catch 100% of issues, but it's a delusion, good security or quality testing is a totally different mode of thought

> never underestimate the ingenuity of a good QA person. "app freezes while triple-clicking About button while changing wifi network when storage is 89% full and screen reader is enabled"

If such feature interactions matter then your application has bigger problems than a QA department.

> it's the same with good security folk. sure you can pretend you'll catch 100% of issues, but it's a delusion, good security or quality testing is a totally different mode of thought

Oh I'm not saying that good QA isn't a valuable skill! Of course it is, it doesn't just happen on its own. What I'm claiming is that it's a skill that should be employed as close as possible to the creation of the thing that it's assuring the quality of. So, ideally within the developer themselves.

Same thing with security. You will have a terrible security in your product if you first design and implement it and then put security in there as an afterthought by a dedicated security team. Ideally it's been at the table from day 1. So, a good security team works on educating your devs to do things right from day 1. Just like QA.

> If such feature interactions matter then your application has bigger problems than a QA department.

QA doesn't exist to check that your crap architecture solves the problem. They're there to be the skeptical person in the chain of custody from developer to production. The old adage "you can't test quality into software" exists for a reason. It's true. And you're right that it's the wrong architecture. But that's dismissing the very real purpose of QA which is to catch these kinds of bad architecture problems by testing for conditions that the dev team possibly didn't consider during development. QA is also there to make sure you're not making any unfounded assumptions about the context the software exists in.

To the extent you actually care about checking all of your assumptions and not just the ones you're cognizant of, QA is very useful.

When I'm not expecting guests my house is always clean

Reply on news.ycombinator.com