Robot Framework: generic open source automation framework | Hacker News Reader

Robot Framework: generic open source automation framework

robotframework.org

163 pointswiradikusuma101 comments

I am less than lukewarm on those sort of "Natural-Language-Like" DSLs that seem to mostly come out of the QA space.

When I worked together with a QA team for automated testing, they used Cucumber, which uses Gherkin as an english-but-not-really DSL. Now the idea is obviously, that someone who doesn't know how to code can still describe and read test scenarios. But what ended up happening eventually is that I spent a lot of time writing code to support different Gherkin expressions, but then had to take over writing the Gherkin expressions as well, because even though Gherkin -looks- like english, it still behaves like a programming language. In the end, it would've been much easier for me to just write the tests in a programming language from the start, instead of using an abstraction layer that turned out to serve no purpose.

What I took away from that frustrating experience is that a lot of people think that learning to code is learning a syntax of a programming language, and if you could just do away with that pesky syntax, everybody could code. But, obviously, the syntax ist just the initial barrier, and actually coding is crystallizing and formulating intent - something which programming languages are much better at than natural languages.

But for some reason, this low/no-code idea never seems to die down, especially in some spaces, and I have the feeling I'm just missing something here.

Yeah, I share the sentiment. What it boils down to is that you need to write the test two times. The supposed feature that once written the keywords can be reused is mostly a joke. There will be few heavily-used keywords, and a really long tail of one-time keywords, which is a pain.

I see a single advantage to these frameworks though. They force you to write a test in a specific way that looks like a real-world scenario with clear delineations between setup stages, assertions, and so on. Not that it's impossible to achieve with regular programming language, obviously.

> The supposed feature that once written the keywords can be reused is mostly a joke.

I've seen well maintained and designed cucumber / rspec / robotframework "code" and i've seen some really horrible shit. Keyword re-use is really a joke if you are not familiar with any good practices to actual software development. Granted, in "low code" scenarios, the worst selling point is "you don't need to know how to write code" - which imho is bullshit and from my pov, statements like i quoted echoes that.

Disclaimer; i partake and contribute to RF ecosystem. And i do prefer to write my tests or process automation with it even thought i can write code in multiple languages..

I like using catch2 (c++ test framework). the keywords are only reused when they group together in the code.

    GIVEN("Some setup") {
        // code to create the setup
        WHEN("some use of the setup") {
            // do something
        }
        WHEN("something else"){
            // do something unrelated to the previous when
       }
    }

I will never understand the reason for Cucumber to exist. As you say, it just introduces unnecessary complexity for no reason.

If you can't code, you shouldn't be writing automated tests. Most of the "benefits" of Cucumber can be gained by using a good reporting framework. I like to use Serenity as I love its reports.

I can count on zero fingers the number of times I've worked with a non-technical QA person who's written tests, let alone benefitting from cucumber. As the GP says, the rest of the time is just wasted maintaining a pseudo-natural language programming language.

Unfortunately, I'd need a few more fingers. Not too many though which makes that argument for Cucumber fairly moot. I did work somewhere where each team had 2 testers. One to write the cucumber and one to write the glue code for it. It was about as successful as you could imagine.

I'd also say "non-coding" rather than "non-technical". Coding is just one aspect of being "technical".

There is benefit in making the tests easy to read but this is better done in actual code, e.g. the screenplay pattern.

yeah we had cucumber at a company I worked for. it was touted that PMs and other non-coding staff could write tests and how great that would be. After we switched tests to the Golang cucumber implementation, exactly no one outside of Dev team wrote any tests. meanwhile Devs have to use this awful new framework everytime we make a change. burn it with fire I say

The fact that gherkin can be be turned into an automated test is a side effect. The real purpose is to get non-programmers to put their requirements in a rigorous form so that programs can implement the spec and know it will do what is intended.

I have never seen it work in practice, but that what it is supposed to be about. If your managers are not looking at the gherkin and making suggestions then there is no point in doing it.

I'm a big fan of the Gherkin syntax and the output of my Example Mapping sessions is generally a big pile of them.

Automated tests are then written based on these but not "using" these. By not "using" the Gherkin scenarios for our tests we can be looser with the language and not have to worry about RegEx.

If a manager, or other non-coder, wants to see the tests, they can look at the very pretty report that was generated.

My experience with trying to use Cucumber for tests pretty much shown me that it doesn't work there. But one thing that I played a bit and seemed that it might work was using Cucumber for writing requirements. At least for when the requirements are written by not technical people, it could give them better structure and for me as a tester make it easier to spot possible problems in them. But I haven't had to chance to try this more, so it might up end not working just like tests themselves...

> I will never understand the reason for Cucumber to exist

In a previous company I had one use-case that I thought cucumber helped out a lot at.

We were writing software that would be vigourously tested and approved by a customer who didn't have any technical knowledge. Their initial plan was the "Excel sheet of death" test case management system with really fuzzily named tests.

I convinced them to write Cucumber tests, which turned out to be around 100, using the very explit syntax that then got called stupid by the cucumber devs and moved to the "training-wheels" gem. I never expressed what we were doing as cucumber tests. I just told them "Write them like this and it is easy to understand" and they agreed and did so, then they were signed off by our project team and them.

After that I then automated them our side so that we could run them quickly and easily in our pipeline while the customer could still use the tests for their tedious manual user acceptance testing at the end.

It worked out quite well. The customer never knew we automated the tests our side, they were just thrilled that something like 99% tests passed first-time and the 1% that didn't were down to interpretation differences.

Congratulations! I think this is actually the first time I've seen a use for Cucumber. Obviously, it would have been better if you didn't have to do this but I've spent enough time as a consultant to know when to just work around the customers.

"Not being able to code" is not black and white. Many can follow basic steps, as long as it gets not too complex. Thus "Low code" tools like Powerautomate,selenium ide, ahk or UIvision are great for people to do test automation. They are a quick way to create an automation. And many users learn/start to code this way.

Does Robot Frarework try to be natural-language-like?

The code consists of a series of invocations of functions ("keywords" in RF terms) with parameters. The syntax is tabular.

When I read the intro, it reminds me of Forth.

Although I only gave it brief reading a couple of times. (RF was used by QA at my past job.)

It's based on "traditional" test plans written in excel, so it's a mix of pseudo NLP and a spreadsheet.

Worst of both worlds, IMO.

How is it based on anything in Excel? There's nothing about test plans in RF. Test plans are about schedules, strategy, deliverables, roles etc. And those are typically garbage. Absolutely nothing related to RF.

Test plans that I've seen go one step farther: written instructions on how to setup the test, and verify results.

I've never seen a manual tester follow the steps that are written down, but I've seen a lot of effort put into writing down the exact steps anyway.

Might be difference in local terminology, but by "test plans" I mean spreadsheets, typically in Excel (sometimes printed and bound), that described step by step every test for manual tester to follow.

And that's how a lot of testing is done in various companies, especially where manual testing is still common, at least in Poland.

From my POV, the idea of Natural-Language-Like tests is it is theoretically more inclusive, can avoid duplication, and can create a more abstract description.

Granted, I've had middling success convincing non technical people to write or even look at tests, but I think it's irresponsible not to try to create this bridge, depending on the audience of the code. Because very few projects are well documented, and this syntax can act as documentation. In my current work, I am using BDD style to create high level flows, where screens are automatically captured, which is useful both for verification and generated documentation, especially when versioned with code.

It can avoid duplication (and its pitfalls), because in unit style tests, there is a text description along with the code, which can drift away from being accurate. If the test is the description, that won't happen.

It creates a more abstract model, because the tests are forced to a natural language, which will tend ot make it less implementation dependant, so the implementation can be swapped out. I've had good success with this approach, seamlessly migrating a complex workflow from one backend to another, and the tests were kinda pricesless.

I also find keyword and flow re-use to be quite high, and it's great how this approach means the more tests you write, the less low level, implementation specific code and scenarios you need to write.

I’ve spent a fair number of years doing corporate cucumber training.

At every client I emphatically insist that if they are not using Gherkin as a “Rosetta stone” for the business folks then they’re wasting their time with cucumber. Don’t go to the trouble of writing gherkin if you’re not using it as living documentation. Save yourself loads of trouble and just use an ordinary test framework.

What you describe may apply more to writing code, rather than reading code. Of course, knowing syntax is not sufficient for programming, but if you want just to review existing code, (unfamiliar) syntax may be an obstacle to non-programmer domain experts.

I've read something along these lines about the choice of OCaml at Jane Street, where they argued that their investor-types are able to proof-read new code, despite not being programmers, because the language "reads like english", similar to SQL or COBOL.

Can you really effectively proof read if you're just guessing about the underlying semantics? If the goal is just to give investors a false sense of security then I suppose it achieves that

I work in an environment which has dozens of products, very significant complexity and legislative needs. Cucumber works very well for us because we use it for specifying outcomes. There's way to much Gherkin out there which tries to use it for describing a journey (I click this, then this, and this....)

It works really well for us to get a shared understanding between a non-technical business, devs and QA.

That said, we then hand it the Gherkin over to the SDETs who fettle it according to our house style (which is strict and must pass enforced linting) and it does take gardening as the product matures. It's our SDETs who write the test code: we wouldn't dream of letting a non-technical person actually drive the test logic itself.

QA should be writing the underlying code along with the BDD-style tests, using something like a Page Object Model design, with BDD-wrappers on top mostly for non-technical analysts like PMs and BAs to understand coverage and add test cases. I think if you're doing it for anyone else's benefit it falls flat. If it's a no-code solution for QA then it becomes a drag and you should hire an SDET to manage it (and train the other staff to write the underlying code too). If BA/PM is not involved in test design it's a pointless wrapper, I agree. Personally I'm not a fan and spend my time focusing on better reporting so people can see what's tested & if they want an additional case I can just add it myself.

The problem is there are three to 4 veins of QA. SDET (largely automators, tend to scale and manage the automation), analysts, who go over test results and pull out trends, manual testers/non-automators, because nothing brings out the bugs/bad assumptions like putting a non-techie in front of something, and then... well, I'd lump software maintainers/DevOps into it too.

I'm not impressed at all by Gherkin, BDD, or Cucumber. Never once has any business person gone "oooh, let me read the tests!", And in the rare circumstance they do, what you'll find is a secondary grammar develops where particular step implementations gain loaded significance, far and above the natural language implementation of a step, making things complicated.

At this point, I just tell people to keep notes, make checklists, get familiar with the data model for test data authoring, then go eith God, and always put the User first.

Mileage varies greatly, but there's a culture problrm in a lot of places where "the fiture must go out" always takes precedence over "the spec must be right".

And no one ever wants to rip apart testing tools to figure out how they work.

This isn't for QA. It's for RPA, which automates processes. UIPath is a billion dollar software company doing this...

I feel you in this regard.

I presume low code/no code comes from the simplest notion of programming language, which is a medium between human and machine. And the easier it is to read and write means the "better" it is. Therefore pitching it to a non-technical or even semi-technical people is as simple as teaching people to write code at surface level.

However what people don't understand deeply is that any language is a set of symbols and rules requiring a certain amount of cognitive space and a certain time to absorb it. The more ambiguous the symbols are, the closer the growth of the number of the rules to exponential. This is a fundamental point that happens in real-life but is rarely understood, moreover to programmers, surprisingly.

I believe a proper low-code/no-code should have natural language processing capability and understanding context, not just providing syntactical aliases.

I think it all boils down to execution and the company’s mindset and skillset.

I was also a Quality Engineer for a time, and one of our bigger projects was to implement precisely a Natural Language system or something so that PMs or whoever wanted to “live in the future” could write their tests directly in the User Story.

What we did was create a large set of commonly used “phrases” to interact with systems, be it UI or APIs, which we coded the actual test framework for, document them for anyone wanting to write User Stories (in Rally platform, like JIRA), the instructions on what to do if a phrase didn’t exist for their case, etc. It took us 6 focused months, but we delivered, and people were happy. Of course, we are talking about technical PMs, PMs that met with designers and set standard guidelines, PMs that required API documentation and read it, and the like; just not “coding much” PMs.

Albeit this being a company where we already wrote automation tests right next to the developer’s timeframe for their own coding, where a feature got both code/tests right around the same time, 99% of the time, so it was easier for people to take advantage of features instead of shallowly complaining.

So, yes, I concur with you that probably your case would be better if everyone were a developer, I don’t really see the need for those kind of DSL, but if you reeeeeeally need to have other people intervene, and want them to focus more on their end of the bargain (not coding), I think these DSLs have their place: Assuming the whole company has the stomach for it.

A QA consultancy used this tool at my previous gig. They were extremely unproductive, as in several months to deliver a few tests.

Honestly if you know your Python you are better off with Selenium, which is probably what Robot uses under the hood anyway.

RF is not in any way only web automation tool. Sure, you can use Selenium with it to automate web ui testing. Or you can use playwright. Or you can just not do web ui testing and do mobile ui testing, or REST, or SAP, or pretty much any other automation you wish.

What RF provides is by far the best reporting, test tagging, test filtering, and test data injection compared to any other test automation framework. And you can automate pretty much anything using it.

I agree whole keyword driven thing is quite messy, but the reporting is the part I love about it. Actually being able to reason why things failed and where, if test case are semi-reasonably written.

Depending what RF is being used for, Selenium may be under the hood wrapped in multi-layers of keywords. In other cases, where testing is not web/REST based (protocol testing, HW testing), what is being used for executing logic is up to devs/testers.

Agree, except: friends, don't let friends use Selenium.

use PlayWright!!

or cypess

or puppeteer

Well, I bet that if they managed to be extremely unproductive with Robot Framework, they'd be equally unproductive with Python ;)

-- Robot Framework builds on top of Python and usually wraps python libraries -- such as Selenium or Playwright -- in a different API (which IMHO is usually easier to use) and provides some better reporting out of the box, so, I can't really see how the tool made people unproductive here...

I had a similar experience. After swapping out robot with pytest on a large project I was at least 4x as productive.

The one thing I did like was the separation of concerns between a "readable" specification written in a non turing complete markup language and test code. I find pytest tests difficult to parse.

The problem is that I found the markup language they built difficult not as expressive as I needed and the tooling around the framework made things difficult to debug.

In reaction to robot/gherkin I wrote my own equivalent framework based using strongly typed YAML with an emphasis on fail fast/crystal clear error messages:

https://hitchdev.com/hitchstory/

(in the process of creating it I discovered that YAML really needs type safety and built a library called strictyaml to underpin it).

Used it extensively for a while. I think all in all, it's a nice, mature framework. The Gherkin stuff is completely optional, I'm not fond of it and never used it. Using Python, you can create a custom DSL tailored to your needs, so that you can write tests in a tabular form. All your tests will need to adhere to this rigid tabular structure. Whether this makes sense, entirely depends on the circumstances. Compared to just using pytest, I think the main advantage is that tests can also be written and extended by people who are not familiar with Python. So if you have a separate QA team with people who are not programmers, this might make sense. Also, I did like that this rigid tabular structure made all tests very much alike, whereas when using pytest, you often end up with a hodge-podge of different ways to test things and people re-inventing the wheel all the time.

Now that's a name I haven't heard in a long time...

Burn. It. With. Fire.

I was at Nokia and we used robot, what an absolute pain it was

The hardware people at my job love it.

Maybe we used it incorrectly but for testing SW on target it's the worst thing I have ever experienced. The 'human readable' keywords are a hidden evil no one should meet. And the sheer amount of them, matched with random Setups and Teardowns, wierd syntax workarounds to do simple stuff, performance...

What would you rather use for end-to-end acceptance testing? Pabot runs multiple robot tests in parallel, meaning dozens of tests can run in a few minutes. It seems acceptable if you start with the assumption that acceptance testing part of the pipeline doesn't need to be run on every commit.

Yeah this is my experience as well, I understand the appeal, hell I was even the one to frontrun/poc it 6 years ago. But in the end it's just worse dev experience, no sane Keywords autocomplete/lookup in any IDE out there(by that time).

In the end you end up programming inside english like DSL with weird quirks, that's somewhere connected to python, with different variable syntax, and not really good debugging.

I guess everyone has to pass through that stage at least once, but I will never go back to it.

Well, nowadays Robot Framework Language Server (which I develop) does have autocomplete/semantic analysis/interactive console/debugger (which can debug Python and Robot Framework at the same time) and a bunch of other things, so, I guess things did improve a bit in the meanwhile...

I suspected LSP would be awesome for it, but nothing existed like 3 years ago - kudos to you sir, I think it would make the experience 10x better.

Same here, hello from ex-Nokia Networks GmbH and NectAct migration into Java project. :)

I don't miss Robot at all.

Well, I don't know how things were back then, but Robot Framework had many changes in the latest times, so, it's possible things improved from there (thus your past experiences may not match the current state of affairs).

I'd say that it does have some quirks, but if well used it can be a good tool to have in your toolbox.

Disclaimer: I'm currently working on the Robot Framework Language Server extension for VSCode (by Robocorp, which uses Robot Framework to work with RPA -- albeit Robocorp supports well both Python as well as Robot Framework in the platform, there's a high traction on the Robot Framework side and many users enjoy it vs using just Python).

Back in the early Robot days you would write the tests on a tabular form, HTML based.

The Python code would parse those HTML tables to execute the code, lovely.

Most common was afaik Excel with export to CSV or TSV, though some people started with Robot IDE (circa 2015-2016 was my last contact)

My last contact with Robot was in 2007.

I was also in nokia and we where pitched to use RF - at the time, we (well, i) made a decision of not to use it - mainly because our actual testing tool was implemented in ruby and using remote library wasnt really possible for our scenario and also i didnt like the test data format then (we where pitched to use a frigin EXCEL or html tables) ..

Things have changed a lot from those times though and I am somewhat happy camper with robot in its current state. Its not without its warts but definitely has come a long way since "nokia days".

Also ex-Nokia employee (in PL) here. I see all of us share the same "love" to RF

I remember this from a gig I did a while ago. Customer wanted to establish their own automated QA and thought Robot Framework was a good fit, because the language „looks easy“. The goal was to enable IT personnel that lacked a technical background, with no programming experience, to write tests. I was sceptical from the start, and was eventually proven right (I did not enjoy this, it created more not-so-enjoyable work for me). Test code was flaky to the point of being useless, and contained alot of redundancy. IMHO it is really hard to write good tests without having a bit of a „programmers mindset“.

"IT personnel that lacked a technical background" : Really interested on what the recruitment team was thinking!

Think support staff, PMs, etc.

Exactly.

You need devs to write good tests in general.

I develop tooling around Robot Framework as my main job, personally I feel biggest tradeoff of RF in Python world is that it competes against Pytest, which is my personal favourite testing framework out of any I have tried.

In JavaScript projects I don't have such a favourite, Jest and Mocha have been okayish when I have tried them but didn't really spark joy. In multilanguage projects (like the Playwright based robotframework-browser which I have been developing) I have enjoyed writing integration tests with Robot Framework.

I don't have excessive experience with other comparable QA tools (like cucumber), but I would say Robot Framework's main advantage and disadvantage are both the fact that it is constantly so close to python. (E.g. you can easily do in-line python expressions, but if you were to have a team working with Robot Framework where some are lacking Python competence those might become very confusing pretty fast.) Also for writing advanced libraries Python usage is pretty much mandatory. But I guess it's pretty rare for a DSL to support writing advanced extensions with the same DSL.

I have used RF for many years now at different companies. We use it for automated integration tests, it does not matter in which language your application is written. I do admit that it requires some programming skills, especially for the keywords. Choose the keywords wisely and ensure that the QA people know how to use them. We also use it as part of regression testing and even for acceptance tests with the customer.

From QA perspective Robot Framework is by far the best tool there is.

First of all, it's not web testing tool, it's not mobile testing tool, it's not REST API testing tool. Or any other specific testing tool. You can automate pretty much any testing activity using RF, including web, mobile and REST, but not in any way limited to those.

The killer feature from quality assurance point of view is test tagging combined with really powerful way of selecting what to include in test run and getting good reports which are supported by many many tools. Another killer feature is dead simple test instrumentation.

Regarding tagging. Let's take this example:

    *** Settings ***
    Force Tags    feature-xyz
    Suite Setup    Initialize Tests


    *** Variables ***
    ${test_environment}    dev    


    *** Test Cases ***
    Foobar
        [Tags]    jira-id-001    jira-test-id-001   smoke
        No Operation

    Lorem
        [Tags]    jira-id-002
        No Operation

    Ipsum
        [Tags]    jira-id-003    bug-in-jira-001
        No Operation


    *** Keywords ***
    Initialize Tests
        Connect To Environment    ${test_environment}

I can select any combination of those test to be included in given test run, specify which environment to connect and send results automatically to Jira. This allows me to run only "smoke" tests against "Pull Request" environment when ever PR is opened. This also allows me to automatically run all tests every hour against "Dev" environment and submit results to Jira.

Like this:

    robot --include smoke --variable test_environment:pr

That would only run the one test tagged smoke and Connect To Environment would get the value "pr".

    robot -i feature-xyz .

Would run all tests with tag feature-xyz (and in the example file that would be all tests) against dev environment. And then I could just `curl` the XML result file from the run to Jira (given it has XRAY installed) and Jira would automatically update all the Jira tickets in mentioned in the tags with the test results. If there is no Jira Test tagged in RF test, Jira would automatically create new Jira Test for me.

And in order to display test statistics in Jenkins, just install RF plugin in Jenkins and instruct your job to read the output XML and you get nice statistics, reporting etc.

That way, when you need to know what is you test coverage, just open Jira and see it yourself.

> And in order to display test statistics in Jenkins, just install RF plugin in Jenkins and instruct your job to read the output XML and you get nice statistics, reporting etc.

Sadly, that same thing can be also a negative side.. There is really no proper way to expose the results if you are not using Jenkins as your main CI. Ofcourse one can use junit reports or whip up a own listener that will generate some sort of test reports for the given ci platform...

As RF already outputs jUnit style XML file, for sure you can instruct your CI to utilize that. And at worst `robot` will return relevant exit code so CI can at least get pass/fail information from the whole process. Also, https://pypi.org/project/allure-robotframework/ generates Allure reports from RF, you might be able to utilize that.

Not sure what else one might need. Surely any relevant CI can also store the HTML files RF generates.

EDIT: Just realized who you are. We've met few times at RF events. I'm not sure what other information one might want besides jUnit, console output and exit code, Allure reports, HTML files and RF output.xml.

My "proud" moment - get recognized in HN discussions - thanks for that ;)

Yeah, point was that jenkins integration and how/what reports are shown is really good but utilizing junit does not provide the same experience. Way back in the past, Peke (the lead dev for those who dont know) was even griefing about RF's build in junit support "there is no standard!" - things have changed now but when i came to RF ecosystem, the buildin junit support was *shit* and it was using a format that wasn't really even close to what some other CI systems where expecting (except jenkins) ..

As someone working as QA Automation I have to admit I hate RF. IMHO Pytest is far better and everything you wrote above can be also done there. It will be a bit more code, that's true, but overall I find pytest better tool for automated testing.

Having had worked with both Pytest and Robot, I have to agree. And to expand on that, Robot framework had multiple disadvantages for us. (Note that this is about 2-3 years, so maybe things improved).

It was pretty slow, just rewriting the same test using Pytest sped it up by about 50% on average and that was despite our test environment being the slow part.

The Robot language was quite error prone as it uses spaces inside single keyword as well as to separate parameters. And the IDE support (VSCode in our case) wasn't particularly great. We ended up having to write custom static analysis tools, to catch some of the more common Robot problems.

And pretty much every time something more complicated was required, it was easier to switch to Python to do it. So in the end two languages were required instead of one, making maintenance more painful.

That is not to say that Robot doesn't have its good points. For example retrying on error was much easier than in pure Python. But in the end it was much more work than just using pure Python with Pytest as testing framework.

    *** Test Cases ***
    Write my test using a DSL
        Read examples of the DSL
        Write a basic test and see that it works
        Write a complex test and find something missing
        Learn quirks about DSL
        Implement missing things on the language that the DSL was implemented on
        Write part of the test in said language
        Write part of the test in DSL matching regexes
        Run the test
        And it works 9 out of 10 times
        Install flaky extension
        Add flaky tag to test
        And it is green
        [Teardown] Reflect about my life as a software developer

I test embedded devices with RF. You write most of your test drivers in python, RF simply orchestrates the tests and collects the results into a report, which works well in simple QA scenarios.

Robot lacks features you'd need to support larger-scale embedded test, such as with a device farm, where you need the concept of tests leasing resources. If you build a test stand with 10 testable devices and need to run a suite of 100 tests, it's ideal to run the tests in parallel as devices become available. Some test setups might require more than one instance of a testable device, or instances of more than one type of device (for example, to ensure that a V2 product can still interoperate with a V3 product.) Robot doesn't really support this.

While a feature to support this could be made extremely general (resource classes, instances, and leases) the RF developers have been uninterested in incorporating this aspect of test into their framework. The result is everyone who does even mid scale embedded device testing has to writes their own.

Another complaint about robot framework is that when you have an expensive setup like a 4 minute flashing operation, you don't want to repeat it more than necessary. So in a file, you might make the expensive setup a suite-level setup, followed by the tests cases that depend upon it. When this file grows, you might want to refactor it into a multi-file test suite in it's own sub-directory. However, these tests no longer share a suite scope (because robot's "suite scope" is actually a file scope" for legacy reasons) so in practice you may need to tolerate 3000 line files to avoid long setups.

> the RF developers have been uninterested in incorporating this aspect of test

Have they been asked? I had the same problem and had different solutions built in house with different levels of success

Pekka has previously expressed that test resource management should not be included in the core and is better implemented as separate keywords. One example would be the limited resource management that Pabot supports (basically a single global semaphore that tests can increment/decrement through a pair of keywords.)

The problem with 3-rd party keyword-based solutions is that the resource acquisition becomes a part of the test process and could a test report to look like "ran for 6 minutes, then failed" when what really happened is the actual test couldn't be run at all because resources weren't found. It would be nice to be able to queue tests based on the resources they'll need.

Also, another feature that's more difficult with custom implementations is static analysis of the test base. If resource requests were a declarative part of the RF language (much like tags) you could do things like "run all tests that exercise resource X" or "run all tests except those which require more than one Y". This is technically possible with a keyword library but requires an additional global lexing pass of the entire test base, and this is the kind of logic you don't want to decouple from RF and it's test discovery behavior.

Perhaps Pekka can be won over in the future.

Have you found a solution to mid-large scale embedded test setup? Could you provide some shallow insight into frameworks or other infrastructure used for embedded testing? I've previously been responsible for firmware testing small production volume devices in aerospace but have since moved to a high volume product with multiple active hardware revisions and no test infrastructure currently in place. It's a different beast to test now while trying to balance schedule and feature dev.

If you're trying to validate new software on known good hardware, then your goal at any scale is to insert a suite of embedded software acceptance tests into your development pipeline usually somewhere between build and release. CI infra is ideal for this, my counterparts all use Jenkins.

Robotframework is a great tool for expressing tests. For simple setups, Jenkins has an RF plugin to kick off tests.

The tricky part is managing many test resources. One jenkins server triggering robot tests on one bench PC with one embedded target is a great way to develop an initial test suite. But once your org gives assigns you more hardware, the tricky part is figuring out how to use it maximally (either optimizing for fastest time to completion or for least time idle of your dedicated test hardware.) This means the tests themselves can no longer "drive" the test resources directly, they need to lease them from a resource manager service.

Larger companies with nearly homogeneous hardware may get by with a basic priority queue. But some test setups with an L1 or L2 switch between them may support a DSL to allow tests to request resources that are physically or logically connected, and may need abstraction layers for common tasks such as "physically disconnect power" which the Resource Manager could implement as a SNMP command to a PDU or as a SCPI command to a DC supply -- the test itself should be declaring what has to happen, not how.

I've scoured the internet and plied counterparts in other industries for any example of commercially available prior art. Everyone I've talked to wrote their own. It took a week to produce something acceptable for my use case. But if the RF devs were to maintain a similar solution, I'd switch to that in a heartbeat.

For other roboticists who might stumble upon this: Robot Framework seems to be an automated testing framework (for testing software)

When I started writing tests for my current project, I looked at what was available and decided forgo existing testing frameworks. I haven't been disappointed.

Writing code to call a bunch of test functions then generate a report is really not hard and having control over the whole thing is nice.

Robot Framework is a general purpose automation framework.

You can use it to automate - Web Applications (with Selenium or Playwright) - Rest APIs - Desktop Applications (Java Swing, WPF, SAP, ..) - All kinds of hardware

It offers control-structures like most programming languages (IF/ELSE/FOR/WHILE/TRY/EXCEPT)

The Extensions for VS Code or PyCharm/IntelliJ offer Code-Completion, step-wise Debugging, Linting.

It is very hackable and can be extended using Python. It has a great API, allowing you to connect it to all kind of other tools.

I guess a lot has happened since some of the people here used it.

But I guess, if somebody is focused on using a single programming Language (like JavaScript) and is not open to learn Python or the RF Syntax I recommend to look into another native test framework

Stop calling things that are not robots, robots.

Does this framework stream sensor data? No. Does this framework control actuators? No. Does this framework represent an actuated device’s configuration space? No. Does this framework perform collision detection? No. Does this framework keep an obstacle map? No. Does this framework have any planners? No.

Just stop already. Stop calling things a robot when they are not.

They're referring to "Robotic Process Automation" (RPA).

I agree the name is a misnomer, but it's now part of the Enterprise Lexicon, which is filled with jargon.

I run an RPA startup, but as much as I hate the term (e.g:https://news.ycombinator.com/item?id=30755118) , once something becomes part of common language patterns, you can't unshift it.

I hate the modern usage of 'AI' as ML, but I can't change the new definition.

Pardon me on advance for continuing to yell at clouds.

I was really excited for a good new ROS like system or something, but instead I got automated testing T.T

Fun to see this is still around! Used this in 2016, at a big bank. I did not enjoy it, but that can largely be attributed to the way management wanted us to use it. The tests had to be written in our native language (not English) which caused a lot of weird mixed language use behind the scenes. Then a manager would sign off on the test report, which we would deliver to him on paper.

The latest 6.0 release finaly adds ability to translate most buildin syntax and afaik, keywords.

I've been using RF for almost 8 years now. I came in touch with it, in the beginning with my C/C++ developer mind: "...what an overhead of work...".

But soon a realize I was wrong and start to notice the power of writing tests in robot. And how it allowed to easy the understand of what operation I did on code in the past... like 6 months before, 1 year before... and also the ability to understand more clearly how other teams' features worked and to fix their test and features many times, because we more clearly wrote test using Gherkins in RF... and see how new team members could ramp MUCH more easily inside our environments, by reading tests, most of time.

I was even able to implant RF inside some other business after that, bringing together that culture of writing clear tests with RF. Some of those places had the heavy culture of writing top-down requirements precisely. On those places, we integrate RF documentation and test case procedure on our process, generating final documentation using RF test case information.

Since them I'm helping to disseminate and evangelizing for the RF use. I saw system analysts embracing the RF, writing the first version of test cases for requirement in matter of minutes. RF provided us with much faster peace in maintaining some very complex environments.

We could grab requirements, and write down cases, that latter could be implemented, edited, removed or adjusted. Whatever is the test environment, testing it is a 'living' environment, and need catering from developers. RF had helped greatly maintaining such places, to produce great reports, to create this live ecosystem easy for uses.

Is perfect? Surely No.

But is much better that all other tools available around. Is open source, easy to expand, allow use of natural language. And when well used, allow us to greatly improve our QA and DEV lives :D

I loved this tool :) ! Few years ago, as a junior I joined one global corporation to my first job and I started to use it with zero knowledge about the python or even testing at all. But learning curve was very easy & fast and as of time, we created more than 3000 E2E & integration tests for our business application, running through the Jenkins. The tests were written in a very human-readable format, so anybody at the team could easily understand the test and identify the issues. Reporting had nice (but quite slow) html & jquery page. Also, Robot editor was a recommended tool to write tests.

After a time, when I was starting a startup with a few junior developers which lacked the expertise, we used this tool in combination with Selenium to crawl data. It was fast & easy.

Always glad to see this tool is still alive. Good job Pekka Klarck!

Robot Framework – test automation in Python - https://news.ycombinator.com/item?id=10631074 - Nov 2015 (5 comments)

I wrote my bachelor thesis about Robot Framework. The biggest issue i had with Robot Framework was the lack of support in IDEs. Initially you were suppose to work on Robot Framework in RIDE (dedicated IDE build on top of Eclipse) or use a Eclipse extension. I believe both of these are not supported anymore. I like Vim, so i wrote a package, which helps users when developing with Robot Framework. I also created some pull requests for Vim ALE to also support rflint.

We also used Robot Framework at work for automated testing, worked like charm.

There's a quite a popular extension called "Robot Code" for vscode:

> An extension which brings support for RobotFramework to Visual Studio Code, including features like code completion, debugging, test explorer, refactoring and more!

Have you tried it?

RIDE is a totally independent RF editor, not written on top of any other IDE. RED is RF editor built on Eclipse. Don't use either of them. https://marketplace.visualstudio.com/items?itemName=robocorp... and https://plugins.jetbrains.com/plugin/16086-robot-framework-l... are the way to go. Personally I use SublimeText with the RF LSP server with this https://github.com/Harri/RobotFrameworkSyntax but that's not "batteries included" like the two other plugins.

@fabioz who also posted to this discussion has been building "language server protocol" support for robot framework for a while now and is being sponsored to do so by a company called Robocorp.

RIDE is not at least officially "not maintained" but my POV is that the current maintainer might not be up to the task -- or at least doesn't have time to dedicate to its maintenance.

Robot has it's peak maybe a couple of years ago. Although my company still uses it extensively there are other Automation frameworks that are proving to be better, like , say Cucumber, Cypress, etc.

But why is Robot Framework being mentioned and getting attention now? It's not like it's something new and doing anything different from the existing frameworks.

Cucumber does not really provide anything RF does not, except for Ruby houses. And Cypress is web automation tool, RF being generic one. One could write Cypress library for RF, but as there already are Selenium and Playwright libraries for RF, I don't think it will happen anytime soon at least.

We use Robot at my current job and it works flawlessly. It can test the web front-end (Browser, wrapper for Playwright), the REST APIs (REST, RestInstance), CLI stuff and much more.

We selected it specifically for its versatility and the DSL is easy enough so even less technically adept testers can pick it up quickly.

The developer experience is fairly nice when it comes to Robot Framework. Nice thing is, that Robot Framework was originally designed as automation tool, so also people who do not know a lot about coding can get up to speed fairly easily.

If you use Robot Framework and have a lot of tests, which can be executed in parallel. Take a look at pabot [1]. It is a parallel executor for Robot Framework.

[1] = https://pabot.org

Blast from the past, I was involved in development of IDE for RF while in Nokia.

My only experience with Robot was at Juniper Networks, where it was a poorly maintained bastard QA org tech. Would have been infinitely better if DEV’s and QA had a shared repo of python scripts.

As an interesting note that it is not surprise that half of showcased business logos are from Finland.

Robot Framework is (was?) heavily used by Nokia and then spread from there to its subcontractor network.

Still is.

If i remember right, RF was Pekka's thesis work inside Nokia and it was later open-sourced and a foundation was formed around it.

No.

Write some code, or get playwright to generate it for you. Don't use DSLs if you value your sanity.

Why is it "generic" and what does that mean? I could not find it in the docs

Reply on news.ycombinator.com