My biggest mistake as an RPA developer
anupam.de
anupam.de
Here's my background in the subject: I have met with team of a dozen people that spent half a year doing something that turned out to be the functional equivalent on an SQL one-liner.
They could not within our meeting slot accurately describe what it is they were doing, apart from "automating tedious work" (yes, but what is that work, specifically?) and kept insisting I was simplyfing their work (which was very likely true, but not from a lack of trying). Not a single line of this code was in git, there were no tests written, and there was no migration path concerning changes in the external data sets.
I have seen a lot of visual programming and business process modelling tools, and written my fair share of LabVIEW back in the days, but with these tools everything old seemed to be new again and evey lesson had to be re-learned. I later learned that certain executives had attended a big Gartner event with RPA in the magic quadrant and promptly set up this team upon returning. Of course, today "AI" has replaced RPA, but that team is still ticking away.
Is there something to these tools, or is it just the latest iteration of preying on C-level executives with budgets to fill?
FWIW: In my world, we have native tools and methods as part of our ERP application, that would accomplish certain automation tasks more efficiently and effectively than RPA. But that would constitute a "change of application" (because it would be); change management and sign-off on these is so onerous; and the perceived risk and potential impact so high; that they never actually materialize.
RPA sales pitch to executives is "This is not changing your application; this is simply doing what your people are already doing in your application, faster and more repeatable with less errors".
This is a powerful risk-reduction pitch and not an untrue one in the real world with processes and risk aversion.
In the end, the RPA effort... had its hiccups and moments, but it DID automate things in a project/environment/client/process that otherwise would wait until heat death of universe to be automated. So it worked great, with some asterisks and for given definition of worked and great :).
----
P.S. That being said, as far as article goes, I feel it's broadly correct:
1. RPA fails in production a lot - which is counter-productive to its primary goal of reducing risk / increasing consistency.
In our application, there's ultimately a reason why something was a human UI and not a batch job - there's edge cases and circumstances and sometimes decision and value judgment, unusual flows and special conditions, however common or rare. And because you're SO externally tied to a GUI, anything breaks it. Anything. You get into this weird position where updating application to fix something has inherent risk of breaking something else (RPA bot)
2. RPA developers I met were intelligent, energetic, excited, positive, hard working, willing. But yes, also not necessarily "software engineer minded". Experience I think is a significant factor - discipline is new, developers are young, and things you gain with experience such as "how do I eek out requirements and edge cases out of user" weren't there just yet; they will be, but I'll agree with a bit of "relearning things we already knew".
That being said, note that there definitely WAS a text-based script editor in tool that we used as well; not sure how primary it was.
The very second one of these runs has some non-intended consequences, and from the nature of the tools that's likely to happen sooner rather than later, change management will latch on to them and never let go.
The tools in question were very similar to Selenium. Any non trivial job is going to have logic in it. Definitively not any less programming involved than writing tests. They were clearly intended for a Microsoft focused audience however, so maybe that's part of it.
> automate things in a project/environment/client/process that otherwise would wait until heat death of universe to be automated
Which is to say "things no one else cares about or thinks are important enohgh to spend time automating."
RPA isn't for major corporate priorities.
RPA is for that thing that takes up 25% of a 3-person team's week. Or 15% of every contact center agent's time. Neither of which are ever going to be prioritized.
Or, to put it another way, RPA is the answer to "That has value, but it isn't important enough to spend software developers' time on."
Which is why "but these people aren't software developers" or "but they didn't use git" or etc simplify to "If we had more software developers, this wouldn't be needed."
Yes.
But that's not true. Nor will likely ever be true. So it is needed.
PS: And fundamentally, it's taking corporate computing back from the "ask IT for anything" to "do it yourself" hacker ethos. Computers exist to do work for you. When did we forget that?
surely much of this could be done already with a tool like AutoHotKey? I mean it won't have some of the fancier stuff like screen scraping, but it gets you 80% of the way there in a lot of cases. the barrier is not the tools, it's the IT policies that prevent ordinary workers from being allowed to use them.
I think RPA exists because every organization is plagued by people management problems that developers and IT often fail to acknowledge.
1. I've updated my post to make it more clear how change management is involved: Using native tools we would be, literally, "changing the application". RPA is not changing application; it is automating what people already do in application. It can get philosophical whether this is a real difference or semantic one; but in corporate world, it's real :). And ultimately in techie world too: ERP is not touched or customized by the RPA (which would have had meaningful and permanent repercussions for troubleshooting, vendor support, upgrades, retrofitting,etc). On Ops side too, we use same process and tools to troubleshoot an error in our application whether a human or RPA bot did it.
RPA bots still go through change management as such; but their risk, cost, implementation time and other profiles are simply vastly different than actually touching the COTS ERP application code. I would not necessarily use RPA to change most of the applications internally made by a company; but they're a valid choice for automating a externally sourced software.
2. I have limited experience with Selenium but I would agree that there are broad similarities. They even remind me of old Mercury/HP LoadRunner or SQA Robot. Just, couple of decades of advancement and different focus.
3. I feel "Clearly intended for a Microsoft focused audience" is some kind of insult, and somewhat uncalled for; but note our application runs on AIX on p-Series, FWIW.
> This is a powerful risk-reduction pitch
Is it? It seems to me a powerful risk creation pitch, where instead of dealing with the process problems that make properly controlled change expensive, you just exploit a loophole in your control requirement that are designed to mitigate risk and implement change with no control.
OTOH, since the pitch is generally to the people responsible for the state of the policies, I guess it is “risk reduction” in that it reduces the short-term risk of admitting the larger error since it enables kicking it down the road a bit, and that is something lots of executives like.
The more you customize a COTS application, the harder your support gets and the more pain you get when you patch or upgrade.
Building on top of it is practically as well as procedurally safer. It is, legitimately, reduced risk and impact. You keep the solid stable core that is fully vendor supported, and you have the piece that you built and own neatly separated.
In that way you can almost think of rpa as a shell script: You don't customize the kernel every time you need something done. You leave kernel well alone (and reduce both risk, and pain of future os upgrades) and build on an easier language something you own on top of it that interacts with it.
Note I don't do rpa and my team doesn't do rpa, but I've seen it around and seen it's value. It does feel I'm talking To folks who want to reduce this to "management sucks" as opposed to any effort to try to actually understand a technology's Place in ecosystem though...
And yes - the exhortation in my article was essentially for RPA developers to max out on the use of the text-based script editor.
The short answer is RPA = screenscraping + keystroke recording. Sometimes you need to programmatically interact with old legacy applications that have no APIs or other interfaces apart from the UI of the application, and this is where RPA comes into play.
would be tedious to do it manually. That said everyting is in git...
If you really do have citizen devs in your org, it's great. I think people are more willing and able to learn Power Automate for Desktop than PowerShell/Python/what have you, then add in the governance, inventory and monitoring/analytics options in an RPA tool like Power Automate, and it can be easier to manage.
Every 10 years or so you will find the industry trying to sell GUI programming tools. Managers will love it. It will sell a lot. Then people will start to see the brittle applications, that it is impossible for 2 people to develop at the same time, that you can't see what changed from 3 versions ago etc.
Then you will have to point your manager to the the 1986 article "No Silver Bullet": https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&c...
RPA replicates the work that these workers do by automating on the UI of the legacy application and replacing them. It is championed by ambitious business professionals who
1) Don't know that a more elegant solution exists in the traditional IT world
2) Don't receive enough support from the IT org to solve this business problem that is crucial to them.
Therefore, such business professionals collaborate with consultants to spear-head RPA projects in the name of digitization, ushering in AI or similarly varnished back-doors.
However, as long as important data tends to accumulate in disparate islands that IT doesn't have the time to pay heed to, RPA projects will continue to be sold at a price that is a little lower than the salaries of the employees they 'liberate'.
There is, although, another wrinkle to this. RPA is also about democritizing IT by handing (ideally) every white-collar worker the ability to write scripts that can automate pieces of their daily-work. Of course, those solutions aren't likely to be elegant, and their code won't be clean, but since their scope is tiny, that isn't a problem.
>1) Don't know that a more elegant solution exists in the traditional IT world
>2) Don't receive enough support from the IT org to solve this business problem that is crucial to them.
>Therefore, such business professionals collaborate with consultants to spear-head RPA projects in the name of digitization, ushering in AI or similarly varnished back-doors.
Sounds like Conway's Law taken to an absurd degree and turned into a multibillion dollar industry.
) Lack of political will/capital to uproot silly/outdated processes.
2) Lack of skills amongst business users to understand development
Many processes in companies are idiotic, outdated and sometimes even unnecessary. Usually these processes are invisible to the decision makers. So whilst the entire HR department keeps sending emails with Excel files to each other all day long, nobody higher up stops and says: well, you know it's bulshit, let's drop Excel and get a proper tool where we could keep records together.
Sometimes the process can't be changed: in my previous job, I was sending Excel files dividing the invoice into 12 different business units in 12 different countries. Guess what, the client was one of top 5 US banks,so we,as a mid size company had zero chances to change it.
Most business users don't understand programming, nor its capabilities or opportunities. For them anything written with code is a black box and ultimately make them feel inferior. So when you go to a business user and say: hey look, YOU can be in charge,YOU can automate. Suddenly they are elevated into the whole new level.
3) The existence of business problems that are valuable in their own right, but not compelling enough for regular developers in the firm to care about
In corporate environments it's super common to experience functional success in parallel to social failure, especially like your anecdote. Most corps/management love terms like "intrapreneur" but have zero actual interest in formalizing + celebrating internal innovations/solutions.
That process is done in 20 separate locations by 20 separate people. With RPA we can automate that entire process (We don’t control the web app and have to wait for features from the vendor).
(The process is more complicated than this, but you get the idea).
Enter a third party vendor I'd worked with before. They specialize in archiving data from EMRs into a standalone searchable database. $50k and two months, they're done. The EMR has an IE only UI, and a newer cross platform UI. We demonstrated how to extract the data required for the new owner, and they created an RPA workflow to "print" to PDF 5k medical records. It's the exact same output the EMR vendor would have given us, but they'd have used their own internal tools to do it, but they'd have taken 3-6 times longer to do it.
RPA is just screen scraping, basically, and sometimes it's still useful.
So a tentative plan forms. Team B meets team A and asks for the API. Team B seeks funding to code on top of the planned API. But it falls through, team A doesn't have time/budget for the project, isn't interested in the idea, or otherwise rejects the idea. This is especially true if team A has other larger, more important users.
Team A regroups and tries RPA to make their own programmatic interface on team B's app without their help. The RPA promise that non-programmers can automate their own jobs is also compelling, and is seen as a cost savings.
What I've seen happen next is that nothing really happens. End users are capable of writing RPA, but don't want to. Any automation is flakey, and may not actually reduce costs. Most folks continue getting paid to shuffle data, who'd want to automate away their own paycheck?
Also, since team B didn't want to build an API and isn't part of the effort, there's a bunch of ugly issues: no programmatic contract/interface, no docs, no versioning, no backwards compatibility, no forward notice of changes, no access to test environments. So the automation can unexpected break or begin corrupting data, it needs regular monitoring.
As a developer, that sounds terrible. But if there's no other way to automate, it's attractive.
Because business processes are poorly documented. And if they are more or less documented they are executed by teams that need to override the underlying technology when a "corner case" happens. As someone else has said, shit needs to get done.
You cannot successfully deploy an RPA project without understanding the business process behind it. And that usually means shadowing the people doing the "clerical" work
An interesting perspective comes from the process mining vendors (which is to say Celonis these days), which goes the other way: look at the application logs to extract event data in order to create a model of the business process. And then they move downstream: perform conformance checking (how is the "as-is" model different from the normative BPMN process) and even perform task mining (what are the users doing for each step of the process)
It's terrifying how much critical functionality at an average company exists only in 1-3 brains.
It's not actually terrifying. It's the way that businesses have worked for centuries. There's always a "high priest[ess]" in the woodpile, somewhere, that has "The Keys to the Kingdom."
Most businesses have some form of continuity or recovery plan, but many would collapse, without the Key Player, and they do.
The flip side, is that with the Key Player, they can do very well, indeed.
It's just that Silicon Valley has developed a dread of "The Bus Factor," so we do things like deliberately design shoddy project plans, so that inexperienced, disloyal, transient, teams can run them.
I have looked at codebases that were designed to be implemented by a team, when it should have been a simple, 1-person job. It made the code brittle, overcomplicated, underperformant, and buggy as a rotten log.
So consequently, the Key Player turns out to be sprinkled throughout the org.
And, the really scary part, awareness of "who that person is" generally doesn't seem to penetrate >2 management layers up.
They had a seriously robust structure and policy. It worked (check what Japan has been through, in the last 100 years).
But it brings massive overhead and rigidity. Most American companies would not want to work that way.
The wait time had to be set appropriately. The tool had no way of detecting if the browser is fully opened and responsive. So if the wait time was too small, it would try to enter the url to a non responsive browser and fail. To make matters worse, each computer will take a different amount of time to load the browser fully. So I had to resort to set a large enough wait time and hope it works. In the end it just seemed ironic that an automation tool could end up making the job take longer than manually doing it. I dont know if the current version of Automation Anywhere still has this issue.
I understand the niche, but I don't particularly care for it. It's an enabler for shortsighted, 'keep the lights on but cut costs and corners' operating strategies that usually go hand-in-hand with technical debt accumulation and brain drain.
Not using RPA is like not using macros in your text editor, because they are automating programmers out of work.
Some data can only be accessed through a legacy gui and it needs to be cross referenced with several proprietary databases that also can only be accessed through a gui.
Even if you begin migrating these it can take years if not decades. Meanwhile shit needs to be done.
But while the 3rd 100million project to rewrite that legacy, proprietary cobolt data app is faltering, the RPA department is actually making it accessible via a rest api that activates a virtual mouse that clicks around and ctr+c,ctr-v the result back to the user.
The curse of building great digital infrastructure in the 80s is you might get stuck with it 40 years later.
Most software products that companies buy have interoperability as either the least prioritized feature or as something to be actively prohibited.
The best, brightest, most modern examples of software, we are not talking about.
Yep, that's the key thing. The ERP systems which have their tentacles all over every part of the organization make changing stuff out effectively intractable.
I don't know if it's intentionally like this (by design on the part of the vendors) or if it's just a consequence of people being terrified of change.
I worked with some folks who were sold on it for streamlining/tying processes from Salesforce that were on a mainframe and an awful peoplesoft thing.
It was ok… but it just felt wrong. Nobody learned anything… they literally recorded a half dozen people doing the tasks and looked for variance.
Spinning the blame around a bit, I've often thought that part of the problem is we keep building GUI toolkits that hate this usecase. If they supported this better, the RPA stuff would be less awful.
Appletalk almost did it, but my impression is that it is dead now.
Websites have decent support for it, but trying to use it in practice still involves a lot of little compromises everywhere. It still breaks pretty easily, and the workflow is bad even when you learn all the tricks.
But there's a lot of useful code buried in GUIs. If there was a way to write scripts against the UI, in a discoverable, testable manner, this would be a much more sensible strategy. And you wouldn't necessarily have to hire developers just to maintain a "real" solution.
But we persist in treating this as a fifth-class citizen in our widget sets, so... here we are.
I mean, yeah, it would always be a bit of a tech debt pit, but it doesn't have to be anywhere near as bad as it is now.
It really seems to be an enterprise (and large-scale professional services) focused tool. It was difficult to get support as a small consulting agency working for a small client.
For the most part, the developer community is very insular and lacks a lot of technical skill. Its primarily Indian and latin american developers using RPA to get into the field. It was frustrating at times needing help on a platform-specific issue and being directed by the same social engagement-focused "experts" to watch their own "tutorial" videos when it didn't answer the question I was posing, and they had nothing to offer besides their video. When I did help answer people's questions in the forums, I was hounded with unrelated questions in my direct messages because I appeared competent. Its very much a "here's my problem please fix" kind of community, where the goal is to solve their immediate issue instead of learn how to do it themselves.
I'm a little jaded to the field of RPA after my experiences. UiPath has some novel solutions to targeting GUI elements using a selector language and introspecting the structure of the GUI itself, but the lock-in and licensing costs really make it inaccessible to most small-medium sized businesses, the performance is really poor, and the development effort of making a resilient solution is pretty high.
I think if I were to get back into it, I would look at workflow languages without a visual workflow designer instead, and avoid GUI interaction in the process at all costs.
Unfortunately, A lot of RPA companies, including UiPath, have co-opted the word RPA into enterprise marketing, alongside words like 'AI', so the words lose meaning. But RPA is just automating the UI, when no API exists.
Disclosure: I'm a co-founder in an RPA startup. I think RPA is a confusing, historical accident of a name.
A lot of unnecessary hype indeed, though you can actually throw in occasional OCR/NER blocks in the workflow you design. Most often it's common NLP tasks on documents being moved around, but also forecasting, etc.
PS: I checked out Axiom yesterday and am excited to see the problem you're solving. Wish you guys all the best!
We think UI automation has a lot of potential as a programming paradigm accessible to non-coders. Once RPA hype and noise dies down, perhaps a few signals like that will remain.
I've always been averse to "low code, no code" solutions. Most of the problems that these seem to solve are mostly already addressed by programming tools. Even version control has GUI nowadays.
As for low-code and no-code, I understand your aversion. Those solutions aren't for somebody like you :)
That doesn't come in Cracker Jack boxes.
Even if we get to the point where all our code is written by AIs, the requirements will still need to be specified with engineering discipline (so they will look a lot like a coding language).
The ideal concept that CEOs have, of "just make it like I dreamed it" will never happen.
It's a nice thought, but DOA.
It'll even make you better at your main skill. There is some progress you simply can not make if you never leave a singular frame of reference.
Anyway circa 2016-2017 at this $BIGCORP, somebody got excited about RPA, created a lot of buzz with the executive team, and secured funding to build an entire RPA department.
Everyone was was excited! People were speaking at conferences after 6 months of scripting, grand claims of thousands of hours of work being eliminated were made, and so on. 2 years later, the entire department was gone. I never met anyone there who actually had work eliminated by the so-called RPA scripts...so it just felt like a big joke to me.
I imagine there are people who have had good experiences, but I've not really heard of them. I'd be interested to hear of legitimate benefits in this space.
The author isn't very clear here and seems to be themselves unclear on how these RPA technologies actually "see" an application.
Every Robotic Operating Model I've ever seen or worked on has always held firm that "surface automation" (think Citrix Receivers and applications built in Silverlight or Flash) should be outside the scope of any RPA solution.
What's left are browser or desktop based "physical" applications that actually have an underlying model used to describe and render a GUI. This is actually what is being utilized by (most) RPA clients.
While correct that the clients do interact with the UI of a program, depending on the application they actually interpret (or "see" it) via the application COM (component object model), or in the case of a webpage, the DOM (document object model). Since these are essentially used to describe what is rendered on screen -usually in a more detailed way then what is actually rendered on screen- the RPA client is able to efficiently and confidently interact with the application.
Take a webpage with a red button to click for example:
Automating purely via UI/surface automation: - Capture 150px by 150px at screen coordinates x and y - Is captured image red with the words "Click me" on it? - Go to x coordinates on screen - Go to y coordinates on screen - Send key 'Right mouse button click" - Pray
Automating via DOM: - Attach to process `Chrome titled "My webpage"` - Is element `<button enabled=true id="superUnique_superDependable" class="clickMe" onClick="navigate(MyOtherWebpage)">Click me</button>` present? - (to browser client) Send key `Right mouse button click` to element id = "superUnique_superDependable" - Wait for process `Chrome titled "My other webpage"`
OR
Automating via DOM: - Attach to process `Chrome titled "My webpage"` - Run function `navigate(MyOtherWebpage)` - Wait for process `Chrome titled "My other webpage"`
However, that app's UI would keep changing due to updates from the IT department, over which the customer service department had no control over. This is why I call the automation 'fragile'.
But if you are automating using a bot (a software program), why does it not have access to the same pool of data as the UI application (some database somewhere)? Would that not be a more robust means to retrieve it rather than via the UI?
If you are automating a business process via the UI (other than for simulating user interactions for testing) there invariably exists a more efficient way to achieve that end without involving the UI. I have found no exception to this rule.
But is that really RPA at that point ? Suppose your task is to scratch some data from a website, calculate something and if some condition is true also perform some form submit.
If you do have access to the actual database you can just use that. If you also have access to the API of the system, you have to "check some box", you might as well just write a normal application that calls some SQL, perform some http-posts and maybe also provides a nice website with that data it crawled, formatted in the way you wanted to calculate the data, in, let's say excel. (Some would say you wrote a microservice.) And if all of this is possible then you're apparently good enough in programming that such a solution is the way to go.
But if you're in the typical corporate environment where the API doesn't exist / you can't access the database / you're not allowed to interact with the checkbox except for the proprietary application, then you're back to what an RPA was supposed to do.
Because otherwise we're down to one question: When is it an RPA ? When you're just using a fancy UI ? When you're not accessing any API ? When you don't script ? Or is the example above with SQL + API also an RPA ?
RPA to me rings "mouse recorder for dummies" or "ugly hack that breaks depending on how good your visual detection system is". At least when you described discovering that you can actually just use real scripts instead of UI indirection, it reads like something like that. A "not invented here syndrome" of that RPA tool, instead of a helper that handles checkbox-detection + clicking for you.
Also that auto-hotkey script for automatic driving in GT is really impressive. ( https://github.com/ByPrinciple/GT7-Scripts/blob/main/PanAmer... ) So that's something between scripting and RPA. Essentially a DSL for exactly this task, which totally destroys my point. And does make a case for learning how to script with a tools that's capable enough, that it is useful for more than the best case.
https://github.com/rohanbaraskar/UiPath-11/tree/master/Autom...
There's not much RPA code on GitHub, which seems like a red flag to me. I suspect there's more hype than actual work happening.