Ask HN: How to Desktop Automate for Locked Down PC-HDMI Capture and Extra PC? | Hacker News Reader

Ask HN: How to Desktop Automate for Locked Down PC-HDMI Capture and Extra PC?

19 pointsairbreather15 comments

My wife has ridiculous amount of manual tasks to do across a bunch of non integrated software every month.

Some of the software is out of date, there is 4 different apps/databases plus excel she needs to click through on to process a single payment, plus checking. She now has the tasked swollen to the point she has to do at least 300-400 every end of month and it is now taking many hours of her time. Mostly unpaid overtime...

So it is entirely able to be automated, except the PC is the most heavily locked down thing I have ever seen, you can't run portable apps, install browser extensions etc etc. Plus everything you try that is verboten sends a report back to HQ and after not many reports questions are asked.

Business is not interested in solving such problems, their answer is "just spend an extra 20-30 hours a month riding the mouse". Well you can guess what I think of that.

So I think to automate this task with an external PC, HDMI capture card and maybe an Arduino or similar to generate mouse and keyboard commands, controlled by USB (serial) commands from the external automation PC.

Obviously with this method it is all shortcut keys and image matching for mouse co-ords to press, so Sikulux or some custom Python maybe.

Has anyone done this? I searched for quite a while and could not come up with any specific examples that anyone else had done anything similar.

I can't see why it would be a big problem ,other than setup time and some potential fragility, but payoffs would be big. I'm surprised not more people have not had similar problems, am I missing something?

I have similar problems, and the security lockdown and lack of support from the IT department. Here are some various hacks I've found. In roughly increasing order of difficulty:

1) Clipboard history. Windows key + v to turn it on. Part of windows, no installation required. Then winkey + v shows your recent clipboard items and you can use them over again. You can pin frequently used commands.

1.5) Windows key + R. Open up a simple gui to run 1 line commands. Has a history you can access with the up and down arrows. You can do amazing things in 1 line of DOS.

2) Dos cmd terminal. Part of windows, no installation required. You can make simple scripts called .bat files

3) Powershell. Part of windows, no installation required. Difficult to learn, frozen in place at version 5 I think.

4) Power Automate. Part of Office Online, no installation required (if they use Office 365) Difficult to learn. The whole Power apps community has a lot of potential, depends on your Windows license.

5) Power Automate Desktop. Free from MS for most MS environments. Can do a lot with this. If I had to pick one this would be it.

6) Power FX. An Excel scripting language from MS, but only available in the Excel web version of Enterprise O365. I guess it's a version of typescript, and feels a bit like Power Query. Worth taking a look at so you know it's there.

6.5) Power Query. Part of Excel Desktop. Not for querying, despite the name, but for getting rid of all the crap in excel files. Not faster than doing it yourself, unless you have to clean up the same excel file over and over, then you can save the query and automate that part.

7) Windows365. Like an azure ms desktop, but a fixed price virtual windows 10. Acccess it through your browser, but can share files and clipboard. About USD 40 a month. But, it's just like win10 home, you can install whatever software you want.

8) AutoHotkey Needs your sysadmin admin to allow a 1 time install. Steep learning curve but it's everything your wife wants. Very deep user base and community.

9) Any of the RPA enverionments like UI-Path and so on. Very expensive and needs big corporate buy in. Never tried any so don't know. MS liked 5) PAD so much they bought it and made it free (their SOP to run competitors out of the business) so that tells you something.

Honorable mention: Programable keyboards (send a string when a key is pressed). Never tried one but would like to.

Looks like she maybe get python installed by IT, as she is a business analyst, in which case I am thinking pywingui or similar.

All the other suggestions are Ok,in their own context, but aren't really going to cut it one way or another.

There is no access to command prompt or powershell, the lockdown is like nothing I have ever seen, often if an app is going to be installed or system change made they courier the laptop back to IT and then back again. It's bizarre.

Great list! I add Excel VBA.

I can see where you are coming from but there is a simpler approach. You wife should be paid for every hour of overtime. If she is not, then, she should only work her paid hours.

When something breaks (and it will) as long as she has made the problem known with management in a traceable and deliberate way. It will be fixed only then (sadly this is how humanity is).

Cheers

Edit: speeling

She is the national manager for a services group with a contract to a charitable group that performs lots of essential and good works in the community through the whole country.

She is the only one with permissions across all the apps that allow the monthly accounts to be settled, the way the systems have worked out.

Upper management is in chaos right now because the ultimate group owners have worked out a bunch of them are dicks and the broom is going thru at the higher levels.

She has asked for a minion, who looks like they will arrive, the three year contract does not support this extra person, but the business is prepared to fund them.

It started out as 4-5 hours a month, then there were system issues etc etc etc.

I have worked contract all my working life and very militant on no pay no work, but this is just one of those situations.

Bright point is she can work any where this an internet connection and to provide some counterpoise to the situation she is currently doing a WFH holiday on a tropical island for a few weeks. But if I can automate this one onerous task she will be back square or better.

Doesn’t a national manager outrank IT? You’re looking for workarounds while you should just force the issue with whoever is in charge and end up with more control over the pc.

You would think so, but no.

Does she have installed locally? Of so she's got VBA, which is very powerful. Alternatively if she has the ability too see file extensions, then create a new text file and rename the extension to .vbs, edit that in notepad, and you can now write vbs scripts that can basically do anything. Google 'VBS sendkeys' for more info.

If you don't have file extensions turned on, you can use VBA inside office to create the vbs file for you, and then open it via notepad. A variant on this is .hta files, again Google these, most people don't know they exist so don't block them.

You can also create .CMD or .bat files for DOS scripting, .PS1 files for powershell, and .URL files to launch apps that may be hidden or blocked.

However, if they know what they are doing with the locked down Group Policies, all of this may be locked out. Theres many exploits you could try, that I've not gone into I've found in the past that typically something has been missed, and you can get what you need.

All of that said, breaking out of the prison her IT dept have created may be cause for her employment to be terminated, so tread carefully.

Too late to edit...

> Does she have installed locally?

Office installed locally.

Sounds like this thing: https://arstechnica.com/gaming/2021/07/cheat-maker-brags-of-...

There are cheap HDMI to USB capture devices on Amazon/eBay: https://sites.duke.edu/ddmc/2020/06/23/ultra-cheap-hdmi-to-u...

This one would be too annoying to have to run (for 1080p) at 30fps with the inherent mouse display lag.

I was considering similar devices for Twitch streaming or other media capture/processing and recognized Elgato and AVerMedia brands. I don't know much about the subject and am just considering getting into OBS etc with something that can do 1080p @60fps passthru capture like the Elgato Game Capture HD60 S (or maybe S+).

https://tinypilotkvm.com/

Can she run power shell scripts?

no, nor batch files

Reply on news.ycombinator.com