Writing a personal automation API
dev.to
dev.to
Really not trying to self-promote, but genuinely interested in laziness as a quality of great developers. I talked about it at a javascript meetup [link redacted] (Ignore the David Foster Wallace hair, the math error. Security folks don't worry that address is not valid).
I bought up lzy.io and wanted to make a lazy manifesto, but I don't think we need any more manifestos, So I just collected some examples of programmers taking laziness to new levels good and bad!
hardwork != productivity.
The idea that "working hard is good" is an outdated notion from a time when manual labor was more closely tied to productivity.
It's easy for the ignorant to ignore but when they say "lazy" they generally mean something closer to "don't want to know" something or "not courageous enough to ask" because it actually takes more work and is thus harder to change for the better through learning things in actuality than it is not to change for the better. As if they desire it to be true that they can succeed by only having to use their brain, it may be a form of greed.
Here's a basic programming example say you're writing some front end javascript to make a call to another site's api... if you have to do it once, you probably will set the headers, and make the call right? Well if you have to do it three or more times, it pays to be lazy and write some sort of function to wrap the fetch call and automatically set the headers for you each time.
This is the laziness that I'm advocating. It's not about people that don't put effort in, it's about maximizing efficiency and taking processes approach. In many cases truly lazy people will work twice as hard now to avoid work in the future.
Totally agree that if you are only doing something exactly three times, probably not worth it. That being said when was the last time you were absolutely sure the requirements won’t change? Only the Sith deal in absolutes.
(I mean, it's a pretty good piece of advice... we just never talk about the irony)
I think you're just not usually around the people who talk about it. Here are some example image macros.
http://img.memecdn.com/only-sith-deals-an-absolutes_fb_51269...
https://img.memecdn.com/who-amp-039-s-really-on-the-dark-sid...
https://i.pinimg.com/originals/d9/32/8c/d9328c625f6f47e8d8a9...
https://pics.me.me/only-a-sith-deals-in-absolutes-that-itsel...
https://pics.onsizzle.com/whats-wrong-obi-wan-kenobi-tells-a...
Great site!
Does anyone use IFTTT heavily and have a better experience? Are you able to rely on it?
Automation makes most sense in the true context where it belongs, "business." Automating a complicated e-mail marketing process with targeting, statistics and all makes sense because you need tens of thousands of instances of that process to be executed to create business value.
Except for rare idiosyncratic use cases, spending time in personal automation is more of a hobbyist and educational activity. Further, if the use case is not "that" rare, there is almost always "an app for that" anyway. Therefore, have no high expectations out of that.
With that disclaimer in mind, if you are into it, check out the open-source project Huginn [3] that can "Create agents that monitor and act on your behalf." for personal automation. Lastly, the industry standard of cloud API integration currently might be Zapier [4], which has more integrations, (business) battle-tested features compared to end-user oriented IFTTT.
----
Why am I posting this here? I am honestly on the fence about whether this has been a good use of my time. It took longer than expected and I had to use Powershell for some of it, which was just dreadful. It was ‘for business’ but I am still not sure it was a great call. Has anyone else out there found themselves automating something like this? Was it worth it?
Main issues were related to RAM amount. It is rock solid since, remaining issues are caused by ancient software running on it which cannot be bumped without reinstalling the server.
The management and ease of deployment made it totally worth it and the system is much simpler than running a "real cluster" like Kubrenetes or especially a Docker setup. Original was a chroot that got messed up on OS updates.
For someone who knows little about cloud computing or VMs but is handy with Python, where would I start? When trying to learn this stuff, I have always come to the conclusion that everybody who uses these technologies is using them for drastically different use cases than what I would use them for. So, I quickly conclude these solutions will involve forcing them to do something they were not designed for.
Learn about modern filesystems with snapshots like ZFS, and PXE. If you don't need native performance, use a VM in preference to PXE.
This is how many cluster/cloud data centers operate.
https://en.wikipedia.org/wiki/Snapshot_(computer_storage)
Unfortunately, I do need native performance. My workstation runs a 5960X, 256 GB RAM, x4 GTX 980 Tis, a stack of SSDs and I use all of it regularly.
I’m very curious about this PXE solution. I am googling now. If anyone has pointers on methods that fit my needs, I would be gracious. Thanks!
UPDATE: Reading the Wikipedia on PXE suggests to me I would be using one of the Microsoft deployment solutions. I messed with these a littleand it felt like cutting bread with a chainsaw. It sort of worked but, on top of many known bugs, it it was designed for big deployments, prioritizing hundreds of features and admin relationships that I have no need for. Maybe I’m looking at the wrong thing, but it’s pretty hard to tell.
My general goal in my scripting solution was to automate out the Windows annoyances like UAC, bloatware, registry configs, and software installation. I learned how to do that with scripting and it seems predictable even after OS updates, and I can modify it very easily. It has a few run-all directories for quick mods, and some .jsons with config data. I made an install USP that puts a script on my desktop which installs git, pulls my setup repo down and runs it, downloading files needed from my google drive... just to give a picture of what worked. It takes about the length of a lunch break to run and I’m back in action, with exception of a few licensed softwares that I chose not to automate. I’m still interested in learning better solutions.
cat /dev/sde3 > /storage/windows-vr-installation.bin
Then when the install dies for some reason, restore it with:
cat /storage/windows-vr-installation.bin > /dev/sde3
You will say it is inefficient. Maybe. On an optane 900, the burst write speed will be decent enough. Keeping .bin images that are a few Tb is not really a problem with modern setup (zfs, raid, etc.)
I know there are commercial solutions doing that, but I wouldn't bother much with that. A simple free software setup will end up being more efficient if you know some basics.
If I were you, I really wouldn't bother more than that with automation for reinstalls. You are not deploying to N identical devices (in which case the above trick may still work by having sufficiently similar workstations, and a special licensing scheme - I often buy a few similar laptops for the same reason).
Regarding native performance, since being able to reconfigure windows quickly seems very important, if I were you I would do a GPU passthrough. KVM on computers with VT-d etc offers near native performance. That's what gamers do, and I trust them for their insane attention to fps performance :-)
It may take some time for the initial tweaking (recompiling kernels, selecting some hardware etc) but it very possible, and also very stable. And it is more and more common.
(disclamer: I sell such solutions, and do various weird things for my clients -- including supporting Wine when their need for a very stable solution is greater than the costs of the time I will bill them + the specific hardware I will have them buy to guarantee the speed of the results)
It makes sense. It sounds like a similar result operationally to using something like Macrium for a master image backup. This is something I wanted to prevent because modifying that master image backup on a regular basis is such a task. With my scripting solution, it's just editing a few json files.
My working environment is constantly changing, especially these days. My preferences also change as I optimize my workflows. My thoughts were to use the approach of For each change in implementation, make a change in code.
The other important thing, to me, is that I am free to install any wacky software I may need to try without worrying what it's doing to my system. Something like intermittent backups will lead to accumulation of garbage files when doing this. And if I choose to adopt the software, I add it to my setup package.
How do I modify the VM on a regular basis?
I trust the input but maybe my understanding is limited by lack of context. I wish I could find a decent training source on this.
I also lack the context to recommend you a better solution. I do not like adding complexity unless it is needed. Hard drives are cheap. It may not worth the hassle to do something more complicated.
Honestly, if the above is sufficient, don't even bother with a VM at first. Just reboot to Linux when you need to go back to a given image. It will take a few minutes, half an hour top, and you're good to go.
A VM makes sense if you are short on space for the snapshot AND you would benefit from it -- like if you are doing half your work in Linux anyway, or if you have some specific security constraints. It could also be because you can't even tolerate the minutes it take to restore an image.
Setting a VM with GPU passthrough/eGPU/kvm/Windows hosts/etc. is very tricky, especially when performance matters. I can't point you to a training source because it depends so much on the situation -- and it's evolving quickly too.
If you want training or a dedicated solution, I offer results oriented consulting services. Let me know your email and we can discuss a solution more tailored to your specific situation - I will recommend a specific setup, get it running for you, and train you on how to use and change it, so that you don't end up needing me after that.
As for the environment reload solution, your solution sounds like the best one for most people, and probably the best one for me in practice, to be honest. But there is some value the scripted solution serves which snapshots cannot.
Updates to the snapshots require some planning. If a wacky software is tried and uninstalled, it is no longer safe to run updates to the snapshot until implementing another refresh. My scripting solution is just a code mod and git push and I can freely test wacky software and update my configuration in any order I like.
I guess you could say image snapshots have an accumulation cost. A refresh is not a refresh to new but a refresh to old on a certain date.
When I have a failure, I want to isolate issues as soon as possible. Determining whether it's a hardware issue or software issue can cost a lot of time and distraction. The fastest and surest way I know to do this is with an OS reinstall from the distributed source (Original Windows ISO). Image snapshots only theoretically isolate from system changes made since original OS install, and that level of isolation is getting harder to confirm as software deliveries get more common. These probably qualify as edge cases, but edge cases are most of what this is about. Edge cases are notorious for undermining production deadlines in an environment not prepared for them. A funny thing about that is they are also one of the hardest for people to admit or understand. Telling a non-technical client the job won't be done in time because of a software bug carries a burden of understanding hard to put into words. Of course, studios are prepared. But freelancers are generally not.
A particular pain point with GPU rendering has been depending depending on driver/hardware/OS compatibility which has different needs than the priorities of gamers. That is changing, thankfully. But, the value of isolating software issues from the hardware is big to me.
I juggle hard drives like nobody's business as it is, so I'm not opposed to that. And while it's nice, source control might be overkill for something I might not even need for months on end.
For the edge cases you describe, including isolating drivers and explaining the precise cause, an itemized diff would make more sense.
I don't know if rsync would work on ntfs, but I would try something like that
You should never, ever shut off UAC.
1) No justification for keeping it enabled on my workstation.
2) A myriad of don't ask why beratements like yours which cause one to wonder if the author is more interested in belittling others or helping them.
I never assumed the warnings were wrong, but they were not helpful. The impositions of UAC on my workstation were severe and I never found an explanation that addressed why I need it.
If I obeyed the warnings of every computer safety expert, I would not even have a workstation to begin with. In life, one must choose their battles. I'm doing my best over here.
At risk of being attacked and down-voted again ,I'll ask a few questions if you don't mind:
In my experience using Linux, I could just type 'sudo' and be root. As far as I can tell, this does not protect as much as impose a level of mindfulness. What am I missing?
If Windows UAC serves an equivalent purpose, why are the consequences so much more severe?
In Linux you don't just type sudo, you also enter your password. In a best case scenario, you should have a different account for administrative actions as your normal use, but for both Linux and Windows the elevation requirement ensures that a program meant to run on your computer can not silently and suddenly make changes to your computer.
If your computer is not connected to the Internet, or you know the contents of every single block of code on every application your computer runs, this is not a problem, but for most people, this is the bedrock of basic security: That if something wants to make a system level change to your computer you must explicitly permit it to do so.
In Windows, UAC has 2 effects. One, some softwares cannot be opened or installed. Two, installing software requires an extra permissions window be clicked on. No password is involved. I don’t mind the second at all. The first however is awful. Numerous softwares I depend on are sacrificed if I enable UAC. These are softwares from very reputable companies which I have used for many years and depended on for many projects.
Linux is not an option (hopefully is obvious why) and Windows with UAC isn’t either. So, I’m using Windows with UAC disabled just like probably a million other people. This is a very pervasive issue.
So, one whose career depends on disabled UAC is left to wonder “what is the risk?” And they are left with no useful answer to that question other than ‘too many to wrap heads around’ what can we expect them to do? It feels like a mortal threat at this point. The world is full of disease but you still go to the grocery store because you need food. Some things are out of my control and effectively using my computer for work with UAC is one of them.
I think your spellbound reaction to people disabling UAC lacks some consideration for their situation. Is it a problem? I’m certain it is. But, different people have different goals and priorities in their life. If yours is to enable UAC on other’s computers, the path to accomplishing that will begin with understanding why it is disabled. It sounds like a noble and important cause that I would be happy to help with, because circumstantial claims assert UAC is helpful to protect (me? my computer?) and that sounds good.
It's generally cheaper to automate something if you can perform it on a command line. With command history, it's easy enough to copy the commands and turn them into a script with appropriate parameterization.
It doesn't help you a great deal with Windows if you've been doing everything with GUIs. But Powershell itself is an investment that can pay off. If you can automate more everyday tasks, the pain of learning is amortized away.
I generally stick with the Unix-like shell that Cygwin gives me, and use the same commands and idioms on as I do on Linux and on Mac, on the very few occasions I use a Mac.
I am scared of the Windows command line - not because it's scary, but because it's so limited. I feel almost helpless, all my tools are missing, it's just so awful.
In practice, the reason I use Windows is mostly games, but it's rare I fire them up these days - I have to upgrade and/or re-verify my Steam account almost every time I start the client. Outside of that the browser and the terminal are almost all I use, and I can have them just as easily elsewhere.
My point being my experience may not translate to your environment quite as well.
Registry in Cygwin is mounted under /proc/registry:
~$ ls -l /proc/registry/HKEY_CURRENT_USER/Software/Microsoft/Windows
total 0
dr-xr-x--- 72 SYSTEM SYSTEM 0 Dec 16 23:57 CurrentVersion
dr-xr-x--- 2 SYSTEM SYSTEM 0 Dec 13 01:58 DWM
dr-xr-x--- 4 SYSTEM SYSTEM 0 Dec 13 01:48 PrivacySettingsBeforeCreatorsUpdate
dr-xr-x--- 6 SYSTEM SYSTEM 0 Dec 13 01:52 Shell
dr-xr-x--- 4 SYSTEM SYSTEM 0 Dec 13 01:47 TabletPC
dr-xr-x--- 3 SYSTEM SYSTEM 0 Dec 13 01:47 'Windows Error Reporting'
dr-xr----- 3 barrkel None 0 Dec 13 01:58 Winlogon
The other aspect of it is that it may help you keep long trains of thought or keep you in the zone longer if there are fewer interruptions.
As someone with ingrown toenails that has ended up at the podiatrist when I stubbed a toe badly enough, I respectfully disagree with your analogy. :)
It doesn't likely save me tons of time, but I feel like it lowers my stress levels by letting me just sit and relax or work or read for longer periods of time uninterrupted.
When I think of personal automation api. It would be to help me with my routines
1. Shower 2. Shave 3. Brush teeth. 4. Check email 5. Plan my day 4. Make breakfast 5. Exercise 6. Sleep
Any of the above could be optimized to be excellent.
For example playing my favorite podcast automatically while I go to the bathroom would be nice. But knowing/using the best shampoo, soap, or toothpaste is also of great value. Same with what breakfast to prepare/eat. Or the gym/workout routine to follow. Technology might help a little, but some active thinking about what and how to optimize is needed.
Every person I asked used it to sync some stuff from A to B, never heard something interesting...
On the other hand, anacron + python + requests + lxml (or beautifulsoup when they've improved) + other nice libraries have never failed me.
You're better off renting a entire VPS and creating code that polls all your data sources and create your own task pipeline.
Curious to hear if anyone else is making IFTTT work for them on a daily basis. My initial experience with IFTTT was buggy, tasks would fire inconsistently, with no clear way to debug.
I tried it but never understood the value they provide for users with basic programming and scripting knowledge.
I either use the API directly, or, if not available or I only need some quick and dirty solution, resort to web browser automation tools like Kantu.
It depends on what you're doing. Most users use it as a simple, time independent trigger mechanism. IE: when they post to twitter, IFTTT picks that up and reposts to tumblr.
Most popular use cases are not time sensitive or high throughput. I once heard a fellow programmer say IFTTT was not designed for people like us. Those two caveats are why. Time sensitive use cases, like getting text messages when it's going to rain, don't work well. Neither do high throughput tasks like filtering the HN firehose.