Huginn: Create agents that monitor and act on your behalf
github.com
github.com
Now that it's no longer a young project, here are some musings about Huginn and responses to people's comments in this thread, in no particular order.
I've found that Huginn excels as a scheduled web scraper with lightweight filtering. That's what I use it for. On the other hand, while you can write custom code in it, Huginn is pretty poor at implementing any sort of complex logic, and is even worse at bidirectional syncing between systems, which is something people often want it to do, but for which it wasn't designed.
If IFTTT or Zapier meet your needs, awesome! No need to run and monitor your own service. I personally choose to run Huginn on my own hardware in large part so that I'm comfortable giving it website cookies and passwords.
Some examples of what I use Huginn for these days:
- Watching Twitter in realtime for high standard deviation spikes in certain keywords, such as "san francisco emergency" or "san francisco tsunami warning", which then sends me a push notification, or "huginn open source", that goes to a digest email (and I imagine will trigger because of this thread).
- Watching Twitter for rare terms and sending me a digest of all tweets that match them. Also sending me all tweets from a few Twitter users who post rarely, but that I don't want to miss.
- Scraping a number of rarely updated blogs that don't have email newsletters and emailing me when they change. Some use RSS, most are just simple HTML scraping.
- Pulling deals from the frontpage and forums of slickdeals and craigslist and filtering them for certain keywords.
- Sending an early morning email if it's going to rain today.
- Watching ebay for some rare items.
- Sending my wife and me an email on Saturday morning with local yardsales from craigslist.
- Watching the HN and producthunt front pages for certain keywords.
Basically, anytime I find myself checking a website more then a few times, I spend 20min making a Huginn Agent to do it for me.
I think one reason Huginn has worked well for me is that I don't try to make it do too much. I use it for scraping data and gentle filtering, and that's about it. It's been super helpful for alerting me to interesting content for The Orbital Index, my current project, a weekly space-industry newsletter. (Last issue: http://orbitalindex.com/archive/2019-12-10-Issue-42/)
And for those unfamiliar, the (also amazingly named) historian Snori Sturluson explains: "Two ravens sit on his (Odin’s) shoulders and whisper all the news which they see and hear into his ear; they are called Huginn and Muninn. He sends them out in the morning to fly around the whole world, and by breakfast they are back again. Thus, he finds out many new things and this is why he is called ‘raven-god’ (hrafnaguð)." [1]
[1] https://norse-mythology.org/gods-and-creatures/others/hugin-...
Huginn is a fun ship to fly.
o7
I wasn't aware of the full details of the lore so thank you for bringing that up!
[1] https://boardgamegeek.com/boardgame/192074/odins-ravens-seco...
https://www.audible.com/pd/Norse-Mythology-Audiobook/B01LWUJ...
I thought what a weird coincidence that they'd choose this very similar weird word ... but now that you point out it's the name of one of Odin's ravens, I guess it's not quite as uncommon of a word.
This, exactly and all of the above.
It really is a fantastic project and kudos to you for starting it.
see Huginn for Newsrooms: http://albertsun.github.io/huginn-newsroom-scenarios/
It's been at least as useful as Yahoo! Pipes, and endlessly more reliable. Thanks a lot!
You might just have fixed a couple of problems with your tool, thanks!
This has been immensely useful to me, and yes, my main uses have been primarily web scraping and then piping it into various channels.
Been running it on a cheapish VM for a couple years, very reliable and lets you monitor things more frequently and reliably than services like IFTTT.
- Self-hosted
- Unlimited and FREE vs $699 / month
- Create your own agents / integrations
- Process data through shell scripts or JavaScript
- Better filtering
- Liquid templating
- Completely private
I save thousands of dollars by using Huginn.
It's incredibly powerful and quite frankly I don't trust Zapier with my data and visibility of what I'm doing because there are commercial implications.
Run it in the cloud on AWS or on an old box at home.
It's very reliable.
Does that take into account the overhead of operating Huginn?
You sure about that? I feel like there are plenty of folks on here whose only computer is a MacBook
It works fine, along with a whole bunch of other stuff in Docker.
I have tried a couple alternatives, e.g. node-red but none really worked the way I wanted them to for these cases. huginn is incredibly flexible and (at least for me) the mental model of it's workflow makes a lot of sense. Sadly more and more pages want you to go through their app/site and make it a bit difficult to work with, e.g. getting content from an instagram account.
One thing I have not figured out about huginn and which all of these automation tools seem to lack are loops. E.g I have page an agent scrapes, from which I want to output the src of an image tag but I also want to check if a certain condition on the page matches (e.g. a "next page" button exists) and then firstly output the found src but then also re-invoke the agent with a new input element. So it would scrape the next page and so on until it does not find the button anymore.
I do some very complex stuff using the "Shell Command Agent". You might want to look into using that if you haven't already.
You can also create your own agent gem https://github.com/huginn/huginn_agent
The "retail" integration space remains underserved and if one of the enterprise players decided to go down-market with a better UI and deeper integrations - they'd mop the floor clean in 18 months.
I think it's a tough market to be "winner" of. Novices are going to want a stupid simple GUI ("wizard mode", as someone else in thread mentioned). Power users are going to want to be able to toss in some code at some point in a workflow to do some fancy ETL you don't support out of the box. When you hit a certain level of complexity, an edge case or integration an automation product doesn't support, or perhaps even an amount of spend that you start looking at annually as painful, it's likely you consider pulling all of your workflows out and have a software engineer build something bespoke for your business line.
This is accurate, there's a significant opportunity in this space but it won't stay that way for long.
There's a well-known entrepreneur I know with significant exits and capital entering this space that's going after Zapier's market and I'm certain he's not alone.
Having seen how the sausage is made and what it takes to build and support such a product, I will say that past success is no guarantee of success in this market. It is a grind, day and day out of dealing with queues, API integrations that fail or degrade either loudly or silently and you have no visibility into them, non-technical users who are are not happy when the magic isn't working, blobs of data that just shouldn't be, spending hours tracking down single edge cases or failures, and so on.
It's not "write some code, here's your $1B exit" (but you will be writing a lot of code!). It is constant tending of a complex piece of machinery, and failures are painful (outages somewhat, data loss exceptionally so).
Anyone attempting to take on this space is going to have their hands more than full and they're far more likely to fail than succeed.
Personally I wouldn't even try, mostly because the tools themselves can enable much easier more likely to succeed lower hanging fruit and that's how I use them.
If you want to make money "using" these tools then Huginn is your best option and it gives you a competitive advantage over those you're competing with that are using Zapier to do similar things.
I'm still laughing :) wth! (My fear is that this might actually be sustainable with ads.)
I enjoyed waking up every morning to an often strange political message posted by my "bot".
@CrowdWisdomBot if anyone is curious :)
Doing it with cat photos is far more clever.
Airflow has been a skill that many companies ask for (especially data engineering), but surprisingly doesn't have many articles written about it.
https://airflow.apache.org/docs/stable/_api/airflow/operator...
https://airflow.apache.org/docs/stable/_api/airflow/contrib/...
Does anybody have useful workflows going on?
Yes would actually be perfect for it, but never had the time to set something like that up. Maybe someday in the future. For now, I only have a simple Google Alert set and regularly check HN and Google Analytics which is a good indicator when it got mentioned somewhere.
I don't think so.
Basically for two things: - Transform rss feeds into instant messages (this way I receive new comics I follow on my phone for example)
- Create simple bots
This way I avoid creating scripts every time and managing a new webserver and stuff, all my small scripts are in one place.
I actually use Huginn to check the weather, compare it with weather conditions for the previous two days and send the result to my Telegram as a simple text ("It should be a bit warmer than yesterday in the first half of the day").
I do this because I have a hard time understanding if wearing a coat would be enough for '-5 deg. Celsium, NW wind 30 kmph'. Comparison with previous days helps a lot since I already know that I was feeling ok in my coat yesterday.
These have required some setup but little to no maintenance after.
1. https://github.com/automaticmode/active_workflow 2. https://github.com/automaticmode/active_workflow/blob/master...
About the "robust" part. A startup I know had a lot of issues with Zapier and switched to n8n and are quite happy as they did not have any issues since. I am sure you are right and you did your research but would love to hear what exactly you are referring to as it will help me to improve n8n. The same with the limitations. Anything specific?
One use case for Zapier (from a developer/company standpoint) is to allow customers to connect their existing services to actions inside your own app. For instance, if a customer updates a CRM record, you can have a custom zap update a record in your own SaaS platform.
To pull that off with huginn, is that as simple as connecting this up to Singer.io? Or would that require a big marketplace of huginn agents for popular integrations?
Once I finally had it working, I found it really difficult to debug "tasks" (I forget the correct, node-red verbiage), for example to find out why an HTTP call was failing.
I also found the available connectors to be lacking, and a lot I tried were quite basic.
Used python simplehttp server and ngrok to replicate a json url and play with the triggers to test it all before pointing it at a real website.
Nice to add a new tool to the belt, thanks!
My home server is pretty minimal and lightweight: a Raspberry Pi. Do you think it will run it fine? (I'm gonna want to try this anyway, didn't know about it until now and it looks amazing!)
You could write python externally (cloud functions/aws lambda) and then use huginn to call that over HTTP. This doesn't require any coding. Huginn has existing agents that you can easily configure to make HTTP requests (PostAgent and WebsiteAgent).
And, depending what you are planning to do with huginn, simply sending data from your python script to huginn using the WebhookAgent may be sufficient.
These are things that I want to control and customize, and I vastly prefer running my own systems over trusting any third party.
Curious how this would turn out.
https://github.com/bilde2910/Hauk
Disclaimer: I have no link with the Hauk project, just found about it by browsing the F-Droid store.
Firebase database for long term persistence.
Huginn for scheduling and event based triggers of those cloud functions.
I like to think of Zapier as "wizard mode" - if you're doing something within their pattern, planned workflow, capabilities, etc it is a great option.
If you need to do something more advanced or outside that narrow scope, you need to go to the APIs (or Huginn) itself.
The important questions are: How many people need to get into advanced mode? How big of an effort/skill/understanding jump is it to go there? (aka will it hurt too much to switch?)
After fighting the Docker images for half an hour, I usually realize that I could've just written a Python script to do whatever I was trying to do by then and proceed to do just that.
Sorry for being lazy! Maybe I'll get around to it next year...
- support remote agents and remote reporting?
- dead-man triggers for missing reports?
- is remote reporting properly secured? (encrypted/signed)
One of the great features of huginn is the diagram. That's where there are green and red indicators. Red indicates both errors and when a longer than expected period has passed. This makes quickly inspecting your workflows easy.
Sorry. I meant running an agent on a machine that is separate from the machine running huginn. The machine running the agent is not accessible publicly but has internet access (doesn't accept connections but can initiate them). huginn would be accessible publicly.
Retrieving data from huginn can be done with the DataOutputAgent.
Sending data to huginn can be achieved through the WebhookAgent.
Huginn can assist in organizing complex workflows into manageable components.
I think this should be positioned more like "A high powered Zapier for technical users" or something similar.
I've been intermittent fasting + keto for the majority of the past five years. The minority is when we cycle off keto for a week here and there by doing intermittent fasting + vegan.
I run Huginn on Heroku. I haven't had any significant issues since it's been running. It's been so long since I've set it up that I would need to review my notes to tell you any more about how it's deployed.
What's a "major issue"?
At this point, I feel that I'm well versed in when to use huginn. That helps me avoid most issues.
I use Zapier all the time for the things it does better/easier. That said, often, one end of the "zap" is integrated with a workflow in huginn. For instance, I commonly use Zapier when I need data gathered by huginn to be continuously logged into a spreadsheet.
Additionally, puppeteer(node) is another fantastic automation tool, which I use to do things that you probably shouldn't try to do with huginn. But, I connect my puppeteer scripts to larger workflows using huginn.
Thanks to the maintainers, I think that huginn does what it was designed to do very well.
Is configuration saved in textfiles which I can put under versioncontrol and backup, or is it some unaccesable binay blob?
How much does it break with each update? And how often does it receive bigger and smaller updates?
How does this compare to something like home assistant or node-red, if you know this software.
Anything you consider missing or lacking on some area?
Primarily Web interface. Though, the CommanderAgent can configure and trigger other agents. You can use that in combination with your external script for configuration.
> Is configuration saved in textfiles which I can put under versioncontrol and backup, or is it some unaccesable binay blob?
Agent configurations are stored in the database and can be exported as text files for your backups.
> How much does it break with each update? And how often does it receive bigger and smaller updates?
Making sure updates don't break existing installations is one of the top priorities of maintainers. Most of the updates are small/incremental. And I haven't had any problems with the updates breaking things.
> How does this compare to something like home assistant or node-red, if you know this software.
I looks at node-red years ago and determined it wasn't as mature of a project. Things could've changed since then. And I haven't heard of home assistant.
> Anything you consider missing or lacking on some area?
The UI is a bit basic. Though, I think this can also be looked at as a feature instead of a flaw.