Show HN: Vaero – Build fast log pipelines in Python

We’ve spent the last few months building Vaero and are now sharing the beta for community feedback: https://github.com/vaerohq/vaero.

Vaero is the first log shipper for developers to write log pipelines with code. You write pipelines in Python, and the pipelines are converted into task graphs that run super fast in Go.

Why we built this: - Current log shippers like Fluentd, Logstash, and Vector.dev only let you specify log pipelines with config files (YAML, JSON, TOML) or domain specific languages (VRL) - This works for simple pipelines but is unwieldy for even mildly complex use cases - It’s annoying for someone on the team to have to learn the specific config file format or DSL of the tool - You can’t use the code-based development tools you’re used to, like IDEs, version control, git, and so on

Our beta version lets you: - Write log pipelines in Python using a Spark-like syntax - Pipelines run blazing fast in Go - Native integrations to collect from log sources like HTTP, Okta, and more and write to destinations like S3 - Built-in transformation functions to reshape and normalize log data - Single binary running on a Docker - Open source

We’d love to hear your feedback and opinions. Feel free to try us out for free - and if there’s a specific integration you want, let us know.

Github: https://github.com/vaerohq/vaero