Show HN: Edge HTTP to S3

edge.mq

2 points·_ben_··2 comments

Hi HN,

Edge.mq makes it very easy to ship data from the edge to S3.

EdgeMQ is a managed HTTP to S3 edge ingest layer that takes events from services, devices, and partners on the public internet and lands them durably in your S3 bucket, ready for tools like Snowflake, Databricks, ClickHouse, DuckDB, and feature pipelines.

Design focus on simplicity, performance and security.

Open articleView on HN

Would you mind sharing a motivating use case for those of us who don't think S3 is complicated or unreliable? Doesn't S3 already include HTTP upload capability? Are ML engineers really avoidant of basic operations like "HTTP retries and S3 multipart uploads"?

Thanks for the question. You’re right that S3 itself is simple and reliable, and yes, most engineers *can* write HTTP retries and multipart uploads. EdgeMQ isn’t trying to replace S3’s API, it’s what you need around S3 when you have lots of producers on the public internet.

It gives you:

* edge HTTPS endpoints (auto-scale, multi-region HA) * a WAL so accepted events aren’t lost * segmentation + compression * explicit commit markers for consumers * backpressure instead of silent data loss * and a standardized way every team lands data in S3

You could build that yourself on top of S3; many companies do. EdgeMQ exists for folks who wants that behavior but dont want to operate a custom HTTP to S3 ingest service forever.

Its also worth noting that its in the early stages and the next features to be developed are transformations whereby you can input format a (say, JSON) and deliver in s3 as format b (e.g. csv, parquet etc).

Reply on news.ycombinator.com