Self-Hosting Moose with Docker Compose, Redis, Temporal, Redpanda and ClickHouse | Hacker News Reader

Self-Hosting Moose with Docker Compose, Redis, Temporal, Redpanda and ClickHouse

docs.fiveonefour.com

50 pointsCallicles15 comments

I put this Docker-Compose recipe together to make kicking the tires on Moose—our open-source data-backend framework—almost friction-less.

What you get:

• A single docker compose up that spins up ClickHouse, Redpanda, Redis and Temporal with health-checks & log-rotation already wired.

• Runs comfortably on an 8 GB / 4-core VPS; scale-out pointers are in the doc if you outgrow single-node.

• No root Docker needed; the stack follows the hardening tips ClickHouse & Temporal recommend.

Why bother?

Moose lets you model data pipelines in TypeScript/Python and auto-provisions the OLAP tables, streams and APIs—cuts a lot of boilerplate. Happy to trade notes on the approach or hear where the defaults feel off.

Docs: https://docs.fiveonefour.com/moose/deploying/self-hosting/de...

18-min walkthrough video: https://www.youtube.com/watch?v=bAKYSrLt8vo

I have a small open-source project, that uses docker compose behind the scenes, to help startup any service. You can look to add it in (or I am also happy to add it in) and then users are one command away from running it (insta moose). Recently just added in lakekeeper and various data annotation tools.

insta-infra: https://github.com/data-catering/insta-infra

Interesting. How do you do dependencies between those pieces of infrastructure if there's any? For example, in our Docker Compose file, we have temporal that depends on progress and then moose depends on temporal. How is that expressed in Insta-Infra?

It leverages docker compose 'depends_on' for the dependencies (https://docs.docker.com/compose/how-tos/startup-order/). For example, airflow depends on airflow-init container to be completed successfully which then depends on postgres.

https://github.com/data-catering/insta-infra/blob/main/cmd/i...

For everyone else confused too…think moose in this context is probably this:

https://mooseframework.inl.gov/

Actually, this https://github.com/514-labs/moose

I thought of https://moosetechnology.org/ and wondered why I'd need all that fancy other stuff?

Maybe this is the greybeard in me, but I first thought about https://metacpan.org/pod/Moose, and catalyst (http://catalyst.perl.org/)

hehehehe At least I'm not alone...

Clearly I'm showing my age here too

this is too perfect. as an SRE who often needs to hand roll my own deployments in k8s or w/e medium, these are the docs that really accelerate my path to production.

You don't publish a ready-made image anywhere? That would be easier to spin it up without installing locally moose first. Kind of defeats the purpose of Docker Compose recipe.

And those ports bindings, is it really necessary to expose it on 0.0.0.0 by default.

Not sure if this is what you are asking about, so if I misread feel free to correct me. You don’t have to install moose first on the deployment machine, in the tutorial I go through that to generate a dummy moose application to be deployed.

It is the same idea as a nextjs application you deploy through docker, you have your application and then you build your docker container that contains your code, then you can deploy that.

I tried to limit the port bindings, we usually expose moose itself since one of the use case is collecting data for product analytics from a web front end, which pushes data to moose. And then usually people want to expose rest apis on top of the data they have collected. The clickhouse ports could be fully closed, this was an example of if you want to connect PowerBook to it

How Moose compares to more traditional ELT data pipeline orchestration frameworks, like Airflow, Dagster, dbt, DuckDB for transformation steps.

I think one of the reasons to use an orchestration framework is integations.

Hi!

We are built on top of them. Right now the techs above are what’s backing the implementation but we want to add different compatibilities. So that you can eventually have for example airflow backing up your orchestration instead of temporal.

You can think of moose as the pre-built glue between those components with the equivalent UX of a web framework (ie you get hit reloading, instant feedback, etc…)

Reply on news.ycombinator.com