250 karma · joined April 20, 2015
github.com/callicles
Thanks for the follow-up :)
https://x.com/thatsFrScience/status/2073741209592295866
Thanks for the feedback, though, and for taking the time to look at the code. I can ship a round of cleanup.
Agreed, this is a cache that expires and refreshes from the source DNS server. It just looks like a virus that propagates when the cache expires.
We went from the get go to that infrastructure for multiple reasons in the first place:
* Having a durable buffer before ensures if you have big spikes that gets eaten by the buffer, not OLAP which when it is powering your online dashboard you want to keep responsive. Clickhouse cloud now has compute/compute that addresses that but open source users' don't.
* When we shipped this for the first time, clickhouse did not have the async buffering in place, so not doing some kind of buffered inserts was forwned upon. * As oatsandsugar mentioned, since them we also shipped direct insert where you don't need a kafka buffer if you don't want it
* From an architecture standpoint, with that architecture you can have multiple consumers
* Finally, having kafka enables having streaming function written in your favorite language vs using SQL. Definitely will be less performance to task ratio, but depending on the task might be faster to setup or even you can do things you couldn't directly in the database.
Disclaimer I am the CTO at Fiveonefour
It is the same idea as a nextjs application you deploy through docker, you have your application and then you build your docker container that contains your code, then you can deploy that.
I tried to limit the port bindings, we usually expose moose itself since one of the use case is collecting data for product analytics from a web front end, which pushes data to moose. And then usually people want to expose rest apis on top of the data they have collected. The clickhouse ports could be fully closed, this was an example of if you want to connect PowerBook to it
We are built on top of them. Right now the techs above are what’s backing the implementation but we want to add different compatibilities. So that you can eventually have for example airflow backing up your orchestration instead of temporal.
You can think of moose as the pre-built glue between those components with the equivalent UX of a web framework (ie you get hit reloading, instant feedback, etc…)
What you get:
• A single docker compose up that spins up ClickHouse, Redpanda, Redis and Temporal with health-checks & log-rotation already wired.
• Runs comfortably on an 8 GB / 4-core VPS; scale-out pointers are in the doc if you outgrow single-node.
• No root Docker needed; the stack follows the hardening tips ClickHouse & Temporal recommend.
Why bother?
Moose lets you model data pipelines in TypeScript/Python and auto-provisions the OLAP tables, streams and APIs—cuts a lot of boilerplate. Happy to trade notes on the approach or hear where the defaults feel off.
Docs: https://docs.fiveonefour.com/moose/deploying/self-hosting/de...
18-min walkthrough video: https://www.youtube.com/watch?v=bAKYSrLt8vo
We are going towards 1.0 from an API perspective, we have just landed what we internally call DMV2 which is the latest iteration of the abstraction level for the api. Think SST / Terraform CDK vertically integrated for Data.
If you are looking to work with Moose in production we would love to chat with you :)
What makes Moose different is how it simplifies the entire workflow - from ingestion to processing to serving data through APIs. We've found teams spend too much time wiring together different tools rather than focusing on the actual data insights.
The local development experience was a big focus for us. You can instantly test your changes with real data without waiting for deployments. And we've made sure the same code runs identically in production to eliminate those frustrating "works on my machine" moments.
Happy to answer any questions about our technical approach or how we're handling specific use cases. We're particularly interested in hearing about pain points you've experienced with existing data systems or any feedback you might have on Moose.
The docs says that it tries to explore all the branches of the AST, isn't that what coverage is doing?
I might be missing something.
For Linkurious.js there are some examples over here https://github.com/Linkurious/linkurious.js
All infos on the toolkit are available on http://linkurio.us/toolkit/