Hacker News Reader

Comment by josh-sematic | Hacker News Reader

There are several differences, but I'd say these are some of the main ones:

UI: whereas Metaflow provides the ability to build your own result visualizations explicitly in your workflow (via their "cards" feature), Sematic makes it so that your outputs (and inputs) get automatic rich visualizations based on the data type of the data being passed around.

API: Instead of being based around explicitly building up a graph, where you have to explicitly specify the I/O connections between steps, Sematic makes defining your steps look like writing/calling python functions.

Packaging: Whereas metaflow requires you to include packaging information in the code defining your steps (the @conda decorator, etc.), Sematic plugs into your existing dependency management to bundle up dependencies for execution in the cloud.

View on HN

Interesting approach on passing data between steps and constructing the overall graph - it will be interesting to see what the take rate is between the two approaches (of sematic and metaflow). On the UI front, Metaflow generates viz for all objects by default in @card; but how does Sematic package up PyTorch referenced in the example (https://docs.sematic.dev/real-example) for execution on the cloud? IIRC, Metaflow packages the cwd (in addition to @conda, @pip etc.) and relies on existing packages for local execution?

Edit: Digging deeper, Sematic relies on Bazel (https://docs.sematic.dev/execution-modes#dependency-packagin...) and needs a BUILD file to specify all the dependencies for cloud execution. It seems that the entire pipeline will execute as a single (or multiple) k8s pod(s) using the same environment?

I am quite interested in trying out Sematic. Any guidelines on what kind of scale Sematic can support today (and the near future)?

The way packaging is designed to work in Sematic is for us to hook into your existing dependency management solution to determine what your dependencies are, then build a docker image for you based on those. As you point out, right now we only integrate with bazel for this purpose, but we hope to add more. A simple plugin for requirements.txt -> Docker image is probably next on the TODO list.

> It seems that the entire pipeline will execute as a single (or multiple) k8s pod(s) using the same environment?

single docker image, but multiple pods (when you are using the full cloud mode). This was an intentional decision to avoid confusion around what things could be imported in what places (mimicking more what it would be like in one python instance), and also avoid weird version inconsistencies across the pipeline.

> Any guidelines on what kind of scale Sematic can support today (and the near future)?

Based on some prior tooling experiences, the main bottleneck should be what your Kubernetes cluster can handle.

> I am quite interested in trying out Sematic

Glad to hear it! We'd love to hear about your experiences. You can join our discord if you want help while you're trying it out: https://discord.com/invite/4KZJ6kYVax

Reply on news.ycombinator.com