Show HN: Hamilton, a Microframework for Creating Dataframes
github.com
github.com
Hamilton wouldn't replace Dask. Instead Hamilton could use Dask, we even have a prototype PR to do so.
https://github.com/asavinov/prosto
However, Prosto allows for data processing via column operations in many tables (implemented as pandas data frames) by providing a column-oriented equivalents for joins and groupby (hence it has no joins and no groupbys which are known to be quite difficult and require high expertise).
Prosto also provides Column-SQL which might be simpler and more natural in many use cases.
The whole approach is based on the concept-oriented model of data which makes functions first-class elements of the model as opposed to having only sets in the relational model.
* It's solving a really different problem than Spark/Dask/etc. It could definitely use those tools but it's just not the same thing.
* If you're looking at this and thinking it's useless even if you're familiar with Pandas/dataframes it's probably just that you haven't had to work on the types of problems that this particular tool is intended to help with.
One note: we do have ideas to extend Hamilton to "compile" the DAG onto Spark/Dask/etc.
It was in Matlab* though and not as slick as what you've created.
Is the idea with Hamilton to limit it specifically to columnar operations? e.g. no functions that filter the DataFrame? I think that's one thing that I attempted to do that ended up overcomplicating the framework I created.
*womp womp
I didn't get that at all.
Spark is a data-processing engine.
Hamilton seems to be a framework for expressing all of your feature extractions and transformations in such a way that they can be encapsulated and organized into functions and modules. These functions can then be composed into a transformation pipeline instructions. Hamilton can then inspect this composition of functions and understand all of the inter-dependencies, effectively constructing a DAG that can be used to visualize what exactly the pipeline is doing while also creating some feature/column lineage. This DAG can also be used to figure out what functionality can be pruned from your modules if it is never used. Unit tests written easily.
There is nothing within Hamilton that dictates which data-processing engine should be used to do the work. In fact they mention it in their blog that Spark could indeed be used. But I imagine anyone could come along and write an integration for Pandas (if it doesn't already exist), BigQuery, Snowflake, Materialize, etc...
This creates an abstraction between what you're trying to do, and what data processing engine is used to do it. Not to mention being able to create some fantastic visualizations (i assume) of what is actually going on in your transformation or feature-extraction pipeline.
It isn't immediately obvious but that is exactly what Spark does - it builds a DAG out of your operations and only lazily executes what it has to when asked for a result. But on top of that, a tremendous amount of work has gone into auto-optimising that DAG execution so it can be very very smart while allowing relatively natural expression of queries and transformations at the top level. This is all going to have to be re-invented if you try and replace the DAG front end and just use spark's execution back end.
Again, I don't know enough about Hamilton, but I don't think it's a matter of replacing anything Spark does, or any other data execution runtime for that matter. It's just another layer, another abstraction.
Hamilton -> Spark SQL/API -> Spark artifact -> Spark execution
Hamilton -> Snowflake SQL -> Snowflake execution
etc...It’s used a lot in data science applications, particularly for matrix math.
> Two-dimensional, size-mutable, potentially heterogeneous tabular data.
> Data structure also contains labeled axes (rows and columns). Arithmetic operations align on both row and column labels. Can be thought of as a dict-like container for Series objects. The primary pandas data structure.
The easiest way for this is to click is open up Python or R and play around with a dataframe, it really isn't that complicated.
Newly hired data scientists would have a "wtf is this thing?" response. You'd really need to "sell" people on this and it doesn't seem worth it.
So yeah, not for every problem (no framework is), but the problems that it solved it solved quite adeptly, and it was worth the cost (both initial and ongoing) of migration. It particularly proved its worth in the case where pipelines have grown in scale and complexity over time and thus gotten out of control. Particularly when there are thousands of lines of transforms -- that's when the ongoing maintenance burden gets so high that its completely worth the cost of migration. "Weeks going by and something seems off" we found often described the day to day of teams that manage codebases like that -- requiring them to spend tons of time working on the software engineering aspects of testing/visibility rather than developing new features to help the business.
Re: new data scientists, you'd be surprised at how adept they are at picking things up. There are a million frameworks/paradigms to learn when you join a company, so this isn't really an additional burden. And, once they learned that "one simple trick" (that parameter names map to the producing function), it ends up being pretty easy to use.
This reminds me of an internal Google tool for doing async programming in Java (ProducerGraph or something). The idea was that you'd just write annotated functions and the framework would handle all the async stuff. Wasted many thousands of engineering hours while giving an even worse experience than manipulating futures directly.
In our case there’s very little magic, and a dead simple engine (so far). We (from a DS perspective) have found that expressing transforms in this manner simplifies the code making it well worth the added layer of abstraction. As a platform team we’ve focused on debugging (viz, etc…), but this is part of the reason we open sourced it! The more voices we get the more operational concerns we can iron out to avoid painful debugging experiences.
In a spreadsheet you're basically building a DAG (assuming you're using it right) that automatically and efficiently recalculates the downstream nodes whenever you change any of the input nodes.
I think it's actually a pretty hard experience to recreate in many programming languages. I think in Python the thing that comes closest these days is Streamlit, but it can still be a lot slower to put together something than a really fast Excel jockey.
Suppose you had a HamiltonFrame that knew the graph that generated it and knew which columns were inputs and which were outputs. When you update any of the input values it could automatically recalculate any columns downstream of the modified inputs.
This project has similar motivations but explains itself a bit more I think. https://opensource.janestreet.com/incremental/
EDIT: Just realizing belatedly that since it was the author of Tributary* who mentioned it he can probably explain himself better than I can...
*along with roughly 450 cool Jupyter extensions.
The blog post gets into the problems created by doing "something simple".
https://multithreaded.stitchfix.com/blog/2021/10/14/function...
I'm still not seeing the usefulness but I see a lot of potential problems and complications from redesigning your codebase like this. I work with pandas dataframes all the time. It really looks like they're forcing a functional paradigm in a place where it doesn't fit well.
But I believe what this would enable you to do is continue writing your data processing pipelines and then have them execute in Pandas (assuming there is a Pandas implementation) while you're working locally.
When it comes time to deploy your transformations to a production environment that uses Spark, Snowflake, etc... for its data processing engine (assuming integrations exist), it should work just the same way.
Keep in mind, that these are instructions that are being deployed. The power here is that the instructions are pushed down into the data processing engine. Data is not exhumed into the dataframe runtime! Hamilton functions can be "compiled" into data-engine runtime code (pandas, SparkSQL/SparkDF, SQL, etc).
Then on top of all of this, you (and more importantly your teammates and/or replacements) will have a better visualization of what your pipeline is doing. You'll also have a better idea of what unused functionality you can prune out of your code-base. Better unit testing. etc...