SirOibaf

At Hopsworks we built HopsFS-S3 to improve things like listing (becomes a partition pruned scan of a in-memory DB), atomic renames and added a block/object caching layer using NVMe drives.

You can read the research paper here if you are curious: https://www.hopsworks.ai/research-papers/hopsfs-s3-extending...

GAIA-X, is that you?

Mr. Gorbachev, Tear Down This Wall! - https://www.youtube.com/watch?v=GCO9BYCGNeY

Reminds me of this: https://www.youtube.com/watch?v=0KXoBcQER_0

Logical Clocks | Mid/Senior Frontend Developer (React) | Europe (Stockholm or Remote) | Full-time | https://logicalclocks.com

Logical Clocks is the company behind Hopsworks. Hopsworks is an open-source feature store that allows teams to organize and scale their feature engineering efforts. (https://hopsworks.ai) We recently raised our series A (5M euro) and we are expanding the teams. We are looking for a mid/senior frontend developer. Our frontend stack includes Typescript, React JS, Redux.

Your responsibilities will be to lead the development the new Hopsworks frontend, lead the technical discussions/decisions and help with the expansion of the UI/UX team.

If you want to apply/have any questions you can reach me at fabio [at] logicalclocks [dot] com or through our website: https://www.logicalclocks.com/job-positions/senior-front-end...

Looks like the Casio is not web scale :)

A bit of a weird article coming from a VC heavily invested in SaaS providers

I'd say that the main difference with ObjectiveFS are metadata operations. From the documentation of ObjectiveFS:

`doesn’t fully support regular file system semantics or consistency guarantees (e.g. atomic rename of directories, mutual exclusion of open exclusive, append to file requires rewriting the whole file and no hard links).`

HopsFS does provide strongly consistent metadata operations like atomic directory rename, which is essential if you are running frameworks like Apache Spark.

Not really, but you can try it out on https://hopsworks.ai

It's conceptually similar to EMR in the way it works. You connect your AWS account and we'll deploy a cluster there. HopsFS will run on top a S3 bucket in your organization. You get a fully featured Spark environment (With metrics and logging included - no need for cloudwatch). UI with Jupyter notebooks, the Hopsworks feature store and ML capabilities that EMR does not provide.

No, HopsFS namenodes are stateless as the metadata is stored on an in-memory distributed database (NDB)

There are other papers that describe HopsFS architecture in more details if you are interested: https://www.usenix.org/system/files/conference/fast17/fast17...

You can see the Hopsworks feature store as a repository of curated features ready to be used in ML models. Or, a middle layer between data engineers and data scientists: - Data engineers write the data pipelines with the transformations and publish the features on the feature store. - Data scientists browse the feature store, pick which features they need and build the model

In the Hopsworks Feature Store we group features together in feature groups. Feature groups can be then joined to create training datasets. (You can also select a subset of features from a feature group) Training datasets are stored in a ML Framework friendly format (e.g. TFRecords if you are using TensorFlow) and you can feed them directly to your model.

If you are interested, we have a longer blog post explaining the core concepts of the Hopsworks feature store: https://www.logicalclocks.com/blog/feature-store-the-missing...

https://logicalclocks.com with their ML + Feature Store open source platform Hopsworks and their managed cloud version https://hopsworks.ai

"HDFS and S3 are designed around large blocks (optimized to overcome slow random I/O on disks), while new NVMe hardware supports fast random disk I/O (and potentially small blocks sizes). "

Interesting solution presented here - keep the block size constant, but put the small files on NVMe disks.