akarve

Done. You should see a more obvious search bar on the next push. We'll also make search case insensitive :)

Is there an example in the docs that just shows `examples.sales`? If so please let me know and I'll fix it. I searched and couldn't find such an example (though maybe, through the "magic" of Chrome's service-worker, got an old version of the website). As mentioned by Kevin `()` or `_data()` is what you need.

You can absolutely edit in place and that will go into `quilt log` for the package--as long as you are the package owner. Our docs were a bit confusing on this point. I just updated them: https://docs.quiltdata.com/edit-a-package.html

We charge business and on-prem users in TB-sized blocks. So that part is variable cost, not flat. And we sell user seats in blocks of 10. What else should we be thinking about? We want to be fair and also price in a way that encourages sharing behind the firewall (e.g. shouldn't require manager approval to add every new user).

We really like pachyderm and know the founders. Quilt is zero-config focused on storage and versioning, pachyderm is more focused on on-prem and compute (as a Hadoop replacement).

Hi. R is really important to us and we want to add support (probably through SparklyR). If you email me I can add you to our Slack channel and we can talk through extending Quilt (aneesh at quiltdata dot io). There is a sliver in our docs but it's not complete: https://docs.quiltdata.com/basics.html

The hooks for extending build targets are here: https://github.com/quiltdata/quilt/blob/4aa6897f9e33349b7778...

Again it's a bit raw but we're here to help make R support easy.

We support arbitrary data formats in that Quilt falls back to a raw copy if it can't parse the file. On the columnar side (things we convert to Parquet) we support XLS, CSV, TSV, and actually anything that `pandas.read_csv` can parse. We use pandas and pyarrow for column type inference. We want to add a "data linter" that checks data against user-provided rules, and welcome such feature requests on GitHub or in our Slack Channel.

Understood. We can do something very close to that. Right now we give the registry source to our on prem users as part of the license. Email sales at quiltdata dot io and we can discuss.

Data cleaning is so necessary. `build.yml` already supports a limited set of feature (through pandas). In addition to custom data transformations, any "out of the box" cleaning functions you'd like? In the spirit of dplyr? We've looked at e.g. scikit feature for normalization, 1-of-n encoding, etc.

The package name + hash is an implicit DOI. What if we added web support for it so that users could https://quiltdata.com/packages/USER/PKG?doi=SOME_HASH ?

The client is fully open source. You can indeed run your own and we are just starting to roll that out. I can get you started: feedback at quiltdata dot io. We are deliberating open sourcing the registry as well (making everything open source). What do you think?

Hmm. We were able to get the pip handle so didn't see major conflicts in our target space. Are there places in code/cli where we could name conflict with the patch sets manager?

As to your slicing question, yes. Data is lazily loaded. With Parquet as our data store we can do even more (but haven't yet): load only the columns referenced.

Four things: serialization, virtualization, querying, big data (even Git LFS isn't super performant for large files). Quilt actually transforms data into Parquet and wraps it in a virtualization layer so that the data can be injected directly into code. Efficient querying is a function of the serialization.

By contrast, GitHub is a blob store, it doesn't transform the data either for serialization or for virtualization.

Thanks. Where do you see it going in 2-5 years? Sometimes outsiders see things about the future that we haven't thought about :)

To start with, we can add stars (pay with prestige). Getting more into science fiction--but very possible science fiction--we can put data on the blockchain and let people transact. The data owner would get the lion's share of the transaction.

Dat is a distributed transport layer for raw data. Quilt is a centralized (your infrastructure or ours) transport and consumption layer for virtualized data. As such we'll be able to, for example, run efficient queries across all of Quilt, allow users to import data the same way (no data prep scripts) across a variety of platforms, etc.

As Kevin mentioned we can extend support to frictionless (and are acceptign PRs on GitHub :). The thing we didn't love about frictionless is that it requires the user to fully specify the schema. We take a slightly more automated approach: https://docs.quiltdata.com/make-a-package.html https://docs.quiltdata.com/buildyml.html

I think we could generate a frictionless schema pretty easily...

Yes :) Happy to discuss in detail if you have specific types of abuse in mind. Our first line of defense is to get help from the community through downvoting useless content.

Only the owner and designated collaborators can view it. We offer private data in the cloud (S3) and are rolling out on-premise in case you want to run Quilt on your own infrastructure, in which case you control blob storage. Does that answer your question?

Which languages are most interesting to you? We wrote the client with an eye towards supporting R and Scala. A PR or FR on GitHub would be ideal. We also have a Slack channel where we can support you if you want to tackle adding language bindings to Quilt. feedback at quiltdata dot io.

Typo fixed. Thanks :)

GH sign up would be useful. If you email us I will ping you when we add it: feedback at quiltdata dot io.

Agreed. We're considering doing more along these lines, such as generating the Hive DDL. So you would like to browse the schema, e.g. under Contents?

Ah, that would be useful. In the meantime our docs are browsable without JS https://docs.quiltdata.com/. The articles linked at the bottom of the docs landing page are also legible without JS. Unfortunately the docs menu is not available without JS. I will look into fixing these issues. Email is feedback at quiltdata dot io if you have questions :)

Hi, I'm one of the founders of Quilt Data (YCW16). We built Quilt to bring package management to data. The goal is to create a community of versioned, reusable building blocks of data, so that analysts can spend more time analyzing and less time finding, cleaning, and organizing data.

Our general inspiration is to create a new kind of data warehouse based on code management practices that haven't yet reached the data domain.

Feedback welcome. Ask me anything.

Nary a consideration of the possibility that poverty and cheating were correlated. "Equality" is euphemism for "the world how I arbitrarily want it--facts be damned."

Author here. Questions about data engineering, versioning data, or why you shouldn't use git to store data? :) Ask away.

You can also file feature requests or PRs against our GitHub repo https://github.com/quiltdata/quilt

Hi Cassie. Did you feel like you had extra leverage to negotiate equity as employee number one? If so, and if I may ask, did you?

yeah, i'm not planning to "shoot originals" of DNA any time soon. just article headers at the macro-molecular scale :)

after your comment i carefully examined every detail of the picture (not just the handedness) and, as a chemist, was annoyed by enough details (especially the hydrogen bonds) to nix the whole thing.