HNHacker News
TopNewBestAskShowJobs

Shelnutt2

463 karma · joined April 1, 2016

submissionscomments
Shelnutt2··on TileDB closes $15M Series A for universal data engine
Time series data can vary and I'd agree that most time series is immutable. There are however use cases in which updates can happen. For instance, at my previous job before TileDB Inc, we had a case where 99% of our data was immutable and never updated but a very small amount of data could be updated if there were late arriving parts. In order to get near-realtime data we accepted that sometimes some columns might not be available within the window the datapoint represents. In that case that record might reappear at a later time with the complete and correct values.

Of course there are trade offs, we could have forced a longer waiting period until we were confident the data was finalized. We also could have ignored the updates. In the end we used a system of staging tables for loading the last 24 hours of data before merging out into a more finalized table. This kept the load of updating records in the database down, and still allowed us to achieve our goals.

At the time, several years ago, I was not aware of TileDB else we would have considered it instead of a more traditional database vendor.

Shelnutt2··on TileDB closes $15M Series A for universal data engine
Please see my comment above in the thread for a full description of TileDB[1].

Compared to Apache Arrow we have some similarities but also some significant differences. Arrow as a project has many components, the most directly comparable are the in-memory data structure and parquet for on-disk storage. For the in-memory data structure TileDB has similar goals of doing zero-copy for moving data between libraries and applications. In-fact we even use Arrow in our TileDB-VCF[2] genomics project for zero copy into spark and python (pyarrow). We are looking to expand support for arrow into other integrations where appropriate.

For parquet, a brief comparison is that parquet is a one dimensional columnar storage format, where TileDB is multi-dimensional. TileDB subsumes parquet in that we include all of its functionality and more. TileDB natively handles the eventual consistency of cloud object stores, and natively handles updates through its MVCC design. TileDB is a complete storage engine, not only a file format. That said parquet does have some advantages, a primary one it has a several year headstart on TileDB in being integrated into many tools, so its more well known.

> Is this suited for large amounts (~1TB) of time series data?

Yes, it's well suited for time series data. We natively support timestamp/datetime fields, in the core library and in many of our integrations (TileDB-Py, TileDB-R, Spark, MariaDB to name a few). We allow for fast sub-slicing on the dimension. You also have configurable tiling[3] so you can shape the array to fit your timestamp granularity and volume. The support for updates also can help if your timeseries data ever gets updated. Many timeseries databases don't recommend updates to records, or they recommend no primary keys and to have duplicates. TileDB supports fast and efficient updates (and duplicates) so you have full control of your design and implementation.

[1] https://news.ycombinator.com/item?id=23897086

[2] https://github.com/TileDB-Inc/TileDB-VCF

[3] https://docs.tiledb.com/main/performance-tips/choosing-tilin...

Shelnutt2··on TileDB closes $15M Series A for universal data engine
Seth from TileDB here.

The foundational invention is the TileDB universal storage engine based on dense and sparse multi-dimensional arrays. Genomic variants are 2D sparse arrays. Images and video are dense arrays. KV are interestingly sparse arrays as well[1]. TileDB is the first solution that can efficiently model all data with a single powerful data structure. TileDB universal storage engine natively supports cloud object stores, as well as local filesystems and we natively handle the eventual consistency through our MVCC design.

On top of the TileDB storage engine we have built and worked on integrations with a range of existing computation tools and frameworks. Our goal is to make the computation pluggable and provide fast and efficient (zero-copy or apache arrow where possible) integrations to allow you to continue to use your favorite computation tools in your favorite language. A few examples are our integrations with numpy/Pandas, Spark, MariaDB, gdal and more.

In addition, we built TileDB Cloud on our storage engine, which has two more innovations: (1) easy data sharing and logging at global scale (beyond organizations), and (2) a complete serverless infrastructure for scaling out compute, very similar to the DAG approach of dask.delayed.

[1] https://docs.tiledb.com/main/handling-key-value-stores

[2] https://docs.tiledb.com/cloud/client-api/task-graphs

Shelnutt2··on Recently minted database technologies that I find intriguing
You can certainly store ML models in TileDB. I've been working on an example notebook showing how to store an opencv model in TileDB, its not quite finished yet but will be published soon.

Using python as an example language, TileDB-Py offers simple functions, like "from_numpy" [1] for basic and simple models which we can write directly. For more advanced (and more common) use cases of complex models, its easy to create an array to store the model and all associated metadata. TileDB's array metadata and time travel capabilities allow for you to even see how your model changes over time as you update it.

[1] https://github.com/TileDB-Inc/TileDB-Py/blob/bcaee16b194675f...

Disclosure: I'm a member of the TileDB team.

Shelnutt2··on Generate Go model from database design
I've never heard of dbml before, seems like its been out for about a year. I'm always interested in managing schemas in a db-agnostic fashion. Seems like overtime everything always devolves into db specific schemas (spatial indexes, new datatypes like JSON, etc). I'll read into this and add it to my list to watch.

Thanks for sharing this and your work.

Shelnutt2··on TileDB 2.0 and the Future of Data Science
Please see my response here:

https://news.ycombinator.com/item?id=23117117

Shelnutt2··on TileDB 2.0 and the Future of Data Science
Disclosure: I am a member of the TileDB team.

Several important concepts are being conflated here, so I'll elaborate on each separately.

S3 eventual consistency: TileDB is fully aware and designed around the eventual consistency guarantees of cloud object stores [1]. When an array is opened, only the committed writes (up to the specified timestamp if using time traveling) are seen by the reader (each write produces a timestamped "fragment", which is essentially a folder). There will never be partial reads, or corrupt reads. The array is always in a readable state with committed data. This works very similar to Iceberg's method of opening a table at a snapshot for reads [2]. I don't believe that TileDB is any more affected by stall reads than Iceberg (or Delta lake), the user must reopen or re-query a table to see data from a newer timestamp/snapshot.

Handling S3 write failures: TileDB is designed for a lock-free, multi-writer scenario. All writes produce a new timestamped fragment, which is immutable after completion of the write. TileDB performs an atomic write of an "ok" object, which signals when the fragment is complete. Any fragment which is missing the "ok" file, is ignored by the reader [3]. TileDB handles corrupt or incomplete fragments by erroring out on the read. In the future, we could offer a retry mechanism for failed reads, but the important thing is we will never return corrupt or invalid results. Incomplete fragments can happen because of S3's eventual consistency; it is possible for the atomic ok file to show up before an object in the fragment "folder".

Write serializability: TileDB's fragments are written at a (timestamp + uuid) fashion. In the event of a conflicting write at the same timestamp, the uuid provides uniqueness and guarantees that there are no errors by essentially randomly ordering the conflicting fragments. The effect in the end is similar to Iceberg's cancel and retry conflicting writes, except TileDB does not have the penalty of retrying. For Iceberg, if there are two simultaneous conflicting writes, it is effectively random which one would be accepted and which one would be retried.

ACID: TileDB does not support ACID intentionally, as it was not designed to be a transactional database. TileDB was designed with a lock-free multi-writer/multi-reader model, as our use cases up until now involved a one-off massive parallel write, and then multiple concurrent reads. Lack of transactions does not yield inconsistent data though, as stated above the read/write algorithms are specifically designed with this in mind. That said, we have recognized that there are workloads where transactions are important, and we have plans to eventually add transactions to our cloud product where we have the orchestration layer needed to manage them. It is important to keep the transactional layer modular and format-agnostic (Delta Lake is blending it with Parquet) and, therefore, we will build it on top of TileDB, not inside the storage engine.

Interoperability: Obviously nothing stops someone from writing a pandas integration for Iceberg or Delta Lake. An important part of our philosophy at TileDB is extreme integration with existing tools and frameworks, which is why we strive to support things like returning numpy arrays for the Python results and doing zero-copying everywhere possible. In other words, it is not trivial to add a tool integration, just the same as it is not trivial to just add an ACID layer.

[1] https://docs.tiledb.com/main/basic-concepts/consistency

[2] https://iceberg.apache.org/reliability/

[3] https://docs.tiledb.com/main/basic-concepts/physical-storage

Shelnutt2··on TileDB 2.0 and the Future of Data Science
Disclosure: I am a member of the TileDB team.

Iceberg, similar to Delta Lake, is another layer you have to add on top of Parquet, which limits your mobility of compute. You can currently use it either with Spark or Presto, but not directly via language APIs or other compute engines (e.g., pandas, Dask, MariaDB, etc). For example, the pandas "read_parquet" will not be able to read your dataframe, you will have to use pyspark or presto + sql query to fetch the data into pandas. TileDB has updates and data versioning built into its format and storage engine, therefore reading an array directly via a TileDB language API, or Spark, Dask, PrestoDB or MariaDB will have the same behavior. You will be able to see the dataframe after all the changes, or time travel and "activate" only some of the changes in time.

Shelnutt2··on TileDB 2.0 and the Future of Data Science
Disclosure: I am a member of the TileDB team.

TileDB fully supports multi-writer/multi-reader, so scaling compute to multiple nodes is mostly about choosing the computation layer. We have integrations with Spark, Dask and PrestoDB for distributed compute. In our cloud product we also offer serverless UDFs and SQL (via MariaDB) to allow for scaling out computations elastically without managing your own clusters.

On the postgres side, we are looking at eventually adding a postgres storage engine, now that the new storage support is in PostgresSQL 11. To start with we have built a storage engine for MariaDB, as MariaDB has an excellent storage engine API, which I have past experience with. We hope to upstream the storage engine to MariaDB after the MariaDB 10.5 release.

MariaDB integration: https://docs.tiledb.com/mariadb/

Spark integration: https://docs.tiledb.com/spark/

Dask integration: https://docs.tiledb.com/dask/

PrestoDB integration: https://docs.tiledb.com/prestodb/

Cloud Docs: https://docs.tiledb.com/cloud/

Shelnutt2··on TileDB 2.0 and the Future of Data Science
Disclosure: I am a member of the TileDB team.

Yes you are correct, TileDB introduces a storage format, but is also a full-fledged storage engine that implements it. Having the complete storage engine is what allows us to make accessing the data transparent regardless of the API or application you use to access the data. We are able to integrate into different computational tools without having to reimplement or sacrifice features.

Shelnutt2··on TileDB 2.0 and the Future of Data Science
Disclosure: I am a member of the TileDB team.

TileDB is designed primarily for persistent storage, similar to HDF5, Zarr and Parquet. TileDB is a cloud-optimized, dense and sparse multi-dimensional array storage engine, which is broader than Arrow's Dataframe. In addition TileDB also handles updates, time traveling and partitioning at the library level, which are not possible with the Arrow project's current choice of persistent format, Parquet. TileDB removes the need for using extra services like Delta Lake for updates or Hive for cataloging and partitioning, as you get all this functionality in a single, embeddable library.

Shelnutt2··on Clickhouse Local
Another option to consider here is storing your data in TileDB[1] which allows you to access it via Python/Pandas, MariaDB for SQL (embeddable or standalone), Spark, R and more. With embedded MariaDB[2] you can query directly into pandas with minimum overhead. TileDB is similar to parquet in that it can capture sparse dataframe usage, however it is more general in that TileDB supports multi-dimensional datasets and dense arrays. Other major features built directly into TileDB include handling updates, time traveling and partitioning at the library level, removing the need for using extra services like Delta Lake to deal with the numerous Parquet files you may create. TileDB also has native support for S3 and in the next release we'll have native Azure Blob Storage support.

[1] https://github.com/TileDB-Inc/TileDB

[2] https://docs.tiledb.com/developer/api-usage/embedded-sql

Disclosure: I am a member of the TileDB, Inc. team

Shelnutt2··on Loading NumPy arrays from disk: mmap() vs. Zarr/HDF5
TileDB can handle most of your requirements today and we are working on the "row id labels" which we call axes labels[1]. We are looking to get axes labels support completed in the next 2-3 months. TileDB supports fixed size and variable length attributes so you have flexibility in how to handle your 256 byte and UTF-8 fields. On the distributed computing front, TileDB has integrations with Dask[2] and Spark[3]. We are happy to answer any further questions you might have on TileDB or how to adapt it to your use case.

[1] https://feedback.tiledb.com/tiledb-core/p/support-axes-label...

[2] https://docs.tiledb.com/developer/dask/quickstart

[3] https://docs.tiledb.com/developer/spark/quickstart

Disclosure: I am a member of the TileDB team.

Shelnutt2··on Loading NumPy arrays from disk: mmap() vs. Zarr/HDF5
TileDB is more than a format. At its core, it is an engine that allows you to store and access multi-dimensional arrays (dense and sparse) very fast. Similar to Parquet, its sparse array support can capture dataframes[1]. It is more general than Parquet in the sense that it can support fast multi-dimensional slicing, by defining a subset of its columns to act as dimensions. This effectively buys you a primary multi-dimensional index (and in the future we could add secondary indexes as well). Also, TileDB handles updates, time traveling and partitioning at the library level, obviating the need for using extra services like Delta Lake to deal with the numerous Parquet files you may create.

The good news is that we offer efficient integrations with MariaDB, PrestoDB and Spark, so you can directly process SQL queries on TileDB data via those engines (which work even for dense data). With MariaDB, we even have a embedded version which allows running SQL queries directly from Python[2]. This combines the ease of use of sqlite with MariaDB's speed and TileDB's fast access to AWS S3 (and Azure Blob Store in the next version).

[1] https://docs.tiledb.com/main/use-cases/dataframes

[2] https://docs.tiledb.com/developer/api-usage/embedded-sql

Disclosure: I am a member of the TileDB team.

Shelnutt2··on Five methods for Filtering data with multiple conditions in Python
TileDB[1] offers an embedded SQL experience for python[2]. We use MariaDB built in embedded mode to allow running sql against TileDB Arrays. This can be combined with pandas.read_sql to load the results directly into a pandas array. I wrote this implementation for the embedded SQL, so happy to answer any questions on it. The embedded SQL in its early stages, but should be fully functional for any queries that MariaDB itself supports. TileDB and the embedded SQL are both open source, TileDB is MIT licensed and the embedded SQL is under GPL.

[1] https://tiledb.com/

[2] https://docs.tiledb.com/developer/api-usage/embedded-sql

(disclosure: TileDB, inc. employee)

Shelnutt2··on FTC Approves Roughly $5B Facebook Settlement
There is also the fact that the social security card is a flimsy piece of paper, not plastic or other more durable material. Even just getting social security cards on plastic would be an improvement for something you are expected to keep and present for your entire life.
Shelnutt2··on The death watch for the X Window System has probably started
There is an override flag, its linked in the code snippet.

Use "--unsupported-gpu" or "--my-next-gpu-wont-be-nvidia" to bypass the check and allow sway to launch.

Shelnutt2··on Trump declares emergency over IT threats
Samsung is Korean, Alcatel-Lucent was French but now is part of Nokia. Nortel was acquired by Ericsson.

Cisco makes some backbone equipment. They were trying hard to sell a LTE "virtualized" core when I previously worked at $wireless_telco, but cisco does not make RAN components.

The only US based company I know of that is has licensed products in production use by a major telcom is Airspan [1]

[1] https://www.airspan.com/

Shelnutt2··on ZombieLoad: Cross Privilege-Boundary Data Leakage on Intel CPUs
If you don't need a dedicated GPU, the APU offers from AMD are great. They have native linux drivers for everything (on the AMD side, double check the nic/touchscreen/touchpad). I'm using an HP envy x360 15z with a AMD Ryzen 2700u running gentoo and love it. The HP envy has a weird keyboard, but it was a good tradeoff for the AMD setup when I bought it last year.

There is a much larger market in 2019 for AMD laptops, so you should be able to find something to suite your needs.

Shelnutt2··on Google Is Turning Off the Works-with-Nest API
Second, my ecobee3 is great.
Shelnutt2··on Show HN: Mockit – Open-source app to create and configure HTTP mocked endpoints
Tools looks great! I've added an issue here for openapi: https://github.com/boyney123/mockit/issues/5 . I have quite a few specifications in openapi v2 (swagger). It'd be helpful to support both v2 and v3 (as op requested) of the OpenAPI format.

Thanks!

Shelnutt2··on Notre Dame on Fire
dup: https://news.ycombinator.com/item?id=19666991
Shelnutt2··on Ask HN: Who is hiring? (April 2019)
TileDB, Inc. | Full-Time | REMOTE | Cambridge, USA | Athens, Greece | https://tiledb.io

TileDB is a disruptive technology for storing and managing enormous volumes of structured data. TileDB efficiently stores data generated from variety of domains (genomics, finance, imaging, geospatial, etc) in a novel unified format as sparse or dense multidimensional arrays. Users can efficiently access this data via a growing number of language APIs, interfaces and our upcoming web service. TileDB, Inc. has raised $4M in seed funding over the past 18 months.

We are looking for frontend web developers to take ownership of the front end and help launch a scalable service to make data storage, management, and computations easier for practicing data scientists. A few features on the roadmap include designing a web admin console for advanced data management, monitoring resources in real time, integration with JupyterLab and handling user profiles and billing.

We are primarily seeking:

  - Senior UI / UX  Engineer (Javascript / Vue.js / Golang / K8s)

* TileDB GitHub organization: https://github.com/TileDB-Inc

* TileDB has been featured on HN: https://news.ycombinator.com/item?id=15547749

* TileDB recent blog post: http://bit.ly/2SjkeYR

* TileDB talk at PyData 2018: http://bit.ly/2BfbyJ3

Our headquarters are located in Cambridge, MA. The candidates must be US citizens or permanent residents located in the US, or Greek citizens to be located at our subsidiary in Athens, Greece.

Apply today at https://tiledb.workable.com !

Shelnutt2··on CopperheadOS Release: Android 9
Rashed has been in the CyanogenMod -> LineageOS community for a while. He has quite a bit of experience from this and from all my interactions with him he is a solid Android Systems Engineer.

I would generally trust Rashed's development but I would still question the company until they prove themselves, which may or may not happen.

Shelnutt2··on Nasa’s Mars Rover Opportunity Concludes a 15-Year Mission
Today's https://xkcd.com/2111/
Shelnutt2··on CERN day 1: rebuilding the first web browser
I believe it says "server".
Shelnutt2··on Red Hat Enterprise Linux 8 Beta
gcc 8.2! Being on gcc 4.8.5 for RHEL 7 instead of 4.9 was a minor pain point for some c++11 or newer libraries when developing. Nice to see them jump all the way to the 8.x series.

Also great to see xfs gain COW support in RHEL, they are working to make up (in features) for dropping btrfs with xfs. We'll see how stratis works out.

Shelnutt2··on TransitMatters, a transit advocacy group in Boston
You don't need a conductor at every door to allow all door embarking/disembarking. Look at metro north, or SunRail. They both allow all door boarding without a conductor at each door. Raised platforms is something that could help to achieve process improvement.
Shelnutt2··on MariaDB acquires Clustrix
First they picked up infinidb a while back and have been working to mainline "mariadb columnstore". Now the acquisition of Clustrix and discussion of mainlining that also. It looks like MySQLs separation of the storage engines is paying off in their ability to keep one interface but allowing significantly different backends to meet the different workload requirements.

There has been a lot of work expanding the storage engine API so columnstore can be mainlined and work so spider can progress. I imagine that progress has showed them that they are likely to be able to integrate a much larger and less connected (to existing mariadb) code base like clusterix. I can only open they decide to open source (BSL license?) the clusterix solution eventually.

Between MyRocks (replacing tokudb), columnstore, spider and eventually clustrix seems that mariadb is trying to making the case that they can handle any size workload being through at it.

Shelnutt2··on Ask HN: Best robust laptop for everyday software development
I recently purchased a HP Envy x360 15z [1] and it might not be the most rugged I feel it is a solid laptop with good build quality. I also like that its one of the few on the market today with a AMD Ryzen APU (I got it with the 2700U). This gives the advantage of great linux support with the AMDGPU drivers and a nice compromise between Intel integrated graphics and some of the dual gpu offerings. The only issue in linux is the touchscreen is currently not functional out of the box. There is a bugzilla entry and patches but they have not been upstreamed yet [2].

[1] https://store.hp.com/us/en/pdp/hp-envy-x360-15z-touch-laptop...

[2] https://bugzilla.kernel.org/show_bug.cgi?id=198715

← PreviousPage 2 of 4Next →