> SQL vector database
Cool, but does it actually return the correct results for these SQL statements, especially when ORDER BY is concerned?
I.e. does it somehow have a way to get a recall of 100% from its indexes?
Cool, but does it actually return the correct results for these SQL statements, especially when ORDER BY is concerned?
I.e. does it somehow have a way to get a recall of 100% from its indexes?
https://clickhouse.com/docs/en/optimize/sparse-primary-index...
Considering that embedding vectors represent a lossy compression of the original text or images, is achieving a 100% recall necessary? I am interested in understanding its practical implications.
Disclaimer: I am an employee at MyScale.
For the app, maybe not. But as a database absolutist, I think you must be able to dump all rows of a table with
WITH
limit_result AS (SELECT *, {similarity} AS metric FROM table ORDER BY {similarity} ASC LIMIT 10),
dist AS (SELECT MAX(metric) AS max_m FROM limit_result)
SELECT *, {similarity} AS metric FROM table, dist WHERE {similarity} > dist.max_m
UNION ALL
SELECT * FROM limit_result
A recall of <100% may skip some rows in the limit_result, which then also won't show up in the main table's scan result, thus potentially corrupting a data dump process that uses sorted output.