fforflo

Good catch, the whole PDF parsing ecosystem is kinda grey on those things. I tried to be extra-careful and I relied on poppler's official statement:

"Note that Poppler is licensed under the GPL, not the LGPL, so programs which call Poppler must be licensed under the GPL as well. See the section History and GPL licensing for more information."

See https://gitlab.freedesktop.org/poppler/poppler/-/blob/master...

I'm missing something I think. Currently the `pdf` type is indeed a blob of bytes. Read from disk, but not parsed. The type is created with storage=extended to avoid the typical 8KB tuple size limit.

In practice though, a PDF for most cases has text-like semantics, so with ::pdf::text you can have all the text-indexes you want.

Interesting: I posted this a few days back, and certainly not "10 hours ago". Who was kind enough to re-surface this? Thanks :)

Some clarifications on a few comments I see downstream:

The motivating example was to easily support Full-Text Search (FTS) on PDFs with SQL only (see blog post https://tselai.com/full-text-search-pdf-postgres ). You can treat `pdf` as an alias for `text` and do everything possible.

On the next iteration, I made `pdf` a type (typical varlena object of bytes) to avoid hitting disk all the time. The file is loaded from the disk only once (if it's a valid pdf). One can store the `pdf` type (blob of bytes) as a standard Postgres type. And use that for subsequent calls. Postgres will do it's magic as usual. There is a potential next step of storing the parsed document just to save some time from re-parsing the bytes, but I deemed it a premature optimization.

The first iteration of this was actually FTS only https://tselai.com/full-text-search-pdf-postgres

In the second pass I made `pdf` a type.

Whenever I find myself needing /tmp, it's usually as a form of IPC shared memory, in which case I use /dev/shm directly.

I have a couple of read-heavy >2TB Postgres instances, document-oriented too. You're right that bulk updates can be too slow. Too many times I end up doing the updates incremental (in batches) or even use COPY.

Yeah, I've been keeping an eye on the pgsql-hackers discussions. Alexander+team are doing great work.

OrioleDB was supposed to tackle this problem with a new storage engine . https://github.com/orioledb/orioledb

I'll write a follow up about this, But the tl;dr is that you'll have to use more generous params when running the model.

There's also the problem of input data quality which isn't really great

I can see myself using this as alternative to foreign data wrappers and/or pg_dump even.

Well, we know we'll have reached AGI when LLM says "this chat has been marked as duplicate"

Of course.

Metoo kind of... Write right-handed, fork on the right, but knife on the left. Throw with the left hand, kick with the left. Mouse on the right. Also wished the gas pedal was on the left :D

Looked nice so I automatically did git clone && cd postgres_dba && make all install installcheck... and failed :D

Creatures of habit... any plans to make this an extension?

I guess you mean the .gguf examples, not Llamafile, right? What command did you use? Maybe you're using an old llama.cpp ?

Paraphrasing Spencer:

Those who don't understand SQL are doomed to reinvent it, poorly

For starters, I didn't choose the name, but rather those who trained it. Second, what's the point of training lang-specific LLMs and giving it a name using a word nobody uses in practice?

Unless you want to make it sound more human; a pretentious know-it-all human that is.

I love the idea of pushing JQ and other DSLs close to the database. I've written jq extensions for SQLite [0] and Postgres [1], but my approach involves basically embedding=pushing the jq compiler into the db. So you can do `select jq(json, jqprogram)` as an alternative to jsonpath.

Trying to understand: Is the main purpose of this to use jq-syntax for cataloging-like functionality and/or cross-query? I mean it's quite a few lines of code, but you inspect the database catalogs and offer a layer on top of that? I mean, how much data is actually leaving the database?

[0] https://github.com/Florents-Tselai/liteJQ [1] https://github.com/Florents-Tselai/pgJQ

Something I don't see being mentioned in the comments: What's a really frustrating part of working with the C API? Setting up the compile/link flags!

The python3-config works generally well but is only available at the OS level. But you don't want to mess with that (e.g., to access pip-installed packages). Beyond that, everything is a mess! python3 -m venv doesn't even bother creating such a script. anaconda/miniconda? Don't even try!.

So every package pollutes their build scripts with many hardcoded `python3 -c "import sys: print..."` calls.

I've opened a CPython/PR that may help a bit by adding `python3 -m sysconfig --json` flag [0]

[0] https://github.com/python/cpython/pull/123318

What is the go-to place to browse/find catalogued satellite image data?

While we're here: what happened to the GitHub explore newsletter? I really enjoyed this, but I've stopped receiving it for a few months now. And I don't think I unsubscribed.

The C extensions API is a big big very big thing. As someone who routinely write small PG extensions, I'd love to be able to kinda use the same code for multiple backend DBs. And I guess lots of inspiration has come from all the efforts that embed DuckDb in Postgres as a miniOLAP

Ah, was planning to post this as a Show HN in a few days (marketing material needs some polishing) but lmk if anyone tried installing/using this and is having trouble.

I did some mentorships there (as a mentor) a few years back. My experience was "meh." Most of the mentees were last-minute students looking for help with their assignments.

But their SEO must be excellent. I still get web traffic and referrals from there.

ChatGPT says

The documentary he's referring to is likely "Farewell, Etaoin Shrdlu," released in 1980. It chronicles the last day of hot metal typesetting at The New York Times before they transitioned to newer technology. The title comes from the nonsense phrase "etaoin shrdlu," which appeared frequently in Linotype machine errors due to the way the keys were arranged. The documentary provides a fascinating look at the end of an era in newspaper production.

I'd suggest bringing more code snippets from the test cases to documentation as examples.

Nice work though.

I got a feeling that this may be a cultural aspect there too: Backgammon is not considered "sophisticated," I think. I'm from Greece (I don't play backgammon myself, though), and backgammon (tavli) is extremely popular but widely considered "a game for the masses." It is played mostly in tavernas and old-style coffee shops, and each game has a social aspect, too. People discussing (sometimes shouting and finger-pointing "no! You should have done this or that! Ah! just get up and let me play, you're a noob!"

I can't but laugh thinking who would be the GMHikaru or GothamChess of tavli :D :D

Edit: This can be misinterpreted. I don't play tavli not because is not considered sophisticated, I just find the rules confusing for certain setups.

Yes, you're right. I felt the gif wasn't polished but was too lazy to re-run. I removed it for now. Simplified the README a bit too. Hope is better now.

Thanks!

Well it's not:D Sure there are 4-5 fundamental classes in python libs but they're just fundamental ones. They don't impose an OOP approach all the way.

What you're alluding to is people coming from Java to Python in 2010+ and having a use-classes-for-everything approach.

OOP is Java, and Java is OOP, right?

My point is to follow a dogmatic OOP approach (think all the nouns like Agent, Prompt, etc.) to model something rather sequential.