They ingest unstructured data, they have a natural query language, and they compress the data down into manageable sizes.
They might hallucinate, but there are mechanisms for dealing with that.
These won't destroy actual systems of record, but they will obsolete quite a lot of ingestion and search tools.
So, no, LLMs aren't going to replace databases. They are going to replace query systems over those databases. Think more along the lines of Deep Research etc, just with internal classified data sources.
If their job was to process incoming data into a structured form I could see them being useful, but holy cow it will be expensive to in realtime run all the garbage they pick up via surveillance through an AI.
...and eventually it'd persist some knowledge in a context window to not make that mistake for a while...
...and then it'd forget and make the same mistake again...