> and let them run wild.

Yep, that's the most worrying part. For now, at least.

> The moment agents start sharing their embeddings

Embedding is just a model-dependent compressed representation of a context window. It's not that different from sharing a compressed and encrypted text.

Sharing add-on networks (LLM adapters) that encapsulate functionality would be more worrying (for locally run models).