Need advice on going Open source

Hi HN — I’m building an early-stage B2B SaaS (pre-seed, no customers yet). The product is a company-wide “agentic AI layer”: employees have identity-bound agents that coordinate work, and the system relies on an inter-agent secure communication protocol (authenticated, encrypted messages; policy-scoped data sharing; auditability).

We’re debating going open-source to build trust and adoption, but we’re worried about (a) cloning the SaaS, (b) increasing maintenance burden too early, and (c) undermining the business before PMF.

Concrete questions I’d love advice on:

- If you were in our shoes, would you open-source (1) protocol spec + test vectors, (2) client SDKs, (3) a reference server, or keep it closed until later?

- Where’s the best open/closed boundary so open-source helps adoption without giving away the whole product?

- Any hard-learned lessons on licenses for this situation (permissive vs copyleft vs dual-licensing)?

- What are the “minimum hygiene” requirements before open-sourcing (docs/CI/security policy/contribution process)?

- For security protocols specifically: what would you expect to see published (threat model, audits, test vectors, etc.) to take claims seriously?

If helpful, I can share a high-level architecture description (no proprietary customer data / no secrets). Thanks!