For that very reason I tried selfhosting Actual Finance[1] but it is more of a budgeting app than a networth tracking app.
I ended up coding a small exporter[2] since I already had some stack in place that queries SimpleFI[3], which essentially allows querying balance and transaction information for most US-based banks (read only); most similar to plaid but a lot more developer-friendly afaik.