I am building a similar app as a hobby, and have done some research on this. I am also very interested in avoiding Plaid, mostly because they are very expensive, and require a signed contract with them for production use (which I don't want to do under my own name, and forming a LLC is also expensive).
Without Plaid, you only have three options: connect to bank yourself with OFX, let user upload OFX file, or let user upload CSV.
OFX stands for Open Financial Exchange, an open standard to programmatically access and transmit financial data - which is basically Plaid, but provided by the banks themselves, which as you can imagine is 1000x worse than Plaid.
Connecting to bank using OFX is really annoying. First, if you want to do it without storing the user's credential, you need to run the app in the user's computer, so forget about auto update in the background. Second, I don't know a single bank that publishes their OFX connection parameters, so people rely on crowdsourced data, which is iffy at best and totally unreliable at worst. Finally, OFX is a disastrous format to parse. It uses SGML (!!!) and require DTD to parse correctly (!!!!). Furthermore, banks' implementation are, as you can imagine, widely inconsistent. For example, each transaction is supposed to have a unique identifier. However, HSBC decided to reuse them, so you can't completely rely on it to dedupe your input.
To be fair, OFX has improved, and the latest revision abandoned SGML in favor of XML and introduced OAuth for authentication. But few bank support them, and among those that do, the API is still not public, so you probably need to talk to the bank's BD people to use them. For example, Chase says "Access to the Developer Ecosystem is currently by invitation only and limited to developers and businesses that have a relationship with Chase."
Most banks support downloading OFX file yourself, so you can just let user upload them manually. This is obviously a huge hassle for the users, and you have the same problem with parsing OFX format.
For the rare cases where OFX is not available, you can count on CSV. However, since there's no standard, you can't automate the import, and must require user input to annotate the columns, which further increase user friction.
In conclusion, in today's landscape, Plaid is a necessary evil if you want to have a seamless user experience. Anything that preserves user privacy will result in a 10x worse UX.