I would suggest a few things:
- figure out some way to do some kind of quasi-official vetting of listings, perhaps as an add-on for sellers. Especially pay attention to pricing, because FSBOs tend to be wildly out of step with fair market value. You might consider refusing the official seal of approval if a property is listed at a price that doesn't reflect well on your platform (we've all seen a FSBO that's priced 50%+ over market; you don't want to endorse that). Platforms like AirBnB offer similar services - for example, they will even send a pro photographer out to take pictures and a top host to help with your copy. Having a way to distinguish listings where a professional has at least looked at the listing to ensure it is sane would be a big value for buyers.
- Many people have pushed back on the notion that folks want to deal with a person. I tend to agree; our last realtor saved us a giant bill on water repairs by making some observations borne of experience. However, the flat percentage model doesn't make sense given how expensive many homes are today. I would suggest having a market of referrals to attorneys (and others?) who can help advise buyers through a transaction. This can allow folks to save a lot of money while feeling supported through the process.
Really hope you're able to make a dent in the existing monopoly!