In 2011, the bootstrapped startup I was at built the first COTS trial site electronic document exchange portal. Up until then, trial teams would use a mish mash of shipping paper, fax, FTP, E-mail, etc.
Our product was a finalist for the Transcelerate Shared Investigator Portal (SIP) project to unify this process on one platform.
I saw the internal workings of IT teams in many large pharmas and fought to modernize the internal workings of our own team.
I say all this as preface. The industry as a whole is highly political, incredibly conservative when it comes to tech, and has high barriers to innovation.
I use this term "political" broadly to cover an array of issues I observed. One of which is that there's a lot of jostling for influence because that's how consultants and vendors make money. A lot of folks with influence are long-time industry veterans with domain expertise, but no technical prowess paired with it. There are a lot of parallels to politics in that highly experienced individuals who carry weight in the industry will leave and form/join a consultancy (a la lobbying) and use that to extract rent. It is then in their interest that processes remain cumbersome and unwieldy so they can come in with their expertise and "streamline" it. Certainly, if there were technical standards and standard processes and open tools, lots of vendors and highly paid consultants would be out of a paycheck!
In the end of the Transcelerate SIP selection process, Veeva submitted a non-existent product (at the time) that they never delivered to SIP and ultimately Cognizant (the consultancy running the selection) built their own because of course! Millions of dollars in recurring revenue were on the line!
Lots of good ideas have come and gone because the entrenched interests in this space extract rent from the friction. One working group I was a part of was iterating a spec as a spreadsheet instead of a technical format like a standard API...
One point I particularly disagree with the OP is CT.gov. It is fine, IMO, because it is meant for industry and not laypersons. It has an excellent API[0] and there are plenty of third parties that build consumer-facing apps for trial matching. For example, I built a really basic one using LLMs to match natural language criteria to clinical trials and emails you when a matching trial pops up in the change feed[1]. The CTTI working group also provides a Postgres dump of the change feed as a consolidated database[2] (I write about how to use it here: https://charliedigital.com/2021/05/24/the-best-worst-kept-se...). The main problem with CT.gov right now is that it lacks a bit in data quality standards. There's a lot of variation in completeness of data dependent on country of origin and even between companies.
The lack of technical standards in the industry really hurts innovation. For example, clinical trial protocols should arguably be a standard technical template. The FDA has done some work in this area[2], but industry has largely not adopted it. You can find public protocols attached to some trials on CT.gov, but every company has a different structure. Yet it is this document that underpins the design and execution of the trial (on top of being hard for trial sites to adapt between different sponsors).
The source of change may ultimately originate from private equity. The trial site side is seeing a consolidation with sites being acquired by PE because they realize they can extract rent[4]. One positive outcome is likely more standardization and technology solutions installed by PE to actually streamline ops in an attempt to take on more clinical trials. Centralization and separation of many of the functions in a clinical trial in this type of structure may also yield improvements on the execution side.
[0] https://clinicaltrials.gov/data-api/api#extapi
[1] https://zeeq.ai
[2] https://aact.ctti-clinicaltrials.org/
[3] https://osp.od.nih.gov/policies/clinical-research#tab1/
[4] https://www.fiercebiotech.com/cro/private-equity-invests-tri...