This is really a "yes and" situation - improvisation and "playing by ear" are great, and have always been part of music (the original "classical" musicians improvised, a tradition we've sadly mostly lost). Improvisation is even more dependent on theory than written music (many folks who "read music" don't actually understand the theory behind it). But being able to read and write is just a super convenient tool, and it addresses use cases that other tools (including recordings) don't.
As with regular writing, it lets you give persistent form and structure to your thoughts. This enables sharing, reviewing, and coordinating in a way categorically different than recordings (books still have value despite the existence of podcasts). This doesn't mean you're "not a musician" if you can't read sheet music, any more than a classical musician who doesn't improvise "isn't a musician" - I'm not trying to gatekeep in any fashion. I'm just saying that both of these dimensions are valuable, and ultimately, complementary.