The reason people say college is irrelevant is precisely because you can cobble together the curriculum yourself. Obviously there are a lot of complicated benefits and costs both ways that people can debate, but it's not obviously silly.
The support for that position is because of what you said. Namely, if you find yourself needing it later in life you can just look it up. Because you can stitch together the curriculum when needed, doing it all up front isn't necessary. (So goes the argument; but personally I think it's somewhat short-sighted in some ways.)