Nobody is disputing that sense data are necessary. The standard rationalist position is, to quote Leibniz since it's easily accessible on Wikipedia
> The senses, although they are necessary for all our actual knowledge, are not sufficient to give us the whole of it, since the senses never give anything but instances, that is to say particular or individual truths. Now all the instances which confirm a general truth, however numerous they may be, are not sufficient to establish the universal necessity of this same truth, for it does not follow that what happened before will happen in the same way again.
So sense data are necessary but not sufficient, especially for logic. If you believe that empiricism is self-justifying then it does seem like you have to start with the manifold of sense data and somehow build up logic from it. Otherwise you're going to end up #include-ing logic somewhere that may not be obvious, and you've landed to the position you were originally arguing against that you need a reason-based framework to justify empiricism.
A lot of smart people have tried to work out a version of empiricism that attempts to build of logic from sense data. For example, Russell's work and early Wittgenstein. But it always ends up getting pretty crazy.
The easiest way, IMO, to see why it won't work to build up human knowledge from sense data is to realize that the human knowledge system is a distributed system. Empiricists are working with the wrong unit of analysis. The words we have for thought are things like "logic" from logos or "word" and "rational" from ratio or "account" or "reckoning". They're both words that are fundamentally about multiple parties (in CS we can think of them as nodes) doing computation together. Some of the oldest ideas we have about reasoning are dialogs, and that basic idea persists to this day with things like game theoretic semantics. There seems to be something fundamentally "distributed systems" about thought and knowledge.
The individual nodes and their sensors aren't the whole story or even the bulk of the story. That's even more true when you consider that how we interpret sense data depends on billions of years of evolution. Yes we operate on sense data, but we do so using a logical structure that heavily constrains what we see, hear, taste and so forth.
Sense data plays a role, as Leibniz says, just as network cards play an important role in the Google data center. But I don't think anyone would really argue that the value of any large knowledge base is ultimately grounded in network packets.