Benefitting from government spending is not inconsistent with believing in a society free from the government.
I see the government as a parasitic and malevolent organisation who forcefully steals tens of thousands from my profits every year.
This is not inconsistent with getting benefits to reduce my taxation or apply for government grant. The more resources and benefits you can get off from your enemy, the better it is.
His beliefs are inconsistent with working for the government.
I think Peter Thiel should have sided with the libertarian party more than Donald Trump; he probably didn't do it because a libertarian victory is pretty much impossible.
In the case where a libertarian win is impossible, I'd pick republicans vs democrats as well. I don't have any hopes that republicans will reduce spending or reduce the size of the government, but at least they tend to enact redistributive policies less than their left wing counterparts.
I'm not sure if Thiel is an anarcho-capitalist, but something I've found after debating for 20 years is that left wing anarchist and anarcho-capitalists agree on the problem but not on the solution.
Anarcho capitalists think we should have a society without government and with free market capitalism.
Left wing anarchists think we should have a society without government and with individuals that voluntarily share what they have.
Depending on who you are, one explanation will make sense and the other one won't. I invite you to research the subject more and to spend time thinking how your solution could work in practice. The book that finally convinced me that anarcho capitalism might work is The Machinery of Freedom (Friedman).
I personally think that left wing anarchism and communism without a government could work if all the individuals were behaving perfectly.
I also think anarcho-capitalism could work, if we managed to privatise the entire government without creating a power imbalance between armed people and non armed people (creating one or more quasi-governments), because the evil nature of humans would be kept in check by competition in the market and reputation.