"Do you really want to hinge your case on Congress' ability to delegate power?"
At some point, yes. Your other arguments are basically handwaved into existence.
Don't get me wrong, some of them even have precedent for that handwaving, but i'm always in favor of arguing based on first principles than "well, you guys have fucked this up for 50 years, so no point in changing this now"
"I mean that COULD be a fun court case if you want to have it and make it a constitutional issue, because the whole executive branch, and the executive branches of every single state, would be ripped to shreds :D"
You know, of course, that the whole point of having different branches was ... to have different branches :)
"the above argument has a thin chance of surviving in 2017
"
Probably, but like most other arguments based on judicially created rights, i think others are even more tenuous over time, and are just kicking the can down the road.
See right to privacy, etc.
At some point, it needs to be grounded in the actual text of the constitution, or it just erodes. There is nothing in the constitution granting green card/etc holders any rights directly at all.