If you at least have a kernel source for the SoC, it's doable. However, especially if you don't have the source for your specific device (because manufacturers play fast and loose with the GPL), you'll probably spend the first few days getting it working and then many, many weekends getting it to be usable. I did stop before I got to that point.
Here's some of the issues I ran into:
* The parallel RGB to MIPI bridge doesn't have complete publicly available documentation and it requires LCD-specific configuration. I've sniffed it from the Android kernel with a logic analyzer.
* The touchscreen controller doesn't have complete publicly available documentation and it requires device-specific configuration.
* The BMS doesn't have complete publicly available documentation and it requires device-specific configuration
I guess you can see a trend. :) If you have the time, you can RE this stuff with some basic soldering skills and a logic analyzer or scope.
But actually the issues which made me lose enthusiasm were the following:
* Rotating the framebuffer to landscape orientation in software is freaking slow. Android seems to be doing it using OpenGL ES, but I'm not aware of any GNU/Linux window manager capable of this. The display controller didn't support rotation and the only remaining alternative would have been too hacky for my taste, to use the image processing engine to maintain a rotated shadow copy.
* Limited capabilities: processing speed is mostly OK, but only 2 GB of RAM and the slow framebuffer are not great. There's almost no I/O, essentially you only have a USB OTG port. Tablets are not thermally designed to sustain high loads. Battery life will probably not be that great and it seems difficult to upgrade the battery.
Now, some of this stuff doesn't fully apply to Nexus 9. I expect Google to be better at providing a kernel source closer to that's running on the device and the hardware itself it much more capable, it even has OpenGL (non-ES) support. This still leaves quite a few issues. Chromebooks (used for running GNU/Linux, of course) might be worth looking into.