For example, say I ran a command to show disk space usage. Perhaps I would like to pipe the output into a d3 visualisation and see it directly within the terminal without context switching. Why shouldn't I be allowed to do this?

Have you read "The Visual Display of Quantitative Information" by Edward Tufte? Text isn't the best representation of all types of data.

Imagine also that I wanted to represent a table of data within an Excel style spreadsheet. This could be useful as it could define affordances for a user such as the ability to select rows and columns, and to sort or filter in realtime.