This this is the approach that Stephen Wolfram's physics project is exploring. In their model, everything is made of space, and space is a hypergraph.
There are several hints in more "standard" physics that a possible TOE could be based a model where you can think of the "stuff" of the universe more like a crystal with defects.