A thing to think about: often there are many different graphs you can create to represent the same thing. Each graph representation can focus on or hide different aspects of the real system.
Why are you trying to model the building as a graph? What are the use cases? What operations do you want this data structure to be able to perform efficiently?
It might turn out that a single graph (or any graph) is not the most effective way of modelling an approximation of the real system.
How many nodes and edges will your graphs typically have? python & networkx work okay for bashing out prototype code and may be good enough for MVP or even a large number of releases if your data size is small and the operations you perform on the graph are linear in the graph size (e.g. connectivity checks, traversals)
I've also seen C/C++ codebases get pretty far by rolling their own domain specific graph data structures-- eg define your own node and edge struct types, give each node and edge pointers to the edges/nodes they connect to, hack domain specific fields as necessary onto the structs. Then just implement each graph algorithm as you need it. This may end up in an unmaintainable mess after a few years, but I've seen this work well enough so that the product based on this is worth enough money that there's enough cash to hire software engineers to come clean things up!
Another thing to think about: are your graphs dynamic or static? If they are large and static, there's lots in common between graphs and sparse matrices. You can encode your graphs in memory in CSR or CSC like sparse matrix formats-- no objects, just giant arrays full of indices. This isn't a good idea if your want to dynamically add or remove nodes and edges, but it is memory efficient.