The routing protocol is called RPL (pronounced "ripple") and is designed to create a directed acyclic graph to route IPv6 packets in networks where the nodes are severely memory-constrained. It is defined by RFC6550. There is more information in http://www.thingsquare.com/docs/mesh/ and https://tools.ietf.org/html/rfc6550
There is no hard limit to the number of nodes in a RPL network. The protocol is defined so that every node can reach the root of the network, but requires additional work to reach nodes inside the network. The mode of operation that we are using in the Thingsquare system is called storing mode and requires all nodes on a path between two nodes to maintain information about the route. This does not scale to large number of nodes. But this is needed only when setting up a TLS connection, to exchange security secrets, which is normally only done once per node. When the TLS connection goes down, the route is torn down, which allows for another route to take its place.