Cryogenic temperatures make most materials more brittle, hard to get a material that works at a wide enough range of temperatures to make a balloon to work correctly.

If you go for a narrower range of temperatures (ie. not structurally stable above 0C), it would need to be manufactured, transported, stored, tested and installed at seriously low temps which probably negates the possible advantage with the added technical complexity.