If the startup isn't costing you a lot it may be best it leave it running. It'll give you more credibility as if someone says "I Made X and failed" yet there's just an expired domain at X you've basically scrubbed out the past and your work.

There's other sides to this as you won't want an ugly broken site up connected to your name. Case by case basis really.

There isn't a whole lot you could do with the code, i'd just say for most open source it and leave it quietly running, if feasible.