So my understanding is following Netscape's acquisition of DigitalStyle, the decision was made to replace Netscape's existing layout engine with something based on what DigitalStyle had in their editor (which, unlike Netscape's existing one, supported things like reflow), and it was this combination that was open-sourced. Notably, it was only really the layout engine that was replaced: the HTML parser and the JS VM survived unchanged, hence the name NGLayout for what became Gecko.