Hacker News Reader

Comment by Klathmon | Hacker News Reader

I really enjoy this article. It's nice to see one thats not just bashing the format as bad or insulting the developers.

Did Microsoft solve any of these problems in their "newer" file formats (IIRC its something like .docx instead of .doc)? And are those as crazy after a few years or have things gotten better since then?

View on HN

Back in 2007 I was a proponent of the 'No-OOXML' movement [1], which outlined several objections (linked in the left margin on their page), some technical, some political, to the ISO standardization process of the new XML formats.

In my opinion, the new formats are little more than XMLified representations of implementation details of how MS Office works; there is a distinct proliferation of elements that aren't purely semantic, and an awkward-but-inconsistent lack of separation of structure from presentation. But I'm not an expert on the binary formats nor the inner workings of their applications. Though my opinions remain unchanged after 9 years, in retrospect I do like that the new formats are documented by a true standards body, and are interoperable not just by reverse-engineering a closed format.

[1] http://noooxml.wikidot.com/start

The best example on how this "just serialize our program's operations" style of file formats turned out crazy was the WMF exploit. Merely hovering over a malicious wmf file in explorer would trigger the payload. And the bug was literally a GDI method being played back which was "call my callback function".

https://blogs.technet.microsoft.com/markrussinovich/2006/01/...

> Documented by true standard body

Is ECMA that "true", I don't know. At least they document a JavaScript without "JavaScript" in its name. MS Office (2010 at least) isn't compatible with the standard text. Is there even a MS Office nowadays that's better? Also Office 2010, dispite support, is hostile to OpenDocument format - opening a document gives you warnings about legacy format and what not.

The OOXML standatisation process was very controversial and left a bad taste around MS and ECMA Switzerland: https://en.wikipedia.org/wiki/Standardization_of_Office_Open...

ECMA 376 is explicitly intended as documentation of what Office 2007 does. Michael Meeks of LO (then at Novell) worked on it and says it did this job: https://people.gnome.org/~michael/blog/2014-02-26-cabinet-of...

But in practice there's no such thing as "OOXML" - there's only what Office 2007, 2008, 2010, 2011, 2013 and 2016 happen to do. So if you want to work with the stuff, you verify everything. The standard is, like so much Microsoft documentation, best treated as based-on-a-true-story fiction with an unreliable narrator.

But the old formats are also (mostly) documented by Microsoft. Only once did I have to reverse-engineer a magic number, and only to get something working with Excel 97:

http://phpexcel.codeplex.com/discussions/71551

For the most part, with Word I get better compatibility in LO with the old binary formats than with OOXML. If someone sends me .docx and it looks garbled, it's easier to ask them to resave as .doc than to try to get them to produce .odt

If you have the time, you should consider filing a bug report.

I once suggested getting rid of the name and the ISO "standard" but keeping the file format itself in a poorly written blog article.

.docx is a much nicer format than doc.

It is literally a zip file containing a some xml files [0]. Having said that, the spec is still 106 pages, but the entire xml schema is only 8 pages; the rest is exposistion.

[0] Some features (such as embedding images) probably involve including non-xml files.

[1] https://msdn.microsoft.com/en-us/library/dd773189(v=office.1...

You only linked to the "extensions" MS added on top of OOXML, whose specs are in total an order of magnitude bigger. ISO/IEC-29500 or ECMA-376 is the actual spec, and its first part (of 5) is 5000+ pages. In comparison, 349 pages for the main spec + a 9 page auxillary spec for the document "filesystem" seems almost trivial.

http://activityworkshop.net/rants/ooxml.html

The PDF spec is 1000 pages. Nobody seems to complain. Or at least they complain a lot less.

Fortunately the only people who can read 1000-page documents are those who already have working PDF renderers, so the problem is self-limiting.

You can ignore about 800 pages of that for most applications though.

Not really, when you start to implement it, the more and more functionality required as you get more samples.

Much nicer? Not really. It literally a transpilation of the binary format. The XML files a very abnormal for everyone but who are familar how the original older doc/xls/ppt fileformat works. E.g. Look at how the Bold-tag is implemented, no sane developer would implement it that way. It's clearly done to mske it as straight forward port of the old format and make it as difficult as possible for every one else. The XML even contains binary blobs and all. A low-light. Every other modern Office format is nicer.

https://en.wikipedia.org/wiki/Comparison_of_Office_Open_XML_...

https://en.wikipedia.org/wiki/Standardization_of_Office_Open...

Worse, it's designed in a way that works completely differently from every other XML office format to allow Microsoft to patent it. Doing document markup the normal way in XML has too much prior art to be patentable (though that didn't stop Microsoft from trying), but they managed to get a patent on their oddball way of doing things.

I wrote an open-source reader for docx, including for math equations, in a week or two of part-time hobby work. The spec isn't always sensible (though I'm sure there are historical reasons for most of the decisions) but it's always easy enough to look at the xml and figure it out. Most importantly, your language only requires a zip lib and an xml parsing lib. So, yeah, I'd say it's much nicer to deal with than what this article describes.

My understanding is that the .docx format is basically just the .doc format, transcribed to XML, and with a .zip filesystem instead of an OLE Compound Document filesystem. In fact, I recall some Office-competitor developers saying they preferred .doc, because (like Joel says) they could just read it into RAM and use it, instead of having to deserialize XML.

In practice there are assorted binary or encoded lumps.

More fun: you know what happens if you save a .DOC in Office 2007/2010/2013? Any features past 2003 are saved as a lump of OOXML! http://vmiklos.hu/blog/doctok.html#comment-header

There are some parts of it, like VB macros, that are still stored in the old format before being put in the zip file.

It's nice to see one thats not just bashing the format as bad or insulting the developers.

You know the author was part of Excel team, don't you?

The article seems like an explanation, not a justification. Moreover, I totally agree with the conclusion: use OLE to extract information programatically.

But the format is still terrible :-) The fact that there were reasons to do it that way at the time is not the same as saying the decisions were the right ones. Probably there were reasons to do <insert here any despicable historical or technical catastrophe>.

I didn't know that the author was on the Excel team, but it doesn't really change my feelings.

I'd say that they were the right thing to do at the time, but in the long-term they turned out to be less than ideal.

From Microsoft's perspective, the formats allowed them to be faster than the alternatives, seemed easy to maintain internally from the sound of it, and they were able to cut and run from the format when needed.

If they decided to go with a format that would be "good" for the next 30 years, would they have been as competitive back then?

As with many things, time has a way of changing our perspectives. No, Office formats would not have been competitive on earlier machines if they used a lexer/parser. Office documents weren't really documents in as much that they were memory snapshots. This is why early Office products had such high productivity over other competitive formats. If they were being written by scratch today, the formats would be very different, but then they wouldn't be as feature rich as their predecessors.

One thing overlooked is that the transition to the new DOCX formats (and PPTX, XSLX, etc.) was that they were being written about the same time Windows was working on Longhorn. Longhorn featured WinFS. WinFS would open the XML document formats and save assets like embedded images as their own components in the filesystem and then would reassemble the documents as you moved them from a WinFS store back to a traditional filesystem. The XML formats were in a similar way designed to make that process fast, just as the binary images did for earlier versions of Windows. The XML document formats were designed to integrate with a filesystem which never shipped. Without WinFS, the formats weren't built for third party interoperability, but were designed to make it easy to decompose a document into block components which the Office apps otherwise handled like verbose but compressed binary formats. If WinFS had shipped, the OOXML formats might not seem as convoluted.

"the right ones"...what would the "rights ones" have been besides shipping functioning software for their users and making money for their shareholders?

Functioning and right are not the same thing. And right for the shareholders is a strange way of defining what's right. Strange for me anyway.

That's kind of my point. On this forum we tend to define "right" by a certain standard of engineering aesthetics. I say a certain standard of aesthetics because it definitely isn't the only one.

Other definitions of "right" could reasonably include "our software performed the intended functions expected by the people that paid for it."

One common definition of "right" in capitalist countries is the fiduciary responsibility to maximize profit. It's not one that I personally think works in every scenario. That version of "right" can often find it's self in direct contradiction of definitions from moral, social, and legal areas.

My only point was that "right" means a very particular thing on HN. It means Unix-y...but not actually Unix...its this Unix that inhabits someone's mind. Where X and sockets are also "files" and text is the only RPC system you would ever need. It's also functional programming fetishistic. LISP because of the great PG. Erlang because I heard of this one really awesome app with great up time...and have you heard about Plan 9, it's perfect if it weren't for {insert dark force of mediocracy} that kept it from being a winner.

Thanks for asking. I've been needing to say that to someone for a long time.

I think it is better than the older binary formats, being based on XML.

And are those as crazy after a few years or have things gotten better since then?

If anything, I'd say they're worse.

https://en.wikipedia.org/wiki/Office_Open_XML

XML'ized everything (yuck!), and instead of "just" Microsoft having control over the format, it's now a whole design-by-committee bureaucratically bloated mess with other parties trying to stuff things in too.

XML is a perfectly reasonable format for documents.

Perhaps for documents which are intended to be manually edited in a text editor, which is hardly the use-case here. Otherwise it's just extremely inefficient because of all the parsing involved. You can't know when one element ends or another begins, or skip through the parts you're not interested in, without reading every single byte and parsing it. Your XML parser has to handle extra whitespace (which will almost never be present), various quoting options, and arbitrary attributes/tags; all "decision points" within the code that add complexity. With a TLV format like the Excel binary one, you can seek through the records without knowing what they contain just by reading the header and adding its length field to the current offset. Then when you get to the record you want (identified by a simple integer comparison), just do a length check and read into the appropriate structure. There's far less to do, far less decision points, and far less to go wrong.

Except, of course, if malicious files are supplied or they are corrupted somehow.

Regardless of XML-ness, you still have to check that the values are valid. With XML, you also have to check whether they're of the right type, because the format is so much more flexible.

You have to check if they are the right type if they are binary files as well.

It is far easier to repair and recover an xml file than a binary format.

except you have to move the rest of the data when you edit the document or you can fragment the data but the old data will still be on memory / memory mapped file / file.

Reply on news.ycombinator.com