Why can't we all use standard libraries for commonly needed algorithms?

If there's a usable "libgpg", I'm unaware of it; the closest thing I know of to it (in C) is GPG/ME, which actually forks and execs GPG.

It would be incredibly useful to have such a Zlib-caliber GPG library.

Reminds me a bit of SQLite in those respects.

Why come up with your own binary filestructure? Just use SQLite!

It's not just the quality of the library, compressing data isn't likely to be tightly coupled to your internal data structures and program logic.

We tried to use something other than zlib once. The application needed to make a copy of a CD-ROM and compress the copy. This was back in the day when a couple gigs was a big hard drive, and 200 MHz was a fast CPU. We first tried zlib but it wasn't fast enough for the client. We looked around, and found another library, written by a graduate student, that was a lot faster. It didn't compress as well, but it was good enough for the client, given the speed. The only problem was it was under GPL, and no way was the client going to open source the application.

We contacted the grad student, and worked out a tentative deal to license the library for around $2k and started working out the details. Then he got all paranoid about Microsoft. Most of our products were Windows products--so what if Microsoft bought us? Would they end up with a Microsoft-wide site license for his library? He wanted to put into the contract terms that would terminate the license if our company changed ownership. Those kind of terms are tricky. Would we lose the license if we went public? What if one of the two major owners (who together owned about 99% of the stock) died and the stock went to their kids?

The CEO (who was one of those two owners) didn't have time to deal with that, said screw it, and told me to find something else.

So I went back to zlib, and put a slider in the ripper settings to control speed vs. compression. The implementation was to simply skip compressing some of the blocks when ripping. The slider controlled what percentage of blocks were compressed.

PS: ~12 years later, we're still owned by the same owner. Microsoft has not ever tried to buy us.

What's wrong with copying zlib around to every project? Well, have you forgotten the 2002 zlib vulnerability where 100s of copies of zlib source had to be fixed up? I believe there were half a dozen of separate copies of zlib source code in the Linux kernel alone (e.g. ipsec, ppp). Tools had to be developed to scan the binaries on your system to determined whether they stealthily used zlib for something and thus were potentially exploitable.

Interesting, indeed.

> But it's odd that someone that feels the need to write about how people should do this hasn't done it in his own project yet.

I'm afraid it's not that odd at all. When I hear people talk about code reuse these days it means something about how they design things, not about whether they actually reuse the code or make it available for reuse by someone else. This detachment from actual reuse has also allowed some drift in the design principles, so that "reusable code" may not be. Sigh.

This is ACM Queue rather than CACM, which is more software-engineering/practitioner oriented rather than CS-oriented.

On the other hand, CACM isn't really that CS-oriented anymore, either: http://cacm.acm.org/magazines/2011/3. Partly it's because "CS" is so big a field that it's not clear what a technical yet completely general CS journal would look like. So it writes more about computing as a field, with a few articles on emerging areas thrown in, while the technical articles go to more specialized venues (like a PLs, systems, or AI journal).

To be fair, Bill, qmail doesn't avoid libraries; it avoid libc. Much of qmail is structured reusably, and, indeed, the core libc-replacement libraries are reused in djbdns as well.

Knuth has argued specifically against reuse in the past. He prefers "re-editable code".

That being said, the sorts of problems concerning the Knuths of this world are decidedly not the sorts of problems I find myself facing every day. I'm happy enough not hacking an HTTP client into shape for every project that needs one, for instance.

Hmm. An interesting observation, but is mutation really the problem here? I'm not sure it is... we are able to transform any program with mutation into an equivalent program where each variable is assigned once upon allocation. (If you don't already know what it is, this transformation is called static single assignment.)

Anyway, it's really hard (undecidable in the general case) to compare two programs and determine whether they do the same thing. Is that what you're proposing?

And even if you could compare programs like that reliably, it seems impractical to apply it to the giant body of today's source code, but maybe you have an idea for doing that?

Can you explain in more detail what you were looking for?

> I once emailed Dr. Bjarne Stroustrup about this; more precisely non-existence of even simple tree algorithms from C++ standard library. He just pointed me to implementation of some sorted list using red/black tree "internally".

What tree algorithms did you need to use?

Also, I suspect "some sorted list" was std::set or std::map.

Reading his pseudo code with its explicit size information ("pad_byte = pad_byte plus32 1;") made me think, of all things, of LLVM's SSA assembler syntax ("pad% = add i32 %pad_byte, 1"). :) Not surprising of course, making the type visible in each operation is typically necessary in assembly.

I have to disagree. For its length and scope it summed up the current problem nicely and clearly pointed out that the thinking that got us into this mess won't get us out of it, to paraphrase Einstein.

I don't think the intent of the article was to answer the question of what to do, but rather to point out its currently intractable nature. Furthermore, he points out that he's even "done something about it," wrote a catch-all library, but that it hasn't had the anticipated or desired effect.

Clearly, something else is afoot here. Where next?

A lot of languages simply take parts of the C standard library, and make it their own. For example, hardly anyone re-implements system signals or processing forking--they just re-use the C library, worts and all.

What problems did you have with FFmpeg libraries?

Assuming you mean the libraries that come with FFmpeg, and not turning ffmpeg.c into a library.

> In addition, why must every language have it's own standard library. Is it possible to have a source code format for standard libraries which can translated into other languages?

It exists. It's called C.

You wouldn't want a standard library of standard widgets because different languages have different approaches to problems. What's suitable for Lisp, isn't suitable for C++, isn't suitable for prolog, isn't suitable for awk.

To be fair, Unicode didn't exist until the 1990s. C was originally invented in the late 60s and early 70s.

The C language and standard library are meant to be minimal. It assumes very little. It barely assumes that Unix/POSIX exist. String.h has to work on server, desktop, console and mobile CPUs as well as microcontrollers. And this is the minimal set of functions that will work almost everywhere. http://en.wikipedia.org/wiki/String.h