Hacker News Reader

Comment by abnercoimbre | Hacker News Reader

Can someone elaborate? I always treated FILE as opaque, but never imagined people could poke into it?

The MH and nmh mail clients used to directly look into FILE internals. If you look for LINUX_STDIO in this old version of the relevant file you can see the kind of ugliness that resulted:

https://cgit.git.savannah.gnu.org/cgit/nmh.git/tree/sbr/m_ge...

It's basically searching an email file to find the contents of either a given header or the mail body. These days there is no need to go under the hood of libc for this (and this code got ripped out over a decade ago), but back when the mail client was running on elderly VAXen this ate up significant time. Sneaking in and reading directly from the internal stdio buffer lets you avoid copying all the data the way an fread would. The same function also used to have a bit of inline vax assembly for string searching...

The only reason this "works" is that traditionally the FILE struct is declared in a public header so libc can have some of its own functions implemented as macros for speed, and that there was not (when this hack was originally put in in the 1980s) yet much divergence in libc implementations.

In gnulib, there is code that patches FILE internals for various platforms to modify behavior of <stdio.h> functions, or implement new functionality.

https://cgit.git.savannah.gnu.org/cgit/gnulib.git/tree/lib/s...

Yes, it's not a good idea to do this. There are more questionable pieces in gnulib, like closing stdin/stdout/stderr (because fflush and fsync is deemed too slow, and regular close reports some errors on NFS on some systems that would otherwise go unreported).

Yes, that part of Gnulib has caused some problems previously. It is mostly used to implement <stdio_ext.h> functions on non-glibc systems. However, it is also needed for some buggy implementations of ftello, fseeko, and fflush.

P.S. Hi Florian :)

And now updated for this change.

https://git.savannah.gnu.org/cgit/gnulib.git/commit/?id=69a0...

> Yes, it's not a good idea to do this. There are more questionable pieces in gnulib, like closing stdin/stdout/stderr (because fflush and fsync is deemed too slow, and regular close reports some errors on NFS on some systems that would otherwise go unreported).

Hyrum's law strikes again. People cast dl_info and poke at internal bits all the time too.

glibc and others should be using kernel-style compiler-driven struct layout randomization to fight it.

> Hyrum's law strikes again.

Is there a name for APIs that are drawn directly from some subset of observed behaviors?

Like Crockford going, "Hey, there's a nice little data format buried in these JS objects. Schloink"

> Is there a name for APIs that are drawn directly from some subset of observed behaviors?

Desire paths. https://en.wikipedia.org/wiki/Desire_path

Hyrum's Law applies: the API of any software component is the entire exposed surface, not just what you've documented. Hence, if you have FILE well-defined somewhere in a programmer-accessible header, somebody somewhere can and will poke at the internal bits in order to achieve some hack or optimization.

OTOH, yes.

OTOH, when coding, I consider FILE to be effectively opaque in the sense that it probably is not portable, and that the implementers might change it at any time.

I am reminded of this fine article by Raymond Chen, which covers a similar situation on Windows way back when: https://devblogs.microsoft.com/oldnewthing/20031015-00/?p=42...

Yes, it would not be sane to depend on implementation details of something like this.

But the sad reality is that many developers (myself included earlier in my career) will do insane things to fix a critical bug or performance problem when faced with a tight deadline.

Such is life, yes.

> to achieve some hack or optimization.

Or functionality. Happens to me all the time I have some Java class that's marked Final, so instead of just extending the class and moving on, I have to copy/paste the entire class wholesale to accomplish my goal.

Personally I hate "nanny" languages that block you from accessing things. It's my computer, and my code, and my compiler. Please don't do things "for my own good", I can decide that for myself.

(And yes, I am aware of the argument that this lets the original programmer change the internals, in practice it's not such a big problem. Or the cure is worse than the problem - for example my copy/paste example.)

Another example is a private constant. Instead of allowing me to reference it, I have to copy it. How is that any better? If the programmer has to change how the constant works then they can do so, and at that point my code will break and I'll .... copy the constant. But until then I can just use the constant.

Typical "early-in-carrier" thinking. Copying implementation is totally correct move here.

All projects mentioned should have forked stdio and added their hacks/optimisations/functionality to that.

They were just too lazy. Can't blame them though. Writing C code is torture after all. One should cut all the corners they could.

> Typical "early-in-carrier" thinking.

Other way around. When I first started I thought these access restrictions were a great idea. Make sure, I, and other, only program "correctly".

When I got more mature I found how often they are impediments, and how little they actually help.

I don't want someone else protecting me from myself. I just don't want that, no matter how well intentioned.

What I meant are "avoid copy paste at all cost" and "I'm center of the world. What I want is the only thing that matters"-thinking.

About access restrictions. We have two nice examples here

1. Stdio devs can't freely make modifications because someone's code depends on private implementation details

2. And yours example. You tried to do wrong thing and those access restriction made you suffer and do the right thing.

Works as intended I'd say.

Access modifiers exist so ppl dont shoot themselves

I never asked anyone for such protection, and I don't want it.

Fortunately as time showed such things are over all beneficial for software engineers worldwide

You can usually use other langs

The OpenBSD answer to this is: fuck them they should've known better. The few pieces of software that do this and have an active port maintainer will get patched. The rest will stay broken until somebody cares to deal with the change.

And for everything that people say about backwards compatibility, and for all the times that something has broken on me... man, I'm still glad that there are people who will stand up and defend that attitude. It isn't breaking things if you point out where they were already broken. And moving fast is defensible when the only alternative is standing still.

One time, when we were discussing a potential internal change that threatened to break ill-behaved software, Steve Summit told a story like this:

"Once upon a time, pointers on the Macintosh had 24 bits. The upper 8 bits were reserved for flags. Apple warned developers not to look directly at the flags in the upper 8 bits, but to use the macros that were supplied as part of the API -- but third-party developers looked directly at the upper 8 bits anyway. When System 7 came out with full 32-bit pointers, a lot of old applications broke because of this!"

Of course, what he didn't mention at the time was that System 7 provided a toggle that allowed these programs to run with old-school 24-bit pointers -- the equivalent concession is something I don't think OpenBSD is willing to make.

Nevertheless, vendors can and have broken full backward compatibility in cases where the developers "should've known better". Hyrum's Law just states that there will be a few that don't get the message and will watch their software break when these changes are made...

The standard doesn't specify any serviceable parts, and I don't think there are any internals of the struct defined in musl libc on Linux (glibc may be a different story). However, on OpenBSD, it did seem to have some user-visible bits:

https://github.com/openbsd/src/commit/b7f6c2eb760a2da367dd51...

If you expose it, someone will probably sooner or later use it, but probably not in any sane / portable code. On the face of it, it doesn't seem like a consequential change, but maybe they're mopping up after some vulnerability in that one weird package that did touch this.

Historically some FILE designs exposed the structure somewhere so that some of the f* methods could be implemented as macros or inline functions (e.g., `fileno()`).

I guess these things usually boil down to performance concerns right?

I've seen old code do this over the years. When you consider for example that snprintf() didn't used to be standardized until the late 1990s. People would mock up a fake FILE* and use fprintf.

*BSD stdio.h used to include macro versions of some stdio functions (feof, ferror, clearerr, fileno, getc, putc) so they would be inlined.

    /*
     * This has been tuned to generate reasonable code on the vax using pcc.
     */*

One issue – ISO C defines setvbuf (configure buffer mode and size of a stdio stream) but not getvbuf (get current buffer mode and size). On many platforms with non-opaque FILE, you can implement your own getvbuf by peering inside its undocumented fields.

I guess part of why it is not in the standard is that it is rarely requested functionality, but there are rare use cases where it may have value. And I think it is an unfortunate lack of orthogonality to have a setter but no corresponding getter.

stdio_ext.h offers some functionality like a "getvbuf", but not quite – e.g. __fbufsize tells you a stream's buffer size, and __flbf whether it is line-buffered – but it isn't clear how to distinguish fully buffered and unbuffered streams. And stdio_ext.h has never been standardised, it is an extension invented on Solaris and copied by Linux (and a few other platforms too, e.g. IBM z/OS).

People use reflection for monkey patching and complain when using compiled languages less supportive of such approaches.

So it wouldn't surprise me, that a few folks would do some tricks with FILE internals.

I always assumed that people could poke into it, but shuddered at the thought.

Reply on news.ycombinator.com