Building and Linking Libraries in C | Hacker News Reader

Building and Linking Libraries in C

cs.swarthmore.edu

142 pointsvyuh96 comments

> LD_LIBRARY_PATH

While this works, it's generally a bad idea to depend on LD_LIBRARY_PATH. This type of environment override should generally be left as a tool for sysadmins and other integration/workaround needs.

Instead, you can bake the library search path into the executable with ld's "-rpath" option.

    gcc -o foo -Wall -Wl,-rpath,/lib/search/dir

> -shared

Building a shared library may also require building position independent object files. (CFLAGS += -fPIC). If the library is being distributed, it is also a good idea to set the soname[1].

This can get complicated very quickly, so it's probably better to let something like libtool[2] handle building and managing the library. (Autotools is a lot easier[2] to use that its reputation suggests. Way too many configure.ac scripts are a cargo cult mess)

[1] http://tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries...

[2] https://autotools.io/libtool/index.html

[3] https://autotools.io/whosafraid.html

Building a shared library may also require building position independent object files. (CFLAGS += -fPIC).

And -DPIC.

on MacOS you can do:

  install_name_tool -change new_path old_link executable

and you can check paths with

  otool -L executable

You can also bake in a relative rpath with $ORIGIN, although the fact that it requires a dollar sign causes endless makefile quoting nightmares:

https://enchildfone.wordpress.com/2010/03/23/a-description-o...

This is pretty much the only sensible way to build an executable that statically relies on having a library shipped alongside it.

I found libtool extremely painful to work with; it will hide your compiled libs in a hidden subdir, will wrap your compiled executable in a fscking shell script that then calls your real binary, it's unable to isolate a build against libs that may or may not be installed on your build system etc. Yet for all magic it does, libtool still doesn't solve the problem of building for a PREFIX (eg. /usr/local) in a temporary directory, and you need a chroot-jail or something for that.

From the libtool manual ([1]):

    Currently it is not generally possible to install into a temporary
    staging area that contains needed third-party libraries that are not
    yet visible at their final location.

So to build against a shared lib provided by another package, that other package has to be installed at the final destination already.

I seriously considered going back to just static libraries at some point. I believe the build system of git does away with autotools alltogether, consisting of just a simple Makefile.

Also, a link to phk's libtool rant seems apt here ([2]).

[1]: https://www.gnu.org/software/libtool/manual/html_node/Instal...

[2]: http://queue.acm.org/detail.cfm?id=2349257&ref=fullrss

Edit: typo; make links work

both of those links are 404 (at least for me)

I'm sorry, first one should work now.

Second one works for me, but has already been discussed here back in 2012 eg. https://news.ycombinator.com/item?id=4407188 .

I've used libtool for 20+ years, and found it to be opaque, baroque, slow, incomprehensible, and just plain broken.

Previous comments: https://news.ycombinator.com/item?id=12253252

Wish I could upvote your more.

The jlibtool I know (and that you seem to have implemented) however is a drop-in replacement for libtool to merely speed up libtool (rather than tar-and-feather libtool out of existence as it deserves to) isn't it? It seems to be used by apache httpd last I checked.

Anyway, cudos to those who do something about it. I tried a couple years ago, but haven't followed through.

I took jlibtool from Apache. I don't think they're using it any more.

I fixed a lot of bugs. It's now a 95% drop-in replacement for libtool. The builds are MUCH faster than with libtool.

> Autotools is a lot easier[2] to use that its reputation suggests. Way too many configure.ac scripts are a cargo cult mess

While also utterly horrible, I find CMake to be marginally less utterly horrible than Autotools, and most new projects seem to use CMake over Autotools.

Using libtool just shifts your learning some (mild) arcana about how libraries work to learning the arcana of libtool.

A library is not rocket science, and knowing how they work is important to understand how your compiler works anyway.

Increasingly popular are the "single header libraries", which were popularised by Sean Barrett[0]. It's as simple as downloading the header, putting it somewhere in your tree and #including it where you need it. It's especially useful for redistributing libraries, but I've also found it useful to create these in my own projects.

[0]https://github.com/nothings/stb

Oh, this kind of horror I've met lately has got a name? Well, I learned something today. Thanks ;-)

You can copy the functions into a .c file if it makes you feel better.

Despite what the code may look like, they all have a very easy to use API. Take stb_image.h for example. Most people need just two functions, stbi_load(), which loads any supported image format to a byte array. And stbi_image_free(), which frees the data.

But if you need anything more than that it's all there. e.g. loading from memory, loading via callbacks, support for HDR images, support for custom allocators, preprocessor flags let you exclude code for unused image formats, etc etc

truetype font rendering: stb_truetype.h: LOC=3287

Wow. I am speechless. Going to dive in another day.

Doesn't this hurt compile times and increase exe size?

Yes, but unfortunately for C++ libraries that make heavy use of templates, it's the only solution. Because templates need to be specialized to generate any code, they can't be compiled ahead of time into shared or static libraries. So in C++ you have this stupid non-sense.

Stupid nonsense? What would your alternative be, mind you, an alternative that would be as powerful and performant? Is there even such a thing?

Lisp does fine with separately compiled macros, which are more powerful than templates.

With Lisp you also need to have the macro definition ready when compiling code that depends on it. This is essentially the problem that all the various "system definition facilities" try to solve in at least somewhat usable manner.

Also, the package definition.

You need the package definition for the source-code-read time of whatever you're compiling, and the macros for its macro-expansion time. Practically speaking, these are tied together.

Instantiating the templates at link time would probably make more sense. With LTO we're already deferring a significant portion of the compilation process to link time so this is an obvious extension.

This issue was also on the mind of the C++ standards committee when they began work on the new module system. I'm not sure how the current spec behaves with respect to template instantiation, though.

C++98 contained some provisions for that ("export template"), but anecdotally there is exactly one compiler that supports it and the whole mechanisms was found to cause more problems than it solves. See: http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2003/n142...

Problem with template instantiation is that the mechanism is too general to be amendable to some kind of meaningful precomputation.

> as powerful and performant?

as the C preprocessor? Are you kidding?

It is not equally powerful, but Java's approach is easure. This means that at compile time the language can differentiate List<String> and List<Buffer>, and will type check them correctly, but at runtime the generic type gets "erased" and all the code turns into List<Object> (where in Java all types derive from Object).

As a result, there do not need to be multiple specializations of the List type for strings and buffers. No matter how many different parameterizations there are of a given generic class, there is only one implementation. I think this is probably a good thing most of the time, since on average the performance gains of this form of specialization are insignificant compared to the costs of code size. You wouldn't want 100 or more different implementations of List clogging memory and needing to be JIT-compiled.

One of Java's constraints is also that all generic types are reference types. So one reason why it wouldn't be too useful to instantiate separate List<String> and List<Buffer> is because these are basically just List<ReferenceToObject> anyway. C++ templates can provide both value and reference semantics, and can work on primitive types such as `int` which may reside on the stack, whereas Java generics only operate on full-fledged garbage collected objects that are dynamically allocated on the heap.

The disadvantage of this approach is that it precludes some constructs that are possible in other languages. For example, in C++ a template method could take some type `T` and construct a new instance of it. In C++ you could do this, but in Java it's not legal:

    <T> T broken() { return new T(); }

In C++ you could also create a new local variable of type `T` that will be allocated on the stack. This `T` can represent anything from a primitive `int` to a complex struct or class, or a point or reference to a dyanmically allocated object. The C++ code could then dynamically allocate a new instance of `T`.

This makes it more difficult to use implement generic programming patterns analogous to things like std::allocator in C++, and to deliberately specify different implementations of parameterized types and algorithms. Fortunately these are not too significant of practical downsides in Java.

I'd rather have code in headers and longer compiler times than type erasure.

C# is capable to instantiate templates at runtime/dynamic link-time so there is no need to compromise.

Instantiation at static link time would be feasible in C++, but runtime instantiation would require an heavy runtime which is frowned upon in C++.

You could put metadata in the object file which then allows the templates to be expanded at link time.

I tend to use a single compilation unit, and use plain C. For a 50k line project I have build times of 1-1.2sec. On top of that I'm also linking to opengl, windows.h and a couple of headers in the standard lib.

The speed up is easy to explain. Instead of launching gcc 50 times, it's now once for all.

Exactly :) I come from a .net background, where compile times are kinda bad compared to what I get in C. I had also heard of big C++ projects that could take more than 30 mins to build. So I was always scared to move to native code. Until I learned the single compilation unit trick! And if compile times were to get slow again, there are many more ways to bring it down even further,

Since when are .Net compile times worse than c? Csc compiles extremely fast, most long compile times are caused by msbuild doing... not much at all really.

I work on a much bigger project and we use that method (unity builds) - it helps for full builds and on small projects, but it makes incremental changes very slow on bigger projects.

If the code is included in the headers, then yes; if only definitions are included, then no.

All modern compilers support precompiled headers. If it's a library external to your project, you just compile it once into .pch file and when you change your own code, the .pch is just loaded without having to compile the headers again.

Not really!

Exe size -- Sean usually includes a macro that lets you include the implementation only once (so you can use it as a pure header in multiple places, then as a definition just once).

Compile times -- if you use that macro, the implementation will be skipped by the preprocesser which is really fast.

Even if you build the whole code multiple times in your project, C compilers are also very very fast these days.

Many people follow Sean's style but use C++. That can be much slower to compile if you use fancy features. But if you avoid templates it's usually fine.

It's interesting that many big C++ projects are also header-only, but for totally the opposite reason -- they use templates everywhere so nothing can be compiled separately. That approach definitely does slow down compilation and increase your code size.

Is having large amounts of code in headers common in real world C code (aside from stb)? Seems like it would be a nightmare to chase down issues, though maybe compiler error messages have gotten smarter since I've last dealt with C.

Header files are not special in C. It is a common convention to put interface in .h and implementation in .c files. But the preprocessor expands all the #includes together into one big file before the compiler proper attempts to compile it.

Modern compilers have error messages that show you the whole chain of #includes, and on top is the filename and line number, whether it ends with ".c" or ".h" or anything else.

Doesn't Boost have a number of header-only libs? I believe there is a header-only regex implementation for example.

C++ is a little different in that templates have to go in header files. A lot of template-based C++ libraries are entirely templates in header files.

It's weird because it does feel like it violates the separation of interface and implementation. On the other hand, it's really efficient. It has the advantages of doing generics with the C preprocessor (efficiency and de-duplication of code, if nothing else), minus many of the disadvantages of doing generics with the C preprocessor.

Putting non-template code or data in headers is something I've seen people do for expediency, but it's more trouble than it's worth IMO. As soon as two files in the same project include it, you risk a future headache.

The compiled result of template-heavy C++ can be very efficient, but the compilation process is notoriously inefficient. Just to be clear :)

Technically you can put the template definition in a separate translation unit and rely on explicit instantiation. The linker will tell you exactly what you need to instantiate. It is tedious and seldomly done.

/Pedantic

It is worth noting that header files should not include code, just macro and variable definitions.

There is nothing wrong with having code in headers.

If you put a bare function or variable in a header, then use it in multiple compilation units, the linker will reject your multiply-defined symbols.

(Templates get de-duplicated).

Unless you mark them 'static', which is common practice for constants and small inlined functions.

Inline is the keyword here (litterally). Inline, in C++, is defined as disabling the requirement of having exactly one definition of a function (it is also a weak hint for the compiler to perform inlining).

Template functions are implicitly inline.

Inline in C has a similar but subtly different definition.

It's called a header for a reason, unless we are now redefining the meaning and intent of headers, so that people could stick C++ template code in them?

Oh, where is the officially sanctioned definition of the meaning and intent of a header?

What we call them doesn't matter. There is nothing wrong with putting code in one. You can call them source files if you prefer.

Oh, where is the officially sanctioned definition of the meaning and intent of a header?

So glad you asked: The C Programming language, 2nd edition, page 82, chapter "FUNCTIONS AND PROGRAM STRUCTURE", as follows:

"There is one more thing to worry about--the definitions and declarations shared among the files. As much as possible, we want to centralize this, so that there is only one copy to get right and keep right as the program evolves. Accordingly, we will place this common material in a header file, calc.h, which will be included as necessary."

https://youtu.be/4PaWFYm0kEw?t=2237

Right, definitions and declarations. Common material. Code.

If you create your own shared object files and do not install them in /usr/lib, then you need to set your LD_LIBRARY_PATH environment variable so that the runtime linker can find them and load them at run time.

A better approach would be to use the $ORIGIN linker keyword (-R works in gcc by special dispensation for compatibility with every other linker in existence, and is portable, as opposed to -rpath):

  gcc -L/home/newhall/lib -R'$ORIGIN:$ORIGIN/../lib' -lmine -o myprog myprog.c

because unfortunately the $ORIGIN keyword includes the $, to embed it effectively into Makefiles and shell scripts, the following is needed in the environment:

  O='$$O'; export O
  ORIGIN='$ORIGIN'; export ORIGIN

this way, both shell and make will expand $ORIGIN to $ORIGIN, without the need to recursively escape anything. $ORIGIN linker keyword is supported by GNU ld, Solaris / illumos ld, and HP-UX's ld.

https://blogs.oracle.com/ali/entry/avoiding_ld_library_path_...

https://blogs.oracle.com/rie/entry/tt_ld_library_path_tt

Also, unless one is an operating system vendor, one may not install anything into /usr, as that is the vendor's namespace and having a clash with the vendor could bring one's production down in case of upgrades, were the vendor to deliver the same software. The System V and LSB FHS specifications state that /opt is for 3rd party and unbundled applications, /etc/opt for configuration, and /var/opt for data.

Unfortunately, the essay says nothing about library interface versioning with linker map files, for backward compatibility, so that one single shared object library can be used by all versions of the consuming software simultaneously.

Also: https://www.akkadia.org/drepper/dsohowto.pdf

drepper to the rescue

I hope there is a tool like Cargo (https://crates.io) in C

There is, your bog standard package manager (in unix and unix-like systems).

That sort of package manager has a totally different purpose and does absolutely nothing to help build your code, manage dependency versions, or control how dependencies are built.

Totally different ? Are you sure about that ?

>.. does absolutely nothing to help build your code ..

I don't know much about cargo nor rust. How does it help build your code ? edit: nvm, i found how [0]

>.. manage dependency versions ..

All package managers do. Granted not implicitly when building your source, as you have to install them yourself. Good libraries can have multiple versions installed at the same time (i won't go into nix and such).

>.. control how dependencies are built ..

Most (of the used) programming languages don't need to build dependent libraries, they load them dynamically.

Note that I don't differentiate between code in headers, code in libraries, or even code in other programs that the specific program would communicate with via some IPC. (side note: i have grown fond of single header libraries, for their simplicity)

Also to quote the rust docs "Rust has two distinct terms that relate to the module system: ‘crate’ and ‘module’. A crate is synonymous with a ‘library’ or ‘package’ in other languages." Not 100% what i was saying, but it does hint on how similar cargo is to a "normal" package manager.

It seems that everyone wants to build in a package manager these days. Rust, racket, haskell, python... What is wrong with copy-ing code directly ? Do we need these ? It just reminds me of leftpad.

[0] http://doc.crates.io/index.html

Everyone wants to build a package manager these days because OS-wide package managers don't provide the same feature set that language-specific package managers do.

Which feature set would that be?

At the very least (npm, conda) multiple independent environments, allowing conflicting versions of a library to exist simultaneously in different environments. If the language allows it, even in the same environment.

To get that from apt/yum/pacman you need chroots, containers or virtualization.

At the very least (npm, conda) multiple independent environments, allowing conflicting versions of a library to exist simultaneously in different environments.

One should never, ever bypass the operating system's software management subsystem, because then, one needs a hack like Docker, since the OS will have no notion of the software installed, and a system specifically invented and designed for managing said software won't be able to manage it. I could author a book on just how bad of an idea using a packaging system proprietary to some language or application is. By utilizing a packaging system outside of the operating system's packaging, anyone doing so undermines operational repeatability (see capability maturity model level 2 and higher).

allowing conflicting versions of a library to exist simultaneously in different environments.

That's another very bad thing: it's a workaround for lack of thinking about backwards compatibility, and designing a single library with multiple versions of the API, and, and, and... I happen to work on an application which utilizes 300 such environments, and it's a nightmare.

Finally, by using a packaging system proprietary to a language, anyone doing so is now forcing everyone else who wants to use said software to wrestle with multiple packaging systems; and if every developer thought that way, the end consumer would have to wrestle with all of them. I don't think I need to point out just how irresponsible, not to mention inconsiderate that is towards users. Users of the application, not developers should come first, and only then everything else. In this day and age of automated builds, it is trivial to set up automatic generation of operating system packages for various systems. Trivial, and developers must take responsibility for that, instead of putting the onus on the consumer of the software.

> One should never, ever bypass the operating system's software management subsystem,

Neither conda nor npm does something like that. They set up a localized environment, insulated from the host os installed software as far as packages they manage are concerned.

> By utilizing a packaging system outside of the operating system's packaging, anyone doing so undermines operational repeatability (see capability maturity model level 2 and higher).

No they don't. I'm not sure about NPM, but conda documents exactly what packages are installed inside it, generates a specification file you can use on another install to recreate the same environment, and insulates the inside install from changes in the underlying OS.

> That's another very bad thing: it's a workaround for lack of thinking about backwards compatibility, and designing a single library with multiple versions of the API, and, and, and... I happen to work on an application which utilizes 300 such environments, and it's a nightmare.

Feel sorry for you. But I have to maintain two projects, one requires an older version of the library, one requires the newer. There is no way to achieve this with "apt-get" or "yum". Conda lets me develop both on the same machine in a repeatable way. And it's way better than any other solution I'm aware of.

> I need to point out just how irresponsible, not to mention inconsiderate that is towards users. Users of the application, not developers should come first, and only then everything else. In this day and age of automated builds, it is trivial to set up automatic generation of operating system packages for various systems. Trivial, and developers must take responsibility for that, instead of putting the onus on the consumer of the software.

What?

There are many packaging systems for Linux/BSD/Unix I use often (apt/deb, yum/dnf/rpm, pacman, ports, brew, cygwin), even more repositories (ubuntu, debian, ...) and there is absolutely zero chance that I can rely on dependencies to be provided by the OS package manager - they don't update at the same rate, and sometimes not in a compatible way (e.g. PIL vs. Pillow migration for Python; ffmpeg vs libav).

Conda, however, makes that a simple "conda install" away, once you have conda installed. And it also works on Windows - which has no canonical package manager (chocolatey is the closest, but it is still very far).

What you are suggestion is useless in my situation, and for at least 95% of the developers and deployment situations I'm familiar with. If it works for you, then ... good for you. But you're pretty unique.

Neither conda nor npm does something like that. They set up a localized environment, insulated from the host os installed software as far as packages they manage are concerned.

That is exactly what I mean by bypassing the operating system's software management subsystem. No software or configuration should ever be installed by anything other than the operating system's software management subsystem. It is the responsibility of the developer (you or anyone else doing so) to convert content pulled in by npm or anything that is not an OS package into one, but the best solution is to fix it upstream by abolishing these parallel software management subsystems, and automatically generating native OS packages for the software through continuous integration build processes.

Not doing so puts the onus on the consumer of the software, and to wit, the purpose of software is to make it easier, not harder or more time consuming, to do something. Imagine forcing the system administrator to have to replicate your development environment, through any mechanism, just to be able to run your application and all her dependencies, when she or he could have just plugged in OS packages of your software into the OS provisioning repository.

Feel sorry for you. But I have to maintain two projects, one requires an older version of the library, one requires the newer. There is no way to achieve this with "apt-get" or "yum".

Actually, yes there is: make the library interfaces versioned, and fuse them into one library for all versions of the consuming software. This is, post action, considerably more work, but that is what developing software is all about. The idea of versioned interfaces is not new, and originates on UNIX. Since the initial topic of discussion is building and linking libraries in C, the link editor can consume linker map files, which define the library's semantics based on the version of the interface, and will encode this information into the executable and linking format header of the binary. The encoded information will be consumed by the run time linker in order to present the correct version of the interface to the consuming application. Same principle applies to any software in any programming language.

Then the developer must provide the additional dependencies as OS packages, and can do so without interfering with the OS vendor by adhering to the System V / Linux Standards Base - Filesystem Hierarchy Standard: by delivering the payload in /opt[/namespace], the configuration in /etc/opt[/namespace], and coding the application to use data from /var/opt[/namespace]. The fathers of UNIX considered this, and provided specifications to address it accordingly.

http://refspecs.linuxfoundation.org/fhs.shtml

What you are suggestion is useless in my situation, and for at least 95% of the developers and deployment situations I'm familiar with.

As you can probably imagine, I disagree vehemently with your assertion, and it is my hope that through this dialog, I've managed to illustrate the reasons because of which that is so, as well as provided practical solutions to the problem at hand.

If it works for you, then ... good for you. But you're pretty unique.

You mean, I am unique in wishing to deliver software to the highest standards of quality enabled by our current technology? Having started as a system administrator, being a developer, I can completely understand the gargantuan amounts of work system administrators face trying to make developers' work repeatable, essentially being forced to complete developers' work for them. And that is simply not just, nor is it right. We as developers should do that.

Empathy is a core engineering value:

https://www.listbox.com/member/archive/182179/2013/05/sort/t...

respect your users and empathize with them. If you don't quite know how to solve the problem, ask system administrators to help you produce higher quality software, or even better, learn system administration yourself, just don't punt on the problem because you perceive it as too much work.

> Actually, yes there is: make the library interfaces versioned, and fuse them into one library for all versions of the consuming software. This is, post action, considerably more work, but that is what developing software is all about.

You assume I am the library maintainer, but I am not. I am just the library user. ergo, no amount of apt-get will fix it for me.

> Then the developer must provide the additional dependencies as OS packages, and can do so without interfering with the OS vendor by adhering to the System V / Linux Standards Base - Filesystem Hierarchy Standard: by delivering the payload in /opt[/namespace], the configuration in /etc/opt[/namespace], and coding the application to use data from /var/opt[/namespace]. The fathers of UNIX considered this, and provided specifications to address it accordingly.

No amount of apt-get will do that if the library/distribution/upstream developer didn't.

> You mean, I am unique in wishing to deliver software to the highest standards of quality enabled by our current technology? Having started as a system administrator, being a developer, I can completely understand the gargantuan amounts of work system administrators face trying to make developers' work repeatable, essentially being forced to complete developers' work for them. And that is simply not just, nor is it right. We as developers should do that.

Your solutions only work if you are the packager for every piece of software you use. That is a unique situation. Most of us have to use packages written and maintained by others.

Furthermore, Docker makes everything work magically for system administrators, and yet you are against it. Why?

And conda makes everything much more repeatable than the underlying OS apt-get can; I have the same, perfectly repeatable, conda-base app running on ubuntu 10 (yes, it needs to go), ubuntu 12, ubuntu 14, ubuntu 16, debian 6, debian 7 and debian 8 right now. It is impossible to do that with underlying OS package management - but conda is perfectly repeatable.

I've single handedly administered north of 200 servers of various makes (linux, windows, solaris), I understand the need for repeatability, and I get it from conda. I can't get it from the underlying package managers. My experience is so different from what you are proposing that I suspect we are coming from different worlds.

You assume I am the library maintainer, but I am not. I am just the library user. ergo, no amount of apt-get will fix it for me.

Then take the Apple Computer, vertically integrated stack approach: package all the libraries you require into OS packages, so that the entire installation is encapsulated.

No amount of apt-get will do that if the library/distribution/upstream developer didn't.

How about working with the upstream developers on it? But I agree on this point with you: you, as the user, have a right to expect to be treated with dignity and respect: if the developers aren't providing OS packages for you, and they are on the same platform as you, and refuse to do so in favor of making their lives easier, then that's just wrong.

Furthermore, Docker makes everything work magically for system administrators, and yet you are against it. Why?

Because it doesn't. Docker makes a system administrator's job a living hell: when production kicks the bucket, and the system administrator is called in the middle of the night to troubleshoot the application, he cannot ask one subsystem designed for this purpose to tell it about the software: the software management subsystem. System administrators hate being burdened with artificial dependencies which make developers' lives easier, but make system administrator's jobs living hell. Change and configuration management with Docker is hell, because it doesn't exist. Docker isn't designed to support the change management process, because the developers of it have never heard of such a thing. How would you model the change management process with Docker? What about software lifecycle management, roll a several gigabyte image every time you need to fix your database component? What if you have thousands of such applications that you're responsible for? And why should you be forced to use Docker, when there are zones which provide far superior containerization?

As if that weren't bad enough, if developers delivered their software as modular units -- as OS packages, Docker would be completely and utterly unnecessary. In fact, it's unnecessary now, and only artificial insistence from developers makes it necessary. This is not a good situation.

Dumping a bunch of files into a format foreign to the operating system is never a good solution. OS packaging is designed from the ground up and optimized for managing software. Provisioning technologies like JumpStart, Kickstart, AutoYaST, Ignite-UX and NIM make it possible to go from bare metal to fully functioning systems, ready to accept and serve data. When one has OS packages, all one needs to do is plug them into systems already optimized for that. No need for arbitrary 3rd party solutions.

Packaging was and is, actually, in the strictest sense, designed for software developers, and yet, I have worked with many, many software developers who actively fight even looking at how to make an OS package, for no better reason than learning something new, and yet they'll spring for every shiny new programming language which shows up.

I don't know what to tell you. I have almost 73,000 systems across various countries around the globe, and growing daily with no end in sight. I specialized, and have formal education in UNIX system administration (in addition to formal education and professional experience in programming and algorithms). Several decades of professional experience in both, plus database engineering experience, plus large scale system engineering and architecture experience, as you can probably infer from the number of systems. But really, I don't think our worlds are that different, I don't think they're different at all: if it works on almost 73,000 systems, it will work on "north of 200".

Let me clarify. I managed, on my own, on metal, north of 200 in 2006, as a developer, before all modern orchestration software and even EC2 wasn't even announced until later that year. And when that started taking a nontrivial amount of time, I hired a sysadmin (which is not my job). I didn't bring it up as a pissing contest, just to indicate that I have relevant experience and appreciation of consistency. It's great you have 73,000 in your fleet, but I'm not at all sure you have more work to do than i did with 200 a decade ago.

That said, we do live in completely different worlds. I deliver software that works, repeatedly, on many flavors and historical versions of Linux, some out of vendor support or that never had vendor support, and windows. And does that without inconveniencing the user, and without costing me much in development time as to make it uneconomical.

None of your suggested solutions make any sense in my world. I need to run python 3.5 and a variety of its packages, but the client runs ubuntu 10.4; you suggest I either get the python maintainers to compile it for 10.4, or the ubuntu devs to, which couldn't have happened even when 10.4 was still supported; and then do it for every package I need. Alternatively, you want me to compile and maintain it myself as an 10.4 package, that shouldn't interfere with system python - but also 12.04, 14.04, 16.04 windows and a couple of centos. Which would also make me liable for tracking upstream security for every package I use, rebuild and distribute.

Alternatively, I can ask my customer to install miniconda (one download, one command to execute, regardless of platform), and then install my package through conda (two commands - create env and install). No interference with system packages, no breakage when they upgrade their system to a newer distribution.

Where as in your world you can run a mini Apple to vertically integrate all your packages, take responsibility for security updates for code you don't own, possibly force your customers to a uniform setup (and let go those who don't/can't).

Totally different worlds.

but I'm not at all sure you have more work to do than i did with 200 a decade ago.

I don't, and that is exactly my point: it's not more or as much work as you perceive it to be, making one's own packages. An experienced packager can get 98% of the software packaged in under five minutes. I do that all the time.

I would have never picked Python precisely because of the pathologies you describe (and which became obvious to me when building and packaging Python). In my experience, your choice of the programming environment led down the rabbit hole you describe.

What is your language of choice then?

How would you support the breadth of environments in that language of choice? Also, you fail to address the question of multiple incompatible environments and of windows support. Do give examples of how you package everything you produce as deb/apt, rpm/yum, exe/msi/chocholatey in a way that works on multiple OS versions and distributions. Please, concrete examples, not general "I would build it the way Apple would integrate verically". I have explained how I solve my problems, and as far as I can tell, (mostly because you avoid any specific answer to any of my questions) your solution is to specify the clients environment to a t, which is impossible in my world.

Do tell how you repackage e.g. the just-released Python 3.6 with Pillow and 20 other libraries for all these environments in a way that does not in any way conflict with or is influenced by the system Python. Or the same with Java8, or Ruby. or Tcl. Or Perl, or PostGRES, or MySQL, or nginx, or apache, for that matter. And in under 5 minutes. I wish to learn.

I am amused at your choice of words and perception. Life is easy for me, extremely easy, and it is so for my customers who run and administer their own machines. And yet you seem to believe I am tormented, or they are. I can easily support 10 very different environments without thinking, and enjoy the best tool for this job (which happens to be Python), the latest version thereof if I need it.

I perceive your reasoning to be dogmatic, and mine pragmatic. To each his own, I guess.

What is your language of choice then?

Depends on the problem at hand; these days, mostly control logic in Bourne (not even bash!) shell and an AWK processing / compute core.

Do give examples of how you package everything you produce as deb/apt, rpm/yum, exe/msi/chocholatey in a way that works on multiple OS versions and distributions.

A piece of software I develop always starts out as a Solaris System V package. Always. However, the code is written such that it's completely portable to all other UNIX like systems, which makes it trivial to build an RPM. I then take alien to build DPKG's for debian, and building HP-UX's PSF's is pretty straightforward too.

I do not, and will never support Windows: first, I don't want to support something that I want to end, and second, I don't do consumer space. It's not worth it. Not having to support a braindead platform (no /usr/bin and applications in it without Windows Services for UNIX or Cygwin) like Windows makes the entire thing manageable.

I don't touch Python 3, because it's not backward compatible, therefore I boycott it, and the Python software I use requires Python 2 anyway. I don't do Java, in fact, if I catch anyone using it, that's immediate grounds for dismissal, no questions asked, not that people I work with would ever do such a thing.

PostgreSQL, Apache, and so on: we developed a generic build engine, which builds this software, so that apart from patching / porting it will compile and install the built software under a builds/postgresql-#.#-SunOS-i386[.64], for example. Then I wrote a whole bunch of tools which automate package metadata files' generation, based on command line options, like so:

  % mkpkginfo -d description -n name -v ...

these then become interfaces, which a continuous integration system (or even cron, if you wanted to!) can call. Once these are finalized, they're checked into the source code repository.

There is much more to it than this that I'm not describing, like change requests, configuration management, and so on, for example, configuration packages which, when installed, automatically configure Apache, or any other application, without any human interaction whatsoever.

You seem to me to be in consumer space, writing "fat client" software for the desktop in Python, whereas I'm in the "backend" server space, developing for the infrastructure and web applications.

Life is easy for me, extremely easy, and it is so for my customers who run and administer their own machines.

Perhaps that is because they only have a handful of systems, and perhaps it is because they accept a salad of solutions, like using your proprietary packaging. I (and luckily the employers I used to work for in the past) allow no such salad. The environment is engineered to a T, and that is how we guarantee that what we deliver will JustWork(SM). Zoos and salads are strictly forbidden. It's as if you bought a laundry machine, or a Blueray player: it works, and noone thinks to mess with it, because it's an appliance. Messing with an appliance would be pointless, especially in my problem domain.

I perceive your reasoning to be dogmatic, and mine pragmatic.

And rightly so; my dogmas were shaped by years of extreme suffering in programming "zoos" and being paid to make such "salads" seamlessly work together. I'm the guy who got stuck on the brain detail of figuring out how to convert your proprietary package format into an OS package, integrate that OS package 100% with the operating system substrate, plug it into a provisioning technology like Kickstart, JumpStart(TM) or AutoYaST, and generally make it JustWork(SM). It is a gargantuan amount of work, and all because developers think exactly like you, when in fact, this was the developers' responsibility in the first place. I resent it, and view such software as incomplete. If the developers delivered their software as native OS packages conformant to System V / LSB-FHS and linked with $ORIGIN linker keyword, and if their packages contained code to fully integrate with the OS and start and stop automatically, things would be grand!

Thanks for the detail, it is interesting --

but it just drives the point that we live in completely different worlds - I cannot turn down Windows using customers (or I could, but I'd have to start a new business), and you apparently can. I believe the vast majority of developers live closer to my world than yours.

They clearly do manage "dependency versions" (that's what they do: that's the primary purpose of say, apt). As for the other things, I am pretty certain those are anti-features? I don't want the way I find and manage libraries and depenendencies to assume I am going to use some specific build environment, and the entire point of good package management is to be able to reuse packages so encouraging developers to "control how dependencies are built" is asking for trouble.

System-wide package managers give you a single, global version of a package built to interact solely with the rest of that one distribution.

I absolutely do want to control how dependencies are built- I want to build them in debug mode, link them using LTO, cross-compile them, or (Rust-specific, since someone mentioned Cargo) switch the panic strategy. Further, many packages I depend on via language-specific tools likely will never be in any distro's repository, nor should they be.

As for assuming a build environment, that may be an issue with a language like C or C++ that has already gone a different route, but it's better when a language has a single, established way of building things that a package manager can integrate with.

I haven't experimented with it too much and I believe there are a few others as well, but you could take a look at https://www.conan.io/

Nice simple guide. Perfect for people like me that are "relearning" c.

You might also like http://david.rothlis.net/c/compilation_model/ -- a tutorial I wrote that goes into some detail on what the preprocessor, compiler, assembler & linker do.

> (3b) you could also build an ARCHIVE FILE (a statically linked library, libmylib.a) from one or more .o files. If you link with a static library, its code is copied into the a.out file at runtime.

Huh? There is no copying at runtime when using a static library. Or have I misunderstood this all these years?

An alternative to using LD_LIBRARY_PATH is to bake the rpath into the executable. It's a little nicer for the user, IMO.

Tangential question: why aren't we using '#pragma once' more? I've adopted it in personal stuff and it's obviously Good. Is anyone still really coding for some arcane compiler that doesn't support it? Or is there some good reason for not just switching?

It's not standard. There are more compilers out there than gcc and clang.

What compilers don't support it?

The Portland Group's compiler (PGI) is the only modern compiler I know of, but there's probably a few embedded system compilers too.

One compiler I've encountered is SDCC. But think about new compilers - they shouldn't have to implement non-standard features to compile portable C. #pragma is non-portable.

There are proprietary compilers for embedded chipsets that don't support this feature. Granted, these are fewer and further between than they once were, but they are out there. It really depends on how portable you want your code to be. Even though it is widely supported, it still isn't a standard language feature.

If your goal is to have a library that runs on most Unix-like and Windows systems, then '#pragma once' works just fine. But, if there's a possibility that your code would be deployed to an odd platform, it's probably best to use traditional header guards. I'm so used to using header guards that the appeal of '#pragma once' isn't that great for me.

You don't have to go that far.. Oracle Studio (nee Sun Studio) for Solaris doesn't support it.

Isn't that more niche than embedded toolchains?

Wikipedia lists it as supported since version 12.5.

https://en.wikipedia.org/wiki/Pragma_once#Portability

#pragma once is great. I use it in new code. Clang and GCC both support it, don't know about other compilers.

1 - it isn't obvious what it does (if a file has a symbolic link, then is the link the same file? what about a hard link?)

2 - it offers no performance benefit on sensible compilers (which have a special case in them that can detect the typical header guard pattern and so avoid reading a header twice)

3- it is non standard, and the feature it provides is one which is easily implemented within the standard, where the only disadvantage seems to be that you need to type more characters.

Can't say about the others, but my IDE just adds #ifndef/#endif automatically, so I don't bother to change it manually.

Is there some performance gain from #pragma once ?

Does it update automatically when you copy and/or rename files?

> Is there some performance gain from #pragma once?

No. Include guards can be detected and optimized just the same.

> Does it update automatically when you copy and/or rename files?

IIRC, it does only if I use the Refactor option inside the IDE.

Do yourself a favour and use libtool through Autotools: https://autotools.io/libtool/index.html

Reply on news.ycombinator.com