mpixorg

Another interesting comparison between clang/llvm and cparser/firm has to do with their respective preprocessors and underlying AST structures. cparser's entity_t and related structures are clean, clear, and very easy to work with. That, together with the fact that cparser is written in C, allows you to effortlessly walk the AST while having access to its entire informationwithout the help of an iterator, and without having to call any specialized functions. Having worked with both (lib)clang and (lib)cparser, I consider cparser the better and easier library for source analysis, as well source-to-source transformation tools. Other areas where cparser's AST could be nicely exploited: a source editor that provides, in addition to syntax highlighting, information about linkage and effective optimization, and a source generation tool based on the user's own code gallery.

A quick update regarding toolchain work and release time: we have now taught binutils and gcc to do the right thing for the target with respect to weak symbols and GOT entries. That has been quite a ride, but gladly dynamic linking is now working exactly as desired (that is, without depending on import/export annotation in the libc headers). Focus has now shifted back to the runtime layer, and we are working in full speed towards the pre-pre-alpha release.

Almost forgot to mention: the current cross-compiler is based on gcc-4.6.4 since that is the last modern gcc which does not depend on C++, and is therefore easier and faster to build and run. Porting subsequent gcc versions, and likewise clang and cparser, is a high priority, and will follow the initial release of the runtime components.

The general design allows for building the toolchain in its entirety (libc, libcgcc, libstdc++) independent of the runtime components. The idea behind this is that changes to the toolchain will fairly soon become rare [but see below], whereas changes to the runtime will be very frequent at least in the first couple of years.

Building the cross-compiler requires a native compiler and a shell environemnt that is capable of building gcc. The build process is trivial, and thus far has been tested on several Linux flavors (just asked that this also gets tested on BSD and OSX). Building gcc in an msys/cygwin environemnt is always a bit more tricky, so we have not spent much time trying that. Instead, we are working hard to become self-hosted as soon as possible.

+ building the cross-compiler:

git clone git://midipix.org/cbb/cbb-gcc && cd cbb-gcc && ./cbb-midipix-cross-gcc.sh

yep, that's that. This will use $HOME/temp as a temporary folder, and install the toolchain to $HOME/midipix. Make sure you add $HOME/midipix/bin to your path in order to use the toolchain. As with all other gcc builds, you need to have the usual dependencies on the build system (gmp,mpfr,mpc,libelf,texinfo) and a working shell environment.

+ pre-pre-alpha, radical changes: some major changes to the toolchain are underway. If you find it hard to believe that building the cross-toolchain is that easy please go ahead and run the above commands from a nearby shell, but please also rebuild it in a day or two (look for a commit message mentioning "automatic creation of GOT entries" :))

+ as a kind reminder, cross-compiling is just for building applications; testing them can only be done with the runtime library, which is not out yet.

Actually no... the application makes all calls using utf-8, and is expected to provide it in a well-formed manner so that the system call layer could convert it to utf-16. In the reverse route, where utf-16 is read by the system call layer and then converted to utf-8 (getdents(2) and friends), it is expected that file names be in well-formed utf-16. For a file-system volume to have ill-formed utf-16 name entries would make for an interesting case... have not encountered that yet, but will certainly look into that.

There are two main scenarios for fork(), and the lack of distinction between them is what I suspect led to some of the articulated worries.

The first scenario, in the spirit of traditional unix, consists of the sequence fork+execve. This scenraio is supported in midipix, yet requires that the (optional) subsystem performs execve on behalf of the forkee. As noted earlier, if all the child does is call execve, then application authors should consider switching to posix_spawn not only due to its better portability, but also for performance reasons.

The second, and arguably more interesting scenario, takes place when the child brnaches away from the parent process, performs (in parallel) a designated portion of a larger task, and then exits. This second scenario has been thoroughly tested and works as expected with respect to IPC, I/O, signal delivery, and nested forking.

In either of the above scenarios, there is no expectation that the child could interact with csrss, create or operate on GDI handles, or otherwise use the WIN32 API directly. Then again, it is expected that a process that calls fork() will not have any of the GUI libraries loaded at the time the call is made. Any limitation here, if any, has little to no relevance to existing applications that depend of fork, but only to what I fondly call the "fetishization of fork" in discussions about posix-to-windows portability. Applications that truly depend on fork were written with neither WIN32 nor GDI in mind, and therefore will not break due to such windows-specific interfaces not being available (as an aside, an X11 application calling fork() from within its event loop is an equally bad design regardless of whether the operating system supports that or not).

Looking forward to further cooperation! I hope this addresses at least some of the questions you have raised.

With FILE_SHARE_DELETE there shouldn't be any particular problem, but I certainly want to test this with the posix flags you had in mind. Do you happen to have a minimal example that you deem problematic on Windows?

The overall approach is to license cross-platform tools under the MIT license, and the Windows-specific runtime components under GPLv2 and GPLv3 that could be supplemented with a commercial license.

The project's pre-pre-alpha will be out very soon (in a week or less) and will already give a nice taste of its design and speed. The primary focus thus far has been major challenges (toolchain, process creation and initialization, signals, glue layer between posix system call layer and libc, etc.). This means that while you will be able to test functions such as fork(), execve(), mmap(), or fopen() with a utf-8 path, some of the easy/easier-to-implement system calls will "surprisingly" still be missing.

For one or the other reason, fork() has become in many discussion of posix on windows something of a fetish. The interface surely has its place, and figuring out how to efficiently implement it took a huge amount of effort, yet the vast majority of applications do not truly need it. Matter of the fact is that even on linux, where fork(2) is natively supported, the sequence fork+execve is more costly than clone+execve (where clone's flags are CLONE_VM && !CLONE_THREAD). For additional reference, see for instance the implementation of posix_spawn in musl libc (http://git.musl-libc.org/cgit/musl/tree/src/process/posix_sp...).

If high performance of the posix layer were not possible the project would have not existed. Among the factors that make high performance possible are 1) direct use of kernel interfaces (aka the Native API, where most of the runtime layer is written as a user-space driver), 2) utf-8 as the primary supported multibyte encoding as a foundational concept rather than an afterthought, and 3) tls implementation that matches in speed the native tls facility.