Interoperability Between Swift and C++
github.com
github.com
Currently this is done through C APIs (and it works well enough TBH), but plain C APIs lack annotations for higher-level concepts (like lifetime, bounds checking, etc). Why not work on a modern language interoperability standard instead? Being able to mix many different smaller and specialized languages in a single project instead of trying to create the "one universal language to rule them all" also would speed up language evolution, because less effort is wasted copying features from one "universal language" into other "universal languages", and as a bonus, operating system APIs following that standard would become language-agnostic.
[0]: https://twitter.com/bitshiftmask/status/1286411475658178561
Also, C, Objective-C and C++ cover a large part of the need (certainly when looking at Apple’s code base for their OS)
I also think any other interoperability standard for native code that works well for a broad range of languages is a pipe dream. There simply are way too many language idioms. Even for managed runtimes, you get lots of friction, for example between languages with mutable state and languages without.
What I mean with "LLVM for language interoperability" is not using LLVM itself (although it could be part of the wider LLVM project), but the idea of LLVM. LLVM decoupled language frontends and code-generation backends by defining a common intermediate layer. Suddenly it was possible for individuals to create new languages that automatically run on all CPUs and ABIs supported by LLVM. What's missing for those small new languages is the same effect for connecting those languages to all the other languages that already exist. Doing this for each other language is impossible for small teams. Doing this for a single common interface specification on the other hand is fairly trivial.
Such an "hourglass shape" would also make sense for language interoperability. But instead of having a common intermediate bytecode representation, have a common intermediate API-definition + ABI specification (somewhat similar to existing IDLs, but without a separate IDL-language that must be manually maintained, but instead automatically derived from the "native"-language interfaces.
Also, it doesn't need nor should support all sorts of special language idioms, just a basic common set of "interface features". It would be very close to what we already have with the C API/ABI interoperability, just flavoured with additional annotations for some common higher-level concepts that languages have agreed on since C was invented (e.g. different types of references: unrestricted pointer, non-nullable pointer, ranged pointer, ownership annotations, etc...).
Once you have such a common set of annotations, languages can create more idiomatic (not perfectly idiomatic) API bindings automatically. It doesn't have to be perfect, just an evolution of the current state (basically "rich C-APIs" instead of "plain C-APIs"). And plain old C would still be part of the language mix that's supported, it would just ignore most of the high-level annotations.
Let’s say you use library A, get a string out of it, and want to pass that to a function in library B. That will only work if both libraries have the same idea about what a string is. The same argument applies to arrays, hash tables, etc.
Similarly, the wish to be able to have a serialization library that works across languages will add restrictions.
Also, do you support bare functions? If so, you’ll have to tweak things for Java. Do you support function overloading, and if so, how? Do you have multimethods? If not, you’re putting languages that do at a disadvantage. If you do, languages that don’t will be problematic.
The only examples I can think of where that works reasonably well are C/C++/Objective-C (with the C ABI as the intermediate), Java/scala (but note that scala was designed to work on the JVM, in the same way C++ and Objective-C was designed to work ‘on’ C), and C#/F# (again, F# was designed for .NET)
Swift is designed to work with Objective-C and, because of it, with C and, to a lesser extent, C++. This tries to make that a larger intent.
There's a whole lot of code written in both; their toolchains are highly portable; for a long time they were seen as the only languages that gave programmers the ability to write high-performing code; and they are both unsafe by default.
Providing seamless bindings with a safe-by-default language like Swift is a way to move the whole systems programming field forward to a safe language, while maintaining compatibility with the enormous existing code base in the world.
Apart from that "grand idea" I think a more realistic real-world solution for the C++ => Swift situation is to simply wrap the C++ API into a C API first, and then create a Swift API on top of the C API, instead of trying to somehow translate the C++ API directly to Swift. E.g. every C++ library should come with an optional C-API to simplify language bindings, or even just compiling that library into a DLL.
Among those remaining, Swift has substantially more corporate backing, what with Apple relying on it for their massively successful iOS platform and Google trying to use it for TensorFlow.
C wrapper for C++ APIs has long been standard practice, but adds a substantial maintenance burden to C++ library authors and does not address semantics in the host language, which is a major part of this proposal. The whole proposal is based around the idea of seamless Swift to C++ interoperability. C++ has a lot of higher-level concepts that potentially can translate to similar higher-level concepts in other languages; round-tripping through a C API loses all of that.
To be clear, I like your idea - it's a grand one, and potentially solves the N+M problem once and for all. The real stumbling block for it is the C++ ABI and sheer mass of existing C++ code.
hard to do without incurring additional dynamic allocations though. e.g. how would you make, say, an API that wraps construction / doing stuff / destruction of something that'd look like this :
namespace mylib {
struct node {
std::vector<node> children;
std::optional<float> value;
};
}
mylib_node* mylib_node_alloc(void);
void mylib_node_free(mylib_node*);
// and then...
float* mylib_node_value(mylib_node*); // this ?
float mylib_node_value(mylib_node*, int* ok); // or this ?
void mylib_node_set_value(mylib_node*, float value, int set); // then there's this ?
void mylib_node_set_value(mylib_node*, float* value); // or this ?
void mylib_node_unset_value(mylib_node*); // or maybe this ?
struct mylib_node { char storage[sizeof c++ struct that was computed manually)]; };
// Or replace with something like a Strong Type; https://www.fluentcpp.com/2017/05/05/news-strong-types-are-free/
using NodeHandle = int;
struct OptFloat { float Value; bool bInitialized = false; };
NodeHandle CreateNode(NodeHandle* Children, int NumChildren, OptFloat Value);
void DestroyNode(NodeHandle ToDestroy);
void SetValue(NodeHandle ToSet, OptFloat NewValue);
> struct mylib_node { char storage[sizeof c++ struct that was computed manually)]; };
You can do;
// IN C api, fill this in manually
#define mylib_storagsize = ?
struct mylib_node {char storage[MYLIB_STORAGESIZE]; };
// or;
struct mylib_node {void* storage; }
// in node.cpp - the underlying C++ abstraction;
static_assert(sizeof(mylib::node) == MYLIB_STORAGESIZE);
My somewhat limited understanding is that the WASM interface types proposal is hoping to serve that sort of role. Note that in this case, the "web" part of "web assembly" is kind of a red herring; as I understand it, this proposal doesn't really have anything to do with the web and is more of an attempt to utilize WASM to create a sort of "universal ABI" that can replace the current de facto choice of using the C ABI everywhere.
While COM is Windows specific, other implementations with similar ideas do exist.
IBM and Unisys mainframe language environment, gRPC, D-Bus, XPCOM, SOM, Android IPC, Fuschia IPC.
My experience is that grand solutions can be attractive in theory but awful in reality.
> trying to create the "one universal language to rule them all"
I feel this rebuttal of your proposal should be made, but I'm not sure how to articulate it more convincingly than you did.
Every language that supports C FFI can talk with any other language that also supports it.
All these other efforts are just really higher level wrappers around that, but given that there are a million tools for generating the glue code to bind APIs using C for most languages in use it is hard for me to see the value here.
It's kinda like saying all you need is bytes. From the perspective of the kernel this might be fine because it isn't supposed to prescribe a particular method but user processes must still agree on a character encoding or a protocol they are going to run on top of those bytes.
The intermediate specification would be another language, so it would have the same impedance mismatch as the other N languages do.
You have memory management issues, OOP inheritance issues, semantic differences. Just trying to do a interoperability layer for the different versions of character strings would be incredibly hard.
Microsoft did something similar to this with CLI, but most people will note that JScript.Net and VB.Net are really new languages that just are optimized for porting Javascript and Visual Basic code over. They also defined a proprietary extension to C++ that allowed it to deal with managed objects so that it could interoperate with CLI code - but this did not aim to expose unmodified C++ code. Rather, it allowed you to use proprietary extensions to manually build bridges between the two.
Yes, but even though this is a limitation, every language only needs to care about connecting to this one intermediate "language". This is also why going through a C API already works quite well.
Also it's interesting to note that Microsoft's latest version of the WinRT APIs are providing a standard C++17 wrapper on top of COM APIs and don't require C++ language extensions anymore.
Right now it is hardly better than ATL/WRL, but so is life when Windows team wins its political battles with DevTools.
I’ve got a feeling that swift is becoming a very polluted mashup of features that come from parties with conflicting interests. AFAIK, internally at Apple, teams pull in different directions (e.g. to get SwiftUI), at the same time there’s this half-baked differentiable programming / swift4tf manifesto, then there’s the backend swift initiative with vapour.
This reminds of a hotch-potch that Scala’s framework and feature landscape is. The overhead of getting into it is quite substantial and for some, definitely not worth the investment if not just downright scary.
None of this is new per se; weird Apple specific edges like @UIApplicationMain and @NSManaged have always polluted the language. But, it's hard to buy in on "Swift world domination" when server-side Swift is stillborn, Rust is more appealing as a better bare metal language, Catalyst and SwiftUI are weaker than AppKit and UIKit, the last time Swift got a useful language feature was guard let in Swift 2.
On the point of "swift world domination", I still think this is largely a PR problem. I think Swift's identity as "compiled python" could make it a very good fit for many use cases where scripting languages are currently being used, due to it's clean, high level syntax and powerful type system. It still suffers a chicken-and-egg problem with respect to broader tooling and library support since it's viewed overwhelmingly as a language for making iOS apps.
Swift as a "compiled python" loses to Go, and that not even Go's design goal (though it is a fairly common use of it).
Does it have to be "ideal" to be worthwhile? I think it fits a similar use-case to something like C# but it's a lot nicer to work with. I don't understand why Swift shouldn't be perfectly usable for data science, or writing server-less functions for example: it would be a perfectly suitable stand-in for Python or Go in these cases, but with a much nicer type system in my opinion thanks to the first-class optional handling.
> Swift as a "compiled python" loses to Go, and that not even Go's design goal (though it is a fairly common use of it).
I totally disagree. Go is a jail. It's optimized for big teams with a lot of turnover, where you want to prevent complexity in the code-base by choosing a language with very limited facilities for abstraction.
Swift on the other hand has a really flexible syntax, which allows you to hide a lot of complexity behind simple interfaces. In that way, it can be a lot more like Python, where the code you're writing is very much at the domain level, and you don't have to think about the implementation details very often. There are many things you can express in swift which are no expressible in go.
It runs circles around storyboards when it comes to collaboration and traceability. Disregarding bugginess, the previews are a lot better than what was possible with IBDesignables.
The integration with UIKit is actually quite well made, and for most part not needed.
> there's no new hardware forcing its adoption.
While there is no explicit new hardware, there are several popular features that are only doable using SwiftUI: home screen widgets and complications. It will also become the language in which to write universal iOS/macOS programs.
> when server-side Swift is stillborn, Rust is more appealing as a better bare metal language
I agree with this, at least from Apple's part most effort is clearly done on "application level" and usefulness of Swift outside there seems more exploratory at the moment. Personally I'd like them to start adding higher, rather than lower level functionality, and steer towards scripting.
This here makes sense. I can see Swift becoming more focused on the application-level side at the moment and in the future becoming a way of writing cross-platform GUI apps in Swift, more likely if it is with Qt5 with direct C++ interop with a SwiftUI-like DSL.
As a Rust user, the server-side use-case for Rust is far more developed unlike Swift and would much rather see Rust pushing for low-level development as well.
However, I've seen many attempts for a cross-platform GUI library for Rust for a long time and it has been proven to be difficult in terms of adoption, implementation and maintenance. It would take an enormous amount of effort to cover all three of these difficulties from scratch and using bindgen bindings to wrap and maintain it and I'm afraid for them it would be more trouble than it is worth. But who knows?
I just don't see a mature cross-platform GUI library coming from the Rust ecosystem anytime soon.
It doesn't interact well with other language features (e.x. if let wasn't supported for a long time), it isn't really extensible (transforms are specified by the language), and yet it adds a bunch of complexity in both the language, implementation and reading code.
It seems clear, at least to me, that this is something that happens if you try to make a feature 'generic' without a lot of time and design consideration, due to Apple's internal politics on SwiftUI. I like SwiftUI, and some features that support it are great (like custom attributes), but I firmly think that they should have taked the time to produce a better design.
[0]: https://github.com/apple/swift-evolution/blob/main/proposals...
https://www.quora.com/Which-features-overcomplicate-Swift-Wh...
That was 2015, when Swift was far simpler than today.
Heck, they added almost all of Smalltalk syntax recently to address one special case of a special case.
https://blog.metaobject.com/2020/06/the-curious-case-of-swif...
And the method syntax that this is a special case of a special case of is many pages of spec.
And of course there's initialization, which was 14 pages of the Swift spec at the time, with crazy overlaps and gotchas. Unsurprisingly and predictably (as in: I predicted it), it wasn't enough, or rather far too specific, so for SwiftUI they had to create an entirely new way of creating/initializing: function builders.
https://blog.metaobject.com/2020/04/swift-initialization-swi...
Multiple trailing closures fit a common use case and seem like a natural extension of the language.
Absolutely.
But Swift wasn't parsimonious at first. Not even close. It unwisely rejected the parsimonious case and chose tremendous complexity.
Keywords? Or parens? Both! And we'll throw in a bunch of other stuff as well. And the weird case of having two keywords, one with a colon and one without, because we decided to make the keywords the names of the parameters. Sometimes.
And then afterwards they noticed that the parsimonious choice that they had initially rejected in favor of their much more complex choice was needed in addition to all that initial complexity.
Swift's keyword syntax isn't complex at all.
> Keywords? Or parens? Both!
I know you like Smalltalk, but keywords and parens are used together in a lot of languages (e.g. Python) so it's a tested idea.
> And the weird case of having two keywords, one with a colon and one without, because we decided to make the keywords the names of the parameters. Sometimes.
You don't have two keywords, you have an optional variable name. It's intuitive at the first time you see it - you're phrasing makes it seem much more complex than it really is.
I think you're trying to imply that Swift isn't an elegant language with it's syntax fitting in a postcard, but that doesn't mean it's 'tremendous complexity'. This whole comment is a bit too harsh to Swift, calling the legitimate differences between Swift and Smalltalk as 'complexity'.
> Swift's keyword syntax isn't complex at all.
Yes it is, particularly compared to the parsimonious options. Either parens or Smalltalk-style keywords can do the job, so you absolutely, positively do not need both.
Particularly, if you are going to have keywords, you just don't need parens. So it's already more complex than it needs to be (not parsimonious). Just explaining the various omitted or extra keywords takes 4 pages in the Swift book, the entire Smalltalk syntax fits on a Postcard.
What is complex or not obviously depends on what your baseline is. If it's C++, Swift might seem simple (though many disagree).
> (e.g. Python) so it's a tested idea.
The question was not whether it was tested, but whether it was a parsimonious design, which it is not. It was also not a good design, because it led to a number of follow-on problems that they had to incrementally fix while adding even more complexity.
Python is not a good comparison because they didn't really have a choice: they already had the paren syntax, which they could hardly deprecate.
However, choosing that design without a backwards compatibility restriction is unconscionable, IMHO, particularly if you already have a different keyword design all over your code-base. That's cutting off your nose to spite your face.
Anyway, even the baseline keywords-in-parens is complex and more complex than it needs to be (because it isn't parsimonious), but then you get follow-on problems: blocks inside parens are really ugly, so you add trailing closure syntax. Yet another special case, which is very different from arguments inside the parens. But then you discover that sometimes you need more than one trailing closure, and then you need yet another special case, the open-keyword-trailing-block syntax, which is different yet again.
Of course, instead of having 3-4 different syntactic mechanisms, you could have chosen just the one and not needed any of the others.
> You don't have two keywords, you have an optional variable name. It's intuitive at the first time you see it
No, it's absolutely positively not the least bit intuitive. And it makes no sense, because internal variable names have no business leaking to the interface. If you want to rename the names of your variables inside your method, it changes the signature! Not even C does this.
> that doesn't mean it's 'tremendous complexity'
Although the argument could be made, I wouldn't, not by itself, no. However, this is just one example of countless others, see for example Rob Rix, whom I quoted before:
Swift is a crescendo of special cases stopping just short of the general; the result is complexity in the semantics, complexity in the behaviour (i.e. bugs), and complexity in use (i.e. workarounds).
I also talk more about it here:
https://blog.metaobject.com/2015/05/i-am-jealous-of-swift.ht...
> is a bit too harsh to Swift,
Nope. The comment I replied to made the indefensible claim that Swift was parsimonious. Which it's not. Not even close.
> Keywords? Or parens? Both!
Why should they be mutually exclusive?
Swift's named parameters are great! For instance they allow for default parameters, which is something I really miss in Rust for instance where I have to use the builder pattern everywhere.
> And the weird case of having two keywords, one with a colon and one without, because we decided to make the keywords the names of the parameters. Sometimes.
Swift's rules for parameter naming are very simple, consistent, and easy to understand. Given the signature:
func foo(first second: Type)
func foo(x: Int)
func foo(x x: Int)
Yes, regretfully.
>> Keywords? Or parens? Both!
> Why should they be mutually exclusive?
Because of parsimony. The comment I was replying to made the claim that the original Swift design war parsimonious. It wasn't.
> Swift's named parameters are great!
Named parameters are great. Swift's implementation of named parameters is deficient, and not parsimonious. Yes, even that design is better than not having named parameters, but once you have named parameters, you don't need the parens.
Can you share an example of what this would look like in practice? I'm having trouble thinking of how this syntax would work.
var a, b, c;
...
a = b.fooBar(1, 2);
c = a.fooBarBaz(1, 2, a);
|a b c|
...
a := b foo:1 bar:2.
c := a foo:1 bar:2 baz:a.
...
The Smalltalk syntax is hard for me to parse, but it's probably just a familiarity thing. I remember when I first learned Swift I wasn't fond of not ending lines with semicolons. Now I dislike having to type them in languages where they're required.
1 to: 20 do:[ :i | Transcript show:i. ]
1 to: 20 do:{ stdout println:$0. }
While binary and unary messages chain w/o problems, it's not possible to disentangle two keyword messages:
receiver msg1:arg1 msg2:arg1
receiver msg1:arg1 msg1:arg2
But not always, it turns out. If you actually chain messages to the same receiver, you can use the semicolon to chain those messages:
myRect x:20;
y:10;
width:100;
height:100.
In Objective-S[1], you can use the pipe ("|") to chain message-sends that go to the result of the previous message, rather than its receiver. For example:
self dictionariesForQuery: "select {column} from {table}" | collect | at:column.
> delineating the argument set for a single function is the main job parens fulfill.
Right. Which is one of the many reasons why the case of Swift is so mind-boggling: there some of the parameters go inside the parens, some outside. I think you can make a good case for either all in or all out (better for all-out, IMHO, but whatever). But partly in and partly out?
I do agree in general that Swift can benefit from a reduction of special cases, but my personal feeling is that Swift is getting more consistent every version (at least in the features that already existed). YMMV, of course.
How so? That is literally what they did. The trailing closure syntax is Smalltalk keyword syntax, and so Smalltalk keyword syntax is used for that one special case. Which exists in addition to the non-keyword trailing closure syntax and the non-trailing (regular argument) closure syntax.
> but it's not fair to say Swift 'added Smalltalk syntax to address a special case'.
Can you explain how it is "unfair" when that is exactly what happened? Smalltalk syntax is really compact, there just isn't that much there.
Exactly. Which is why I wrote "they adopted the syntax", not "they adopted the ideas". It would have been really wonderful had they adopted the ideas, and it would also have made their marketing slogan "Objective-C without the C" actually truthful.
> the ideas are different
Yes, and that's the problem.
> Trailing closure syntax is basically 'you can remove the parens if there are trailing closures'
Not really, no. The parens are still all there, you just move one (original trailing closure syntax) or more (new keyword trailing closure syntax) of the parameters outside the parens.
Which is horribly inconsistent just by itself. I mean, the point of the parens is that they delimit the arguments. If you have parens, then the arguments go inside the parens. Except for this one...did I say one, I meant these two special cases where the arguments do not go inside the parens.
If you are going to have some of the arguments be outside the parens, why not just go all in and have all of them "outside" by just getting rid of the parens and be (a) consistent and (b) dramatically simpler and (c) readable?
> which was inspired by Objective-C, which was inspired by Smalltalk.
> I don't think Smalltalk influenced that syntax any more ...
Which is it?
Anyway, the point of the article wasn't that they consciously decided to adopt Smalltalk syntax. The point is that they bungled this so badly that they ended up adopting the very Smalltalk syntax that they initially rejected, and the reason they ended up adopting it was that they couldn't make things work reasonably any other way. Except that had they done this from the beginning, they would have not run into any of these problems.
There some truth to that I guess, but I'd also add that a lot of these modern multiparadigm languages feel like they're converging feature-wise (with a few outliers). I feel like Swift still has a unique enough feel to it and I enjoy working with it a lot. I think Chris Lattner has also expressed some concern about the focus on features over everything else.
That said, automatic differentiation could be a game-changer and should absolutely be included in the core language.
Seems to me like there's at least an argument that it should be possible grow diff programming into a general purpose meta programming system rather than use general purpose meta programming as implementation mechanism for differentiable programming ...
Cathedral and the Bazar..
You are basically complaining about how messy the bazar model is, and yet the bazar gave us Linux and in the end it won the race against all the others working over the Cathedral model and with millions of dollars invested in them.
Its fine to work at the bazar model, and it seems messy at first, but as long theres a 'cathedralization' phase from time to time it will work as a living entity, with constant organic evolution.
I've developed and maintain 3 or so swift apps in the last 4 years.
I also assist a colleague who ports and maintains them in Kotlin.
At first I liked Swift. As a dynamic OO languages guy, I didn't like C++, and felt that Swift was a fresh approach to the non dynamic OO language. Over time, my enthusiasm has waned.
For a while, I drank the value /struct programming koolaid. Too much. It turns out, that references are really really handy when your modeling real world things. Time and time again I've pushed myself to go the protocol route with structs, only to pull it out and just use simple single inheritance with real class based objects.
The real trap I keep falling into is this. The Swift memory management story for objects is a fricking joke. First you have to constantly stress about ownership relationships, so that you don't get burned by cycles. And then you get to discover the various behind the scenes implicit refcointing rules around closures or passing methods directly in lieu of closures. And having to put [weak self] everywhere. So you react by using structs more. Because you believe it will be more efficient, etc. But structs come with their own problems. You put ids in your structs, because you need to assert identity of a given thing. Abstraction is more difficult. You get to dance with generics more.
I've played with SwiftUI, I get the basic dream. It demos well. I could use it for simple CRUD style apps (which is what all the tutorials seem to be for). I'm just not sure how well it will scale to more complicated user interactions.
As much as I don't like Android development and loathe Java, I'm forced to often just wish I could use Kotlin in iOS. It has its warts. And often there are too many ways to do one thing so that they can streamline a tiny amount of boilerplate away at the expense of consistency. But I fight it less.
- API
- ABI, including threading, exception, function call, and memory models at the ABI level
- build specification
- packaging
This article mostly focuses on API, there seem to be some nods to the other three, but those seem like the harder and more pernicious problems to me.
For instance, how would the build system know that a library is being included that requires the swift runtime? Or even the C++ runtime. If any library was built against a different set of standard library interfaces (of any language), do any tools fail with a useful diagnostic?
That's not to say this isn't helpful research, but there's a lot of work to do before all of this is robust enough to be straightforward for end users.
The Swift compiler is in C++ targeting the LLVM and therefore with great interoperability with C and C++ through Clang.
The Swift compiler also ships with Clang, and you can link C and C++ projects that are dependencies of Swift packages using the Swift package manager with little effort.