XCurl | Hacker News Reader

XCurl

daniel.haxx.se

334 pointsTangerineDream90 comments

Microsoft is certainly within the rights afforded to them by the curl license to do this, but:

> The provided functionality is certainly a very stripped down and limited version of the libcurl API. A fun detail is that the quite bluntly just link to the libcurl API documentation to describe how xCurl works.

is IMO a major dick move. It seems inevitable that doing this will cause users with questions about xCurl to seek out curl maintainers/forums for support with xCurl (as with the email that prompted this article), even though they have no insight at all into how xCurl works. And some of those people might not be very happy with the response "we have no idea - try contacting Microsoft".

Now, that might not have been Microsoft's deliberate intent, but I don't see how it's anything but completely forseeable, and should have been avoided.

(Yes, Microsoft need to credit curl with the copyright info and license, but the place for that is the "Credits" and/or "License" parts of the documentation, not the "Usage"/"API reference" parts.)

Oh that's a huge dick move. And this behaviour you mention, "people seeking help from libcurl for xcurl" is an already documented and detrimental problem. You can see why Sqlite temp files has an extension of "etilqs". That's a grandiose example of this. That was not even a documentation link!

https://github.com/mackyle/sqlite/blob/3cf493d/src/os.h#L52-...

Another common example is people seeking help at upstream projects for packages provided by their linux distro that have subtle changes which upstream knows nothing about.

Honestly though, that's life. I don't think its that big a deal, if you've ever done open source user support you'll know that people ask about all sorts of crazy things. You're lucky if they are even asking about software and not how to wash a car.

Wow, fun one, thanks!

> It seems inevitable that doing this will cause users with questions about xCurl to seek out curl maintainers/forums for support with xCurl

I doubt this will be a large number of people. If xCurl is only available via the GDK, then you get access to it you need to be working for a company that has developer contract signed with Microsoft. If you are such a dev, you will also have access to the Xbox dev forums which _should_ be your first stop for GDK-related questions.

GDK is much more open nowadays (AIUI you can basically deploy stuff to your own Xbox by just signing few things online?), but you're basically right.

People are assuming this is meant to be a general-purpose, generic replacement; but it is quite narrowly focused on gamedev.

I think that's the case for 'Apps' but not games. Like I said in another post, my info is several years dated at this point, so things may have changed, but when I was in this field, if you wanted to publish a game vs a non-game app, there were many more requirements. Apps had pretty limited amounts of scrutiny and certification hoisted on them, while games had several hundred certification requirements for a ton of different things they had to meet. Apps also did not get access to the full capabilities of the console, whereas games did. Under the hood there were quite a few differences between how apps and games were provisioned, published and run -- games are almost like a container image rather than a regular windows app.

Oh if you want to _publish_ I'm sure it's a whole different ballgame, but if you're just playing around and/or developing a game (and don't need a full on DevKit), it's relatively straightforward.

> you can basically deploy stuff to your own Xbox by just signing few things online

Is this true? I've been wanting to be able to develop my own console stuff for a long time, but the hoops have seemed insurmountable so far. Can I really deploy my own stuff to Xbox without having to pay huge fees and have calls with a sales team?

For publishing to some store, that's fine, but just for development I just wanna be able to get a build running without too much red tape, otherwise it's not fun and therefore not worth it.

Yep, just install the Devmode app and follow the steps.

https://learn.microsoft.com/en-us/windows/uwp/xbox-apps/devk...

Lots of guides available for different scenarios (C++, Unity or HTML/JS)

Ah, this is specifically for Xbox apps, not games. You're not getting access to a bunch of things with this method (especially w.r.t. graphics APIs.) As far as I can tell, the full GDK is still behind a contract and NDA.

Ah damn, seems you're right.

From the GDK GitHub (https://github.com/microsoft/gdk):

> How do I run a GDK game built for the Xbox App on Windows 10 or Xbox Game Pass for PC on Xbox Consoles?

> Xbox console development requires the “Microsoft Game Development Kit with Xbox Extensions (GDKX)”. Games will need to retarget and rebuild for Xbox One or Xbox Series X|S with the GDKX installed.

> You need to download, install, and retarget your project for Xbox consoles using the GDKX

> The GDKX is currently only available under confidential license within an NDA Xbox program (e.g. ID@Xbox).

Sorry for giving you false hope!

Yeah, its a somewhat unfortunate reality that if you want to get full access to the consoles, you need to be an actual gamedev and sign some contracts. Microsoft does not suffer poor fools just trying to learn how to get the most out of the Xbox. Sony is much the same in this regard, and nevermind Nintendo.

I do wish they made Xbox more available to students in gamedev/cs degrees. When I was in school, Microsoft actually sponsored the program I was in, and I had a chance to work on Xbox stuff that as a student I would not have had any access to otherwise; That ended up being really useful to my early career -- not a lot of college grads have experience with developing and publishing for real on Xbox.

AFAIK this isn't really true. Microsoft made the GDK public, but only for targeting Windows 10/11. The portion for targeting the Xbox is still under NDA. You can get access to developer mode for your Xbox by signing a few things online and paying a nominal fee, but that level of access only lets you deploy UWP apps to your Xbox, not do native development.

There was at least one already, who did confuse it with something curl project does indeed.

I don't know what it's like on Xbox, but I recently had to set up an identity provider in AWS that delegates to Azure AD, and Microsoft's documentation claims the integration is single-click with IAM Identity Center to generate the SAML config using an app tile available in their app catalog in Azure. This may be true in general, but it's not true for their FedRamp environment, and the devs with access to Azure, which I did not have, couldn't figure out how to do it manually. This customer obviously had a paid account in Azure, along with plenty of other enterprise IT services purchased from Microsoft, and it still took four weeks for Microsoft to finally get them a scheduled meeting with a solutions architect who knew how to do what they needed to do.

I would disagree

Yes, they should make it clear that xCurl has nothing to do with real curl.

But if it intends to implement the same interface, I don't see any problem with saying look at the curl docs for what its supposed to do. I don't see how that would be much different then if you implemented your own web browser and was like, look at MDN for how its supposed to work.

"Hey, your docs say that curl should be doing this, but it's not working for me?"

<2 pages of back-and-forth posts later>

"Oh you're using xCurl, yeah you need to talk to Microsoft."

It probably depends on the reach this has within the intended audience (people wanting to deploy libcurl on Xbox?) as to how much of a support burden it will end up, but I suspect more than Mozilla has to deal with from similar situations.

If this was an Alpha software that was created a few months ago and is rapidly changing then sure, link back to the original.

But we’re talking about curl here. MS could have an intern copy paste the curl document, delete the stuff that doesn’t apply to XCurl, and host it on a shared OneDrive Word document all within a day, and it would be orders of magnitude a significantly better experience.

Instead curl maintainers will end up spending man years supporting XCurl issues instead.

Yea, but we're talking about Microsoft here. Why would you expect anything else?

I get the feeling people think MS is different now it has embraced open source.

What if you called your browser xFirefox, and suggested it would be equivalent to "Firefox" on the MDN "Browser compatibility" tables. And you have 1000x the resources of Mozilla at your disposal.

Well, you’d need to adhere to the license terms of the MPL which generally requires changes to be shared back.

You'd probably have a trademark lawyer calling you up

They should at least mirror the documentation on their own domain to make it clear that there's no (direct) link between the projects.

Based on the limited implementation surface and marriage to WinHTTP I would've imagined simply writing their own API distinct from curl's would've been a smarter choice, albeit with worse SEO to leech from curl.

...Is SEO even a thing now in the GPT era?

Then your could just use winhttp. The point of this is so you don't have to change anything in your game when porting to this platform/SDK.

fair enough in theory though I guess that didn't pan out if the "don't have to change anything" is leading to my man Dan receiving grief.

I agree it is. A fitting return volley would be to insert a note in the documentation along the lines of "derivative implementations of curl may have important differences from the documented implementation. Please refer to the derivative implementation's documentation for any behaviors that don't appear to conform to curl's implementation".

Not at all shocking coming from the same company who replaced its OS-wide help system with a browser that displays Bing search engine results.

Ludicrous and insulting, like it is saying LMGTFY.

BuT ThEy'Re ThE NeW MiCroSoFt!!

The mentioned security requirements likely have to do with certification requirements for games released for the Xbox. You couldn't just make network connections willy-nilly, there was some amount of having to declare ahead of time what ports you would be using and for what. The normal socket API is also severely limited, and largely cannot even be used without obtaining a special exception from Microsoft for your games.

I was actually at one point (years ago) tasked with determining if we could use curl in a project I was working on. There was indeed much surgery needed to ensure that the code even compiled, much less worked correctly. We ended up just using WinHTTP directly instead.

As for getting access to the source code for xCurl, that will likely not be possible without actually signing up as a dev with Microsoft and going through the whole process. And even then, you'd be under a NDA.

(My information is several years old though, so it might be somewhat outdated)

It sounds like xCurl is more like a libcurl-compatible wrapper API on top of WinHTTP rather than an actual fork of curl.

That is highly likely to be the case. I no longer have access to take a look, but from my experience, it would be silly to do otherwise. There were a handful of other xSomething libraries that they had that were effectively wrappers or simplified versions of libSomething.

Quoted in the article:

> xCurl differs from libCurl in that xCurl is implemented on top of WinHttp and automatically follows all the extra Microsoft Game Development Kit (GDK) requirements and best practices. While libCurl itself doesn’t meet the security requirements for the Microsoft Game Development Kit (GDK) console platform, use xCurl to maintain your same libCurl HTTP implementation across all platforms while changing only one header include and library linkage.

Yet the author of the article (and curl itself) seems to think it's a fork:

"With large invasive changes of this kind we can certainly hope that the team making it has invested time and spent serious effort on additional testing, both before release and ongoing."

"...since I can’t find the source code I cannot really get a grip of exactly how much and how invasive Microsoft has patched this."

It's not a patch of libcurl if they just implement a mostly compatible API from scratch.

The presence of the license strongly suggests that it's a fork.

"Fork" and "API compatible wrapper" are also not two mutually exclusive things. Often, one of the easiest ways to build an API compatible wrapper is to start by forking the original project, scoop out the internals and replace them with what you are trying to wrap, and then discard all the dangling code (e.g. features that you couldn't wire up because your custom internals don't support it).

> ”The presence of the license strongly suggests that it's a fork.”

It could just mean that Microsoft copied over the header file from libcurl without using any of the implementation.

This was my assumption reading it; it very nearly has to be exactly that to work as described by Microsoft. Which IMO makes all this a complete nothing burger.

This sort of reason is also why gRPC isn’t usable in games that also release on consoles. It comes with its own HTTP implementation which you can’t use and even if you wanted to reimplement it, it requires HTTP/2 which isn’t supported by the “blessed” console libraries.

> certification requirements for games released for the Xbox

Could it be simply that WinHTTP collects usage telemetry and MS wants to ensure their valuable feed by forcing all traffic through their library?

I doubt it. If you've ever worked on a console game, you'll know that these requirements are no joke - they're pretty hard to consistently comply with, and using APIs from the manufacturer is often much much easier than to do your own thing. It will also be much cheaper, because failing certification is unimaginably expensive - in the worst case, it can even lead to cancellation.

A lot of studios that ship on multiple platforms have abstraction layers that can internally be switched out to every platform specific API.

> A lot of studios that ship on multiple platforms have abstraction layers that can internally be switched out to every platform specific API.

Exactly this. At one point, I was the man responsible for that abstraction layer. Even most of our engine devs never even saw those low layers of the console APIs -- the single exception being the graphics peeps who by necessity had to work at the same layer I was. I basically wrote a Xbox/Playstation/Nintendo/PC libc/runtime that the rest of our engine was based on. I Imagine a lot of other studios that shipped cross-platform had some level of this as well.

The voice of the battle-weary maintainer comes through clearly in this one. I can't imagine the burden of being responsible for something as widely used as curl. Ok, I can kind of imagine it, or I wouldn't be writing this comment. That feeling of "oh no, someone has just created a support nightmare for me" is familiar, even if I've only experienced it at a minuscule scale in comparison.

As a cynical Enterprise person, my take is that some team within M$ wanted to force some other team to no longer use Curl/Libcurl for "reasons", so they had to gut it and replace pieces with some other team's cruft. There is no interest in upstreaming, supporting, extending, documenting, releasing, etc this monstrosity as it was probably an ugly kludge to meet an internal bureaucratic requirement.

This is why the MIT license is great. If it was GPL'd or something, they would have to rip out curl entirely and develop a completely in-house library, which takes more time, costs more, adds bugs, etc. On top of adapting any programs that normally use libcurl to use their custom thing, or creating a shim library (either way more work). MIT-licensed code allows corporations to still build on top of open source, and monkey-patch their own shittiness in the process. Many of us OSS devs choose MIT for this reason: we just want more people to be able to use it.

It is purely due to fact that developers on Xbox (where GDK is mostly used) do not have free access to sockets but instead have to clear every network connection with Microsoft. This is much easier to do if every http call goes through WinHttp which they can control/monitor easily on the console.

Obvious question, why didn't they just implement the security features necessary? They could have contacted Daniel to get his cooperation.

Edit: Answer is in the comments. So they should have just implemented a wrapper ontop of libcurl. But I guess the real effort here is to promote WinHttp as a replacement for libcurl. And since the developer community at large is so used to libcurl they made a wrapper for WinHttp instead. Oh lord.

Embrace.

Extend.

Extinguish.

...standard MS behavior.

As far as triple E goes this embrace seems pretty limp.

The purpose is to restrict HTTP features and where it can connect to and proxy usage to restrict "wrong" doing. (for some definition of wrong)

WinHttp itself is relatively high level, comparable to curl. Most likely this is using curl headers, but a complete custom implementation, nothing upstreamable. (Especially as Daniel probably won't like the burden of supporting a proprietary platform he doesn't have access to)

So in summary, this seems like it could be a security issue waiting to happen for games that use that xcurl lib from Microsoft, since it seems they don't keep up with upstream and will miss important patches.

How that improves "Microsoft security requirements" is beyond me, really.

My interpretation is this is an implementation of the interface, basically wrapping WinHttp for existing code that uses curl.

So security vulnerabilities in curl are not necessarily present here, since the two implementations would have matching forward declarations but little in common in terms of implementation.

> this seems like it could be a security issue waiting to happen for games that use that xcurl lib from Microsoft, since it seems they don't keep up with upstream and will miss important patches

And at the same time, missing out on the continual stream of bugs that are being put in. (I found it interesting how Daniel was able to give such a positive spin to the comment, "We merge bugfixes at a rate of around three bugfixes per day.")

This was my thought as well. Software that is as mature and well-defined as curl, I would be shocked to see three bug fixes per day. That means it is getting lots of new features / new code. That's not a bad thing necessarily, but most of the people that I know who use curl have been using it the exact same way for many years. A version that is a few years old would definitely not need three bug fixes a day

Speaking of strange Microsoft CURL variants, have you ever faced problem with HTTP2 in practically any network client (for example Eclipse strangely failing all attempts to fetch from new HTTP2-enabled Maven) that could be traced to

    C:\Windows\System32\curl.exe

failing handshakes with such HTTP2 servers, whilst any other CURL (from e.g. git-for-windows, even with lover version) could connect to the same server correctly?

The Windows provided curl is not for user consumption IIRC. If any application is using it, that’s a bug.

Do you have any evidence of this? IIRC it is completely expected for users to consume it — https://learn.microsoft.com/en-us/virtualization/community/t...

From what I recall from aforementioned incident, I was not able to reliably prove that client app (Eclipse) really used Windows' curl (I would have guessed that such software would have it's own internal stack for network connections, but my knowledge in this area is non-existent), only that it exhibited suspiciously similar problems. So I came to conclusion that it either uses that CURL or CURL that is similarly broken as the Windows' one. I was unable (or more precisely haven't even tried) to swap Windows curl with the "better" one to see if it fixes Eclipse Maven. Result back then was only that HTTP2 was disabled on the Maven server and all (windows) clients (both of them) were happy. (Other might become less happy since it presumably slowed something down.)

> I have not been able to figure out how to download xCurl in any form

I think it will be somewhere in here. It will probably get installed on your PC with this thing.

https://github.com/microsoft/GDK

edit: you can see references to xurl.dll and xurl.h here, which is GDK examples

https://github.com/search?q=repo%3Amicrosoft%2FXbox-GDK-Samp...

so the xcurl.dll and xurl.h is really probably in the installer.

If you excuse some speculation: (clarifications appreciated of course.)

If i remember correctly, ages ago (Windows Server 2003?) Microsoft introduced in-kernel HTTP-handling for the IIS web-server. I think it was for performance improvements with less copying between kernel- and user-space memory.

I suspect the for me unknown WinHTTP apis mentioned here could use these optimizations? Maybe that's why they mention security requirements? (Which would obviously be needed when doing parsing in the kernel.)

The security requirements are more due to trying to keep nasty things from being downloaded in the context of a game running on the Xbox. Games tend to run at fairly high privilege levels on Xbox, so Microsoft is probably worried mainly about jailbreaks.

The thing you're thinking about is http.sys and it was introduced in Windows 2008/IIS7 to support the new "Integrated Mode" request pipeline.

WinHTTP doesn't have anything to do with http.sys it just listens for HTTP requests and then hands them off to the right bits inside IIS.

WinHTTP is essentially a HTTP stack for client services running on Windows Server to allow them to make HTTP requests. It has a sibling API named WinINet which is aimed at use in desktop environments. I think the threading models are main differentiator (I been a while since I looked at this).

WinHTTP is fairly well documented:

https://learn.microsoft.com/en-us/windows/win32/winhttp/abou...

While doing these code things are fine, is the reuse of the name ok? That's what I would be concerned about.

Relevant: https://curl.se/docs/thename.html (page is linked from the copyright page linked in the article)

Their position on a previous name collision involving an overseas trademark sounds like "we are bigger, nobody ever confuses us for you, but we'll be sure to redirect anyone who we notice is looking for your project!"

Not sure what that says about the inverse, but since the post mentions nothing about trademark concerns, I imagine the author isn't too worried now either

I'm aware that this is utterly childish, but still wondering how far one would get, forking Visual Studio Code and calling it XVisual Studio Code.

To be fair, there is vscodium[1] which is only a few letters off vscode:

https://vscodium.com/

The tone of this article is really strange. It somehow sounds like the author is upset that someone forked his project. (There are several indicators of that throughout the post.) That's the whole point of free software. If you didn't want that, you shouldn't have chosen a free software license.

Also, as explained in other comments, it's unlikely this is actually a fork of curl. It's more likely it's just a wrapper around winhttp with a curl-like API.

> It somehow sounds like the author is upset that someone forked his project.

I didn't get that at all. It reads like he is slightly annoyed that he is getting detail requests for a half baked fork though.

He got an email requesting support for software with the same name as his, that points to his documentation, while having zero access to the software itself. I'd say being mildly annoyed at that is warranted, especially since we're talking about a gigantic corporation on the other side.

This isn't even the first time[1] Microsoft has done something like this with the curl name. And, then as now, people bother Daniel with support requests.

[1]: https://daniel.haxx.se/blog/2016/08/19/removing-the-powershe...

Yeah those corporate kindlings deeming themselves "senior developers" or whatnot have those ideas all the time. In this case the schizophrenia is that on one hand Microsoft is using unmodified curl in Windows (used by "billions"), and it's okay. But on the other hand for their game blahblah SDK the "requirements" are higher so they do need to have to maintain separate fork. That's not curable I think.

Given that their own implementation of curl itself answers with "curl : The remote name could not be resolved: '--help'" just for trying to figure out how to use the damn thing, I wouldn't count on XCurl to be implemented with any degree of competence either.

I'm surprised that they don't just use Invoke-Webrequest from PowerShell.

That has to be good enough since Microsoft insists on having "curl" as an alias for Invoke-Webrequest.

This hasn't been the case for years. Only in old versions of windows powershell.

Also, this is about the library libcurl, not a command line program that uses the library.

It's still there in the powershell 5 witch is still the default in Windows 10

Windows' Powershell is in fact an old version that they still ship because it's the last one compatible with .NET Framework. Newer versions must be installed separately (so they've ironically had more adoption in Linux).

But curl.exe should still bring the actual curl, I think.

eXtend.

Is that a reference to EEE? I feel like it is, but the capitalisation is confusing me, and now I'm wondering if I'm missing the actual intended meaning.

I think it is. The capitalization is like that because the project name is xCurl, with x. Maybe they should have called it eCurl?

Microsoft's console gaming stuff is very "X"-centric (Xbox, XDK, FATX and so on).

DirectX, ActiveX, etc

I believe that they're relating the X in XCurl to eXtend.

i'm thinking two themes here: the X in xcurl and the famous MS-ism "embrace, extend, extinguish"

Yes.

As impressive as I find the curl project, I find it almost equally remarkable that Daniel had the courage to introduce its own license.

From the sounds of this blog post, in retrospect I think Daniel finds it remarkable too!

> It is an MIT license that I was unclever enough to slightly modify many years ago

(emphasis mine)

How is changing a few words in a licence file courageous?

Because now, see, you need an army of very expensive lawyers to very carefully analyze the changes to determine if you can still be using the thing. The answer is always tricky, boils down to “it depends, now pls give us more of those sweet hourly fees and your own project description in detail so we may determine if maybe probably it’s ok in your particular case. Not guaranteed, of course, gotta have a court case to guarantee anything.”

So legally tricky, much legal busywork. Very courageous.

He's hardly the first person to do this. Often when people do this it causes real problems, as most people aren't lawyers and sometimes their edits have unintended consequences. It also tends to mean that reusers have to re-evaluate if the trivial change affects them, possibly get separate legal approval if they are in a company, etc. Tends to just cause a huge headache. Anyways, I'm sure Daniel meant no harm with this, but i still think people should be strongly discouraged from doing that sort of thing.

Reply on news.ycombinator.com