You Too Could Have Made Curl | Hacker News Reader

You Too Could Have Made Curl

daniel.haxx.se

143 pointssebg93 comments

I saw this talk in person. It was delightful. My favorite part was the stories of helping random people who contacted him to help fix things like car electronics. They found his email from the OSS license acknowledgments, which apparently are accessible to ordinary users.

(As they should be.)

Well he also got "I will slaughter you" through the exposure of his email in OSS license acknowledgements, so it's not always fun after all.

https://daniel.haxx.se/blog/2021/02/19/i-will-slaughter-you/

After reading that I thought the guy who sent that email must have been crazy, and it turns out he was: https://daniel.haxx.se/blog/2021/08/09/nocais-apology/:

> I had to retire from my career do to schizophrenia. Again, I should have not let my delusions go to the point they did nor should I have acted the way it does. My illness doesn’t detract from the rashness of my actions.

There a lot of technical word salad there, like "When I say hacked, I lost every device. They rooted my charge arbitrator, I was bios bonded and I basically lost every document and all my software."

It seems like getting projectors to run is really the hardest problem in computer science.

I've been in Computer Science graduate courses and lectures in one of the top programs in the US where it seemed like every week the first 5 to 10 minutes was spent trying to get the projector to work with the lecturers laptop.

What amazes me on conference calls is how no one can explain audio issues. Really, really smart people on a call that can explain registers, cryptography, kernel tuning.

Sounds like you are having network issues/bit errors? You are breaking up.

Sounds like your bluetooth is having issues? You are breaking up.

Sounds like they are mobile, and in a weak service area? You are breaking up.

As a person who is an audio engineer, dsp programmer and supporting such systems professionally: That is because the breaking up for the most part sounds very much the same. It is a CODEC specialized for low bitrate transmission going into low bitrate compressions, packets being dropped entirely or arriving with a delay etc.

Who knows where such a thing comes from? It could be any number of things on a huge technological stack distributed over a large geographical area.

A domain expert should of course be able to distinguish various bad conditions by ear, e.g. clipping, saturation, wromg microphone distances/orientation, signal interferences, hum, bad grounding etc.

But not all people are good at analytical hearing regardless of them being engineers or not. That is why people pay the likes of me.

To say the network guru should automatically know the cause of a packet drop is like to say the postal worker automatically knows why the letter never arrived, or a doctor automatically knows the cause of that cough.

This is mostly a consequence of the complexity of all the abstraction layers between "conference call" and "physical routed and switched connection". At best you might be able to identify that the visible/audible symptom is due to packet loss, but proving a root cause of the packet loss is extremely difficult. Through some tooling, like ThousandEyes (which I work on), you might be able to identify the hop in the path that's causing that forwarding loss, but unless you have access to that device it'd be impossible to prove exactly /why/ it has forwarding loss.

Any type of problem like that ultimately becomes a "5 Whys?" kind of troubleshooting to get to a real root cause, and from a end-user device you generally don't have the necessary access or data to answer more than 2-3 layers of abstraction.

I try to be more specific, but there's just so much going into modern sound processing. Told a guy once his bluetooth connection was failing, but the real issue was that he had a plane taking off overhead and the VC software was noise-cancelling everything.

Are those things distinguishable? I thought people say you're breaking up because they only see the symptom of audio cutting out and have no way to distinguish what the cause is.

Mediatec professional here: hard/impossible to distinguish especially, because knowing precisely which signal processing your video-conferencing solution might be doing under which conditions and in which version is even hard to know for people who analyze one version of software in a lab environment.

That is like a relative saying you are not an IT expert because you don't know immidiately what the distinct root cause of "the screen being black" is.

I just have a ChromeCast in each room that I lecture in. Easy to bounce a tab over to the big screen, and it turns on the TV/projector. It also helps prevent notification or other screen sharing related mishaps.

My biggest complaints about this approach are that I can't easily see speaker notes (I have a fiddly workaround that I can use if I need it, but it would be nice if Google Slides would support the ChromeCast use case a little better..) and that the TVs/projectors tend to screw around for 10-15 seconds before automatically choosing the right source.

would phone work?

I know I can open other tabs to look at the presentation, so I assume it would. But it's not good optics for me to have a phone out when I ask my students not to do so.

I can get a speaker notes and a normal presentation tab open and cast one of them, but they all jump around and do the wrong thing on my screen (wanting to maximize, etc) and require a lot of coercion; too much time to start a normal lecture.

Harder than getting printers to print?

Typically, you aren't trying to print in front of a large audience

Hey that's what we get CUPS auto-discovery for.

Good one!

It is unreal how tech as mature as printers is so fucking unreliable

Arguably it still hasn't matured yet. Or at least it's nowhere near to be a solved problem and we'll probably sidestep it before ever coming up with a real solution.

I'd put it in the same bucket as TODO lists. After centuries we're still not set.

Depends on the printer. I still remember the LaserJet 4000 series with great fondness. They were absolute tanks that almost never broke. You did have to replace wear items like rollers, but that was it.

At least printers give you some sort of error message.

EDID negotiation or signal integrity issues in HDMI, good luck diagnosing actual causes without highly specialized (and expensive...) equipment.

> At least printers give you some sort of error message.

Inscrutable, and infamous, though they may be:

https://en.wikipedia.org/wiki/PC_LOAD_LETTER

I used to joke it was ironic how people in management would always have their presentations up and running almost immediately while programmers seemed to always struggle with projectors.

These days however nothing seems to work for anyone.

Management is typically running a stock Windows Dell/HP/Lenovo or a Mac.

Programmers are running some custom Linux distro with a custom window manager and other customizations.

The window manager won't matter at all in the HDMI/EDID negotiation with the projector.

Windows vs the X server may make a difference. But which X11 WM you use should be irrelevant.

Technically yes, but since the WM usually comes as part of a desktop environment that also includes utilities and daemons which help with that sort of thing, to many folks "my WM makes a big difference in how easy it is to set up external monitors" is often true.

It drives me crazy that there's not a long running named process in e.g. GNOME the DE that if I run will cause my fn-volume keys on my laptop keyboard to change the volume any WM, the same as when I'm running GNOME? Why isn't that piece separated out? Or at least, where is the code so I can separate it out?

Questions and experiences like that cause many to conflate behavior and experience with WM/DE.

I once helped Andrew Tannenbaum get his laptop to work with the projector in the room. His presentation was only slightly delayed because of it, but the irony ...

This feels like such a fixable problem. Presenters should be required to show up early and test the av or send the conference their slides in advance.

The burden shouldn't be on users. Why are computer manufacturers putting out a hardware/OS combination that can't work reliably with an external display? Don't they even test this common use case?

Even my Mac (which, to Apple's credit, does work 100% reliably with projectors) still struggles when I plug in multiple external monitors, blanking the screen, turning on one external display, blanking them both, turning on the other one, and so on. This is a manufacturer who normally accepts only a smooth, polished user experience, and they still can't get it perfect. What chance do Lenovo or similar garbage-tier plastic box manufacturers have of getting it right?

Right after getting the screen layout right when you move between home office and work office :-)

Seriously should be possible to save those in a layouts pane to force adjust the monitors

Say what you want about Apple but I've never had a problem printing or connecting a monitor/project (as long as I have the correct dongle :P )

Meanwhile, "this scratchy audio is fine!".

I guess it's something one can throw AI at. It probably doesn't need GPUs, just some audio filters.

I've had good luck slapping an Extron IN-1606 in front of a projector to make all the compatibility problems go away.

A/V is the final frontier, not P ?= NP

> Several people eventually got involved, things were rebooted multiple times, cables were yanked and replugged in again. First after I installed arandr and forced-updated the resolution of my HDMI output to 1920×1080 the projector would suddenly show my presentation. (Later on I was told that people had the same problem in this room the day before…)

That's why it is a good idea if you're a presenter to always have a Decimator [1] or its cheaper and less-capable alternative from BMD [2] in your pocket. No matter what, both of them will make sure you can get at least a signal to any sink with an HDMI or SDI port. Even 20 years after EDID, negotiation issues reign supreme...

Personally, when I'm responsible for inhouse conferences, I always have the venue place a Decimator at the presenter desk and run the cable to the projector using SDI - particularly MacBooks and their USB-C dongles tend to have issues with cable lengths above 5-10 meters, I'd guess too low voltage swing.

I literally never ran into incompatibility issues with this setup, only issue I had was some dumbass thinking they could unplug the Decimator to charge their phone. Additionally, SDI makes it very easy to install a split-off run to a central recording site, even if it's 100 meters of cable run away, or a split screen at the presenter's feet. No negotiation issues, everything Just F...ing Works.

[1] https://decimator.com/Products/MiniConverters/12G-CROSS/12G-...

[2] https://www.blackmagicdesign.com/de/products/miniconverters/...

Interesting, what exactly does the Decimator do?

(Side note: I don't think it is a particularly clever idea of Decimator Design to fill their logo and homepage with images of large spiders. They are effectively eliminating around 10% of the population with arachnophobia as customers, and are making the experience of visiting their website slightly uncomfortable for large parts of the remaining population.)

> Interesting, what exactly does the Decimator do?

A multitude of things, depending on how you configure it:

- signal boosting of low-voltage or otherwise problematic HDMI or SDI signals, including reclocking. That's imposing virtually no latency and if you're running in a TV broadcast environment with centralized clocking it makes work easier for the downstream mixer. Primarily, it avoids issues with mis-designed sources such as cheap-ass adapter dongles that have signal integrity issues - as long as it's "good enough" for the Decimator to recognize it will output a clean and high-strength signal.

- format conversion from HDMI/DVI to SDI and vice versa. That imposes a tiny bit of latency and allows the usage of "consumer" equipment with broadcast equipment, or if you use two Decimators linked by SDI, you can "just" go for far longer cable runs than you could ever do with HDMI - up to 100m, in practice way more.

- up and downscaling as well as framerate conversions. This is where things really get interesting - no matter what kind of source and sink you use or what their limitations are, the scaler engine can make them talk. That includes color space conversions and bandwidth as well, and it's particularly useful if you, say, have a 12G SDI link from the source to the sink, and you want to use an older display for additional monitoring. Splice a Decimator in there and it will output the full quality signal on one output while providing a scaled-down signal on the second output.

Decimator Design excluding 10% of the population is a historically accurate decimation.

Yeah, that’s the perfect case of a good product overrun by a questionable design idea. And its name sounds 13yo too, tbh.

Yes, SDI was just designed for this kind of thing (moving video long distances between cameras in studios, tape decks and patch bays in technical rooms, vision switches and capture equipment in editing bays, etc.), whereas HDMI was only ever designed for connections of a few metres (and then active optical cables had to be made to try and fix the mess)…

I agree. I personally leave my Decimator MD-HX’s in ‘Free Run’ scaling mode at my preferred resolution. The in-built ‘no signal colour’ of my choosing helps me troubleshoot the problem in a signal chain quickly.

Worth mentioning for the novices that SDI does not do HDCP, so you can’t pass copy-protected content via this setup without workarounds.

Of course, it really should be up to the conference AV suppliers to organise all this, not the presenters themselves.

> Worth mentioning for the novices that SDI does not do HDCP, so you can’t pass copy-protected content via this setup without workarounds.

Well, the workaround is a 10$ HDCP stripper from ebay. Easy enough. (And I think that both Decimators and BMD converters also strip HDCP, but don't quote me on that)

> Of course, it really should be up to the conference AV suppliers to organise all this, not the presenters themselves.

Well, conference setups tend to be a hit and miss IME. Some where the venue and the conference is run by dedicated professionals, you don't even have to tell them to place Decimators and SDI cable runs, they'll come in with these from the start in their offer... but a lot of smaller venues and conferences (especially those ran by volunteers / NGOs) don't have that knowledge and just place some random home cinema projector together with a 20 meter HDMI cable. Corporate offices are just as bad IME, I've never seen a corp conference room with anything but HDMI or, in the worst cases, VGA. Corporate just doesn't give a flying fuck unless they got a dedicated team and a proper budget working on conference room setups...

It looks like the MD-HX is what you want for this use case?

I do a bunch of international travel, and the HD-LX seems ideal sized, but it seems like it doesn't do most of the useful functions?

It's "only" 3G-SDI so FullHD only.

The problem is not to make a tool like curl.

The problem is the marketing and getting people to use it.

You can easily make the best product and still not being able to get the word out there.

I don't think Daniel "marketed" curl?

I couldn't say when curl was released without looking it up, but I know it's been around for at least a couple of decades. There were not many command-line tools for dealing with web stuff, and curl got picked up by most Linux distributions, the BSDs, and various embedded things because it was the only tool that did what it did, as well as it did. And then it became famous essentially for being ubiquitous, like bash or vim.

"Marketing" seems to have had very little (if anything at all) to do with it.

Best-in-class vs First-in-class (not that I have any criticisms of curl). Sometimes just being first is all it takes to win.

I am pretty sure wget existed before curl, insofar as I remember people back in the day talking about curl being a newer/better version of wget. From my limited usage it didn't seem like a better version because I had to remember a bunch of switches to get it to just download the file which wget already did out of the box, but somehow it became the industry standard anyway, perhaps because of the more permissive license. I don't have any particular complaints with it either, although I still always forget the switches I need to do to make it download a file instead of print a bunch of stuff to stdout. Oh well.

They overlap but wget focuses on downloading by default, while curl is more general purpose. Once I found httpie I didn’t really use either anymore, however.

And being consistent too.

I call that "last man standing" strategy, works really well in mature SaaS markets as well.

There was a bit of PR "war" between curl and wget a long time ago. At some point curl just became everyone's goto and even the reference implementation for http.

Many moons ago, I was in the small ISP business, and I discovered the aftermath of an attempted script kiddie hack on one of our servers. When I examined the logs I realized we were extremely vulnerable to the remote code execution exploit but had been completely saved by two things: they kept trying to use curl to install the payload, but we only had wget installed; and their scripts were extremely Linux-centric but we were using FreeBSD.

That's funny, I remember the exact same thing. It was either Joomla or Wordpress for us, and we were saved only by virtue of having everything in jails and those jails having a very limited (and un-Linux-like) userland.

Curl's success has more to do with it's quality (and ubiquity) as a C library. I think the curl cli somewhat got "taken along for a ride" with all the improvements that curl lib had by becoming the de-facto standard http library.

Because the library gets ported to every new platform anyways (it's what people use so will be the first to be ported to a new architecture or whatever) so the cli gets support for everything new "for free" so can outcompete wget (because it's always _there_ and works the same).

Now, there's also the fact that the curl cli is a fantastic piece of software, but in terms of features I don't think there's that much between curl and wget for simple cli usecases (but I still use curl).

wget (and htrack) is able to parse downloaded HTML and CSS for scanning additional URLs to fetch recursively and mirror entire sites that way, while curl is "just" a very very complete HTTP client.

Though wget isn't perfect for mirroring, as it will typically download large amounts of resources redundantly only differing in URL query params when those refer to the same content, such as on typical WeirdPress sites with comment links. Ideal would be an HTTP client with http/3, auth token, and keepalive support etc. based on libcurl that can be customized, such as via "href handlers" triggered by event-driven markup parsers for SGML (though CSS and JS imports need special treatment).

I never did really figure out why curl became a thing when we’d been wget to do that since 2001.

Because MacOS included curl because wget is GPL: https://daniel.haxx.se/blog/2021/09/25/curls-first-twenty-ye...

Because you could use curl as a library in PHP and some other languages.

They're actually very different tools: wget is mostly useful only for downloading of web resources, curl is for making arbitrary requests.

Curl certainly isn't my go-to, because it's so complicated. For most uses I can just "wget http://blahblah" and it just works. With curl I have to look up the flags to save the file, etc. Curl is certainly more powerful, but that power comes with a steep usability loss for the common case.

"curl http://blahblah > blahblah.html"

What other common cases take flags on curl but not wget?

-L (follow redirects) very commonly appears in curl commands, whereas it's the default in wget without needing a flag.

If curl was created today? Sure.

But curl was something new when it came out, and it truly filled a void both as a cli tool and as a library. It needed no marketing. We needed it.

It's hard now to imagine a world without a tool like curl, but quite a few of us are old enough to remember it still.

Word of mouth can still help you, but you at least need to find the community where people have the shared problem, so their recommendations have weight

Overcoming the initial "why should I even look at this" to me is the hardest part

Amen. Marketing is so important for software, even for OS software.

It seems like the first-mover advantage really plays a significant role here.

Daniel is a unique OSS builder. We asked him to give a talk to our community and he accepted, no questions asked, just showed up :-)

His commitment to spread the word of curl and OSS is really admirable.

arandr mentioned! I use it all the time. Never had an issue.

TIL: ARandR is a tool to control output positioning and properties of multiple monitors using XRandR.

https://christian.amsuess.com/tools/arandr/

Recently set up something similar for Windows as a win in the battle against time wasted thanks to auto-logout.

https://github.com/caoyue/WindowResizer

... Having used arandr for ~forever, it suddenly occurs to me that people complaining about display issues on Linux maybe just don't know it exists? That would explain a bit

The man built a software used by billions and still gets nervous talking in public :)

It was a delightful presentation.

"Very few things are an immediate success". Wise words.

Reply on news.ycombinator.com