Twenty-five years of curl
daniel.haxx.se
daniel.haxx.se
How large would the team be behind it? How many stars on Github? How many open pull-requests?
How big would the patreon be? The Github sponsors? What Fortune100 would sponsor it?
The open issues count? The build steps? The installer packages? The docker image? The docker-compose? The rust rewrite?
Wiki pages? The reactjs website? The nextjs rewrite? The CSS framework that would be born from the rewrite?
The incomplete documentation, because of the Discord? How big would the Discord be? What about the subreddit born to cry about the lack of documentation because of the Discord?
What about the startup that would offer you curl as a service? Curl as a lambda? Curl one click install in CPanel?
How many milliseconds would curl take to just start up? How much RAM would it take?
How many developer resume would claim to be proficient at curl?
Somethings are amazing because they were made at a time where things were simply made... Not taking a stab at anyone or anything in particular, just reminiscing and imagining...
If I knew as much about programming 25 years ago as I know now, maybe I may have thought of writing something like curl or wget too and make a successful project out of it. But alas! Those days are gone. And developing a new project these days comes with so much baggage. Developing something like curl from scratch today would be a mammoth task. The HTTP/HTTPS protocol spec has become huge! And so many compatibility issues to consider and resolve even for creating an MVP that can get some minimal adoption.
What good problems are there today that curious developers like me can work on? Problems that do not have a huge historical baggage and compatibility issues to worry about? Problems that are still new and tractable enough for a single person to work on and get an MVP out?
I'll take 10% and I'll see you at the IPO!
Nothing I can think of that will make you important though.
Curl seems to pop up here amazingly often. I don't think wget for instance has the same prominence, or gpg, or many other tools.
For one off requests, yeah there’s no point to something like that.
A set of curl scripts or a .http file (httpie) held in your project's git repository is IMO preferable, most editors allow you to run these directly from the editor.
I see stuff like curl as a more advanced tool for people who already know what they are doing. In fact Postman even offers a curl "export" option that lets you see how to do whatever you have done in Postman, using curl instead. So arguably, it's also a good introduction to curl as well.
[1] https://hurl.dev
HTTP is in the base, but HTTPS is on a subscription model with payment based on volume, cipher, and number of hosts communicated with. Hosts can pay to make their endpoints cheaper to talk to, but curl adds sponsored fields to all JSON sent to endpoints unless a "clean JSON" subscription is purchased. curl of course sends telemetry to its developers' endpoint, at the usual rate, and refuses to work if that endpoint sends a 5xx or 4xx response. Better hope Cloudflare doesn't have one of its hiccups while you're debugging a RESTful endpoint!
(Telemetry may be used to ensure curl isn't employed in violating the law, trade secrets, business model, or terms of service of any business partners. All of this is strictly necessary because the curl developers have no legal or financial presence in the EU or California. Your continued use of curl constitutes your assent to these practices.)
I don't get it. Those are ethical ways to reward people for their valuable work. Are you implying people should be ashamed of this?
"Someone who is driven is so determined to achieve something or be successful that all of their behavior is directed toward this aim"
That fits with this early part of the article:
"The concept behind curl development has always been to gradually and iteratively improve all aspects of it. Keep behavior, but enhance the code, add test cases, improve the documentation. Over and over, year after year. It never stops. As the timeline below helps showing.
Similarly, there was no sudden specific moment when suddenly curl became popular and the number of users skyrocketed. Instead, the number of users and the popularity of the tool and library has gradually and continuously grown. In 1998 there were few users. By 2010 there were hundreds of millions."
That very much typifies "driven".
However, there is this at the end of his about page at https://daniel.haxx.se/about.html
## Future
I never plan never far ahead.
Good to see someone who had such a positive influence get recognised outside of our bubble
Wget to download stuff in an interactive shell (shorter, and I know its -c switch to continue interrupted downloads), curl for anything else.
Also the "2019" section made me lol:
> I started working for wolfSSL, doing curl full-time. It just took 21 years to make curl my job.
Finally: thank you to Daniel Stenberg, Rafael Sagula, and the many, many contributors for making this tool.
In November, in the middle of Covid, I finally got a US visa.[1]
[1] https://daniel.haxx.se/blog/2020/11/09/a-us-visa-in-937-days...
Edit: formating
International conference organizers ought to plan a bit in advance if they want to schedule things in the US.
… or just don't organize stuff in the US.
Of course, it's not as easy if you're not from one of the 40 countries eligible for the Visa Waiver Program. But from what I can see, Daniel is in and from Sweden, which is one of those countries.
It's apparently not that easy when the US decides it's not enough.
It's not like bureaucrats actually understand what curl does, but they have heard about "hacks".
There are a ton more people for whom it is very bad. ESTA only includes maybe 600-700 million people. There are probably 7 billion people for which it is hard to very hard to get any US visa.
But I should not have said "most people". And now knowing the context of Daniel's situation, my comment was probably unnecessary in general.
That was not really my intent.
My comment was quite snarky, this was the intent. I got a visa myself quite easily, but a lot of people actually argue for not organizing conferences there because of visa issues.
A former colleague of mine wanted to present a paper and got his visa issued after the conference, most likely because his passport is a Lebanese one. He's been living in France for years. His PhD supervisor had to present for him instead. This colleague wont best paper awards but no, he is not going to present them in the US.
My (more) serious argument is yes, it is this bad, and good enough for a lot of people is not satisfactory. We need places that don't arbitrarily exclude people so much. The US needs to fix this visa issue and until then, it would be best if events were organized elsewhere.
Not all interesting people doing research or involved in programming are American or EU citizen. This happens everywhere, including from countries if you happen to leave have set foot in or lived, you are practically banned from going to the US.
If I ever go to Iran, I can say goodbye to ESTA for the rest of my life. Even if it's because of a plane issue and the plane needs to do an emergency landing.
The US are too unreliable.
Of course.
So don’t.
Bringing this up unmprompted is a bad-faith response. As if someone needed to be reminded that two unrelated things are indeed unrelated.
-- Curl claims that it supports so many protocols but I do not see many examples where they are in action, for instance, how to use ssh protocol to perform transfer between two machines? Where can I find such examples?
-- Are there any examples of a fully functional high-level data motion tool that is based on Curl?
-- Has anyone used Curl to perform data transfers at large scale such as terabytes of data? I would love to read more about it if there are any pointers describing such transfers.
That's out of scope for cURL, use `scp` for that.
> Are there any examples of a fully functional high-level data motion tool that is based on Curl?
What's data motion?
> Has anyone used Curl to perform data transfers at large scale such as terabytes of data? I would love to read more about it if there are any pointers describing such transfers.
There's a few posts on Stack Overflow about that, but they go up to gigabytes. cURL is not the best solution for transfering terabytes of data. http://moo.nac.uci.edu/~hjm/HOWTO_move_data.html seems to be a good overview of the challenges (and tools) to use to securely and reliably transfer terabytes of data, referencing rsync, bbcp, bittorrent, etc.
You could try it yourself! The man page is pretty good.
SSH file transfers use either SCP:// or SFTP:// urls, not SSH:// urls.
>> curl sftp://public.example.com/file.txt