Beej's Guide to Network Programming
beej.us
beej.us
If you search Algolia for Beej you'll see his material has been on hacker news numerous times.
So, BGP, autonomous systems, peering agreements and other site-to-site routing protocols that keep the entire Internet infrastructure ticking along.
Some bookmarks I have regarding BGP and peering that dives into it a bit:
- https://www.fortinet.com/resources/cyberglossary/bgp-border-...
- https://aws.amazon.com/blogs/architecture/internet-routing-a...
- https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/iproute_bg...
I found Cantrill’s course pretty solid. It is free and you can just skip to DNS & DNSSEC.
Or, as I just recently learned, you can click on the (beej.us) domain name next to the title at the top of this page and it will take you to a list of all the HN submissions for the site:
I’m generally available to help out with email questions so feel free to write in, but I can’t guarantee a response.
I lead a pretty busy life and there are times when I just can’t answer a question you have.
When that’s the case, I usually just delete the message. It’s nothing personal;
I just won’t ever have the time to give the detailed answer you require. [...]
That's how you do it!
Tssk tssk. Big mistake. Always do the "If it's urgent, contact <someone else>" approach ;)
[1] at the beginning of each month, May 2023 https://www.youtube.com/watch?v=h_GD4E27Bt8
We don't have 64 Kilobytes of limited memory on our email servers anymore.
You should understand when they say "delete" that they mean "remove from their queue", whether it still exists in an "archive" folder, backups, trash, ... is irrelevant.
OK but providing value isn't the same as having power. It's almost the opposite thing.
Providing value is absolutely a way of having power. You change the world, that's power.
Having power isn't the same as yielding benefit (for yourself).
Beej's teaching power only answers "Yes" to the first question here - and that's only if your goals have something to do with teaching or improving the world overall, somewhere, for some people.
Power that isn't attached to the desire to reap a direct benefit is a strange thing to witness, but it's power nonetheless!
It's whether or not a particular type of power you have can help you and yours in adversarial situations.
Can your Power of Teaching help you if a local thug wants you gone? Or, can your Power of Teaching help you when local governor decides to close down and raze your school, to be replaced with a factory? In both cases, someone else is exercising their power to directly or indirectly take away your livelihood (or life) - can your power prevent it, or at least mitigate the blow?
No power is good against all other powers, but some kinds of power are better at this in general. For instance, a wealthy businessman or politician could answer "Yes" to both scenarios above (replacing "school" with closest equivalent) - their money and influence can buy protection and work as a deterrent against the thug, and it can outweigh the influence of the local governor (or just allow for an effective bribe).
> I think beej's teaching power fulfills the second question in a very radical way which is that it is an act of caring for a large audience of strangers and does improve their lives.
Caring about large audience of strangers (or humanity in general) resonates with me, so I'll give this a passing mark here :).
> Power that isn't attached to the desire to reap a direct benefit is a strange thing to witness, but it's power nonetheless!
True. What I am trying to highlight here is that, when people talk about "power", in the sense of seeking more of it, or comparing people by how much of it they have, they usually don't count magnitudes of raw power, but rather how generic and how controllable it is.
This is similar to wealth, btw. Having $1M in cash makes you more wealthy in the immediate term than someone who has $1M in stocks, equipment, etc., as that other person won't get anything near a $1M if they try to convert their assets to cash.
Anyway, in an age of information abundance and videos, educators are overvalued if they just give instruction alone. It's the whole package, the checks and balances on the student, that possibly provides some sort of value. Then again, there are irregularities, like the programmer who's never been to college who contributes as much or even more.
Beej's Guide to Network Programming (1994-2020) - https://news.ycombinator.com/item?id=26100075 - Feb 2021 (165 comments)
Beej's Guide to Network Programming - https://news.ycombinator.com/item?id=13983212 - March 2017 (44 comments)
Beej's Guide to Network Programming - https://news.ycombinator.com/item?id=13670971 - Feb 2017 (6 comments)
Beej's Guide to Network Programming - https://news.ycombinator.com/item?id=12402313 - Aug 2016 (4 comments)
Beej's Guide to Network Programming (2012) - https://news.ycombinator.com/item?id=9445692 - April 2015 (31 comments)
Beej's Guide to Network Programming - https://news.ycombinator.com/item?id=7584974 - April 2014 (1 comment)
Beej's Guide to Network Programming - https://news.ycombinator.com/item?id=5241220 - Feb 2013 (47 comments)
Beej's Guide to Network Programming - https://news.ycombinator.com/item?id=584557 - April 2009 (22 comments)
Guide to Network Programming - https://news.ycombinator.com/item?id=337371 - Oct 2008 (7 comments)
I will always bring up “but you know who the real beej is, right?” if the other person is a nerd.
I miss WON. Man my friends and I hated Steam when it came out...
So, to me it was beej. But in reality, to the public [you can’t nickname yourself].
So it’s always been b33j0r. My friend who had the first dedicated server I encountered named me that because my name is Brian Jor—-
Finding me has been trivial since I first AWP’d in a CAL-O match. This was before scoutznknives were more than merit badges.
1. A library exists to abstract over the bytes-into-sockets layer, meaning I don't need to deal with it; or
2. A library probably exists to abstract over the bytes-into-sockets layer, but I've chosen to ignore it either because I'm truly ignorant of it, or for intellectual stimulation.
(Note that in category 2 I count things like "not using protobuf/thrift/etc. for IPC.)
When are these things not true? Genuinely curious! I'm guessing interfacing with existing obscure third parties, or using obscure language environments, but what are examples of that? And what other usages are there?
E.g. interfacing a research machine with an OEM machine or a machine to a standalone embedded target, sometimes it makes sense to just serialize, send over a socket, ethernet crossover cable and be done.
How would you implement, say, netcat? Launch a jango rest service with kubernetes, which will run in a wm if it's a wrong OS, create a certificate with letsencrypt and access it with a react electron app?
E.g.: Learn the basics Transport Layer protocols (TCP/UDP) if you work with HTTP
2. Speaking unusual protocols for which (good) libraries don’t exist.
3. I think interacting with the sockets api can also be relevant for tweaking various options to get better performance (though often there are Linux defaults that can be tweaked instead)
i also wrote a similar event-driven nonblocking http client in c for benchmarking and load-testing both that server and a different compatible one being written by another team in c++. this was nonblocking because it was important for the client to be faster than the server. at the time there was no libevent/libev/libuv so i wrote my own
at satellogic i maintained the upper layers of our cubesat space protocol stack, which provided a socket-like interface to our coap stack, which i also maintained part of
i also had to tweak kernel socket buffering settings for rabbitmq there
for wercam i'm experimenting with different approaches to ipc for efficiency; the data being transferred are pixel buffers so i don't think protobuf/thrift/etc. is going to help
for the mail server i share with some friends i wrote a milter in python to smtp-reject outside email to certain recipients unless it matched a whitelist; milter doesn't speak pb either and i couldn't find a python library
i wrote a multicast file transfer program in python (bccpo/bccpi) for copying files across the lan without having to fiddle around with ip addresses; the data being transferred is just (multicast) the ip and port and (unicast) the filename, file size, and file contents, so again i'm not sure pb would simplify anything. the hard part of getting it working was just multicast
i wrote a chat client for icb in python with asyncore, and again i couldn't find a suitable library
even when i'm using a library, it's often pretty useful to be able to read strace output or use select or poll with the file descriptor it's trying to hide from me
i don't think any of these qualify as either your category 1 or 2
i've also done a huge number of things that fall into category 2, of course, like writing an irc client in bash, writing a web server in assembly, and writing a chat server in three 80-column lines of c, writing a mud, and so on
Writing servers in Elixir is straightforward because of the foundation exposed by Erlang's gen_tcp and gen_udp libraries. When you want to take your TCP server up a notch, try TCP acceptor pooling. There are two viable TCP acceptor pool libraries: Thousand Island, written in pure Elixir, and Ranch, written in Erlang.
Because of these well- abstracted foundation libraries and acceptor pool libraries, it's possible to write robust servers without venturing much into low level network programming. Beej explains the lowest level details well. I've done some of the equivalent work in Rust but found the slow progress of my work unsatisfying and moved back to Elixir quickly.
[1] protohackers.com
Beej possibly tops that list.
And I couldn't be happier, Beej's guide is a seminal read for anyone looking to get into net programming. Exposing as many newbies to it as possible helps strengthen the field.
You certainly can program raw sockets, the techniques are almost orthogonal to the beej guide. It's the same API and Structs (using FFI) and you would change some glue logic to make it more rust idiomatic. It would just be less useful in those languages, generally.
I followed Beej's linked guide to transfer data over udp sometime ago, written in C on a Windows machine. I then compiled the code on Mac os x but could never get it to work properly. Tried all the Google suggestions at the time, posting this comment in case there's an obvious "oh did you set X?" tip from someone that has done a similar task.
Genuinely inspirational hacker. Feel's great to have had that experience at a state school.
One of my favorite modules he designed was on computer architecture. We simulated the CPU and registers in Python (they switched from C right before I got there), and each class extended our instruction set.
Teaching his modules taught me enough that I was able to start the MSCS program at UT Austin.
First, CS programs that went 100% into Java did not age well and most of these courses were taught in a way that led students on to believe this stuff was static and won’t really improve. The concept of new languages beyond what was available in circa 2006 was unimaginable.
Another quirk that a Java-centric education brought on was that low-level courses were often taught in Java and you ended up writing all your data structures coursework in Java taught by someone with a C background. But you couldn’t use C or anything sensible for this because universities promised students would know object oriented programming. This created a lot of disconnect and meaningless ritual around a paradigm that did not fit the language.
CS programs that succeeded understood that a foundational knowledge coupled with language flexibility. They had to understand that the specific languages did not matter so much, rather it should fit the paradigm and principles of the context in which it was written. Once students are able to think in terms of computer procedures, instructors can leverage this knowledge to introduce abstractions and a grand unification principle.
It's more or less magical when you find just the right conjugate match between tutorial material and your own learning style. I can see how Beej's Guide might do the same for someone new to both socket programming and C.