I sent an Ethernet packet
github.com
github.com
> Strangely, I've found that there is a not-insignificant number of people are against this - especially in a professional environment. I think this has something to do with the JIRA-fication of development process, where all work is divided and sub-divided, and anything that does not directly check a box and produce a deliverable is wasted effort. It's disheartening that there is a whole group of people - developers and managers alike - who do not understand or see the value in exploration as part of the process.
So true, being able to create your own small tools is a superpower which is often at the heart of 10x programmers. It is sadly an art often practiced in the shadow.
That said, writing your own tools if applicable is useful, however I'd argue that a productive developer writes the least code and instead uses what's off the shelf. To use this article as an example, nobody should be writing their own TCP stack, there are perfectly fine existing implementations.
That said that said, writing your own stuff is probably the best way to get a deep understanding, and a deep understanding of things is a constituent part of the mythical 10x developer. I just wouldn't expect any employer to pay you for it.
But sibling comment is also right. The 10x developer has more "free" time outside of their Jira tickets. Some choose to focus on the next business problem, others drift and experiment.
Either way the business problem gets solved and you retain a very skilled employee. Their explorations may turn up something valuable, or just add to their own personal dragon hoard of skills, when then usually still benefits you in the end
But if you're not that person (and I like to think I'm not) and you are curious and stop to think about second order effects, future features that are a little obvious, and so on ... it's really painful working with those developers who do not.
I worked with one recently, not a bad person at all, but now I'm cleaning up his project that 100% did exactly what the tickets told him to do. And the project that he worked on that was designed to replace a piece of software that had issues ... has every single issue the old one had for all the same reasons (someone coded it in the easiest possible way + only what the ticket told them to).
So frustrating.
I will say this though, I find the use of the term "10x developer" a sort of buzzword smell for the kind of reckless developer with no oversight who "gets the job done" at great cost ... because they can do what they want and then everything is siloed and potentially a disaster when anyone has to touch it or they bolt when it becomes clear their tower of cards is ready to come down.
Not to say I disagree with your statement generally, just that term tends to make me worry.
1. From the perspective of the money folks (managers & up): they get their jobs done very quickly, thus spending minimal time, meaning minimal money.
2. From the perspective of excellent systems design folks: they design an excellent system that addresses subtle problems and foresees and addresses system problems that will occur down the road.
Type #2 folks tend to not add to the org's technical debt; Type #1 folks don't GAF about such considerations -- they meet their quotas, get their perf reviews, and are on their way to their next promotion/position/project.
I'd say there is excellence for excellence's sake, and then there's money-making excellence. Sure, orgs need to make money (or at least break even) to survive, but there's a lot of overlap where being too much one way or the other will be an impediment to long-term viability.
"That’s not very typical, I’d like to make that point."
In this case, the managers knew that if they went back and removed the door plug then it would have to be inspected again. That would cost time, and the plane was already behind schedule. Their bonuses were in the line, so they got together to find a solution. The solution that they found was to skirt the inspection rules using semantics. They decided to have their employees loosen the bolts, shift the door plug a little, do the required work, and then put everything back. This allowed them to claim with a straight face that they hadn't ”removed” the door plug, and therefore it didn't need to be inspected.
Yes the root cause might stem from management but good engineering would not have the doors flying off... thus bad engineering. Regardless of everything else, engineers are responsible for their designs at the end of the day. (Yes when management only approves cheap unsafe designs)
Otherwise you are "just following orders" which is not a viable leg to stand on.
Second, the steps for assembly of an airplane are all very important. If any of them are skipped or left out somehow, the plane will break! You can’t engineer your way out of this problem either: the more ways you add of attaching that door plug to the airframe, the more possibilities there are for mistakes. That’s why the assembly process requires one team to install the plug and another team to verify that installation was completed correctly and according to the specifications.
Any time you have managers using semantics to weasel their way out of the inspections that verify that the plane was assembled correctly, that’s the mistake. Full stop. Fire those idiots.
https://x.com/SpaceX/status/1819772716339339664
I think what happened is, Boeing codified all these labor-intensive manual processes back when they were riding high. The planes were selling well, they were state of the art. Now, 30 years later, it takes the same amount of effort to put everything together without mistakes, but the relative value of the finished product is less.
Not only that, but it's also really painful working in a system which doesn't really value your artwork because you're not on the product team, and shouldn't be making product decisions.
"Wow that's really helpful. Your tool confirmed our model."
And
"Did you ask the PM first? You have to be careful with rabbit holes."
I usually just do it if it'll be under a day's worth of work. It's never gone wrong and never gone to waste. At worst I and up copy/pasting the code into something else later
Example: I used to work for a company that basically made slot machines. We needed to produce something called parsheets to give to submit to get our slots certified and to the casino. I was given the project but no manager wanted to give any developers to implement it. No one wanted to give the money to buy a computer with enough power to run simulations to produce the theoretical hold. But this was something we NEEDED to help get our games approved.
To steelman, because, as Frank is sending ethernet packets, we're stuck picking up the slack, doing things that are part of the job description, that are needed by the org as a whole. Why doesn't he just innovate in the actual problem we're working on, since there's a near infinity backlog?
I think, ideally, everyone is given exploration time, but that requires a manager that can strictly defend it, even when upper management is made aware of it and inevitably says "we need <project> done sooner". It's also a problem when other managers become aware of it with "You're allowing that team to hire more to compensate for the "lost time"? We need those people!". It really needs to be an org level thing, which even Google gave up on [1].
"Unethical" solution: Pad your development time to include some of it.
[1] https://hrzone.com/why-did-google-abandon-20-time-for-innova...
You may be on a team that has decided "developer burnout" is just an inevitable and acceptable cost of business.
> since there's a near infinity backlog?
Which is a problem in and of itself. In any case I'm only going to be able to give you like 4, real, solid hours of work a day on that log. The other 4 will be team coordination, managing that log, and stress management maybe while I hate eat my lunch.
> everyone is given exploration time
Call it "skill investment time." We're in a fast moving industry and keeping heads down for too long is destructive personally and organizationally. It's also the pathway to getting more than the basic level of engagement above.
No. Burnout is a work life balance thing more than a "doing work at work that is related to the business" thing. Fun can be had outside of work hours, just as everyone else who is employed handles it. You can give people meaningful work they're interested in and still keep it related to the actual business.
> Which is a problem in and of itself.
Absolutely not. If you don't have a near infinity list, then that means you don't have a roadmap. Why aren't you thinking about the future?
> Call it "skill investment time."
Yes, and this can always be done in the context of the work. If it's not related to the work, or the business, then it's not an investment for the people paying you.
It's more than that. Perhaps you haven't been in a position to accept a lot of resignations in your career. Burnout is most definitely implicated by the work place itself, specific work assignments, and general office culture.
> don't have a near infinity list, then that means you don't have a roadmap
Are you talking only about startups? That would make sense. In the more general case I doubt your assertion here. Does this fragment truly sound rational to you on a second reading?
> and this can always be done in the context of the work.
The very article this thread is attached to clearly shows how false this is.
Then he started asking me to make them accessible to non programmers, and suddenly those tools seemed a lot more than I bargained for.
(I was hoping it would be about bit-banging Ethernet on some improbable bit of hardware.)
Just 10Mb/s can be done even dirtier. I'm still waiting for someone to combine an RP2040 with DVI/HDMI video output and Ethernet into a "modern" glass terminal (just telnet, SSH is probably asking too much).
I made half a terminal with that output, aiming for something that ran the original VT100 ROM. I never finished the emulator part, but I did write video output (including scan line emulation) for VT100 memory -> VGA. Adding SSH once the rest works should be perfectly possible. (Not bit banging ethernet at the same time, but using some external chip.)
I should probably put that online somewhere. Or finish it and then put it online.
https://hackaday.com/2014/08/29/bit-banging-ethernet-on-an-a...
The most useful resource while I was learning was RFC1812 "Requirements for IPv4 Routers" [1]
Its an ancient document written the same year I was born detailing how future routers should be built on this relatively new thing called the internet. The language is highly approachable and detailed, often explaining WHY things are done. It is an awesome read.
To be honest you don't need to finish it. I only read the first few chapters, but I googled EVERYTHING I did not understand. The first few paragraphs took several hours. Talk to LLM's if you need a concept explained. Take notes. In a few days you'll have a very solid grasp.
Also,
> Due to the complexity of the signalling involved with ethernet, a dedicated ASIC is generally used
In the context of microcontrollers, I think this is generally not true. In most cases Ethernet functionality is incorporated as a embedded peripheral within the microcontroller itself, as it is with the STM32F407 and the ESP32 (both of which happen to use the same Ethernet peripheral sourced from an external provider).
But his stated goal is to build a TCP/IP stack, not futz around with SPI and the particulars of an idiosyncratic network chip. There will be plenty of work (and learning) to do once he starts climbing up the network stack.
Over a decade ago, when I was just learning Linux, I set out on a quest to turn a CentOS box into a router that could perform NAT between two different networks. I spent an entire weekend researching and following every suggestion I could find until it finally worked. I was so proud when my pings reached their respective destinations.
I took it apart the next day and never did anything more with it, but the journey and the reward was the fun of it.
Be the change you want to see in the world.
I'll look into the STM32F407 for the main series. Thanks
It's not too bad as far as such things go. The documentation on how the DMA system works leaves something to be desired, but it's not bad (and it's a heck of a lot faster than spitting packets over SPI).
Packets are an IP concept :)
But RFC 791 dates from when the underlying L2 network was most likely ARPAnet, which had 128 byte packets. Today the distinction is (or should be) moot - if you're relying on IP fragmentation to send your large IP datagram over multiple L2 packets, you're doing something wrong. IPv6 doesn't even support in-network fragmentation.
In practice, the distinction has largely disappeared over time. Everyone I know talks about packets and very rarely bothers to distinguish between frames, datagrams and packets, because in practice one Ethernet frame generally carries one IP datagram, and everyone calls it a packet.
Yet the article uses frame throughout where technically correct, and what was important to the author wasn't that any frame was transmitted, but that they got their packet to show up correctly.
Their 'STM32Cube' software gets mixed reviews, but it will spit out a working example of ethernet communication.
[1] https://www.st.com/en/evaluation-tools/nucleo-f439zi.html
Thankfully there's more than one CMake projects on github as an alternative.
https://liliputing.com/waveshare-esp32-p4-nano-is-a-tiny-ris...
Edit: I just saw there are way cheaper modules, although probably not as performant, like the WT32-ETH01
Sometimes. On some hardware it's broken.
When I last used Cube-MX, it was a very unpleasant experience throughout. I'd use stm32-hal or libopencm3 out of preference if I was using the parts again. The tool itself and the code it spat out had all sorts of nasty bugs and edge cases that cost days of debugging. Maybe it's improved since.
I recently discovered the ESP32 and my only complaint so far is that Ethernet (and with it PoE) is not a first class citizen of the platform.
SOCK_RAW packets are passed to and from the device driver without
any changes in the packet data. When receiving a packet, the
address is still parsed and passed in a standard sockaddr_ll
address structure. When transmitting a packet, the user-supplied
buffer should contain the physical-layer header. That packet is
then queued unmodified to the network driver of the interface
defined by the destination address.
It took me a while to realize the data consumers (those who need to look at and understand data) didnt have any orchestration tools, except a creaky UI cron type of scheduler.
For my work I refused to compromise and spend all my days clicking in a UI, so I searched around for a good tool, decided on Prefect. It served me well, but I didn't realize going in that Prefect/Airflow/Argo etc are really orchestration engines and you still need to write the client software around it to serve your own (or the team) productivity.
For example, connections to sources, methods to extract from source with a sql file, where to put the data after, injecting into publication channels, etc. I all gradually ended up writing functions to support this for my own personal IC work. I sank a ton of time into learning, had to learn how to develop python packages (I chose poetry), on an on. And yet despite all that time spent grinding away on indirectly productive learning I have still been more productive than my peers.
It was a mystery so I spent time learning about my peer workflows, which people are so cagey about at my company. Anyway, everybody just crams 100s of lines of business logic into Tableau custom sql sources to avoid scrutiny of data artifacts like tables they might create on the data lake. I guess these are tableau flavored views, but its so hard to read and understand all this logic in the UI -- oh and the calculated fields also.
I guess to sum up, if I am keeping up with peers using the latest expensive enterprise 'data storytelling' service while I self educate on basically data engineering to stats presentation plots, spark, julia, python, etc. I think I have concluded that Tableau should be considered harmful.
[1] https://www.kernel.org/doc/html/v6.9/networking/af_xdp.html
I was going to implement TCP, but got bored (it suddenly becomes way more difficult)...
It's just so amazing to me to behold the entire network stack and the fact it actually works (even if IPv4 is clinging on at this stage).
That's a microcontroller. I wonder if more powerful chips can do more.
The PIO allows for single-cycle or dual-cycle data output, which essentially just pulls bits from a FIFO and puts them on some pins. The "side-set pin" makes it really easy to generate a clock signal as well. If the target devices samples on rising edge, it becomes essentially "output data, clock low; nop, clock high; implicit repeat" or even "output data; implicit repeat" if the target can do its own clock recovery or uses DDR. This is great, because you can output a lot of data at quite high speeds.
But the opposite is a lot harder. The PIO has no native clock support, so you're left simulating it yourself with a multi-instruction operation. If you need even as little as two instructions to receive a bit, that 133MHz MCU clock already limits you to a 66MHz signal! In theory you could use an external clock input - but those pins are limited to 50MHz. You could also feed the external clock into the main oscillator input - but that breaks a bunch of other functionality, especially if the external clock isn't reliable. Oversampling isn't an option either: you'd have to read the pins at 4x the signal frequency (limiting you to 33MHz) and dedicate a lot of computing resources to that.
In other words, high-speed input is only really an option if you don't care about synchronization, such as a free-running logic analyzer. But bitbanging 100BASE-T Ethernet? Don't count on it. Even the 2-bit-wide 50MHz RMII interface has proven to be quite challenging.
Now that I think about it more, you're right. Best case scenario for reading a bit would be to wait for the pin to go high/low, then push the corresponding bit into the input shift register, and have that automatically flush to DMA every 32 bits, say. Can't do better than some fraction of the clock speed because of the multiple instructions needed.
Ref: https://en.wikipedia.org/wiki/Undersampling (funny enough, this article explicitly calls out the FM radio use case).
100GHz is .. well, can anyone find any part with that capability? Seems to top out around 10GHz for ethernet transceivers.
When it comes to tcp connections linux seems more robust.
I don't want to sound bad, but it's a bit like writing BASIC programs on C64 today? Fun, entertaining, but is it actually useful for developing skills in this domain?
Yes. Yes, always. You always learn, no matter what you do. Why build an 8-Bit CPU if you we have very complex 64 bit ones nowadays? Because the fundamentals are mostly the same.
Pretty much by definition, the engineering required for "high performance" anything is completely divorced from the knowledge required to implement basic systems.
Doing Ethernet at the logic design level is a lot of hard work but it isn't exactly a mystery at this point unless you are working on next generation speeds and feeds. A book on digital design, a book on HDL of choice, and a willingness to read specifications (IEEE 802, SFP MSA) and datasheets (FPGA MAC, third party PHY chips) is what you need to be successful.
NetFPGA collates a lot of teaching materials and example code for this but the major FPGA vendors also have relevant works. Ignore any suggestions that you need formal education in the space, it's simply not true.
The moment I received my first packet on a cut-up wired headset I used as a transceiver makeshift tenceiver it felt like something clicked and I just began to understand more how the universe works. I recommend projects of this type and wish more folks did it.
I wrote a little framework (bootloader, utilities, etc.) to allow for updating firmware and tunneling stdout on AVRs over pure Ethernet.
I also had (a different) issue with SPI communication (the W6100 uses a little different protocol).
I still couldn’t get any Ethernet packets to send/receive with a real machine, so I did basically the same debugging step: bought an off the shelf W6100 dev board and planned to use their arduino library to compare against. That’s where I left that project.
I admire this kind of low-level, first principles exploration which I personally don't do enough of.
I'll never forget Packetfu (https://rubygems.org/gems/packetfu)
It was really fun and surprisingly helped me get ahead in the networking class I'm taking.
Like others in this thread I was hoping I’d see an RJ45 breakout board directly hooked into an STM32 and doing onerous bit banging and keeping it as low level and basic as possible.