Framework Laptop 2.5Gbps Ethernet Expansion Card
frame.work
frame.work
I don't think it's entirely a Mikrotik issue as I believe this is just a physics thing - you're firing off one port at a certain speed so the buffer of the other port fills up too quickly unless the faster one slows down (resulting in excessive packet loss.) But it looks like Mikrotik has the most complaints about this.
The solution appears to be to enable Flow Control, but it's never clear in which direction or which port it needs to be enabled on, and I haven't really had any success with any combination.
[0] https://forum.mikrotik.com/viewtopic.php?t=182691
[1] https://forum.mikrotik.com/viewtopic.php?t=181881
[2] https://www.reddit.com/r/mikrotik/comments/rq7ytu/rb5009_25g...
[3] https://www.reddit.com/r/mikrotik/comments/rza5u3/slow_multi...
I swapped a more expensive "low end" (for the family) Broadcom Trident 3 based switch in in place of 2 different 10G MikroTik boxes and the mixed speed transfers are now behaving exactly like one would expect without any flow control.
I'm curious though: Going by your statement, wouldn't even a large buffer fill eventually? And by eventually, in computer terms, that might be milliseconds/a few seconds if we're talking transfer speeds hundreds of megabytes or gigabytes per second. How big is a buffer supposed to be exactly?
There are endless rabbit holes of details though. You want a buffer with at least 1 hardware queue per network queue (in advanced situations you may even want a hardware queue set per flow) and you want these queues to be intelligent - priority queues, weighted round robin queues, best effort queues. In each of these queues you also probably want things like WRED which counterintuitively helps avoid congestion by starting to drop packets _before_ the buffer is full. You also want the buffer to be as "flat" among interfaces as possible, e.g. if the switch has 12 interfaces it's best it's all 1 line rate ASIC with 1 shared buffer not an 8 port + 4 port with interconnect and it's own set of internal QoS. For non local things latency also starts to become as big a factor in buffer sizes as interface speeds.
I forget what I ended up finding the buffer size of my MikroTik switch to be but I remember thinking it should have been 4x-8x the size - whatever it was it was truly tiny and didn't seem to be operating well.
As for why MikroTik makes things with known too small buffers... it's cheaper than proper high speed interfaces but more performant than just using low speed interfaces. Sure, it's not a perfect 2.5 Gbps solid but for the price it'd take for them to do that "right" they would be able to do 10 Gbps ports "wrong" instead. I still use my MikroTiks though I just make sure not to mix interface speeds or if I do it's with something on the side where I don't really care about having perfect performance to it just a generally fast connection when I do use it.
Buffer counts and buffer sizing is also a notoriously cut-throat game in vendor marketing, almost on par with (mis-)use of the term "wire speed".
This is not an easy/cheap problem to solve, and I fully agree with the conclusion: Mikrotik hardware is good up to the point where the price/performance curve falls the wrong way in a particular situation.
I also get a lot of use from Mikrotik hardware, particularly the passively-cooled models where performance isn't as important as price + silence.
And yes, make your buffers too big (we had relatively tons of memory) and all you get is weird latency cycles as the higher level protocols over-send, then fall back. Make your buffers to small, and the total throughput falls off, bu the latency is far more 'normal'. We ended up buffer scaling based on what the upstream connection really achieved, so larger buffers for a 1Mb/S link than for a 128Kb/S link. We would start the router at 1Mb/S buffer sizing, and scale down if the DSL link throughput was less than that.
At the time we were working through that, a number of people would scream 'just give it more buffer', not understanding how badly that broke something like TCP.
Unless overly large buffers are covering up the real problems with TCP?
Having too large of a buffer, particularly on gigabit class software routers/nat points which use CPU+RAM instead of expensive hardware, is definitely the more common problem. There it's too easy for the trap "I have 512 MB of RAM, why shouldn't I let as much as is available be used as buffer, it's helping" which then makes you wish you had 4 MB of buffer instead.
We're probably going to be in for a bit of pain until everything can reliably function at 10Gb/s
It's kind of infuriating, to the point I bought a separate dedicated 2.5G switch with 10G uplinks and plugged that over to the Mikrotik with a DAC.
Transceivers matter. This is the one I used: https://www.amazon.com/gp/product/B06XQBFHNL/ref=ppx_yo_dt_b...
What you want is a small amount of buffer, but when the WAN is sending packets too fast, they should be dropped and TCP will figure it out. But maybe the buffers are too big and you get latency spikes. Or maybe something else is funky when packets are getting dropped?
Flow control usually causes more issues as things will get delayed or dropped in bunches.
This is something I really want to do, actually.
In general though unless you need >10G you'll come behind on (good) COTS offerings in price and performance. Particularly if you need features like NAT or firewalling where software starts adding latency or performance cliffs at certain intervals while things like a low end Fortigate have high levels of hardware offload.
I wonder why there was never a standardized PCIE 'card' format? Seems like the perfect thing for extra storage with NVME being so common these days.
The best thing about standards is that there are so many of them to choose from. The closest analogue is the ExpressCard (https://en.wikipedia.org/wiki/ExpressCard), but there's also the internal-only M.2 (https://en.wikipedia.org/wiki/M.2), which is the most common non-server format for NVMe; mini-PCIe (similar to M.2, used mostly for wireless); SD express (SD cards, but with PCIe); and CFexpress (CompactFlash cards, but with PCIe).
It would have been nice if framework adopted this existing standard and started making their own express card modules instead of making their own USBC thing that doesn't fit in any other laptop.
for those that haven't seen these thick credit card sized adapters from the last century..
Harder times, sure. But also simpler, because at least the mainstream distros back then weren't so damn convoluted and it could never take me more than 30 seconds to change the DNS servers permanently. I set up NextDNS on my Pop OS laptop recently and it took me 2 hours of dicking around with systemd.
Starting to yearn back to Arch Linux again... At least with that I actually know what I'm running under the hood...
What can tb do that express card could not?
The best part about EC was that there was no "handshakes" or any other software layer nonsense like you get with Thunderbolt, it was just plain old PCIe and it just worked. Every device you plugged in was fully "native" to the system and didn't require any screwy stuff to work right. SAS controllers just worked. 4x network cards just worked. Serial cards just worked.
These days, my old Dell workstation is on its last legs, it's looking like EC is a dead end. It's a shame, because I doubt we'll ever see that level of simplicity for connecting random devices to our computers again. Yes, the connector sucked and the form factor was awkward, but it really had a special place to me.
[1] https://images-fe.ssl-images-amazon.com/images/I/51cyhJ95LsL...
Its real replacement is Thunderbolt though! That's basically PCIe.
The framework adapters are grooved to secure them to the laptop chassis.
With the small caveat that you need enough space around the port; on other computers, it might block other nearby ports, or even not fit if there's any protuberance. Passive USB-C plug to USB-C socket cables (that is, USB-C "extension cords") are AFAIK forbidden by the standard (and ignoring the standard and making one anyway would, as far as I understand, allow for dangerous combinations like putting 5A into a 3A cable), so they cannot be used to workaround this limitation.
Is there an advantage to this over an external ethernet to USB-C adapter (e.g. https://www.amazon.com/Anker-Ethernet-Portable-1-Gigabit-Chr...)? These days, I find that when using my laptop at home, a decent USB-C hub/dock is pretty much a pre-requisite given the number of things I need to connect (for me, monitor + mouse + keyboard + webcam + webcam light) and many of these have ethernet adapters built in anyway (which I don't personally use since my WiFi is fast enough).
I personally don't find the card very appealing. I thought Ethernet ports in thin laptops are a solved problem.
In that case, an expansion slot sounds nice.
I'd much rather have more battery volume inside the computer with all the USB-C ports directly exposed and then have the option to carry around a USB-C dongle with the extra ports I might need.
Hiding a USB-C hub inside a laptop and calling it "modularity" seems pretty silly.
Maybe if that hub connected to some kind of riser card PCI-E interface and was replaceable I'd take this modularity claim more seriously. At least then I'd be able to replace the whole contraption with NVMe slots, a GPU, 10GigE, etc.
Well congrats, it is connected to PCIe because it's thunderbolt, and there's no hub to need to replace because it has a direct line to the CPU (you can replace the main board if you want).
https://d3t0tbmlie281e.cloudfront.net/igi/framework/t5KLkw4x...
I'd rather either have a bank of ports that can be removed as one module and replaced with a single large component or just drop the concept and put more battery in there. These tiny individual modules waste space and don't accomplish much.
That's still missing the point. There isn't a thing to remove. You just want bigger slots.
There is no specific thing that could be removed to satisfy your desire. The difference between removing something and the type of redesign they would have to do isn't pedantry trolling.
On the one hand, they're too small for some very obvious standard ports you might want on a laptop, like for example an RJ45 port. Or multiple/vertical USB-A ports. And they're too small by a relatively small amount, so if the laptop had aimed more at Thinkpad T size thickness (still not terribly thick) rather than Macbook Air thickness, the modules would have been much more flexible (and maybe we could have had even better keyboard options).
A really significantly sized bay would have been quite interesting too - old T-series used to allow you to add a second battery, a big bay could have allowed breakouts for sensors, fpga add ons, etc. Perhaps allowing a double-wide might have been a good way of addressing both.
On the other hand, the modules are still pretty chunky. You don't want to be using one out of its slot, I don't know what sacrifices were made in terms of space in order to have what in many peoples laptops are going to be pass throughs, but the Framework has a relatively disappointing battery, so perhaps that's an effect of the reserved space.
It's more securely attached to your laptop: You don't need to deal with it when you put your laptop in a backpack, or worry about losing it when you disconnect it.
It seems crazy to me that the only way to get a Linux ARM laptop that isn't essentially a Raspberry Pi (or equivalent) is to buy a Mac.
I'd be happy to be wrong about this. If I am, please let me know where I can spend my money.
[0] https://www.notebookcheck.net/AMD-Ryzen-7-6800U-Efficiency-R...
[1] https://gadgetversus.com/processor/apple-m1-vs-qualcomm-sm84...
Part of might be that -- and this is just an observation, speculation -- nobody seems to be licensing newer core designs in high volumes outside of server-class chips or explicit mobile SKUs. That means there's no volume to drip down to consumer parts. Everything is either a mobile SoC design or a high-margin server SKU, there's no mid-range option for a modern ARMv8.5 core -- which happens to be exactly the kind of device Apple targets with something like the Macbook Air.
The rumor is that Nvidia is aiming to get the Xavier/Orin series devices to have full upstream Linux support (GPU acceleration is a different story and needs Mesa, but may come later I suspect, since both the mobile and desktop GPU drivers are now open.) If that happens I think these would probably be the best alternative options you could get, in potentially mobile form factors -- but they are still much pricier when considering performance.
You can't drive a Nvidia GPU with anywhere near the same performance/functionality with an open source driver.
Corrections welcome.
Most ARM manufacturers for consumer hardware don't seem to think that working with upstream is a good idea. Intel and AMD actively work with linux developers to provide the drivers, and some other companies provide partial (with closed firmware) or complete documentation so developers can create drivers. On ARM side, some companies are fine while others are refusing to provide documentation: Qualcomm's modems and GPUs and Broadcom's wifi hardware are some examples.
You can always force the new Intel processors in low power mode (by activating only the low power cores, at least with Linux) if you want to save power.
Of course there are the other things a laptop is compose of that consume power.
Maybe Framework is the right company to bring an ARM + Linux laptop with decent performance and battery life.
Another option is an Android tablet with a keyboard, just pick one with an unlockable bootloader so that you can run your own software.
That said, I'd love to see an ARM option from Framework.
The likely OEM's the SOC for a laptop like this (rockchip, amlogic, etc) are mostly non-helpful. Though rockchip is getting upstream reasonably well now (IE you could probably build a mainline kernel that works).
But honestly, I would expect a linux laptop built with any on-market ARM SOC to be a mess right now.
There is actually one SBC vendor who runs linux in a container under the android kernel because the android kernel has much better ARM hardware support :)
(which is probably right - most of the SOC's you'd put in a laptop are being built mainly for android based set top boxes, tablets, etc)
It would take a couple iterations and dedicated work to get somewhere good.
Edit: Since this got a lot of replies, I do get the benefits of an expansion slot. Put one on each side, that should be plenty. But fill the rest of the laptop with the usual set of ports in a motherboard that I can easily repair.
[1] https://www.lenovo.com/us/en/p/laptops/thinkpad/thinkpadt/th...
And that is what frustrates me so much. Sure, have an expansion slot on each side, they seem cool and I get the benefits. But give me more ports in a motherboard I can replace. This didn't seem like a technical decision or one grounded in values of repairability and sustainability, it seems like they made a design decision that is objectively worse.
They might simply be bandwidth-constrained. Driving 4 individual Thunderbolt 4 channels requires an insane amount of IO bandwidth, more than even the M1 or M2 can provide.
If they added another port, it probably wouldn't be any faster than USB 2.0.
Maybe they're hoping a 3rd party does it (it's effectively a regular hub in a specific form factor).
Is that really the niche that Framework is targeting? I thought it was meant to be a generally appealing laptop with expansion options like laptops used to have.
How much physical space does 1TB of NAND really take up anyway?
I know nothing about industrial design, and no doubt this must have been considered. What would be the trade offs the designers faced? Seems this modular design uses a lot of layers of chassis and space.
For devices that don't have external connectors, modular devices are already common, like M.2.
They could have maybe laid them out in a way that would have made making double-wide modules easier or something though, maybe.
Where exactly does "easily repair" mean? It all depends on who the end user is. Consider three very different users:
1. Expert: Replacing connectors on motherboards is not particularly hard for someone with experiencing soldering surface mount components.
2. Intermediate: Many laptops have modular barrel connector subcomponents that are easy to replace for anyone who is comfortable opening laptops.
3. Non-technical: Many consumers are uncomfortable opening laptops at all. Many have thrown away laptops with modular batteries.
Because there are lots of different users on different levels of technical ability, creating a "repairable" laptop can mean a lot of different things depending on where you draw the line.
That's one thing that makes me a bit uncomfortable about the current trend to use only USB-C for charging. At least on my current laptop (and I believe many other models from the same manufacturer), the "traditional" barrel connector is separate from the motherboard, connected to it by a short length of wire, while the USB-C connector is soldered directly to the motherboard and reinforced by a piece of metal. Any mechanical stress to the charging cable (for instance, from tripping on it) will go to the chassis for the barrel connector, but for the USB-C connector it will go directly to the motherboard.
Repeat this process as every external connector changes physical layout, non backwards compatible change, etc.
There is no perfectly forward and backward compatible laptop port layout. Lenovo does it one way (which some people like). Framework has gone a different direction (which a lot of other people seem to like).
Edit: Also, this ethernet expansion card this thread is about also physically hangs off the laptop.
Would be great if they could release a dual USB-A expansion slot though, the bandwidth should not be a problem. Or maybe redesign the expansion slots to be more slim so they could fit 6 in a laptop.
Being able to reuse the chassis is good idea but the implementation is lacking.
I find this statement interesting - could you expand on this? IIRC they just delivered on this promise which included (1) a new intel 12th gen mainboard as a drop-in replacement for their 11th gen mainboard and (2) an optional new chassis lid to resolve issues with the original chassis.
Disclaimer: I'm both a Thinkpad and Framework fanboy.
What they could have easily done instead is to use the same port layout for as long as possible. Let's say Ethernet, 2x USB-A, HDMI, 3.5mm jack, and as many USB-C ports as they can reasonably fit. That would be good enough for at least 3 years and probably way longer. In my opinion 3-5+ years backwards compatibility is good enough. Of course this is only me saying this from my armchair.
But it isn't "bad" in any sense of the word. More cards and they'd have to make the motherboard bigger. Maybe on a future 15"?
But on my Dell two of those ports the USB-C and the RJ45 no longer work anymore and I can't fix them ...
Something that won't happen (at least not as fast) on my frame.work.
I also have to wonder at the longevity of that design - it’s gonna hang up every time you shove it in a backpack, I’d be concerned about the port hole cracking at some stress points.
Really wanted to keep it but it arrived with a litany of quality control issues. Most of those are probably fixable with replacement parts but I don't think there's anything that can be done about the screen wobble.
A laptop I can't actually use on my lap..
Unfortunately, we didn't get an extended warranty, so its a loss. Hoping Framework can address their quality issues because the notion of extensibility and repairability is fantastic. We'll look forward to checking in on them in a few years after they work through the QA issues. Gotta having a laptop that works.
> If you have a laptop where the lid angle drops on its own while the laptop is stationary, write into support with a video of it, and we’ll send you a new Hinge Kit.
I think it should be covered by warranty :)
Yes
I never have a good experience with Realtek networking chipsets, especially on 2.5Gbps.
How long until 120hz+ displays?
2 to 5 watts for ethernet, less than 1 watt for SFP+.[0]
[0]: https://community.fs.com/blog/10gbase-t-vs-sfp-which-one-is-...
I really wish SFP+ and SFP28 would be more common in high end desktop motherboards. They’re more efficient and just better, and they can be adapted to plain old ethernet if needed (at a power penalty imposed by switching away from fiber).
Do you think that power requirements are different because SFP often uses direct-attach copper?
I'm honestly curious to reconcile my experience with other, more experience persons' reality here.
Handling the SFP module itself will more directly expose you to the heat than an integrated ethernet port, so maybe that’s the difference.
This would explain a lot, TBH. Thanks for the productive discussion -- it helped me realize a few gaps in my knowledge
And on the minor note - external SAS connectors (6/12gbps) are more then just warm, too.
My own experience, YMMW.
Also the crap with vendor locking SFP slots by Dell and Cisco especially. At least Intel cards are quite universal.
USB-C to SFP+ is also way to expensive and you can't do SFP28 on USB 3 speeds.
Or something similar if anyone else gets with the programme and stops to produce for the landfill.