Several people including me have had throughput issues with the RB5009. The download speed via the SFP+ to any of the Ethernet ports is terrible. Usually around 200-400mbps.

Well, I have finally figured out an interesting way to reproduce the issue... So the RB5009 has no config at all.. I literally cleared it, removed the router config, created a bridge and added the SFP+ and first couple Ethernet ports. It's just a switch basically.

I have this setup: SERVER - SWITCH - (sfp+)RB5009(eth2) - client

The server runs iperf3 server and the client is where I am testing from… The client is connected at 1Gbps. The server is connected at 10Gbps

I can send and receive data at around 900Mbps+ which is full speed. Seems fine.
The issue appears to be something to do with delay. I will then simulate delay on the linux client using linux tc with the command `sudo tc qdisc add dev eno1 root netem delay 5ms`
This adds 5ms of latency.. With 5ms of latency I can still send data at 900Mbps+ but I can only receive data at around 200-300Mbps through the RB5009.

To verify this, I remove the RB5009 and I plug directly into the switch. I still have the latency. I can send and receive at a normal 900Mbps+ rate.

So it seems like it's something with latency + TCP + RB5009 is what is causing the issue. I was having a horrible time with my AP connected to the 2.5G port. I'm guessing since the AP has a lot more latency 2-5ms it's why clients seem to be having issues. It's just a guess but an interesting observation. I tried with vlan filtering on and off just to see. Made no difference. Ethernet flow control is off everywhere and I tried with CPU Flow control on and off under the switch settings.

So I've found that if I disable hardware offload of the 1G Ethernet ports in the bridge config, it fixes the issue.

Could it be bufferbloat in the switch chip? Some sort of buffer overrun? Where the ACKs aren't getting back through? Total speculation, no idea.. But at least I can reproduce it now reliably. I can probably live with HW offloading disabled for now. When running iperf the CPU is around 8%. As long as the SFP+ can route at full speed I'm pretty happy because it's a router on a stick for me.