Community discussions

MikroTik App
 
dgrippin
just joined
Topic Author
Posts: 11
Joined: Fri Sep 16, 2022 6:20 pm
Location: North Carolina
Contact:

Multi RIOM testing behind indiv. routers, same private IPs, data loss

Mon Sep 19, 2022 6:03 pm

Problem
We have been having issues with dropped packets, missing data points, and comms errors during testing of sensor board product (aka RIOMs) causing lost time and repeated testing errors. The test set up does work sometimes, however not reliably enough. I believe there may be issues with the router config. Being that UDP is primarily used by the RIOMs, maybe there's collision or some problem with bukets filling up?
The more tests we run per day, the more failure warnings and dropped/missing data points we get each time we run a batch.
I am taking on this project from someone who left the company a long time ago, and support is limited for this.
I have completely ruled out any physcal/cable connection issues.
Due to the nature of our work and contracts with customers, the product can not be changed in any way or the GUI used for testing. The IP addresses used are also unchangeable.

The test fixture
10 RIOMs, each one behind a single Router.
Routers are RB750UPr2 hex poe lite (mipsbe) v6.47.9
(config script below)
Diagram of test set up (below)

The process
Automated via interactive desktop GUI in Windows10.
Computer NIC is set to 192.168.129.250/24 and 10.10.10.250/24. This allows the GUI to send commands to the ACU for move-to, and also allows the GUI to send and receive sensor data from the individual RIOMs.
The GUI first sends a payload program to each RIOM for basic functionality.
After a power-cycle, the GUI puts the RIOMs into a "calibration" mode, and commands the ACU to move to various positions, sending "known" position data to each RIOM. At each position, the RIOMs set their own internal offsets/calibration.
After calibration, the GUI commands the ACU to move the fixture into various known test point orientations.
At each test position, raw data is received by the GUI from all of the sensors/RIOMs.
The GUI calculates pass/fail based on the recorded data and predetermined tolerances at the end of the procedure.

The RIOMs that are being tested all have identical fixed IP address which is set in the hardware: 192.168.0.63.
The public IP of each RIOM must be 10.10.10.x, as seen by the GUI. (10.10.10.14 is RIOM#1. 10.10.10.24 is RIOM#2. 10.10.10.34 is RIOM#3. 10.10.10.104 is RIOM#10)

I was able to obtain the following info from Engineering regarding the protocols:
"The RIOMs use a custom UDP protocol on port 21001 for normal communications. However the GUI will also use a TCP connection to start the payload on port 9760. We add our own psudo-TCP checksum and structure on the UDP packets. If you're wanting to put some rules on the routers you'll need to allow ports 21001 and 9760.
The GUI only knows about the router's reported IP address. So if a router has 10.10.10.14 on it's WAN port that will be what the GUI is looking for. From the GUI's perspective all the IPs in its table are reachable RIOMs, it's up to the router to translate. The routers really just translate the WAN (GUI network) to LAN (individual RIOM network). Each router has a unique IP and will convert requests/responses back and forth."

I have attached a simple illustration of my test set up (keep in mind I have 10 routers, but this only shows 5).
Thank you very much for any advice. I have been having such difficulty with these as networking is not my area of expertise, and Routerboards have so many options my head spins.

mybackup.cfg.rsc
test setup image.jpg
You do not have the required permissions to view the files attached to this post.
 
Sob
Forum Guru
Forum Guru
Posts: 9049
Joined: Mon Apr 20, 2009 9:11 pm

Re: Multi RIOM testing behind indiv. routers, same private IPs, data loss

Tue Sep 20, 2022 2:12 am

Seems simple, description and image are fine, I'd expect (for first router):

- 10.10.10.14/24 on ether1
- 192.168.0.1/24 on ether5
- some src/dstnats mappings between 10.10.10.14 and 192.168.0.63

And that's pretty much it. But posted config seems to be for something slightly different. What's with all those other addresses? Meaning all 10.10.10.x, 192.168.129.1/24, 192.168.0.x in NAT?
 
dgrippin
just joined
Topic Author
Posts: 11
Joined: Fri Sep 16, 2022 6:20 pm
Location: North Carolina
Contact:

Re: Multi RIOM testing behind indiv. routers, same private IPs, data loss

Tue Sep 20, 2022 3:34 pm

... posted config seems to be for something slightly different. What's with all those other addresses? Meaning all 10.10.10.x, 192.168.129.1/24, 192.168.0.x in NAT?
Thank you Sob,
The config is something that was intended to work generally for a number of set ups and installations.
-192.168.0.x is the Internal network for the ACU. These boards I am testing are normally meant to communicate directly with an ACU on a finished assy. I only need an ACU in this case for controlling the motors on the test fixture.
-192.168.129.1/24 would normally serve as the Public network for the ACU, allowing the GUI command and control of the ACU.
-All the other 10.10.10.X IPs are for various unrelated test set ups.
I left them all there because that is how the automated Router setup is configured by the GUI. Just in case any of those extra items could be contributing to my issues.
The router is configured by a separate tool in our GUI, and is meant to allow basic functionality. However, I want to upgrade to the latest RouterOS, and our GUI is not able to perform the config on any routers with the latest RouterOS (v7..) installed.

Are there items you see in the config that I can do away with?
Currently, ports 1-4 are bridged. Can I remove the bridge?
I think all ports are switched. Can I remove that?
Is the src-nat required? I read in wiki that only dst-nat is needed as the return path is automatic. Or is that not accurate?
I see HW offloading, Fast Path, and Fast Forward are enabled in various windows. Are those helpful?
Defconf Firewall filters are set up. Are they needed?

Are there any queues or buffers I can implement in the router for UDP that would help fix my losses? Is it possible to add some rules for this?
The only thing I can control in the GUI during testing is sample rate (ms). Default is 100ms. Sometimes increasing it marginally reduces the number of missing or dropped data cells, but in this scenario any dropped data that results in a blank value in the GUI is unacceptable. Keep in mind this is only useful as a troubleshooting aide, as increasing the sample rate is only something I can do as an admin. Our techs, who normally perform this test, will not be able to change the smple rate.
During test, at each test point, all the RIOMs send their readings to the GUI at once. I can not tell if it is a limitation of my NIC, or a limit of the Cisco switch, MTU limit somewhere, or what. Would Wireshark on my PC or via an external computer work for diganosing that? All I have to work with is my PC workstation and another laptop, and basically an unlimited supply of these routers and RIOMs.

The ethernet controller on the RIOM is a DigiKey Stand Alone 10/100 with SPI or parallel - ENC424J600T-I/ML. I have requested from Engineering the config specs or code, but it's highly unlikely I will get it. I've attached the DigiKey specs page in case any insights can be gleaned.

Again I appreciate your help with this. I feel like I've been banging my head against a wall trying to get this process smoothed out.
You do not have the required permissions to view the files attached to this post.
 
Sob
Forum Guru
Forum Guru
Posts: 9049
Joined: Mon Apr 20, 2009 9:11 pm

Re: Multi RIOM testing behind indiv. routers, same private IPs, data loss

Tue Sep 20, 2022 9:29 pm

For testing, I'd make it as simple as possible, the less there is, the smaller chance you'll overlook something. Minimal working config should be:
/interface ethernet
set [ find default-name=ether5 ] poe-out=forced-on
/ip address
add interface=ether1 address=10.10.10.14/24
add interface=ether5 address=192.168.0.1/24
/ip firewall nat
add chain=dstnat dst-address=10.10.10.14 action=netmap to-addresses=192.168.0.63
add chain=srcnat src-address=192.168.0.63 action=netmap to-addresses=10.10.10.14
Srcnat may not be required if RIOM only ever responds to incoming connections, but it can't hurt if you keep it anyway (it just won't do anything).

But truth is, I don't see anything in original config that should make it work differently. It can't be completely broken if it sometimes works. I'm probably missing something, but I'm not sure what it might be. Packet sniffer can help, if you know what data you're expecting, you'll see if they come or not.

I also don't understand what you want to do with v7, it doesn't seem like it could bring anything useful over v6.
 
dgrippin
just joined
Topic Author
Posts: 11
Joined: Fri Sep 16, 2022 6:20 pm
Location: North Carolina
Contact:

Re: Multi RIOM testing behind indiv. routers, same private IPs, data loss

Wed Sep 21, 2022 6:39 pm


Srcnat may not be required if RIOM only ever responds to incoming connections, but it can't hurt if you keep it anyway (it just won't do anything).

But truth is, I don't see anything in original config that should make it work differently. It can't be completely broken if it sometimes works. I'm probably missing something, but I'm not sure what it might be. Packet sniffer can help, if you know what data you're expecting, you'll see if they come or not.

I also don't understand what you want to do with v7, it doesn't seem like it could bring anything useful over v6.
v7 enables a few more options for these routers I hoped might be useful. But you are right, it is not necessary as far as baseline functionality is concerned.

Consistently 2 things have led me to believe my issues might be solved by altering router configs:

1) If the test fixture sits idle and powered on for some (undetermined) amount of time during, before, or after the procedure (like taking a break or going to lunch), all of the ROIMs will not respond to the GUI and/or command line pings, and must be power-cycled, causing us to start the procedure all over again.
Restart only GUI and/or PC does not fix it.
Restart only ACU and/or RIOMs does not fix it.
Power-cycle Routers is the only thing that fixes it.

2) I have gotten very different results depending on the quality of the NIC on the computer. "Nicer" NICs can go longer or can perform more test runs before things get screwy/GUI stops getting any readins from some of the RIOMs. The cheapo Amazon ethernet dongle 3.0 USB and similar devices usually start to lose communicaion from one or two RIOMs somewhere during the Test phase.
I'll get maybe 1-3 good test points from all 10 RIOMs, then one RIOM will drop out, or two, or more, and won't respond to pings. The ones that drop out are random. Swapping cables or RIOMs does not result in predictable changes. Nicer NICs can go longer/more batches before this happens, but it always happens given enough time.

if some tools in RouterOS can better manage packet flow to the GUI on the PC, maybe that is the solution? or maybe some time out setting we aren't thinking of? Perhaps it is not the size of the individual packets/data that's at fault, but the fact that there are 10 RIOMs all sending data through 10 routers into the PC GUI all at once is enough to do this? Could this cause an aggregated overflow of sorts that gets worse over time or affects the ability of the NIC to handle it?
 
Sob
Forum Guru
Forum Guru
Posts: 9049
Joined: Mon Apr 20, 2009 9:11 pm

Re: Multi RIOM testing behind indiv. routers, same private IPs, data loss

Thu Sep 22, 2022 12:39 am

How much data are we talking about and how exactly it behaves (bursts, ...)? I can imagine that if ten devices at once tried to send lots of UDP traffic at maximum speed, something would get dropped. But if they have only 10/100 NICs anyway, connected to gigabit switch and to (most likely) gigabit NIC in computer, there shouldn't be any bottleneck.

What could be interesting is when it gets stuck and restarting routers helps. You can debug that. When it happens, don't restart router, instead connect to it (laptop to free port, WinBox to MAC address, or add another subnet for it on that port) and watch what happens using Tools->Torch on interfaces (for a quick look) or Tools->Packet Sniffer (for details; it can also stream packets to Wireshark on laptop). Keep e.g. ping from PC to RIOM running and see where exactly it gets lost (if you can see requests on one interface, then on the other, if there's any response, ...).

As for managing packet flow, I'm pretty happy with "router receives packet, router sends packet further", I have great success with that. :) You can influence it with queues (https://wiki.mikrotik.com/wiki/Manual:Queue), I'm no expert on that, but it's mainly when you have more flows, but here on single router is should be just one. You can limit maximum speed, but it will just drop excessive packets that won't fit in buffer, so it's probably not what you want.
 
dgrippin
just joined
Topic Author
Posts: 11
Joined: Fri Sep 16, 2022 6:20 pm
Location: North Carolina
Contact:

Re: Multi RIOM testing behind indiv. routers, same private IPs, data loss

Thu Sep 22, 2022 4:13 am

Excellent advice. I will do that and report back with debug logs. Some clarification I need first:

Can I "ping 10.10.10.X -t" from the main pc (to perform debug as described), even if ping responses are showing no reply or unreachable? Should Torch still report those?

I wanted to know why you used "action=netmap" in NAT? Rather than "action=dstnat/action=srcnat" which I had previously.

Yes, all RIOMs are 10/100, and the routers are seeing that. Im very happy with the settings you gave me. Initial testing this evening showed good comms. I still need to run the actual procedure though.

As far as "how much data", I'm not sure how to quantify that here. Bytes per second leaving each device?
 
Sob
Forum Guru
Forum Guru
Posts: 9049
Joined: Mon Apr 20, 2009 9:11 pm

Re: Multi RIOM testing behind indiv. routers, same private IPs, data loss

Thu Sep 22, 2022 5:50 am

- Ping - Yes, you can keep it running with -t and PC will be sending ICMP packets to router. If switch doesn't eat them, you must see them on router, first as inbound on ether1 and then outbound on ether5. If RIOM responds, you'll see it as inbound packets on ether5 and outbound on ether1 (and finally they should reach PC too). It seems most likely that requests will be ok and there won't be responses from RIOM for some reason. You can sniff what other traffic is on ether5 and maybe it will show something useful. You can also try to ping RIOM's 192.168.0.63 from router to see if it responds to that. Also check if there's MAC address for RIOM in IP->ARP.

- Netmap - It's a habit, I use it when I want all ports. It's different from src/dstnat when used with subnets (i.e. multiple addresses), but I'm not sure if there's any real difference when it's just one (maybe there is, but so far I didn't find it).

- How much data - It's just that I don't have even rough idea, this data point, is it few bytes, few kilobytes, megabyte, ... one packet, ten packets, thousand packets...
 
dgrippin
just joined
Topic Author
Posts: 11
Joined: Fri Sep 16, 2022 6:20 pm
Location: North Carolina
Contact:

Re: Multi RIOM testing behind indiv. routers, same private IPs, data loss

Thu Oct 20, 2022 9:50 pm

- Ping - Yes, you can keep it running with -t and PC will be sending ICMP packets to router. If switch doesn't eat them, you must see them on router, first as inbound on ether1 and then outbound on ether5. If RIOM responds, you'll see it as inbound packets on ether5 and outbound on ether1 (and finally they should reach PC too). It seems most likely that requests will be ok and there won't be responses from RIOM for some reason. You can sniff what other traffic is on ether5 and maybe it will show something useful. You can also try to ping RIOM's 192.168.0.63 from router to see if it responds to that. Also check if there's MAC address for RIOM in IP->ARP.

- Netmap - It's a habit, I use it when I want all ports. It's different from src/dstnat when used with subnets (i.e. multiple addresses), but I'm not sure if there's any real difference when it's just one (maybe there is, but so far I didn't find it).

- How much data - It's just that I don't have even rough idea, this data point, is it few bytes, few kilobytes, megabyte, ... one packet, ten packets, thousand packets...

Here is the config that finally worked from one of the routers.
/interface ethernet
set [ find default-name=ether5 ] poe-out=forced-on
/interface bridge
add name=bridge1
/interface list
add name="riom net"
/interface bridge port
add bridge=bridge1 interface=ether5 trusted=yes
add bridge=bridge1 interface=ether4 trusted=yes
/ip settings
set icmp-rate-limit=5 send-redirects=no
/interface detect-internet
set detect-interface-list="riom net" lan-interface-list="riom net"
/interface list member
add comment="for detect interface" interface=ether5 list="riom net"
add comment="for detect interface" interface=ether4 list="riom net"
add comment="for detect interface" interface=ether3 list="riom net"
add comment="for detect interface" interface=ether2 list="riom net"
/ip address
add address=10.10.10.64/24 interface=ether1 network=10.10.10.0
add address=192.168.0.1/24 interface=ether5 network=192.168.0.0
/ip firewall mangle
add action=mark-routing chain=prerouting dst-address=10.10.10.64 \
    new-routing-mark=main passthrough=no
/ip firewall nat
add action=dst-nat chain=dstnat dst-address=10.10.10.64 in-interface=ether1 \
    to-addresses=192.168.0.63
/ip route
add distance=1 gateway=ether1
add check-gateway=ping distance=1 dst-address=192.168.0.63/32 gateway=bridge1
/system watchdog
set watchdog-timer=no
/tool bandwidth-server
set authenticate=no enabled=no
/user aaa
set default-group=full
 
Sob
Forum Guru
Forum Guru
Posts: 9049
Joined: Mon Apr 20, 2009 9:11 pm

Re: Multi RIOM testing behind indiv. routers, same private IPs, data loss

Fri Oct 21, 2022 3:37 am

If it works, great. But:

- "riom net" list and everything related to it does nothing, you don't actually use it for anything (so it doesn't even matter whether internet detect does anything useful, which is generally questionable)
- same goes for your mangle rule, you have only one routing table, default one called "main", so assigning "main" routing mark has the same effect as not doing it
- it's not breaking anything, but IP address on ether5 is wrong when ether5 is bridge port, it should be on bridge1
- it doesn't look that the first route is doing anything useful either
 
dgrippin
just joined
Topic Author
Posts: 11
Joined: Fri Sep 16, 2022 6:20 pm
Location: North Carolina
Contact:

Re: Multi RIOM testing behind indiv. routers, same private IPs, data loss

Fri Oct 21, 2022 4:10 pm

I will try removing those 1st two items and see what happens.
From my experiments, assigning that IP to the bridge rather than ether5, the RIOM would not reply to icmp. I tested by watching Winbox @ether1. I sent "ping 10.10.10.14 -t" via CLI on my PC. I would see timeouts or "host not reachabl;e" when IP address was changed to the bridge. It would come back as soon as I changed it back to ether5.
The RIOMs are powered with PoE, so I could also test that the RIOM was actually responding, and not the bridge or interface itself. Inducing a power cycle of ether5 PoE would show timeouts when IP address is assigned to ether5, then replies would continue when power came back.
With IP address assigned to the bridge instead, power cycling the PoE, I continued to receive icmp replies. Indicating to me that it was actually the bridge interface replying and not the RIOM itself.
The interface lists I made intially when I was testing various NAT rules and firewall rules. So they are just a remnant of that and continued testing.

I do know the "detect interface" menu is questionable among the forum gurus, but I found if I allow detection for ether5 or the bridge, that the RIOM shows up in ARP, switch, & bridge "Hosts" tables more readily. If I don't use it, I fail to get the RIOM's MAC address to show in any tables.
I also tried various proxy arp settings on the bridge or ether5 but can't ever figure out if that's working or not. Seems like everything I have tried, I either see no RIOM MAC address at all, or I only see the RIOM MAC address and the proxy isn't working like I expect. But maybe I'm just clueless as to how that works entirely.

The 1st route is manual "default route" per the MikrTik documentation. is that what you mean? The 0.0.0.0/0 gateway ether1?
If instead you refer to the 10.10.10.X gateway ether1, that route populates automatically, maybe because of my mangle rule? I also found that without certain routes in place, the RIOM will not respond to ICMP.

Maybe this will help. I will attach Google Drive shared folder with 3 wireshark captures of successful runs.

https://drive.google.com/drive/folders/ ... sp=sharing

1st; a successful send-payload programming of just one of the RIOMs.
2nd; a successful Calibration run of all 10 RIOMs.
3rd; a successful Test run after calibration. (RIOMs all send sensor readings at each position. This file contains only 2 out of 4 total tests we run).
These captures were taken using my new router config posted above.
note: anything using address 192.168.129.200 & 192.168.129.51 is for move-to commands to the test fixure ACU only.
 
Sob
Forum Guru
Forum Guru
Posts: 9049
Joined: Mon Apr 20, 2009 9:11 pm

Re: Multi RIOM testing behind indiv. routers, same private IPs, data loss

Mon Oct 24, 2022 12:19 am

Address on bridge or interface, obviously I can't argue with you seeing a difference, but there really shouldn't be any.

Similar with detect internet. But I guess that probably could do something, even if unintended, because it's sending some packets to interface, so even if they are not for the device itself, they could possibly have some effect.

As for routes, you shouldn't actually need any static ones, if the router is routing only between two subnets. Dynamic routes created based on IP addresses and their masks are enough. But again, the one with check-gateway=ping, even though it's doing nothing as route, is pinging 192.168.0.63 from time to time, which can have some effect. But I'd rather replace that with Netwatch pinging same address, there would be configurable interval as bonus.
 
dgrippin
just joined
Topic Author
Posts: 11
Joined: Fri Sep 16, 2022 6:20 pm
Location: North Carolina
Contact:

Re: Multi RIOM testing behind indiv. routers, same private IPs, data loss

Mon Oct 24, 2022 5:18 pm

Similar with detect internet...

...the one with check-gateway=ping, even though it's doing nothing as route, is pinging 192.168.0.63 from time to time, which can have some effect. But I'd rather replace that with Netwatch pinging same address, there would be configurable interval as bonus.

I was using Netwatch as a tool during config and found it to be very useful. Are you saying I can leave it running without detriment? That is something I wanted to try.
Suggestions for interval to set?
How frequently does "check-gateway" ping interface?

I had problems with connections dropping or being blocked under various use cases within our test GUI, typically during transitions from calibration mode to testing mode, or immediately before beginning a test, where the GUI sends icmp to RIOMS 1-10 before beginnig it's testing. By shortening some of the connection tracking imeout values, the frequency of problems seemed to be less. Any issues you can see from this?
Is there a preferred way to configure RouterOS connection tracking to be fully open and allow all connections/ports without any restrictions? (as long as they follow my NAT rules)
I know that when it keeps connections in the tracking list, it will prevent "new" ones that match. Can I minimize tracking without totally disabling tracking? I guess, what is the preferred way to still allow tracking without having it block connections?

Image of Current timeouts:
conntracktime.jpg
Additional issue, some of my Wireshark logs show "previously used port" errors with my PC as the source address. I am not sure if this means it is being blocked or not. I saw less of these when decreasing timeout values above. I have no firewall configured on my PC and full administrator privileges. Are you aware of any Win10 settings that might cause a message like this?

EDIT:
I experimented with assigning address to bridge only, vs assigning to an interface on a bridge, vs no bridge, and did get different results.
Without changing the rest of my existing configuration:
I created a bridge with ether1 assigned as the only port. I assigned address 10.10.10.14/24 to ether1 only (not bridge) -> could not log in to router using Winbox via ether1 IP address. Can only connect using MAC.
I then reassigned the IP address 10.10.10.14/24 to the bridge containing ether1 -> I could now log in using Winbox IP address @ ether1.
I then removed ether1 from the bridge, deactivated that bridge interface, and assigned 10.10.10.14/24 to ether1 -> behaves the same as assigning Ip to the port, can not log in with IP address.
You do not have the required permissions to view the files attached to this post.
Last edited by dgrippin on Mon Oct 24, 2022 7:27 pm, edited 1 time in total.
 
Sob
Forum Guru
Forum Guru
Posts: 9049
Joined: Mon Apr 20, 2009 9:11 pm

Re: Multi RIOM testing behind indiv. routers, same private IPs, data loss

Mon Oct 24, 2022 7:20 pm

Quick test says that check-gateway=ping uses 10 seconds interval. Using Netwatch should be ok, assuming that RIOM doesn't have some problem with pings.

Connection tracking has some limits, but if you use any kind of NAT, you can't get rid of it.

Duplicate ports could be a problem for TCP, where you can see when connection starts. So if old one would be still active, new one using same source port (same everything, source/destination addresses/ports) would fail. But normally it shouldn't be a problem, because Windows changes source port for each new connection.

UDP wouldn't mind at all, as it doesn't have connections, so "duplicate" would be seen as part of what conntrack already knows.

But there could be problem with too low timeouts. If there was a pause without packets, connection would time out and following response from RIOM would be seen as new, and router wouldn't know that it should change its source to 10.10.10.x. UDP could be fixed with srcnat rule (see my simple config at the beginning). TCP probably not.
 
dgrippin
just joined
Topic Author
Posts: 11
Joined: Fri Sep 16, 2022 6:20 pm
Location: North Carolina
Contact:

Re: Multi RIOM testing behind indiv. routers, same private IPs, data loss

Mon Oct 24, 2022 7:49 pm

Connection tracking has some limits, but if you use any kind of NAT, you can't get rid of it.

Which I do (use NAT), because PC/GUI expects all replies to come from a 10.10.10.X address. And RIOMs all expect everything to come from 192.168.0.X address. So I am stuck having to disguise both internal and external addresses. I did try "netmap" per your original. No dice.
I've only been able to get dst-nat to work, and it seems to take care of disguising the reverse path simultaneously.
I did have a src-nat rule under the dst-nat rule, but looking at packet counters, it did not appear to ever get used.

Duplicate ports could be a problem for TCP, where you can see when connection starts. So if old one would be still active, new one using same source port (same everything, source/destination addresses/ports) would fail. But normally it shouldn't be a problem, because Windows changes source port for each new connection.
UDP wouldn't mind at all, as it doesn't have connections, so "duplicate" would be seen as part of what conntrack already knows.


I did see that behavior, but perhaps 10 devices plus many connection "transactions" could be causing ports to get used up? It could be a benign warning, as I was still able to complete my test runs with those errors being intermittent.
What do you mean UDP does not have connections? Would that not mean conntrack does not track them, therefore could not be part of what it "already knows"? Forgive my nit picking. I assumed everything was tracked in some way.

But there could be problem with too low timeouts. If there was a pause without packets, connection would time out and following response from RIOM would be seen as new, and router wouldn't know that it should change its source to 10.10.10.x. UDP could be fixed with srcnat rule (see my simple config at the beginning). TCP probably not.


I predicted this, and did have a firewall filter rule to accept "new" dst-nat connections from certain input interfaces, but I found that deleting all firewall filters instead did the job just as well, and seemed simpler overall due to less CPU load. Just a guess. Regarding the src-nat rule, see above. It seemed to not do anything (zero bytes/packets).


EDIT: overwhelmed by the amount of options in Wireshark, I am seeing a lot of variations in how it interprets packets and bytes. Choosing different default configs will result in more or less errors or flags being captured. are there Wireshark filters or profiles/dissectors/protocol options you suggest?
 
Sob
Forum Guru
Forum Guru
Posts: 9049
Joined: Mon Apr 20, 2009 9:11 pm

Re: Multi RIOM testing behind indiv. routers, same private IPs, data loss

Mon Oct 24, 2022 8:15 pm

You can check (in Windows) how many source ports you have and judge whether it's possible that you're exhausting them too quickly:
netsh int ipv4 show dynamicport tcp
netsh int ipv4 show dynamicport udp
TCP has real connections, each begins with clearly recognizable handshake. UDP doesn't have anything like that, it's just packets from x.x.x.x:x to y.y.y.y:x, there's no clear start or end. When conntrack sees UDP packets with exactly same source and destination, it assumes that they belong together and treats them as "virtual connection". But if packet with exactly same source and destination appeared from somewhere else, conntrack couldn't tell that it's something else.
 
dgrippin
just joined
Topic Author
Posts: 11
Joined: Fri Sep 16, 2022 6:20 pm
Location: North Carolina
Contact:

Re: Multi RIOM testing behind indiv. routers, same private IPs, data loss

Mon Oct 24, 2022 9:41 pm

So in my case, best to set UDP timeouts higher, and TCP timeouts lower?

port range shows 49152 start port/16384 #of ports - for both udp/tcp. I expected more detail. should I run that cmd while running test fixture?
 
Sob
Forum Guru
Forum Guru
Posts: 9049
Joined: Mon Apr 20, 2009 9:11 pm

Re: Multi RIOM testing behind indiv. routers, same private IPs, data loss

Mon Oct 24, 2022 10:20 pm

Yes and no. If you'd be expecting such problems, then yes. But if those captures you shared are representative example, there's about ten TCP connections in them, total. And most of them to 192.168.129.x. So 16k ports should last for a while, until they start to get reused.
 
mcasanas
just joined
Posts: 5
Joined: Tue Nov 29, 2022 12:29 am

Re: Multi RIOM testing behind indiv. routers, same private IPs, data loss

Tue Nov 29, 2022 5:18 pm

Having a similar scenario here, I´d appreciate some help!
1. The posted configuration is applied to routers running their default configuration? Or a total wipe is needed before?
2. Can be used less routers to achieve the same results (no PoE needed)? (ideally, just one switch running RouterOS)
3. Did the RIOM project finally reach production stage?

I must control several devices from one PC. These devices have all of them the same 192.168.x.x private IP, that cannot be changed. Don´t know from where to start, any help is welcome!
 
dgrippin
just joined
Topic Author
Posts: 11
Joined: Fri Sep 16, 2022 6:20 pm
Location: North Carolina
Contact:

Re: Multi RIOM testing behind indiv. routers, same private IPs, data loss

Wed Nov 30, 2022 3:42 pm

1. The posted configuration is applied to routers running their default configuration? Or a total wipe is needed before?

The config is applied after a complete system reset with no default configuration. You may need to connect via MAC address in Winbox until the custom config is applied.

2. Can be used less routers to achieve the same results (no PoE needed)? (ideally, just one switch running RouterOS)

I'm not sure if I understand this question, so I'll answer as best I can.
You must use one router per device you are operating. If you have a number of devices to test at the same time, you CAN NOT use one router for all of them.
Each router's ether1 10.10.10.X address must be unique. You use the ether1 address to communicate to your devices. This is also how you can tell them apart. If they were all behind the same router, you'd have no way to know which device is sending or receiving specific data, or in the event of errors, which device was the culprit.

PoE is optional. I needed it here because our RIOMs only recieve power that way.
If your devices have external power supply you can turn off the PoE option on Ether5.

I should note that PoE output is equal to the voltage of each router's external power supply. Make sure your devices are compatible with that. The great thing is that Routerboards can operate with a wide range of voltage, about 9-30 VDC if I recall.

3. Did the RIOM project finally reach production stage?
I must control several devices from one PC. These devices have all of them the same 192.168.x.x private IP, that cannot be changed. Don´t know from where to start, any help is welcome!

Yes, our test fixture is operating using this configuration. No issues.
Our devices also have a 192.168.x.x fixed private IP that's unchangeable, so I think my config is a perfect place to start.
Use the latest long-term release RouterOS if you can.
Once your routers are updated, reset by selecting "no default config" and "do not back up".
If you are using a copy of my config, make sure you edit it for the appropriate ether1 IP address for each router you're using. Do a "Ctrl+F" in the text file to find all instances of 10.10.10.X and change appropriately.
Be careful with editing the firewall IPs so you don't block yourself.
You will need to edit the "RouterID" and the MAC addresses for each interface/bridge as well, to match your router.
After router reset, connect via MAC address and send your edited config via the terminal interface window, or however you feel comfortable.
 
mcasanas
just joined
Posts: 5
Joined: Tue Nov 29, 2022 12:29 am

Re: Multi RIOM testing behind indiv. routers, same private IPs, data loss

Thu Dec 01, 2022 3:04 pm

Thanks a lot for your answer!
Trying to get things working, but no luck. Can you post your most up-to-date-and-working config?
In the second question, I tried to find is there´s a chance to route more than one ethernet port (maybe adding virtual interfaces), since PoE is not needed here.
Thanks again, best regards!
 
dgrippin
just joined
Topic Author
Posts: 11
Joined: Fri Sep 16, 2022 6:20 pm
Location: North Carolina
Contact:

Re: Multi RIOM testing behind indiv. routers, same private IPs, data loss

Thu Dec 01, 2022 7:54 pm

Here it is.
change the IP addresses, PoE settings, etc for your set up.
/interface ethernet
set [ find default-name=ether1 ] advertise=\
    10M-half,10M-full,100M-half,100M-full,1000M-half,1000M-full
set [ find default-name=ether5 ] advertise=\
    10M-half,10M-full,100M-half,100M-full,1000M-half,1000M-full poe-out=\
    forced-on
/ip neighbor discovery-settings
set discover-interface-list=all
/ip settings
set send-redirects=no
/ip address
add address=10.10.10.104/24 interface=ether1 network=10.10.10.0
add address=192.168.88.1/24 interface=ether2 network=192.168.88.0
add address=192.168.0.200/24 interface=ether5 network=192.168.0.0
/ip firewall nat
add action=dst-nat chain=dstnat in-interface=ether1 to-addresses=192.168.0.63
add action=src-nat chain=srcnat out-interface=ether1 to-addresses=\
    10.10.10.104
/ip ipsec settings
set accounting=no
/snmp
set enabled=yes
/system identity
set name=router10
/system watchdog
set ping-start-after-boot=5h watchdog-timer=no
/tool bandwidth-server
set authenticate=no enabled=no
/tool netwatch
add host=192.168.0.63 interval=30s
/user aaa
set default-group=full use-radius=yes
 
Sob
Forum Guru
Forum Guru
Posts: 9049
Joined: Mon Apr 20, 2009 9:11 pm

Re: Multi RIOM testing behind indiv. routers, same private IPs, data loss

Fri Dec 02, 2022 2:33 pm

@mcasanas: About multiple ports on same router:

viewtopic.php?t=107142
viewtopic.php?t=130127
 
mcasanas
just joined
Posts: 5
Joined: Tue Nov 29, 2022 12:29 am

Re: Multi RIOM testing behind indiv. routers, same private IPs, data loss

Fri Dec 02, 2022 7:21 pm

Still trying, thanks for your answers! Testing with 2 PC´s with same IP, rdesktop sometimes goes to one, some other times goes to the other...

Who is online

Users browsing this forum: No registered users and 5 guests