Community discussions

MikroTik App
 
server8
Long time Member
Long time Member
Topic Author
Posts: 591
Joined: Fri Apr 22, 2011 1:27 pm

CCR2004 packet loss

Mon Oct 12, 2020 11:00 pm

I have a seriuos problem with all my CCR2004 when traffic pass from giga interface to 10 giga interface and viceversa I have a packet loss aorund 1% or little less.

According with the block diagram https://i.mt.lv/cdn/product_files/CCR20 ... 200459.png I am thinking the problem is generated in the passive intelligent port extender

CCR2004 is not usable to mix different interface speeds :-(

how is it possible?
 
mkx
Forum Guru
Forum Guru
Posts: 8955
Joined: Thu Mar 03, 2016 10:23 pm

Re: CCR2004 packet loss

Mon Oct 12, 2020 11:24 pm

Do you experience packet loss in both directions equllky or is one direction distinctively worse?

I'd expect problems in packet direction 10G->1G where quite some buffering occurs and enabled flow control should help a lot (another matter is whether flow control actually works).
 
server8
Long time Member
Long time Member
Topic Author
Posts: 591
Joined: Fri Apr 22, 2011 1:27 pm

Re: CCR2004 packet loss

Mon Oct 12, 2020 11:37 pm

We have mainly packet loss pinging from the remote end to ccr2004 gigabit interface, the problem probably occurs when the upstream traffic from the 10G->1G fullfill the ethernet..

Enabling flow control not solve the problem.
 
paulct
Member
Member
Posts: 329
Joined: Fri Jul 12, 2013 5:38 pm

Re: CCR2004 packet loss

Thu Oct 15, 2020 10:28 am

buffer memory issue? I.e not enough buffer space when going from 10Gbps to 1Gbps and vice versa. Not sure how to get the stats from a Mikrotik on this.
 
jebz
Member
Member
Posts: 362
Joined: Sun May 01, 2011 12:03 pm
Location: Australia

Re: CCR2004 packet loss

Thu Oct 15, 2020 11:38 am

There was a similar issue on the CCR1072 where the ethernet port was claimed only to be suitable for management traffic.
viewtopic.php?t=125361
 
User avatar
IPANetEngineer
Trainer
Trainer
Posts: 1676
Joined: Fri Aug 10, 2012 6:46 am
Location: iparchitechs.com
Contact:

Re: CCR2004 packet loss

Thu Oct 15, 2020 4:30 pm

We have mainly packet loss pinging from the remote end to ccr2004 gigabit interface, the problem probably occurs when the upstream traffic from the 10G->1G fullfill the ethernet..

So does the packet loss only occur when the 1G interface is full?
 
server8
Long time Member
Long time Member
Topic Author
Posts: 591
Joined: Fri Apr 22, 2011 1:27 pm

Re: CCR2004 packet loss

Thu Oct 15, 2020 11:49 pm

We have mainly packet loss pinging from the remote end to ccr2004 gigabit interface, the problem probably occurs when the upstream traffic from the 10G->1G fullfill the ethernet..

So does the packet loss only occur when the 1G interface is full?
No when the giga interface receive data from 10 giga interface: the slower port sometimes has to keep in pause the faster one before to receive new packets because the buffer is full or not have the buffer.
 
User avatar
IPANetEngineer
Trainer
Trainer
Posts: 1676
Joined: Fri Aug 10, 2012 6:46 am
Location: iparchitechs.com
Contact:

Re: CCR2004 packet loss

Thu Oct 15, 2020 11:52 pm

What's the lowest throughput (on the 1G) you've seen the packet loss?
 
server8
Long time Member
Long time Member
Topic Author
Posts: 591
Joined: Fri Apr 22, 2011 1:27 pm

Re: CCR2004 packet loss

Fri Oct 16, 2020 8:46 am

The problem is how fast the data arrive on the giga port from the 10 giga port, you can loose packet also whit 100kb or less. If the number of packets arriving exeeds tha capibility of gigaport to send it you will lose packet.
Generally is around 1% or littble bit less. This is a big hardware poblem for ccr2004 and it is not usable to mix different speed ports.
 
User avatar
normis
MikroTik Support
MikroTik Support
Posts: 25389
Joined: Fri May 28, 2004 11:04 am
Location: Riga, Latvia

Re: CCR2004 packet loss

Fri Oct 16, 2020 10:32 am

The 1G port is for management (winbox, ssh to the router), as it's also labeled on the front panel. This port is somewhat limited, and is meant for what it says. The diagram also gives a clue to this.
 
server8
Long time Member
Long time Member
Topic Author
Posts: 591
Joined: Fri Apr 22, 2011 1:27 pm

Re: CCR2004 packet loss

Fri Oct 16, 2020 11:58 pm

Normis we are speaking about SFP ports
 
User avatar
Maggiore81
Trainer
Trainer
Posts: 477
Joined: Sun Apr 15, 2012 12:10 pm
Location: Italy
Contact:

Re: CCR2004 packet loss

Sat Oct 17, 2020 8:09 am

Sincerely I am very disappointed for CCR2004's performance.
We replaced a CCR1036 + 10G switch (with trunks) to a CCR2004 with directly put in DACs cables 10G.

We do about 4.5gb traffic through it.
20 firewall rules, most in RAW for "simple" packet filter at the border.
fasttrack enabled.


4x99% cpu, one 100%.
I tried disabling everything, just fastpath, 80% cpu.
even with 2g traffic the voip was chopping.
Replaced back the 1036, and we are with it now. I could consider the 1072 but I fear the watchdog issues that are on the forum. Till now the 1036 is 2+ years working solid.
we use the long-term ROS.
 
server8
Long time Member
Long time Member
Topic Author
Posts: 591
Joined: Fri Apr 22, 2011 1:27 pm

Re: CCR2004 packet loss

Sat Oct 17, 2020 11:32 am

A router where you can't mix different sfp ports speeds it's not usable in production, if I deploy fiber I need to have 0% packets loss and no 1% like now.

I opened a ticket [SUP-30387] but for now no solution to the issue :-(
 
User avatar
IPANetEngineer
Trainer
Trainer
Posts: 1676
Joined: Fri Aug 10, 2012 6:46 am
Location: iparchitechs.com
Contact:

Re: CCR2004 packet loss

Sat Oct 17, 2020 4:55 pm

Have you tried changing the interface queues for both ports from hardware-only to ethernet-default ?

You need buffering when mixing interface speeds on any router and i'm not sure what the default buffer capabilities are for the CCR2004.

I would at least try a few different queue types for the interfaces to see if performance improves.
 
User avatar
Maggiore81
Trainer
Trainer
Posts: 477
Joined: Sun Apr 15, 2012 12:10 pm
Location: Italy
Contact:

Re: CCR2004 packet loss

Sat Oct 17, 2020 6:05 pm

Where can I find precise documentation about queue types?
Well I read the wiki... but I have not found any REAL example of WHERE to use wich queue type or interface type
Last edited by Maggiore81 on Sat Oct 17, 2020 9:00 pm, edited 1 time in total.
 
server8
Long time Member
Long time Member
Topic Author
Posts: 591
Joined: Fri Apr 22, 2011 1:27 pm

Re: CCR2004 packet loss

Sat Oct 17, 2020 9:00 pm

Have you tried changing the interface queues for both ports from hardware-only to ethernet-default ?

You need buffering when mixing interface speeds on any router and i'm not sure what the default buffer capabilities are for the CCR2004.

I would at least try a few different queue types for the interfaces to see if performance improves.
The first think that I have tried before to write here and to open a support ticket.
 
User avatar
Paternot
Forum Veteran
Forum Veteran
Posts: 931
Joined: Thu Jun 02, 2016 4:01 am
Location: Niterói / Brazil

Re: CCR2004 packet loss

Sun Oct 18, 2020 12:59 am

Sincerely I am very disappointed for CCR2004's performance.
We replaced a CCR1036 + 10G switch (with trunks) to a CCR2004 with directly put in DACs cables 10G.
But You did a downgrade! Take a look at the performance figures:

https://mikrotik.com/product/CCR1036-12 ... estresults
https://mikrotik.com/product/ccr2004_1g ... estresults

Yes, the 2004 is faster with pure fastpath and big packets, but only in this situation. Everything else, and the 1036 wipes the floor with it. It isn't meant as a direct substitute to the 1036 - it is more something above 1009 and just below 1016.

Yes, each one of its 4 cores is much faster than the 10XX ones - but not enough to offset 4 cores versus 36 cores. It will outperform the CCR10XX series in single threaded activities, but not in firewalling.
 
User avatar
Maggiore81
Trainer
Trainer
Posts: 477
Joined: Sun Apr 15, 2012 12:10 pm
Location: Italy
Contact:

Re: CCR2004 packet loss

Sun Oct 18, 2020 1:05 am

Hello Paternot, I have read carefully the performance figures, but sincerely I expected a lot more.
I did not expect a 100% cpu load !!!
However I am planning to replace the 1036 witha 1072 because I need the pure 10G ports for each link.
I have some fear to get a 1072 that has issues as in the threads about "watchdog and 1072".

I use conntrack, I use Fasttrack, it is one of my border router.
 
User avatar
Paternot
Forum Veteran
Forum Veteran
Posts: 931
Joined: Thu Jun 02, 2016 4:01 am
Location: Niterói / Brazil

Re: CCR2004 packet loss

Sun Oct 18, 2020 6:05 am

Hello Paternot, I have read carefully the performance figures, but sincerely I expected a lot more.
I did not expect a 100% cpu load !!!
However I am planning to replace the 1036 witha 1072 because I need the pure 10G ports for each link.
I have some fear to get a 1072 that has issues as in the threads about "watchdog and 1072".

I use conntrack, I use Fasttrack, it is one of my border router.
Well, I can't speak for the usage model on You network - but if it was taxing one CCR1036, I find remarkable that one CCR2004 can barely cope with the load. I mean, take a look at the results for 512 bytes and 25 ip rules: it's about 1/3 of the 1036 speed!
 
User avatar
Maggiore81
Trainer
Trainer
Posts: 477
Joined: Sun Apr 15, 2012 12:10 pm
Location: Italy
Contact:

Re: CCR2004 packet loss

Sun Oct 18, 2020 11:39 am

You may be right, I may have chosen the wrong product.
The 1036 was at about 15% average cpu...
 
server8
Long time Member
Long time Member
Topic Author
Posts: 591
Joined: Fri Apr 22, 2011 1:27 pm

Re: CCR2004 packet loss

Sun Oct 25, 2020 9:38 pm

Any update on this issue?
 
server8
Long time Member
Long time Member
Topic Author
Posts: 591
Joined: Fri Apr 22, 2011 1:27 pm

Re: CCR2004 packet loss

Tue Oct 27, 2020 11:28 am

Any update on this issue?
 
server8
Long time Member
Long time Member
Topic Author
Posts: 591
Joined: Fri Apr 22, 2011 1:27 pm

Re: CCR2004 packet loss

Thu Oct 29, 2020 10:34 pm

Any update on this issue?
 
ozar
just joined
Posts: 12
Joined: Fri Jun 15, 2018 4:28 am

Re: CCR2004 packet loss

Fri Oct 30, 2020 11:45 am

I would just like to say that I experienced the same issues.

I have x4 CCR2004's and when they were in my lab before being shipped to production I had them connected to each other using Mikrotik 1G S-RJ01 Ethernet modules. NSW-RTR-01 (172.16.4.1 - CCR2004) was connected to NSW-RTR-02 (172.16.4.2 - CCR2004) via a Mikrotik 1G S-RJ01 Ethernet module. I ran "ping 172.16.4.2 interface=sfp-sfpplus2 do-not-fragment count=100 size=1500" and received high packet loss. Since I did this a while ago now I would have to dig up this packet loss data if you want proof.

Alternatively Mikrotik you could grab x2 CCR2004's and connect them to each other via Mikrotik 1G S-RJ01 Ethernet modules and run the same ping test and see the packet loss yourself. I don't see any packet loss if the Router to Router connections use 10G interfaces instead. I haven't tested this but I assume 1G Fibre SFPs will be affected too. Hopefully shaping or setting the Tx/Rx Flow Control to on will resolve this issue.

Worth investigating.
 
ozar
just joined
Posts: 12
Joined: Fri Jun 15, 2018 4:28 am

Re: CCR2004 packet loss

Fri Oct 30, 2020 2:27 pm

I still have x2 CCR2004's that aren't in production yet so I did some testing on them. The CCR2004's are directly connected to each other via sfp-sfpplus2.

I tested with:
- 1G Ethernet module - Mikrotik S-RJ01 - 17% packet loss (First attempt)
- DIFFERNT 1G Ethernet module & Ethernet cable - Mikrotik S-RJ01 - 3% packet loss (Second attempt)
- 1G SM Fibre SFP - Ubqiuiti UF-SM-1G-S - 0% packet loss
- 1G MM Fibre SFP - Ubiquiti UF-MM-1G - 0% packet loss

Obviously something is wrong with the CCR2004 and 1G ethernet modules. In production I have the Router to Router connections as 10Gbps DAC cables and they get 0% packet loss.

Mikrotik S-RJ01:
[x@NSW-RTR-01] > ping 172.16.4.2 count=420 do-not-fragment size=1500 interface=sfp-sfpplus2
419 172.16.4.2 1500 64 0ms
sent=420 received=347 packet-loss=17% min-rtt=0ms avg-rtt=0ms max-rtt=0ms

Mikrotik S-RJ01:
[x@NSW-RTR-01] > ping 172.16.4.2 count=420 do-not-fragment size=1500 interface=sfp-sfpplus2
419 172.16.4.2 1500 64 0ms
sent=420 received=406 packet-loss=3% min-rtt=0ms avg-rtt=0ms max-rtt=0ms

Ubqiuiti UF-SM-1G-S:
[x@NSW-RTR-01] > ping 172.16.4.2 count=420 do-not-fragment size=1500 interface=sfp-sfpplus2
419 172.16.4.2 1500 64 0ms
sent=420 received=420 packet-loss=0% min-rtt=0ms avg-rtt=0ms max-rtt=0ms

Ubiquiti UF-MM-1G:
[x@NSW-RTR-01] > ping 172.16.4.2 count=420 do-not-fragment size=1500 interface=sfp-sfpplus2
419 172.16.4.2 1500 64 0ms
sent=420 received=420 packet-loss=0% min-rtt=0ms avg-rtt=0ms max-rtt=0ms
 
ozar
just joined
Posts: 12
Joined: Fri Jun 15, 2018 4:28 am

Re: CCR2004 packet loss

Fri Oct 30, 2020 2:49 pm

If I was you I would give shaping just below 1Gbps a shot and see if you still get the packet loss.

I don't use 1G Ethernet modules in productions so I don't have too much skin in the game, goodluck.
 
server8
Long time Member
Long time Member
Topic Author
Posts: 591
Joined: Fri Apr 22, 2011 1:27 pm

Re: CCR2004 packet loss

Fri Oct 30, 2020 6:20 pm

If I was you I would give shaping just below 1Gbps a shot and see if you still get the packet loss.

I don't use 1G Ethernet modules in productions so I don't have too much skin in the game, goodluck.
it not work
 
server8
Long time Member
Long time Member
Topic Author
Posts: 591
Joined: Fri Apr 22, 2011 1:27 pm

Re: CCR2004 packet loss

Fri Oct 30, 2020 6:22 pm

Can I have an official answer about this problem by mickrotik support? I am pinging every day for ticket [SUP-30387] wihtout any answer :-(
 
server8
Long time Member
Long time Member
Topic Author
Posts: 591
Joined: Fri Apr 22, 2011 1:27 pm

Re: CCR2004 packet loss

Mon Nov 02, 2020 9:23 am

Any update on this issue?
 
mkx
Forum Guru
Forum Guru
Posts: 8955
Joined: Thu Mar 03, 2016 10:23 pm

Re: CCR2004 packet loss

Mon Nov 02, 2020 3:27 pm

Can I have an official answer about this problem by mickrotik support? I am pinging every day for ticket [SUP-30387] wihtout any answer :-(

If you're not getting response through official channels, then you surely won't get any response here. This is user forum (where MT employees might check from time to time and answer some questions) not official support forum.
 
server8
Long time Member
Long time Member
Topic Author
Posts: 591
Joined: Fri Apr 22, 2011 1:27 pm

Re: CCR2004 packet loss

Tue Nov 03, 2020 1:00 am

I have a support tikcet opened without any answer too
 
oouups
just joined
Posts: 3
Joined: Tue Mar 27, 2012 7:33 pm

Re: CCR2004 packet loss

Wed Nov 04, 2020 2:03 pm

Exactly the same problem here !
Tired of playing "Betatester" for Mikrotik. Paid for something that really not Works.. and no one help of them.... THX Mikrotik.
Lost time, lost patience, lost money... and you loose a costumer.
 
server8
Long time Member
Long time Member
Topic Author
Posts: 591
Joined: Fri Apr 22, 2011 1:27 pm

Re: CCR2004 packet loss

Thu Nov 05, 2020 9:03 am

I am using back old CCR016 to connect gigabit site. CCR2004 is not usable in real because if we have problem to use different port speeds we can't use 25 Gb/s uplink ports with 10 gbis/s ports.
 
server8
Long time Member
Long time Member
Topic Author
Posts: 591
Joined: Fri Apr 22, 2011 1:27 pm

Re: CCR2004 packet loss

Fri Nov 06, 2020 8:31 am

Any update on this issue?
 
server8
Long time Member
Long time Member
Topic Author
Posts: 591
Joined: Fri Apr 22, 2011 1:27 pm

Re: CCR2004 packet loss

Mon Nov 09, 2020 8:29 am

Any update on this issue?
 
mkx
Forum Guru
Forum Guru
Posts: 8955
Joined: Thu Mar 03, 2016 10:23 pm

Re: CCR2004 packet loss

Mon Nov 09, 2020 10:21 am

Not much will happen in 3 days ... specially so not over weekend.
 
server8
Long time Member
Long time Member
Topic Author
Posts: 591
Joined: Fri Apr 22, 2011 1:27 pm

Re: CCR2004 packet loss

Sat Nov 21, 2020 9:39 am

There are a lot of users with same problem around and after 2 months no answer from mikrotik :-(

At this point I have to suppose that is è hw problem without solutions. CCR2004 is an expensive paperweight
 
User avatar
Maggiore81
Trainer
Trainer
Posts: 477
Joined: Sun Apr 15, 2012 12:10 pm
Location: Italy
Contact:

Re: CCR2004 packet loss

Sat Nov 21, 2020 10:21 am

Hello. I want to tell you my personal experience.
My experience with 2004 was bad because I replaced a 1036, but maybe the 2004 was too "light" for the purpose.

Now I have installed two 2004 with:

1x10g uplink
1x rj01 to AF24 ubiquiti
2x SFP+ 10G to other sites
everything in routing via BGP
0 issues and cpu low.
I tried also to disable conntrack for the routing (previously I used fasttrack everywhere) and 0 issues, just a higher cpu.
 
server8
Long time Member
Long time Member
Topic Author
Posts: 591
Joined: Fri Apr 22, 2011 1:27 pm

Re: CCR2004 packet loss

Sat Nov 21, 2020 5:33 pm

Please check the tx drop under the tab Tx Stats
 
User avatar
Maggiore81
Trainer
Trainer
Posts: 477
Joined: Sun Apr 15, 2012 12:10 pm
Location: Italy
Contact:

Re: CCR2004 packet loss

Sat Nov 21, 2020 5:39 pm

Hello
I got 1649 tx drop on the interface with an RJ01 with AIRFIBER24 from UBNT.
it 4 days and 4 hours.

it is a 500mbit link.
no issues from the customers.

maybe the issue you are describing?
The interfaces are routed between each other, so no mixed speeds in the same bridge.
 
server8
Long time Member
Long time Member
Topic Author
Posts: 591
Joined: Fri Apr 22, 2011 1:27 pm

Re: CCR2004 packet loss

Sat Nov 21, 2020 8:38 pm

Hello
I got 1649 tx drop on the interface with an RJ01 with AIRFIBER24 from UBNT.
it 4 days and 4 hours.

it is a 500mbit link.
no issues from the customers.

maybe the issue you are describing?
The interfaces are routed between each other, so no mixed speeds in the same bridge.
You are loosing packets if you check on SFP28 you 'll find zero tx drops.

Now I have all 10 gigabit pots but I loose packets onyl on sfplus ports.
 
User avatar
Maggiore81
Trainer
Trainer
Posts: 477
Joined: Sun Apr 15, 2012 12:10 pm
Location: Italy
Contact:

Re: CCR2004 packet loss

Sun Nov 22, 2020 12:32 am

Hello.
I use:
Sfp1 1g sfp
Sfp2 10g link
Sfp3 1g on rj01 with airfiber
Sfp11 and sfp12 on a bridge. Two 1g links

I have drops only on the sfp3 1g.
I don't use 25g ports
 
server8
Long time Member
Long time Member
Topic Author
Posts: 591
Joined: Fri Apr 22, 2011 1:27 pm

Re: CCR2004 packet loss

Tue Nov 24, 2020 3:25 pm

It depens how fast the trafffic coming on the 10 gibi/s port to be routed or swithed to the other ports.

We are speaking about 0,1% of packet loss that is not to much for a wireless link but is too much for a fiber link.
 
User avatar
Maggiore81
Trainer
Trainer
Posts: 477
Joined: Sun Apr 15, 2012 12:10 pm
Location: Italy
Contact:

Re: CCR2004 packet loss

Tue Nov 24, 2020 3:33 pm

Everything is routed!
 
wojrep
newbie
Posts: 42
Joined: Thu Apr 07, 2005 12:27 pm

Re: CCR2004 packet loss

Wed Nov 25, 2020 8:56 am

I have a similar problem, CCR2004 post sfp-sfplus2 - SFP 1G insert, no auto, flow controll On ... and I have a lot of drop packages.
They start to appear when I exceed the speed of 100Mbit / s, at 200-500Mbit / s I have several thousand of them per minute, multicast transmission is interrupted (frame stop and IPTV artifacts).
Changing queue improves things a bit, but fails to get stability.
 
User avatar
Maggiore81
Trainer
Trainer
Posts: 477
Joined: Sun Apr 15, 2012 12:10 pm
Location: Italy
Contact:

Re: CCR2004 packet loss

Wed Nov 25, 2020 10:01 am

Hello
I never found final documentation in how changing queue helps.
I leave them at only-hardware-queue
 
wojrep
newbie
Posts: 42
Joined: Thu Apr 07, 2005 12:27 pm

Re: CCR2004 packet loss

Wed Nov 25, 2020 10:30 am

Today I have tested all combinations with different queue sizes.
There is a difference, if you set ethernet-default and experiment with the size of the queue it may be enough for you.
I checked between 50-1000 packets and there was always a TX DROP problem, with queue values of 150-250 the least problem was. However, I am using multicast for IPTV, so dropping packets too often is unacceptable.
This problem occurs only with optical inserts with 1Gbps link speed, no matter if the insert itself is 1Gbps or 10Gbps.

I went back to CCR1036, wait for the new original MikroTik inserts and check it again.
 
User avatar
Maggiore81
Trainer
Trainer
Posts: 477
Joined: Sun Apr 15, 2012 12:10 pm
Location: Italy
Contact:

Re: CCR2004 packet loss

Wed Nov 25, 2020 2:27 pm

We use only hardware queue
on the wiki I didnt found any interesting data about changing queue types on interfaces.
 
User avatar
Maggiore81
Trainer
Trainer
Posts: 477
Joined: Sun Apr 15, 2012 12:10 pm
Location: Italy
Contact:

Re: CCR2004 packet loss

Sat Dec 19, 2020 12:56 pm

I have replaced both 2004 in heavy duty enveroniment (testing in a core section)

I installed 2x 2004 with FASTPATH and bgp only.
heavy packet loss on 10G interfaces
replaced with a UBNT edgeswitch 16XP and 1036+vlans, work flawlessy
with just 1.5g traffic, cpu at 30% (fastpath) and irq at 15, heavy packet loss.
changed cables, ports. no change.
latest longterm, tried also stable, no differencies.
Last edited by Maggiore81 on Sat Jun 05, 2021 10:52 am, edited 1 time in total.
 
mhugo
Member Candidate
Member Candidate
Posts: 176
Joined: Mon Sep 19, 2005 11:48 am

Re: CCR2004 packet loss

Sat Dec 19, 2020 5:50 pm

Hi,

We had issues with interfaces dropping using rj01 on 2004s. After replacing with fiber sfps the issues are gone. Could perhaps be power consumption.

/Mikael
 
User avatar
andrewe02000
Frequent Visitor
Frequent Visitor
Posts: 68
Joined: Tue Aug 28, 2012 6:33 am
Location: Canton, OH
Contact:

Re: CCR2004 packet loss

Wed Dec 30, 2020 1:16 am

I have the same problem and opened a ticket with no response over 2 weeks ago. Very annoying. The odd thing is that im only getting the tx drops on the interface going to my elan. At first I thought it was an mti mismatch but have since ruled that out. I even put a CRS309 in between my ELAN and my CCR2004 and the tx drops moved to the interface going to the ELAN on the CRS309 and stoped on my cloud core. I've tried finisar, mikrotik, and fiberstore modules with the same results. Even tried a 1G and 10G DAC from multiple vendors. My ISP for my ELAN checked their equipment and show no drops on their side or FCS errors. It's odd that the problem moved down to the interface on my CRS309 instead of staying on my CCR2004. Even odder is when I unplug the 10G DIA and switch over to my backup 1G DIA the TX drops go away and as soon as I plug my 10G DIA back in they come back. It is weird that the TX drops only show up once you open the interface and not in the column view when I select them. It always shows 0 in the column view on both the crs2004 and crs309 depending on whether or not im using the crs309 inline but shows up after double clicking the interface and going to stats. The other weird thing is that the tx drops don't show up on any other 1G interface on the CCR2004. I would expect it to be consistent at the very least. I set a lab up with all the smae routers and have yet to be able to reproduce the result in the lab and I believe this to be due to the fact that I am not able to tax the CPU load high enough to perturbate the issue. Please Mikrotik. HELP!!!!! My CCR2004 like many others is a paperweight as it is causing out of order packets and voice quality issues even thought the packet loss through the ccr2004 is low at 0.08 to 0.12 percent.
 
server8
Long time Member
Long time Member
Topic Author
Posts: 591
Joined: Fri Apr 22, 2011 1:27 pm

Re: CCR2004 packet loss

Wed Dec 30, 2020 9:37 am

Give it up I have opened a ticket 3 months a go without any answer. No chance to solve this issue.
 
markonen
just joined
Posts: 18
Joined: Tue Aug 11, 2020 4:28 pm

Re: CCR2004 packet loss

Wed Dec 30, 2020 11:55 am

I even put a CRS309 in between my ELAN and my CCR2004 and the tx drops moved to the interface going to the ELAN on the CRS309 and stoped on my cloud core. … It's odd that the problem moved down to the interface on my CRS309 instead of staying on my CCR2004.
So you think the CCR2004 is causing tx drops on a completely separate device (the 309)? I don't think that's possible.
 
wojrep
newbie
Posts: 42
Joined: Thu Apr 07, 2005 12:27 pm

Re: CCR2004 packet loss

Sat Jan 02, 2021 3:00 pm

There is a problem with Tx-Drop on mikrotik 10Gbps interfaces in combination with any 1Gbps interface.
Even at speeds of 100Mbps, TX-drops arrive quickly.
 
User avatar
andrewe02000
Frequent Visitor
Frequent Visitor
Posts: 68
Joined: Tue Aug 28, 2012 6:33 am
Location: Canton, OH
Contact:

Re: CCR2004 packet loss

Fri Jan 08, 2021 9:00 am

I opened a support ticket on 12/21/2020 and still no response! This is obviously an issue and we deserve answers. Anyone else had luck with a support ticket?
 
wojrep
newbie
Posts: 42
Joined: Thu Apr 07, 2005 12:27 pm

Re: CCR2004 packet loss

Fri Jan 08, 2021 11:13 am

I also have an open ticket, but I'm waiting longer ... no reaction.
 
mhaluska
just joined
Posts: 22
Joined: Sat Jun 13, 2020 1:20 pm

Re: CCR2004 packet loss

Sat Jan 09, 2021 12:41 am

I even put a CRS309 in between my ELAN and my CCR2004 and the tx drops moved to the interface going to the ELAN on the CRS309 and stoped on my cloud core. … It's odd that the problem moved down to the interface on my CRS309 instead of staying on my CCR2004.
So you think the CCR2004 is causing tx drops on a completely separate device (the 309)? I don't think that's possible.
I've also problem with tx drops on switch (CRS317). Never had this problem before with RB4011. Tx drop are only on interfaces connected to CCR2004. Tested "overloading" ports connected to server with CHR -> not drop, no issue.
 
wojrep
newbie
Posts: 42
Joined: Thu Apr 07, 2005 12:27 pm

Re: CCR2004 packet loss

Sat Jan 09, 2021 2:29 pm

I've also problem with tx drops on switch (CRS317). Never had this problem before with RB4011. Tx drop are only on interfaces connected to CCR2004. Tested "overloading" ports connected to server with CHR -> not drop, no issue.
In the case of connecting the 10G interface to the 1G interface and additionally overloading the 1G link with the test, the appearance of Tx-Drop is a rather natural situation.

It is not natural for them to appear even further up to 1G, e.g. at 200Mbps.
 
User avatar
andrewe02000
Frequent Visitor
Frequent Visitor
Posts: 68
Joined: Tue Aug 28, 2012 6:33 am
Location: Canton, OH
Contact:

Re: CCR2004 packet loss

Mon Jan 11, 2021 8:28 pm

At this point both units are bricks to me and I am going to return the units I purchased as they are paper weights and no response from the vendor. I get that we have COVID going on right now but a brand new product with multiple people having the same issue deserves an answer I would think. My network can't take much more of this before I have to break down and spend big bucks for Juniper.
 
User avatar
andrewe02000
Frequent Visitor
Frequent Visitor
Posts: 68
Joined: Tue Aug 28, 2012 6:33 am
Location: Canton, OH
Contact:

Re: CCR2004 packet loss

Mon Jan 11, 2021 8:44 pm

My carrier has been helpful on scheduling a 10G port upgrade on my 1G ELAN going to the CCR2004 to see if this helps on the 22nd. If this does not fix the issues I am shipping these back. Resume pulling hair out.
 
User avatar
andrewe02000
Frequent Visitor
Frequent Visitor
Posts: 68
Joined: Tue Aug 28, 2012 6:33 am
Location: Canton, OH
Contact:

Re: CCR2004 packet loss

Mon Jan 11, 2021 8:47 pm

Oh also of note trying to generate support.rif crashes the router horribly and requires physically removing power from the unit. Lovely.
 
server8
Long time Member
Long time Member
Topic Author
Posts: 591
Joined: Fri Apr 22, 2011 1:27 pm

Re: CCR2004 packet loss

Tue Jan 12, 2021 8:52 am

We have in production several CCR2004 and all have problems about packet loss on 10 Giga sfp+ ports no problem of packet loss on sfp28 ports used at 10 Giga.

It's an hardware problem without solutions.... :-(
 
mblfone
just joined
Posts: 24
Joined: Sun Feb 02, 2014 2:22 am

Re: CCR2004 packet loss

Tue Jan 12, 2021 10:03 pm

Hello,

I appreciate this post but do not appreciate the lack of response from MT on this. I need these CCR2004 routers but am NOT ordering them from MT due to this issue being unresolved.

Again, thanks for the posts!
 
server8
Long time Member
Long time Member
Topic Author
Posts: 591
Joined: Fri Apr 22, 2011 1:27 pm

Re: CCR2004 packet loss

Wed Jan 13, 2021 12:14 pm

Hello,

I appreciate this post but do not appreciate the lack of response from MT on this. I need these CCR2004 routers but am NOT ordering them from MT due to this issue being unresolved.

Again, thanks for the posts!
Give it up it's normal from mikrotik the lack of response when they can't resolve the problem.
 
User avatar
andrewe02000
Frequent Visitor
Frequent Visitor
Posts: 68
Joined: Tue Aug 28, 2012 6:33 am
Location: Canton, OH
Contact:

Re: CCR2004 packet loss

Thu Jan 21, 2021 6:02 pm

I am just going to post here every day this goes unanswered in the forum and my support ticket.SUP-37056 opened on Dec 21, 2020 11:30 EST goes unanswered. STILL NO ANSWER 1 MONTH LATER.
 
User avatar
andrewe02000
Frequent Visitor
Frequent Visitor
Posts: 68
Joined: Tue Aug 28, 2012 6:33 am
Location: Canton, OH
Contact:

Re: CCR2004 packet loss

Fri Jan 22, 2021 10:30 pm

I understand this. But the fact was we weren't maxing out our 1036 it was overpowered but we still didn't have a 10-gig interface. Even after disabling all of the firewall rules on this unit we still have problems with it being able to do line rate with what it says it can do without errors. Just last night I upgraded all of my circuits to 10 gig ports which took an arm and a leg to get my service provider to upgrade a one gig service to a 10 gig port but I digress. After all of that last night I still have TX drops on the interfaces and about 0.1% packet loss. I even tried turning off mpls and any other feature like ospf and did static routing and still had the same problem. it's not that the box is overloaded it's that the hardware has problems and the vendor is not acknowledging it.yes I am looking at the solution that you implemented as a possible solution compared to just buying an mx80 DC unit at 3K and being done with my life. if a vendor sells a product and says that it can do line rate 10 gig it should be able to do it. Anything besides that is either a lie or gross negligence. Not answering support tickets for customers or telling them that they're working on the problem or acknowledging them in anyway for over a month now. Day 31 for me. That's a problem. Network service providers expect communication from their communication vendors. Even if it's just a little bit instead of being left completely in the dark. There is my post for the day.

Sent from my Pixel 4a using Tapatalk


 
User avatar
andrewe02000
Frequent Visitor
Frequent Visitor
Posts: 68
Joined: Tue Aug 28, 2012 6:33 am
Location: Canton, OH
Contact:

Re: CCR2004 packet loss

Sat Jan 23, 2021 11:11 pm

@normis 1 month 2 days. Can we get a response to this in the forums or on our support tickets?
 
server8
Long time Member
Long time Member
Topic Author
Posts: 591
Joined: Fri Apr 22, 2011 1:27 pm

Re: CCR2004 packet loss

Sun Jan 24, 2021 1:37 pm

I have a ticket, on this issue, opened 5 months ago and I am still vaiting for an answer
 
User avatar
andrewe02000
Frequent Visitor
Frequent Visitor
Posts: 68
Joined: Tue Aug 28, 2012 6:33 am
Location: Canton, OH
Contact:

Re: CCR2004 packet loss

Sun Jan 24, 2021 9:18 pm

Another post for the day. 1 month and 3 days.

Sent from my Pixel 4a using Tapatalk

 
User avatar
andrewe02000
Frequent Visitor
Frequent Visitor
Posts: 68
Joined: Tue Aug 28, 2012 6:33 am
Location: Canton, OH
Contact:

Re: CCR2004 packet loss

Mon Jan 25, 2021 5:17 pm

And another day. 1 month 4 days.
 
User avatar
Paternot
Forum Veteran
Forum Veteran
Posts: 931
Joined: Thu Jun 02, 2016 4:01 am
Location: Niterói / Brazil

Re: CCR2004 packet loss

Mon Jan 25, 2021 8:24 pm

And another day. 1 month 4 days.
Stop this nonsense. You are just polluting the topic, and this helps no one - far from it.
 
User avatar
andrewe02000
Frequent Visitor
Frequent Visitor
Posts: 68
Joined: Tue Aug 28, 2012 6:33 am
Location: Canton, OH
Contact:

Re: CCR2004 packet loss

Fri Jan 29, 2021 5:48 pm

I'll stop when I get a response.
1 Month 8 days no response.
 
User avatar
andrewe02000
Frequent Visitor
Frequent Visitor
Posts: 68
Joined: Tue Aug 28, 2012 6:33 am
Location: Canton, OH
Contact:

Re: CCR2004 packet loss

Fri Jan 29, 2021 6:10 pm

If no one says anything then nothing will get done. We don't deserve to be left in the dark. Regardless of what's going on. The squeaky wheel gets the grease. I'm not going to stop squeaking until I get some grease.
 
mhugo
Member Candidate
Member Candidate
Posts: 176
Joined: Mon Sep 19, 2005 11:48 am

Re: CCR2004 packet loss

Sat Jan 30, 2021 12:10 am

If no one says anything then nothing will get done. We don't deserve to be left in the dark. Regardless of what's going on. The squeaky wheel gets the grease. I'm not going to stop squeaking until I get some grease.
You might be squeaking on your own - I unsubscribed because of the noise.

6.48.1 is out soon - lets hope for some more 2004 improvements in that.

/Mikael
Last edited by mhugo on Sat Jan 30, 2021 3:03 pm, edited 1 time in total.
 
User avatar
Paternot
Forum Veteran
Forum Veteran
Posts: 931
Joined: Thu Jun 02, 2016 4:01 am
Location: Niterói / Brazil

Re: CCR2004 packet loss

Sat Jan 30, 2021 2:15 am

If no one says anything then nothing will get done. We don't deserve to be left in the dark. Regardless of what's going on. The squeaky wheel gets the grease. I'm not going to stop squeaking until I get some grease.
Do a constructive squeaking, don't be this person. Post tests, ideas, suggestions. One post a day, just saying "I'm a victim, Mikrotik doesn't love me" don't help anyone - and will not solve the problem faster either.
 
Robervaldd
newbie
Posts: 40
Joined: Tue Nov 15, 2016 2:11 am

Re: CCR2004 packet loss

Mon Feb 01, 2021 11:42 pm

I have two ccr2004 here with the same packet loss problem, almost certainly the problem is in the hardware with no solution!
 
mikeeg02
Member Candidate
Member Candidate
Posts: 162
Joined: Fri Mar 30, 2018 2:28 am
Location: Pennsylvania

Re: CCR2004 packet loss

Wed Feb 03, 2021 3:04 pm

Could be interesting

6.49 released.

switch - improved packet transmit between CPU and 98PX1012 for CCR2004-1G-12S+2XS device;

viewtopic.php?f=21&t=172259&sid=b361e20 ... ddcaa12707

It's a beta, so test it out first.
 
mhugo
Member Candidate
Member Candidate
Posts: 176
Joined: Mon Sep 19, 2005 11:48 am

Re: CCR2004 packet loss

Mon Feb 08, 2021 5:21 pm

Could be interesting

6.49 released.

switch - improved packet transmit between CPU and 98PX1012 for CCR2004-1G-12S+2XS device;

viewtopic.php?f=21&t=172259&sid=b361e20 ... ddcaa12707

It's a beta, so test it out first.
Hi,

Anybody dared trying it?

/Mikael
 
User avatar
andrewe02000
Frequent Visitor
Frequent Visitor
Posts: 68
Joined: Tue Aug 28, 2012 6:33 am
Location: Canton, OH
Contact:

Re: CCR2004 packet loss

Mon Feb 08, 2021 7:46 pm

I ended up pulling my units and not a fan of running beta in production. Maybe I'll get a lab going and try to reproduce this.
 
mhugo
Member Candidate
Member Candidate
Posts: 176
Joined: Mon Sep 19, 2005 11:48 am

Re: CCR2004 packet loss

Mon Feb 15, 2021 7:07 pm

Could be interesting

6.49 released.

switch - improved packet transmit between CPU and 98PX1012 for CCR2004-1G-12S+2XS device;

viewtopic.php?f=21&t=172259&sid=b361e20 ... ddcaa12707

It's a beta, so test it out first.
6.49beta11 did not fix packetloss but interface instability. BUT beware - all ports negotiate in full 10G even if SFP is 1G. Works kind of until it becomes unstable causing ospf to flap.

Also got this which Im testing on one box right now. Kind of defeats the purpose on 10G interfaces but very fast initial testing seems promising and mostly we are below 1 gig so its peferable to packetloss.

Regarding packetloss apply these settings on SFP+ ports which report tx-drop

/queue type
add kind=pcq name=queue1 pcq-burst-rate=990M pcq-burst-time=20s \
pcq-limit=2000KiB pcq-rate=980M
/queue interface
set sfp-sfpplus1 queue=queue1

/Mikael
 
User avatar
andrewe02000
Frequent Visitor
Frequent Visitor
Posts: 68
Joined: Tue Aug 28, 2012 6:33 am
Location: Canton, OH
Contact:

Re: CCR2004 packet loss

Wed Feb 17, 2021 8:18 pm

Could be interesting

6.49 released.

switch - improved packet transmit between CPU and 98PX1012 for CCR2004-1G-12S+2XS device;

viewtopic.php?f=21&t=172259&sid=b361e20 ... ddcaa12707

It's a beta, so test it out first.
6.49beta11 did not fix packetloss but interface instability. BUT beware - all ports negotiate in full 10G even if SFP is 1G. Works kind of until it becomes unstable causing ospf to flap.

Also got this which Im testing on one box right now. Kind of defeats the purpose on 10G interfaces but very fast initial testing seems promising and mostly we are below 1 gig so its peferable to packetloss.

Regarding packetloss apply these settings on SFP+ ports which report tx-drop

/queue type
add kind=pcq name=queue1 pcq-burst-rate=990M pcq-burst-time=20s \
pcq-limit=2000KiB pcq-rate=980M
/queue interface
set sfp-sfpplus1 queue=queue1

/Mikael
I got this same message from tech support to try this exact same thing. It actually made the packet loss worse from 1 to 2 packets out of 1000 to 8 to 10 packets out of 10000 on any interface I put the rule on so yeah. Also did not fix tx drop errors.
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 207
Joined: Tue May 05, 2015 11:12 am
Location: 74, FR / SA48, UK
Contact:

Re: CCR2004 packet loss

Thu Feb 18, 2021 11:53 am

6.49beta11 did not fix packetloss but interface instability. BUT beware - all ports negotiate in full 10G even if SFP is 1G. Works kind of until it becomes unstable causing ospf to flap.
I got this same message from tech support to try this exact same thing. It actually made the packet loss worse from 1 to 2 packets out of 1000 to 8 to 10 packets out of 10000 on any interface I put the rule on so yeah. Also did not fix tx drop errors.
Same here (though what we were sent didn't look like a full command). We're having packet loss when using only two 10G interfaces, but only when routing production-like traffic (i.e. lots of src and dst addresses). A single iperf runs fine.

Support ticket to MikroTik updated with full details. We've been working with them for a month now — hopefully the test cases we've got are helping.
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 207
Joined: Tue May 05, 2015 11:12 am
Location: 74, FR / SA48, UK
Contact:

Re: CCR2004 packet loss

Thu Feb 18, 2021 4:49 pm

Same here (though what we were sent didn't look like a full command). We're having packet loss when using only two 10G interfaces, but only when routing production-like traffic (i.e. lots of src and dst addresses). A single iperf runs fine.

Support ticket to MikroTik updated with full details. We've been working with them for a month now — hopefully the test cases we've got are helping.
…and now same here with the 6.49beta11 "SFP at 1G is not stable" bug.
 
mhugo
Member Candidate
Member Candidate
Posts: 176
Joined: Mon Sep 19, 2005 11:48 am

Re: CCR2004 packet loss

Thu Feb 18, 2021 4:51 pm

…and now same here with the 6.49beta11 "SFP at 1G is not stable" bug.
For us the 1Gs report 10G. Same for you?

I think Mikrotik should make a seperate 2004 release based on long-term and fix this without mixing it with all other fixes.

/M
Last edited by mhugo on Thu Feb 18, 2021 6:24 pm, edited 3 times in total.
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 207
Joined: Tue May 05, 2015 11:12 am
Location: 74, FR / SA48, UK
Contact:

Re: CCR2004 packet loss

Thu Feb 18, 2021 5:06 pm

For us the 1Gs report 10G. Same for you?
Hard down, will not establish link. Light is received, but port does not negotiate or come up when set to 1G.
 
mhugo
Member Candidate
Member Candidate
Posts: 176
Joined: Mon Sep 19, 2005 11:48 am

Re: CCR2004 packet loss

Wed Mar 03, 2021 10:20 pm

For us the 1Gs report 10G. Same for you?
Hard down, will not establish link. Light is received, but port does not negotiate or come up when set to 1G.
Is anybody getting any feedback on support tickets from Mikrotik? I was told that they are working on it but not idea if any fix seems imminent or if this will first be resolved in ROS7...

We have replaced the worst offenders with 1072s but are stuck with 20 2004s we had hoped to roll out and a need to upgrade 25 more sites with 2004s.

/M
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 207
Joined: Tue May 05, 2015 11:12 am
Location: 74, FR / SA48, UK
Contact:

Re: CCR2004 packet loss

Wed Mar 03, 2021 10:23 pm

For us the 1Gs report 10G. Same for you?
Hard down, will not establish link. Light is received, but port does not negotiate or come up when set to 1G.
Is anybody getting any feedback on support tickets from Mikrotik? I was told that they are working on it but not idea if any fix seems imminent or if this will first be resolved in ROS7...

We have replaced the worst offenders with 1072s but are stuck with 20 2004s we had hoped to roll out and a need to upgrade 25 more sites with 2004s.

/M
Occasionally getting responses — we're being told to send supouts, console logs, change L2MTU, try setting ports to fixed speeds — but no visibility of what progress is being made.
 
mhugo
Member Candidate
Member Candidate
Posts: 176
Joined: Mon Sep 19, 2005 11:48 am

Re: CCR2004 packet loss

Mon Mar 08, 2021 10:22 pm

A week ago I was told by MT support they were hoping to have a fix out in the next beta. Seems one was launched today but nothing in release notes, but could be non-documented. Anybody have any more information?

/Mikael
 
onnoossendrijver
Member
Member
Posts: 468
Joined: Mon Jul 14, 2008 11:10 am
Location: The Netherlands

Re: CCR2004 packet loss

Mon Mar 08, 2021 10:57 pm

A week ago I was told by MT support they were hoping to have a fix out in the next beta. Seems one was launched today but nothing in release notes, but could be non-documented. Anybody have any more information?

/Mikael
What about: *) switch - improved packet transmit between CPU and 98PX1012 for CCR2004-1G-12S+2XS device
 
mhugo
Member Candidate
Member Candidate
Posts: 176
Joined: Mon Sep 19, 2005 11:48 am

Re: CCR2004 packet loss

Tue Mar 09, 2021 11:00 am

A week ago I was told by MT support they were hoping to have a fix out in the next beta. Seems one was launched today but nothing in release notes, but could be non-documented. Anybody have any more information?

/Mikael
What about: *) switch - improved packet transmit between CPU and 98PX1012 for CCR2004-1G-12S+2XS device
That was only for port stability according to MT support and already present in beta 11.

/M
 
markonen
just joined
Posts: 18
Joined: Tue Aug 11, 2020 4:28 pm

Re: CCR2004 packet loss

Tue Mar 09, 2021 11:17 am

There is a new related item in the latest beta:
*) switch - improved resource allocation on 98PX1012 switch chip for CCR2004-1G-12S+2XS device;
 
mhugo
Member Candidate
Member Candidate
Posts: 176
Joined: Mon Sep 19, 2005 11:48 am

Re: CCR2004 packet loss

Tue Mar 09, 2021 11:19 am

There is a new related item in the latest beta:
*) switch - improved resource allocation on 98PX1012 switch chip for CCR2004-1G-12S+2XS device;
Just got confirmation

The latest RouterOS v6.49beta22 has improved resource allocation which has been extensively tested before the release and addresses CCR2004 performance issues in various setups. Please upgrade CCR2004 to see if it now works fine in your planned application.

From MT support.

/M
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 207
Joined: Tue May 05, 2015 11:12 am
Location: 74, FR / SA48, UK
Contact:

Re: CCR2004 packet loss

Tue Mar 09, 2021 2:36 pm

There is a new related item in the latest beta:
*) switch - improved resource allocation on 98PX1012 switch chip for CCR2004-1G-12S+2XS device;
Just got confirmation

The latest RouterOS v6.49beta22 has improved resource allocation which has been extensively tested before the release and addresses CCR2004 performance issues in various setups. Please upgrade CCR2004 to see if it now works fine in your planned application.
We've had the same, and will be deploying it on pre-production routers tonight.
 
server8
Long time Member
Long time Member
Topic Author
Posts: 591
Joined: Fri Apr 22, 2011 1:27 pm

Re: CCR2004 packet loss

Tue Mar 09, 2021 4:12 pm

We've had the same, and will be deploying it on pre-production routers tonight.
Maznu let know us if it works thx Giuseppe
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 207
Joined: Tue May 05, 2015 11:12 am
Location: 74, FR / SA48, UK
Contact:

Re: CCR2004 packet loss

Tue Mar 09, 2021 4:14 pm

We've had the same, and will be deploying it on pre-production routers tonight.
Maznu let know us if it works thx Giuseppe
We'll know if it doesn't immediate crash and burn later tonight… but from experience it might be a few weeks before we know if it's stable or not. Will keep this thread updated as we go.
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 207
Joined: Tue May 05, 2015 11:12 am
Location: 74, FR / SA48, UK
Contact:

Re: CCR2004 packet loss

Tue Mar 09, 2021 11:27 pm

We've had the same, and will be deploying it on pre-production routers tonight.
Maznu let know us if it works thx Giuseppe
We'll know if it doesn't immediate crash and burn later tonight… but from experience it might be a few weeks before we know if it's stable or not. Will keep this thread updated as we go.
10 minutes on 6.49beta22 so far, no issues to report. This device only has 10G links in the SFP+ ports, though, so your mileage may vary with SFP or SFP28.
 
server8
Long time Member
Long time Member
Topic Author
Posts: 591
Joined: Fri Apr 22, 2011 1:27 pm

Re: CCR2004 packet loss

Wed Mar 10, 2021 8:13 am

10 minutes on 6.49beta22 so far, no issues to report. This device only has 10G links in the SFP+ ports, though, so your mileage may vary with SFP or SFP28.
So ROS 6.49.22 version solve the packet loss problem?
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 207
Joined: Tue May 05, 2015 11:12 am
Location: 74, FR / SA48, UK
Contact:

Re: CCR2004 packet loss

Wed Mar 10, 2021 8:47 am

10 minutes on 6.49beta22 so far, no issues to report. This device only has 10G links in the SFP+ ports, though, so your mileage may vary with SFP or SFP28.
So ROS 6.49.22 version solve the packet loss problem?
I don't think it does, at least not if you have many thousands of routes with nexthops on different physical interfaces.
 
User avatar
IPANetEngineer
Trainer
Trainer
Posts: 1676
Joined: Fri Aug 10, 2012 6:46 am
Location: iparchitechs.com
Contact:

Re: CCR2004 packet loss

Wed Mar 10, 2021 5:59 pm

I've been testing the 2004 with the 25G interfaces only on an fs.com 5860-20SQ switch stack in our lab and haven't seen the same issues as people using the 10G ports, so i'll be interested to load this version and try the 10G interfaces.
 
server8
Long time Member
Long time Member
Topic Author
Posts: 591
Joined: Fri Apr 22, 2011 1:27 pm

Re: CCR2004 packet loss

Tue Mar 16, 2021 9:40 am

Anyone has good news? :-)
 
mhugo
Member Candidate
Member Candidate
Posts: 176
Joined: Mon Sep 19, 2005 11:48 am

Re: CCR2004 packet loss

Wed Mar 24, 2021 12:48 am

Anyone has good news? :-)
Most has been fixed in last RC but still some left. MT says they are improving it still but the fix itself is well tested.

/M
 
server8
Long time Member
Long time Member
Topic Author
Posts: 591
Joined: Fri Apr 22, 2011 1:27 pm

Re: CCR2004 packet loss

Wed Mar 24, 2021 1:05 am

Anyone has good news? :-)
Most has been fixed in last RC but still some left. MT says they are improving it still but the fix itself is well tested.

/M
thank you
 
mikeeg02
Member Candidate
Member Candidate
Posts: 162
Joined: Fri Mar 30, 2018 2:28 am
Location: Pennsylvania

Re: CCR2004 packet loss

Wed Mar 24, 2021 2:30 am


Most has been fixed in last RC but still some left. MT says they are improving it still but the fix itself is well tested.

/M
Hows the uptime look so far?
 
mhugo
Member Candidate
Member Candidate
Posts: 176
Joined: Mon Sep 19, 2005 11:48 am

Re: CCR2004 packet loss

Sun Apr 11, 2021 11:53 pm

Hows the uptime look so far?
No reboots yet on beta22.

/M
 
asadsayyad123
just joined
Posts: 4
Joined: Tue Feb 14, 2017 12:38 pm

Re: CCR2004 packet loss

Sat Apr 17, 2021 6:34 pm

Hello,

The router causes packet loss at regular intervals when the routing process eats up the CPU up to 24%.

I have observed packet loss only when the BGP routing updates are sent/received it causes the routing process on the router to spike up to 24% and hence causing packets to drop.

ROS 6.48.2

Thank you.
 
server8
Long time Member
Long time Member
Topic Author
Posts: 591
Joined: Fri Apr 22, 2011 1:27 pm

Re: CCR2004 packet loss

Sun Apr 18, 2021 12:59 pm

I can confrim we upgraded all our CCR2004 and the issue is still here on the PtP beetween them, old CCR1016 works perfectly
 
User avatar
andrewe02000
Frequent Visitor
Frequent Visitor
Posts: 68
Joined: Tue Aug 28, 2012 6:33 am
Location: Canton, OH
Contact:

Re: CCR2004 packet loss

Wed Jun 09, 2021 10:38 pm

The latest use for your CCR2004!
Image
 
mikeeg02
Member Candidate
Member Candidate
Posts: 162
Joined: Fri Mar 30, 2018 2:28 am
Location: Pennsylvania

Re: CCR2004 packet loss

Thu Jun 10, 2021 12:00 am

Have they implemented the self destruct option in software for them yet?

I have 8 or 10 of them sitting behind me taken out of production.

Still frustrated with that financial contribution.

Hoping someone reports life is good with them again so I can use them for anything.
 
User avatar
Maggiore81
Trainer
Trainer
Posts: 477
Joined: Sun Apr 15, 2012 12:10 pm
Location: Italy
Contact:

Re: CCR2004 packet loss

Thu Jun 10, 2021 8:10 am

We sent them in rma... they come back untouched with a message that states: rhe issues you have for is a issue that can be resolved with a software fix...
 
biziiac
just joined
Posts: 2
Joined: Mon Nov 30, 2020 12:13 pm

Re: CCR2004 packet loss

Tue Nov 16, 2021 9:10 am

I have the same problem and opened a ticket with no response over 2 weeks ago. Very annoying. The odd thing is that im only getting the tx drops on the interface going to my elan. At first I thought it was an mti mismatch but have since ruled that out. I even put a CRS309 in between my ELAN and my CCR2004 and the tx drops moved to the interface going to the ELAN on the CRS309 and stoped on my cloud core. I've tried finisar, mikrotik, and fiberstore modules with the same results. Even tried a 1G and 10G DAC from multiple vendors. My ISP for my ELAN checked their equipment and show no drops on their side or FCS errors. It's odd that the problem moved down to the interface on my CRS309 instead of staying on my CCR2004. Even odder is when I unplug the 10G DIA and switch over to my backup 1G DIA the TX drops go away and as soon as I plug my 10G DIA back in they come back. It is weird that the TX drops only show up once you open the interface and not in the column view when I select them. It always shows 0 in the column view on both the crs2004 and crs309 depending on whether or not im using the crs309 inline but shows up after double clicking the interface and going to stats. The other weird thing is that the tx drops don't show up on any other 1G interface on the CCR2004. I would expect it to be consistent at the very least. I set a lab up with all the smae routers and have yet to be able to reproduce the result in the lab and I believe this to be due to the fact that I am not able to tax the CPU load high enough to perturbate the issue. Please Mikrotik. HELP!!!!! My CCR2004 like many others is a paperweight as it is causing out of order packets and voice quality issues even thought the packet loss through the ccr2004 is low at 0.08 to 0.12 percent.
Hi,

I have exactly the same situation. Did you ever managed to make it work?

Thanks
Adrian
 
mhugo
Member Candidate
Member Candidate
Posts: 176
Joined: Mon Sep 19, 2005 11:48 am

Re: CCR2004 packet loss

Mon Nov 22, 2021 5:14 pm

I have exactly the same situation. Did you ever managed to make it work?
Seems to still be the same in ROS7 too. Just did testing with a couple.
 
ksoze
just joined
Posts: 8
Joined: Fri Nov 26, 2021 11:28 pm

Re: CCR2004 packet loss

Fri Nov 26, 2021 11:31 pm

I have the same issue on ROS 7.1. It's maddening: you can't use a router which drops packets between two 10 GBE SFP+ interfaces. I'll likely switch vendors if this cannot be resolved.
 
User avatar
Splash
Member Candidate
Member Candidate
Posts: 200
Joined: Fri Oct 16, 2015 10:09 am
Location: Johannesburg, South Africa

Re: CCR2004 packet loss

Mon Dec 06, 2021 12:40 pm

Running 6.49.1 with the queue fix I am picking up packet loss only when BGP is syncing routes (100k). If I disable the BGP session then the router shows no signs of packet loss.

Packet loss is also experienced between the router and the internal network.

Anyone know of a way to stop BGP impacting the packet flow when syncing... ie it seems maybe when the CPU running the BGP process, packets in that CPU's queue timeout/drop/die.
 
mhugo
Member Candidate
Member Candidate
Posts: 176
Joined: Mon Sep 19, 2005 11:48 am

Re: CCR2004 packet loss

Mon Dec 06, 2021 1:32 pm

Running 6.49.1 with the queue fix I am picking up packet loss only when BGP is syncing routes (100k). If I disable the BGP session then the router shows no signs of packet loss.

Packet loss is also experienced between the router and the internal network.

Anyone know of a way to stop BGP impacting the packet flow when syncing... ie it seems maybe when the CPU running the BGP process, packets in that CPU's queue timeout/drop/die.
Sounds like something 7.1 will fix unfortunatly.

What queuefix are you running?
 
User avatar
Splash
Member Candidate
Member Candidate
Posts: 200
Joined: Fri Oct 16, 2015 10:09 am
Location: Johannesburg, South Africa

Re: CCR2004 packet loss

Mon Dec 06, 2021 2:47 pm

/queue type set ethernet-default pfifo-limit=300
/queue interface set [find where queue!=no-queue] queue=ethernet-default

viewtopic.php?p=895484#p895484
 
User avatar
Splash
Member Candidate
Member Candidate
Posts: 200
Joined: Fri Oct 16, 2015 10:09 am
Location: Johannesburg, South Africa

Re: CCR2004 packet loss

Mon Dec 06, 2021 2:48 pm

Running 6.49.1 with the queue fix I am picking up packet loss only when BGP is syncing routes (100k). If I disable the BGP session then the router shows no signs of packet loss.

Packet loss is also experienced between the router and the internal network.

Anyone know of a way to stop BGP impacting the packet flow when syncing... ie it seems maybe when the CPU running the BGP process, packets in that CPU's queue timeout/drop/die.
Sounds like something 7.1 will fix unfortunatly.

What queuefix are you running?
I see 7.1 stable has been released.... so might look at trying it, but how to upgrade remotely will be they key, since BGP route filters are not updated in to 7.1
 
mhugo
Member Candidate
Member Candidate
Posts: 176
Joined: Mon Sep 19, 2005 11:48 am

Re: CCR2004 packet loss

Mon Dec 06, 2021 2:57 pm

I see 7.1 stable has been released.... so might look at trying it, but how to upgrade remotely will be they key, since BGP route filters are not updated in to 7.1
Beware a little - we had issues with something like L2 in the routers after upgrade. We managed to get into it using 30 attempts of mac-telnet and a couple of reboots after just made it work.

Only on 2004s and random 2004s all through upgrading between different rcs and now 7.1.

But after the 2004s has been rock solid for us.

/M
 
mhugo
Member Candidate
Member Candidate
Posts: 176
Joined: Mon Sep 19, 2005 11:48 am

Re: CCR2004 packet loss

Mon Dec 06, 2021 2:58 pm

/queue type set ethernet-default pfifo-limit=300
/queue interface set [find where queue!=no-queue] queue=ethernet-default

viewtopic.php?p=895484#p895484
We were instructed to use 500 and doing so on 7.x too.

/M
 
mhugo
Member Candidate
Member Candidate
Posts: 176
Joined: Mon Sep 19, 2005 11:48 am

Re: CCR2004 packet loss

Tue Jan 11, 2022 8:00 pm

/queue type set ethernet-default pfifo-limit=300
/queue interface set [find where queue!=no-queue] queue=ethernet-default

viewtopic.php?p=895484#p895484
We were instructed to use 500 and doing so on 7.x too.

/M
Just a clarification if someone reads this thread and sees my "rock solid". We dont have any reboots anymore, but we still have the packetloss on 2004s running 7.1.1.
 
ivicask
Member
Member
Posts: 380
Joined: Tue Jul 07, 2015 2:40 pm
Location: Croatia, Zagreb

Re: CCR2004 packet loss

Tue Jan 11, 2022 8:52 pm

I have also alot of packet lost on ethernet interfaces and some on SFP when going from 10gb to 1gb, the fifo suggestion made it even worse, some ports have 100 000 drops in few days with it...
CRS354-48G-4S+2Q+RM
 
User avatar
andrewe02000
Frequent Visitor
Frequent Visitor
Posts: 68
Joined: Tue Aug 28, 2012 6:33 am
Location: Canton, OH
Contact:

Re: CCR2004 packet loss

Wed Mar 02, 2022 5:57 pm

I have the same problem and opened a ticket with no response over 2 weeks ago. Very annoying. The odd thing is that im only getting the tx drops on the interface going to my elan. At first I thought it was an mti mismatch but have since ruled that out. I even put a CRS309 in between my ELAN and my CCR2004 and the tx drops moved to the interface going to the ELAN on the CRS309 and stoped on my cloud core. I've tried finisar, mikrotik, and fiberstore modules with the same results. Even tried a 1G and 10G DAC from multiple vendors. My ISP for my ELAN checked their equipment and show no drops on their side or FCS errors. It's odd that the problem moved down to the interface on my CRS309 instead of staying on my CCR2004. Even odder is when I unplug the 10G DIA and switch over to my backup 1G DIA the TX drops go away and as soon as I plug my 10G DIA back in they come back. It is weird that the TX drops only show up once you open the interface and not in the column view when I select them. It always shows 0 in the column view on both the crs2004 and crs309 depending on whether or not im using the crs309 inline but shows up after double clicking the interface and going to stats. The other weird thing is that the tx drops don't show up on any other 1G interface on the CCR2004. I would expect it to be consistent at the very least. I set a lab up with all the smae routers and have yet to be able to reproduce the result in the lab and I believe this to be due to the fact that I am not able to tax the CPU load high enough to perturbate the issue. Please Mikrotik. HELP!!!!! My CCR2004 like many others is a paperweight as it is causing out of order packets and voice quality issues even thought the packet loss through the ccr2004 is low at 0.08 to 0.12 percent.
Hi,

I have exactly the same situation. Did you ever managed to make it work?

Thanks
Adrian
I have not. I am still waiting for route aggregation to get added to ROS7 and for ROS7 to get a little more seasoned with time before I try again. Hopefully in a few months I will be ready to bring the big brick out to see if it is still a paperweight for my desk or not.
 
elmemis
just joined
Posts: 8
Joined: Sat Oct 01, 2016 6:49 am

Re: CCR2004 packet loss

Tue Jul 26, 2022 12:20 am

hi,

Does anybody have a packets losses?
We have several CCR2004 devices, and all have packet losses (few, but does exist). In our devices, we are working with 6.48.2, 6.48.3, 6.49.2, and 7.4. All have packet losses.
The load in all devices is less than 10%.

We tried with the queue fix but not working, the packet losses still persist.

We have generate supout.rif files and see in `rif viewer` tool tx/rx pauses between switch0-switch-1. Is normal this situation?
 
User avatar
chechito
Forum Guru
Forum Guru
Posts: 2548
Joined: Sun Aug 24, 2014 3:14 am
Location: Bogota Colombia
Contact:

Re: CCR2004 packet loss

Tue Jul 26, 2022 2:28 am

What percentage of packets lost do you have?
 
elmemis
just joined
Posts: 8
Joined: Sat Oct 01, 2016 6:49 am

Re: CCR2004 packet loss

Tue Jul 26, 2022 2:44 am

What percentage of packets lost do you have?
1%-2%, but is bad, with ccr1009 or ccr1036 we haven't any issue.
 
server8
Long time Member
Long time Member
Topic Author
Posts: 591
Joined: Fri Apr 22, 2011 1:27 pm

Re: CCR2004 packet loss

Tue Jul 26, 2022 8:27 am

It's ha hardware problem due to cache missing on the switch chip, it can not be solved
 
User avatar
chechito
Forum Guru
Forum Guru
Posts: 2548
Joined: Sun Aug 24, 2014 3:14 am
Location: Bogota Colombia
Contact:

Re: CCR2004 packet loss

Tue Jul 26, 2022 3:26 pm

Looking all the CCR 2xxx product line i think is a very strange move from MikroTik the way CCR2004 products where designed (both)

i think It would have been better to use a simple switch chip like 98DX3236 (same used in crs 328) and follow the trend of integrating switch chip on CCR 2xxx product line
 
mhugo
Member Candidate
Member Candidate
Posts: 176
Joined: Mon Sep 19, 2005 11:48 am

Re: CCR2004 packet loss

Wed Jul 27, 2022 9:12 pm

It's ha hardware problem due to cache missing on the switch chip, it can not be solved
The packetloss? Then the 2004 wont be usable is the conclusion?
 
server8
Long time Member
Long time Member
Topic Author
Posts: 591
Joined: Fri Apr 22, 2011 1:27 pm

Re: CCR2004 packet loss

Thu Jul 28, 2022 8:35 am

To avoid the packet loss you must use the same speed on all sfp ports
 
flapviv
just joined
Posts: 7
Joined: Wed Oct 13, 2021 7:50 am

Re: CCR2004 packet loss

Thu Jul 28, 2022 10:58 am

To avoid the packet loss you must use the same speed on all sfp ports
Very interesting information...
Thanx.
 
issme
just joined
Posts: 12
Joined: Wed Sep 16, 2020 11:38 pm

Re: CCR2004 packet loss

Thu Jul 28, 2022 11:04 am

I haven't seen an issue with packet loss when running mismatched speeds, but have definitely experienced packet loss with 10G DACs from FS.com (the same FS DACs work without issue on gear from other vendors). Have not experienced any issues when using 10G StarTech DACs.
 
User avatar
carl0s
Member Candidate
Member Candidate
Posts: 151
Joined: Thu Jun 25, 2009 7:18 pm

Re: CCR2004 packet loss

Sat Aug 13, 2022 2:44 am

So, as well as the CRS354 being firmly on the avoid list, the CCR2004 is as well?

Is there any high end Mikrotik kit that is safe to buy? I would like a high end router with at least two SFP+ 10G ports (I had to go elsewhere for the switches :-( )
 
mtsphere
just joined
Posts: 4
Joined: Thu Jun 22, 2017 3:53 am

Re: CCR2004 packet loss

Wed Dec 28, 2022 2:53 am

Just to add that these crapy tx drops are also present in ROSv7.6.

Cheers
 
mhugo
Member Candidate
Member Candidate
Posts: 176
Joined: Mon Sep 19, 2005 11:48 am

Re: CCR2004 packet loss

Wed Dec 28, 2022 4:04 am

Im told 7.7 will fix it but haven't tried yet.
 
User avatar
sirbryan
Member Candidate
Member Candidate
Posts: 139
Joined: Fri May 29, 2020 6:40 pm

Re: CCR2004 packet loss

Wed Dec 28, 2022 6:55 am

I had much better success in tests under 7.4.1 (up to 19.4Gbps full duplex) than 7.6 and some of the 7.7 betas. Here's hoping the 7.7 finals will help with that.

Who is online

Users browsing this forum: No registered users and 5 guests