19 Replies Latest reply on Feb 18, 2013 11:50 AM by david

    Seeing Errored Seconds and Controlled Slip Seconds on TA-904

    sthenge New Member

      We recently starting using software release A5-03-00 on newly installed 900 IADs.  We are seeing Errored seconds and Controlled Slip Seconds on the network T1.  We are connecting to a 5000 shelf to a DS1 PPP 32port card.  All IADs using this software release on a number of different 5000 shelves is experiencing these errors.  Customers are also complaining of difficulty sending and receiving faxes.  We connect faxes on the FXS ports using mgcp.  We were not experiencing this on A02.07 versions of the software.  Are others seeing this problem and is there a solution short of going back to an earlier version of the software?

       

      Thanks,

       

      Bill Stoner

      Daystar Communications

      941-206-7845

        • Re: Seeing Errored Seconds and Controlled Slip Seconds on TA-904
          unified Past_Featured_Member

          Please send the output of debug isdn l2-formatted and check the event log to see if there are any errors.

           

          You can try downgrading to A4.11.00.

          • Re: Seeing Errored Seconds and Controlled Slip Seconds on TA-904
            david Employee

            Mr. Stoner,

             

            Thanks for posting!  I believe I have found another reported issue in A5.03.00 firmware that indicates CSS on the web GUI when doing continuous monitoring of the T-1 interface.  At this time, this appears to be a cosmetic/display issue and not indicative of actual clock slips on the T-1 interface.  Can you verify how and where exactly you are seeing the slips?  Also, do you know if A4.11.00 or A5.02.00 firmware exhibit the problem?

             

            Thanks,

            David

              • Re: Seeing Errored Seconds and Controlled Slip Seconds on TA-904
                sthenge New Member

                David,

                 

                 

                 

                I am seeing the slips on both the GUI and in the cli, however, I wondered if it was just a display error too.  I don’t see this on version A4.11 and I haven’t tried A5.02.  We were first alerted to this when a customer started complaining about fax calls not working on a B1 line on one of the FXS ports configured for MGCP.  Again, not sure if the slips are real or just cosmetic or are a factor in the faxing issues too. 

                 

                 

                 

                The other symptom is on the Call Quality Stats where we are seeing hundreds of thousands of “Out of Order” packets as noted in the screenshot below many calls.  I’m not sure when this field was introduced though as I went from A2.07 to A5.03, but this is way not right if it is real.

                 

                 

                 

                Thanks for any insight you can provide.

                 

                 

                 

                Bill

                 

                 

                 

                 

                 

                 

                 

                Bill

                 

                Network Engineer

                 

                Daystar Communications

                 

                941-206-7845

                • Re: Seeing Errored Seconds and Controlled Slip Seconds on TA-904
                  sthenge New Member

                  Dave,

                   

                  Since my last post I had a customer that was seeing the slips in his PBX.  I rolled the firmware back to A2.07.  The slips stopped on both the Network and DSX-1 interfaces on the IAD and the slips stopped on his PBX.  They were complaining of faxing issues too, but have not determined if the rollback has fixed this issue too.

                   

                  This is offered as information in case Atran wishes to investigate the slipping issues.

                   

                  Thanks,

                   

                  Bill

                    • Re: Seeing Errored Seconds and Controlled Slip Seconds on TA-904
                      david Employee

                      Bill,

                       

                      At this point, we believe the CSS counter increments because of web GUI monitoring of T-1 interfaces in A5.03.00 firmware.  This would show up in the CLI after the fact.  To test this, we could set one of your units to A5.03.00 firmware and disable HTTP and HTTPS on the unit, then issue a reboot.  If you still see CSS, we will likely need to troubleshoot directly on one of your units.  In this case, it would be better to handle this with a regular Technical Support ticket by calling 888-423-8726.  We could also look into the voice quality statistics and any fax issues.

                       

                      Thanks!

                      David

                        • Re: Seeing Errored Seconds and Controlled Slip Seconds on TA-904
                          sthenge New Member

                          David,

                           

                           

                           

                          I believe that there is a problem in the system clock.  The T1 clock parameter changed somewhere between A4.11 and A5-03 replacing the system clock with a choice of Line, Internal, or Through.  This tells me something changed in that portion of the firmware enough to suspect there may be a problem with it.

                           

                           

                           

                          When the PBX vendor said he was getting slips on his pri and after downgrading the IAD to A2.07, his slips stopped, I can only assume the problem is not just cosmetic.  Adtran can set this up in their lab to test it if they want to pursue a fix. I will continue to run A2.07, which seems to be error free enough for my use.

                           

                           

                           

                          Thanks for your interest and suggestions… I hope Adtran will take my observations and look into it.

                           

                           

                           

                          Bill

                           

                          Network Engineer

                           

                          Daystar Communications

                           

                          941-206-7845

                    • Re: Seeing Errored Seconds and Controlled Slip Seconds on TA-904
                      jbiegert New Member

                      I'm using a not yet released code by Adtran for the TA908 (R10-2-1) to fix another issue they admitted to, starting in A4.08. and seeing the same Errored Seconds & Controlled Slip Seconds in this version. My T-1 is clean. The issue seems to be software related. I 'm sure Adtran is aware of the matter but unsure what they are doing to address it.

                      • Re: Seeing Errored Seconds and Controlled Slip Seconds on TA-904
                        david Employee

                        Bill,

                         

                        The only issue we have been able to reproduce with clock slips occurs only when the web GUI is used to monitor or change settings on the T-1 interface.  This issue is currently being investigated and will be fixed in a future firmware release. 

                         

                        If you disable the web GUI, save, and then reboot the unit, you should not see this known problem.  The save and reboot is important because the unit will continue to report clock slips if the T-1 interface is accessed within the GUI just one time.  If you are still seeing clock slips, please let us know because we may have another issue to troubleshoot.

                         

                        Thanks,

                        David

                          • Re: Seeing Errored Seconds and Controlled Slip Seconds on TA-904
                            sthenge New Member

                            David,

                             

                             

                             

                            I see what you are saying.  Slips don’t occur until you go the T1 interface on the GUI.  So although this is what triggers it, is it correct to say that once triggered the slips are real and will affect packet transmission on the wire?

                             

                             

                             

                            Thanks,

                             

                             

                             

                             

                             

                            Bill

                             

                            Network Engineer

                             

                            Daystar Communications

                             

                            941-206-7845

                              • Re: Seeing Errored Seconds and Controlled Slip Seconds on TA-904
                                david Employee

                                Bill,

                                 

                                It appears that the CSS counts increasing is just a cosmetic issue.  It is not that actual slips are occurring on the T-1, they are just being falsely reported in the interface statistics.

                                 

                                Thanks,

                                David

                                  • Re: Seeing Errored Seconds and Controlled Slip Seconds on TA-904
                                    sthenge New Member

                                    David,

                                     

                                     

                                     

                                    Just a follow up to our conversation.  I put a T-Berd T1 tester on the circuit and observed that the transmit frequency on the Network T1 0/1 drops from 1544000 bps to 1543986 bps. Not a  significant change, but is it enough cause slips?  I don’t know.  I get the same results when I place the system clock source on internal.  To me this is indicative of the clock source for the Network T1 being changed from “Line” to “Internal” when you look  at the T1 0/1 interface in the GUI.  Note that this can be reversed or corrected by cycling the system clock source Primary and Secondary to internal and then back to Primary T1 0/1 and Secondary T1 0/2.  Note also that since the T1 0/2 (DSX-1 ) interface is clock source is set to “through” its transmit frequency follows the Network clock source and I observed this on the T-Berd.

                                     

                                     

                                     

                                    Again, just food for thought for the engineers looking into the problem.

                                     

                                     

                                     

                                    Bill

                                     

                                    Network Engineer

                                     

                                    Daystar Communications

                                     

                                    941-206-7845

                              • Re: Seeing Errored Seconds and Controlled Slip Seconds on TA-904
                                jessiebryan New Member

                                This is also a known issue with A5.03 - which is why it was suddenly removed for download.   We also ran into the pesky bug, so I asked Adtran about it, and an ETA:

                                 

                                 

                                "AOS R10.3.1 (the next release) is scheduled for Monday, 9/24.  The issue with slips only occurs if you choose the "continuous monitoring" function in the GUI on a T1 interface.  Simply logging into the GUI and doing anything else does not result in slips."

                                • Re: Seeing Errored Seconds and Controlled Slip Seconds on TA-904
                                  david Employee

                                  Bill et al.,

                                   

                                  I wanted to provide an update on this issue.  During our testing, we were able to verify that if the web GUI was accessed to view or modify settings on a T-1 interface, the clock source setting would change to internal and potentially cause clock slips (CSS).   This issue should be resolved with the R10.3.2 release which is now posted on our main website, www.adtran.com.  This issue will also be fixed in R10.5.0 and R10.4.1, when it is released.

                                   

                                  Best regards,

                                  David

                                  • Re: Seeing Errored Seconds and Controlled Slip Seconds on TA-904
                                    david Employee

                                    Bill,


                                    I went ahead and flagged this post as "Assumed Answered".  If the response on this thread assisted you, please mark it as Correct or Helpful as the case may be with the applicable buttons.  This will make it visible and help other members of the community find solutions more easily.  If you still need assistance, I would be more than happy to continue working with you on this - just let me know in a reply.

                                     

                                    Thanks!

                                    David

                                    • Re: Seeing Errored Seconds and Controlled Slip Seconds on TA-904
                                      david Employee

                                      Sthenge,

                                       

                                      I went ahead and flagged the "Correct Answer" on this post to make it more visible and help other members of the community find solutions more easily. If you don't feel like the answer I marked was correct, feel free to come back to this post and unmark it and select another in its place with the applicable buttons.  If you still need assistance, we would be more than happy to continue working with you on this - just let us know in a reply.

                                       

                                      Thanks,

                                      David