20 Replies Latest reply on Nov 22, 2013 10:58 AM by jonathanblack

    TA 908e "DM/CSS/ES 15 min threshold exceeded"

    jonathanblack New Member

      I have a couple of 908e units configured with data in on ETH 0/1, and PRI connections on NET 0/3 and 0/4 to legacy IVR equipment.  We have configured SIP service with a couple of carriers over the data connection.  This is working. 

       

      However, we are getting a lot of these "threshold exceeded" errors.  Here's a sampling:

      2013.09.25 12:45:03 T1.t1 0/4 CSS 15 min threshold exceeded

      2013.09.25 12:46:13 T1.t1 0/4 ES 15 min threshold exceeded

      2013.09.25 12:46:25 T1.t1 0/3 CSS 15 min threshold exceeded

      2013.09.25 12:46:30 T1.t1 0/4 DM 15 min threshold exceeded

      2013.09.25 12:46:43 T1.t1 0/3 DM 15 min threshold exceeded

      2013.09.25 13:00:02 T1.t1 0/4 CSS 15 min threshold exceeded

      2013.09.25 13:01:12 T1.t1 0/4 ES 15 min threshold exceeded

      2013.09.25 13:01:41 T1.t1 0/3 CSS 15 min threshold exceeded

      2013.09.25 13:01:54 T1.t1 0/4 DM 15 min threshold exceeded

      2013.09.25 13:02:07 T1.t1 0/3 DM 15 min threshold exceeded

      2013.09.25 13:15:02 T1.t1 0/4 CSS 15 min threshold exceeded

      2013.09.25 13:16:09 T1.t1 0/3 CSS 15 min threshold exceeded

      2013.09.25 13:16:13 T1.t1 0/4 DM 15 min threshold exceeded, ES 15 min threshold exceeded

      2013.09.25 13:16:26 T1.t1 0/3 DM 15 min threshold exceeded

      2013.09.25 13:30:02 T1.t1 0/4 CSS 15 min threshold exceeded

      2013.09.25 13:31:12 T1.t1 0/4 ES 15 min threshold exceeded

      2013.09.25 13:31:28 T1.t1 0/3 CSS 15 min threshold exceeded

      2013.09.25 13:31:37 T1.t1 0/4 DM 15 min threshold exceeded

      2013.09.25 13:31:51 T1.t1 0/3 DM 15 min threshold exceeded

      2013.09.25 13:45:02 T1.t1 0/4 CSS 15 min threshold exceeded

      2013.09.25 13:46:03 T1.t1 0/3 CSS 15 min threshold exceeded

      2013.09.25 13:46:09 T1.t1 0/3 DM 15 min threshold exceeded

      2013.09.25 13:46:13 T1.t1 0/4 ES 15 min threshold exceeded

      2013.09.25 13:47:02 T1.t1 0/4 DM 15 min threshold exceeded

      2013.09.25 14:00:03 T1.t1 0/4 CSS 15 min threshold exceeded

      2013.09.25 14:01:13 T1.t1 0/4 ES 15 min threshold exceeded

      2013.09.25 14:01:21 T1.t1 0/4 DM 15 min threshold exceeded

      2013.09.25 14:01:32 T1.t1 0/3 CSS 15 min threshold exceeded

      2013.09.25 14:01:34 T1.t1 0/3 DM 15 min threshold exceeded

      2013.09.25 14:15:02 T1.t1 0/4 CSS 15 min threshold exceeded

      2013.09.25 14:16:10 T1.t1 0/3 CSS 15 min threshold exceeded

      2013.09.25 14:16:12 T1.t1 0/4 ES 15 min threshold exceeded

      2013.09.25 14:16:45 T1.t1 0/4 DM 15 min threshold exceeded

      2013.09.25 14:16:58 T1.t1 0/3 DM 15 min threshold exceeded

      2013.09.25 14:30:02 T1.t1 0/4 CSS 15 min threshold exceeded

      2013.09.25 14:31:13 T1.t1 0/4 ES 15 min threshold exceeded

      2013.09.25 14:31:17 T1.t1 0/3 DM 15 min threshold exceeded

      2013.09.25 14:31:40 T1.t1 0/3 CSS 15 min threshold exceeded

      2013.09.25 14:32:10 T1.t1 0/4 DM 15 min threshold exceeded

      2013.09.25 14:45:02 T1.t1 0/4 CSS 15 min threshold exceeded

      2013.09.25 14:46:13 T1.t1 0/4 ES 15 min threshold exceeded

      2013.09.25 14:46:18 T1.t1 0/3 CSS 15 min threshold exceeded

      2013.09.25 14:46:28 T1.t1 0/4 DM 15 min threshold exceeded

      2013.09.25 14:46:42 T1.t1 0/3 DM 15 min threshold exceeded

      2013.09.25 15:00:02 T1.t1 0/4 CSS 15 min threshold exceeded

      2013.09.25 15:00:56 T1.t1 0/3 CSS 15 min threshold exceeded

      2013.09.25 15:01:12 T1.t1 0/4 ES 15 min threshold exceeded

      2013.09.25 15:01:53 T1.t1 0/4 DM 15 min threshold exceeded

      2013.09.25 15:02:06 T1.t1 0/3 DM 15 min threshold exceeded

      2013.09.25 15:15:02 T1.t1 0/4 CSS 15 min threshold exceeded

      2013.09.25 15:16:11 T1.t1 0/4 DM 15 min threshold exceeded

      2013.09.25 15:16:13 T1.t1 0/4 ES 15 min threshold exceeded

      2013.09.25 15:16:25 T1.t1 0/3 CSS 15 min threshold exceeded, DM 15 min threshold exceeded

      2013.09.25 15:30:03 T1.t1 0/4 CSS 15 min threshold exceeded

      2013.09.25 15:31:05 T1.t1 0/3 CSS 15 min threshold exceeded

      2013.09.25 15:31:13 T1.t1 0/4 ES 15 min threshold exceeded

      2013.09.25 15:31:37 T1.t1 0/4 DM 15 min threshold exceeded

      2013.09.25 15:31:50 T1.t1 0/3 DM 15 min threshold exceeded

      2013.09.25 15:45:02 T1.t1 0/4 CSS 15 min threshold exceeded

      2013.09.25 15:46:09 T1.t1 0/3 DM 15 min threshold exceeded

      2013.09.25 15:46:12 T1.t1 0/4 ES 15 min threshold exceeded

      2013.09.25 15:46:35 T1.t1 0/3 CSS 15 min threshold exceeded

      2013.09.25 15:47:02 T1.t1 0/4 DM 15 min threshold exceeded

       

      From my research, it seems that these could be caused by a clock source problem.  Our clock source is set to internal, since there is no T1 from a carrier involved.  Is this a correct assumption?

       

      We are also experiencing lags in the audio between caller and recipient.  It can reach as much as a second or more, which results in the parties talking over each other.  I'm trying to determine if these two could be related.  I also have the carrier exploring this issue from their end.

        • Re: TA 908e "DM/CSS/ES 15 min threshold exceeded"
          jayh Hall_of_Fame

          You definitely have a clocking issue.  CSS is controlled slip seconds which is likely the cause of the errored and degraded alerts.

           

          The IVR is probably also set to internal, or to another T1 elsewhere.

           

          Each T1 should have exactly one source of clock.  If the legacy IVR equipment is connected to a carrier or another source of T1 clock, then you probably want to have the IVR clock from that carrier and the TA908e clock from the IVR.

           

          If the TA908e is the only TDM connection to the IVR, then you can really go either way, either set the IVR to clock from the TA908e or have the TA908e clock from the IVR, whichever is easier.

           

          You don't want them both internal nor do you want each clocking from the other.

           

          As far as the latency, fix the clock slips and see if it goes away.  There will be a small amount of latency inherent in an RTP-to-TDM conversion but rarely to that extent.  What do the MOS scores look like in VQM on the TA908e?

          1 of 1 people found this helpful
            • Re: TA 908e "DM/CSS/ES 15 min threshold exceeded"
              jonathanblack New Member

              Thanks, jayh.

              We do not have carrier T1s plugged in to the IVR equipment, which is why I was a bit confused about where to get clock source.  However, your explanation does make sense.  I am analyzing my entire setup to ensure that the clock sources are consistent throughout.

               

              I'm obviously fairly new to this, so I wasn't aware of the RTP monitoring built into AOS.  Thanks for the heads up on that.  It's not turned on, so I'll do some testing with it turned on.  Do you know if there is a performance penalty to turning on RTP monitoring?

                • Re: TA 908e "DM/CSS/ES 15 min threshold exceeded"
                  jayh Hall_of_Fame

                  jonathanblack wrote:

                   

                  We do not have carrier T1s plugged in to the IVR equipment, which is why I was a bit confused about where to get clock source.  However, your explanation does make sense.  I am analyzing my entire setup to ensure that the clock sources are consistent throughout.

                   

                  You're on the right track.  The rule is that T-1 circuits are point-to-point with two ends.  Exactly one of those ends must source clock for the span.

                   

                  I'm obviously fairly new to this, so I wasn't aware of the RTP monitoring built into AOS.  Thanks for the heads up on that.  It's not turned on, so I'll do some testing with it turned on.  Do you know if there is a performance penalty to turning on RTP monitoring?

                   

                  Nothing significant in our experience and we push the TA900 series pretty hard.  Probably more impact rendering the flash on the page to view it than collecting the data but it's very well-behaved.

                    • Re: TA 908e "DM/CSS/ES 15 min threshold exceeded"
                      jonathanblack New Member

                      I turned on RTP monitoring (and the firewall) over the weekend.  Unfortunately, on two occasions one of my carriers started getting 504 errors (server timeout) from our end.  This wasn't immediate, but after running for several hours, with a couple of additional hours between the incidents.  After the second time, I turned it back off, and since then no further 504 errors.  I'm not 100% sure the problem was this, but I'm now reluctant to turn it back on.

                       

                      The two Adtran 908e units are 1st generation.  Is it possible that RTP monitoring on the 1st gen units could cause an overload and this was fixed in 2nd gen?

                        • Re: TA 908e "DM/CSS/ES 15 min threshold exceeded"
                          jayh Hall_of_Fame

                          It could be a resource issue.  There was a software version that resulted in SIP resources not being released but that was fixed a while back.

                          The latest firmware for Gen. 1 is A4.11 .

                           

                          Were you able to fix the timing slips and errors?

                            • Re: TA 908e "DM/CSS/ES 15 min threshold exceeded"
                              jonathanblack New Member

                              We're on A4.11.00.E for both.

                               

                              Still working on changing the clock sources throughout.  Since these are live systems, I have to do it during maintenance windows, which aren't always the same for each system.  I actually have a total of three 908e's and 4 IVR servers all interconnected at some level.

                                • Re: TA 908e "DM/CSS/ES 15 min threshold exceeded"
                                  david Employee

                                  Jonathan,

                                   

                                  I just thought I would check in with you and see if you still needed assistance.  I would make clearing the CSS the first priority.  This can affect voice quality and, in some cases, call control.  Once that is resolved we can work on the 504 responses.  We may need a long term debug capture in order to understand the reason for those failures.  Below are the common debug commands we use to determine the point of failure.

                                   

                                  debug sip stack message

                                  debug sip cldu

                                  debug voice verbose

                                  debug isdn L2-formatted

                                   

                                  Our document Enabling Persistent Debug Logging can help you setup a debug capture that can run without closing down before the event occurs.

                                   

                                  Thanks!

                                  David

                                    • Re: TA 908e "DM/CSS/ES 15 min threshold exceeded"
                                      jonathanblack New Member

                                      David,

                                      Thanks for your response.  I'm still working on the clock source change.  We have a particular Dialogic board on which I'm having trouble setting the clock source.  (It's a D/480JCT-2T1, and I can't figure out how make it get clock source from one T1 versus the other.)  However, that's not an Adtran issue.

                                      Regards,

                                      Jonathan

                          • Re: TA 908e "DM/CSS/ES 15 min threshold exceeded"
                            jonathanblack New Member

                            Ok, I've changed the clock source to what I believe is correct (only one source per span).  This seems to have cleared the Degraded Minutes, but I'm still getting Controlled Slip Seconds and Errored Seconds (1411 of each in the last 24 hours on one PRI).

                          • Re: TA 908e "DM/CSS/ES 15 min threshold exceeded"
                            geo Employee

                            Hello Jonathanblack,

                             

                            I went ahead and flagged the "Correct Answer" on this post to make it more visible and help other members of the community find solutions more easily. If you don't feel like the answer I marked was correct, feel free to come back to this post and unmark it and follow up with additional questions. 

                             

                            Thanks,

                            Geoff