9 Replies Latest reply on Mar 27, 2014 8:07 AM by juad

    Netvanta 4430 WAN Encapsulation Link Status Down

    juad New Member


      Hi all,

       

      Our bandwidth monitoring tool shows we are only using up to 4.5mb at a certain site while the total bandwidth should be 6.0mb; initially I figured that perhaps 1.5mb was dedicated for VOIP.  While checking the Netvanta 4430 router at the location I noticed that T1 3/1 Link is Down under WAN Encapsulation Interfaces.  I don't know what that means as there are no alarms at the SmartJack and our monitoring tool shows the link to be up.

      2014-03-05_0904.png

       

      So I did a little bit more digging and compared configurations and statistics for each of the T1 interfaces.  While configurations for all 4 interfaces are the same, statistics are different for T1 3/1 when compared to the other 3.  The highlighted area shows a big difference in the Input/output rate of T1 3/1.  Is someone able to explain the difference and advise if this might be a problem?  I rebooted the router yesterday to see if that would "fix" it but nothing changed.

       

      Appreciate the help.

       

      T1_3_1_2014-03-05_0915.png

       

      T1_3_2_2014-03-05_0917.png

       

      T1_3_3_2014-03-05_0919.png

       

      T1_3_42014-03-05_0920.png

        • Re: Netvanta 4430 WAN Encapsulation Link Status Down
          levi Employee

          juad:

           

          Thank you for asking this question in the Support Community.  From the screenshots you supplied, it appears the Frame Relay interface is down for T1 3/1.  When you get a change will you reply with a copy of the current configuration? (Please, remember to remove any information that may be sensitive to the organization).

           

          Also, would you reply with the output from the following commands:

           

          show interfaces

          sho frame-relay lmi

          show frame-relay pvc

           

          Please, do not hesitate to reply to this email with any additional questions or information.  I will be happy to help in any way I can.

           

          Levi

          1 of 1 people found this helpful
          • Re: Netvanta 4430 WAN Encapsulation Link Status Down
            jayh Hall_of_Fame

            It looks like you're up on the physical layer but the other end my not have your T1 3/1 in the bundle or be misconfigured.  If you have access to the other end of the circuit, take a look there.  If not, ask the carrier to have a look. 

             

            Some other CLI tools you can use are:

             

            show frame-relay multilink

            (You'll probably see that T1 3/1 is up with link-state down)

             

            debug frame-relay multilink t1 3/1

             

               and compare with

             

            debug frame-relay multilink t1 3/2


            You should see hellos sent outbound (O) every ten seconds followed by a hello_ack inbound (I) in both directions every ten seconds.

             

            My suspicion is that the other end hasn't put the first T1 into the multilink bundle.  Did you originally have a single frame-relay T1 on 3/1 and then grow it to a 4-loop bundle? 

            1 of 1 people found this helpful
              • Re: Netvanta 4430 WAN Encapsulation Link Status Down
                juad New Member

                I got the following from running the commands you suggested:

                 

                show frame-relay multilink

                 

                Bundle: frame-relay 1 is UP; class A bundle

                 

                        Near-end BID:
                MFR1; Far-end BID: Multilink0/4/0/0/6

                 

                  Bundle links:

                 

                    Link thru t1 3/1
                is UP, link state is ADD_RX

                 

                    Link thru t1 3/2
                is UP, link state is UP

                 

                    Link thru t1 3/3
                is UP, link state is UP

                 

                    Link thru t1 3/4
                is UP, link state is UP

                 

                 

                t1 3/1 Debug

                 

                1. 2014.03.06 15:47:08 FRAME_RELAY.MULTILINK_INT t1 3/1 (I):
                  msg=ADD_LINK, Link=t1 3/1 1, Bundle=MFR1, BL state=ADD_RX
                2. 2014.03.06 15:47:08 FRAME_RELAY.MULTILINK_INT t1 3/1 (O):
                  msg=ADD_LINK_ACK, Link=t1 3/1 1, Bundle=MFR1, BL state=ADD_RX
                3. 2014.03.06 15:47:15 FRAME_RELAY.MULTILINK_INT t1 3/1 (O):
                  msg=ADD_LINK, Link=t1 3/1 1, Bundle=MFR1, BL state=ADD_RX
                4. 2014.03.06 15:47:22 FRAME_RELAY.MULTILINK_INT t1 3/1 (I):
                  msg=ADD_LINK, Link=t1 3/1 1, Bundle=MFR1, BL state=ADD_RX      

                 

                t1 3/2 Debug

                 

                1. 2014.03.06 15:56:07 FRAME_RELAY.MULTILINK_INT t1 3/2 (I):
                  msg=HELLO, Link=t1 3/2 1, Bundle=MFR1, BL state=UP
                2. 2014.03.06 15:56:07 FRAME_RELAY.MULTILINK_INT t1 3/2 (O):
                  msg=HELLO_ACK, Link=t1 3/2 1, Bundle=MFR1, BL state=UP
                3. 2014.03.06 15:56:10 FRAME_RELAY.MULTILINK_INT t1 3/2 (O):
                  msg=HELLO, Link=t1 3/2 1, Bundle=MFR1, BL state=UP
                4. 2014.03.06 15:56:10 FRAME_RELAY.MULTILINK_INT t1 3/2 (I):
                  msg=HELLO_ACK, Link=t1 3/2 1, Bundle=MFR1, BL state=UP
                5. 2014.03.06 15:56:16 FRAME_RELAY.MULTILINK_INT t1 3/2 (I):
                  msg=HELLO, Link=t1 3/2 1, Bundle=MFR1, BL state=UP

                 

                 

                 

                No Hellos for interface T1 3/1; I'm wondering if it is a problem on our device or the ISP.

                  • Re: Netvanta 4430 WAN Encapsulation Link Status Down
                    jayh Hall_of_Fame

                    juad wrote:

                     

                    No Hellos for interface T1 3/1; I'm wondering if it is a problem on our device or the ISP.

                     

                    Most likely the ISP.  Could be the transport carrier but I would expect to see errors if this were the case.

                     

                    To test, during a period of low traffic swap the cables between T1 3/1 and T1 3/2 and see if the problem follows the circuit or stays on T1 3/1. Note that can take up to a minute for frame-relay circuits to handshake and train.

                     

                    Another thought, have you or the ISP changed router hostnames or multilink identifier names since the equipment was first installed? 

                     

                    We had a really strange similar scenario a while back with multilink PPP.  Cisco uses the router hostname as an "endpoint identifier" to ensure that all of the links in a bundle go to the same place.  We had changed the hostname on a customer router which caused no immediate problems.  Some weeks later one of the T1s in the bundle went down.  When it came back up it refused to join the multilink bundle because the "endpoint identifier" was different.  Very subtle problem which we struggled with for a while until debugs clued us in.  We wound up having to shut/no-shut all of the links and let the bundle sort out that the other end now had a different identifier.

                     

                    You may need to do something similar.  If your swap test results in BOTH 3/1 and 3/2 now being unhappy, then you'll need to tear down all of the links and let the bundle negotiation start over.  Obviously best done after hours.

                      • Re: Netvanta 4430 WAN Encapsulation Link Status Down
                        juad New Member

                        I contacted the ISP and provided them the same information.  They said the circuit tested good to the Smartjack and therefore was a problem on our end since they could not loop to the CSU.  I wonder if a shut/ no shut on interface t1 3/1 might resolve the issue...or would I have to do that for the other 3?  Would doing a shut/ no shut on just that interface cause any problems for the rest?

                          • Re: Netvanta 4430 WAN Encapsulation Link Status Down
                            jayh Hall_of_Fame

                            juad wrote:

                             

                            I contacted the ISP and provided them the same information.  They said the circuit tested good to the Smartjack and therefore was a problem on our end since they could not loop to the CSU.  I wonder if a shut/ no shut on interface t1 3/1 might resolve the issue...or would I have to do that for the other 3?  Would doing a shut/ no shut on just that interface cause any problems for the rest?

                             

                            The physical link looks good, so I would expect their test to the smartjack would be fine.  Their loopback testing would have had to take that T1 out of service so it has already been "bounced" and I suspect that a shut/no-shut isn't likely to help.

                             

                            You can indeed do a shut/no-shut on that link without affecting traffic.  Before doing so verify that this T1 isn't the only external timing source.  When you shut it down you'll want to take clock from  a different T1.  "Show system" should list a primary and secondary clock source.  Both should be configured to different T1s in the bundle so that when you shut down 3/1 there is another T1 clock source available.

                             

                            A shut/no-shut may fix it simply by forcing both sides to renegotiate, but likely not.

                             

                            If this does not fix it, the next step as I suggested earlier would be to temporarily swap the affected T1 with one of the others.  Do this during a period of low traffic.  If the problem remains on T1 3/1 then the problem would seem to be within the Adtran (but it isn't obvious where).  If the problem moves to the other physical interface on the Adtran then contact the carrier again.  Let them know that it isn't a physical issue but something with the frame-relay configuration on their end.  Note that it may take up to 60 seconds for frame-relay LMI to settle down when testing.

                             

                            Swap the links back after testing so that your documentation is correct with regard to circuit IDs for future repair tickets.

                    • Re: Netvanta 4430 WAN Encapsulation Link Status Down
                      juad New Member

                      Just a quick follow-up.  Our ISP found the issue on their network and was able to correct the problem.  Thanks for all the help.