4 Replies Latest reply on Jul 7, 2014 1:15 PM by noor

    ETHERNET_INTERFACE.Switch Failed to process Link Notify Callback

    davshark New Member

      Received this error log and am not sure if this indicates a hardware failure or reset occurred.  Shortly after the entire network went down and restored by itself after two hours of down time.

       

      The switch had the correct time implemented and NTP configured.  Not sure how it would reset, Maybe battery fail on the device reset the time?  No configuration loss was found.  The system went down about 1:30PM and restored 3pm.

       

      The Gigabit links are connected to other switches.  there are 11 switches on the network and this is a core switch were half of them combine.

       

      I did a hard reset of the switch about 2pm (power plug)

       

      Any ideas and help would be useful.   I am interested in the failed to process Link Notify Call Back.  I am curious if a Spanning Tree Election was triggered, but I don't see that in the logs (or if it would be).  The syslog is configured log everything at the debug level.  And the firmware was updated on this switch about a months ago.

       

      This is an Adtran NetVanta 1238 PoE Switch

      Firmware Version18.01.06.00
      Part Number1700599G1
      MemoryTotal Heap: 47,496,176 Bytes
      Free Heap: 40,115,184 Bytes
      CPU UtilizationSystem Load: 35.32%
      1 Min Avg Load: 37.34%     5 Min Avg Load: 38.34%
      Min Load: 0%     Max Load: 100%
      Context Switch Load: 0.19%
      File SystemTotal: 6,697,320 Bytes
      Used: 4,686,696 Bytes
      Free: 2,010,624 Bytes

       

       

      00 duplex:1

      1970.01.01 00:00:05 ETHERNET_INTERFACE.Switch Failed to process Link Notify Callback: unit:0 port:52 status:1 enable:1 speed:1000 duplex:1

      1970.01.01 00:00:05 ETHERNET_INTERFACE.Switch Failed to process Link Notify Callback: unit:0 port:41 status:1 enable:1 speed:100 duplex:1

      1970.01.01 00:00:06 ETHERNET_INTERFACE.Switch Failed to process Link Notify Callback: unit:0 port:52 status:0 enable:1 speed:1000 duplex:1

      1970.01.01 00:00:06 ETHERNET_INTERFACE.swx 0/1 link up

      1970.01.01 00:00:06 ETHERNET_INTERFACE.swx 0/4 link up

      1970.01.01 00:00:06 ETHERNET_INTERFACE.swx 0/8 link up

      1970.01.01 00:00:06 ETHERNET_INTERFACE.swx 0/42 link down

      1970.01.01 00:00:06 ETHERNET_INTERFACE.swx 0/47 link down

      1970.01.01 00:00:06 ETHERNET_INTERFACE.giga-swx 0/1 link down

      1970.01.01 00:00:06 ETHERNET_INTERFACE.giga-swx 0/2 link down

      2014.05.07 01:01:25 OPERATING_SYSTEM.USER Created user 'admin'

      2014.05.07 01:01:25 ETHERNET_INTERFACE.swx 0/42 link up

      2014.05.07 01:01:26 ETHERNET_INTERFACE.giga-swx 0/1 link up

      2014.05.07 01:01:26 ETHERNET_INTERFACE.swx 0/47 link up

      2014.05.07 01:01:27 ETHERNET_INTERFACE.giga-swx 0/2 link up

      2014.05.07 01:01:27 ETHERNET_INTERFACE.giga-swx 0/4 link up

      2014.05.07 01:01:41 INTERFACE_STATUS.swx 0/1 changed state to up

      2014.05.07 01:01:41 INTERFACE_STATUS.swx 0/4 changed state to up

      2014.05.07 01:01:41 INTERFACE_STATUS.swx 0/8 changed state to up

      2014.05.07 01:01:41 INTERFACE_STATUS.swx 0/42 changed state to up

      2014.05.07 01:01:41 INTERFACE_STATUS.swx 0/47 changed state to up

      2014.05.07 01:01:41 INTERFACE_STATUS.giga-swx 0/1 changed state to up

      2014.05.07 01:01:41 INTERFACE_STATUS.giga-swx 0/2 changed state to up

      2014.05.07 01:01:41 INTERFACE_STATUS.giga-swx 0/4 changed state to up

      2014.05.07 01:01:41 INTERFACE_STATUS.vlan 1 changed state to up

      2014.05.07 01:01:41 OPERATING_SYSTEM Cold Start

      2014.05.07 01:01:47 POWER_OVER_ETHERNET.swx 0/10 changed state from SEARCHING to DELIVERING

      2014.05.07 01:01:47 POWER_OVER_ETHERNET.swx 0/10 classified to IEEE Class 0

      2014.05.07 01:01:48 POWER_OVER_ETHERNET.swx 0/16 changed state from SEARCHING to DELIVERING

      2014.05.07 01:01:48 POWER_OVER_ETHERNET.swx 0/16 classified to IEEE Class 0

      2014.05.07 01:01:48 POWER_OVER_ETHERNET.swx 0/17 changed state from SEARCHING to DELIVERING

      2014.05.07 01:01:48 POWER_OVER_ETHERNET.swx 0/17 classified to IEEE Class 0

      2014.05.07 01:01:48 POWER_OVER_ETHERNET.swx 0/19 changed state from SEARCHING to DELIVERING

      2014.05.07 01:01:48 POWER_OVER_ETHERNET.swx 0/19 classified to IEEE Class 0

      2014.05.07 01:01:49 POWER_OVER_ETHERNET.swx 0/30 changed state from SEARCHING to DELIVERING

      2014.05.07 01:01:49 POWER_OVER_ETHERNET.swx 0/30 classified to IEEE Class 0

      2014.05.07 01:01:49 ETHERNET_INTERFACE.swx 0/10 link up

      2014.05.07 01:01:49 ETHERNET_INTERFACE.swx 0/16 link up

      2014.05.07 01:01:49 ETHERNET_INTERFACE.swx 0/30 link up

      2014.05.07 01:01:49 ETHERNET_INTERFACE.swx 0/17 link up

      2014.05.07 01:01:49 INTERFACE_STATUS.swx 0/10 changed state to up

      2014.05.07 01:01:49 INTERFACE_STATUS.swx 0/16 changed state to up

      2014.05.07 01:01:49 INTERFACE_STATUS.swx 0/17 changed state to up

      2014.05.07 01:01:49 INTERFACE_STATUS.swx 0/30 changed state to up

      2014.05.07 01:01:49 ETHERNET_INTERFACE.swx 0/19 link up

      2014.05.07 01:01:50 INTERFACE_STATUS.swx 0/19 changed state to up

      2014.05.07 01:01:56 ETHERNET_INTERFACE.swx 0/10 link down

      2014.05.07 01:01:56 ETHERNET_INTERFACE.swx 0/16 link down

      2014.05.07 01:01:56 ETHERNET_INTERFACE.swx 0/30 link down

      2014.05.07 01:01:56 ETHERNET_INTERFACE.swx 0/17 link down

      2014.05.07 01:01:56 ETHERNET_INTERFACE.swx 0/19 link down

      2014.05.07 01:01:56 INTERFACE_STATUS.swx 0/10 changed state to down

      2014.05.07 01:01:56 INTERFACE_STATUS.swx 0/16 changed state to down

      2014.05.07 01:01:56 INTERFACE_STATUS.swx 0/17 changed state to down

      2014.05.07 01:01:56 INTERFACE_STATUS.swx 0/19 changed state to down

      2014.05.07 01:01:56 INTERFACE_STATUS.swx 0/30 changed state to down

      2014.05.07 01:01:58 ETHERNET_INTERFACE.swx 0/16 link up

      2014.05.07 01:01:58 ETHERNET_INTERFACE.swx 0/10 link up

      2014.05.07 01:01:58 ETHERNET_INTERFACE.swx 0/17 link up

      2014.05.07 01:01:58 ETHERNET_INTERFACE.swx 0/30 link up

      2014.05.07 01:01:58 ETHERNET_INTERFACE.swx 0/19 link up

      2014.05.07 01:01:58 INTERFACE_STATUS.swx 0/10 changed state to up

      2014.05.07 01:01:58 INTERFACE_STATUS.swx 0/16 changed state to up

      2014.05.07 01:01:58 INTERFACE_STATUS.swx 0/17 changed state to up

      2014.05.07 01:01:58 INTERFACE_STATUS.swx 0/19 changed state to up

      2014.05.07 01:01:58 INTERFACE_STATUS.swx 0/30 changed state to up

      2014.05.07 14:31:30 ETHERNET_INTERFACE.giga-swx 0/4 link down

      2014.05.07 14:31:31 INTERFACE_STATUS.giga-swx 0/4 changed state to down

      2014.05.07 14:55:08 ETHERNET_INTERFACE.giga-swx 0/4 link up

      2014.05.07 14:55:09 INTERFACE_STATUS.giga-swx 0/4 changed state to up

      2014.05.07 15:00:22 ETHERNET_INTERFACE.giga-swx 0/3 link up

      2014.05.07 15:00:22 INTERFACE_STATUS.giga-swx 0/3 changed state to up

      2014.05.07 15:15:57 ETHERNET_INTERFACE.swx 0/47 link down

      2014.05.07 15:15:58 INTERFACE_STATUS.swx 0/47 changed state to down

      2014.05.07 15:15:59 ETHERNET_INTERFACE.swx 0/47 link up

      2014.05.07 15:15:59 INTERFACE_STATUS.swx 0/47 changed state to up

      2014.05.07 15:46:38 ETHERNET_INTERFACE.swx 0/47 link down

      2014.05.07 15:46:39 INTERFACE_STATUS.swx 0/47 changed state to down

      2014.05.07 15:46:40 ETHERNET_INTERFACE.swx 0/47 link up

      2014.05.07 15:46:40 INTERFACE_STATUS.swx 0/47 changed state to up

      2014.05.07 15:50:35 ETHERNET_INTERFACE.swx 0/47 link down

      2014.05.07 15:50:35 INTERFACE_STATUS.swx 0/47 changed state to down

      2014.05.07 15:50:37 ETHERNET_INTERFACE.swx 0/47 link up

      2014.05.07 15:50:37 INTERFACE_STATUS.swx 0/47 changed state to up

      2014.05.07 15:51:42 ETHERNET_INTERFACE.swx 0/47 link down

      2014.05.07 15:51:43 INTERFACE_STATUS.swx 0/47 changed state to down

      2014.05.07 15:51:44 ETHERNET_INTERFACE.swx 0/47 link up

      2014.05.07 15:51:44 INTERFACE_STATUS.swx 0/47 changed state to up

        • Re: ETHERNET_INTERFACE.Switch Failed to process Link Notify Callback
          levi Employee

          davshark:

           

          Thank you for asking this question in the support community.  How long was the switch up?  The "link notify callback" event messages, appeared to have happened immediately after the switch was booting up.  Is this the only time you have had a similar network outage like this?  Do you know what brought the network back?  It appears that a hard reboot on the ADTRAN did not make any difference.  If you get a chance, to reply with a current copy of the configuration (please, remember to remove any information that may be sensitive to the organization), I will be happy to review it for you.

           

          Levi

            • Re: ETHERNET_INTERFACE.Switch Failed to process Link Notify Callback
              davshark New Member

              Thank you very much for the reply.  We basically have 11 switches a primary VLAN for all PCs and server and a VLAN for VOIP phones.  THe PBX, Router, and 4 other switches connect to this switch below.   I had a "sniffer" port enabled on Trunk Port 47.  Router is on 1, PBX (VLAN) is on G1.

               

              We have had the switches go crazy 4 times in about 4 months.  This is first time i have seen this error message on any of the switches.  The first two times logging was not enabled so it may have occurred. 

               

              The switch had been on for 4 weeks since it's last power cycle.  This network outage occurred 5 weeks ago, then occurred one week before that.  Strangly they seem to be similar parts of the day (12 to 2pm).

               

              I don't know what brought the network.  Previously we had restarted some or all of the switches and normally after 30 minutes it came back up.  But I couldn't tell if was the result of anything we did.  This most recent time we were changing wires and it took two hours to come back, it normally was 45min to 1 hour.  So maybe our attempts to repair and reset delays to system from coming back up.

               

              I though it was spanning tree flipping out and closing all traffic for a while, but i don't see any indication in the logs that it was even doing a spanning  tree election.  In some cases PCs were not able to ping the switch they were directly connected to.  And some times some people on a switch were fine and other completely dropped... (this was on multiple switches, I'm focusing on this one because it has the unique error message and it is a core switch.)

               

              !

              !

              ! ADTRAN, Inc. OS version 18.01.06.00

              ! Boot ROM version 17.03.02.SB

              ! Platform: NetVanta 1238 PoE, part number 1700599G1

              ! Serial number #################

              !

              !

              hostname "MDFSwitch01"

              enable password ##############

              !

              clock timezone -5-Eastern-Time

              ip default-gateway ##.##.1.1

              !

              !

              no auto-config

              !

              event-history on

              logging forwarding on

              logging forwarding priority-level debug

              logging forwarding receiver-ip ###.50.1.207

              no logging email

              !

              no service password-encryption

              !

              username "########" password "##########"

              !

              banner motd #

                               ****** Important ******

               

               

              Enable and Telnet passwords are configured to "#######".

              Please change them immediately.

               

               

              The vlan 1 interface is enabled with an address of ###.10.10.1

              Telnet access is also enabled.

               

               

               

               

               

               

              #

              !

              banner exec <

              MDF Switch 1 >

               

               

              <

              !

              !

              !

              !

              !

              !

              no dot11ap access-point-control

              !

              !

              !

              !

              !

              !

              !

              !

              vlan 1

                name "Default"

              !

              vlan 200

                name "AllworxPhones"

              !

              interface switchport 0/1

                description Router / PBX

                spanning-tree edgeport

                no shutdown

                switchport mode trunk

                switchport trunk allowed vlan 1,200

              !

              interface switchport 0/2

                spanning-tree edgeport

                no shutdown

                switchport voice vlan 200

                switchport voice-signaling vlan 200

              !

              interface switchport 0/3

                spanning-tree edgeport

                no shutdown

                switchport voice vlan 200

                switchport voice-signaling vlan 200

              !

              interface switchport 0/4

                spanning-tree edgeport

                no shutdown

                switchport voice vlan 200

                switchport voice-signaling vlan 200

              !

              interface switchport 0/5

                spanning-tree edgeport

                no shutdown

                switchport voice vlan 200

                switchport voice-signaling vlan 200

              !

              interface switchport 0/6

                spanning-tree edgeport

                no shutdown

                switchport voice vlan 200

                switchport voice-signaling vlan 200

              !

              interface switchport 0/7

                spanning-tree edgeport

                no shutdown

                switchport voice vlan 200

                switchport voice-signaling vlan 200

              !

              interface switchport 0/8

                description ASA5505 Firewall

                speed 100

                spanning-tree edgeport

                no shutdown

              !

              interface switchport 0/9

                spanning-tree edgeport

                no shutdown

                switchport voice vlan 200

                switchport voice-signaling vlan 200

              !

              interface switchport 0/10

                spanning-tree edgeport

                no shutdown

                switchport voice vlan 200

                switchport voice-signaling vlan 200

              !

              interface switchport 0/11

                spanning-tree edgeport

                no shutdown

                switchport voice vlan 200

                switchport voice-signaling vlan 200

              !

              interface switchport 0/12

                spanning-tree edgeport

                no shutdown

                switchport voice vlan 200

                switchport voice-signaling vlan 200

              !

              interface switchport 0/13

                spanning-tree edgeport

                no shutdown

                switchport voice vlan 200

                switchport voice-signaling vlan 200

              !

              interface switchport 0/14

                spanning-tree edgeport

                no shutdown

                switchport voice vlan 200

                switchport voice-signaling vlan 200

              !

              interface switchport 0/15

                spanning-tree edgeport

                no shutdown

                switchport voice vlan 200

                switchport voice-signaling vlan 200

              !

              interface switchport 0/16

                spanning-tree edgeport

                no shutdown

                switchport voice vlan 200

                switchport voice-signaling vlan 200

              !

              interface switchport 0/17

                spanning-tree edgeport

                no shutdown

                switchport voice vlan 200

                switchport voice-signaling vlan 200

              !

              interface switchport 0/18

                spanning-tree edgeport

                no shutdown

                switchport voice vlan 200

                switchport voice-signaling vlan 200

              !

              interface switchport 0/19

                spanning-tree edgeport

                no shutdown

                switchport voice vlan 200

                switchport voice-signaling vlan 200

              !

              interface switchport 0/20

                spanning-tree edgeport

                no shutdown

                switchport voice vlan 200

                switchport voice-signaling vlan 200

              !

              interface switchport 0/21

                spanning-tree edgeport

                no shutdown

                switchport voice vlan 200

                switchport voice-signaling vlan 200

              !

              interface switchport 0/22

                spanning-tree edgeport

                no shutdown

                switchport voice vlan 200

                switchport voice-signaling vlan 200

              !

              interface switchport 0/23

                spanning-tree edgeport

                no shutdown

                switchport voice vlan 200

                switchport voice-signaling vlan 200

              !

              interface switchport 0/24

                spanning-tree edgeport

                no shutdown

                switchport voice vlan 200

                switchport voice-signaling vlan 200

              !

              interface switchport 0/25

                spanning-tree edgeport

                no shutdown

                switchport voice vlan 200

                switchport voice-signaling vlan 200

              !

              interface switchport 0/26

                spanning-tree edgeport

                no shutdown

                switchport voice vlan 200

                switchport voice-signaling vlan 200

              !

              interface switchport 0/27

                spanning-tree edgeport

                no shutdown

                switchport voice vlan 200

                switchport voice-signaling vlan 200

              !

              interface switchport 0/28

                spanning-tree edgeport

                no shutdown

                switchport voice vlan 200

                switchport voice-signaling vlan 200

              !

              interface switchport 0/29

                spanning-tree edgeport

                no shutdown

                switchport voice vlan 200

                switchport voice-signaling vlan 200

              !

              interface switchport 0/30

                spanning-tree edgeport

                no shutdown

                switchport voice vlan 200

                switchport voice-signaling vlan 200

              !

              interface switchport 0/31

                spanning-tree edgeport

                no shutdown

                switchport voice vlan 200

                switchport voice-signaling vlan 200

              !

              interface switchport 0/32

                spanning-tree edgeport

                no shutdown

                switchport voice vlan 200

                switchport voice-signaling vlan 200

              !

              interface switchport 0/33

                spanning-tree edgeport

                no shutdown

                switchport voice vlan 200

                switchport voice-signaling vlan 200

              !

              interface switchport 0/34

                spanning-tree edgeport

                no shutdown

                switchport voice vlan 200

                switchport voice-signaling vlan 200

              !

              interface switchport 0/35

                spanning-tree edgeport

                no shutdown

                switchport voice vlan 200

                switchport voice-signaling vlan 200

              !

              interface switchport 0/36

                spanning-tree edgeport

                no shutdown

                switchport voice vlan 200

                switchport voice-signaling vlan 200

              !

              interface switchport 0/37

                spanning-tree edgeport

                no shutdown

                switchport voice vlan 200

                switchport voice-signaling vlan 200

              !

              interface switchport 0/38

                spanning-tree edgeport

                no shutdown

                switchport voice vlan 200

                switchport voice-signaling vlan 200

              !

              interface switchport 0/39

                spanning-tree edgeport

                no shutdown

                switchport voice vlan 200

                switchport voice-signaling vlan 200

              !

              interface switchport 0/40

                spanning-tree edgeport

                no shutdown

                switchport voice vlan 200

                switchport voice-signaling vlan 200

              !

              interface switchport 0/41

                spanning-tree edgeport

                no shutdown

                switchport voice vlan 200

                switchport voice-signaling vlan 200

              !

              interface switchport 0/42

                spanning-tree edgeport

                no shutdown

                switchport voice vlan 200

                switchport voice-signaling vlan 200

              !

              interface switchport 0/43

                spanning-tree edgeport

                no shutdown

                switchport voice vlan 200

                switchport voice-signaling vlan 200

              !

              interface switchport 0/44

                spanning-tree edgeport

                no shutdown

                switchport voice vlan 200

                switchport voice-signaling vlan 200

              !

              interface switchport 0/45

                spanning-tree edgeport

                no shutdown

                switchport voice vlan 200

                switchport voice-signaling vlan 200

              !

              interface switchport 0/46

                spanning-tree edgeport

                no shutdown

                switchport voice vlan 200

                switchport voice-signaling vlan 200

              !

              interface switchport 0/47

                spanning-tree edgeport

                no shutdown

                switchport voice vlan 200

                switchport voice-signaling vlan 200

              !

              interface switchport 0/48

                description Allworx Phone System

                spanning-tree edgeport

                no shutdown

              !

              !

              interface gigabit-switchport 0/1

                description Allworx Phone System

                spanning-tree edgeport

                no shutdown

                switchport access vlan 200

              !

              interface gigabit-switchport 0/2

                spanning-tree edgeport

                no shutdown

                switchport mode trunk

                switchport trunk allowed vlan 1,200

              !

              interface gigabit-switchport 0/3

                spanning-tree edgeport

                no shutdown

                switchport mode trunk

                switchport trunk allowed vlan 1,200

              !

              interface gigabit-switchport 0/4

                spanning-tree edgeport

                no shutdown

                switchport mode trunk

                switchport trunk allowed vlan 1,200

              !

              !

              !

              interface vlan 1

                ip address  ###.###.1.253  255.255.0.0

                no shutdown

              !

              interface vlan 200

                no ip address

                shutdown

              !

              !

              !

              no tftp server

              no tftp server overwrite

              ip http server

              ip http secure-server

              no ip snmp agent

              no ip ftp server

              ip ftp server default-filesystem flash

              no ip scp server

              no ip sntp server

              !

              !

              !

              !

              !

              !

              !

              !

              !

              line con 0

                no login

              !

              line telnet 0 4

                login

                password ########

                no shutdown

              line ssh 0 4

                login local-userlist

                no shutdown

              !

              sntp server #############

              !

              !

              !

              end