8 Replies Latest reply on Apr 3, 2018 7:02 AM by mforrence

    Lose network access to admin when connected to customer network

    mforrence New Member

      Hi all,

      We're scratching our head.  Customer added a new building where we've installed four Netvanta 1550P switches.  We pre-configured and deployed, and all are working like a champ except for admin via LAN (http, ssh, etc).

      When these four switches are disconnected from the rest of the customer's network, all admin works exactly as expected.  However, when we connect to the rest of customer's network, LAN admin simply stops working.  Note that all OTHER network devices in this new building remain accessible via http, as do all other non-Adtran devices in their "old" buildings.   Connection to "old" building is via fiber with yet another Adtran 1550P at far end.  We cannot access that switch either when rest of network is connected. Firmware on all switches is 12.3.4.  Serial admin functions fine at all times.  We have addresses assigned to both VLANs, no joy accessing either with rest of network connected. Customer has a third-party company managing their core network, we haven't gone too far down that road as all other devices appear to be normally accessible - only the Netvanta switches are negatively affected.  We have a fairly straight-forward config - all switches use essentially the same config, with different IP addresses:

       

       

      hostname "BldgC Data cabinet B"

      enable password XXXX

      !

      clock timezone -5-Eastern-Time

      !

      ip subnet-zero

      ip classless

      ip default-gateway 172.16.0.254

      ip routing

      name-server 75.75.75.75 8.8.8.8

      !

      ip route-cache express

      !

      no auto-config

      !

      event-history on

      no logging forwarding

      no logging email

      !

      no service password-encryption

      !

      username "XXXX" password "YYYY"

      !

      !

      no dot11ap access-point-control

       

      no dos-protection

       

      no desktop-auditing dhcp

       

      no network-forensics ip dhcp

      !

      vlan 1

        name "Default"

      !

      vlan 18

        name "Public Network"

      !

      interface gigabit-switchport 0/1

        no shutdown

        switchport access vlan 18

      !

      interface gigabit-switchport 0/2

        no shutdown

        switchport access vlan 18

      !

      interface gigabit-switchport 0/3

        no shutdown

      !

      interface gigabit-switchport 0/4

        no shutdown

      !

      interface gigabit-switchport 0/5

        no shutdown

      !

      interface gigabit-switchport 0/6

        no shutdown

      !

      interface gigabit-switchport 0/7

        no shutdown

      !

      interface gigabit-switchport 0/8

        no shutdown

      !

      interface gigabit-switchport 0/9

        no shutdown

      !

      interface gigabit-switchport 0/10

        no shutdown

      !

      interface gigabit-switchport 0/11

        no shutdown

      !

      interface gigabit-switchport 0/12

        no shutdown

      !

      interface gigabit-switchport 0/13

        no shutdown

      !

      interface gigabit-switchport 0/14

        no shutdown

      !

      interface gigabit-switchport 0/15

        no shutdown

      !

      interface gigabit-switchport 0/16

        no shutdown

      !

      interface gigabit-switchport 0/17

        no shutdown

      !

      interface gigabit-switchport 0/18

        no shutdown

      !

      interface gigabit-switchport 0/19

        no shutdown

      !

      interface gigabit-switchport 0/20

        no shutdown

      !

      interface gigabit-switchport 0/21

        no shutdown

      !

      interface gigabit-switchport 0/22

        no shutdown

      !

      interface gigabit-switchport 0/23

        no shutdown

        switchport mode trunk

      !

      interface gigabit-switchport 0/24

        no shutdown

        switchport mode trunk

      !

      interface xgigabit-switchport 0/1

        no shutdown

        switchport mode trunk

        speed auto

      !

      interface xgigabit-switchport 0/2

        no shutdown

        switchport mode trunk

        speed auto

      !

      interface xgigabit-switchport 0/3

        no shutdown

        switchport mode trunk

        speed auto

      !

      interface xgigabit-switchport 0/4

        no shutdown

        switchport mode trunk

        speed auto

      !

      interface vlan 1

        ip address  172.16.0.142  255.255.0.0

        no awcp

        no shutdown

      !

      interface vlan 18

        ip address  172.18.0.62  255.255.0.0

        no awcp

        no shutdown

      !

      no tftp server

      no tftp server overwrite

      http server

      http secure-server

      no snmp agent

      no ip ftp server

      no ip scp server

      no ip sntp server

      !

      snmp-server engine local 8000029803000000000000

      !

      line con 0

        no login

      !

      line telnet 0 4

        login

        password XXXX

        no shutdown

      line ssh 0 4

        login local-userlist

        no shutdown

      !

      sntp server us.pool.ntp.org

      !

      end

       

      I'd be thrilled if anyone can point out our boneheaded mistake.  Thanks

        • Re: Lose network access to admin when connected to customer network
          jroad Employee

          This sounds like an issue where the CPU is handling network traffic and not responding to management interfaces ( as it is designed to do network traffic first priority).

          switch# sh proc cpu will show what processes that are running.  And switch# sh proc cpu hist will show the CPU load over time.

           

          This will sometimes give you an idea of the type of network traffic.   It could be as simple as a very high level of Multicast or Broadcast traffic.

           

          Also the VLAN interfaces do not have -

          ip route-cache express

           

          This should not matter if they are not routing any traffic though.

           

          To troubleshoot further, please provide this info to support.  

          • Re: Lose network access to admin when connected to customer network
            jayh Hall_of_Fame

            Is it your intent that this switch act as a layer 3 router between VLAN 1 and VLAN 18? If not, disable IP routing or remove the IP address from the VLAN interface you don't want to route.

             

            From what IP address or VLAN are you trying to access the devices?

              • Re: Lose network access to admin when connected to customer network
                mforrence New Member

                I've disabled IP routing and greatly dumbed down the config for diagnostics.  Current version below.

                 

                In testing we have:

                • Added another Adtran 1531P with very basic config - same phenomenon occurs:  When stand-alone all admin works normally.  Once connected to customer's network, IP admin stops working. We tried to mimic their network in our shop - all works perfectly normally.
                • Via serial connection, realized that we cannot ping other devices from the 1550.  We CAN ping our own address.  The 1550 does not reply to a ping.
                • Tried changing IP address to something way out of the normal ranges (this is a /16 network - lots of open territory), no joy/difference
                • All connected devices continue to work normally
                • Other web-enabled devices are fully and normally accessible.  It is ONLY the Netvanta switches that stop responding to IP services.

                 

                At this point, I'm fairly certain that it is something about the customer's network - but I"m at a loss to what may be wrong.  I'd welcome suggestions on a valid test.  We can insert a network monitor somewhere and capture packets for wireshark analysis.

                 

                 

                Current config:

                !

                !

                ! ADTRAN, Inc. OS version R12.3.4

                ! Boot ROM version BVS1.0

                ! Platform: NetVanta 1550-24P, part number 17101524PF1

                ! Serial number XXXX

                !

                !

                hostname "Switch"

                enable password YYYY

                !

                !

                !

                ip subnet-zero

                ip classless

                no ip routing

                !

                !

                no ip route-cache express

                !

                no auto-config

                !

                event-history on

                no logging forwarding

                no logging email

                !

                no service password-encryption

                !

                username "xxxx" password "yyyy"

                !

                !

                !

                !

                !

                !

                no dot11ap access-point-control

                 

                 

                no dos-protection

                 

                 

                no desktop-auditing dhcp

                 

                 

                no network-forensics ip dhcp

                 

                !

                vlan 1

                  name "Default"

                !

                interface gigabit-switchport 0/1

                  no shutdown

                !

                interface gigabit-switchport 0/2

                  no shutdown

                !

                interface gigabit-switchport 0/3

                  no shutdown

                !

                interface gigabit-switchport 0/4

                  no shutdown

                !

                interface gigabit-switchport 0/5

                  no shutdown

                !

                interface gigabit-switchport 0/6

                  no shutdown

                !

                interface gigabit-switchport 0/7

                  no shutdown

                !

                interface gigabit-switchport 0/8

                  no shutdown

                !

                interface gigabit-switchport 0/9

                  no shutdown

                !

                interface gigabit-switchport 0/10

                  no shutdown

                !

                interface gigabit-switchport 0/11

                  no shutdown

                !

                interface gigabit-switchport 0/12

                  no shutdown

                !

                interface gigabit-switchport 0/13

                  no shutdown

                !

                interface gigabit-switchport 0/14

                  no shutdown

                !

                interface gigabit-switchport 0/15

                  no shutdown

                !

                interface gigabit-switchport 0/16

                  no shutdown

                !

                interface gigabit-switchport 0/17

                  no shutdown

                !

                interface gigabit-switchport 0/18

                  no shutdown

                !

                interface gigabit-switchport 0/19

                  no shutdown

                !

                interface gigabit-switchport 0/20

                  no shutdown

                !

                interface gigabit-switchport 0/21

                  no shutdown

                !

                interface gigabit-switchport 0/22

                  no shutdown

                !

                interface gigabit-switchport 0/23

                  no shutdown

                !

                interface gigabit-switchport 0/24

                  no shutdown

                !

                !

                interface xgigabit-switchport 0/1

                  no shutdown

                  switchport mode access

                  speed auto

                !

                interface xgigabit-switchport 0/2

                  no shutdown

                  switchport mode access

                  speed auto

                !

                interface xgigabit-switchport 0/3

                  no shutdown

                  switchport mode access

                  speed auto

                !

                interface xgigabit-switchport 0/4

                  no shutdown

                  switchport mode access

                  speed auto

                !

                !

                !

                interface vlan 1

                  ip address  172.16.0.143  255.255.0.0

                  no awcp

                  no shutdown

                !

                !

                !

                !

                !

                no tftp server

                no tftp server overwrite

                http server

                http secure-server

                no snmp agent

                no ip ftp server

                no ip scp server

                no ip sntp server

                !

                !

                !

                !

                !

                snmp-server engine local 8000029803000000000000

                !

                !

                !

                !

                line con 0

                  no login

                !

                line telnet 0 4

                  login

                  password XXXX

                  no shutdown

                line ssh 0 4

                  login local-userlist

                  no shutdown

                !

                !

                !

                !

                end

                  • Re: Lose network access to admin when connected to customer network
                    jayh Hall_of_Fame

                    Is this a flat /16 network? You don't have a default route in your latest configuration. Connected management host on same subnet? Connected management host getting the proper /16 mask from customer's DHCP server? Firewall or similar doing proxy-arp perhaps? When you try to ping the unresponsive Adtran device, does the correct MAC show up in the management host's ARP table?

                      • Re: Lose network access to admin when connected to customer network
                        mforrence New Member

                        Indeed, a flat /16 network.  Latest config was one of many, in hopes of finding a silver bullet - tried without default route, as goal is truly a layer two operation with management.  Host PC is on same subnet, and physically connected to the switch that we're doing most diagnostics on.  PC IP address is assigned via DHCP, and getting a /16 address in same subnet.  We had tried assigning static address to PC for testing, no difference.  When I try to ping unit, correct MAC shows in PC's arp table.

                  • Re: Lose network access to admin when connected to customer network
                    mforrence New Member

                    Just to close this out, the culprit was overwhelming multicast and broadcast traffic.  The good folks at support provided fabulous assistance in identifying the issues as well as some possible resolutions.  We still need to deal with the culprits, but at least we can now appropriately manage the switches as we work thru the process.