15 Replies Latest reply on Oct 17, 2014 8:50 AM by kylem

    Problem with SNMP Polling

    ccmh New Member

      Two of our NetVanta 3448 Routers have suddenly started going to nearly 100% CPU Utilization when we enable SNMP polling to communicate with a network monitoring software we use.  It works on the other routers, one of which is also a NetVanta 3448.  Settings all appear to be identical.  The setup worked fine for 2 years............. 

       

      Any thoughts would be appreciated................

       

      Thanks!

        • Re: Problem with SNMP Polling
          levi Employee

          ccmh:

           

          Thank you for asking this question in the support community.  Without seeing the configuration or the output from the show process cpu command, it would appear that you may be receiving an SNMP denial of service attack.  Here are a few recommendations:

           

          • Disable SNMP if you are not using it (no snmp agent or no ip snmp agent (depending on firmware version))
          • Configure an SNMP access-group and apply it to the public facing Internet connection to block SNMP traffic from reaching the unit's processor:

                    ip access-list extended BLOCK-SNMP

                      deny udp any  any eq snmp

                      permit ip any  any

                    !

                    interface eth 0/2

                     ip access-group BLOCK-SNMP in

          • Upgrade the firmware to R10.7.0 and allow the ADTRAN's firewall to block the attack

           

          I hope that makes sense, but please do not hesitate to reply to this post with any additional questions or information.  I will be happy to help in any way I can.

           

          Levi

          1 of 1 people found this helpful
          • Re: Problem with SNMP Polling
            levi Employee

            ccmh:

             

            Do you have any further questions on this post? 

             

            Levi

              • Re: Problem with SNMP Polling
                ccmh New Member

                We updated to R10.7.2 but this was not helpful. We had to shut down Solarwinds Orion Network Performance Monitor to resolve the problem. Solarwinds claims this is a bug in Adtrans's AOS but we cannot confirm why this worked for 2 years without this issue.

                  • Re: Problem with SNMP Polling
                    levi Employee

                    ccmh:

                     

                    Did you try using the ACL and access-group to see if the problem was caused by a SNMP DoS attack?

                     

                    Levi

                      • Re: Problem with SNMP Polling
                        ccmh New Member

                        We know the Solarwinds software is causing the problem as when we shut down the software (Orion Network Performance Monitor) the problem with he Netvanta 3448 CPU usage stops. There is no external attack. Unfortunatly the purpose of the offending software is to monitor the network appliances for the same problems it seems to be causing...
                        Currently we have hidden the Netvantas in our network from the software. I am not very happy with Solarwinds at this time as they are pointing at Adtran's AOS as the problem claiming there is a "known bug".

                  • Re: Problem with SNMP Polling
                    levi Employee

                    ccmh:

                     

                    I marked this post as "assumed answered," but please do not hesitate to reply if you have further questions.

                    Levi

                    • Re: Problem with SNMP Polling
                      chazh New Member

                      I have a customer with nearly the exact same problem. They are running Solarwinds 10.6.1, and also collecting cache flow data. Exactly every four hours they say the "new" routers, 3430's running r10.5.1 code nearly stop routing. We can not log into them, and pings are not returned.

                       

                      They turned off SNMP monitoring and cache flow collection, and there are no problems anymore. However, they want to get SNMP and cach flow data. SolarWinds to my CSR that it is a problem with the Adtran version of OS. They did not offer a solution. We have opened a ticket with Adtran, and called several times, and sent about half dozen emails, and we have not received information back.

                       

                      Has anyone found a solution to this yet? Is there an AOS that specifically addresses this issue?

                      • Re: Problem with SNMP Polling
                        billingslyd New Member

                        I am currently experiencing the same problem. Our network consists of 700+ Adtran 1224 and 1335 (POE and non POE) We recently installed SolarWinds as our Network Monitoring application.  After reviewing this thread we have disabled Adtran polling via Solarwinds. We had approx. 18 switches lock up over a period of 4 days before disabling in Solarwinds We thought this had corrected the issue then  have had 4-8 more lock up over the past 3 days. Wondering if the snmp walk or snmp like DOS attack left its impression on the Adtran in tying up memory resources or CPU. And this just took a few days to set it over the edge and cause it to lockup. We are running the blow FW versions on the majority of these switches and see some bugs in the software release notes where an snmp walk can cause a system lockup. I'm wondering how If this theory makes sense to anyone and what cpu or memory would be a good place to look for compare free memory heap cpu etc to give me a warning that a switch is about to lockup. Seems like a memory error moreso then a cpu but I cannot get in to 1 after it locks up so cannot be sure,   Any assistance is appreciated.

                        "NV1335A-17-08-02-00-E.biz"
                        "NONVOL:/NV1335A-17-01-01-00-E.biz"

                        a random site switch stats

                         

                        #show memory heap
                        Memory Heap:
                          HeapFree:   78584816
                          HeapSize:   96136176

                        Block Managers:
                          Mgr         Size        Used        Free    Max-Used    Overhead
                                   (bytes)    (blocks)    (blocks)    (blocks)     (bytes)
                          0              0           1           3           4         128
                          1             32       23291        1789       25080      802560
                          2             96        7834         966        8800      281600
                          3            224        1438          11        1449       46368
                          4            480        1383         933        2316       74112
                          5            992         215           2         217        6944
                          6           2016         103           0         103        3296
                          7           4064          94           6         100        3200
                          8           8160          42           3          45        1440
                          9          16352          33           5          38        1216
                          10         32736          11           0          11         352
                          11         65504          12           0          12         384
                          12        131040           2           0           2          64
                          13        262112           2           0           2          64
                          14        524256           0           0           0           0
                          15       1048544           0           0           0           0
                          16       2097120           0           0           0           0
                          17       4194272           0           0           0           0
                          18       8388576           0           0           0           0
                          19      16777184           0           0           0           0

                          Total Overhead (bytes):    1221728
                          Total Used (bytes):        6101056
                          Total Free (bytes):         732896

                         

                         

                         

                         

                        #show processes cpu

                        System load: 1sec:7.38%  1min:7.43%  5min:7.40%  Min: 0.00%  Max: 100.00%

                        Context switch load: 0.17%

                                                              Invoked  Exec Time    Runtime    Load %%

                        Task Id    Task Name        PRI STA   (count)     (usec)     (usec)     (1sec)

                        1          Idle               0 W    70846543       2003     924498      92.45

                        2          Thread Pool        2 W         793        289          0       0.00

                        3          PC Config          5 S    12847152       1022      49432       4.94

                        4          PacketRouting     36 W     5212274         15       3010       0.30

                        5          Timer-00           8 W    49871152          5       1189       0.12

                        6          Nm01               3 W           0     186296          0       0.00

                        7          Clock              7 W      300829         29         49       0.00

                        8          FrontPanel        35 W     1982627        422       8559       0.86

                        9          con0              37 W         483         11          0       0.00

                        10         CF Manager         7 W      189961         24         31       0.00

                        11         PCI Bridge        25 W      988712          4         67       0.01

                        12         Switch            37 W     4544858         13        646       0.06

                        13         Stacking           7 W      101948         19         19       0.00

                        14         SwitchQ           13 W     9356001         16       1021       0.10

                        15         RSTP              35 W     1224035         13       2714       0.27

                        16         RouteTableTick     4 W      165421         89        124       0.01

                        17         OSPF               4 W      233047         32        544       0.05

                        18         IGMPTick           4 W      100844         35         35       0.00

                        19         IGMP-Receiver      4 W           0    2485173          0       0.00

                         

                         

                         

                         

                         

                         


                        • Re: Problem with SNMP Polling
                          kylem New Member

                          I am also having this problem on a 924e first gen. It does not appear to be an SNMP attack based on the diagnosis procedures found here: https://supportforums.adtran.com/docs/DOC-6376#CLI

                           

                          Here is my config:

                           

                          ! ADTRAN, Inc. OS version A4.11.00.E

                          ! Boot ROM version 14.04.00

                          ! Platform: Total Access 924e (1st Gen), part number 4240924L1

                          ! Serial number LBADTN0730AF192

                          !

                          !

                          hostname "BMB-530BB207"

                          enable password encrypted 141250aff3047ff9916b87537a57e90a8b95

                          !

                          clock timezone -5-Eastern-Time

                          !

                          ip subnet-zero

                          ip classless

                          ip routing

                          !

                          !

                          !

                          !

                          no auto-config

                          !

                          event-history on

                          no logging forwarding

                          no logging email

                          !

                          service password-encryption

                          !

                          username "i123" password encrypted "292f6b4a7bd8e8efd48be17ad71b507967cd"

                          !

                          !

                          ip firewall

                          no ip firewall alg msn

                          no ip firewall alg mszone

                          no ip firewall alg h323

                          !

                          !      

                          !

                          !

                          !

                          no dot11ap access-point-control

                          !

                          !

                          !

                          !

                          !

                          !

                          !

                          !

                          !

                          !

                          qos map WAN-EDGE 10

                            match dscp 26 46

                            priority unlimited

                          !

                          !

                          !

                          !

                          interface eth 0/1

                            description Uplink to Radio

                            ip address  152.160.44.142  255.255.255.252

                            ip ffe

                            ip access-group BlockSNMP in

                            media-gateway ip primary

                            traffic-shape rate 3000000

                            qos-policy out WAN-EDGE

                            no awcp

                            no shutdown

                          !

                          !

                          interface eth 0/2

                            description LAN Handoff

                            ip address  152.160.47.25  255.255.255.248

                            ip ffe

                            no shutdown

                          !

                          !

                          !

                          !

                          interface t1 0/1

                            shutdown

                          !

                          interface t1 0/2

                            shutdown

                          !

                          interface t1 0/3

                            shutdown

                          !

                          interface t1 0/4

                            shutdown

                          !

                          !

                          interface fxs 0/1

                            rx-gain +0.0

                            tx-gain +0.0

                            no shutdown

                          !

                          interface fxs 0/2

                            rx-gain +0.0

                            tx-gain +0.0

                            no shutdown

                          !

                          interface fxs 0/3

                            rx-gain +0.0

                            tx-gain +0.0

                            no shutdown

                          !

                          interface fxs 0/4

                            shutdown

                          !

                          interface fxs 0/5

                            shutdown

                          !

                          interface fxs 0/6

                            shutdown

                          !

                          interface fxs 0/7

                            shutdown

                          !

                          interface fxs 0/8

                            shutdown

                          !

                          interface fxs 0/9

                            shutdown

                          !

                          interface fxs 0/10

                            shutdown

                          !

                          interface fxs 0/11

                            shutdown

                          !

                          interface fxs 0/12

                            shutdown

                          !

                          interface fxs 0/13

                            shutdown

                          !

                          interface fxs 0/14

                            shutdown

                          !

                          interface fxs 0/15

                            shutdown

                          !

                          interface fxs 0/16

                            shutdown

                          !

                          interface fxs 0/17

                            shutdown

                          !

                          interface fxs 0/18

                            shutdown

                          !

                          interface fxs 0/19

                            shutdown

                          !

                          interface fxs 0/20

                            shutdown

                          !

                          interface fxs 0/21

                            shutdown

                          !

                          interface fxs 0/22

                            shutdown

                          !

                          interface fxs 0/23

                            shutdown

                          !

                          interface fxs 0/24

                            shutdown

                          !

                          !

                          interface fxo 0/0

                            no shutdown

                          !

                          !

                          !

                          !

                          !

                          !

                          !

                          !

                          ip access-list standard ACCESS-IN

                            permit 216.234.96.0 0.0.1.255 log

                            permit host 66.103.225.122 log

                            deny   any log

                            permit host 66.103.225.123 log

                            permit host 216.234.103.118 log

                          !

                          ip access-list standard SIP-IN

                            permit 216.234.105.72 0.0.0.7

                            permit 66.103.225.88 0.0.0.7

                            deny   any log

                          !

                          !

                          ip access-list extended BlockSNMP

                            permit udp 216.234.96.0 0.0.1.255  host 152.160.44.142 eq snmp  

                            deny   udp any  host 152.160.44.142 eq snmp  

                            permit ip any  any   

                          !

                          !

                          !

                          !

                          ip route 0.0.0.0 0.0.0.0 152.160.44.141

                          !

                          no ip tftp server

                          no ip tftp server overwrite

                          ip http server

                          no ip http secure-server

                          ip snmp agent

                          no ip ftp server

                          no ip scp server

                          no ip sntp server

                          !

                          ip http access-class ACCESS-IN in

                          !

                          snmp-server community public RO

                          !

                          !

                          !

                          !

                          ip sip

                          ip sip udp 5060

                          no ip sip tcp

                          !

                          !

                          !

                          voice feature-mode local

                          voice transfer-mode local

                          voice forward-mode local

                          !

                          !

                          !

                          !

                          !

                          !

                          !

                          voice dial-plan 1 local NXX-XXXX

                          voice dial-plan 2 long-distance NXX-NXX-XXXX

                          !

                          !

                          !

                          !

                          voice class-of-service GLOBAL

                            call-privilege all

                          !

                          voice codec-list GLOBAL

                            default

                            codec g711ulaw

                          !

                          !

                          !

                          voice trunk T01 type sip

                            description "123-SIP"

                            match dnis "NXX-XXXX" substitute "248-NXX-XXXX"

                            sip-server primary 216.234.105.74

                            registrar primary 216.234.105.74

                            codec-group GLOBAL

                          !

                          !      

                          voice grouped-trunk SIP

                            description "Outbound Calls"

                            trunk T01

                            accept $ cost 0

                          !

                          !

                          voice user 01

                            connect fxs 0/1

                            password encrypted "3f3b802c6899a4539120d2dbd1042cd80022"

                            no call-waiting

                            caller-id-override external-number 2487240441

                            did "2487240441"

                            no special-ring-cadences

                            forward-disconnect delay 1000

                            sip-authentication password encrypted "2226c76989b7cc49da5ab592196a551b8c23"

                            codec-group GLOBAL

                          !

                          !

                          voice user 02

                            connect fxs 0/2

                            password encrypted "1410d98cc24448d33d16eaa509eb5aef8ed3"

                            no call-waiting

                            caller-id-override external-number 2487240442

                            did "2487240442"

                            no special-ring-cadences

                            forward-disconnect delay 1000

                            sip-authentication password encrypted "191d6d2607a0b620443811b254d8f6fb7ecb"

                            codec-group GLOBAL

                          !

                          !

                          voice user 03

                            connect fxs 0/3

                            password encrypted "1e1aa742c2891bd23ae4dfcabae276c7347e"

                            no call-waiting

                            caller-id-override external-number 2487240443

                            did "2487240443"

                            no special-ring-cadences

                            forward-disconnect delay 1000

                            sip-authentication password encrypted "23276f427bfc89780d7b16d61c9b89691d63"

                            codec-group GLOBAL

                          !

                          !

                          !

                          !

                          !

                          !

                          !

                          !

                          !

                          ip sip access-class SIP-IN in

                          !

                          !

                          !

                          !

                          !

                          !

                          !

                          !

                          !

                          !

                          !

                          ip sip qos dscp 46

                          !

                          !

                          !

                          !

                          ip rtp quality-monitoring

                          ip rtp quality-monitoring udp

                          ip rtp quality-monitoring sip

                          ip rtp quality-monitoring history max-streams 500

                          !

                          line con 0

                            login

                          !      

                          line telnet 0 4

                            login local-userlist

                            no shutdown

                            access-class ACCESS-IN in

                          line ssh 0 4

                            login local-userlist

                            no shutdown

                            access-class ACCESS-IN in

                          !

                          sntp server 216.234.97.3

                          !

                          !

                          !

                          !

                          end