11 Replies Latest reply on May 31, 2013 12:27 AM by fish

    Undesired Layer-3 Romaing Issue  - Help!

    fish New Member

      Hello,

       

      Basic topology:

      I have a deployment where I have 14 buildings, each behind its own layer-3 routed network.  Each building has an Student VLAN tagged 200.  We are using captive portal web-auth with a 10 minute idle-timeout.  Running 2.1.0.14, and 6.5.4.-11.

       

      Problem:

      Users associating to an AP in building X get placed into a location in building Y, with AP from building X tunneling all traffic to building Y.  It is ALWAYS the same building that they are getting incorrectly placed into.  The users are random.  Essentially, the user should have a 10.209.x.x IP from an AP on a 172.23.9.x VLAN, but instead, they are getting a 10.210.x.x IP from the AP at 172.23.10.x - even thought they haven't been to that building in days.

        • Re: Undesired Layer-3 Romaing Issue  - Help!
          dalicea New Member

          Are all the APs showing the proper “Active  locations”  under Status > Details > Active Connections > Active Locations column? Usually, I have seen the tunneling if the AP cannot see the DHCP scope for the location it is handing out IPs. I’ve also seen it where the switchport configuration does not have the vlan for the location, so the AP does not know where to grab IPs from for a particular vlan.

           

           

          -Dave

            • Re: Undesired Layer-3 Romaing Issue  - Help!
              fish New Member

              Dave,

               

              Wow!  Actually, it doesn't show any active locations other than the AP VLAN itself.  Now that you've given me a major clue, can you help with a solution for this?

              They are connected to Cisco switches (3750).  All of the proper VLANs are configured on the switch  My port config looks like this:

               

              interface FastEthernet1/0/1
              power inline consumption 9600
              switchport trunk encapsulation dot1q
              switchport trunk native vlan 15
              switchport mode trunk
              ip arp inspection trust
              no mdix auto
              ip dhcp snooping trust

                • Re: Undesired Layer-3 Romaing Issue  - Help!
                  dalicea New Member

                  That might be the issue.

                   

                  This is an example of one location using a 6513-E chassis.

                  To explain what the VLANs below are:

                  Vlan 71: The actual AP vlan:

                  Vlan 70: Wireless vlan for our admin users (has its own active location in vWLAN)

                  Vlan 68: Wireless vlan for our students and guests (has its own active location in vWLAN)

                   

                   

                  interface GigabitEthernet6/7

                  description Wireless-APs

                  switchport

                  switchport trunk encapsulation dot1q

                  switchport trunk native vlan 71

                  switchport trunk allowed vlan 68,70,71

                  switchport mode trunk

                  switchport nonegotiate

                  spanning-tree portfast edge trunk

                   

                   

                  This is what we have at all sites and have not had the issue UNLESS the DHCP scope goes down. Once we allow those specific vlans on the actual AP interface, and reboot the AP, it works. They pick up the locations we had created on the vWLAN appliance.

                   

                   

                  David Alicea, MBA

                  Network Engineer, National Team

                  DeVry Inc. | p: 630.645.1145 | dalicea@devry.edu<mailto:dalicea@devry.edu>

                    • Re: Undesired Layer-3 Romaing Issue  - Help!
                      fish New Member

                      Dave,

                       

                      I'm trying those configs on a small area.  I'll let you know if this works.

                        • Re: Undesired Layer-3 Romaing Issue  - Help!
                          dalicea New Member

                          I was just wondering if you had a chance to try the config change out.

                           

                           

                          -Dave

                            • Re: Undesired Layer-3 Romaing Issue  - Help!
                              fish New Member

                              Dave,

                               

                              Thank you for your continued interest and follow-up.  As I said, my issued was really sparked by users being dumped into a building they were never at, behind a T1.  I've have been working very closely with Adtran support.  They had me reboot the vWLAN.  Since the reboot, users are no longer being directed to that remote building on layer-3 roam.  There are still layer-3 roams, but these are expected.

                               

                              I discussed your recommendation with their engineers.  They aid that it shouldn't really make a difference to the AP, although logically, I understand that it would streamline the AP's detection process of available networks.  This is something that I will be implementing campus-wide over the next week, as well as removing the power restriction statement.  It will take time to make these changes.

                               

                              They think the issue may lie in the fact that I have two DHCP servers (one running on each HSRP router), despite the fact that they do not have over-lapping IP ranges to assign.  They are at a loss.

                               

                              As far as a solution goes........  They said that implementing 2.2.1.20 should solve any layer-3 roam issues due to the clear layout of Domains and Platform.  However, this needs to be done with caution.  Additionally, there have been many things integrated into the 6.6.0.30 firmware that would mitigate some adverse functionality.

                               

                              Adtran states that they may release a patch to add the functionality of disabling layer-3 roam, but it will take time, and there is no rush.  Apparently, they think that no one aside from us would like to have this disabled......

                      • Re: Undesired Layer-3 Romaing Issue  - Help!
                        daniel.blackmon Employee

                        The BSAP's will send out a DHCP discover message to determine if it can service a location. That would explain why the AP says it cannot service a location if the DHCP server is down.

                         

                        There are also times when a switch may not fully initialize before the AP does. The main instance we have seen this occur is with PoE switches. Some switches will initialize their PoE controller as soon as power is applied to the switch. This means the AP will power on almost immediately afterwards. When this happens, the AP may initialize before the switch, and the switchports may not be tagging traffic properly at that time. It is a rare situation, but it happens.

                          • Re: Undesired Layer-3 Romaing Issue  - Help!
                            fish New Member

                            Charlie,

                             

                                 Thank you, but this isn't my situation.  Our DHCP servers are the routers in the building's MDF.  They are running HSRP, and each has a non-overlapping range on the subnet to assign.  The switches have not rebooted, and the ports they APs are on have not bounced.

                              • Re: Undesired Layer-3 Romaing Issue  - Help!
                                fish New Member

                                S.D.C., and all,

                                 

                                First, let's summarize some issues:

                                 

                                1) Dual routers are present running DHCP with exclusion ranges that prevent each router from giving out a duplicate IP address.

                                These routers are running as router on a stick config, with sub-interfaces that are also the default gateways of the given IP pools.

                                 

                                2) New port config that looks like this:

                                 

                                interface GigabitEthernet1/0/1
                                switchport trunk encapsulation dot1q
                                switchport trunk native vlan 15
                                switchport mode trunk
                                ip arp inspection trust
                                no mdix auto
                                ip dhcp snooping trust

                                 

                                3) Original issue is that Layer-3 roam occurs when you don't want it to.

                                 

                                I believe that I have found a resolution.  First, let's analyze what's going.  The router answer DHCP w/o issue.  However, they BOTH answer because it is a layer-2 UDP broadcast.  Debugging confirms that there is quite a lot of chattiness until they finally decide which router should give the address.  This appears to be determined by the client finally "accepting" one of the router's offers.  Computers, however, are patient because they actually want to receive an IP address.  They then notify you if they have a duplicate IP.  The APs don't work quite the same way.  They do a quick DHCP UDP broadcast to see what networks are serviceable by them.  They don't go through the same process as a PC.  My theory is that since both routers have this "argument", the AP just sometimes doesn't learn what it needs to, and simply doesn't mark that network location as serviceable by that AP.  The only way for the DHCP to follow the HSRP virtual address and not have both routers answer, while at the same time, maintaining redundancy, is to run the MDF switch as a layer-3 switch.  Configure the interface VLANs on the switch.  Use an ip-helper command in the interface config, and point it to the HSRP IP of the two WAN routers.  Now the DHCP UDP broadcast packet becomes a UDP unicast packet.   The routers have the sub-interface removed, and run only with 2 IPs (a WAN and a LAN).  The switchport that the router is connected to is no longer a trunk port.  Instead the ports is an access VLAN port on the same VLAN that you are using for infrastructure equipment.  Sample configs are:

                                 

                                Switch

                                interface Vlan1
                                description MANAGEMENT-VLAN
                                ip address 172.16.5.11 255.255.255.0
                                no ip redirects
                                no ip proxy-arp

                                !

                                 

                                Additionally, I clarified that the idle timeout setting in the vWLAN refers to how long you go without being connected to an AP, regardless of whether or not you are actively using it.  This allows us to drop the timer to as low as 5 minutes.  Doing this, plus the nework changes above, seem to have virtually eliminated all undesired layer-3 roaming.

                                 

                                If anyone out there wants more detail or info, please feel free to contact me.