-
Re: GRE Tunnel randomly dropping
levi May 8, 2012 6:33 AM (in response to smross)Thank you for asking this question in the support community. It may be helpful if you attach a copy of the running configuration of the ADTRAN unit (please make sure to remove any information that may be sensitive to the organization).
First, is the message you are receiving on the Cisco from the remote ADTRAN device's IP address? The "Invalid Security Parameter Index (SPI)" message is typically related to an IPSec VPN with IKE, and is not used in standard GRE tunnels. Is this a GRE/IPSec tunnel?
One thing I would suggest is to perform a debug on the GRE tunnel to monitor the GRE keepalive transmissions/reception. You can view this with the debug interface tunnel x command (where "x" represents the GRE tunnel interface number).
Please, reply with any additional information or questions. I will be happy to help in any way I can.
Levi
-
Re: GRE Tunnel randomly dropping
smross May 8, 2012 7:26 AM (in response to levi)Hi Levi, thanks for responding. yes you are correct its GRE/IPSec tunnel.
The ip address in the SPI message has the source addr as the 1335 end and the dest addr as the cisco end.
I've attached the config of the 1335
While i was on the switch the tunnel happened to go down, i then turned on debugging and this is the output minus the public IP's
2012.05.08 10:13:48 INTERFACE_STATUS.tunnel 1 changed state to down
CHC-RDGOH-IDF-L3SWT-RIDGEWOOD-1#debug interface tunnel 1
CHC-RDGOH-IDF-L3SWT-RIDGEWOOD-1#
2012.05.08 10:14:19 TUNNEL.1 Keepalive retries exceeded without Rx a keepalive.
2012.05.08 10:14:21 TUNNEL.1 GRE/IP encapsulated X.X.X.X->X.X.X.X (len=48).
2012.05.08 10:14:21 TUNNEL.1 Keepalive Tx.
2012.05.08 10:14:31 TUNNEL.1 GRE/IP encapsulated X.X.X.X->X.X.X.X (len=48).
2012.05.08 10:14:31 TUNNEL.1 Keepalive Tx.
2012.05.08 10:14:31 TUNNEL.1 GRE to decaps X.X.X.X->X.X.X.X (len=24 ttl=253).
2012.05.08 10:14:31 TUNNEL.1 Keepalive Rx.
2012.05.08 10:14:32 TUNNEL.1 GRE/IP to decaps X.X.X.X->X.X.X.X (len=48 ttl=254).
2012.05.08 10:14:32 TUNNEL.1 GRE decapsulated IP X.X.X.X->X.X.X.X (len=24 ttl=255).
2012.05.08 10:14:33 INTERFACE_STATUS.tunnel 1 changed state to up
2012.05.08 10:14:33 TUNNEL.1 GRE/LLDP encapsulated X.X.X.X->X.X.X.X (len=239).
2012.05.08 10:14:40 TUNNEL.1 GRE/IP encapsulatedX.X.X.X->X.X.X.X (len=48).
2012.05.08 10:14:40 TUNNEL.1 Keepalive Tx.
2012.05.08 10:14:40 TUNNEL.1 GRE to decaps X.X.X.X->X.X.X.X (len=24 ttl=253).
2012.05.08 10:14:40 TUNNEL.1 Keepalive Rx.
From the debug output it appears that the 1335 is transmitting keep alives but it didnt receive keepalives back within a certain period.
With this issue being intermittent, is it best practice to have this debug running for long periods of time? Its a very busy debug.
Thanks,
Sean
-
1335config.zip 3.1 K
-
Re: GRE Tunnel randomly dropping
levi May 8, 2012 8:53 AM (in response to smross)Sean:
Thank you for replying with the configuration and the debug output. As you said above, the debug indicates that this unit is not receiving the keepalive messages, and thus tears the GRE tunnel down. By default the keepalives are sent every 10 seconds, and it takes three missed retries before it is declares the peer unreachable. For something as intermittent as this, you will most likely have to monitor the link long term. It would also be beneficial to verify if the other end of the tunnel was receiving the keepalives the ADTRAN is transmitting, when it is in the failed state.
Also, I would recommend disabling LLDP on the tunnel interface, because the Cisco does not have it enabled by default. The command is no lldp send-and-receive.
Levi
-
Re: GRE Tunnel randomly dropping
adam May 15, 2012 10:06 AM (in response to smross)Do you also have the Cisco side of the config? I am trying to do something similar but am having troubles just getting the tunnel up.
-
Re: GRE Tunnel randomly dropping
levi May 15, 2012 11:25 AM (in response to adam)adam:
No, I do not have an example configuration for GRE/IPSec tunnels on a Cisco, but here is a document on Configuring a GRE over IPSec Tunnel in AOS.
Also, here is another post on when to use GRE tunnels: https://supportforums.adtran.com/message/2133#2133
Levi
-
-
-
-
Re: GRE Tunnel randomly dropping
smross May 8, 2012 1:38 PM (in response to smross)Hi Levi, is there an Adtran equivalent of Cisco's IP SLA.
Thanks,
Sean
-
Re: GRE Tunnel randomly dropping
levi May 9, 2012 7:20 AM (in response to smross)Sean:
From what I know of Cisco's IP SLA, our Network Quality Monitoring and Integrated Traffic Monitoring features are similar. Feel free to review the documents I linked on NQM and ITM, and if you have questions about it I will be happy to branch this to a new discussion.
Levi
-
Re: GRE Tunnel randomly dropping
smross May 9, 2012 1:05 PM (in response to levi)Levi, thanks for that resource regarding NQM. After more troubleshooting we found out that the tunnel is going down every hour but it comes back up quick enough not to send an alert. We noticed this based off our BGP notifications(hold time expired) from the 1335 side of the tunnel.
I've attached a debug output from debug crypto ike client and crypto ipsec, if you can assist in interpreting the debug
we think that the issue is when the keys timeout and try to re-negotiate the tunnel drops during the process.
-
ike_deb.rtf 81.5 K
-
Re: GRE Tunnel randomly dropping
levi May 9, 2012 1:56 PM (in response to smross)Sean:
The IPSec debug you attached is the output from a successful IPSec tunnel negotiation. Unfortunately, the portion where the tunnel went down was not captured. I would need to see the debug messages when the tunnel was terminated to facilitate you in diagnosing the problem. Also, with GRE/IPSec tunnels, the GRE is encapsulated within the IPSec tunnel, so if the IPSec goes down the GRE will also go down. Therefore, it is important to determine if IPSec or only GRE is dropping.
Levi
-
-
-
-
Re: GRE Tunnel randomly dropping
levi May 24, 2012 2:38 PM (in response to smross)I went ahead and flagged this post as “Assumed Answered.” If any of the responses on this thread assisted you, please mark them as either Correct or Helpful answers with the applicable buttons. This will make them visible and help other members of the community find solutions more easily. If you still need assistance, I would be more than happy to continue working with you on this - just let me know in a reply.
Levi