Symptom
Receive “Negotiating IP Security” when testing connectivity from ISA Server across a Remote Site connection
Consider the following scenario : ISA-A has a Remote Site connection to ISA-B (or a 3rd party IPSec gateway) using IPSec Tunnel Mode.
Figure 1
After creating the Remote Site and creating the Firewall Rule to allow the Local Host Network (at the ISA firewall) access to the Remote Site, you are unable to establish a connection with any protocol. If you test connectivity with PING, you receive the Negotiating IP Security response indefinitely.
Solution
On each ISA Server (or 3rd party VPN gateway), add the external IP address of the opposing ISA firewall into the Addresses tab of the connection.
Description
If the ISA firewall is installed on Windows 2003, you can use netsh ipsec dynamic show qmfilters all command to see the filters referenced below.
SubnetA -- ISA-A -- Internet -- ISA-B -- SubnetB
ISA-A has a Remote Site for SubnetB containing the addresses of that subnet.
This results in ISA-A having an IPSec Filter List of:
A1 SubnetA > SubnetB
A3 ISA-A > SubnetB
A2 SubnetA < SubnetB
A4 ISA-A < SubnetB
ISA-B has a Remote Site for SubnetA containing the addresses of that subnet.
B1 SubnetB > SubnetA
B3 ISA-B > SubnetA
B2 SubnetB < SubnetA
B4 ISA-B < SubnetA
When you PING from ISA-A to SubnetB, the traffic sources from ISA-A's external IP address. Because of this, ISA-A has a matching filter for the traffic (A3 above) but ISA-B doesn't have a matching filter for this (B1 through B4 don't match the traffic). As a result, ISA-A continues trying to negotiate IP Security with ISA-B but this will never complete as there is not a match for the traffic on ISA-B.
To fix this, on ISA-A, you'll need to add ISA-B's external IP address into the Addresses tab of the Remote Site. On ISA-B, you'll need to add ISA-A's external IP address.
What happens is now ISA will now have the following filters...
ISA-A
A1 SubnetA > SubnetB
A3 ISA-A > SubnetB
A5 ISA-B > SubnetA
A2 SubnetA < SubnetB
A4 ISA-A < SubnetB
A6 ISA-B < SubnetA
ISA-B
B1 SubnetB > SubnetA
B3 ISA-B > SubnetA
B5 ISA-A > SubnetB
B2 SubnetB < SubnetA
B4 ISA-B < SubnetA
B6 ISA-A < SubnetB
With this setup, when ISA-A tries to communicate with SubnetB, A3 now matches B5 and A4 matches B6 and the Security Associations can come online.
