Fast Down Detection


Fast down detection or sub-second link failure detection is needed in a modern networks. Network operators of modern networks require that they can detect failure in sub-second and react to either soft or hard failures as quickly as possible.

The following are the two categories of fast down detection.

1. Polling

One of the method that polling uses is routing protocol hellos.

By default, EIGRP sends hello packets every 5 seconds on high bandwidth link and every 60 seconds on low bandwidth multipoint links.
The speed at which the EIGRP sends hello packets is called hello interval.
The hello interval is configurable using the command ip hello-interval eigrp.

The hold time is three times the hello interval. The hold time is the duration that a router will consider a neighbour is up without receiving a hello packet.
Hold time is configurable using the command ip hold-interval eigrp.

EIGRP neighbours can establish adjacency even their hello interval and hold time are different.

EIGRP does not support fast hellos or sub-second hello unlike OSPF and IS-IS.

For 5-second hello:
broadcast media, such as Ethernet, Token Ring, and FDDI
point-to-point serial links, such as PPP or HDLC leased circuits, Frame Relay point-to-point subinterfaces, and ATM point-to-point subinterface
high bandwidth (greater than T1) multipoint circuits, such as ISDN PRI and Frame Relay

For 60-second hello:
multipoint circuits T1 bandwidth or slower, such as Frame Relay multipoint interfaces, ATM multipoint interfaces, ATM switched virtual circuits and ISDN BRIs

OSPF sends hello packet every 10 seconds for broadcast media (e.g. ethernet) and every 30 seconds for a non-broadcast media (e.g.frame relay).
The dead interval (similar to hold time in EIGRP) is four times the value of the hello interval.
OSPF neighbours should have the same hello and dead interval otherwise, adjacency does not come up.

Unlike EIGRP, fast hello or sub-second hello is supported in OSPF. The benefit of fast hello is for fast down detection of neighbour particularly beneficial in broadcast media (.e.g ethernet).
Fast hello is configurable using this command ip ospf dead-interval minimal hello-multiplier multiplier.

IS-IS sends hello packet every 10 seconds. The hello interval can set differently for Level 1 and 2 except on point-to-point interfaces.
Hello interval is configurable using this command isis hello-interval {seconds} [level-1 | level-2].
IS-IS supports fast hello for faster convergence same with OSPF. To configure the fast hello, use this command isis hello-multiplier multiplier [level-1 | level-2].

BGP uses keepalive for fast down detection. By default, BGP sends keepalive every 60 seconds with a hold time of three times the keepalive which is 180 seconds.
These parameters are configurable using this command neighbor [ip-address | peer-group-name] timers keepalive holdtime [min-holdtime]

Fine tuning hello timer, hold down timer and keepalive makes the link failure detection faster. In return, faster network convergence.
However, tuning timers/keepalive must be carefully examined in big networks as this is a CPU intensive.

Consider a network of ten point-to-point links with ten neighbours and using OSPF as its routing protocol. Hello and hold down timers are on its default.

1 OSPF hello per second x 10 point-to-point link = 10 hello packets per second

What happened if we enable OSPF fast hello. In this example, let us assume that hello is sent every 330ms. Thus, three hellos in one second.

3 OSPF hello per second x 10 point-to-point link = 30 hello packets per second

The numbers shown above is not that big but how about if you had more than 50 neighbours? or 75? or 100?

Second method is protocol’s built-in hellos for fast down detection.

Unidirectional Link Detection protocol also known as UDLD for short is a proprietary protocol developed by Cisco to determine the physical status of the link. UDLD is good in detecting these scenarios:

1. Links are up on both sides, however, packets are only received by either side.
2. Miswire when receive and transmit fibers are not connected to the same port on the remote side.

Fast UDLD is the latest enhancement of UDLD. Fast UDLD is created for sub-second fast down detection.

To enable UDLD and Fast UDLD, all switches must support these two protocols.

Spanning Tree Protocol Bridge Assurance (STP BA) is use to protect against problems that can cause bridging loops in the network, specifically, unidirectional link failure (wiring mistake) or other software failure like when it continues to forward data traffic when it is no longer running the spanning tree algorithm.

To enable STP BA, all switches must support this protocol.

Etherchannel is a port aggregation technology to bundle two or more physical ports to form into one logical port. This is use to increase bandwidth, load balance traffic and link redundancy. PAgP and LACP are the two etherchannel protocols. Port Aggregation Protocol (PAgP) is Cisco proprietary and Link Aggregation Control Protocol (LACP) is open standard.
These protocols have built-in timers and if one of the physical links is down for whatever reason, this port is automatically place taken out of the etherchannel bundle.

Continue reading Fast Down Detection

UNetLab Common Issues


These are the common issues I encountered in setting up UNetLab and how I fixed these issues. I’m sharing this so UNetLab users will benefit from it. I will constantly update this list as I progress.

Issue 1: UNetLab is complaining about – Neither intel VT-X or AMD-V found.
Solution: Find the .vmx file for the UNetLab and add this line at the end.

vhv.enable = "TRUE".

Issue 2: After UNetLab install, running apt-get update and apt-get upgrade won’t update the UNetLab package.
Solution: I had to force update using these commands.

apt-get update
apt-get -o Dpkg::Options::="--force-overwrite" install unetlab unetlab-qemu

Continue reading UNetLab Common Issues

Access VMware ESXi Console from Ubuntu 14.04


VMware has no native VSphere client for Linux hosts and VMware pushes its users to use the VMware web client. There is a workaround for this but it doesn’t give you the full benefits of using a full pledge VSphere client in Windows, however it gets the job done.

To access VMware ESXi console from ubuntu download the free VMware Player for Linux 64-bit . Move it to your preferred install folder, give execution privilege and install. Follow the installation prompt all through out the process.

mv ~/Desktop/ ~/
chmod u+x VMware-Player-7.1.0-2496824.x86_64.bundle

Continue reading Access VMware ESXi Console from Ubuntu 14.04

Cisco TelePresence Touch Panel Stuck in Downloading Software


Issue: Cisco TelePresence touch panel stuck in downloading software after system boots up.

TelePresence Touch Panel stuck in downloading software

Platform: Any Cisco TelePresence endpoints that support touch Panel.

Findings: I didn’t really find out what causes the Cisco TelePresence Touch Panel stuck in downloading software after systems boots up. I have tried three ways in resolving this issue and they are ordered in preference below:

1. Reboot the Cisco TelePresence endpoint.
– Most of the times this issue is resolved after power cycle.
2. Factory reset the TelePresence Touch Panel.
– If the first option didn’t fix the issue, try the factory reset option. Steps provided.

> Hold exclamation (!) button for 10 seconds until it lights up.
> Release exclamation (!) button.
> Press mute button twice (it will light up).

3. Upgrade to newer release.
– Last resort! Make sure you have CCO account and a valid contract t before you can access the software off Cisco site and upgrade to newer release. If your version is TC5 and like to upgrade to TC7, you need a release TC7 key otherwise you can’t upgrade. Other option is stay on TC5 and upgrade to latest minor version (e.g. TC5.1.13).

H323 NAT address not configured


Issue: Cisco TelePresence endpoint cannot make H323 point-to-point call to another Cisco TelePresence endpoint. Both endpoints are standalone H323 and no Gatekeeper. Symptom is when calling from either endpoint the call automatically get disconnected after few seconds.

Platform: EX, SX, MX and Integrator C series TelePresence endpoints
Software version: Any version as long it supports H323.

Findings: Checking for any system errors in the logs revealed the issue was caused by the H323 NAT was turned on. Note that H323 NAT is on by default.

Mar 1 15:43:03 (none) main: H323CC: Inc Setup, NAT enabled but H323 NAT address not configured, disconnecting call

To fix the issue, login to administration web page. Go to Configuration > Advanced Configuration > H323 > NAT > Then set the mode to off.

Anonymous SIP Calls


This document walks you through on how to implement anonymous SIP calls in a Cisco Voice Gateway (e.g 28xx/38xx, 29xx/39xx).

1. Enable privacy at the header level in the SIP message. This setting affects all calls (applied globally).

voice service voip
   privacy header

2. This dial-peer configuration prevents the calling-party number from being shown. Per dial-peer configuration.

dial-peer voice 10 voip
clid restrict

Continue reading Anonymous SIP Calls

OpenVPN DNS Issue


Issue: OpenVPN connects successfully but cannot resolved hostnames when browsing. It looks like an OpenVPN DNS issue.

Platform: Ubuntu 14.04 LTS (Trusty Tahr)
Software version: OpenVPN 2.3.4

OpenVPN logs indicated the DNS configuration was successfully pushed from OpenVPN server to client.

Tue Oct 14 05:08:39 2014 SENT CONTROL [changeme]: 'PUSH_REQUEST' (status=1)
Tue Oct 14 05:08:40 2014 PUSH: Received control message: 'PUSH_REPLY,redirect-gateway def1 bypass-dhcp,dhcp-option DNS,dhcp-option DNS,route,topology net30,ping 10,ping-restart 120,ifconfig'

I tested nslookup but failed.

rejohn@R007:/opt/openvpn-2.3.4$ nslookup

** server can't find REFUSED

The error above indicated that my local ISP didn’t allow lookup from non-AU IPs. This make sense because my IP now changed to a non-AU IP.

Continue reading OpenVPN DNS Issue

Extension Mobility Login is Unavailable (23)


Issue: Cisco 7821 SIP IP Phone intermittently login to Extension Mobility service. Error displayed on the phone was Login is Unavailable (23).

Cisco 7821 SIP IP Phone running on firmware version 10.1(1)SR1
Cisco Unified Communications Manager version

Time        Source                Destination           Protocol Length Info
0.000000       TCP      70     51493 > http-alt [SYN] Seq=0 Win=14480 Len=0 MSS=1460 TSval=195791 TSecr=619720525
0.000095        TCP      70     http-alt > 51493 [SYN, ACK] Seq=0 Ack=1 Win=14480 Len=0 MSS=1460 TSval=619767712 TSecr=195791
0.000686       TCP      66     51493 > http-alt [ACK] Seq=1 Ack=1 Win=14480 Len=0 TSval=195791 TSecr=619767712
0.006832       HTTP     510    GET /emapp/EMAppServlet?device=SEP00082F1B6653 HTTP/1.1 
0.006880        TCP      66     http-alt > 51493 [ACK] Seq=1 Ack=445 Win=15544 Len=0 TSval=619767719 TSecr=195792
0.062434        HTTP/XML 814    HTTP/1.1 200 OK 
0.063115       TCP      66     51493 > http-alt [ACK] Seq=445 Ack=749 Win=15708 Len=0 TSval=195797 TSecr=619767774
0.065465       TCP      66     51493 > http-alt [FIN, ACK] Seq=445 Ack=749 Win=15708 Len=0 TSval=195797 TSecr=619767774
0.065564        TCP      66     http-alt > 51493 [FIN, ACK] Seq=749 Ack=446 Win=15544 Len=0 TSval=619767777 TSecr=195797
0.065973       TCP      66     51493 > http-alt [ACK] Seq=446 Ack=750 Win=15708 Len=0 TSval=195798 TSecr=619767777
10.020709       TCP      70     51494 > http-alt [SYN] Seq=0 Win=14480 Len=0 MSS=1460 TSval=196793 TSecr=619767777
10.020821        TCP      70     http-alt > 51494 [SYN, ACK] Seq=0 Ack=1 Win=14480 Len=0 MSS=1460 TSval=619777733 TSecr=196793
10.021377       TCP      66     51494 > http-alt [ACK] Seq=1 Ack=1 Win=14480 Len=0 TSval=196793 TSecr=619777733
10.026852       HTTP     534    GET /emapp/EMAppServlet?device=SEP00082F1B6653&seq=35600&userid=35600 HTTP/1.1 
10.026900        TCP      66     http-alt > 51494 [ACK] Seq=1 Ack=469 Win=15544 Len=0 TSval=619777739 TSecr=196794
10.166359        HTTP/XML 576    HTTP/1.1 200 OK 
10.167019       TCP      66     51494 > http-alt [ACK] Seq=469 Ack=511 Win=15544 Len=0 TSval=196808 TSecr=619777878
10.176926       TCP      66     51494 > http-alt [FIN, ACK] Seq=469 Ack=511 Win=15544 Len=0 TSval=196809 TSecr=619777878
10.177023        TCP      66     http-alt > 51494 [FIN, ACK] Seq=511 Ack=470 Win=15544 Len=0 TSval=619777889 TSecr=196809
10.177447       TCP      66     51494 > http-alt [ACK] Seq=470 Ack=512 Win=15544 Len=0 TSval=196809 TSecr=619777889

Hypertext Transfer Protocol
    GET /emapp/EMAppServlet?device=SEP00082F1B6653&seq=35600&userid=switch HTTP/1.1\r\n
        [Expert Info (Chat/Sequence): GET /emapp/EMAppServlet?device=SEP00082F1B6653&seq=35600&userid=switch HTTP/1.1\r\n]
            [Message: GET /emapp/EMAppServlet?device=SEP00082F1B6653&seq=35600&userid=switch HTTP/1.1\r\n]
            [Severity level: Chat]
            [Group: Sequence]
        Request Method: GET
        Request URI: /emapp/EMAppServlet?device=SEP00082F1B6653&seq=35600&userid=35600
        Request Version: HTTP/1.1
    Accept: x-CiscoIPPhone/Menu, x-CiscoIPPhone/Text,  x-CiscoIPPhone/IconMenu, x-CiscoIPPhone/IconFileMenu, x-CiscoIPPhone/Directory, x-CiscoIPPhone/Input, x-CiscoIPPhone/Execute, text/*, */*\r\n
    Accept-Language: en_GB\r\n
    Accept-Charset: utf-8,iso-8859-1;q=0.8\r\n
    x-CiscoIPPhoneModelName: CP-7821\r\n
    x-CiscoIPPhoneSDKVersion: 8.5.1\r\n
    x-CiscoIPPhoneDisplay: 396,133,1,G\r\n
    [Full request URI:]
    [HTTP request 1/1]
    [Response in frame: 16]

Continue reading Extension Mobility Login is Unavailable (23)