Telnet Connection Timed Out

Posted by
Facebooktwitterredditpinterestlinkedintumblr

Issue: Telnet connection timed out when accessing the CPE via the secondary management interface loop1 as shown in the figure. The disconnection happens when the user stop sending data across the telnet session (e.g. stop typing commands).

Telnet connection timed out

Findings: Started my investigation by taking a packet capture on the Fortinet Fortigate 100D. The packet capture shown in Exhibit 2 reveals that the Firewall was only getting one way traffic, that is from 172.31.100.1 to 172.31.1.2. This behavior indicated the network exhibits asymmetric routing but still this doesn’t explains why the Telnet connection timed out.

Next step was to run debug command (debug ip tcp transactions) on the CPE, Cisco 881 Router. Upon checking ACK packets were timing out as well (Exhibit 3). This was a strange behavior. I went back to the Fortigate 100D to check traffic flow (diagnose debug flow trace). What I found out was valuable information. The Fortigate was resetting the connection due to no session matched (Exhibit 4). This was a normal behavior of the Fortigate 100D because in the first place it was not getting the complete TCP 3-way handshake. Hence was only seeing one way traffic in the packet capture (Exhibit 2). This was why the Telnet connection was timing out.


Solution: To prove this hypothesis, I disabled stateful inspection on the Fortigate by entering this command set asymroute enable under config system settings. After applying the said command, the Telnet connection wasn’t timing out anymore. Please take note that the disabling stateful inspection reduces Firewall’s capability in detecting attacks. Each packet is now treated individually which makes it stateless Firewall.

|Time     | 172.31.100.1                          |
|         |                   | 172.31.1.2        |                   
|0.000000 |         1478→23 [SYN] Seq=0           |TCP: 1478→23 [SYN] Seq=0 Win=8192 Len=0 MSS=1300 WS=4 SACK_PERM=1
|         |(1478)   ------------------>  (23)     |
|0.224379 |         1478→23 [ACK] Seq=1           |TCP: 1478→23 [ACK] Seq=1 Ack=1 Win=261568 Len=0
|         |(1478)   ------------------>  (23)     |
|0.225379 |         Telnet Data ...               |TELNET: Telnet Data ...
|         |(1478)   ------------------>  (23)     |
|0.282443 |         Telnet Data ...               |TELNET: Telnet Data ...
|         |(1478)   ------------------>  (23)     |
|0.343053 |         1478→23 [ACK] Seq=31          |TCP: 1478→23 [ACK] Seq=31 Ack=538 Win=261568 Len=0
|         |(1478)   ------------------>  (23)     |
|0.399259 |         1478→23 [ACK] Seq=31          |TCP: 1478→23 [ACK] Seq=31 Ack=680 Win=261000 Len=0
|         |(1478)   ------------------>  (23)     |
|0.401203 |         1478→23 [ACK] Seq=31          |TCP: 1478→23 [ACK] Seq=31 Ack=689 Win=260964 Len=0
|         |(1478)   ------------------>  (23)     |
|0.401677 |         Telnet Data ...               |TELNET: Telnet Data ...
|         |(1478)   ------------------>  (23)     |
|0.401970 |         Telnet Data ...               |TELNET: Telnet Data ...
|         |(1478)   ------------------>  (23)     |
|0.507840 |         1478→23 [ACK] Seq=45          |TCP: 1478→23 [ACK] Seq=45 Ack=698 Win=260928 Len=0
|         |(1478)   ------------------>  (23)     |
|0.976703 |         Telnet Data ...               |TELNET: Telnet Data ...
|         |(1478)   ------------------>  (23)     |
|1.200812 |         Telnet Data ...               |TELNET: Telnet Data ...
|         |(1478)   ------------------>  (23)     |
|1.306026 |         1478→23 [ACK] Seq=47          |TCP: 1478→23 [ACK] Seq=47 Ack=700 Win=260920 Len=0
|         |(1478)   ------------------>  (23)     |
|1.354071 |         Telnet Data ...               |TELNET: Telnet Data ...
|         |(1478)   ------------------>  (23)     |
|1.448583 |         Telnet Data ...               |TELNET: Telnet Data ...
|         |(1478)   ------------------>  (23)     |
|1.554383 |         1478→23 [ACK] Seq=49          |TCP: 1478→23 [ACK] Seq=49 Ack=702 Win=260912 Len=0
|         |(1478)   ------------------>  (23)     |
|1.600429 |         Telnet Data ...               |TELNET: Telnet Data ...
|         |(1478)   ------------------>  (23)     |
|1.713811 |         1478→23 [ACK] Seq=50          |TCP: 1478→23 [ACK] Seq=50 Ack=703 Win=260908 Len=0
|         |(1478)   ------------------>  (23)     |
|1.784601 |         Telnet Data ...               |TELNET: Telnet Data ...
|         |(1478)   ------------------>  (23)     |
|1.892420 |         1478→23 [ACK] Seq=51          |TCP: 1478→23 [ACK] Seq=51 Ack=704 Win=260904 Len=0
|         |(1478)   ------------------>  (23)     |
|1.944876 |         Telnet Data ...               |TELNET: Telnet Data ...
|         |(1478)   ------------------>  (23)     |
|2.050144 |         1478→23 [ACK] Seq=52          |TCP: 1478→23 [ACK] Seq=52 Ack=705 Win=260900 Len=0
|         |(1478)   ------------------>  (23)     |
|2.104959 |         Telnet Data ...               |TELNET: Telnet Data ...
|         |(1478)   ------------------>  (23)     |
|2.209701 |         1478→23 [ACK] Seq=54          |TCP: 1478→23 [ACK] Seq=54 Ack=717 Win=260852 Len=0
|         |(1478)   ------------------>  (23)     |
|2.776770 |         Telnet Data ...               |TELNET: Telnet Data ...
|         |(1478)   ------------------>  (23)     |
|3.250085 |         [TCP Retransmission]          |TELNET: [TCP Retransmission] Telnet Data ...
|         |(1478)   ------------------>  (23)     |
|3.294536 |         Telnet Data ...               |TELNET: Telnet Data ...
|         |(1478)   ------------------>  (23)     |
|3.354375 |         Telnet Data ...               |TELNET: Telnet Data ...
|         |(1478)   ------------------>  (23)     |
|3.721951 |         Telnet Data ...               |TELNET: Telnet Data ...
|         |(1478)   ------------------>  (23)     |
|3.974734 |         Telnet Data ...               |TELNET: Telnet Data ...
|         |(1478)   ------------------>  (23)     |
|4.255355 |         Telnet Data ...               |TELNET: Telnet Data ...
|         |(1478)   ------------------>  (23)     |
|4.574883 |         Telnet Data ...               |TELNET: Telnet Data ...
|         |(1478)   ------------------>  (23)     |
|4.895467 |         Telnet Data ...               |TELNET: Telnet Data ...
|         |(1478)   ------------------>  (23)     |
|5.215591 |         Telnet Data ...               |TELNET: Telnet Data ...
|         |(1478)   ------------------>  (23)     |
|7.876209 |         1478→23 [ACK] Seq=67          |TCP: 1478→23 [ACK] Seq=67 Ack=1348 Win=261568 Len=0
|         |(1478)   ------------------>  (23)     |

000986: Sep  3 17:50:48.063 EST10: Reserved port 0 in Transport Port Agent for TCP IP type 0
000987: Sep  3 17:50:48.063 EST10: TCP0: state was LISTEN -> SYNRCVD [23 -> 172.31.100.1 (1478)]
000988: Sep  3 17:50:48.063 EST10: TCP: tcb 872EFA7C connection to 172.31.100.1 :1478, peer MSS 1300, MSS is 516
000989: Sep  3 17:50:48.063 EST10: TCP: Selective ack is disabled from the CLI
000990: Sep  3 17:50:48.063 EST10: TCP: sending SYN, seq 1462041879, ack 45725672
000991: Sep  3 17:50:48.063 EST10: TCP0: Connection to 172.31.100.1:1478, advertising MSS 536
000992: Sep  3 17:50:48.119 EST10: TCP0: state was SYNRCVD -> ESTAB [23 -> 172.31.100.1(1478)]
000993: Sep  3 17:50:48.119 EST10: TCB872EFA7C setting property TCP_TOS (11) 854C2BAC
000994: Sep  3 17:50:48.119 EST10: TCB872EFA7C setting property TCP_RTRANSTMO (36) 89B49060
000995: Sep  3 17:50:48.119 EST10: TCB872EFA7C setting property TCP_GIVEUP (41) 89B49064
000996: Sep  3 17:50:48.119 EST10: TCB872EFA7C setting property TCP_KEEPALIVE (17) 89B49038
000997: Sep  3 17:50:48.119 EST10: TCP: Setting Keepalive interval and retries to 60 and 4
000998: Sep  3 17:50:51.135 EST10: TCP9: ACK timeout timer expired
000999: Sep  3 17:50:51.447 EST10: TCP9: ACK timeout timer expired
001000: Sep  3 17:50:51.815 EST10: TCP9: ACK timeout timer expired
001001: Sep  3 17:50:52.095 EST10: TCP9: ACK timeout timer expired
001002: Sep  3 17:50:52.415 EST10: TCP9: ACK timeout timer expired
001003: Sep  3 17:50:52.735 EST10: TCP9: ACK timeout timer expired
001004: Sep  3 17:50:53.055 EST10: TCP9: ACK timeout timer expired
001005: Sep  3 17:50:53.375 EST10: TCP9: ACK timeout timer expired

id=20085 trace_id=12839 msg="vd-hitn_krpy received a packet(proto=6, 172.31.100.1:1177->172.31.1.2:23) from port1.3039."
id=20085 trace_id=12839 msg="find a route: gw-172.16.255.15 via krpy-bne1-gre-8"
id=20085 trace_id=12839 msg="no session matched"
id=20085 trace_id=12840 msg="vd-hitn_krpy received a packet(proto=6, 172.31.100.1:1177->172.31.1.2:23) from port1.3039."
id=20085 trace_id=12840 msg="find a route: gw-172.16.255.15 via krpy-bne1-gre-8"
id=20085 trace_id=12840 msg="no session matched"
id=20085 trace_id=12841 msg="vd-hitn_krpy received a packet(proto=6, 172.31.100.1:1177->172.31.1.2:23) from port1.3039."
id=20085 trace_id=12841 msg="find a route: gw-172.16.255.15 via krpy-bne1-gre-8"
id=20085 trace_id=12841 msg="no session matched"
id=20085 trace_id=12842 msg="vd-hitn_krpy received a packet(proto=6, 172.31.100.13:1177->172.31.1.2:23) from port1.3039."
id=20085 trace_id=12842 msg="find a route: gw-172.16.255.15 via krpy-bne1-gre-8"
id=20085 trace_id=12842 msg="no session matched"

Leave a Reply

Your email address will not be published. Required fields are marked *

*