Symptom
The combination of error messages below is the
result of a non-C:D application trying to connect to the C:D UNIX port.
This could be a health check from a system monitor facility, a hardware
load balancer, or something like an FTP Client being mistakenly used to
connect to C:D UNIX.
e.g. A FTP client trying to connect to C:D port 1364.
The TCP session will be established ( SYN,
SYN+ACK, ACK ) but then the session will hang. This is because the FTP
Client is waiting as it expects the next packet to be sent by the FTP
Server. The Server side is actually C:D UNIX which is waiting as it is
expecting a FMH68 from the Client side. The session ends when the FTP
Client times out. The message "XIPT016I -TCP lost the connection. System
error is Broken pipe." is issued when the timeout occurs.
A similar sequence of messages are issued for
outbound C:D Secure+ sessions when the remote system has no C:D sessions
available. The difference is that there will be TCQ status change
messages in the C:D UNIX stats before the XIPT016I and after the
XSMG600I. See article NFX9956 for further details.
C:D UNIX stats when non-C:D application attempts to connect to to C:D UNIX port.
E XIPT 01/06/2011 16:31:44 ndm_error_set(): rc=16 fdbk=0 msgid=XIPT016I stext=TCP lost the connection. System error is Broken pipe.
E XSMG 01/06/2011 16:31:44 ndm_error_set(): rc=8 fdbk=0 msgid=XSMG621I stext=A receive from the remote node operation failed.
E XSMG 01/06/2011 16:31:44 ndm_error_set(): rc=16 fdbk=16 msgid=XSMG631I stext=SMGR failed to receive an FMH. Diag=XIPT016I.
E XSMG 01/06/2011 16:31:44 ndm_error_set(): rc=16 fdbk=16 msgid=XSMG600I stext=SMGR (snode) didn't receive FMH68 during session start.
E CXIT 01/06/2011 16:31:44 Snode SMGR exited. Pid=1200220. Exitcode=0.
E XSMG 01/06/2011 16:31:44 ndm_error_set(): rc=8 fdbk=0 msgid=XSMG621I stext=A receive from the remote node operation failed.
E XSMG 01/06/2011 16:31:44 ndm_error_set(): rc=16 fdbk=16 msgid=XSMG631I stext=SMGR failed to receive an FMH. Diag=XIPT016I.
E XSMG 01/06/2011 16:31:44 ndm_error_set(): rc=16 fdbk=16 msgid=XSMG600I stext=SMGR (snode) didn't receive FMH68 during session start.
E CXIT 01/06/2011 16:31:44 Snode SMGR exited. Pid=1200220. Exitcode=0.
Messages when outbound session using Secure+ is
rejected by remote C:D because no sessions are available (e.g. remote
C:D zOS rejects session because either MAXSESSIONS, MAXSECONDARY, or
SESS.SNODE.MAX exceeded.)
E QCEX 12/15/2010 13:11:54 TCQ queue change from WAIT to EXEC, status PE.
E XIPT 12/15/2010 13:11:54 ndm_error_set(): rc=16 fdbk=0 msgid=XIPT016I stext=TCP lost the connection. System error is Broken pipe.
E XSMG 12/15/2010 13:11:54 ndm_error_set(): rc=8 fdbk=0 msgid=XSMG621I stext=A receive from the remote node operation failed.
E XSMG 12/15/2010 13:11:54 ndm_error_set(): rc=16 fdbk=16 msgid=XSMG631I stext=SMGR failed to receive an FMH. Diag=XIPT016I.
E XSMG 12/15/2010 13:11:55 ndm_error_set(): rc=16 fdbk=16 msgid=XSMG600I stext=SMGR (snode) didn't receive FMH68 during session start.
E QCHO 12/15/2010 13:11:55 TCQ queue change from EXEC to HOLD, status HE.
E CXIT 12/15/2010 13:11:55 Pnode SMGR exited. Pid=446576. Exitcode=0.
E XIPT 12/15/2010 13:11:54 ndm_error_set(): rc=16 fdbk=0 msgid=XIPT016I stext=TCP lost the connection. System error is Broken pipe.
E XSMG 12/15/2010 13:11:54 ndm_error_set(): rc=8 fdbk=0 msgid=XSMG621I stext=A receive from the remote node operation failed.
E XSMG 12/15/2010 13:11:54 ndm_error_set(): rc=16 fdbk=16 msgid=XSMG631I stext=SMGR failed to receive an FMH. Diag=XIPT016I.
E XSMG 12/15/2010 13:11:55 ndm_error_set(): rc=16 fdbk=16 msgid=XSMG600I stext=SMGR (snode) didn't receive FMH68 during session start.
E QCHO 12/15/2010 13:11:55 TCQ queue change from EXEC to HOLD, status HE.
E CXIT 12/15/2010 13:11:55 Pnode SMGR exited. Pid=446576. Exitcode=0.
Resolving the problem
Ask your network team to identify the remote
non-C:D application that is connecting to the C:D port. Stop the remote
application from connecting to the C:D UNIX port..
Note: tcpdump command could be used to identify
the remote IP address when the problem is occuring (Not suitable if the
problem is intermitant)
tcpdump –n –l –s 100 port 1364
Dump all inbound &
outbound sessions using port 1364. This will output to the screen
tcpdump –n –l –s 100 port 1364 -w output.file.txt This will output to a file.
tcpdump –n –l –s 100 port 1364 -r output.file.txt
> output.file.formated.txt This will format the the contents of
the tcpdump output file and write it into a new file.
Example tcpdump output:
13:32:10.172133 IP 90.80.70.60.1704 > 10.20.30.40.1364: S 469734307:46973430>
13:32:10.172190 IP 10.20.30.40.1364 > 90.80.70.60.1704: S 577235407:57723540>
13:32:10.402660 IP 90.80.70.60.1704 > 10.20.30.40.1364: . ack 1 win 1260
13:32:31.171828 IP 90.80.70.60.1704 > 10.20.30.40.1364: F 1:1(0) ack 1 win 65 <<< XIPT016I will be issued in C:D stats at this point. Remote aplication has timed out and issued a FIN.</strong></span>
190.80.70.60.1704: . ack 2 win 65535
13:32:31.172492 IP 10.20.30.40.1364 > 90.80.70.60.1704: F 1:1(0) ack 2 win 65
13:32:31.403852 IP 90.80.70.60.1704 > 10.20.30.40.1364: . ack 2 win 65535
( S = SYN , F= FIN )
13:32:10.172190 IP 10.20.30.40.1364 > 90.80.70.60.1704: S 577235407:57723540>
13:32:10.402660 IP 90.80.70.60.1704 > 10.20.30.40.1364: . ack 1 win 1260
13:32:31.171828 IP 90.80.70.60.1704 > 10.20.30.40.1364: F 1:1(0) ack 1 win 65 <<< XIPT016I will be issued in C:D stats at this point. Remote aplication has timed out and issued a FIN.</strong></span>
190.80.70.60.1704: . ack 2 win 65535
13:32:31.172492 IP 10.20.30.40.1364 > 90.80.70.60.1704: F 1:1(0) ack 2 win 65
13:32:31.403852 IP 90.80.70.60.1704 > 10.20.30.40.1364: . ack 2 win 65535
( S = SYN , F= FIN )
No comments:
Post a Comment