C H A P T E R 2 Troubleshooting Switch System Issues; Working With A Firmware Upgrade Failure; Working With Recoverable Restarts - Cisco MDS 9000 Series Troubleshooting Manual

Troubleshooting System Restarts
S e n d c o m m e n t s t o m d s f e e d b a c k - d o c @ c i s c o . c o m .
Most system restarts generate a Call Home event, but the condition causing a restart may become so
severe that a Call Home event is not generated. Be sure that you configure the Call Home feature
properly, follow up on any initial messages regarding system restarts, and fix the problem before it
becomes so severe. For information about configuring Call Home, refer to the Cisco MDS 9000 Family
Configuration Guide or the Cisco MDS 9000 Family Fabric Manager User Guide.

Working with a Firmware Upgrade Failure

When you perform a hitless upgrade (either by using the install all command, or by using Cisco Fabric
Manager), and the upgrade fails with no error message displayed, please call technical support to
determine the cause of the failure. Do not attempt to reboot the switch.

Working with Recoverable Restarts

Every process restart generates a Syslog message and a Call Home event. Even if the event is not service
affecting you should identify and resolve the condition immediately because future occurrences could
cause service interruption.
To respond to a recoverable system restart, follow these steps:
Step 1 Enter the following command to check the Syslog file to see which process restarted and why it restarted.
switch# sh log logfile | include error
For information about the meaning of each message, refer to the Cisco MDS 9000 Family System
Messages Guide
The system output looks like the following:
Sep 10 23:31:31 dot-6 % LOG_SYSMGR-3-SERVICE_TERMINATED: Service "sensor" (PID 704) has
finished with error code SYSMGR_EXITCODE_SY.
switch# show logging logfile | include fail
Jan 27 04:08:42 88 %LOG_DAEMON-3-SYSTEM_MSG: bind() fd 4, family 2, port 123, ad
dr 0.0.0.0, in_classd=0 flags=1 fails: Address already in use
Jan 27 04:08:42 88 %LOG_DAEMON-3-SYSTEM_MSG: bind() fd 4, family 2, port 123, ad
dr 127.0.0.1, in_classd=0 flags=0 fails: Address already in use
Jan 27 04:08:42 88 %LOG_DAEMON-3-SYSTEM_MSG: bind() fd 4, family 2, port 123, ad
dr 127.1.1.1, in_classd=0 flags=1 fails: Address already in use
Jan 27 04:08:42 88 %LOG_DAEMON-3-SYSTEM_MSG: bind() fd 4, family 2, port 123, ad
dr 172.22.93.88, in_classd=0 flags=1 fails: Address already in use
Jan 27 23:18:59 88 % LOG_PORT-5-IF_DOWN: Interface fc1/13 is down (Link failure
or not-connected)
Jan 27 23:18:59 88 % LOG_PORT-5-IF_DOWN: Interface fc1/14 is down (Link failure
or not-connected)
Jan 28 00:55:12 88 % LOG_PORT-5-IF_DOWN: Interface fc1/1 is down (Link failure o
r not-connected)
Jan 28 00:58:06 88 % LOG_ZONE-2-ZS_MERGE_FAILED: Zone merge failure, Isolating p
ort fc1/1 (VSAN 100)
Jan 28 00:58:44 88 % LOG_ZONE-2-ZS_MERGE_FAILED: Zone merge failure, Isolating p
ort fc1/1 (VSAN 100)
Jan 28 03:26:38 88 % LOG_ZONE-2-ZS_MERGE_FAILED: Zone merge failure, Isolating p
ort fc1/1 (VSAN 100)
Jan 29 19:01:34 88 % LOG_PORT-5-IF_DOWN: Interface fc1/1 is down (Link failure o
r not-connected)
switch#
Step 2 Enter the following command to identify the processes that are running and the status of each process.
Cisco MDS 9000 Family Troubleshooting Guide
2-2
Chapter 2 Troubleshooting Switch System Issues
OL-5183-02, Cisco MDS SAN-OS Release 1.3