Fault Manager is part of self-healing functionality that provides fault isolation and component restart, in this case hardware component
(SMF will take care of software components).
Make sure that you run the service and have required packages.
# pkginfo | grep fmd
system SUNWfmd Fault Management Daemon and Utilities
system SUNWfmdr Fault Management Daemon and Utilities (Root)
#
# fmadm config
MODULE VERSION STATUS DESCRIPTION
cpumem-diagnosis 1.7 active CPU/Memory Diagnosis
cpumem-retire 1.1 active CPU/Memory Retire Agent
disk-transport 1.0 active Disk Transport Agent
eft 1.16 active eft diagnosis engine
etm 1.2 active FMA Event Transport Module
ext-event-transport 0.1 active External FM event transport
fabric-xlate 1.0 active Fabric Ereport Translater
fmd-self-diagnosis 1.0 active Fault Manager Self-Diagnosis
generic-mem 1.0 active SPARC-Generic-Memory Diagnosis
io-retire 1.0 active I/O Retire Agent
pri-monitor 1.0 active PRI Update Monitor
snmp-trapgen 1.0 active SNMP Trap Generation Agent
sp-monitor 1.0 active Service Processor Monitor
sysevent-transport 1.0 active SysEvent Transport Agent
syslog-msgs 1.0 active Syslog Messaging Agent
zfs-diagnosis 1.0 active ZFS Diagnosis Engine
zfs-retire 1.0 active ZFS Retire Agent
#
To check the fmd error or any hardware failure on system
# fmdump -v
# fmadm faulty
# fmadm faulty -r
# fmadm faulty -a
For example :
# fmdump
TIME UUID SUNW-MSG-ID
May 02 09:40:33.3713 f01fb317-2383-caad-9883-83ea390a5bba PCIEX-8000-KP
May 02 10:15:27.9764 f01fb317-2383-caad-9883-83ea390a5bba FMD-8000-58 Updated
May 02 10:15:47.2144 4123dedb-9bbb-c58d-e88a-f12d0b28fa44 PCIEX-8000-KP
#
# fmadm faulty -r
dev:////pci@0,0/pci8086,340a@3/pci111d,806c@0 degraded
dev:////pci@0,0/pci8086,340a@3 degraded
#
# fmadm faulty -r
mem:///unum=MB/CMP0/CH3:R0/D1/J2201 degraded
#
To rotate the fmd errors use below script ( run at least two times) .
#GZ:
fmdump
#Clean up old entries:
for i in `/usr/sbin/fmdump|awk '{print $4}'`
do
/usr/sbin/fmadm repair $i
done
sleep 2
/usr/sbin/fmadm rotate fltlog
/usr/sbin/fmadm rotate errlog
If the errors are still exist in # fmadm faulty –a then we need to clear the cache of fmd using the below steps.
Clear ereports and resource cache:
# cd /var/fm/fmd
# rm e* f* c*/eft/* r*/*
Clearing out FMA files with no reboot needed:
svcadm disable -s svc:/system/fmd:default
cd /var/fm/fmd
find /var/fm/fmd -type f -exec ls {} \;
find /var/fm/fmd -type f -exec rm {} \;
svcadm enable svc:/system/fmd:default
And monitor the system for few hrs (or one day) if the errors are came again then we need raise the oracle SR to get it fix.
To resolved Correctable DIMM errors.
If we received memory DIMM alert and all memory is show as enabled on the system controller and the FMA event was logged on the motherboard.
# prtconf -v | grep -i Mem
Memory size: 16256 Megabytes
memory (driver not attached)
virtual-memory (driver not attached)
#
I think it would be best if the FMA events are cleared and repaired on operating system first then we can go ahead for PSRs.
From Solaris level :Display the fault in the OS
# fmadm faulty
Repair all FMA events from the output from above
# fmadm repair <UUID>
Example:
# fmadm repair a5e24c9f-65c5-ee14-8148-a1ff8b35dc5a
# fmadm repair 07b01c83-c720-481c-cb9a-85774abb30f5
Or use below script to clear the fmd errors:
#GZ:
fmdump
#Clean up old entries:
for i in `/usr/sbin/fmdump|awk '{print $4}'`
do
/usr/sbin/fmadm repair $i
done
sleep 2
/usr/sbin/fmadm rotate fltlog
/usr/sbin/fmadm rotate errlog
Check to see that faults are cleared
# fmdump -v
# fmadm faulty
# fmadm faulty -r
# fmadm faulty -a
Clear error reports and resource cache with following commands:
cd /var/fm/fmd
rm e* f* c*/eft/* r*/*
Reset the fmd serd modules:
fmadm reset cpumem-diagnosis
fmadm reset cpumem-retire
fmadm reset eft
fmadm reset io-retire
Check with the below command to confirm that the DIMM is not faulty and wait for few days and again check for confirmation.
sc> showfaults -v
(SMF will take care of software components).
Make sure that you run the service and have required packages.
# pkginfo | grep fmd
system SUNWfmd Fault Management Daemon and Utilities
system SUNWfmdr Fault Management Daemon and Utilities (Root)
#
# fmadm config
MODULE VERSION STATUS DESCRIPTION
cpumem-diagnosis 1.7 active CPU/Memory Diagnosis
cpumem-retire 1.1 active CPU/Memory Retire Agent
disk-transport 1.0 active Disk Transport Agent
eft 1.16 active eft diagnosis engine
etm 1.2 active FMA Event Transport Module
ext-event-transport 0.1 active External FM event transport
fabric-xlate 1.0 active Fabric Ereport Translater
fmd-self-diagnosis 1.0 active Fault Manager Self-Diagnosis
generic-mem 1.0 active SPARC-Generic-Memory Diagnosis
io-retire 1.0 active I/O Retire Agent
pri-monitor 1.0 active PRI Update Monitor
snmp-trapgen 1.0 active SNMP Trap Generation Agent
sp-monitor 1.0 active Service Processor Monitor
sysevent-transport 1.0 active SysEvent Transport Agent
syslog-msgs 1.0 active Syslog Messaging Agent
zfs-diagnosis 1.0 active ZFS Diagnosis Engine
zfs-retire 1.0 active ZFS Retire Agent
#
To check the fmd error or any hardware failure on system
# fmdump -v
# fmadm faulty
# fmadm faulty -r
# fmadm faulty -a
For example :
# fmdump
TIME UUID SUNW-MSG-ID
May 02 09:40:33.3713 f01fb317-2383-caad-9883-83ea390a5bba PCIEX-8000-KP
May 02 10:15:27.9764 f01fb317-2383-caad-9883-83ea390a5bba FMD-8000-58 Updated
May 02 10:15:47.2144 4123dedb-9bbb-c58d-e88a-f12d0b28fa44 PCIEX-8000-KP
#
# fmadm faulty -r
dev:////pci@0,0/pci8086,340a@3/pci111d,806c@0 degraded
dev:////pci@0,0/pci8086,340a@3 degraded
#
# fmadm faulty -r
mem:///unum=MB/CMP0/CH3:R0/D1/J2201 degraded
#
To rotate the fmd errors use below script ( run at least two times) .
#GZ:
fmdump
#Clean up old entries:
for i in `/usr/sbin/fmdump|awk '{print $4}'`
do
/usr/sbin/fmadm repair $i
done
sleep 2
/usr/sbin/fmadm rotate fltlog
/usr/sbin/fmadm rotate errlog
If the errors are still exist in # fmadm faulty –a then we need to clear the cache of fmd using the below steps.
Clear ereports and resource cache:
# cd /var/fm/fmd
# rm e* f* c*/eft/* r*/*
Clearing out FMA files with no reboot needed:
svcadm disable -s svc:/system/fmd:default
cd /var/fm/fmd
find /var/fm/fmd -type f -exec ls {} \;
find /var/fm/fmd -type f -exec rm {} \;
svcadm enable svc:/system/fmd:default
And monitor the system for few hrs (or one day) if the errors are came again then we need raise the oracle SR to get it fix.
To resolved Correctable DIMM errors.
If we received memory DIMM alert and all memory is show as enabled on the system controller and the FMA event was logged on the motherboard.
# prtconf -v | grep -i Mem
Memory size: 16256 Megabytes
memory (driver not attached)
virtual-memory (driver not attached)
#
I think it would be best if the FMA events are cleared and repaired on operating system first then we can go ahead for PSRs.
From Solaris level :Display the fault in the OS
# fmadm faulty
Repair all FMA events from the output from above
# fmadm repair <UUID>
Example:
# fmadm repair a5e24c9f-65c5-ee14-8148-a1ff8b35dc5a
# fmadm repair 07b01c83-c720-481c-cb9a-85774abb30f5
Or use below script to clear the fmd errors:
#GZ:
fmdump
#Clean up old entries:
for i in `/usr/sbin/fmdump|awk '{print $4}'`
do
/usr/sbin/fmadm repair $i
done
sleep 2
/usr/sbin/fmadm rotate fltlog
/usr/sbin/fmadm rotate errlog
Check to see that faults are cleared
# fmdump -v
# fmadm faulty
# fmadm faulty -r
# fmadm faulty -a
Clear error reports and resource cache with following commands:
cd /var/fm/fmd
rm e* f* c*/eft/* r*/*
Reset the fmd serd modules:
fmadm reset cpumem-diagnosis
fmadm reset cpumem-retire
fmadm reset eft
fmadm reset io-retire
Check with the below command to confirm that the DIMM is not faulty and wait for few days and again check for confirmation.
sc> showfaults -v
