Issue
Veritas Volume Manager(VxVM) is reporting duplicate (ghost) disk entries for Veritas disk access (da) name "c1t1d0" following a recent LUN (bootdisk) replacement attempt.
Sample output
# vxdisk -eo alldgs list
<snippet>
VxVM vxconfigd ERROR V-5-1-13992 open failed on path: /dev/vx/rdmp/c1t1d0s2
VxVM vxconfigd ERROR V-5-1-13992 open failed on path: /dev/vx/rdmp/c1t1d0s2
VxVM vxconfigd ERROR V-5-1-13992 open failed on path: /dev/vx/rdmp/c1t1d0s2
DEVICE TYPE DISK GROUP STATUS OS_NATIVE_NAME ATTR
c1t0d0s2 auto:sliced rootdisk rootdg online c1t0d0s2 -
c1t1d0s2 auto - - error c1t1d0s2 -
c1t1d0s2 auto:none - - online invalid c1t1d0s2 -
c1t1d0s2 auto - - error c1t1d0s2 -
.
.
<snippet>
NOTE: Veritas disk access (da) name "c1t1d0" is reported three times by VxVM.
Environment
Solaris Sparc.
Cause
In this instance, the Solaris cfgadm (leadville) stack is reporting the access points for the removed LUN via two different WWN references.
# cfgadm -alo show_FCP_dev
Ap_Id Type Receptacle Occupant Condition
c1 fc-private connected configured unknown
c1::500000e0139f7621,0 disk connected configured failing <<<< reference to c1t1d0
c1::500000e013f04cb1,0 disk connected configured unknown
c1::5000cca004ac2c85,0 disk connected configured unknown <<<< reference to c1t1d0
The bootdisk (c1t1d0) was removed incorrectly from the server, hence the stale disk entry recorded in the /etc/vx/disk.info file.
# cat /etc/vx/disk.info
FUJITSU%5FMAX3147FCSUN146G%5FDISKS%5F500000E013F04CB0 c1t0d0 0x4680000 0x1 c1t0d0 Disk DISKS
HITACHI%5FOPEN-V%20%20%20%20%20%20-SUN%5F05C13%5F177E c4t60d3 0x4680010 0x1 c4t60d3 TagmaStore-USP 05C13
HITACHI%5FOPEN-V%20%20%20%20%20%20-SUN%5F05C13%5F177C c4t60d1 0x4680018 0x1 c4t60d1 TagmaStore-USP 05C13
HITACHI%5FOPEN-V%20%20%20%20%20%20-SUN%5F05C13%5F177B c4t60d0 0x4680020 0x1 c4t60d0 TagmaStore-USP 05C13
HITACHI%5FOPEN-V%20%20%20%20%20%20-SUN%5F05C13%5F177D c4t60d2 0x4680028 0x1 c4t60d2 TagmaStore-USP 05C13
HITACHI%5FOPEN-V%20%20%20%20%20%20-SUN%5F05C13%5F177F c4t60d4 0x4680030 0x1 c4t60d4 TagmaStore-USP 05C13
INVALID c1t1d0 0x4680008 0x1 c1t1d0 Disk DISKS <<<<< stale disk entry
#
Upon further investigation, c1t1d0 refers to WWN's "5000cca004ac2c85" and "500000e0139f7621".
# ls -la /dev/rdsk/c1t1d0*
<snippet>
lrwxrwxrwx 1 root root 74 Jan 26 05:32 /dev/rdsk/c1t1d0s0 -> ../../devices/pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@w5000cca004ac2c85,0:a,raw
lrwxrwxrwx 1 root root 74 Jan 26 05:33 /dev/rdsk/c1t1d0s1 -> ../../devices/pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@w5000cca004ac2c85,0:b,raw
lrwxrwxrwx 1 root root 74 Jan 26 05:33 /dev/rdsk/c1t1d0s2 -> ../../devices/pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@w5000cca004ac2c85,0:c,raw
lrwxrwxrwx 1 root root 74 Jan 26 05:33 /dev/rdsk/c1t1d0s3 -> ../../devices/pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@w5000cca004ac2c85,0:d,raw
lrwxrwxrwx 1 root root 74 Jan 26 05:33 /dev/rdsk/c1t1d0s4 -> ../../devices/pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@w5000cca004ac2c85,0:e,raw
lrwxrwxrwx 1 root root 74 Jan 26 05:33 /dev/rdsk/c1t1d0s5 -> ../../devices/pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@w5000cca004ac2c85,0:f,raw
.
<snippet>
# ls -la /dev/rdsk/* | grep -i 7621
<snippet>
lrwxrwxrwx 1 root root 74 Jan 26 06:15 /dev/rdsk/c1t1d0s0 -> ../../devices/pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@w500000e0139f7621,0:a,raw
lrwxrwxrwx 1 root root 74 Jan 26 06:15 /dev/rdsk/c1t1d0s1 -> ../../devices/pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@w500000e0139f7621,0:b,raw
lrwxrwxrwx 1 root root 74 Jan 26 06:15 /dev/rdsk/c1t1d0s2 -> ../../devices/pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@w500000e0139f7621,0:c,raw
lrwxrwxrwx 1 root root 74 Jan 26 06:15 /dev/rdsk/c1t1d0s3 -> ../../devices/pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@w500000e0139f7621,0:d,raw
lrwxrwxrwx 1 root root 74 Jan 26 06:15 /dev/rdsk/c1t1d0s4 -> ../../devices/pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@w500000e0139f7621,0:e,raw
lrwxrwxrwx 1 root root 74 Jan 26 06:15 /dev/rdsk/c1t1d0s5 -> ../../devices/pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@w500000e0139f7621,0:f,raw
.
.
<snippet>
Solution
1.] Remove the Veritas disk access (da) name duplicated by "vxdisk list".
# vxdisk list
<snippet>
DEVICE TYPE DISK GROUP STATUS OS_NATIVE_NAME ATTR
c1t0d0s2 auto:sliced rootdisk rootdg online c1t0d0s2 -
c1t1d0s2 auto - - error c1t1d0s2 -
c1t1d0s2 auto:none - - online invalid c1t1d0s2 -
c1t1d0s2 auto - - error c1t1d0s2 -
<snippet>
In this instance, there are three entries for the same Veritas disk access name "c1t1d0", therefore, the "vxdisk rm <da-name> " needs to be issued three times:
# vxdisk rm c1t1d0
# vxdisk rm c1t1d0
# vxdisk rm c1t1d0
2.] Confirm using the "vxdisk list" command, that VxVM no longer reports the problematic Veritas disk access name.
# vxdisk list
<snippet>
DEVICE TYPE DISK GROUP STATUS OS_NATIVE_NAME ATTR
c1t0d0s2 auto:sliced rootdisk rootdg online c1t0d0s2 -
<snippet>
3.] Using the Solaris luxadm interface, mark all the paths to the disk as offline.
# luxadm -e offline /dev/rdsk/c#t#d#s2
NOTE: In this instance, there is only a single path via both WWN's to c1t1d0.
4.] Clean-up the stale OS device handles using Solaris command "devfsadm -Cvc disk".
Sample output
# devfsadm -Cvc disk
devfsadm[5935]: verbose: removing file: /dev/dsk/c1t1d0s0
devfsadm[5935]: verbose: removing file: /dev/dsk/c1t1d0s1
devfsadm[5935]: verbose: removing file: /dev/dsk/c1t1d0s2
devfsadm[5935]: verbose: removing file: /dev/dsk/c1t1d0s3
devfsadm[5935]: verbose: removing file: /dev/dsk/c1t1d0s4
devfsadm[5935]: verbose: removing file: /dev/dsk/c1t1d0s5
devfsadm[5935]: verbose: removing file: /dev/dsk/c1t1d0s6
devfsadm[5935]: verbose: removing file: /dev/dsk/c1t1d0s7
devfsadm[5935]: verbose: removing file: /dev/rdsk/c1t1d0s0
devfsadm[5935]: verbose: removing file: /dev/rdsk/c1t1d0s1
devfsadm[5935]: verbose: removing file: /dev/rdsk/c1t1d0s2
devfsadm[5935]: verbose: removing file: /dev/rdsk/c1t1d0s3
devfsadm[5935]: verbose: removing file: /dev/rdsk/c1t1d0s4
devfsadm[5935]: verbose: removing file: /dev/rdsk/c1t1d0s5
devfsadm[5935]: verbose: removing file: /dev/rdsk/c1t1d0s6
devfsadm[5935]: verbose: removing file: /dev/rdsk/c1t1d0s7
5.] In this instance, the Solaris cfgadm (leadville) stack was reporting the multiple access points for the removed LUN (c1t1d0).
Refresh the cfgadm output, by running "cfgadm -alo show_FCP_dev".
# cfgadm -alo show_FCP_dev
Ap_Id Type Receptacle Occupant Condition
c1 fc-private connected configured unknown
c1::500000e013f04cb1,0 disk connected configured unknown
c1::5000cca004ac2c85,0 disk connected configured unusable <<<< reference to c1t1d0
NOTE: The previous access point for "c1::500000e0139f7621,0" is no longer listed by the cfgadm interface:
c1::500000e0139f7621,0 disk connected configured failing <<<< did reference c1t1d0
6.] Force Fibre Channel SAN disk rescan using the Solaris "luxadm" interface.
a.] Establish the port reference to be rescanned.
# luxadm -e port
/devices/pci@9,600000/SUNW,qlc@2/fp@0,0:devctl CONNECTED
b.] Force the rescan using the port reference.
# luxadm -e forcelip /devices/pci@9,600000/SUNW,qlc@2/fp@0,0:devctl
7.] Refresh the cfgadm (leadville) stack
# cfgadm -al
Ap_Id Type Receptacle Occupant Condition
c0 scsi-bus connected configured unknown
c0::dsk/c0t0d0 CD-ROM connected configured unknown
c1 fc-private connected configured unknown
c1::500000e013f04cb1 disk connected configured unknown
c1::5000cca004ac2c85 disk connected configured unknown <<<<<<< Updated from unusable to "unknown"
usb0/1 unknown empty unconfigured ok
usb0/2 unknown empty unconfigured ok
usb0/3 unknown empty unconfigured ok
usb0/4 unknown empty unconfigured ok
8.] Refresh both VxVM and the /etc/vx/disk.info file contents.
# vxdisk scandisks
# vxddladm assign names ( Introduced in 5.0 MP3 onwards)
9.] Execute "vxdisk scandisks" an additional time to ensure the duplicate (ghost) disk entries do not reappear.
# vxdisk scandisks
# vxdisk list
<snippet>
DEVICE TYPE DISK GROUP STATUS OS_NATIVE_NAME ATTR
c1t0d0s2 auto:sliced rootdisk rootdg online c1t0d0s2 -
c1t1d0s2 auto:none - - online invalid c1t1d0s2 -
<snippet>
NOTE: Veritas disk access name "c1t1d0s2" contains a single line entry now, therefore it can be safely initialized for VxVM use.
Process complete
Veritas Volume Manager(VxVM) is reporting duplicate (ghost) disk entries for Veritas disk access (da) name "c1t1d0" following a recent LUN (bootdisk) replacement attempt.
Sample output
# vxdisk -eo alldgs list
<snippet>
VxVM vxconfigd ERROR V-5-1-13992 open failed on path: /dev/vx/rdmp/c1t1d0s2
VxVM vxconfigd ERROR V-5-1-13992 open failed on path: /dev/vx/rdmp/c1t1d0s2
VxVM vxconfigd ERROR V-5-1-13992 open failed on path: /dev/vx/rdmp/c1t1d0s2
DEVICE TYPE DISK GROUP STATUS OS_NATIVE_NAME ATTR
c1t0d0s2 auto:sliced rootdisk rootdg online c1t0d0s2 -
c1t1d0s2 auto - - error c1t1d0s2 -
c1t1d0s2 auto:none - - online invalid c1t1d0s2 -
c1t1d0s2 auto - - error c1t1d0s2 -
.
.
<snippet>
NOTE: Veritas disk access (da) name "c1t1d0" is reported three times by VxVM.
Environment
Solaris Sparc.
Cause
In this instance, the Solaris cfgadm (leadville) stack is reporting the access points for the removed LUN via two different WWN references.
# cfgadm -alo show_FCP_dev
Ap_Id Type Receptacle Occupant Condition
c1 fc-private connected configured unknown
c1::500000e0139f7621,0 disk connected configured failing <<<< reference to c1t1d0
c1::500000e013f04cb1,0 disk connected configured unknown
c1::5000cca004ac2c85,0 disk connected configured unknown <<<< reference to c1t1d0
The bootdisk (c1t1d0) was removed incorrectly from the server, hence the stale disk entry recorded in the /etc/vx/disk.info file.
# cat /etc/vx/disk.info
FUJITSU%5FMAX3147FCSUN146G%5FDISKS%5F500000E013F04CB0 c1t0d0 0x4680000 0x1 c1t0d0 Disk DISKS
HITACHI%5FOPEN-V%20%20%20%20%20%20-SUN%5F05C13%5F177E c4t60d3 0x4680010 0x1 c4t60d3 TagmaStore-USP 05C13
HITACHI%5FOPEN-V%20%20%20%20%20%20-SUN%5F05C13%5F177C c4t60d1 0x4680018 0x1 c4t60d1 TagmaStore-USP 05C13
HITACHI%5FOPEN-V%20%20%20%20%20%20-SUN%5F05C13%5F177B c4t60d0 0x4680020 0x1 c4t60d0 TagmaStore-USP 05C13
HITACHI%5FOPEN-V%20%20%20%20%20%20-SUN%5F05C13%5F177D c4t60d2 0x4680028 0x1 c4t60d2 TagmaStore-USP 05C13
HITACHI%5FOPEN-V%20%20%20%20%20%20-SUN%5F05C13%5F177F c4t60d4 0x4680030 0x1 c4t60d4 TagmaStore-USP 05C13
INVALID c1t1d0 0x4680008 0x1 c1t1d0 Disk DISKS <<<<< stale disk entry
#
Upon further investigation, c1t1d0 refers to WWN's "5000cca004ac2c85" and "500000e0139f7621".
# ls -la /dev/rdsk/c1t1d0*
<snippet>
lrwxrwxrwx 1 root root 74 Jan 26 05:32 /dev/rdsk/c1t1d0s0 -> ../../devices/pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@w5000cca004ac2c85,0:a,raw
lrwxrwxrwx 1 root root 74 Jan 26 05:33 /dev/rdsk/c1t1d0s1 -> ../../devices/pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@w5000cca004ac2c85,0:b,raw
lrwxrwxrwx 1 root root 74 Jan 26 05:33 /dev/rdsk/c1t1d0s2 -> ../../devices/pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@w5000cca004ac2c85,0:c,raw
lrwxrwxrwx 1 root root 74 Jan 26 05:33 /dev/rdsk/c1t1d0s3 -> ../../devices/pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@w5000cca004ac2c85,0:d,raw
lrwxrwxrwx 1 root root 74 Jan 26 05:33 /dev/rdsk/c1t1d0s4 -> ../../devices/pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@w5000cca004ac2c85,0:e,raw
lrwxrwxrwx 1 root root 74 Jan 26 05:33 /dev/rdsk/c1t1d0s5 -> ../../devices/pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@w5000cca004ac2c85,0:f,raw
.
<snippet>
# ls -la /dev/rdsk/* | grep -i 7621
<snippet>
lrwxrwxrwx 1 root root 74 Jan 26 06:15 /dev/rdsk/c1t1d0s0 -> ../../devices/pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@w500000e0139f7621,0:a,raw
lrwxrwxrwx 1 root root 74 Jan 26 06:15 /dev/rdsk/c1t1d0s1 -> ../../devices/pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@w500000e0139f7621,0:b,raw
lrwxrwxrwx 1 root root 74 Jan 26 06:15 /dev/rdsk/c1t1d0s2 -> ../../devices/pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@w500000e0139f7621,0:c,raw
lrwxrwxrwx 1 root root 74 Jan 26 06:15 /dev/rdsk/c1t1d0s3 -> ../../devices/pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@w500000e0139f7621,0:d,raw
lrwxrwxrwx 1 root root 74 Jan 26 06:15 /dev/rdsk/c1t1d0s4 -> ../../devices/pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@w500000e0139f7621,0:e,raw
lrwxrwxrwx 1 root root 74 Jan 26 06:15 /dev/rdsk/c1t1d0s5 -> ../../devices/pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@w500000e0139f7621,0:f,raw
.
.
<snippet>
Solution
1.] Remove the Veritas disk access (da) name duplicated by "vxdisk list".
# vxdisk list
<snippet>
DEVICE TYPE DISK GROUP STATUS OS_NATIVE_NAME ATTR
c1t0d0s2 auto:sliced rootdisk rootdg online c1t0d0s2 -
c1t1d0s2 auto - - error c1t1d0s2 -
c1t1d0s2 auto:none - - online invalid c1t1d0s2 -
c1t1d0s2 auto - - error c1t1d0s2 -
<snippet>
In this instance, there are three entries for the same Veritas disk access name "c1t1d0", therefore, the "vxdisk rm <da-name> " needs to be issued three times:
# vxdisk rm c1t1d0
# vxdisk rm c1t1d0
# vxdisk rm c1t1d0
2.] Confirm using the "vxdisk list" command, that VxVM no longer reports the problematic Veritas disk access name.
# vxdisk list
<snippet>
DEVICE TYPE DISK GROUP STATUS OS_NATIVE_NAME ATTR
c1t0d0s2 auto:sliced rootdisk rootdg online c1t0d0s2 -
<snippet>
3.] Using the Solaris luxadm interface, mark all the paths to the disk as offline.
# luxadm -e offline /dev/rdsk/c#t#d#s2
NOTE: In this instance, there is only a single path via both WWN's to c1t1d0.
4.] Clean-up the stale OS device handles using Solaris command "devfsadm -Cvc disk".
Sample output
# devfsadm -Cvc disk
devfsadm[5935]: verbose: removing file: /dev/dsk/c1t1d0s0
devfsadm[5935]: verbose: removing file: /dev/dsk/c1t1d0s1
devfsadm[5935]: verbose: removing file: /dev/dsk/c1t1d0s2
devfsadm[5935]: verbose: removing file: /dev/dsk/c1t1d0s3
devfsadm[5935]: verbose: removing file: /dev/dsk/c1t1d0s4
devfsadm[5935]: verbose: removing file: /dev/dsk/c1t1d0s5
devfsadm[5935]: verbose: removing file: /dev/dsk/c1t1d0s6
devfsadm[5935]: verbose: removing file: /dev/dsk/c1t1d0s7
devfsadm[5935]: verbose: removing file: /dev/rdsk/c1t1d0s0
devfsadm[5935]: verbose: removing file: /dev/rdsk/c1t1d0s1
devfsadm[5935]: verbose: removing file: /dev/rdsk/c1t1d0s2
devfsadm[5935]: verbose: removing file: /dev/rdsk/c1t1d0s3
devfsadm[5935]: verbose: removing file: /dev/rdsk/c1t1d0s4
devfsadm[5935]: verbose: removing file: /dev/rdsk/c1t1d0s5
devfsadm[5935]: verbose: removing file: /dev/rdsk/c1t1d0s6
devfsadm[5935]: verbose: removing file: /dev/rdsk/c1t1d0s7
5.] In this instance, the Solaris cfgadm (leadville) stack was reporting the multiple access points for the removed LUN (c1t1d0).
Refresh the cfgadm output, by running "cfgadm -alo show_FCP_dev".
# cfgadm -alo show_FCP_dev
Ap_Id Type Receptacle Occupant Condition
c1 fc-private connected configured unknown
c1::500000e013f04cb1,0 disk connected configured unknown
c1::5000cca004ac2c85,0 disk connected configured unusable <<<< reference to c1t1d0
NOTE: The previous access point for "c1::500000e0139f7621,0" is no longer listed by the cfgadm interface:
c1::500000e0139f7621,0 disk connected configured failing <<<< did reference c1t1d0
6.] Force Fibre Channel SAN disk rescan using the Solaris "luxadm" interface.
a.] Establish the port reference to be rescanned.
# luxadm -e port
/devices/pci@9,600000/SUNW,qlc@2/fp@0,0:devctl CONNECTED
b.] Force the rescan using the port reference.
# luxadm -e forcelip /devices/pci@9,600000/SUNW,qlc@2/fp@0,0:devctl
7.] Refresh the cfgadm (leadville) stack
# cfgadm -al
Ap_Id Type Receptacle Occupant Condition
c0 scsi-bus connected configured unknown
c0::dsk/c0t0d0 CD-ROM connected configured unknown
c1 fc-private connected configured unknown
c1::500000e013f04cb1 disk connected configured unknown
c1::5000cca004ac2c85 disk connected configured unknown <<<<<<< Updated from unusable to "unknown"
usb0/1 unknown empty unconfigured ok
usb0/2 unknown empty unconfigured ok
usb0/3 unknown empty unconfigured ok
usb0/4 unknown empty unconfigured ok
8.] Refresh both VxVM and the /etc/vx/disk.info file contents.
# vxdisk scandisks
# vxddladm assign names ( Introduced in 5.0 MP3 onwards)
9.] Execute "vxdisk scandisks" an additional time to ensure the duplicate (ghost) disk entries do not reappear.
# vxdisk scandisks
# vxdisk list
<snippet>
DEVICE TYPE DISK GROUP STATUS OS_NATIVE_NAME ATTR
c1t0d0s2 auto:sliced rootdisk rootdg online c1t0d0s2 -
c1t1d0s2 auto:none - - online invalid c1t1d0s2 -
<snippet>
NOTE: Veritas disk access name "c1t1d0s2" contains a single line entry now, therefore it can be safely initialized for VxVM use.
Process complete
No comments:
Post a Comment