All-Paths-Down (APD) Dead paths on esxi host

This article describes how to troubleshooting the issue.

I recently encountered the issue, few paths are dead after power outage of site.

Rescanned the esxi hosts and no luck and find the below screenshot.

Command line troubleshooting of issue:-

> esxcli storage core adapter rescan --all

> esxcli storage filesystem rescan

> esxcli storage core adapter list

> esxcli storage vmfs extent list

> esxcli storage filesystem list

> esxcli storage vmfs extent list | grep -i datastorename

> esxcli storage core device list -d naa.6000dfk9870000000000000000089

> esxcli storage san iscsi list

> esxcli storage nmp device list -d naa.6000dfk9870000000000000000089

> less /var/run/log/vmkernel.log

" 2023-01-06T14:17:11.001Z cpu10:2098182)NMP: nmp_ThrottleLogForDevice:3867: Cmd 0xa3 (0x45d9b6ded908, 0) to dev "c" on path "vmhba64:C0:T7:L5" Failed:

2023-01-06T14:17:11.001Z cpu10:2098182)NMP: nmp_ThrottleLogForDevice:3875: H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x3f 0xe. Act:NONE. cmdId.initiator=0x45398ab9bbc8 CmdSN 0x0

2023-01-06T14:17:11.001Z cpu10:2097741)ScsiDevice: 3792: Handle REPORTED LUNS CHANGED DATA unit attention

2023-01-11T06:04:36.008Z cpu15:2097294)ScsiDeviceIO: 4176: Cmd(0x45b99bf0a088) 0x28, CmdSN 0x2849a27 from world 6241205 to dev "naa.6000d7100d117e0000000000000000

07" failed H:0x1 D:0x0 P:0x0"

2023-01-11T06:04:36.008Z cpu28:6241205)Fil6: 4092: 'datastorename': Fil6 file IO (<FD c0 r1>) : Device is permanently unavailable

> # esxcli storage filesystem list

> # lsof | grep -i datastore

> # lsof | grep -i 616ff9b2-c4014970-f1fb-bc97e1050890

> # esxcli storage core adapter rescan --all