APD (All Path Down) Task Hangings and Solutions

We faced APD (All path down) issue and tried rescan storage/Datastore/LUN from GUI , command line , storageRM nothing help us just only restart the ESXi host helped.
You can see the logs and article about APD, PDL below

========================================================

vmkernel.all:2013-02-06T09:40:02.033Z cpu3:8732)WARNING: NMP: nmpDeviceAttemptFailover:658:Retry world

 

failover device “naa.600507680187053498000000000000c3” – failed to issue command due to Not found (APD),

 

try again…

vmkernel.all:2013-02-06T09:40:02.033Z cpu3:8732)WARNING: NMP: nmpDeviceAttemptFailover:658:Retry world

 

failover device “naa.600507680187053498000000000000c7” – failed to issue command due to Not found (APD),

 

try again…

vmkernel.all:2013-02-06T09:40:02.033Z cpu1:8732)WARNING: NMP: nmpDeviceAttemptFailover:658:Retry world

 

failover device “naa.600507680187053498000000000000c6” – failed to issue command due to Not found (APD),

 

try again…

vmkernel.all:2013-02-06T09:40:02.033Z cpu1:8732)WARNING: NMP: nmpDeviceAttemptFailover:658:Retry world

 

failover device “naa.600507680187053498000000000000c5” – failed to issue command due to Not found (APD),

 

try again…

vmkernel.all:2013-02-06T09:40:02.033Z cpu2:14962)WARNING: NMP: nmpDeviceAttemptFailover:658:Retry world

 

failover device “naa.600507680187053498000000000000c9” – failed to issue command due to Not found (APD),

 

try again…

vmkernel.all:2013-02-06T09:40:02.033Z cpu1:8732)WARNING: NMP: nmpDeviceAttemptFailover:658:Retry world

 

failover device “naa.600507680187053498000000000000c8” – failed to issue command due to Not found (APD),

 

try again…

vmkernel.all:2013-02-06T09:40:03.032Z cpu1:8732)WARNING: NMP: nmpDeviceAttemptFailover:658:Retry world

 

failover device “naa.600507680187053498000000000000c0” – failed to issue command due to Not found (APD),

 

try again…

vmkernel.all:2013-02-06T09:40:03.033Z cpu2:14962)WARNING: NMP: nmpDeviceAttemptFailover:658:Retry world

 

failover device “naa.600507680187053498000000000000c4” – failed to issue command due to Not found (APD),

 

try again…

vmkernel.all:2013-02-06T09:40:03.033Z cpu2:14962)WARNING: NMP: nmpDeviceAttemptFailover:658:Retry world

 

failover device “naa.600507680187053498000000000000c6” – failed to issue command due to Not found (APD),

 

try again…

vmkernel.all:2013-02-06T09:40:03.033Z cpu2:14962)WARNING: NMP: nmpDeviceAttemptFailover:658:Retry world

 

failover device “naa.600507680187053498000000000000c9” – failed to issue command due to Not found (APD),

 

try again…

vmkernel.all:2013-02-06T09:40:07.032Z cpu5:14962)WARNING: NMP: nmpDeviceAttemptFailover:658:Retry world

 

failover device “naa.6005076801870534980000000000011e” – failed to issue command due to Not found (APD),

 

try again…

vmkernel.all:2013-02-06T09:40:07.032Z cpu5:14962)WARNING: NMP: nmpDeviceAttemptFailover:658:Retry world

 

failover device “naa.600507680187053498000000000000c5” – failed to issue command due to Not found (APD),

 

try again…

vmkernel.all:2013-02-06T09:40:07.032Z cpu4:8732)WARNING: NMP: nmpDeviceAttemptFailover:658:Retry world

 

failover device “naa.600507680187053498000000000000c2” – failed to issue command due to Not found (APD),

 

try again…

vmkernel.all:2013-02-06T09:40:08.033Z cpu1:8732)WARNING: NMP: nmpDeviceAttemptFailover:658:Retry world

 

failover device “naa.600507680187053498000000000000c2” – failed to issue command due to Not found (APD),

 

try again…

vmkernel.all:2013-02-06T09:40:08.033Z cpu1:8732)WARNING: NMP: nmpDeviceAttemptFailover:658:Retry world

 

failover device “naa.600507680187053498000000000000c5” – failed to issue command due to Not found (APD),

 

try again…

vmkernel.all:2013-02-06T09:40:08.033Z cpu1:8732)WARNING: NMP: nmpDeviceAttemptFailover:658:Retry world

 

failover device “naa.6005076801870534980000000000011e” – failed to issue command due to Not found (APD),

 

try again…

================================================

Related article : http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2004684

Also we faced task hanging on and locking issues

A task like restarting VM or kind of things is hang then you need to restart vpxd service

A  task like vMotion is hang then you need to restart vCenter service

Also locking issues, when try to start VM, it can give an error like cannot read vmdk file or locking issue then for our case we used standart linux lsof command , very interesting to up th VM we moved VM on different ESXi nodes but it didn’t start after investigation we saw that on one of the ESXi node a files of VM looks like open and this cause locking issue after kill it everything was OK

VM

 

Posted on 08/02/2013, in VMware and tagged , , , , . Bookmark the permalink. Leave a comment.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: