dskMon[19086]: FS /dev/mapper/emc_vg_lun_5-emc_lv_nas_var_emcsupport mounted on /nbsnas/var/emcsupport filling up

Aaa before sleep one more article

We got such warning on Unisphere

dskMon[19086]: FS /dev/mapper/emc_vg_lun_5-emc_lv_nas_var_emcsupport mounted on /nbsnas/var/emcsupport filling up

On that partition some logs file stored 😀 , need to be deleted if its full

/dev/mapper/emc_vg_lun_5-emc_lv_nas_var_emcsupport

552M  472M   52M  91% /nbsnas/var/emcsupport

[nasadmin@ASLANVNX01 ~]$ cd /nbsnas/var/emcsupport

[nasadmin@ASLANVNX01 emcsupport]$ ls -al

total 466265

drwxrwxr-x 3 nasadmin nasadmin     4096 Aug 22 13:02 .

drwxrwxr-x 8 nasadmin nasadmin     1024 Aug 23 22:00 ..

drwx—— 2 root     root        16384 Apr 22  2011 lost+found

-rw-rw-r– 1 nasadmin nasadmin 45309242 Jul  1 03:05 support_materials_CKM00105000413.120701_0257.zip

-rw-rw-r– 1 nasadmin nasadmin 43483801 Jul  8 03:05 support_materials_CKM00105000413.120708_0257.zip

-rw-rw-r– 1 nasadmin nasadmin 46748284 Jul 11 06:39 support_materials_CKM00105000413.120711_0631.zip

-rw-rw-r– 1 nasadmin nasadmin 43792585 Jul 15 03:05 support_materials_CKM00105000413.120715_0256.zip

-rw-rw-r– 1 nasadmin nasadmin 44872969 Jul 22 03:05 support_materials_CKM00105000413.120722_0257.zip

-rw-rw-r– 1 nasadmin nasadmin 46031689 Jul 23 06:09 support_materials_CKM00105000413.120723_0600.zip

-rw-rw-r– 1 nasadmin nasadmin 46466604 Jul 23 07:23 support_materials_CKM00105000413.120723_0715.zip

-rw-rw-r– 1 nasadmin nasadmin 47372360 Jul 29 03:08 support_materials_CKM00105000413.120729_0257.zip

Connect “Control Station” via ssh , go to under  /nbsnas/var/emcsupport folder and delete the oldest file 😀 like below

rm support_materials_CKM00105000413.120701_0257.zip

attantion , rm command will not ask to you (y/n) , delete , thats all

VM

 

Can not delete host entry on VNX via EMC Unisphere

Today we faced deregistered initiator hosts are still on Host List of VNX, no way to remove such records because “Remove” option was Grey

Again, no way to solve such issue without restart service 🙂 very funny

Do step by step

Close EMC Unisphere windows first
Go https://SPA/setup/
Login with credentials
Click on Restart Management Server button
Click Yes
and Submit

wait and control the page again its accessible or not, then switch SPB, do same thing for SPB

then try to access EMC Unisphere , you will see that all entries which need to be removed before are gone 😀

Perfect
VM

Think twice before purchase VNX

After use more then 3 or 4 different storage vender really you can understand whats happening 😀

EMC is leader storage vender and selling more storage from other vendors but this not means its perfect, if you are thinking to purchase think twice …

(I’m writing this article 10 March 2012 and first update 28 March 2012, if they improve something pls do not get my words 🙂 AND PLS DO NOT FORGET I’M WORKING ON ISP AND INTERNET SERVICE PROVIDERS ARE DIFFERENT THEN ENTERPRISE CUSTOMERS )

Update 3 Sep 2012, new update …… new subject is : Think third times before purchase VNX

Why !

• Block device software and File Server software are should be upgraded together, if you upgrade Block software you MUST upgrade FLARE too ! Means if Block device has a issue your have a double upgrade and 5-6 minute downtime for FileServices (Good thing EMC guys are making the upgrade maybe you can like this but its taking too much time, i can upgrade my IBM SVC Cluster in a 30 mins, thats it)
• Think again again and again when you set raid level, because later can not change it why ?! because….. i don’t know, ask EMC experts, they always have something to tell about you
• If you want performance, then forget to use -Disk Pool- , because when you have problem and send nar files(Kind of debug log) to EMC Support they will say ‘The pool configuration is just used for the effective storage management, it wouldn’t benefit the storage performance.’ 
• If you will not use Disk Pool then its means ready to categories your workload and purchase more more disk then what you think before
• Try to do everything with professional service because at the end they will say ‘Suggest to engage the EMC Professional Service to reconfigure the storage.‘
• Try to get bigger controllers then what you expecting and try to purchase max cache because at the end you will start to deal with cache watermarks, if cache is low and full more quickly then host I/O will be stop for a while and this will couse delay.
• Do not believe SSD with Fast Cache, do not think that put more more SSD drive as a Fast Cache because you will wait that system will use closer %90 percent of Fast Cache SSD layer but in our environment its stay on %40-50 means for us no need it and you will get such answer from EMC guys ‘SSD drives are more efficient especially if your applications have small, random  and read weighted I/Os.’ SORRY 😦 but this words are wrong, pls check well your analyzer to determine how many cache miss you have, if its a lot then you should increase the number of SSD drive in Fast Cache, also recommendation of EMC is %5 of total storage space determine Fast Cache size.
• DeDup – NO They do not have it
• Pls do not mix the compression with dedup because you should only compress something if it will not used any more or maybe 10 years later you need it because every access time EMC should unzip it before use it this means you push the CPU , EMC do not have ASIC like 3PAR pls do not tire it 🙂
• DO NOT, but pls DO DO NOT  create a thin LUN in production , EMC say 40-50% performance degradation could be and that are preparing an update of firmware on June or closer to improve thin performance , but our contact from EMC say, do not use thin in production, its also eat memory from SP Caches 🙂
• if you have a money try to purchase more disks or if you still want to create a pool purchase SSD and put it with FC and SATA pull because EMC tearing is not working well , rule is very easy if if performance layer is empty use it , very easy decision but this is not what i want ! also chunks should be 1GB to move between layers , how much data should be stay on tier zero only 20 MB of it ?!
• Update 3 Sep 2012 – Be careful when you upgrade the system , if you have heavy loaded system think more then twice, last week we did an upgrade to latest version and SPA did not back because of LCC upgrade and more then half business day we worked with single SP ,  its back again and working well right now, very nice 🙂

The world is going on another way, pls keep looking my posts, i will tell about that.

VM

lsfabric out active/inactive issue with EMC VNX Series

Hi ,

We found that SVC do not understand EMC VNX fabric active or inactive, after long time IBM developers find out the issue and they will fix it.

VM

APAR IC80749:

ERROR DESCRIPTION:

On controllers (for example EMC Clariion) that have fibre

channel logins as both target and initiator the active/inactive

state may be incorrectly displayed.

The reason is that both the target logins (which are being used)

and the initiator logins (which are not) may update the status.

Hence the status may be shown as “inactive” becuase the

initiator logins are not active, while the target ones are in

fact being used.

CMVC 142997 is tracking this.

LOCAL FIX:

No local fix or workaround.

Don’t Restart Or Shutdown Because Data Mover Do Not Failover Like That

VNX data movers are not failover if such conditions are not occurred (You can expect in many situation but not ! )

• Failure of both internal Ethernets …
• Data Mover internal power failure
• Manually failover with the server_standby command
• Data Mover panic
• Lost of SCSI connection
• Lost of Fiber connection
• Memory errors

Otherwise do not wait something like me , right clicked on data mover server_2 and shut it down and wait for server_3 will take off but it was a dream 🙂

VM

Don’t Wait L3 Redundancy From You VNX Data Movers

EMC and File Services …..

When i heard that multiple data movers can run on blade and if you need a additional performance and capacity ( because of data mover are limited to serve a space ) you can add move. I’m really impressed

Today i’m thinking again that how EMC developers can’t or why do not implement Layer3 redundancy in to data mover or that environment ….
Also more funny thing is this movers can not failover if Control Station is not available 😦

You should care about L3 yourself , you do not have to chance to made a mistake, you can not set wrong vlan id to switch port (of course you did not but we are human and can make a mistake)? also you should always watch you switch uplinks which data mover are connected that switch 🙂 Where is the real redundancy ? Okay, there is Fail Safe configuration but does it understand your switch uplink problem ?

Redhat Cluster Suite and NFS/CIFS , Netapp , GlusterFS , Object Storages and all other vendors almost have such option but EMC not …

Think Again ..
VM

VNX Disk Fails Couse IBM SVC Cluster Error

Yesterday we saw that disk failures on VNX side are effecting IBM SVC.

Yes very interesting but it is. You can find out  related explanation below (pls check number 4 because until that step you believe the its a controller issue why disk). I believe that this is really hard issue for storage admins which try to fix both side, first fix the VNX second fix the IBM SVC. The same issue is not available on DS series storages.

Error ID = 10011 : Remote Port excluded for a specific Managed Disk
Error Code = 1220 : Remote FC port excluded

THIS IS AN EXTERNAL ISSUE REPORTED BY THE SVC. NOT AN SVC FAULT

Possible Cause: A remote fibre-channel port has been excluded.

SAN Volume Controller 2145-8G4: N/A

SAN Volume Controller 2145-8F4: N/A

SAN Volume Controller 2145-8F2: N/A

SAN Volume Controller 2145-4F2: N/A

Other: Enclosure/controller fault (50%); Fibre-channel network fabric (50%)
Action:
1. View the error log. Note the MDisk ID associated with the error code.
2. From the MDisk, determine the failing disk controller ID.
3. Refer to the service documentation for the disk controller and the fibre-channel network to resolve the reported problem.
4. After the disk drive is repaired, start a cluster discovery operation to recover the excluded fibre-channel port by rescanning the fibre-channel network.
5. To restore MDisk online status, include the managed disk that you noted in step 1.
6. Check the status of the disk controller. If all disk controllers show a ″good″ status, mark the error that you have just repaired, ″fixed.″
7. If all disk controllers do not show a good status, contact your support center to resolve the problem with the disk controller.
8. Go to repair verification MAP.

VM

IBM SVC really support VNX and VNX working together with IBM SVC

After a month later, VNX 5500 and IBM SVC is working together how i want instead of little status issue which is not have a difficulty, but i will investigate

The story is starting when i create a ticket to IBM for to be sure about the support and L2 answer was IBM do not support VNX 5500 😦
L2 reason is cx5 and related FLARE version is not on http://www-01.ibm.com/support/docview.wss?rs=591&uid=ssg1S1003797
but web site said that we support “EMC CLARiiON CX-series models” which VNX is not a so different things then CX-5 and Celerra, its only bundle solution http://www-03.ibm.com/systems/storage/software/virtualization/svc/specifications.html