Category Archives: VNX
EMC VNX Series Storages
dskMon: FS /dev/mapper/emc_vg_lun_5-emc_lv_nas_var_emcsupport mounted on /nbsnas/var/emcsupport filling up
Aaa before sleep one more article
We got such warning on Unisphere
dskMon: FS /dev/mapper/emc_vg_lun_5-emc_lv_nas_var_emcsupport mounted on /nbsnas/var/emcsupport filling up
On that partition some logs file stored 😀 , need to be deleted if its full
552M 472M 52M 91% /nbsnas/var/emcsupport
[nasadmin@ASLANVNX01 ~]$ cd /nbsnas/var/emcsupport
[nasadmin@ASLANVNX01 emcsupport]$ ls -al
drwxrwxr-x 3 nasadmin nasadmin 4096 Aug 22 13:02 .
drwxrwxr-x 8 nasadmin nasadmin 1024 Aug 23 22:00 ..
drwx—— 2 root root 16384 Apr 22 2011 lost+found
-rw-rw-r– 1 nasadmin nasadmin 45309242 Jul 1 03:05 support_materials_CKM00105000413.120701_0257.zip
-rw-rw-r– 1 nasadmin nasadmin 43483801 Jul 8 03:05 support_materials_CKM00105000413.120708_0257.zip
-rw-rw-r– 1 nasadmin nasadmin 46748284 Jul 11 06:39 support_materials_CKM00105000413.120711_0631.zip
-rw-rw-r– 1 nasadmin nasadmin 43792585 Jul 15 03:05 support_materials_CKM00105000413.120715_0256.zip
-rw-rw-r– 1 nasadmin nasadmin 44872969 Jul 22 03:05 support_materials_CKM00105000413.120722_0257.zip
-rw-rw-r– 1 nasadmin nasadmin 46031689 Jul 23 06:09 support_materials_CKM00105000413.120723_0600.zip
-rw-rw-r– 1 nasadmin nasadmin 46466604 Jul 23 07:23 support_materials_CKM00105000413.120723_0715.zip
-rw-rw-r– 1 nasadmin nasadmin 47372360 Jul 29 03:08 support_materials_CKM00105000413.120729_0257.zip
attantion , rm command will not ask to you (y/n) , delete , thats all
Today we faced deregistered initiator hosts are still on Host List of VNX, no way to remove such records because “Remove” option was Grey
Again, no way to solve such issue without restart service 🙂 very funny
Do step by step
Close EMC Unisphere windows first
Login with credentials
Click on Restart Management Server button
wait and control the page again its accessible or not, then switch SPB, do same thing for SPB
then try to access EMC Unisphere , you will see that all entries which need to be removed before are gone 😀
After use more then 3 or 4 different storage vender really you can understand whats happening 😀
EMC is leader storage vender and selling more storage from other vendors but this not means its perfect, if you are thinking to purchase think twice …
(I’m writing this article 10 March 2012 and first update 28 March 2012, if they improve something pls do not get my words 🙂 AND PLS DO NOT FORGET I’M WORKING ON ISP AND INTERNET SERVICE PROVIDERS ARE DIFFERENT THEN ENTERPRISE CUSTOMERS )
Update 3 Sep 2012, new update …… new subject is : Think third times before purchase VNX
- Block device software and File Server software are should be upgraded together, if you upgrade Block software you MUST upgrade FLARE too ! Means if Block device has a issue your have a double upgrade and 5-6 minute downtime for FileServices (Good thing EMC guys are making the upgrade maybe you can like this but its taking too much time, i can upgrade my IBM SVC Cluster in a 30 mins, thats it)
- Think again again and again when you set raid level, because later can not change it why ?! because….. i don’t know, ask EMC experts, they always have something to tell about you
- If you want performance, then forget to use -Disk Pool- , because when you have problem and send nar files(Kind of debug log) to EMC Support they will say ‘The pool configuration is just used for the effective storage management, it wouldn’t benefit the storage performance.’
- If you will not use Disk Pool then its means ready to categories your workload and purchase more more disk then what you think before
- Try to do everything with professional service because at the end they will say ‘Suggest to engage the EMC Professional Service to reconfigure the storage.‘
- Try to get bigger controllers then what you expecting and try to purchase max cache because at the end you will start to deal with cache watermarks, if cache is low and full more quickly then host I/O will be stop for a while and this will couse delay.
Do not believe SSD with Fast Cache, do not think that put more more SSD drive as a Fast Cache because you will wait that system will use closer %90 percent of Fast Cache SSD layer but in our environment its stay on %40-50 means for us no need it and you will get such answer from EMC guys ‘SSD drives are more efficient especially if your applications have small, random and read weighted I/Os.’SORRY 😦 but this words are wrong, pls check well your analyzer to determine how many cache miss you have, if its a lot then you should increase the number of SSD drive in Fast Cache, also recommendation of EMC is %5 of total storage space determine Fast Cache size.
- DeDup – NO They do not have it
- Pls do not mix the compression with dedup because you should only compress something if it will not used any more or maybe 10 years later you need it because every access time EMC should unzip it before use it this means you push the CPU , EMC do not have ASIC like 3PAR pls do not tire it 🙂
- DO NOT, but pls DO DO NOT create a thin LUN in production , EMC say 40-50% performance degradation could be and that are preparing an update of firmware on June or closer to improve thin performance , but our contact from EMC say, do not use thin in production, its also eat memory from SP Caches 🙂
- if you have a money try to purchase more disks or if you still want to create a pool purchase SSD and put it with FC and SATA pull because EMC tearing is not working well , rule is very easy if if performance layer is empty use it , very easy decision but this is not what i want ! also chunks should be 1GB to move between layers , how much data should be stay on tier zero only 20 MB of it ?!
- Update 3 Sep 2012 – Be careful when you upgrade the system , if you have heavy loaded system think more then twice, last week we did an upgrade to latest version and SPA did not back because of LCC upgrade and more then half business day we worked with single SP , its back again and working well right now, very nice 🙂
The world is going on another way, pls keep looking my posts, i will tell about that.
APAR IC80749:ERROR DESCRIPTION:On controllers (for example EMC Clariion) that have fibrechannel logins as both target and initiator the active/inactivestate may be incorrectly displayed.The reason is that both the target logins (which are being used)and the initiator logins (which are not) may update the status.Hence the status may be shown as “inactive” becuase theinitiator logins are not active, while the target ones are infact being used.CMVC 142997 is tracking this.LOCAL FIX:No local fix or workaround.
VNX data movers are not failover if such conditions are not occurred (You can expect in many situation but not ! )
- Failure of both internal Ethernets …
- Data Mover internal power failure
- Manually failover with the server_standby command
- Data Mover panic
- Lost of SCSI connection
- Lost of Fiber connection
- Memory errors
Otherwise do not wait something like me , right clicked on data mover server_2 and shut it down and wait for server_3 will take off but it was a dream 🙂
EMC and File Services …..
When i heard that multiple data movers can run on blade and if you need a additional performance and capacity ( because of data mover are limited to serve a space ) you can add move. I’m really impressed
Today i’m thinking again that how EMC developers can’t or why do not implement Layer3 redundancy in to data mover or that environment ….
Also more funny thing is this movers can not failover if Control Station is not available 😦
You should care about L3 yourself , you do not have to chance to made a mistake, you can not set wrong vlan id to switch port (of course you did not but we are human and can make a mistake)? also you should always watch you switch uplinks which data mover are connected that switch 🙂 Where is the real redundancy ? Okay, there is Fail Safe configuration but does it understand your switch uplink problem ?
Redhat Cluster Suite and NFS/CIFS , Netapp , GlusterFS , Object Storages and all other vendors almost have such option but EMC not …
Think Again ..
Yesterday we saw that disk failures on VNX side are effecting IBM SVC.
Yes very interesting but it is. You can find out related explanation below (pls check number 4 because until that step you believe the its a controller issue why disk). I believe that this is really hard issue for storage admins which try to fix both side, first fix the VNX second fix the IBM SVC. The same issue is not available on DS series storages.
Error ID = 10011 : Remote Port excluded for a specific Managed Disk
Error Code = 1220 : Remote FC port excluded
THIS IS AN EXTERNAL ISSUE REPORTED BY THE SVC. NOT AN SVC FAULT
Possible Cause: A remote fibre-channel port has been excluded.
SAN Volume Controller 2145-8G4: N/A
SAN Volume Controller 2145-8F4: N/A
SAN Volume Controller 2145-8F2: N/A
SAN Volume Controller 2145-4F2: N/A
Other: Enclosure/controller fault (50%); Fibre-channel network fabric (50%)
1. View the error log. Note the MDisk ID associated with the error code.
2. From the MDisk, determine the failing disk controller ID.
3. Refer to the service documentation for the disk controller and the fibre-channel network to resolve the reported problem.
4. After the disk drive is repaired, start a cluster discovery operation to recover the excluded fibre-channel port by rescanning the fibre-channel network.
5. To restore MDisk online status, include the managed disk that you noted in step 1.
6. Check the status of the disk controller. If all disk controllers show a ″good″ status, mark the error that you have just repaired, ″fixed.″
7. If all disk controllers do not show a good status, contact your support center to resolve the problem with the disk controller.
8. Go to repair verification MAP.
Sometimes its hard to adapt VNX if you are coming from Netapp.
VNX do not support MCS type iscsi connection, only allowed MPIO reason is bellow from EMC Global Service.
To answer your question MCS Is not supported on EMC.
You cannot aggregate the iSCSI ports and they should be on separate subnets for redundancy.
MCS was designed to help with failover on the iSCSI level.
This is not handled by Failover software on the OS , either MPIO or Powerpath.
MC/S was designed at time, when most OS’ didn’t have standard OS level multipath. Instead, each vendor had its own implementation, which created huge interoperability problems. So, one of the goals of MC/S was to address this issue and standardize the multipath area in a single standard. But nowadays almost all OS’s has OS level multipath implemented using standard SCSI facilities, hence this purpose of MC/S isn’t valid anymore.
You can find the differences between MCS and MPIO
Pls watch the video, you will discover
Too many windows storage server is working all around the world
Sorry for video quality because I was in shock ….
Right now i and Gökhan is working on to introduce iSCSI I/O module to VNX storage. We expected that without stop i/o we can plug and go the iscsi card but not ! We have to reboot both i/o controller together to initialize cards. Somebody can say what you expecting what happened if you have IBM or Netapp storages, pls see below
Check the link http://www.emc.com/storage/vnx/vnx-series.htm , its said that AUTOMATED, what is automated ? I’m still up and waiting the steps over , its take closer a hour
Why recognized I/O modules need to be restart SPs for initialize, this is cloud world and cloud storages need to be more flexible, i believe the programmers and logic can handle it
Is it really hard to created manually powered bus and recognize and initialize the i/o module like kudzu in Linux