SAN Volume Controller / Storwize / Flash System version 8.4.2 allows you to non-disruptively migrate a LUN between array controller clusters. It’s set up like remote copy, except you can map the remote copy to the same host at the same time. The remote copy becomes non-preferred paths for the same vdisk and vdisk ID. Then you can switch who is primary. Then you can remove the old copy.
Here is someone who did a demo video: https://www.youtube.com/watch?v=NpcOoshkm4w
This may need to happen if you remove/re-add NPIV mapping, if SAN topology changes, or if you have removed maps, ports, cables, or entire LUNs.
1. Change the configuration on the array to remove the extra ports, etc.
2. Physically remove any fibre cables if needed.
3. IPL the server.
4. Run the STRSST command., then Press ENTER
5. Option 1, Start a service tool, then Press ENTER
6. Option 4, Display/Alter/Dump, then Press ENTER
7. Option 1, Display/Alter storage, then Press ENTER
8. Option 2, Licensed Internal Code (LIC) data, then Press ENTER
9. Option 14, Advanced analysis (scroll down to see this option), then Press ENTER
a. Scroll down and type 1 by MULTIPATHRESETTER, then press ENTER
b. Options, -RESETMP -ALL -CONFIRM, then press ENTER
DISPLAY/ALTER/DUMP Running macro: MULTIPATHRESETTER -CONFIRM -ALL Reset the paths for Multiple Connections ******************************************************************** ***CONFIRM RESET MULTIPATH UNIT PATHSTO NUMBER CURRENTLY ENLISTED*** ******************************************************************** This service function should be run only under the direction of the IBM Hardware Service Support. You have selected to reset the number of paths on a multipath unit to equal the number of paths that have currently enlisted. Attempting to reset path for resource name: DMP004 Attempting to reset path for resource name: DMP075 ... *** Your request completed successfully *** The number of paths connected to your multipath unit have been reset to match the number of paths that are currently enlisted.
REF: https://www.ibm.com/support/pages/reducing-or-removing-paths-multipath-lun
REF: https://www.ibm.com/support/pages/san-disks-missing-paths
When you replace a drive on one of these, mdisk arrays do not auto-rebuild.
If the GUI fix procedures go away, or never show up, or whatever causes the replacement drive to not get included as a new drive in the mdisk, you can do this manually.
First, look for the candidate or spare drive you want to use.
lsdrive | grep -v member
Then, make sure that drive ID is a candidate:
chdrive -use candidate 72
Then, find the missing member:
lsarraymember mdisk1 | grep -v exact
Then, set the new drive to use that missing member ID:
charraymember -member 31 -newdrive 72 mdisk1
You can watch the progress of the rebuild:
lsarraymemberprogress mdisk1
IBM released “Data Reduction Pools” for their SAN Volume Controller, which includes Storwize, FlashSystem etc products back in early 2018. The internal flag for it is “deduplication”, but it can be used for thin, compressed, or deduplicated volumes. You can put thick volumes in them too, but that is more of a “block off this space” thing.
DRP is a “log structured device”, meaning it is random read, but always sequential write. When you enable DRP, it inserts into the high cache (front-end cache). Because of this, it adds 3-4ms of latency right off the bat. Some of the higher end controllers can reduce that to 2-3ms, but it never goes away. Also, this affects ALL storage pools (mdisk groups), not just the ones built with DRP enabled. IBM product engineering says this is not a defect. It’s just they way it is, because you have to twiddle the data on the CPU.
In late 2018, I found one of their code defects, which has been resolved; however, that defect highlighted a troubling design choice. DRP is a whole-cluster thing, not a whole I/O group thing, nor a whole pool thing. That means, if you trigger a defect that kills DRP for one pool, it will kill DRP for every pool. If you’re in a hyperswap or stretched cluster, and something happens (future defect maybe) to take one site’s DRP offline while the nodes are still live, then that could take out BOTH sites.
This year, I found out that their sizing partner, “Intellimagic”, does not understand DRP. When a “Disk Magic” report comes out, it can recommend DRP. It cannot tell how much cache affects things, so it just recommends maximum cache. It cannot tell how much DRP affects things, so it cannot recommend against it for latency sensitive workloads. The IBM technical sales team is trying to work around this by simply recommending the next model up when DRP is to be used.
However, that is not enough. DRP garbage collection is VERY heavy handed. There is no way to throttle it. It’s a log structured array, so when you re-write a block on a LUN, the old block gets invalidated, then the old chunk gets read, then new data is packed in, then it’s written to the end of the array. This is sequential for each major section of the pool, and it has additional latency added by the code. We found that a normal system of 25% writes would be suppressed by 80% in DRP. Latency during normal operations would fluctuate from 6 to 18, and that’s to the client. Any major I/O operations on the client (big rewrites, or big block freeing), or any vdisk (LUN) deletions would cause latency to fluctuate from 16-85ms (completely unusable).
Once DRP is enabled, it affects the whole system. You cannot turn it off for a pool. You have to delete that pool. You cannot migrate a vdisk out of nor into a DRP, you have to mirror to a new pool, then delete or split the old copy. Funny thing is, that’s just what a migrate does at all but the lowest levels. It’s been 2 years, that really should have been folded into the standard commands by now.
Once you have everything split, and all of the hosts are running on the new, non-DRP enabled pool, that is not enough. All of the performance problems will still be there. It is only once you delete the old pool that performance returns to normal. Since vdisk deletions take several hours each, and running more than 4 at a time causes severe, deadly latency, it’s best to just verify nothing is in use and force-delete the whole DRP-enabled pool. Risky, but otherwise, you could spend weeks deleting the old LUNs by hand due to how slow they clear.
The sales materials, technical whitebooks, and the gritty redbooks all recommend DRP as the best thing ever. Staff and books shy away from highlighting any of these limitations. Other products handle deduplication without this sort of negative impact, but here, even with no deduplication in use, just the OPTION of using dedupe, it causes the system to be unusable.
Maybe there will be a “fix” or a “rewrite”, but I’ve been burned hard, with severe customer satisfaction, over two years. This negatively impacts trust, and is not the only IBM storage product causing me trust issues.
I expect that this is the fallout from Ginny Rometty’s command, “Don’t try to protect the past”. IBM’s vision is stated as “Cloud, Data, and Watson”. It will be interesting to see exactly how that translates into a future structure. IBM has been divesting a lot of pieces lately. Rometty ran services, and that’s what she knows as Chair and CEO. Anything she sees as a commodity, or low margin will be spun off. (IBM seems to need at least a 6:1 revenue to cost in order to operate. There are a lot of layers.) I’m starting to wonder if Storage and POWER will be sold off to Lenovo. I see hints, but no roadmaps, nothing concrete.
I did some testing and needed to document command syntaxen, even though I was not successful.
node01 / node02 – cannot remove EMC disks
aps are stopped
The fuser command will not detect processes that have mmap regions where that associated file descriptor has since been closed.
lsof | grep hdisk ### nothing fuser -fx /dev/hdisk2 ### nothing fuser -d /dev/hdisk2 ### nothing sudo filemon -O all -o 2.trc ; sleep 10 ; sudo trcstop ### only shows hottest 2 dsks ### Cannot remove disks after removign from HA, is related to this defect. http://www-01.ibm.com/support/docview.wss?uid=isg1IV65140 /usr/es/sbin/cluster/events/utils/cl_vg_fence_term -c vgname
In PowerHA 7.1.3, with the shared VG varied off, and the
disk in closed state, rmdev may fail and return a
busy error, eg:
# rmdev -dl hdisk2
Method error (/usr/lib/methods/ucfgdevice):
0514-062 Cannot perform the requested function because
the specified device is busy.
.
# cl_set_vg_fence_height
Usage: cl_set_vg_fence_height [-c] [rw|ro|na|ff]
JDSD NOTE: The levels are:
* rw = readwrite
* ro = read only
* na = no access
* ff = fail access
jdsd@node01 /home/jdsd $ sudo ls -laF /usr/es/sbin/cluster/events/utils/cl*fence* -rwxr--r-- 1 root system 12832 Nov 7 2013 /usr/es/sbin/cluster/events/utils/cl_fence_vg* -rwxr--r-- 1 root system 15624 Nov 7 2013 /usr/es/sbin/cluster/events/utils/cl_set_vg_fence_height* -r-x------ 1 root system 5739 Nov 7 2013 /usr/es/sbin/cluster/events/utils/cl_ssa_fence* -rwxr--r-- 1 root system 22508 Nov 7 2013 /usr/es/sbin/cluster/events/utils/cl_vg_fence_init* -rwxr--r-- 1 root system 4035 Feb 26 2015 /usr/es/sbin/cluster/events/utils/cl_vg_fence_redo* -rwxr--r-- 1 root system 15179 Oct 21 2014 /usr/es/sbin/cluster/events/utils/cl_vg_fence_term* jdsd@node01 /home/jdsd $ sudo ls -laF /usr/es/sbin/cluster/events/cspoc/cl*disk* -r-x------ 1 root system 109726 Feb 26 2015 /usr/es/sbin/cluster/cspoc/cl_diskreplace* -rwxr-xr-x 1 root system 20669 Nov 7 2013 /usr/es/sbin/cluster/cspoc/cl_getdisk* -r-x------ 1 root system 105962 Feb 26 2015 /usr/es/sbin/cluster/cspoc/cl_lsreplacementdisks* -r-x------ 1 root system 103433 Feb 26 2015 /usr/es/sbin/cluster/cspoc/cl_lsrgvgdisks* -rwxr-xr-x 1 root system 12259 Feb 26 2015 /usr/es/sbin/cluster/cspoc/cl_pviddisklist* -rwxr-xr-x 1 root system 4929 Nov 7 2013 /usr/es/sbin/cluster/cspoc/cl_vg_non_dhb_disks* jdsd@node01 /home/jdsd $ sudo /usr/es/sbin/cluster/cspoc/cl_lsrgvgdisks #Volume Group hdisk PVID Cluster Node #--------------------------------------------------------------------- caavg_private hdisk38 00deadbeefcaff53 node01 node01,node02datavg hdisk22 00deadbeefca8643 node02 node01,node02 demo_rg datavg hdisk23 00deadbeefca86f9 node02 node01,node02 demo_rg datavg hdisk24 00deadbeefca8752 node02 node01,node02 demo_rg datavg hdisk25 00deadbeefca87ac node02 node01,node02 demo_rg datavg hdisk26 00deadbeefca880e node02 node01,node02 demo_rg datavg hdisk27 00deadbeefca886c node02 node01,node02 demo_rg datavg hdisk28 00deadbeefca88d7 node02 node01,node02 demo_rg datavg hdisk29 00deadbeefca8965 node02 node01,node02 demo_rg datavg hdisk30 00deadbeefca89c5 node02 node01,node02 demo_rg datavg hdisk31 00deadbeefca8a52 node02 node01,node02 demo_rg datavg hdisk32 00deadbeefca8ad2 node02 node01,node02 demo_rg datavg hdisk33 00deadbeefca8b50 node02 node01,node02 demo_rg datavg hdisk34 00deadbeefca8c26 node02 node01,node02 demo_rg datavg hdisk35 00deadbeefca8c9a node02 node01,node02 demo_rg datavg hdisk36 00deadbeefca8cf7 node02 node01,node02 demo_rg journalvg hdisk37 00deadbeefca8d53 node02 node01,node02 demo_rg jdsd@node01 /home/jdsd $ sudo /usr/es/sbin/cluster/cspoc/cl_getdisk hdisk2 Disk name: hdisk2 Disk UUID: 1edeadbeefcafe04 b512d9e3b580fb13 Fence Group UUID: 0000000000000000 0000000000000000 - Not in a Fence Group Disk device major/minor number: 18, 2 Fence height: 2 (Read/Only) Reserve mode: 0 (No Reserve) Disk Type: 0x01 (Local access only) Disk State: 32785
Concurrent vg, so updating on node2 shows up on node1.
From node 2
sudo extendvg journalvg hdisk2 hdisk3 hdisk4 hdisk5 hdisk6 hdisk7 hdisk8 hdisk9 hdisk10 hdisk11 hdisk12 sudo /usr/es/sbin/cluster/cspoc/cl_getdisk hdisk2 sudo /usr/es/sbin/cluster/cspoc/cl_getdisk hdisk37 # Shows RW
From node 1
sudo /usr/es/sbin/cluster/cspoc/cl_getdisk hdisk2 sudo /usr/es/sbin/cluster/cspoc/cl_getdisk hdisk37 # Shows RW
From node1
sudo /usr/es/sbin/cluster/events/utils/cl_set_vg_fence_height -c journalvg rw sudo /usr/es/sbin/cluster/cspoc/cl_getdisk hdisk2 # Shows RW
From node2
sudo reducevg journalvg hdisk2 hdisk3 hdisk4 hdisk5 hdisk6 hdisk7 hdisk8 hdisk9 hdisk10 hdisk11 hdisk12 sudo /usr/es/sbin/cluster/cspoc/cl_getdisk hdisk2 # Shows RO
### OK, try again
From node 1
sudo mkvg -y dummyvg hdisk2 hdisk3 hdisk4 hdisk5 hdisk6 hdisk7 hdisk8 hdisk9 hdisk10 hdisk11 hdisk12 sudo varyoffvg dummyvg
From node 2
sudo importvg -y dummyvg hdisk2 sudo /usr/es/sbin/cluster/events/utils/cl_set_vg_fence_height -c dummyvg rw sudo /usr/es/sbin/cluster/cspoc/cl_getdisk hdisk2 ### Still RO
sudo /usr/es/sbin/cluster/events/utils/cl_vg_fence_term -c dummyvg sudo /usr/es/sbin/cluster/cspoc/cl_getdisk hdisk2 ### Still RO
sudo varyoffvg dummyvg sudo rmdev -Rl hdisk2
Both nodes
sudo exportvg dummyvg sudo importvg -c -y dummyvg hdisk2 sudo /usr/es/sbin/cluster/cspoc/cl_getdisk hdisk2 ### Still RO
sudo /usr/es/sbin/cluster/events/utils/cl_set_vg_fence_height -c dummyvg rw sudo /usr/es/sbin/cluster/events/utils/cl_vg_fence_init -c dummyvg rw hdisk2 cl_vg_fence_init[279]: sfwAddFenceGroup(dummyvg, 1, hdisk2): No such device sudo chvg -c dummyvg sudo varyonvg -n -c -A -O dummyvg sudo /usr/es/sbin/cluster/cspoc/cl_getdisk hdisk2 sudo /usr/es/sbin/cluster/cspoc/cl_getdisk hdisk3 ### Still RO
sudo varyoffvg dummyvg From Node 2
sudo rmdev -Rl hdisk2
Method error (/etc/methods/ucfgdevice):
0514-062 Cannot perform the requested function because the
specified device is busy.
sudo /usr/es/sbin/cluster/events/utils/cl_vg_fence_redo -c dummyvg rw hdisk2
/usr/es/sbin/cluster/events/utils/cl_vg_fence_redo: line 109: cl_vg_fence_init: not found
cl_vg_fence_redo: Volume group dummyvg fence height could not be set to read/write
This is related to this defect, but later version:
http://www-01.ibm.com/support/docview.wss?uid=isg1IV52444
sudo su - export PATH=$PATH:/usr/es/sbin/cluster/utilities:/usr/es/sbin/cluster/events/utils/:/usr/es/sbin/cluster/cspoc/:/usr/es/sbin/cluster/sbin:/usr/es/sbin/cluster /usr/es/sbin/cluster/events/utils/cl_vg_fence_redo -c dummyvg rw hdisk2 cl_vg_fence_init[279]: sfwAddFenceGroup(dummyvg, 11, hdisk2, hdisk3, hdisk4, hdisk5, hdisk6, hdisk7, hdisk8, hdisk9, hdisk10, hdisk11, hdisk12): No such device cl_vg_fence_redo: Volume group dummyvg fence height could not be set to read/write# cd /dev /usr/es/sbin/cluster/events/utils/cl_vg_fence_redo -c dummyvg rw hdisk2 cl_vg_fence_init[279]: sfwAddFenceGroup(dummyvg, 11, hdisk2, hdisk3, hdisk4, hdisk5, hdisk6, hdisk7, hdisk8, hdisk9, hdisk10, hdisk11, hdisk12): No such device cl_vg_fence_redo: Volume group dummyvg fence height could not be set to read/write#
SIGH!
I give up. We will probably have to reboot.
