2.6TB filesystem to be copied or moved to another FS ?

filosophizer · December 18, 2010, 3:30pm

Hi folks,

I have a filesystem which is 2.6TB big and would like to copy/mirror this filesystem on another volume group.

1) What is the fastest way to copy or mirror it to another filesystem without shutting down production vg ( oracle is using the filesystem )

2) Is there any way to estimate how long it will take to copy/mirror the filesystem if it is required to shutdown Oracle ?

/u03 is the filesystem which has to be copied/moved to another new filesystem

two options which i thought of:
(1) to mirrorvg ie., mirror oraclevg to the new vg but this process will be very slow if ORACLE is running is continuously accessing the filesystem /u03 while mirror process is under way.

(2) shutdown oracle, use copy filesystem command and move it, but need to know how long will it take to move 2.6TB since Oracle cannot be down for more than 10 hours

 # lspv
  hdisk0          000f4c011b1e8d57                    rootvg          active
  hdisk1          000f4c017d3b0cd9                    rootvg          active
  hdisk2          000f4c01fc5f525f                    oraclevg        active
  hdisk4          000f4c01b4e565d1                    oraclevg2       active
  hdisk3          000f4c01d80cdc0b                    oraclevg        active
  # lsvg -l oraclevg
  oraclevg:
  LV NAME             TYPE       LPs     PPs     PVs  LV STATE      MOUNT POINT
  loglv00             jfs2log    1       1       1    open/syncd    N/A
  fslv01              jfs2       5178    5178    2    open/syncd    /u03
   
   
  # lsvg -p oraclevg
  oraclevg:
  PV_NAME           PV STATE          TOTAL PPs   FREE PPs    FREE DISTRIBUTION
  hdisk2            active            4079        0           00..00..00..00..00
  hdisk3            active            1100        0           00..00..00..00..00
  #
   
  # lsvg oraclevg
  VOLUME GROUP:       oraclevg                 VG IDENTIFIER:  000f4c010000d9000000011dfc87db92
  VG STATE:           active                   PP SIZE:        512 megabyte(s)
  VG PERMISSION:      read/write               TOTAL PPs:      5179 (2651648 megabytes)
  MAX LVs:            256                      FREE PPs:       0 (0 megabytes)
  LVs:                2                        USED PPs:       5179 (2651648 megabytes)
  OPEN LVs:           2                        QUORUM:         2 (Enabled)
  TOTAL PVs:          2                        VG DESCRIPTORS: 3
  STALE PVs:          0                        STALE PPs:      0
  ACTIVE PVs:         2                        AUTO ON:        yes
  MAX PPs per VG:     30480
  MAX PPs per PV:     5080                     MAX PVs:        6
  LTG size (Dynamic): 256 kilobyte(s)          AUTO SYNC:      no
  HOT SPARE:          no                       BB POLICY:      relocatable

[/FONT]

---------- Post updated at 12:30 PM ---------- Previous update was at 07:30 AM ----------

I did little research on the internet and found few documents which showed the recommended procedure

1) either by backup and restore ( which will preserve the file permissions and inode information )

2) cplv

now regarding cplv, no document points out the following and i hope some knows the answer here on the forum

1) can it be used to copy 2.6 TB ? since it is very big filesystem ?

2) will it preserve file permissions and inode information ?

3) what is the transfer/copy speed ? this is important to know since cplv can be only done when the filesystem is unmounted ( not in use ) so 2.6TB takes like 2 days then have to go with another solution like backup and restore.

thanks

zxmaus · December 18, 2010, 9:45pm

Hi,

if you are running AIX 6.1, create and mount the new filesystem with nolog + noatime option. Than just do cp -hpr the .dbf files and whatever else you have from the old to the new filesystem. When the DB is down, this should not take more than a couple of hours (I copied more than 4 TB in just over 8 hrs not too long ago while a few other DBs were even up and running). You need to have the DB down which you copy, or the copy will be corrupted / useless. Permissions, ownership and even timestamps are preserved.

Hope that helps
regards
zxmaus

Corona688 · December 18, 2010, 9:52pm

As usual, the transfer rate almost certainly has more to do with your disks and disk controllers and I/O buses than what program you use. It will take as long as it takes.

bakunin · December 19, 2010, 9:46am

I suppose that the mirrorlv soution will be on of the fastest methods and probably the one with the least hassle. I would, being in your place, prefer to go for this solution.

In case this is not possible for whatever reason, here is a suggestion, you will have to fill in some blanks. It assumes that there is (almost) nothing else than the DB-files on the filesystem.

Set the database into "hot backup" mode. I don't know exactly how this is done, but every Oracle DBA knows for sure what I'm talking about. Basically this is making the database aware that a file-backup is taken while it is running. The file-backup as such will be considered inconsistent but along with the archive logs taken during the backup it can be made consistent. Oracle has special provisions for doing exactly this built in.

Note that the redo-logs created this way contain more info than normal redo-logs and the rate of logswitches will increase. Your DBA should be able to estimate how big the factor of the increase will be.

Take a file-backup from the running database. You can use whatever system means there are: TSM, tar, cp , ....
After the backup has finished rotate out the active redo-logs and set the database to normal again.
After restoring the copy to some other diskspace start the database and make it consistent by applying the archive-logs taken since the start of the backup-process. The DB then is consistent and equal to the original at the time of the end of the backup process.

filosophizer · December 19, 2010, 10:23am

thank you everyone for the suggestions,

Also came across this IBM tech note on this subject

for archival purpose:

https://www-304.ibm.com/support/docview.wss?uid=isg3T1000167
Moving a JFS/JFS2 File System to a New Volume Group

Technote (FAQ) Question Moving a JFS/JFS2 File System to a New Volume Group
Answer This document describes how to move a journaled file system (JFS) or enhanced journaled file system (JFS2) from one volume group to another. Information in this document applies to AIX Versions 4 and 5.
Moving file systems from one volume group to another
Logical volumes specific to rootvg
Related documentation Moving file systems from one volume group to another

ATTENTION: Make sure a full backup exists of any data you intend to migrate before using these procedures.
In AIX, storage allocation is performed at the volume group level. Storage cannot span volume groups. If space within a volume group becomes constrained, then space that is available in other volume groups cannot be used to resolve storage issues.
The solution to this problem is to add more physical volumes to the relevant volume group. This may not be an option in all environments. If other volume groups contain the required free space, the alternative is to move the required logical volumes to the desired volume group and expand them as needed.
The source logical volume can be moved to another volume group with the cplv command. The following steps achieve this.
ATTENTION: The logical volume should be inactive during these steps to prevent incomplete or inconsistent data. If the logical volume contains a mounted file system, then that file system should be unmounted first. If this logical volume is being used as a RAW storage device, then the application using this logical volume should close the device or be shut down.

Copy the source logical volume to the desired volume group with the cplv command. For example, where myvg is the new volume group and mylv is the name of the user's logical volume, enter:
cplv -v myvg mylv
This will return the name of the new logical volume, such as lv00. If this logical volume was being used for RAW storage, skip to to step 6. If this is a JFS or JFS2 file system, proceed to step 2. Note that RAW storage devices should NOT use the first 512 bytes of the RAW device. This is reserved for the LVCB or logical volume control block. cplv will not copy the first 512 bytes of the RAW logical volume, but it will update fields in the new logical volume's LVCB.

All JFS and JFS2 file systems require a log device. This will be a logical volume with a type of jfslog or jfs2log for JFS2 file systems. Run the lsvg -l <VGNAME> command on your destination volume group. If a JFS or JFS2 log DOES NOT already exist on the new volume group, create one by using the mklv and logform commands as detailed below. If a JFS or JFS2 log DOES exist, proceed to step 3. With a JFS2 filesystem, you also have the option of using an inline log. With inline logs, the jfs2log exists on the filesyster itself. After the cplv command is ran on a JFS2 inline log filesystem, run:
logform /dev/lvname
You should receive a message about formatting the inline log. If you do not receive a message about an inline log, then this filesystem is not a JFS2 inline log filesystem and you should treat it as a regular JFS2 filesystem. After hitting y on formatting the inline log, continue to step 3.
To make a new JFS log, enter the following command, where myvg is the name of the new volume group, enter:
mklv -t jfslog myvg 1
To make a new JFS2 log, enter:
mklv -t jfs2log myvg 1
This will return a new logical volume of either type jfslog or jfs2log, such as loglv00. This new logical volume will need to be formatted with the logform command in order to function properly as either a JFS or JFS2 log. For example:
logform /dev/loglv00
Answer yes to destroy.

Change the filesystem to reference a log device that exists in the new volume group and the new logical volume with the chfs command. For example, where myfilesystem is the name of the user's filesystem, enter:
chfs -a dev=/dev/lv00 -a log=/dev/loglv00 /myfilesystem
With inline logs on JFS2 filesystems this command is also different:
chfs -a dev=/dev/lv00 -a log=INLINE /myfilesystem

Run fsck to ensure filesystem integrity. Enter: fsck -p /dev/lv00
NOTE: It is common to receive errors after running fsck -p /dev/lvname prior to mounting the filesystem. These errors are due to a known bug that development is currently aware of and which will be resolved in a future release of AIX. Once the filesystem is mounted, a future fsck with the filesystem unmounted should no longer produce an error.

Mount the file system. For example, where myfilesystem is the name of the user's file system, enter:
mount /myfilesystem
At this point, the migration is complete, and any applications or users can now access the data in this filesystem. To change the logical volume name, proceed to the following step.
NOTE: If you receive errors from the preceding step, do not continue. Contact you AIX support center.

Remove the source logical volume with the rmlv command. For example, where mylv is the name of the user's logical volume, enter:
rmlv mylv

Rename and reset any needed attributes on the new logical volume with the chlv or chmod commands. In order to rename the logical volume, the filesystem or raw logical volume must be in a closed state. For example, where mylv is the new name you wish to change lv00 to be, enter:
chlv -n mylv lv00

Logical volumes specific to rootvg

The following logical volumes and file systems are specific to the rootvg volume group and cannot be moved to other volume groups:
Logical Volume File System or Description
------------------------------------------------------
hd2 /usr
hd3 /tmp
hd4 /
hd5 <boot logical volume>
hd6 <primary paging space>
hd8 <primary JFS or JFS2 log>
hd9var /var
Related documentation

For more in-depth coverage of this subject, the following IBM publications are recommended:

AIX Version 4.3 System Management Guide: Operating System and Devices

AIX Version 5.1 System Management Guide: Operating System and Devices

AIX Version 4.3 Commands Reference (references for chfs, chlv, and cplv)

AIX Version 5.1 Commands Reference

AIX Version 5.2 Commands Reference

Cross Reference information Segment Product Component Platform Version Edition Operating SystemAIXInstallation- backup- restore

Historical Number isg1pTechnote0280

but please check these fixpacks as CPLV has some problems

http://unix.derkeiler.com/Mailing-Lists/AIX-L/2003-04/0416.html

Moving a JFS FIle to a New Volume Group

backup data you intend to migrate

unmount fs where lv resides

copy the source logical volume to the desired volume group with the cplv
command.
For example, where myvg is the NEW volume group and mylv is the name of
the user's
logical volume, enter:

cplv -v myvg mylv

All JFS require a log device. If JFS log does not already exists on the
new volume group,

mklv -t jfslog myvg 1

This will return a new jfslog such as loglv00. This will need to be
reformatted with the
logform command to function as a JFS log.

logform /dev/loglv00

Change the filesystem to reference a log device that exists in the new vg
and the new
logical volume with the chfs command.

For example where myfilesystem is the name of the user's filesystem,
enter:

chfs -a dev=/dev/lv00 -a log=/dev/loglv00 /myfilesystem

RUn fsck to ensure filesystem integrity,

fsck -p /dev/lv00

Mount the file system. For example, where myfilesystem is the name of the
user's
file system, enter:

mount /myfilesystem

At this point the migration is complete and any applications or users can
now access the
data in this filesystem. To change the logical volume name, proceed to
step 8. Note: If you receive errors from the preceding step do not continue, contact AIX
support center.

Remove the source logical volume with the rmlv command:

rmlv mylv

Rename and reset any needed attributes on the new logical volume with the
chlv or chmod commands.

For example, where mylv is the name of the user's logical volume, enter:

chlv -n mylv lv00
-------------------------------------------------------------

Joe Leyden

-----Original Message-----
From: Green, Simon [mailto:Simon.Green@EU.ALTRIA.COM]
Sent: Wednesday, April 30, 2003 9:36 AM
To: aix-l@Princeton.EDU
Subject: Re: move logical volume and file system to different vg

Sorry: I obviously wasn't clear. They were intended as three completely
independent alternatives, not three related steps.

Simon Green
Altria ITSC Europe s.a.r.l.

AIX-L Archive at http://marc.theaimsgroup.com/?l=aix-l&r=1&w=2
AIX FAQ at AIX FAQ Index

N.B. Unsolicited email from vendors will not be appreciated.

zxmaus · December 19, 2010, 1:26pm

All solutions will work - the question is always what you achieve. cplv will maintain the fileallocation tables - not sure if this is what you want. If your DB exists for a long time, that this is rather big and over time slowing down all filesystem accesses. Creating a new filesystem will create a new clean fileallocation table, so this would always be my choice but that is probably just a matter of personal taste. I like Bakunins solution too as that would achieve the same but it makes me dependant on a DBA and I like to be independant
regards
zxmaus

bakunin · December 19, 2010, 4:45pm

Well, actually it amounts to 2-3 commands typed into SQLPlus - I'd ask the DBA to tell me how its done and put it into my documentation, so i never have to ask him again. ;-))

bakunin

kapilraj · December 20, 2010, 5:07pm

Well, if you can afford it - you may standup a standby (Oracle-Standby) database and decide a date and shutdown the real DB and convert the new database as the production. If you plan it well, you might be done with an Hour downtime. In the worst case, backout is to shutdown the new database and startup the old. ( 5 mins ? ). I am sure change management is going to like this one. Very similar to the hot backup solution someone posted.

If all you want is to move to new disks, the storage vendor might have some utilities as well. BCV for local copies and SRDF for copy across arrays in the EMC world. (I assumed you have SAN storage)

filosophizer · December 22, 2010, 3:02am

okay, cplv has failed

1) I had a filesystem with 200G allocation with actual Data of 100.6 GB
so I created another filesystem with 245 GB and tried to copy with cplv command and i got a message to use chlv with -t parameter ; so i didn't understand why i should change the type when they are both same type of filesystem JFS2 ?

so, i skipped and went to another option , cplv to user-create logical volume ( that is the system will create the logical volume by itself )

2) cplv to user-create logical volume is taking more than 1.1 hour and still the copy was not finished and I had to cancel/abort the operation because users needed the filesystem to copy 100.6 GB (total filesystem size 200GB) Logical Volume to another logical volume is taking more than 1.1 hours
any suggestions ?

some suggested copying files and directories but Oracle team is avoiding this approach.

filosophizer · December 25, 2010, 8:00am

mirrorlv for the above example : filesystem of 200G allocation with actual Data of 100.6 GB takes around 3.5 hours.

Note: the filesystem / logical / volume group is from SAN Storage connected by one 4 GB fiber.

so we are here copying a filesystem/logical volume from SAN Storage to SAN Storage.

So, if 200G logical volume takes 3.5 hours with mirrorlv imagine 2.6TB ?
around 40 hours or so ?

next I will try to cp as zxmaus or as bakunin said, will backup and restore.

filosophizer · December 26, 2010, 8:54am

Folks, there is a small problem. After using cplv and log was created and initialized and a new filesystem ( new name for filesystem) was mounted on that Logical Volume. But after mounting it, I didn't see any data !

but lsvg volume_group shows PPs are in use.

any clue as to what might be the issue ?

zxmaus · December 26, 2010, 10:57am

do you have a new mountpoint or the same - maybe it is just overmounted with something else

BTW - if you only want to move to new SAN - but not necessarily to a new volumegroup - why dont you just add the new disks to the existing VG and do a migratepv olddisk newdisk(s) and drop the old empty disks afterwards from the VG. This is for sure not the fastest but the easiest way as it doesnt need ANY downtime whatsoever.

Regards
zxmaus

filosophizer · December 26, 2010, 1:23pm

It is a new mount point..

Okay here is what happened:

Created a new VG ==> oraclevg6
cplv logical volume from oldvg to newvg ( and the system created lv called fslv06 )
after the cplv was over lsvg -l oraclevg6 ==> showed that there is a log logical volume called loglv01
create a filesystem with a new name and not the existing one for the copied logical_volume call it /u06
mount /u06 on fslv06
doesn't work...umount /u06
logform /dev/loglv01
mount /u06

still no data present.

==========

No, it cannot be done for various administrative reasons.

the array is different and new san is different
Database team want to have the new disks totally seperate.
the above experiment was with 200GB and it took 3.5 - 4 hours....what about 2.6TB...... ? and if something goes wrong then the company application might need a total restore.
too risky !

homeyjoe · December 27, 2010, 11:07am

I think the problem may be the new filesystem on the copied lv.
Here's what I did on a test filesystem and it worked fine:

Created new lv

# mklv -y lvcorenew -t jfs2 data2vg 32

changed new lv to be type 'copy'

# chlv -t copy lvcorenew

Ran cplv

# cplv -e lvcorenew -f lvcore

Created new jfs2log for the new lv

# mklv -y corelog -t jfs2log data2vg 1

Formated the new jfs2log

# logform /dev/corelog

Modified the existing filesystem to use the newly copied lv

# chfs -a dev=/dev/lvcorenew -a log=/dev/corelog /usr/local/corefiles

Mounted fs

# mount /usr/local/corefiles

Check filesystem

# ls /usr/local/corefiles

filosophizer · December 27, 2010, 2:25pm

homeyjoe, thank you for the post.

Just wondering why you have changed the logical volume type to copy ?

changed new lv to be type 'copy'

I didn't do this step, but would like to know the reason behind changing the type. When I created a logical volume and tried to copy it ( as i mentioned in the previous post ) i got an error and the message was :: Change the type of the logical volume with chlv command ...
At that time, i didn't understand why it needs to be changed and into what ...

So, I skipped, and I let the system create the logical volume and copy....So, when I chose the second option, I am not sure if AIX will automatically change the logical volume type to copy ?

Also, after successful mounting do i need to change the type to something else in-order to make the logical volume into write mode ?

homeyjoe · December 27, 2010, 2:42pm

Since I made a new LV with my naming scheme, and not letting AIX create a generic-named LV, I had to set the type to 'copy' (per the cplv error code that you saw).
When you do a 'lsvg -l VGname' you will see that it's a 'copy' type instead of the usual jfs, jfs2, paging, etc. Once the cplv command is complete then the LV will be listed as jfs/jfs2 instead of copy.

Happy AIX-ing!

filosophizer · January 3, 2011, 9:12am

guys the cplv didn't work.
I tried all the steps homeyjoe mentioned.

here is scenario

1) hdisk3 = testvg1 = 100MB = fslv02 = 20 pps = /testv1

2) hdisk4 = testvg2 = 200MB = testlv = 20 pps

did all the steps mentioned by homeyjoe but after mounting with another name /testv2 ; no data was present in the filesystem....
so, I renamed the original mount point /testv1 so that i can mount the copied logical volume filesystem as /testv1 and still no data present !!

Do the logical volumes need to be of exact size ?
Are there any additional steps missing ?

homeyjoe · January 3, 2011, 11:33am

Filosophizer - Did you do a different command than what I listed in Step #6 since you say that you created another filesystem (/testv2)? I'm guessing this may be where the procedure I listed is failing for you.

Just so you know, when I do it my way I ended up with 2 lv's but only one filesystem. I had lvcore in rootvg and lvcorenew in data2vg. Only lvcorenew showed a filesystem mounted on it though (/usr/local/corefiles). So after verifying the files were present in /usr/local/corefiles I deleted the old lv (lvcore) in rootvg.

I've done this procedure multiple times over the years so I'm guessing there's something that you are doing differently.

filosophizer · January 3, 2011, 11:42am

I followed your steps as you listed, and for step #6 i did as you mentioned:
two ways to do
through smitty ... add a filesystem on existing defined logical volume and in the section of LOG i mentioned the log logical form.

or through command you gave.

now step #6
# chfs -a dev=/dev/lvcorenew -a log=/dev/corelog /usr/local/corefiles rather it should be

# chfs -a dev=/dev/lvcorenew -a log=/dev/corelog /filesystem_name

otherwise you will get an error chfs: 0506-915 No record matching /usr/local/corefiles was found in /etc/filesystems.

I think the problem is in step 6
/usr/local/corefiles should be filesystem name or something else ?

homeyjoe · January 3, 2011, 12:01pm

Right - /usr/local/corefiles is the name of my filesystem that I was moving from rootvg to data2vg. You will have to change that accordingly. Same with the lv names I was using. Sorry if I confused you!