How to grow a zfs file system?

Hi

I have the following file system, that needs to be expanded to more 500Gb, so a total of 1Tb:

 df -h /oradata1
Filesystem             Size   Used  Available Capacity  Mounted on
oradata1               587G   517G        69G    89%    /oradata1

I am not familiar with zfs , I am more used to ufs , so how can I go about it?

Depends on how your zpool is setup... You might just add a new disk to grow your pool and you might be done...

Managing ZFS File Systems in Oracle(R) Solaris 11.3

thanks for the reply.
The previous sys admin setup this server like this below:

 zpool list
NAME      SIZE  ALLOC   FREE  CAP  DEDUP    HEALTH  ALTROOT
oradata   500G   473G  27.5G  94%  1.00x  DEGRADED  -
oradata1  596G   517G  78.8G  86%  1.00x    ONLINE  -
rpool     136G   132G  4.10G  96%  1.00x    ONLINE  -
root@solaris:~# zfs list
NAME                       USED  AVAIL  REFER  MOUNTPOINT
oradata                    473G  19.6G   473G  /oradata
oradata1                   517G  69.5G   517G  /oradata1
rpool                      132G  1.47G  73.5K  /rpool
rpool/ROOT                 105G  1.47G    31K  legacy
rpool/ROOT/solaris         105G  1.47G   104G  /
rpool/ROOT/solaris/var     660M  1.47G   498M  /var
rpool/dump                8.19G  1.72G  7.94G  -
rpool/export              11.3G  1.47G  33.5K  /export
rpool/export/home         11.3G  1.47G    34K  /export/home
rpool/export/home/cmovel  4.66G  1.47G  4.66G  /export/home/cmovel
rpool/export/home/oracle  6.66G  1.47G  6.66G  /export/home/oracle
rpool/swap                2.06G  1.53G  2.00G  -
rpool/swap2               6.19G  1.66G  6.00G  -
root@solaris:~# df -h
Filesystem             Size   Used  Available Capacity  Mounted on
rpool/ROOT/solaris     134G   104G       1.5G    99%    /
/devices                 0K     0K         0K     0%    /devices
/dev                     0K     0K         0K     0%    /dev
ctfs                     0K     0K         0K     0%    /system/contract
proc                     0K     0K         0K     0%    /proc
mnttab                   0K     0K         0K     0%    /etc/mnttab
swap                   268M   2.2M       266M     1%    /system/volatile
objfs                    0K     0K         0K     0%    /system/object
sharefs                  0K     0K         0K     0%    /etc/dfs/sharetab
fd                       0K     0K         0K     0%    /dev/fd
rpool/ROOT/solaris/var
                       134G   498M       1.5G    25%    /var
swap                   266M     0K       266M     0%    /tmp
rpool/export           134G    33K       1.5G     1%    /export
rpool/export/home      134G    34K       1.5G     1%    /export/home
rpool/export/home/cmovel
                       134G   4.7G       1.5G    77%    /export/home/cmovel
rpool/export/home/oracle
                       134G   6.7G       1.5G    82%    /export/home/oracle
oradata                492G   473G        20G    97%    /oradata
oradata1               587G   517G        69G    89%    /oradata1
rpool                  134G    73K       1.5G     1%    /rpool
/export/home/cmovel    6.1G   4.7G       1.5G    77%    /home/cmovel
root@solaris:~#

Now, the oracle dba wants me to either expand /oradata1 or create a new /oradata2

Hi,

Sorry to butt in, but as an aside to your main question, I'd be far more worried about this before thinking about doing anything else:

NAME      SIZE  ALLOC   FREE  CAP  DEDUP    HEALTH  ALTROOT
oradata   500G   473G  27.5G  94%  1.00x  DEGRADED  -

The oradata pool is in a degraded state, which means it's experienced a failure of some kind and so its redundancy is likely to be severely reduced or non-existent. As a priority, you probably want to run zpool status -v oradata and see what has actually gone wrong and sort that out.

Of course, if the oradata pool isn't in use any longer or you otherwise have reasons not to care about its data then that's fine I suppose. But I just thought I'd mention this, since if it were me, and this is a live pool, I'd want to sort this out urgently before doing anything else.

Hi

They are not using that, at the moment. the output of the status is:

 zpool status -v oradata
  pool: oradata
 state: DEGRADED
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://www.sun.com/msg/ZFS-8000-9P
  scan: none requested
config:

        NAME                     STATE     READ WRITE CKSUM
        oradata                  DEGRADED     0     0     0
          c4t500A09828DE3E799d0  DEGRADED     0     0     0  too many errors

errors: No known data errors
root@solaris:~#

Hi,

OK. Well, if you're 100% absolutely sure you don't care about anything that happens to the data on that pool, then I suppose there's nothing you need to do about it.

Returning to your original question: firstly, I have to say I'm relatively new to ZFS myself, having only recently come back to Solaris on a regular basis. But from what I've learned thus far, the first thing is (as per the earlier advice from DukeNuke2) to figure out what kind of pool you've got. The Oracle documentation he linked you to this morning is also excellent, and covers pretty much everything you need to know in terms of how to add things.

So take a look at the output of zpool status -v oradata1 , find out if it's made of mirrors, some kind of RAID, what devices are in it, and then once you know that you'll be able to determine the best way to add a device to it.

Another comment about your ZFS pools.

Your root pool is almost full, has only 4% left:

rpool     136G   132G  4.10G  96%  1.00x    ONLINE  -

You are strongly advised to leave more free space in active ZFS pools as otherwise, the performance will suffer a lot due to the system trying to find free space and the data fragmentation resulting from the lack of contiguous areas. Not exceeding 80% is a safe bet.

Back to the topic, the "long" name reported for the oradata disk (c4t500A09828DE3E799d0) suggests a storage array is used to build this pool.

Exanding the oradata1 pool might then be achieved simply by enlarging the underlying LUN. Depending on your ZFS settings, nothing more might be required.

You might want to post these commands output for us to get a better idea about your configuration:

cat /etc/release
zpool status oradata1
zpool get autoexpand oradata1
cfgadm -al -o show_SCSI_LUNS c4

please find the output of the requested commands below:

root@solaris:~# cat /etc/release
                          Oracle Solaris 11 11/11 SPARC
  Copyright (c) 1983, 2011, Oracle and/or its affiliates.  All rights reserved.
                            Assembled 18 October 2011
root@solaris:~#
root@solaris:~# zpool status oradata1
  pool: oradata1
 state: ONLINE
  scan: none requested
config:

        NAME                       STATE     READ WRITE CKSUM
        oradata1                   ONLINE       0     0     0
          c5t500A09819DE3E799d1s6  ONLINE       0     0     0

errors: No known data errors
root@solaris:~#
root@solaris:~# zpool get autoexpand oradata1
NAME      PROPERTY    VALUE   SOURCE
oradata1  autoexpand  off     default
root@solaris:~#
root@solaris:~# cfgadm -al -o show_SCSI_LUNS c4
cfgadm: Hardware specific failure: invalid option: show_SCSI_LUNS
root@solaris:~#

Hi,

It now looks pretty much certain that you're using a SAN or some other kind of managed storage solution at the back-end here, rather than internal disks or other simple local storage. As per jlliagre's reply earlier, using that storage solution to enlarge the underlying LUN would be the best way forward here in all likelihood.

What is it you're using as a storage solution in this platform ? Whatever it is, it almost certainly provides a way to enlarge a LUN by one means or another. Once that's been done, you can then look at growing the ZFS pool to fill the new available size.

We are using NetApp as storage

Hi,

OK. I'm not myself familiar with NetApp storage solutions, but I'm sure there will be some management interface you can use to administer it no doubt. If NetApp permits you to non-destructively enlarge the underlying LUN without interrupting normal operation (you need to be totally sure about both aspects of that before doing this with the filesystem still mounted), then once you've enlarged the appropriate LUN you can come back to looking at expanding the ZFS pool.

Hi,

On the NetApp front you can go two ways.

  1. Expand the NetApp LUN and grow ZFS into it.
  2. Provision a new LUN of the appropriate size and use it to replace the existing LUN.

To expand the LUN on the NetApp, you would run the following from the NetApp CLI;

vol size VolName + 500g

Then you would expand into the allocated space.

Or create a new 1Tb LUN, present it to the system and use the zfs commands to replace as follows;

zpool replace oradata1 c5t500A09819DE3E799d1s6 <newdiskname>
zpool set autoexpand=on oradata1 

Regards

Gull04

1 Like

Thanks Gull04!

I dont have access to NetApp cli, the NetApp admin does it for me, on my side (operating system wise) If I am going to replace current oradata1 disk with newdiskname would I not be destroying current data on oradata1 ?

Hi,

When it comes to replacing a disk in a zfs pool, the replace action does the following;

The "replace" action migrates data to the new device from the damaged device or from other devices in the pool if it is in a redundant configuration.

When the command is finished, it detaches the damaged device from the configuration, at which point the device can be removed from the system.

Hopefully your disk source is undamaged, so it will just migrate the data without significant risk.

Regards

Gull04

1 Like

But the oradata1 pool is not in a redundant configuration.

---------- Post updated at 18:34 ---------- Previous update was at 18:31 ----------

You can't replace a non redundant disk.

Assuming you have proper backups, the simple way is to extend the size of the LUN. The data already stored on it won't be lost.

Hi,

There is an excellent guide on the oracle site here - - as I read it (but I've been known to be wrong). I can't really understand the logic of a single disk in a zpool, as there are no data replicas available that allow the benefits of zfs - you might as well use UFS at least you have more recovery tools.

Obviously the best approach would be to grow the original LUN or have the SAN admin people do this, then grow the FS into it. In any case you should ensure that you have a backup of the data before you start doing anything.

Regards

Gull04

A single disk pool is a poor practice, but unfortunately widespread when storage arrays are used. ZFS is still beneficial when using a single disk pool though.

  • There is double or triple redundancy in the metadata, that makes the file system resilient to moderate disk corruption.
  • You can also enable ditto blocks (copies) so data itself would recover from some disk blocks issues.
  • ZFS will immediately spot corrupted data or metadata while UFS will return corrupted data without notice and might panic the OS if metadata is corrupted.
  • Finally, telling you have more UFS recovering tools is questionable. With ZFS, you can recover from situations where UFS would be helpless.
1 Like

a couple of notes because I run this exact setup.

#1. I would never build a zpool without the following conditions being met.
a. multipathing on the netapp AND the solaris box being configured and working correctly.
#2. At a minium 2 luns in a zpool mirror, a raidZ would be better.

if these conditions are met, you'll probably never lose data. While it's all fine and good to say that Netapp handles the redundancy using huge aggregates, all that really does is buy you insulation from hard disk failure, not corruption or loss of connection. the zpool is likely in a degraded state due to loss of comms or poor network performance. Thus the multipathing suggestion.

to answer your specific question though, the way I have handled this in the past is to grow the lun and then just turn on autoexpand. that will grow the zpool to meet the new lun size.