Need help to understand zfs rpool space allocation

Hi,

I am unable to understand that, in one of my servers while

df -kh

Filesystem                     Size   Used  Available Capacity  Mounted on
rpool/ROOT/solaris-2     98G	11G	29G      29%              /

Even the Root FS filled on 40gb and system becomes unstable.

it is showing 40gb usable and whole size is 98gb.

root@xxx # zpool iostat -v rpool
   capacity	operations	bandwidth
pool	alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
rpool	   67.7G  31.8G	  0	23  5.59K   414K
  c2d0	  67.7G  31.8G	  0	23  5.59K   414K

root@xxx # zfs get volsize rpool/dump
NAME        PROPERTY  VALUE  SOURCE
rpool/dump  volsize   15G    local
root@xxx # zfs get volsize rpool/swap
NAME        PROPERTY  VALUE  SOURCE
rpool/swap  volsize   32G    local
root@xxx # zfs list
NAME                         USED  AVAIL  REFER  MOUNTPOINT
ldom1pool                    500G  1.12T    32K  legacy
ldom1pool/opt               36.1G  1.12T  36.1G  /opt
ldom1pool/opt-abc          149G  1.12T   149G  /opt/abc
ldom1pool/opt-def        84.2G  1.12T  84.2G  /opt/def
ldom1pool/opt-ghi          31K  1.12T    31K  /opt/ghi
ldom1pool/swap               231G  1.12T   224G  -
ldom1pool/u02                 31K  1.12T    31K  /u02
ldom1pool/u03                 31K  1.12T    31K  /u03
ldom1pool/u04                 31K  1.12T    31K  /u04
rpool                       69.2G  28.7G  73.5K  /rpool
rpool/ROOT                  14.5G  28.7G    31K  legacy
rpool/ROOT/solaris-2        14.5G  28.7G  11.5G  /
rpool/ROOT/solaris-2/var    2.07G  28.7G  2.00G  /var
rpool/VARSHARE              1.39G  28.7G  1.39G  /var/share
rpool/dump                  15.5G  29.2G  15.0G  -
rpool/export                4.79G  28.7G    32K  /export
rpool/export/home           4.79G  28.7G  9.75M  /export/home
rpool/export/home/u#      143M  28.7G   143M  /export/home/u1
rpool/export/home/u#    32.5M  28.7G  32.5M  /export/home/u#
rpool/export/home/u#     35K  28.7G    35K  /export/home/u#
rpool/export/home/u#  4.61G  28.7G  4.61G  /export/home/u#
rpool/export/home/u#    35K  28.7G    35K  /export/home/u#
rpool/export/home/u#    38.5K  28.7G  38.5K  /export/home/u#
rpool/export/home/u#     35K  28.7G    35K  /export/home/u#
rpool/swap                  33.0G  29.7G  32.0G  -

Please help me to understand where the rest space gone.

Take a look of the output of following commands :

zfs get -r reservation,refreservation,quota,refquota rpool
zfs list -t snapshot -r rpool
beadm list

Perhaps that will offer a clearer picture of status of your rpool zpool regarding space, reservation, quotas and such.

On the first glance, you have 32 GB of swap space defined inside rpool, do you require that much for some reason ?
You might want to define a new device or even a swap file on different disk/pool and free the rpool of mentioned 32GB.

Regards
Peasant.

1 Like

Just have a look to the column labeled "REFER" in "zfs list" output. It shows for each dataset the actual size occupied by that dataset.

Out of a 100 GB pool, 32 GB is taken by the swap area, 15 GB by the dump area, 11.5 by the root file system, 4.61 Gb by a user's file system, 2 GB by /var, 1.39 GB by /var/share. All remaining datasets are much smaller. The sum of all REFER values is 70 GB which match the 29 GB available you got with "df".

I would recommend being careful with reducing the swap size unless you really know how much swap your system might need in the future. Relocating the swap elsewhere, as already suggested by Peasant, would on the other hand be a good idea and the simplest way to recover 32 GB of disk space in your root pool. You might also relocate your 15 GB dump area elsewhere but there are more constraints about the receiving pool.

1 Like

Hi,

Please find the o/p of said commands,

root@xxx # zfs get -r reservation,refreservation,quota,refquota rpool
NAME                              PROPERTY        VALUE  SOURCE
rpool                             reservation     none   default
rpool                             refreservation  none   default
rpool                             quota           none   default
rpool                             refquota        none   default
rpool/ROOT                        reservation     none   default
rpool/ROOT                        refreservation  none   default
rpool/ROOT                        quota           none   default
rpool/ROOT                        refquota        none   default
rpool/ROOT/solaris-2              reservation     none   default
rpool/ROOT/solaris-2              refreservation  none   default
rpool/ROOT/solaris-2              quota           none   default
rpool/ROOT/solaris-2              refquota        none   default
rpool/ROOT/solaris-2@install      reservation     -      -
rpool/ROOT/solaris-2@install      refreservation  -      -
rpool/ROOT/solaris-2@install      quota           -      -
rpool/ROOT/solaris-2@install      refquota        -      -
rpool/ROOT/solaris-2/var          reservation     none   default
rpool/ROOT/solaris-2/var          refreservation  none   default
rpool/ROOT/solaris-2/var          quota           none   default
rpool/ROOT/solaris-2/var          refquota        none   default
rpool/ROOT/solaris-2/var@install  reservation     -      -
rpool/ROOT/solaris-2/var@install  refreservation  -      -
rpool/ROOT/solaris-2/var@install  quota           -      -
rpool/ROOT/solaris-2/var@install  refquota        -      -
rpool/VARSHARE                    reservation     none   default
rpool/VARSHARE                    refreservation  none   default
rpool/VARSHARE                    quota           none   default
rpool/VARSHARE                    refquota        none   default
rpool/dump                        reservation     none   default
rpool/dump                        refreservation  15.5G  local
rpool/dump                        quota           -      -
rpool/dump                        refquota        -      -
rpool/export                      reservation     none   default
rpool/export                      refreservation  none   default
rpool/export                      quota           none   default
rpool/export                      refquota        none   default
rpool/export/home                 reservation     none   default
rpool/export/home                 refreservation  none   default
rpool/export/home                 quota           none   default
rpool/export/home                 refquota        none   default
rpool/export/home/u#           reservation     none   default
rpool/export/home/u#           refreservation  none   default
rpool/export/home/u#           quota           none   default
rpool/export/home/u#           refquota        none   default
rpool/export/home/u#          reservation     none   default
rpool/export/home/u#          refreservation  none   default
rpool/export/home/u#          quota           none   default
rpool/export/home/u#          refquota        none   default
rpool/export/home/u#         reservation     none   default
rpool/export/home/u#         refreservation  none   default
rpool/export/home/u#         quota           none   default
rpool/export/home/u#         refquota        none   default
rpool/export/home/u#        reservation     none   default
rpool/export/home/u#        refreservation  none   default
rpool/export/home/u#        quota           none   default
rpool/export/home/u#        refquota        none   default
rpool/export/home/u#        reservation     none   default
rpool/export/home/u#        refreservation  none   default
rpool/export/home/u#        quota           none   default
rpool/export/home/u#        refquota        none   default
rpool/export/home/u#          reservation     none   default
rpool/export/home/u#          refreservation  none   default
rpool/export/home/u#          quota           none   default
rpool/export/home/u#          refquota        none   default
rpool/export/home/u#         reservation     none   default
rpool/export/home/u#         refreservation  none   default
rpool/export/home/u#         quota           none   default
rpool/export/home/u#         refquota        none   default
rpool/swap                        reservation     none   default
rpool/swap                        refreservation  33.0G  local
rpool/swap                        quota           -      -
rpool/swap                        refquota        -      -
You have new mail in /var/mail/root
root@xxx # zfs list -t snapshot -r rpool
NAME                               USED  AVAIL  REFER  MOUNTPOINT
rpool/ROOT/solaris-2@install      1.01G      -  2.03G  -
rpool/ROOT/solaris-2/var@install  70.8M      -  96.8M  -
root@xxx # beadm list
BE        Active Mountpoint Space  Policy Created
--        ------ ---------- -----  ------ -------
solaris-2 NR     /          17.51G static 2015-02-26 22:47

---------- Post updated 03-23-16 at 12:45 AM ---------- Previous update was 03-22-16 at 11:40 PM ----------

Hi,

I am completely new to zfs, can you plz suggest the way by which i can first remove and then recreate swap and dump to other place which can be referred by system.

I know i am asking for too much but plz help, if it would be solaris 10 then i can do all but here i need help or any reference for it.

**This system is very very critical

Here are some useful starting points:

https://docs.oracle.com/cd/E26502_01/html/E29006/fsswap-14677.html\#scrolltoc

https://docs.oracle.com/cd/E53394_01/html/E54764/dumpadm-1m.html

The method would have been essentially the same with Solaris 10 if installed with root ZFS.

Then, build a test system where you experience and familiarize with the technology before attempting any change on that system. Note that a "very very critical system" should have an alternate boot environment ready to take over should you need to rollback for some reason. This is doesn't look to be the case with yours.