Solaris 10 booted to maintenance mode Reason: Start method exited with $SMF_EXIT_ERR_FATAL

Hello dear community.
Help me please

Boot device: rootdisk  File and args:
SunOS Release 5.10 Version Generic_150400-44 64-bit
Copyright (c) 1983, 2016, Oracle and/or its affiliates. All rights reserved.
Hostname: saromea4
Dec 15 19:19:48 svc.startd[9]: svc:/system/sysevent:default: Method "/lib/svc/method/svc-syseventd start" failed with exit status 95.
Dec 15 19:19:48 svc.startd[9]: system/sysevent:default failed fatally: transitioned to maintenance (see 'svcs -xv' for details)
Requesting System Maintenance Mode
(See /lib/svc/share/README for more information.)
Console login service(s) cannot run

Root password for system maintenance (control-d to bypass):
single-user privilege assigned to /dev/console.
Entering System Maintenance Mode

Dec 15 19:20:45 su: 'su root' succeeded for root on /dev/console
Sun Microsystems Inc.   SunOS 5.10      Generic January 2005
# svcs -xv
svc:/system/sysevent:default (system event notification)
 State: maintenance since Thu Dec 15 19:19:48 2022
Reason: Start method exited with $SMF_EXIT_ERR_FATAL.
   See: http://sun.com/msg/SMF-8000-KS
   See: man -M /usr/share/man -s 1M syseventd
   See: /etc/svc/volatile/system-sysevent:default.log
Impact: 41 dependent services are not running:
        svc:/system/device/fc-fabric:default
        svc:/milestone/devices:default
        svc:/milestone/single-user:default
        svc:/milestone/sysconfig:default
        svc:/milestone/multi-user:default
        svc:/application/ciena/kls_server:default
        svc:/application/onecontrol/onedb_wd:default
        svc:/application/stosreg:default
        svc:/milestone/multi-user-server:default
        svc:/system/basicreg:default
        svc:/system/zones:default
        svc:/application/graphical-login/cde-login:default
        svc:/application/cde-printinfo:default
        svc:/system/system-log:default
        svc:/network/smtp:sendmail
        svc:/network/sendmail-client:default
        svc:/system/utmp:default
        svc:/network/ssh:default
        svc:/system/console-login:default
        svc:/network/inetd:default
        svc:/application/management/wbem:default
        svc:/system/sac:default
        svc:/system/sysidtool:system
        svc:/system/postrun:default
        svc:/system/filesystem/local:default
        svc:/system/filesystem/autofs:default
        svc:/system/dumpadm:default
        svc:/system/fmd:default
        svc:/system/cron:default
        svc:/network/nfs/status:default
        svc:/network/nfs/nlockmgr:default
        svc:/system/hotplug:default
        svc:/application/font/fc-cache:default
        svc:/system/boot-archive-update:default
        svc:/network/shares/group:default
        svc:/system/filesystem/volfs:default
        svc:/system/sysidtool:net
        svc:/network/rpc/bind:default
        svc:/platform/sun4v/efdaemon:default
        svc:/network/routing-setup:default
        svc:/system/picl:default

svc:/network/rpc/meta:default (SVM remote metaset services)
 State: uninitialized since Thu Dec 15 19:19:36 2022
Reason: Restarter svc:/network/inetd:default is not running.
   See: http://sun.com/msg/SMF-8000-5H
   See: man -M /usr/share/man -s 1M rpc.metad
Impact: 10 dependent services are not running:
        svc:/system/mdmonitor:default
        svc:/milestone/multi-user:default
        svc:/application/ciena/kls_server:default
        svc:/application/onecontrol/onedb_wd:default
        svc:/application/stosreg:default
        svc:/milestone/multi-user-server:default
        svc:/system/basicreg:default
        svc:/system/zones:default
        svc:/application/graphical-login/cde-login:default
        svc:/application/cde-printinfo:default

svc:/network/rpc/smserver:default (removable media management)
 State: uninitialized since Thu Dec 15 19:19:38 2022
Reason: Restarter svc:/network/inetd:default is not running.
   See: http://sun.com/msg/SMF-8000-5H
   See: man -M /usr/share/man -s 1M rpc.smserverd
Impact: 1 dependent service is not running:
        svc:/system/filesystem/volfs:default

svc:/network/ilomconfig-interconnect:default (ilomconfig-interconnect)
 State: offline since Thu Dec 15 19:19:47 2022
Reason: Start method is running.
   See: http://sun.com/msg/SMF-8000-C4
   See: man -M /usr/share/man -s 1M ilomconfig
   See: /etc/svc/volatile/network-ilomconfig-interconnect:default.log
Impact: This service is not running.
#



#
#
#
# cat /etc/svc/volatile/system-sysevent:default.log
[ Dec 15 19:19:34 Enabled. ]
[ Dec 15 19:19:47 Executing start method ("/lib/svc/method/svc-syseventd start") ]
WARNING: /usr/lib/sysevent/syseventd failed: exit status 1
[ Dec 15 19:19:48 Method "start" exited with status 95 ]
#

Welcome to the community.

I assume that this is a X86 (not SPARC) platform?

If so, the manufacturer (Oracle) has issued instructions to be followed in the event of this crash.

Ref:

Or have you already tried that and are telling us it didn't work?

This is a known problem (you're not on your own here) and you will find other helpful information if you search the web for the error "SMF_EXIT_ERR_FATAL"

1 Like

Thank you for the quick response.

It’s Solaris 10 SPARC installed on T5220 platform.
I have found several solutions for this issue, but they are not applicable to my case.

I have tried to expand root partition located on RAID1 mirror and then I got this issue.

So it's a ZFS filesystem, yes?

So what steps did you take to do that?

Did you have spare unallocated space on the RAID 1 volume immediately after the root filesystem location? AKAIK, you cannot dynamically expand a RAID 1 volume on the on-board RAID controller on the T5220.

Did you use Volume Manager to expand a ZFS root volume? AFAIK, you cannot do that whilst booted from that root filesystem. You would need to boot from CD/DVD to do that.

Please explain what you did.

Did you take a full backup before you did it?

I made it using the documentation:

https://docs.oracle.com/cd/E19683-01/806-6111/6jf2ve3hp/index.html

And after attach new slices to both submirrors boot to single user mode and made growfs -M / /dev/md/dsk/d10.
No, I don't have backup(((

Can I somehow roll back?

That link refers to Sun Volume Manager if that is what you are using? Being a T5220 it has an onboard hardware RAID controller configured using the 'raidctl' command so I thought it would be using that. Do you know which the system is using??

Please do

metastat d10

does it look okay?

If yes then...
If your growfs command did not give an error then I suspect a bug in the Solaris volume manager.
The following article suggests to run a forced fsck, and that would be

fsck -o f /

and, if successful, reboot.

https://comp.unix.solaris.narkive.com/triglVVF/solaris-10-failed-to-boot-console-login-service-cannot-run

Yes, SVM. I not used raidctl.

bash-3.2# metastat
d10: Mirror
    Submirror 0: d11
      State: Okay
    Submirror 1: d12
      State: Okay
    Pass: 1
    Read option: roundrobin (default)
    Write option: parallel (default)
    Size: 286596864 blocks (136 GB)

d11: Submirror of d10
    State: Okay
    Size: 286596864 blocks (136 GB)
    Stripe 0:
        Device     Start Block  Dbase        State Reloc Hot Spare
        c1t0d0s0          0     No            Okay   Yes
    Stripe 1:
        Device     Start Block  Dbase        State Reloc Hot Spare
        c1t0d0s1          0     No            Okay   Yes


d12: Submirror of d10
    State: Okay
    Size: 286596864 blocks (136 GB)
    Stripe 0:
        Device     Start Block  Dbase        State Reloc Hot Spare
        c1t1d0s0          0     No            Okay   Yes
    Stripe 1:
        Device     Start Block  Dbase        State Reloc Hot Spare
        c1t1d0s1          0     No            Okay   Yes


Device Relocation Information:
Device   Reloc  Device ID
c1t0d0   Yes    id1,sd@n5000cca00a1932fc
c1t1d0   Yes    id1,sd@n5000c50032388e17
bash-3.2#

I checked this solution, but i have another problem.
I can't remount / fs.

bash-3.2# fsck -o f /
** /dev/md/rdsk/d10
** Last Mounted on /
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3a - Check Connectivity
** Phase 3b - Verify Shadows/ACLs
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cylinder Groups
271689 files, 64952493 used, 76036421 free (123309 frags, 9489139 blocks, 0.1% fragmentation)
bash-3.2#

But

bash-3.2# mount -o rw,remount /dev/md/dsk/d10 /
mount: The state of /dev/md/dsk/d10 is not okay
        and it was attempted to be mounted read/write
mount: Please run fsck and try again
bash-3.2#

But after reboot system starting to maintenance mode


Boot device: rootdisk  File and args:
SunOS Release 5.10 Version Generic_150400-44 64-bit
Copyright (c) 1983, 2016, Oracle and/or its affiliates. All rights reserved.
Hostname: saromea4
Dec 16 16:55:06 svc.startd[9]: svc:/system/sysevent:default: Method "/lib/svc/method/svc-syseventd start" failed with exit status 95.
Dec 16 16:55:06 svc.startd[9]: system/sysevent:default failed fatally: transitioned to maintenance (see 'svcs -xv' for details)
Requesting System Maintenance Mode
(See /lib/svc/share/README for more information.)
Console login service(s) cannot run

Root password for system maintenance (control-d to bypass):

I think the problem is in the file system, but I can't find how to see what the problem is.

Please give output of

metastat d10
bash-3.2# metastat
d10: Mirror
    Submirror 0: d11
      State: Okay
    Submirror 1: d12
      State: Okay
    Pass: 1
    Read option: roundrobin (default)
    Write option: parallel (default)
    Size: 286596864 blocks (136 GB)

d11: Submirror of d10
    State: Okay
    Size: 286596864 blocks (136 GB)
    Stripe 0:
        Device     Start Block  Dbase        State Reloc Hot Spare
        c1t0d0s0          0     No            Okay   Yes
    Stripe 1:
        Device     Start Block  Dbase        State Reloc Hot Spare
        c1t0d0s1          0     No            Okay   Yes


d12: Submirror of d10
    State: Okay
    Size: 286596864 blocks (136 GB)
    Stripe 0:
        Device     Start Block  Dbase        State Reloc Hot Spare
        c1t1d0s0          0     No            Okay   Yes
    Stripe 1:
        Device     Start Block  Dbase        State Reloc Hot Spare
        c1t1d0s1          0     No            Okay   Yes


Device Relocation Information:
Device   Reloc  Device ID
c1t0d0   Yes    id1,sd@n5000cca00a1932fc
c1t1d0   Yes    id1,sd@n5000c50032388e17

After reboot:

# metastat d10
d10: Mirror
    Submirror 0: d11
      State: Needs maintenance
    Submirror 1: d12
      State: Needs maintenance
    Pass: 1
    Read option: roundrobin (default)
    Write option: parallel (default)
    Size: 286596864 blocks (136 GB)

d11: Submirror of d10
    State: Needs maintenance
    Invoke: metasync d10
    Size: 286596864 blocks (136 GB)
    Stripe 0:
        Device     Start Block  Dbase        State Reloc Hot Spare
        c1t0d0s0          0     No            Okay   Yes
    Stripe 1:
        Device     Start Block  Dbase        State Reloc Hot Spare
        c1t0d0s1          0     No            Okay   Yes


d12: Submirror of d10
    State: Needs maintenance
    Invoke: metasync d10
    Size: 286596864 blocks (136 GB)
    Stripe 0:
        Device     Start Block  Dbase        State Reloc Hot Spare
        c1t1d0s0          0     No            Okay   Yes
    Stripe 1:
        Device     Start Block  Dbase        State Reloc Hot Spare
        c1t1d0s1          0     No            Okay   Yes


Device Relocation Information:
Device   Reloc  Device ID
c1t0d0   Yes    id1,sd@n5000cca00a1932fc
c1t1d0   Yes    id1,sd@n5000c50032388e17
#

Some additional info about FS state:

# metastat d10
d10: Mirror
    Submirror 0: d11
      State: Needs maintenance
    Submirror 1: d12
      State: Needs maintenance
    Pass: 1
    Read option: roundrobin (default)
    Write option: parallel (default)
    Size: 286596864 blocks (136 GB)

d11: Submirror of d10
    State: Needs maintenance
    Invoke: metasync d10
    Size: 286596864 blocks (136 GB)
    Stripe 0:
        Device     Start Block  Dbase        State Reloc Hot Spare
        c1t0d0s0          0     No            Okay   Yes
    Stripe 1:
        Device     Start Block  Dbase        State Reloc Hot Spare
        c1t0d0s1          0     No            Okay   Yes


d12: Submirror of d10
    State: Needs maintenance
    Invoke: metasync d10
    Size: 286596864 blocks (136 GB)
    Stripe 0:
        Device     Start Block  Dbase        State Reloc Hot Spare
        c1t1d0s0          0     No            Okay   Yes
    Stripe 1:
        Device     Start Block  Dbase        State Reloc Hot Spare
        c1t1d0s1          0     No            Okay   Yes


Device Relocation Information:
Device   Reloc  Device ID
c1t0d0   Yes    id1,sd@n5000cca00a1932fc
c1t1d0   Yes    id1,sd@n5000c50032388e17
# iostat -En | grep -i hard
c1t0d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
c1t1d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
c0t0d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
c1t3d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
# metastat -p
d10 -m d11 d12 1
d11 2 1 c1t0d0s0 \
         1 c1t0d0s1
d12 2 1 c1t1d0s0 \
         1 c1t1d0s1
#

# metastat -c
d10              m  136GB d11 (maint) d12 (maint)
    d11          s  136GB c1t0d0s0 c1t0d0s1
    d12          s  136GB c1t1d0s0 c1t1d0s1
#
#
#
#
# prtvtoc /dev/rdsk/c1t0d0s1
* /dev/rdsk/c1t0d0s1 partition map
*
* Dimensions:
*     512 bytes/sector
*     848 sectors/track
*      24 tracks/cylinder
*   20352 sectors/cylinder
*   14089 cylinders
*   14087 accessible cylinders
*
* Flags:
*   1: unmountable
*  10: read-only
*
*                          First     Sector    Last
* Partition  Tag  Flags    Sector     Count    Sector  Mount Directory
       0      2    00          0 155611392 155611391
       1      3    00  155611392 130985472 286596863
       2      5    00          0 286698624 286698623
       7      0    00  286596864    101760 286698623
# prtvtoc /dev/rdsk/c1t1d0s1
* /dev/rdsk/c1t1d0s1 partition map
*
* Dimensions:
*     512 bytes/sector
*     848 sectors/track
*      24 tracks/cylinder
*   20352 sectors/cylinder
*   14089 cylinders
*   14087 accessible cylinders
*
* Flags:
*   1: unmountable
*  10: read-only
*
*                          First     Sector    Last
* Partition  Tag  Flags    Sector     Count    Sector  Mount Directory
       0      2    00          0 155611392 155611391
       1      3    00  155611392 130985472 286596863
       2      5    00          0 286698624 286698623
       7      0    00  286596864    101760 286698623
#
#
#
# prtvtoc /dev/rdsk/c1t0d0s0
* /dev/rdsk/c1t0d0s0 partition map
*
* Dimensions:
*     512 bytes/sector
*     848 sectors/track
*      24 tracks/cylinder
*   20352 sectors/cylinder
*   14089 cylinders
*   14087 accessible cylinders
*
* Flags:
*   1: unmountable
*  10: read-only
*
*                          First     Sector    Last
* Partition  Tag  Flags    Sector     Count    Sector  Mount Directory
       0      2    00          0 155611392 155611391
       1      3    00  155611392 130985472 286596863
       2      5    00          0 286698624 286698623
       7      0    00  286596864    101760 286698623
#
#
#
#
# prtvtoc /dev/rdsk/c1t1d0s0
* /dev/rdsk/c1t1d0s0 partition map
*
* Dimensions:
*     512 bytes/sector
*     848 sectors/track
*      24 tracks/cylinder
*   20352 sectors/cylinder
*   14089 cylinders
*   14087 accessible cylinders
*
* Flags:
*   1: unmountable
*  10: read-only
*
*                          First     Sector    Last
* Partition  Tag  Flags    Sector     Count    Sector  Mount Directory
       0      2    00          0 155611392 155611391
       1      3    00  155611392 130985472 286596863
       2      5    00          0 286698624 286698623
       7      0    00  286596864    101760 286698623
#

May be simple i need wait some time?

# metadetach d10 d11
metadetach: saromea4: d10: resync in progress

Yes, "resync in progress" looks good, should not be interrupted.
Check with metastat that there is some progress.

Please give output of

metadb

Does /etc/system have two entries that refer to the devices?

1 Like

Hello.
After 24 hours metastat in the same state:

# metadb
        flags           first blk       block count
     a m  p  luo        16              8192            /dev/dsk/c1t0d0s7
     a    p  luo        8208            8192            /dev/dsk/c1t0d0s7
     a    p  luo        16400           8192            /dev/dsk/c1t0d0s7
     a    p  luo        16              8192            /dev/dsk/c1t1d0s7
     a    p  luo        8208            8192            /dev/dsk/c1t1d0s7
     a    p  luo        16400           8192            /dev/dsk/c1t1d0s7
#
____________________________________________________________________________
# metastat d10
d10: Mirror
    Submirror 0: d11
      State: Needs maintenance
    Submirror 1: d12
      State: Needs maintenance
    Pass: 1
    Read option: roundrobin (default)
    Write option: parallel (default)
    Size: 286596864 blocks (136 GB)

d11: Submirror of d10
    State: Needs maintenance
    Invoke: metasync d10
    Size: 286596864 blocks (136 GB)
    Stripe 0:
        Device     Start Block  Dbase        State Reloc Hot Spare
        c1t0d0s0          0     No            Okay   Yes
    Stripe 1:
        Device     Start Block  Dbase        State Reloc Hot Spare
        c1t0d0s1          0     No            Okay   Yes


d12: Submirror of d10
    State: Needs maintenance
    Invoke: metasync d10
    Size: 286596864 blocks (136 GB)
    Stripe 0:
        Device     Start Block  Dbase        State Reloc Hot Spare
        c1t1d0s0          0     No            Okay   Yes
    Stripe 1:
        Device     Start Block  Dbase        State Reloc Hot Spare
        c1t1d0s1          0     No            Okay   Yes


Device Relocation Information:
Device   Reloc  Device ID
c1t0d0   Yes    id1,sd@n5000cca00a1932fc
c1t1d0   Yes    id1,sd@n5000c50032388e17
#


And /etc/system contain only mirror device

# cat /etc/system
*ident  "@(#)system     1.18    97/06/27 SMI" /* SVR4 1.5 */
*
* SYSTEM SPECIFICATION FILE
*

* moddir:
*
*       Set the search path for modules.  This has a format similar to the
*       csh path variable. If the module isn't found in the first directory
*       it tries the second and so on. The default is /kernel /usr/kernel
*
*       Example:
*               moddir: /kernel /usr/kernel /other/modules



* root device and root filesystem configuration:
*
*       The following may be used to override the defaults provided by
*       the boot program:
*
*       rootfs:         Set the filesystem type of the root.
*
*       rootdev:        Set the root device.  This should be a fully
*                       expanded physical pathname.  The default is the
*                       physical pathname of the device where the boot
*                       program resides.  The physical pathname is
*                       highly platform and configuration dependent.
*
*       Example:
*               rootfs:ufs
*               rootdev:/sbus@1,f8000000/esp@0,800000/sd@3,0:a
*
*       (Swap device configuration should be specified in /etc/vfstab.)



* exclude:
*
*       Modules appearing in the moddir path which are NOT to be loaded,
*       even if referenced. Note that `exclude' accepts either a module name,
*       or a filename which includes the directory.
*
*       Examples:
*               exclude: win
*               exclude: sys/shmsys



* forceload:
*
*       Cause these modules to be loaded at boot time, (just before mounting
*       the root filesystem) rather than at first reference. Note that
*       forceload expects a filename which includes the directory. Also
*       note that loading a module does not necessarily imply that it will
*       be installed.
*
*       Example:
*               forceload: drv/foo



* set:
set msgsys:msginfo_msgtql=1024
set msgsys:msginfo_msgmnb=65536
set shmsys:shminfo_shmmax=4294967295
set shmsys:shminfo_shmmni=400
set semsys:seminfo_semmns=2048
set semsys:seminfo_semmni=126
set semsys:seminfo_semmsl=260
set semsys:seminfo_semopm=100
set semsys:seminfo_semvmx=32767
set rlim_fd_cur=8193
set rlim_fd_max=8193
*       Set an integer variable in the kernel or a module to a new value.
*       This facility should be used with caution.  See system(4).
*
*       Examples:
*
*       To set variables in 'unix':
*
*               set nautopush=32
*               set maxusers=40
*
*       To set a variable named 'debug' in the module named 'test_module'
*
*               set test_module:debug = 0x13


set md:mirrored_root_flag = 1
* Begin MDD root info (do not edit)
rootdev:/pseudo/md@0:0,10,blk
* End MDD root info (do not edit)

I don't know what else I can do.
Because it all looks kind of weird.
I have another disk in this server.
Can I transfer everything to it and make the disk bootable and then, and then rebuild the raid?

# prtvtoc /dev/md/rdsk/d10
* /dev/md/rdsk/d10 partition map
*
* Dimensions:
*     512 bytes/sector
*     848 sectors/track
*      24 tracks/cylinder
*   20352 sectors/cylinder
*   14082 cylinders
*   14082 accessible cylinders
*
* Flags:
*   1: unmountable
*  10: read-only
*
*                          First     Sector    Last
* Partition  Tag  Flags    Sector     Count    Sector  Mount Directory
       0      0    00          0 286596864 286596863
# mount
/ on /pseudo/md@0:0,10,blk read/write/setuid/devices/rstchown/dev=154000a on Thu Jan  1 03:00:00 1970
/devices on /devices read/write/setuid/devices/rstchown/dev=5840000 on Fri Dec 16 17:57:44 2022
/system/contract on ctfs read/write/setuid/devices/rstchown/dev=5880001 on Fri Dec 16 17:57:44 2022
/proc on proc read/write/setuid/devices/rstchown/dev=58c0000 on Fri Dec 16 17:57:44 2022
/etc/mnttab on mnttab read/write/setuid/devices/rstchown/dev=5900001 on Fri Dec 16 17:57:44 2022
/etc/svc/volatile on swap read/write/setuid/devices/rstchown/xattr/dev=5940001 on Fri Dec 16 17:57:44 2022
/system/object on objfs read/write/setuid/devices/rstchown/dev=5980001 on Fri Dec 16 17:57:44 2022
/etc/dfs/sharetab on sharefs read/write/setuid/devices/rstchown/dev=59c0001 on Fri Dec 16 17:57:44 2022
/platform/sun4v/lib/libc_psr.so.1 on /platform/SUNW,SPARC-Enterprise-T5220/lib/libc_psr/libc_psr_hwcap2.so.1 read/write/setuid/devices/rstchown/dev=154000a on Fri Dec 16 17:58:01 2022
/platform/sun4v/lib/sparcv9/libc_psr.so.1 on /platform/SUNW,SPARC-Enterprise-T5220/lib/sparcv9/libc_psr/libc_psr_hwcap2.so.1 read/write/setuid/devices/rstchown/dev=154000a on Fri Dec 16 17:58:01 2022
/dev/fd on fd read/write/setuid/devices/rstchown/dev=5b40001 on Fri Dec 16 17:58:04 2022
/tmp on swap read/write/setuid/devices/rstchown/xattr/dev=5940002 on Fri Dec 16 17:58:05 2022
/var/run on swap read/write/setuid/devices/rstchown/xattr/dev=5940003 on Fri Dec 16 17:58:05 2022
#
#
#
# pwd
/
# touch test.txt
touch: cannot create test.txt: Read-only file system
#