How to fix and manage data in rescue mode?

baris35 · June 28, 2019, 3:36pm

Hello,
I am running ubuntu 14.04
My server has problems again. I need to manage system files in rescue mode, backup some files, edit /etc/crontab and rc.local files etc.

root@rescue:~# cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath] [faulty]
md2 : active raid1 sda2[0] sdb2[1]
      4193216 blocks [2/2] [UU]

md3 : active raid0 sda3[0] sdb3[1]
      7787813888 blocks 512k chunks

unused devices: <none>

root@rescue:~# fdisk -l

Disk /dev/ram0: 50 MiB, 52428800 bytes, 102400 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/ram1: 50 MiB, 52428800 bytes, 102400 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/ram2: 50 MiB, 52428800 bytes, 102400 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/ram3: 50 MiB, 52428800 bytes, 102400 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/ram4: 50 MiB, 52428800 bytes, 102400 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/ram5: 50 MiB, 52428800 bytes, 102400 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/ram6: 50 MiB, 52428800 bytes, 102400 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/ram7: 50 MiB, 52428800 bytes, 102400 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/ram8: 50 MiB, 52428800 bytes, 102400 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/ram9: 50 MiB, 52428800 bytes, 102400 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/ram10: 50 MiB, 52428800 bytes, 102400 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/ram11: 50 MiB, 52428800 bytes, 102400 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/ram12: 50 MiB, 52428800 bytes, 102400 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/ram13: 50 MiB, 52428800 bytes, 102400 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/ram14: 50 MiB, 52428800 bytes, 102400 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/ram15: 50 MiB, 52428800 bytes, 102400 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/sda: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 9F2BE807-05E5-4247-B2A3-E5C013A64B46

Device          Start        End    Sectors  Size Type
/dev/sda1        2048    1048575    1046528  511M EFI System
/dev/sda2     1048576    9435135    8386560    4G Linux RAID
/dev/sda3     9435136 7797250047 7787814912  3.6T Linux RAID
/dev/sda4  7797250048 7814025215   16775168    8G Linux swap

Disk /dev/sdb: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: F60F460A-BE4D-40EB-915B-BAE53BEF4758

Device          Start        End    Sectors  Size Type
/dev/sdb1        2048    1048575    1046528  511M EFI System
/dev/sdb2     1048576    9435135    8386560    4G Linux RAID
/dev/sdb3     9435136 7797250047 7787814912  3.6T Linux RAID
/dev/sdb4  7797250048 7814025215   16775168    8G Linux swap

Disk /dev/md3: 7.3 TiB, 7974721421312 bytes, 15575627776 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 524288 bytes / 1048576 bytes
Disk /dev/md2: 4 GiB, 4293853184 bytes, 8386432 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

How may I proceed step by step ?

Thank you
Boris

RudiC · June 28, 2019, 3:46pm

And WHAT EXACTLY is your problem? What errors do you encounter, where do you run into difficulties? What is your system disk? Show the mounted devices.

baris35 · June 28, 2019, 4:14pm

Hello Rudic,
CPU is 100% and server is not accessible via ssh in normal boot mode.

The problem started after nfs installation. I received a warning message from authorities emphasizing that nfs usage has some risks as attackers could sniff the system via udp ports. Not sure about that. Company replaced the motherboard in the morning which did not solve the issue. After an eight hours of intervention process they said, server is under attack.
I am still not sure about that so I am planning to edit my settings in rescue mode.

Regarding system disk details:

root@rescue:~# df -aTh
Filesystem                        Type        Size  Used Avail Use% Mounted on
none                              proc           0     0     0    - /proc
none                              sysfs          0     0     0    - /sys
devtmpfs                          devtmpfs     16G     0   16G   0% /dev
178.33.124.49:/home/pub/rescue.v8 nfs         1.8T  298G  1.4T  18% /nfs
tmpfs                             tmpfs        16G  1.7M   16G   1% /rw
aufs                              aufs         16G  1.7M   16G   1% /
178.33.124.49:/home/pub/pro-power nfs         1.8T  298G  1.4T  18% /power
178.33.124.49:/home/pub/commonnfs nfs         1.8T  298G  1.4T  18% /common
securityfs                        securityfs     0     0     0    - /sys/kernel/security
tmpfs                             tmpfs        16G     0   16G   0% /dev/shm
devpts                            devpts         0     0     0    - /dev/pts
tmpfs                             tmpfs        16G  9.7M   16G   1% /run
tmpfs                             tmpfs       5.0M     0  5.0M   0% /run/lock
tmpfs                             tmpfs        16G     0   16G   0% /sys/fs/cgroup
cgroup                            cgroup         0     0     0    - /sys/fs/cgroup/systemd
pstore                            pstore         0     0     0    - /sys/fs/pstore
cgroup                            cgroup         0     0     0    - /sys/fs/cgroup/cpuset
cgroup                            cgroup         0     0     0    - /sys/fs/cgroup/cpu,cpuacct
cgroup                            cgroup         0     0     0    - /sys/fs/cgroup/blkio
cgroup                            cgroup         0     0     0    - /sys/fs/cgroup/memory
cgroup                            cgroup         0     0     0    - /sys/fs/cgroup/devices
cgroup                            cgroup         0     0     0    - /sys/fs/cgroup/freezer
cgroup                            cgroup         0     0     0    - /sys/fs/cgroup/net_cls,net_prio
cgroup                            cgroup         0     0     0    - /sys/fs/cgroup/pids
systemd-1                         autofs         0     0     0    - /proc/sys/fs/binfmt_misc
hugetlbfs                         hugetlbfs      0     0     0    - /dev/hugepages
debugfs                           debugfs        0     0     0    - /sys/kernel/debug
mqueue                            mqueue         0     0     0    - /dev/mqueue
tmpfs                             tmpfs        16G   12K   16G   1% /tmp
configfs                          configfs       0     0     0    - /sys/kernel/config
fusectl                           fusectl        0     0     0    - /sys/fs/fuse/connections
rpc_pipefs                        rpc_pipefs     0     0     0    - /run/rpc_pipefs

Kind regards
Boris

baris35 · June 28, 2019, 6:12pm

Hello,

fdisk -l

Output:

Device          Start        End    Sectors  Size Type
/dev/sda1        2048    1048575    1046528  511M EFI System
/dev/sda2     1048576    9435135    8386560    4G Linux RAID
/dev/sda3     9435136 7797250047 7787814912  3.6T Linux RAID
/dev/sda4  7797250048 7814025215   16775168    8G Linux swap

Disk /dev/sdb: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: F60F460A-BE4D-40EB-915B-BAE53BEF4758

Device          Start        End    Sectors  Size Type
/dev/sdb1        2048    1048575    1046528  511M EFI System
/dev/sdb2     1048576    9435135    8386560    4G Linux RAID
/dev/sdb3     9435136 7797250047 7787814912  3.6T Linux RAID
/dev/sdb4  7797250048 7814025215   16775168    8G Linux swap

Disk /dev/md3: 7.3 TiB, 7974721421312 bytes, 15575627776 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 524288 bytes / 1048576 bytes
Disk /dev/md2: 4 GiB, 4293853184 bytes, 8386432 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Then I run:

mount /dev/md3 /mnt/

now I can see my files inside mnt . So when I edit related files, I hope it will be auto-edited after reboot in normal mode with one exception.
I do not know how to edit " crontab -e " in this environment.
I had created some rules with that command. Strongly believe that to surf in google will give me an idea.

Last edit:
To stop crontab rules:

update-rc.d cron remove

Then, I turned server to normal boot
After restart, in command line:

update-rc.d cron defaults

After I edit some files, I have changed the boot mode. Now system is up&running.

Kind regards
Boris

MadeInGermany · June 29, 2019, 6:45am

Thanks for sharing this.
If you know where the crontab file is located - usually under /var/spool/cron/, in your case /mnt/var/spool/cron/ - then you can directly edit it. (That method is not good on a live system because the running cron daemon would not be notified.)
Furthermore, a chroot /mnt crontab -e might work.