Mistakenly renamed libdl.so: system got corrupted

Hi,

I am using Ubuntu 8.04 64-bit (Hardy Heron LTS Desktop edition) OS on a 64-bit intel hardware (x86_64). I have wrongly renamed the /lib64/libdl-2.7.so shared library file and now hardly few commands are working. My Gnome UI display has gone and I could not establish any new connection via SSH. I cannot run "ls" or "mv", or "cp" as these basic commands rely on libdl library for dlopen, dlclose operations etc,. It is a build server machine for our project and I am reluctant to reboot the system as it has to be run 24/7 (Hudson setup is there).

How I can resolve the system now? I guess recovery mode does not help in this case.
What if I boot via Ubuntu installation Live CD and mount the hard-disk and do the rename? Will it help?

What other options I have been left out?

This is really an urgent issue. Your help is highly solicited and appreciated.

At this point, I'd be booting the system from my handy dandy knoppix CD.

You going to need something running that allows you to rename the shares library file back to it's original name.

Boot another OS, mount broken system, and repair ..... That's what most of us would do.

I believe ubuntu comes with busybox. If ubuntu's version of busybox is compiled statically, it will provide a minimal shell and minimal toolset without any libraries. ...which is kind of the point of having it, so I hope so.

On my Gentoo system, /bin/bb is a minimal shell, and /bin/busybox can be used to call the builtin programs:

$ /bin/busybox
BusyBox v1.15.3 (2010-12-30 13:47:04 CST) multi-call binary
Copyright (C) 1998-2008 Erik Andersen, Rob Landley, Denys Vlasenko
and others. Licensed under GPLv2.
See source distribution for full notice.

Usage: busybox [function] [arguments]...
   or: function [arguments]...

	BusyBox is a multi-call binary that combines many common Unix
	utilities into a single executable.  Most people will create a
	link to busybox for each function they wish to use and BusyBox
	will act like whatever it was invoked as!

Currently defined functions:
	[, [[, acpid, addgroup, adduser, adjtimex, ar, arp, arping, ash, awk,
	basename, bb, bbconfig, bbsh, beep, blkid, brctl, bunzip2, bzcat,
	bzip2, cal, cat, catv, chat, chattr, chgrp, chmod, chown, chpasswd,
	chpst, chroot, chrt, chvt, cksum, clear, cmp, comm, cp, cpio, crond,
	cryptpw, cttyhack, cut, date, dd, deallocvt, delgroup, deluser, depmod,
	devmem, df, dhcprelay, diff, dirname, dmesg, dnsdomainname, dos2unix,
	dpkg-deb, du, dumpkmap, dumpleases, echo, ed, egrep, eject, env,
	envdir, envuidgid, ether-wake, expand, expr, false, fbset, fdflush,
	fdformat, fdisk, fgrep, find, findfs, flash_eraseall, flash_lock,
	flash_unlock, free, freeramdisk, fsck, fsck.minix, fsync, ftpd, fuser,
	getopt, getty, grep, gunzip, gzip, halt, hd, hdparm, head, hexdump,
	hostname, httpd, hwclock, id, ifconfig, ifdown, ifenslave, ifplugd,
	ifup, init, insmod, install, ionice, ip, ipaddr, ipcrm, ipcs, iplink,
	iproute, iprule, iptunnel, kbd_mode, kill, killall, killall5, klogd,
	last, length, less, linux32, linux64, linuxrc, ln, loadfont, loadkmap,
	logger, login, logread, losetup, lpq, lpr, ls, lsattr, lsmod, lzmacat,
	lzop, lzopcat, makedevs, makemime, man, md5sum, mdev, mesg, microcom,
	mkdir, mkdosfs, mkfifo, mkfs.minix, mkfs.vfat, mknod, mkpasswd, mkswap,
	mktemp, modprobe, more, mount, mountpoint, mt, mv, nameif, nc, netstat,
	nice, nmeter, nohup, nslookup, openvt, passwd, patch, pgrep, pidof,
	ping, ping6, pipe_progress, pivot_root, pkill, popmaildir, poweroff,
	printenv, printf, ps, pscan, pwd, raidautorun, rdate, rdev, readahead,
	readlink, readprofile, realpath, reboot, reformime, renice, reset,
	resize, rm, rmdir, rmmod, route, rtcwake, run-parts, runlevel, runsv,
	runsvdir, rx, script, scriptreplay, sed, sendmail, seq, setarch,
	setconsole, setfont, setkeycodes, setlogcons, setsid, setuidgid, sh,
	sha1sum, sha256sum, sha512sum, showkey, sleep, softlimit, sort, split,
	start-stop-daemon, stat, strings, stty, su, sum, sv, svlogd, swapoff,
	swapon, switch_root, sync, sysctl, syslogd, tac, tail, tar, tee,
	telnet, telnetd, test, tftp, tftpd, time, timeout, top, touch, tr,
	traceroute, true, tty, ttysize, tunctl, udhcpc, udhcpd, umount, uname,
	uncompress, unexpand, uniq, unix2dos, unlzma, unlzop, unzip, uptime,
	usleep, vconfig, vi, vlock, volname, watch, watchdog, wc, wget, which,
	who, whoami, xargs, yes, zcat, zcip

$ ls -l /bin/bb
lrwxrwxrwx 1 root root 7 Dec 30 13:48 /bin/bb -> busybox
$ ldd /bin/busybox
        not a dynamic executable
$ /bin/bb
~ # /bin/busybox touch file
~ # /bin/busybox ls -l file
-rw-r--r--    1 root     root            0 Dec 30 13:52 file
~ # /bin/busybox mv file file2
~ # /bin/busybox ls -l file2
-rw-r--r--    1 root     root            0 Dec 30 13:52 file2
~ # 

If you have a shell and nothing else, this might be able to save you. If not, you might be able to boot with "init=/bin/bb", maybe.

Otherwise, you're back at the livecd option.

Hi.

The busybox details on a 32-bit Ubuntu system I use in a virtual machine:

% file /bin/busybox
/bin/busybox: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), statically linked, for GNU/Linux 2.6.15, stripped

context of system:

(Versions displayed with local utility "version")
OS, ker|rel, machine: Linux, 2.6.32-26-generic, i686
Distribution        : Ubuntu 10.04.1 LTS (lucid) 
GNU bash 4.1.5
busybox - ( /bin/busybox Apr 22 2010 )

Some examples:

vm-ubuntu ~ % /bin/busybox ls -li t1
    340 -rw-r--r--    1 drl  drl       732 Jun 12  2010 t1
vm-ubuntu ~ % /bin/busybox mv t1 t2
vm-ubuntu ~ % /bin/busybox ls -li t2
    340 -rw-r--r--    1 drl  drl       732 Jun 12  2010 t2

Good luck ... cheers, drl

Hmmm....

I think most people don't have busybox installed on their ubuntu systems. In order to illustrate this, I will login to three ubuntu servers on the net and see if busybox has been installed:

First server (Dedicated Ubuntu server in EU):

user@dedicated:~# /bin/busybox
-bash: /bin/busybox: No such file or directory

Second server (Slicehost Ubuntu server in US, East Coast):

user@slicehost:~# /bin/busybox
-bash: /bin/busybox: No such file or directory

Third server (Linode Ubuntu server, Central US):

user@linode:~# /bin/busybox 
-bash: /bin/busybox: No such file or directory

---------- Post updated at 13:36 ---------- Previous update was at 13:25 ----------

Also, FWIW, I just did a standard apt-get install busybox install on a 64 bit Ubuntu server, and got an ELF dynamic installation:

user@linode:~# apt-get install busybox
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  busybox
0 upgraded, 1 newly installed, 0 to remove and 36 not upgraded.
Need to get 335kB of archives.
After this operation, 586kB of additional disk space will be used.
WARNING: The following packages cannot be authenticated!
  busybox
Install these packages without verification [y/N]? yes
Get:1 http://ftp2.de.debian.org lenny/main busybox 1:1.10.2-2 [335kB]
Fetched 335kB in 1s (307kB/s)   
Selecting previously deselected package busybox.
(Reading database ... 18074 files and directories currently installed.)
Unpacking busybox (from .../busybox_1%3a1.10.2-2_amd64.deb) ...
Processing triggers for man-db ...
Setting up busybox (1:1.10.2-2) ... 
user@linode:~# /bin/busybox
BusyBox v1.10.2 (Debian 1:1.10.2-2) multi-call binary
Copyright (C) 1998-2007 Erik Andersen, Rob Landley, Denys Vlasenko
and others. Licensed under GPLv2.
See source distribution for full notice.

Usage: busybox [function] [arguments]...
   or: function [arguments]...

        BusyBox is a multi-call binary that combines many common Unix
        utilities into a single executable.  Most people will create a
        link to busybox for each function they wish to use and BusyBox
        will act like whatever it was invoked as!

Currently defined functions:
        [, [[, adjtimex, arping, ash, awk, basename, brctl, bunzip2, bzcat, bzip2, cal, cat, chgrp, chmod, chown, chroot,
        chvt, clear, cmp, cp, cpio, cut, date, dc, dd, deallocvt, df, dirname, dmesg, dos2unix, du, dumpkmap, echo, egrep,
        env, expr, false, fgrep, find, fold, free, ftpget, ftpput, getopt, grep, gunzip, gzip, head, hexdump, hostid, hostname,
        httpd, id, ifconfig, ip, ipcalc, kill, killall, klogd, last, length, ln, loadfont, loadkmap, logger, logname, logread,
        losetup, ls, lzmacat, md5sum, mkdir, mkfifo, mknod, mktemp, more, mount, mt, mv, nameif, nc, netstat, nslookup, od,
        openvt, patch, pidof, ping, ping6, printf, ps, pwd, rdate, readlink, realpath, renice, reset, rm, rmdir, route, rpm,
        rpm2cpio, run-parts, sed, setkeycodes, sh, sha1sum, sleep, sort, start-stop-daemon, strings, stty, swapoff, swapon,
        sync, sysctl, syslogd, tac, tail, tar, tee, telnet, test, tftp, time, top, touch, tr, traceroute, true, tty, umount,
        uname, uncompress, uniq, unix2dos, unlzma, unzip, uptime, usleep, uudecode, uuencode, vi, watch, watchdog, wc, wget,
        which, who, whoami, xargs, yes, zcat
user@linode:~# ldd /bin/busybox
        linux-vdso.so.1 =>  (0x00007fff01dff000)
        libm.so.6 => /lib/libm.so.6 (0x00007fbf29a6d000)
        libc.so.6 => /lib/libc.so.6 (0x00007fbf296fd000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fbf29cf1000)
user@linode:~# file /bin/busybox
/bin/busybox: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), stripped

---------- Post updated at 13:45 ---------- Previous update was at 13:36 ----------

And just a note, I just checked and to get a static version of busybox on ubuntu, you need to

apt-get install busybox-static

Resulting in:

user@linode:~# file /bin/busybox
/bin/busybox: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, for GNU/Linux 2.6.8, stripped

user@linode:~# ldd /bin/busybox
        not a dynamic executable

Hi.

One needs to install / have installed the static version:

% apt-cache search busybox
busybox-initramfs - Standalone shell setup for initramfs
busybox-static - Standalone rescue shell with tons of builtin utilities
busybox - Tiny utilities for small and embedded systems
...

Juding by the maintainers, I'd guess that this is in a standard install:

Description: Standalone rescue shell with tons of builtin utilities
 BusyBox combines tiny versions of many common UNIX utilities into a single
 small executable. It provides minimalist replacements for the most common
 utilities you would usually find on your desktop system (i.e., ls, cp, mv,
 mount, tar, etc.).  The utilities in BusyBox generally have fewer options than
 their full-featured GNU cousins; however, the options that are included
 provide the expected functionality and behave very much like their GNU
 counterparts.
 .
 busybox-static provides you with a statically linked simple stand alone shell
 that provides all the utilities available in BusyBox. This package is
 intended to be used as a rescue shell, in the event that you screw up your
 system. Invoke "busybox sh" and you have a standalone shell ready to save
 your system from certain destruction. Invoke "busybox", and it will list the
 available builtin commands.
Original-Maintainer: Debian Install System Team <debian-boot@lists.debian.org>

I was not using server, but 32-bit Desktop. I don't recall installing busybox separately. If the OP has a more-or-less standard install, then I'd expect a statically-linked version. However, there are differences (OP vs. mine): 64 vs. 32, 8.04 vs. 10.04, so the only way to tell is for the OP to try it ... cheers, drl

I don't think it is in a standard install to have a static version of busybox installed. Even if busybox was installed, it is more than likely the dynamic, shared lib version was installed.

Frankly, I don't think it is good to "expect" anything; it is best to look and examine, not assume. You have one example of one desktop version. I just checked three servers, one we build directly from CD and two VM from two different companies. In addition, our three example versions were 64 bit and the original poster's version is 64 bit as well.

So far, the evidence and facts we know seem to suggest that the 64 bit versions of ubuntu do not have busybox installed by default.

Hi, Neo.

I expect the worst, but hope for the best :slight_smile:

The busybox-static utility is included in some meta-packages:

% apt-cache rdepends busybox-static
busybox-static
Reverse Depends:
 |e2fsck-static
 |e2fsck-static
 |debirf
  busybox
  busybox
 |bootcd-mkinitramfs
  ubuntu-standard

and the ubuntu-standard package is a meta-package:

% apt-cache show ubuntu-standard
Package: ubuntu-standard
Priority: standard
Section: metapackages
Installed-Size: 60
Maintainer: Matt Zimmerman <mdz@ubuntu.com>
Architecture: i386
Source: ubuntu-meta
Version: 1.197
Depends: aptitude, at, busybox-static, cpio, cron, dmidecode, dnsutils, dosfstools, ed, file, ftp, hdparm, info, iptables, language-selector-common, logrotate, lshw, lsof, ltrace, man-db, memtest86+, mime-support, parted, pciutils, popularity-contest, psmisc, rsync, strace, time, usbutils, wget
Recommends: apparmor-utils, apt-transport-https, bash-completion, command-not-found, friendly-recovery, iputils-arping, iputils-tracepath, irqbalance, manpages, mlocate, mtr-tiny, nano, ntfs-3g, openssh-client, plymouth, plymouth-theme-ubuntu-text, ppp, pppconfig, pppoeconf, tcpdump, telnet, ufw, update-manager-core, uuid-runtime, w3m
Filename: pool/main/u/ubuntu-meta/ubuntu-standard_1.197_i386.deb
Size: 30794
MD5sum: 1319acb51e9202c85521aa6651e9025a
SHA1: 1d1f38ebbd4cb4ed700d2a7d147780977bdbd89a
SHA256: 24b6b6790849e20eec300b4f0bc82f549153e424e4ed54babed95a7fa41ac88a
Description: The Ubuntu standard system
 This package depends on all of the packages in the Ubuntu standard system.
 This set of packages provides a comfortable command-line Unix-like
 environment.
 .
 It is also used to help ensure proper upgrades, so it is recommended that
 it not be removed.

So if the OP had a standard install, then busybox-static should be there -- 64-bit, 8.04 still being the differences ... cheers, drl

On 64-bit Fedora/Redhat/Centos, Busybox is statically linked.

Busybox is not installed by default, one has to manually install the RPM.

Well, we have 3 standard 64-bit Ubuntu installs, and none of them have busybox installed, either static or dynamic. One we installed from CD ourselves (as I recall, has been a while) and the other two are recent, 2010, VM installs.

---------- Post updated at 16:44 ---------- Previous update was at 16:41 ----------

And also,

user@linode:~# apt-cache show ubuntu-standard
Package: ubuntu-standard
Priority: standard
Section: metapackages
Installed-Size: 56
Maintainer: Matt Zimmerman <mdz@ubuntu.com>
Architecture: amd64
Source: ubuntu-meta
Version: 1.175
Depends: aptitude, at, cpio, cron, dmidecode, dnsutils, dosfstools, ed, file, ftp, hdparm, info, iptables, logrotate, lshw, lsof, ltrace, man-db, memtest86+, mime-support, parted, pciutils, popularity-contest, psmisc, rsync, strace, time, usbutils, wget
Recommends: apparmor-utils, apt-transport-https, bash-completion, command-not-found, friendly-recovery, iputils-arping, iputils-tracepath, manpages, mlocate, mtr-tiny, nano, ntfs-3g, openssh-client, ppp, pppconfig, pppoeconf, tcpdump, telnet, ufw, update-manager-core, uuid-runtime, w3m
Filename: pool/main/u/ubuntu-meta/ubuntu-standard_1.175_amd64.deb
Size: 29638
MD5sum: 31f2967ffcc4ca7a26c51efd3bccdcf7
SHA1: 6ff4eaeae9f541e9174fa47e3b1c32bf6014ef54
SHA256: 6a210bfba1e515a99f6ee6c28cfc428a0106902e0eb8bcdca50bb93f52e0108a
Description: The Ubuntu standard system
 This package depends on all of the packages in the Ubuntu standard system.
 This set of packages provides a comfortable command-line Unix-like
 environment.
 .
 It is also used to help ensure proper upgrades, so it is recommended that
 it not be removed.
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Origin: Ubuntu
Task: standard

No busybox ...

---------- Post updated at 16:46 ---------- Previous update was at 16:44 ----------

And one more... no busybox of any kind:

user@www:~# apt-cache show ubuntu-standard
Package: ubuntu-standard
Priority: standard
Section: metapackages
Installed-Size: 56
Maintainer: Matt Zimmerman <mdz@ubuntu.com>
Architecture: amd64
Source: ubuntu-meta
Version: 1.175
Depends: aptitude, at, cpio, cron, dmidecode, dnsutils, dosfstools, ed, file, ftp, hdparm, info, iptables, logrotate, lshw, lsof, ltrace, man-db, memtest86+, mime-support, parted, pciutils, popularity-contest, psmisc, rsync, strace, time, usbutils, wget
Recommends: apparmor-utils, apt-transport-https, bash-completion, command-not-found, friendly-recovery, iputils-arping, iputils-tracepath, manpages, mlocate, mtr-tiny, nano, ntfs-3g, openssh-client, ppp, pppconfig, pppoeconf, tcpdump, telnet, ufw, update-manager-core, uuid-runtime, w3m
Filename: pool/main/u/ubuntu-meta/ubuntu-standard_1.175_amd64.deb
Size: 29638
MD5sum: 31f2967ffcc4ca7a26c51efd3bccdcf7
SHA1: 6ff4eaeae9f541e9174fa47e3b1c32bf6014ef54
SHA256: 6a210bfba1e515a99f6ee6c28cfc428a0106902e0eb8bcdca50bb93f52e0108a
Description: The Ubuntu standard system
 This package depends on all of the packages in the Ubuntu standard system.
 This set of packages provides a comfortable command-line Unix-like
 environment.
 .
 It is also used to help ensure proper upgrades, so it is recommended that
 it not be removed.
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Origin: Ubuntu
Task: standard

---------- Post updated at 16:47 ---------- Previous update was at 16:46 ----------

Seems "Ubuntu Standard Install" has little meaning, LOL :smiley:

FWIW, on our end, I am talking:

user@www:~# cat /etc/issue
Ubuntu 9.10 \n \l

Hi, Neo.

Extracting the depends from each post, separating into lines, sorting, and running through comm yields:

				Version:1.175
Version:1.197
busybox-static,
language-selector-common,

suggesting that the recent version of a standard install has busybox-static, but not installs of versions as old as the ones into which you logged-in, and possibly not as old as that of the OP ... cheers, drl

( Again with the possible additional differences of Desktop vs Server, 32 vs. 64 )