Hi,
I have a production solaris 10 SPARC system \(portal\). Yesterday legato/Networker gave an I/O Error on one of the files on its SAN mounted disk.
I went to that particular file on the system, did an ls and it showed the file. However, ls -l did not work and it said IO error.
What is the solution to fix this IO error. fsck?
can you post relevant dmesg output?
Nov 22 18:03:07 plumcrt01 ufs: [ID 879645 kern.notice] NOTICE: /opt/luminis: unexpected free inode 8677, run fsck(1M) -o f
Nov 22 18:04:37 plumcrt01 last message repeated 3 times
Nov 23 10:35:22 plumcrt01 ufs: [ID 879645 kern.notice] NOTICE: /opt/luminis: unexpected free inode 8677, run fsck(1M) -o f
Nov 23 10:36:17 plumcrt01 last message repeated 2 times
Nov 23 10:36:53 plumcrt01 ufs: [ID 879645 kern.notice] NOTICE: /opt/luminis: unexpected free inode 8677, run fsck(1M) -o f
Nov 23 11:03:40 plumcrt01 ufs: [ID 879645 kern.notice] NOTICE: /opt/luminis: unexpected free inode 8677, run fsck(1M) -o f
Nov 23 11:05:45 plumcrt01 last message repeated 3 times
Nov 23 12:33:28 plumcrt01 ufs: [ID 879645 kern.notice] NOTICE: /opt/luminis: unexpected free inode 8677, run fsck(1M) -o f
Nov 23 20:07:38 plumcrt01 ufs: [ID 879645 kern.notice] NOTICE: /opt/luminis: unexpected free inode 8677, run fsck(1M) -o f
Nov 23 20:10:27 plumcrt01 last message repeated 3 times
Nov 24 03:00:12 plumcrt01 ufs: [ID 879645 kern.notice] NOTICE: /opt/luminis: unexpected free inode 8612, run fsck(1M) -o f
Nov 24 08:58:32 plumcrt01 ufs: [ID 879645 kern.notice] NOTICE: /opt/luminis: unexpected free inode 8677, run fsck(1M) -o f
Nov 24 18:01:53 plumcrt01 ufs: [ID 879645 kern.notice] NOTICE: /opt/luminis: unexpected free inode 8612, run fsck(1M) -o f
Nov 24 18:04:59 plumcrt01 last message repeated 4 times
Nov 25 10:01:13 plumcrt01 ufs: [ID 879645 kern.notice] NOTICE: /opt/luminis: unexpected free inode 8677, run fsck(1M) -o f
What was the original error? is the message from 11/22 the first one logged?
unmount, fsck and remount
We saw the first error via legato. On the server, the following:
[root@plumcrt01 /opt/luminis/products/ds/slapd-cp/locks/server] # ls
1198
[root@plumcrt01 /opt/luminis/products/ds/slapd-cp/locks/server] # ls -l
./1198: I/O error
total 0
The second error via legato was an ldif data file. When we cated the file, it had garbage values at the end
---------- Post updated 11-26-10 at 10:13 AM ---------- Previous update was 11-25-10 at 12:02 PM ----------
Well, ran fsck it found some errors and fixed it. Everything appears to be fine, however, the cause of the IO error is still a mystery.
First Error (occurred during movement)
Nov 12 21:57:19 plumcrt01 savecore: [ID 570001 auth.error] reboot after panic: free: freeing free frag, dev:0x14800000008, blk:33717, cg:2, ino:13487, fs:/opt/luminis
Nov 12 22:46:02 plumcrt01 savecore: [ID 570001 auth.error] reboot after panic: free: freeing free frag, dev:0x14800000008, blk:33717, cg:2, ino:13487, fs:/opt/luminis
Nov 12 22:46:50 plumcrt01 /opt/luminisShared/common/imq/instances/imqbroker/lock
Nov 12 23:51:29 plumcrt01 savecore: [ID 570001 auth.error] reboot after panic: free: freeing free frag, dev:0x14800000000, blk:1416, cg:5, ino:30709, fs:/opt/luminisShared
my best guess is it happened when the server crashed. you should find out why the system crashed.
lol, got mail from SUN stating that it might be because I did not connect the fiber cables properly.
Any way all is good till now. I am assuming that /opt being 100% full might be the cause since, the SAN partition was mounted on /opt as /opt/luminis