Unable to preserve hard links. Why?

Hi,

I'm trying to create a Makefile that would automate remastering Knoppix distribution.

As a part of the process I am mounting using linux cloop device a compressed filesystem and copy the content out of it to separate dir. However during that process I need to preserve hard links and it seems it is not working.

I've tried using:

  • cp -a --preserve=all
  • rsync -a -H
  • tar | tar version

All of them are making original hard-linked files a separate files.

The lines in the code are:

	#sudo nice tar -C $(NAME).cloop -cf - . | sudo nice tar -C $(NAME).tree -xvpf -
	sudo cp -av --preserve=all $(NAME).cloop $(NAME).tree
	#sudo rsync -x -a -H --progress --delete $(NAME).cloop/ $(NAME).tree

Can you please advise what can be wrong?

Below is the full content of the Makefile

SITE=ftp://sunsite.icm.edu.pl/pub/Linux/distributions/knoppix/
NAME=KNOPPIX_V6.2.1CD-2010-01-31-EN
PWD=$(shell pwd)

.PHONY: all clean test test-orig

all: $(NAME)-cherry88.iso

$(NAME).iso:
	wget -c -O $(NAME).iso.download $(SITE)/$(NAME).iso && mv $(NAME).iso.download $(NAME).iso

clean:
	if [ ! -z "`mount | grep "$(PWD)/$(NAME).cloop"`" ]; then sudo umount $(NAME).cloop && rm -f $(NAME).cloop.mount || test 1=1; fi
	if [ ! -z "`mount | grep "$(PWD)/$(NAME)"`" ]; then sudo umount $(NAME) && rm -f $(NAME).mount || test 1=1; fi

$(NAME).mount: $(NAME).iso
	mkdir -p $(NAME)
	if [ -z "`mount | grep "$(PWD)/$(NAME)"`" ]; then sudo mount -o loop $(NAME).iso $(NAME); fi
	touch $(NAME).mount

$(NAME).cloop.iso: $(NAME).mount
	nice extract_compressed_fs $(NAME)/KNOPPIX/KNOPPIX - > $(NAME).cloop.iso
	touch $(NAME).cloop.iso

$(NAME).cloop.mount: $(NAME).cloop.iso
	mkdir -p $(NAME).cloop
	if [ -z "`mount | grep "$(PWD)/$(NAME).cloop"`" ]; then sudo mount -o loop $(NAME).cloop.iso $(NAME).cloop; fi
	touch $(NAME).cloop.mount

$(NAME).cdtree.touch: $(NAME).mount
	sudo mkdir -p $(NAME).cdtree
	sudo rm -Rf $(NAME).cdtree
	#sudo nice tar -C $(NAME) --exclude=KNOPPIX/KNOPPIX -cf - . | sudo nice tar -C $(NAME).cdtree -xvpf -
	sudo nice cp -av --preserve=all $(NAME) $(NAME).cdtree
	#sudo rsync -x -a -H --progress --delete $(NAME)/ $(NAME).cdtree
	touch $(NAME).cdtree.touch

$(NAME).tree.touch: $(NAME).cloop.mount
	sudo mkdir -p $(NAME).tree
	sudo rm -Rf $(NAME).tree
	#sudo nice tar -C $(NAME).cloop -cf - . | sudo nice tar -C $(NAME).tree -xvpf -
	sudo cp -av --preserve=all $(NAME).cloop $(NAME).tree
	#sudo rsync -x -a -H --progress --delete $(NAME).cloop/ $(NAME).tree
	touch $(NAME).tree.touch

$(NAME).cdtree/KNOPPIX/KNOPPIX: $(NAME).tree.touch $(NAME).cdtree.touch $(NAME).cdtree.mods
	#nice -n 5 create_compressed_fs - 65536 | \
	#
	sudo rm -f $(NAME).cdtree/KNOPPIX/KNOPPIX || test 1=1
	nice mkisofs -R -U -V "KNOPPIX.net filesystem" -publisher "KNOPPIX www.knoppix.net" -v  \
		-hide-rr-moved -cache-inodes -pad $(NAME).tree | \
	sudo nice -n 5 create_compressed_fs -B 65536 -f tmp - $(NAME).cdtree/KNOPPIX/KNOPPIX

$(NAME)-cherry88.iso: $(NAME).cdtree/KNOPPIX/KNOPPIX
	sudo chmod a+w $(NAME).cdtree/KNOPPIX/sha1sums
	cd $(NAME).cdtree; sudo find -type f -not -name sha1sums -not -name boot.cat -not -name isolinux.bin \
		   -exec sha1sum '{}' \; > KNOPPIX/sha1sums
	sudo chmod a-w $(NAME).cdtree/KNOPPIX/sha1sums
	sudo nice mkisofs -pad -l -r -J -v -V "KNOPPIX" -no-emul-boot -boot-load-size 4 \
		-boot-info-table -b boot/isolinux/isolinux.bin -c boot/isolinux/boot.cat \
		-hide-rr-moved -o $(NAME)-cherry88.iso $(NAME).cdtree

$(NAME).cloop.md5sums: $(NAME).cloop.mount
	cd $(NAME).cloop && sudo nice find . -type f -print0 | nice sort -z | sudo nice xargs -0 ls -l | awk '{ print $$1" "$$2" "$$3" "$$4" "$$8 }' > ../$(NAME).cloop.ls
	cd $(NAME).cloop && sudo nice find . -type f -print0 | nice sort -z | sudo nice xargs -0 md5sum > ../$(NAME).cloop.md5sums

$(NAME).md5sums: $(NAME).mount
	cd $(NAME) && sudo nice find . -type f -print0 | nice sort -z | sudo nice xargs -0 ls -l | awk '{ print $$1" "$$2" "$$3" "$$4" "$$8 }' > ../$(NAME).ls
	cd $(NAME) && sudo nice find . -type f -print0 | nice sort -z | sudo nice xargs -0 md5sum > ../$(NAME).md5sums

$(NAME).cdtree.md5sums: $(NAME).cdtree.touch
	cd $(NAME).cdtree && sudo nice find . -type f -print0 | nice sort -z | sudo nice xargs -0 ls -l | awk '{ print $$1" "$$2" "$$3" "$$4" "$$8 }'> ../$(NAME).cdtree.ls
	cd $(NAME).cdtree && sudo nice find . -type f -print0 | nice sort -z | sudo nice xargs -0 md5sum > ../$(NAME).cdtree.md5sums

$(NAME).tree.md5sums: $(NAME).tree.touch
	cd $(NAME).tree && sudo nice find . -type f -print0 | nice sort -z | sudo nice xargs -0 ls -l | awk '{ print $$1" "$$2" "$$3" "$$4" "$$8 }' > ../$(NAME).tree.ls
	cd $(NAME).tree && sudo nice find . -type f -print0 | nice sort -z | sudo nice xargs -0 md5sum > ../$(NAME).tree.md5sums


# MODYFIKACJE
$(NAME).cdtree.mods: $(NAME).cdtree.touch
	# no 3d
	#	sudo sed -i '' -e 's/window_manager=compiz/window_manager=openbox-lxde/g' $(NAME).tree/etc/xdg/lxsession/LXDE/desktop.conf
	# touch
	touch $(NAME).cdtree.mods

test: $(NAME)-cherry88.iso
	qemu -m 512 -cdrom $(NAME)-cherry88.iso -boot d

test-orig:
	qemu -m 512 -cdrom $(NAME).iso -boot d

changes: $(NAME).cloop.md5sums $(NAME).tree.md5sums
	diff -u $(NAME).cloop.md5sums $(NAME).tree.md5sums > changes; test 1=1
	diff -u $(NAME).cloop.ls $(NAME).tree.ls > changes-ls; test 1=1

changes-cd: $(NAME).md5sums $(NAME).cdtree.md5sums
	diff -u $(NAME).md5sums $(NAME).cdtree.md5sums >  changes-cd
	diff -u $(NAME).ls $(NAME).cdtree.ls > changes-cd-ls

The command:

make changes

will work for a longer moment and produce changes file which is a diff of mounted cloop device and copyied tree content.

tar preserves hardlinked files fine:

$ touch a
$ ln a b
$ tar -cf hard.tar [ab]
$ mkdir wtf
$ tar -C wtf -xf ./hard.tar
$ ls -li wtf/
total 0
1920716 -rw-r--r-- 2 tyler users 0 Aug 26 12:23 a
1920716 -rw-r--r-- 2 tyler users 0 Aug 26 12:23 b

I think part of your problem with tar may be you are trying to use the -C option when creating the tarball. -C does nothing when creating the file, you have to cd. Meaning, your hardlinks may be ending up a directory or two deeper than you thought they were.

actually hard link is like directory record..so inode nums can change when in new dir..

 
touch a
ln a b
tar -cvf hard.tar *
a
b
# ls -li *
65194 -rw-r--r-- 2 root root 10240 Jun 25 03:20 a
65194 -rw-r--r-- 2 root root 10240 Jun 25 03:20 b
65423 -rw-r--r-- 1 root root 20480 Jun 25 03:22 hard.tar
rsync -H -avve "ssh localhost" /xx/dirrss/test1 /xx/dirrss/oba/*

building file list ...
done
delta-transmission disabled for local transfer or --whole-file
"a" is a hard link
b
hard.tar
test1/
a => b
total: matches=0  hash_hits=0  false_alarms=0 data=30720
...............
....................
ls -li test1/
total 48
65485 -rw-r--r-- 2 root root 10240 Jun 25 03:20 a
65485 -rw-r--r-- 2 root root 10240 Jun 25 03:20 b
65497 -rw-r--r-- 1 root root 20480 Jun 25 03:22 hard.tar
65484 drwxr-xr-x 2 root root  4096 Jun 25 03:22 test1

at the result pair of files are same data and same inode and hard link preserve :wink:

Guys, I know how it should work, but it ain't happening. Possibly due to the fact that the source filesystem is RO cloop device based.

You may want to copy a Makefile, run make changes yourself and see wha I mean.

The way you were using tar to create archives was completely wrong: It would create an archive with files in the wrong place, or even the wrong files. Have you tried it using my suggestions?

Could you give an ls -li listing of the input files you want preserved, and the output files not properly linked?

I have three methods that should work. Non of them is. I've just rerun the whole script from the scratch with tar method enabled and it does work as I want except preserving the hardlinks.

$ ls -li */bin/dnsdomainname
   10419 -rwxr-xr-x 4 root root 12992 2010-01-19 10:40 KNOPPIX_V6.2.1CD-2010-01-31-EN.cloop/bin/dnsdomainname
10641687 -rwxr-xr-x 1 root root 12992 2010-01-19 10:40 KNOPPIX_V6.2.1CD-2010-01-31-EN.tree/bin/dnsdomainname

You're trying to create hardlinks between different filesystems!?

That won't ever work, period. You'll have to find a different way to do whatever you're trying to do.

Oh /dev/full. This thread is not any help because everybody thinks that I've did something plainly stupid. I'm feeling like I've just called an IT support. :smiley:

Please, before posting anything stop for a moment and think what I'm doing or just fire the script, give it 30 minutes and see the results.

I'm trying to copy filesystem from the read-only compressed loop device to normal filesystem to be able to modify it and then compress it to replace the old one. That's what Knoppix remastering is all about. I have to preserve all the details about the filesystem i.e. recreate hard link structure from one filesystem on the other filesystem. The ls -li that I've pasted is the same file in two different filesystem to show that source had hardlinks and the destination have lost them.

This is obviously possible -- if the rsync is able to preserve hardlinks between different hosts it should be able to do it on between filestems on the same host.

For some strange reason this is not working using neither of three methods that I'm aware that should do the job and I'm puzzled what is wrong. I've spent almost a workday to create the script, test it and this seems to be a last piece that stops it from fully working.

I have a test suite and a part of it is counting md5sums from both filesystems and the diff is empty - meaning that content of every file between filesystem is exactly the same. However, the diff between permissions and hard link numbers shows that every file that was refering to shared (hard-linked) inode is now a separate inode which is not what I wanted. So I'm quite sure that the tar syntax is somehow working but is not really important because all the three methods are doing the same thing wrong which is really suspicious and I'm running out of ideas.

So, please stop trying to find typos and stupid bugs and start be creative. :slight_smile:

---------- Post updated at 10:03 AM ---------- Previous update was at 09:02 AM ----------

It seems I've found it!

CD-record mailing list

But what can I do about it?

We only know as much as you tell us, and a makefile without the files it works on is not a lot. A silly makefile doesn't tell me a thing about your file layout, I still don't know after asking three times if you ever fixed your use of tar, in fact the most I've gotten from you -- beyond abuse -- is one ls readout of only a quarter of the files I asked for. A real listing of all the hardlinked input (and output) files could have been helpful(and would still be), but you treat it like an RTFM.

If you're trying to create an ISO with rock-ridge hard links, yes, that is a problem. Is the input truly Rock Ridge? Usually on livecd's I see compressed filesystems done as cramfs and the like.

Preserving hard links doesn't seem to matter to me when creating a read-only filesystem. The permissions, contents, and owners are still all as they should be, and nothing will ever be able to change it.

There is no filesystem layout beside that Makefile. It downloads everything.

I've found the root of the problem with little help from here Unable to preserve hard links. Why? - Super User .

Thanks anyway.