so i was testing something on a test box running linux. i manually vi'ed the /var/log/messages file. and i noticed, the file immediately stopped being updated.
it wasn't until i restarted the syslog process that events started being recorded in it again.
so that tells me, the syslog process knows when that file is modified by a process other than itself, so that prevents it from working.
can someone please show me how i can apply this same tactic to a file of my own?
for instance, if i have a file called /home/skysmart/boldness.txt. I want to know whenever this file is "vi'ed" by a real life user or if someone other than a specific process adds contents to it, as in ">>" /">" .
I'm guessing that you're really saying that new lines being written to the log file did not show up in your vi editing buffer until you closed vi and reloaded the file. It is extremely unlikely that anything stopped writing to a file because vi had that file open.
When you edit a file using vi (or ex or ed or emacs or any other editor) you load a copy of that file into a buffer. You edit the buffer; not the underlying file. If you want to see recent additions to the file while you are editing it, you need to reload the buffer from the file. In vi, the command to reload the buffer is :e . If you have changed the buffer and have not written the updates back to another file; you'll need to use :e! . If you change the file and write those changes back to the file while some other process is writing to it, whether the changes you made to the file or additions added by that other process or some combination of those changes and additions will appear in the file after you exit vi is unspecified.
i'm pretty familiar with vi. what i was saying was that, apparently, when you vi a file thats being written to by syslog, syslog stops updating that file. yeah, when i vied that filed and saved it, i saw my additions in there. that wasn't the issue. the problem was, after my additions, i expected to have syslog continue to update the log file as usual. but no. it just stopped updating, until i restarted it.
this happened on linux red hat 6.2. i'm guessing most people aren't aware of this?
Try the following, and you will see that while you are editing the file, syslog continues to update the log file.
go to the end of the file in vi and see what is there.
wait a few minutes to let something loggable happen.
in a separate window, do a tail on the log file.
the results will be different, because syslog continues to update the file.
Here is another demo:
$ date > date.txt
$ vi date.txt # will see single line
$ date >> date.txt # in another window, while vi open
$ cat date.txt
Wed Apr 17 13:56:14 PDT 2013
Wed Apr 17 13:56:27 PDT 2013
When you saved the log file from within vi, you wiped out the changes that syslog had made. Normally, not a good idea to make changes to syslog log file, unless some over-riding benefit.
syslogd is not "prevented from working", and /var/log/messages is not "stopped being updated".
When you edit that file, AND save it, a new copy is created, of which syslogd does not know. It happily keeps logging to the old file, accessed via inode number, as you can see using the lsof command.
Send a HUP signal to the process to close and reopen all files.
That is incorrect. Whatever changes syslog had made are still there. When saving, vim unlinks the original file and creates a new one. syslog is still working with the original, as RudiC points out.
While the original file is no longer reachable through the filesystem, any process with an open descriptor to the original's contents can still read/write from/to it. Only when the last of those descriptor's is closed will the kernel remove the unreachable file.
To demonstrate this, let's use sed to delete all empty lines from a file, without using a temp file (-i, even when available, uses a temp file):
{ rm file; sed '/./!d' > file; } < file
1) { ... } < file opens a descriptor to the original file contents. As long as this descriptor is open, the original file's contents are accessible.
2) rm file unlinks the file. At this point, the file is no longer reachable through the filesystem hierarchy.
3) The redirection in sed ... > file creates a new file and redirects stdout to it. sed inherits its stdin descriptor from the parent sh, through which it has access to the original file's content.
Such "cleverness" is usually a very bad idea. Not creating the temp file means that, should the system fail at just the right time, you could be left without a reachable version of the data. And even though a temp file isn't created, the amount of storage required is the same (the original version of the file and the version without empty lines will coexist for some finite amount of time).
If instead an editor which does not unlink the original file were used, e.g. ed, there would then be the problem of multiple unsynchronized writers. The resulting file's contents will be some indeterminate, interleaved melange of data written by multiple processes.
Addendum to the unlinked "old file, accessed via inode number": You could try and undel ete the old file using (on linux systems!) debugfs :
man debugfs:
The chance of finding all blocks intact is high as the file as such is still there and held open by the syslogd process. In fact, thinking twice, it should be 100%
When saving, vim unlinks the original file and creates a new one.
I'm trying to understand what you are saying. I think my assertion is very simple:
$ date > date.txt
$ vi date.txt # different window, shows one line
$ date >> date.txt
$ cat date.txt
Thu Apr 18 11:46:30 PDT 2013
Thu Apr 18 11:47:57 PDT 2013
Save and exit vi
$ cat date.txt
Thu Apr 18 11:46:30 PDT 2013
When you saved the log file from within vi,
you wiped out the changes that syslog had made.
Is this incorrect because of something about vi, or something about syslog?
Nothing to do with vi nor syslog; it's just the way the file system works.
If an editor opens a file, it reads its momentary contents, which you then can edit. During that time, another process can modify (= add to) the original file, of which the editor is not aware. When saving your edits, a NEW file is created, and the old one unlinked, which would normally mean: deleted. Compare inode numbers before and after with e.g. ls -i .
Should that other process have the old file open at that time, the link between file name and inode will be lost, but the file still exists, and can be seen by e.g. lsof. linux's debugfs has a command: undel , to recreate the link between inode and file name.
The description of how unlink() works on a file is correct. The statement that vim unlink()s the file is an optimization used by vim under certain circumstances. It isn't clear to me that the standards allow this optimization in the case being discussed in this thread. The standard clearly does not allow this behavior when there are multiple links to the file being edited and vim doesn't unlink() (or more likely rename(tmp_file, original_file)) the original file when there are multiple links.
To verify this, execute the following commands:
$ touch x1 x2
$ ln x2 x3
$ ls -li x[1-3]
You will note at this point that there are two links to x2 and x3 and that they have the same i-node number (in POSIX referred to as the file serial number).
If you then edit x1 with vim to add some text and save the file before exiting vim and rerun the ls, you will indeed see that the i-node number of x1 has changed.
However, if you edit x2 with vim, add some text, save the file before exiting vim, and rerun the ls; you will see that x2 and x3 both still have the same (unchanged) i-node number but the added text is now in this file and visible not matter which link is used to access the file. So, vim only performs this optimization when there is only one link to the file. In my personal opinion, that fact that vim ever does this is a bug.
The standards have a detailed description of how file handles have to be shared between and within processes to get the desired results; and when two processes are updating a file at the same time (as is being described in this thread) the results are undefined. For those of you with access to the 2008 Issue of the POSIX Standards (or the 2013 edition including technical corrigendum #1 to be released tomorrow) this text appears in the System Interfaces volume of the standard in subclause 2.5.1 titled Interaction of File Descriptors and Standard I/O Streams.
On linux, if your kernel supports inotify, you can use inotifywait to watch a file. Upon notification of a process opening the file, you use lsof to see who came calling. There's a race, though. It's possible that the process may have called close() by the time lsof gets around to looking for it.
If that's unacceptable, you can almost certainly accomplish this task with selinux (something I know almost nothing about).
$ date > date.txt
$ cat date.txt
Thu Apr 18 12:49:28 PDT 2013
$ ls -i date.txt
786451 date.txt
$ vi date.txt # different window, shows one line
$ date >> date.txt
$ cat date.txt
Thu Apr 18 12:49:28 PDT 2013
Thu Apr 18 12:50:31 PDT 2013
Save and exit vi
$ cat date.txt
Thu Apr 18 12:49:28 PDT 2013
$ ls -i date.txt
786451 date.txt
The inode did not change. Whenever I read one thing in a man page, and see another thing in practice, I tend to believe what I see. But I'm willing to be educated.
Thank you very much for elaborating on vim's behavior and thanks also for the POSIX update information.
Regards,
Alister
---------- Post updated at 04:03 PM ---------- Previous update was at 04:01 PM ----------
There are many vi implementations. Yours may not have changed the inode, but at least one very popular implementation will under certain circumstances (as detailed in Don Cragun's post).
I would say it's as clear as mud at this point. I examined the inode. It did not change. It doesn't help much to cite long-winded official explanations. I am willing to believe the inode could change when vi writes out the file. Can someone provide an example of that happening?
The previous post from RudiC said "Nothing to do with vi nor syslog; it's just the way the file system works." Nobody contradicted that. My example seems to contradict the assertion.
I have to admit that Hanson44 is right - some editors reuse inodes (which, BTW, was not the topic of this thread).
I'm using joe ; its reusing inodes as well, which I never noticed before. But, this guy is smart: if you edit a file, and another process adds/edits as well, it tells you before allowing to edit:
Notice: File on disk changed! (hit ^C to continue)
and, when trying to finally write the file, it issues:
$ echo foo > file
$ cat file
foo
$ ls -i file
472445 file
$ # vim 7.0 opens file in a different terminal, showing just one line, "foo"
$ echo bar >> file
$ cat file
foo
bar
$ ls -i file
472445 file
$ # vim :wq in the other (after it warns that the file has changed)
$ cat file
foo
$ ls -i file
472449 file
A syscall trace of vim 7.0 reveals the source of the new file with the new inode number:
Before writing, this version of vim is using the rename system call to move the original file to file~ . It then uses the open syscall with the O_CREAT flag to create a different file. It then writes the contents of the buffer to this new file before unlinking the backup copy (which is the original file and points to the original inode).
Your example does not contradict anything. It merely demonstrates that a different sequence of calls is being made.
Thank you for verifying with a different editor. It was obvious from the tests I did that the same file was being written to, regardless of what some man page or specification might suggest.
Just to make sure, I did the same test on syslog. If you edit syslog (normally not a good idea), wait a while for something to be appended, write and exit vi, the newly added content is not there. The original poster thought syslog stopped working only because the newly added content was overwritten with the original file when vi saved/exited.
This could be insidious, actually. If the old file is deleted and a new one created, syslog will apparently stop working, since the old, deleted file will remain on disk -- and keep getting written to -- until syslog rotates the file itself for some reason or the machine restarts.
What puzzles me is why anything under /var/log was being edited at all.
In my posts, including the last one, I have repeatedly said it is NOT a good idea to edit syslog (or other system log file). So I am in 100% agreement. I just did this as a one-time operation (a great sacrifice :D) to test the assertion that syslog would continue to be written due (and the content preserved), even if vi edited and saved the file. There was no other way to test it.
If by "overwritten" you mean that the editor changed the file's contents, then you are wrong. If by "overwritten" you mean that the editor created a completely different file, then you are correct.
If an editor simply overwrites the file's contents, after the edit, syslog's new messages would continue to appear in the file.
If the messages cease to reach the named file, then the original file has been unlinked and replaced by a new file (a different inode) with the same name. Meanwhile, syslog is still happily logging to the original log file (now with no name).
The OP stated having to restart to resume logging. That would not be necessary if the file's inode had not changed.
