Hi Team,
Can anyone help me here:
I have to access server logs via putty and these logs file is a trailing file (continously updating) with ERROR and WARNINGS... I need to know if I can pull this trailing file to a local drive so that I can do some higlighting on some keywords through Notepad ++
But I need to manipulate data in real-time (not on the archived file)... Also, I don't have write access on the server as well, so cannot write logs to new file on the server.
Please help !
I do not know how to automate notepad++ on UNIX remote files.
However, the rest is doable. I'm not giving the best possbile answer because I cannot do that without knowing your system type and your shell.
Some flavors of Linux example:
$ sudo apt-get install sshpass # do this only once
$ sshpass -p your_password ssh user@hostname 'tail -f /path/to/logfile/filename ' > $HOME/somefile &
The & makes the command run in the background
$HOME/somefile will have the data you want work with
Note that you need to be very specific about what you want to find locally. Please. Give us example input and desired ouput. Also!
Please give us the output of:
uname -a && echo $SHELL
My example DOES NOT work everywhere.
Thanks @Jim for the answer.
I'm a beginner and may not know everything about UNIX commands. My question was just that if there's any way we can download any file to my local system from Putty/WinSCP and that file is in continous update mode (logs are getting written everytime on that file).
I need to highlight ERROR word with red color in that file.
Normally I use tail -f server.log to follow this file, and keep a eye to see for ERROR line. I just want a better UI so that ERROR lines is visible easily.
Do you have access to cron / crontab on the remote system?
You can probably do this in your terminal by printing an escape sequence to change color (and another after, to reset it to normal). 1;31 means bold+red, 0 means reset. See this list for other colors.
$ tail -f file | awk '/ERROR/ { $0="\x1b[1;31m" $0 "\x1b[0m"; } 1'
text
text
text
something something ERROR something something
text
text
text
You may even be able to make your computer beep from your terminal, as there's a special nonprinting character \a just for that:
$ tail -f file | awk '/ERROR/ { $0="\a\x1b[1;31m" $0 "\x1b[0m"; } 1'
text
text
text
something something ERROR something something
text
text
text
Yes, I do have the access to crontab command.
------ Post updated at 03:40 AM ------
Thanks for the help @Crona688 It works like a charm.
It fulfills my problem to an extent.
I would me more happy if I can pull the server.log file to my local Windows directory and it should be updating continously as it do on the server. Can you tell if there's any way to sync/download the file to local drive while it is continoulsy updated on the server path.
Thanks for the help always !
Please show us the code you are using now. We especially need to see how you are accessing the remote file.
If you can directly access the file on the remote server, do you actually need a copy of the file on your Windows system, or do you just need to see new entries as they are added to the file on the remote server?
Trying to keep a local copy of a file synchronized with a continuously changing remote file can be tricky if you can't use tail -f /remote/file/pathname . You you can directly access /remote/file/pathname from your Windows server, tail should be all you need to keep with with changes being added to the file on the remote server and should also be able to keep a local copy of the file up-to-date with text appended to that file on the remote system.
If existing text in the remote file is being modified while additional text is being appended to the end of the file, your problem is MUCH more complicated than you have implied so far.
Hi Don Cragun,
I cannot share the file data over here due to company policies, and even there is no code written for this...
I just login into server through Putty and use tail -f server.log
to see the contents of the logs. And this file keeps only appending with new logs (ERROR, WARN and INFO lines), it does not edit the previous lines which have already written in past.
I just wanted to have this file to be copied/synced in my local windows drive so that I can view the logs realtime (but in Windows drive).
when you tail should be all you need to keep with with changes being added to the file on the remote server and should also be able to keep a local copy of the file up-to-date with text appended to that file on the remote system, can you elaborate how this can be achieved.
Thanks in advance and Sorry if I missed anything.
Basically, in modern environments one should have some kind of log shipping service/agents, and process those logs on central server.
There are free, open, proprietary etc. systems, which do the same thing, ship logs you want to central server.
As i see it you have couple of choices on Windows :
This is all, from my point of view, not production ready and could be / is considered hackery.
A regular solution would be to use software written for that purpose.
What do you consider as real time?
I consider that to be almost instant, in regards time needed to read from disk, send it over the wire, and write to disk in windows.
Those ops are measured in milliseconds.
Hopefully we can neglect the CPU time required for that operation, but it is software implementation dependent.
For instance, if you use rsync with compression and checksum, it is bound to take more cpu (and other) resources then an agent made especially for that purpose.
Also, running scheduled rsyncs on large log files will cause that entire file to be cached in memory, changing the memory footprint of the box.
Above can be irrelevant if there is enough memory, but can also cause log files to be cached instead of other things, making those other things visible to users as slowdowns 
Hope that helps
Regards
Peasant.
Files do not work that way.
I didn't ask you to share any file data (other than the sample data you have already shown us and a clearer explanation of the format of the data you are trying to process).
I asked you to explain how you are currently accessing the remote file using applications running on Windows. From your response, I assume that you have no way to access the file. Using NFS to mount the remote file onto your windows server (acting as a cliient in this case) would allow a tail -f command on Windows to access your remote file seeing additions to that file in real-time. Anything else will introduce delays and complications.
As Corona688 said, unless you have established remote access to a file, it won't magically create a copy of itself on another server.
Wouldn't a continuously (or, rather, often enough to meet the "real-time" requirements) updated rsync -ed copy of the file suffice? Depending on the size one could also set up a cron job to do something similar with the complete file via [icode]ssh[i/code]/[icode]scp[i/code].
But all this means there at least some means to do what has to be done. But i am not allowed to touch the file, read it, transfer it or even access it but somehow i should process it is a matter of (stupid) regulations. You can't alleviate regulations via code, just by changing the regulations: "you want me to process the file, then give me access to it" is the correct way of doing things in such a case.
I hope this helps.
bakunin
Thanks @Bakunin for the insight.
I have the read only access over the file as said... I can view the file over Putty by tail -f command.
Can you help me how I can set the sync with NFS... a URL where I can learn how to setup NFS services to sync the file from server with local Windows machine.
Thanks a lot
To repeat: Files do not work that way. Even when you share the file -- which you can't without big changes in your server -- you'll be stuck running tail -f in a local terminal instead of a remote one, because text editors don't work that way either.
Files do not work that way.
Files do not work that way.
Because files don't work that way.
It would be worth your while to reconsider the problem from any other angle besides "magically growing file", or better yet, explain what you actually want. You might be quite surprised when you go through all the trouble of setting up an NFS only to discover that what you thought you needed hasn't actually solved your problems. tail -f in putty actually works surprisingly well.
Perhaps, is the problem the inconvenience of having to type passwords into putty? There are automatic login features you could be using. putty can do logging, also, and even network forwarding, allowing you to connect to services on the server.
Otherwise, you have information you wish to transmit from point A to point B in realtime. Perhaps netcat?
You could do full-out network or centralized logging, perhaps, and have an infrastructure to warn you of certain events.