cpu%/mem% usage, scripting, dzen2: howto learn bash the hard way

broli · August 2, 2008, 7:29pm

I am trying to write a small (and rather simple) script to gather some info about the system and piping it to dzen2

first, i want to explain some things.
I know i could have used conky, but my intention was to expand my knowledge of bash, pipes and redirections inside a script, and to have fun (which i am)
some time later, i also decided to try to be more strict on the resource usage. this is a script that will constantly run on a low-end laptop.

also, dzen is not the important part of the script and can be simple ignored, is just a way to show the info. if you are interested in dzen, you can read more info here gotmor - dzen

so, lets talk about the script itself

i use top in batch mode to get the list of most cpu/mem using, but now, i would like to include the total too. (in percentage)

with the memory, i was thinking maybe using free, and doing some math on the used/cached fields. but that would involve a new app to run, when top already gave me the info.
but taking the info from top's output will require a heavy use of pipes and secondary apps like tail and head

with the cpu%, well, im lost .....
where could i find that info?
im thinking, maybe is better to just use the iddle% field of top

this is what i have done so far
general pastebin - broli - post number 1091305

any suggestion on how to improve what i have done will also be welcomed

era · August 3, 2008, 1:54pm

For a small piece of code like that, I'm taking the liberty to quote

The idiomatic way to code an endless loop is simply while true although you also see the obscure while : which avoids an external process (even though true is often a shell built-in in modern shells).

do
	top -b -n 1 > /tmp/salidatop
	
	#primero cpu
	echo -n "^fg(green)CPU **"
	tail -n +8 /tmp/salidatop | sort -r -n -k9 | head -n 5 | while read line
	do 
		echo "$line" | awk ' { print "[^fg(cyan)",$12,"(^fg(red)",$9,"^fg(green)]--" } ' | tr -d ' ' | tr -d '\n'	 
	done

Is there a reason to feed awk a line at a time? Why not just

tail -n +8 /tmp/salidatop | sort -r -n -k9 | head -n 5 |
awk ' { printf "[^fg(cyan)",$12,"(^fg(red)",$9,"^fg(green)]--" } '

I don't see any spaces in the output, so tr -d ' ' seems superfluous. By using printf instead of print, there will be no trailing newline. As an aside, you could have combined the two tr:s to one: tr -d ' \n'

The same change could be applied below:

	echo ">>"
	echo -n "^fg(green)MEM **"
	#segundo mem
	tail -n +8 /tmp/salidatop | sort -r -n -k10 | head -n 5 | while read line
	do 
		echo "$line" | awk ' { print "[^fg(cyan)",$12,"(^fg(red)",$10,"^fg(green)]--" } ' | tr -d ' ' | tr -d '\n'	 
	done

	echo ">>"
	
	sleep 5

done | dzen2 -ta l -u -l 1 -x 20 -y 710 -w 660 -e 'onstart=lower,uncollapse'

I don't understand the question about CPU percentage. You can add the user, system, and nice percentages, or just subtract the idle percentage from 100% as you note.

If you can replace the complex tail | head etc with a simple sed or awk script, that will probably help reduce the resource requirements. Perhaps you could pass some option to top to order the output like you want it, so you can avoid the separate sort -- that's probably the main bottleneck here (albeit a very minor one, with so little input).

broli · August 3, 2008, 7:16pm

great, this sort of things is what i was expecting when i posted here
i didn't used true, because i had the idea that it doesn't work (i might have got that form an old aix/tru64 at work)
i will google for more info on ":"

well, the main reason is cause i didnt tough of it
specially because of the approach i had with the while

well, i had spaces on my tests, dont know when they came from. i will check again

great !! i didnt knew this. this is really good

didnt knew you could use tr like that

my main problem is how to get it, or where to get that info.

i have the vague idea that i could use awk to avoid head | tail | cut , but i dont know how

could you give me some directions ?
specially to a good example

era · August 4, 2008, 1:52am

Obiously, awk is a fairly complete programming language, so you can rewrite simple text-processing utilities in awk, many of them easily.

awk 'NR==10 { exit 0 }1'  # head -10
awk '{ print $2 }'   # cut -f2 (splitting on runs of whitespace though)

tail is harder to emulate generically because the array type is rather crude. Maybe switch to sed (or Perl!) for that.

See also http://www.pement.org/awk/awk1line.txt

The first few lines of top output indicate the CPU usage. Read the manual page to learn what the fields mean.

Cpu(s): 16.8%us,  1.7%sy,  0.0%ni, 80.5%id,  0.8%wa,  0.1%hi,  0.1%si,  0.0%st

broli · August 4, 2008, 8:26am

era:

Obiously, awk is a fairly complete programming language, so you can rewrite simple text-processing utilities in awk, many of them easily.
awk 'NR==10 { exit 0 }1'  # head -10
awk '{ print $2 }'   # cut -f2 (splitting on runs of whitespace though)

so, if i read your post correctly, with awk i can simulate sime functions of tail or head...
what about sort?
i must admint my knowledge of awk is fairly simple. i ave seen more complex ones, and im reading this website AWK Language Programming - Table of Contents
readnig the temporal file with awk and doing all the text procesing will decrease the number of binaries that have to be loaded (less i/o)

well. i see now, that perl is a best solution than bash. after all perl was made to process text right?
and perl has always been in my todo-list
the problem there would be to pipe the info to dzen
because the way dzen works, i need an never ending loop, so dzen dosnt die
is posible to do that piping from a perl script?

thanks for the link. is in my bookmarks now

era:

The first few lines of top output indicate the CPU usage. Read the manual page to learn what the fields mean.
Cpu(s): 16.8%us,  1.7%sy,  0.0%ni, 80.5%id,  0.8%wa,  0.1%hi,  0.1%si,  0.0%st

[/quote]

i know that. the problem is getting that info without using head | tail | cut or awk
spawning all those apps 4 times on the same text input file doesnt strike me as good performance

thanks for the replies !

era · August 4, 2008, 1:19pm

Modern versions of awk have a sort command. I'm not sure if it was included even in the original awk.

Certainly, but I'm not really suggesting you move to Perl just for this. There are really two schools of thought on this; skip awk, and only learn Perl; or move to Perl if you run into situations which awk cannot handle comfortably. If you need to process binary data or very large data sets, Perl was designed to overcome the limitations awk have in those areas.

But the requirements you have shown so far should be easy to handle in awk; it's just not clear from reading the code in which direction you want the script to grow, and/or I was too lazy to rewrite it all.

broli · August 4, 2008, 2:07pm

im on gentoo here, so is the latest stable gawk (sorry, my bad, for not telling you earlier)

the very reason i started with linux, the very reason im still with linux, is that there is something to learn, teh chalenge, and the satisfaction of personal growth.
"moving to perl" doesn't mean anything "bad"
i will finish this with bash (because i wont let this sucker win). and after that, i will try to make it in perl. just because i can (or rather "just because i still dont know how" )

there is only one more thing i want from this script. show the total of memory used, and the total of cpu used, while keeping it light

for the cpu. i had some doubts, but i have decided to show the iddle percentage. this is a personal thing, i will use it, so if i want/know iddle%, thats what im gonna get

for the memory, i have been researching.
i was using wmmem, so i went and read the code to and read how it calculated. i came up with the math behind it.
((total - cache - buffer) * 100 ) / total
now my problem is getting those 3 from top

unfortunately, all my work is in my personal laptop (at home), and usually, i dont spent time in this during the week, so i cant give you any particulars on what i have done so far to retrieve that data
i dont ask you to rewrite my script. actually i would hate that.
as i said, i want to solve this, or at least try

thanks for the great help so far !

era · August 5, 2008, 2:02am

Another couple of notes about the use of a temporary file.

First off, if you will never use the first few lines of the temp file, might as well throw them away already at the start.

top -b -n 1 | tail -n +8 > /tmp/salidatop

Secondly, when you are done, you should remove your temporary file. Better yet, remove it even if you are interrupted.

trap 'rm -f /tmp/salidatop; exit $?' 0
trap 'exit 127' 1 2 3 5 15

Put those lines near the beginning of the script.

Properly speaking, you should probably use something like mktemp to generate a unique, unpredictable temporary file name. There are security issues with using predictable names, and having a static file name means you can't run two instances of the script at the same time.

broli · August 5, 2008, 9:18am

[quote="era,post:8,topic:207221"]
Another couple of notes about the use of a temporary file.

First off, if you will never use the first few lines of the temp file, might as well throw them away already at the start.

top -b -n 1 | tail -n +8 > /tmp/salidatop

[quote]

i will use those lines to have the iddle% and the amount of memory

era;302221664:

Secondly, when you are done, you should remove your temporary file. Better yet, remove it even if you are interrupted.
trap 'rm -f /tmp/salidatop; exit $?' 0
trap 'exit 127' 1 2 3 5 15
Put those lines near the beginning of the script.

Properly speaking, you should probably use something like mktemp to generate a unique, unpredictable temporary file name. There are security issues with using predictable names, and having a static file name means you can't run two instances of the script at the same time.

its by design.

about the security risk, i decided that beeing my personal laptop, always in a secure network, and only the contents of top, that is not thread enough to consider some extra precausions
also, my /tmp is wiped out on poweroff and poweron

the script needs X to run, and the script is run by my user config files (fluxbox startup), so there wont be any other instance.
but deniying the posibilitie is a bad idea.
maybe using $$ in some part of the file can do the trick
and adding the trap to make sure i dont left behind tons of temporal files.

broli · August 10, 2008, 8:06pm

this is what i have now

      #!/bin/bash

      while :
      do
              top -b -n 1 > /tmp/salidatop

              cpuid=`awk ' NR==3 { print $5 }'  /tmp/salidatop`
              echo -n "^fg(green)CPU $cpuid ** "
              awk 'NR == 8, NR == 12 { printf "[^fg(cyan)%s(^fg(red)%s^fg(green)]--",$12,$9 }' /tmp/salidatop
              echo ">>"
       
              echo -n "^fg(green)MEM ** "
              sort -r -n -k10 /tmp/salidatop | awk 'NR<=4 { printf "[^fg(cyan)%s(^fg(red)%s^fg(green)]--",$12,$10 } '
              echo ">>"
       
              sleep 4
       
      done | dzen2 -ta l -u -l 1 -x 20 -y 710 -w 660 -e 'onstart=lower,uncollapse'

and i dont know how to get the info of the memory.
i need to do this math
(( free + buffers + cached) * 100)/total

but i dont know how to get those fields in one awk (and maybe use the internal math?)
thanks again

era · August 11, 2008, 3:12am

Something like this maybe.

#!/bin/sh

while :
do
    top -b -n -1 |
    awk 'NR > 5 { mem[$12] = $10; }
        NR==3 { printf "^fg(green)CPU " $5 " ** "; next; }
        NR==4 { total=$2; free=$6; buffers=$8; next; }
        NR==5 { cached=$8; next; }
        NR==8, NR==12 { printf "[^fg(cyan)" $12 "(^fg(red)" $9 "^fg(green)]--"; next; }
        NR==13 { printf ">>\n"; next; }
        END { ... sort and extract top 4 items from mem array here ...;
            printf "%2.2f\n", (free+buffers+cached)/total*100 }'
    sleep 4
done | dzen2 --options

I haven't finished the mem part but it's not too complex; you should be able to implement the remaining part of the awk script and sort and pick the output lines you want by searching the forums a bit. The mawk manual page has an example of how to implement a simple sort in awk.

My top prints "Mem: total used free buffers" followed by "Swap: total used free cached" on lines 4 and 5; I'm not entirely sure which of the "free" and "total" fields you want, but extracting the fields you want (provided you are sure which ones you want) should be similarly straightforward. I've put in placeholders for those. A complication is that the output has a human-readable suffix like "k" which I imagine might vary, so properly you should parse that before doing the math on the resulting numbers.

broli · August 11, 2008, 10:20am

era:

Something like this maybe.
#!/bin/sh

while :
do
   top -b -n -1 |
   awk 'NR > 5 { mem[$12] = $10; }
   NR==3 { printf "^fg(green)CPU " $5 " ** "; next; }
   NR==4 { total=$2; free=$6; buffers=$8; next; }
   NR==5 { cached=$8; next; }
   NR==8, NR==12 { printf "[^fg(cyan)" $12 "(^fg(red)" $9 "^fg(green)]--"; next; }
   NR==13 { printf ">>\n"; next; }
   END { ... sort and extract top 4 items from mem array here ...;
   printf "%2.2f\n", (free+buffers+cached)/total*100 }'
   sleep 4
done | dzen2 --options
I haven't finished the mem part but it's not too complex; you should be able to implement the remaining part of the awk script and sort and pick the output lines you want by searching the forums a bit. The mawk manual page has an example of how to implement a simple sort in awk.

My top prints "Mem: total used free buffers" followed by "Swap: total used free cached" on lines 4 and 5; I'm not entirely sure which of the "free" and "total" fields you want, but extracting the fields you want (provided you are sure which ones you want) should be similarly straightforward. I've put in placeholders for those. A complication is that the output has a human-readable suffix like "k" which I imagine might vary, so properly you should parse that before doing the math on the resulting numbers.

yeah, this thing is growing faster than my speed to read awk info
i will look at that and run some tests
the problem, is that all this work only o get a simple percentage of free mem, is growing in complexity and footprint
to a point where i dont know if its worth the problem
maybe i can make use of an external program to give me that info.
maybe doit myself in C? .... (actually, stole the related code of wmmem )

era · August 12, 2008, 2:16am

A single awk script is probably way more efficient than the multiple awks and sorts and echos and what not you had before, that's why I'm proposing it.

If you can fork the code from top to give you exactly the output you want, it will of course be more efficient still, but IMHO probably not worth the programmer effort.

Maybe if you could add a module to top to produce XML or CSV or something else more machine-readable, that could perhaps be worth it.

broli · August 12, 2008, 8:57am

i see that using a single awk would be faster, but im still working to uderstand how to doit.

eventually i hope my script is a while loop (needed only for dzen) and an awk calling a script file with the -f parameter.

and what you mean by "if you could add a module to top"

broli · August 17, 2008, 6:40pm

i have been reading about how to sort arrays. and now i understand why you did this

NR > 5 { mem[$12] = $10; }

but the problem is, how do i print the array index ?
i need $12 later on

until i can fix that "retieval" of the value, i dont see how to integrate the memory calculation into the next awk

altho i have integrated the cpu part in one awk, and only that, nor more exhos

im gettign there

#!/bin/bash

while :
do
	top -b -n 1 > /tmp/salidatop
	
	awk '	NR==3 { printf "^fg(green)CPU %2.f%% ** ",$5 } \
		NR==8, NR==12 { printf "[^fg(cyan)%s(^fg(red)%s^fg(green)]--",$12,$9 } \
		END { print ">>" } ' /tmp/salidatop

	memper=$(awk 'NR==4 { total=$2; free=$6; buffers=$8}
			NR==5 { cached=$8 }

			END {   
	        	sub( "k","",total)
		        sub( "k","",free)
			sub( "k","",buffers)
			sub( "k","",cached)

			suma = free + buffers + cached
			porcentaje = (suma * 100) / total
			printf "%2.f\n",porcentaje }' /tmp/salidatop)

	echo -n "^fg(green)MEM $memper% ** "
	sort -r -n -k10 /tmp/salidatop | awk 'NR<=5 { printf "[^fg(cyan)%s(^fg(red)%s^fg(green)]--",$12,$10 } \
						END { print ">>" } ' 

	sleep 4

done | dzen2 -ta l -u -l 1 -x 20 -y 710 -w 660 -e 'onstart=lower,uncollapse'