Need to validate that all 24 hr are in a table

Shell Programming and Scripting
1

I have a table that looks like this, but for a whole year:

Hourly weather history for XXX 7 Jun
0:00	Clear weather    28
1:00	Clear weather    23
2:00	Clear weather    21
3:00	Clear weather    22
4:00	Clear weather    22
5:00	Clear weather    22
6:00	Clear weather    23
7:00	Clear weather    23
9:00	Clear weather    28
10:00	Clear weather    30
11:00	Clear weather    32
12:00	Clear weather    33
14:00	Clear weather    35
15:00	Clear weather    34
16:00	Clear weather    34
17:00	Clear weather    34
18:00	Clear weather    32
19:00	Clear weather    33
20:00	Clear weather    31
21:00	Clear weather    30
22:00	Clear weather    30
23:00	Clear weather    28
Hourly weather history for XXX 8 Jun
0:00	Clear weather    29
1:00	Clear weather    28
2:00	Clear weather    27
6:00	Clear weather    25
8:00	Clear weather    29
9:00	Clear weather    29
10:00	Clear weather    32
11:00	Clear weather    34
12:00	Clear weather    35
13:00	Clear weather    35
15:00	Clear weather    37
17:00	Clear weather    36
18:00	Clear weather    34
19:00	Clear weather    32
20:00	Clear weather    31
21:00	Clear weather    30
22:00	Clear weather    30
23:00	Clear weather    30

If you look carefully, you'll see the first day has all 24 hours accounted for, but the 2nd day is missing 0300-0500, 0700, 1400 and 1600

I'm trying to loop through the list and check if the time on line N+1 is equal to TIME +1. If not, insert that line (just the time)

So far I have this:

#! /bin/awk -f

{
if ($1 ~ /Hourly/)
	{
	print ""  #print a blank line for ease of finding new day
	tmp[1]="" #reset variables.  Necessary?
	lineONE[1]=""	  #
	}
	else
	{
	split($1,lineOne,":")   #split the time so that you're just using the hour portion
	
	if (lineOne[1] <= 23)					#Loop through until all 24 hours are tested
		{
		getline line2
		split(line2, tmp, ":")
		if  (tmp[1]== lineOne[1]+1)
			{
			print $0
			print line2
			}
		else
			{
			print $0
			print lineOne[1]+1 ":00\t!!!!inserted by program!!!!"
			}
		}
	}
}

Which works if there is only one missing hour, and there are 2 consecutive hours for the next set of hours. I can't figure out how to reset so that (getline line2) becomes lineOne. Any help appreciated.

Currently the output is:

paulyester$ awk -f awktest testhrfile.txt

kevin$ awk -f awktest testhrfile.txt

0:00	Clear weather    28
1:00	Clear weather    23
2:00	Clear weather    21
3:00	Clear weather    22
4:00	Clear weather    22
5:00	Clear weather    22
6:00	Clear weather    23
7:00	Clear weather    23
9:00	Clear weather    28
10:00	Clear weather    30
11:00	Clear weather    32
12:00	Clear weather    33
14:00	Clear weather    35
15:00	Clear weather    34
16:00	Clear weather    34
17:00	Clear weather    34
18:00	Clear weather    32
19:00	Clear weather    33
20:00	Clear weather    31
21:00	Clear weather    30
22:00	Clear weather    30
23:00	Clear weather    28

0:00	Clear weather    29
1:00	Clear weather    28
2:00	Clear weather    27
3:00	!!!!inserted by program!!!!    <-- still missing 04000-0500, but even worse, dropped 0600 too
8:00	Clear weather    29
9:00	Clear weather    29
10:00	Clear weather    32
11:00	Clear weather    34
12:00	Clear weather    35
13:00	Clear weather    35
15:00	Clear weather    37
16:00	!!!!inserted by program!!!!  <--dropped 1700
18:00	Clear weather    34
19:00	Clear weather    32
20:00	Clear weather    31
21:00	Clear weather    30
22:00	Clear weather    30
23:00	Clear weather    30
2

how about this?

 echo '0:00 Clear weather 29
1:00 Clear weather 28
2:00 Clear weather 27
6:00 Clear weather 25
8:00 Clear weather 29
9:00 Clear weather 29
10:00 Clear weather 32
11:00 Clear weather 34
12:00 Clear weather 35
13:00 Clear weather 35
15:00 Clear weather 37
17:00 Clear weather 36
18:00 Clear weather 34
19:00 Clear weather 32
20:00 Clear weather 31
21:00 Clear weather 30
22:00 Clear weather 30
23:00 Clear weather 30' |awk -F: 'NR>1{if($1-var1==1){var2=$1}else{while (var2<$1-1) print ++var2":00"}}{var1=$1}1'
0:00 Clear weather 29
1:00 Clear weather 28
2:00 Clear weather 27
3:00
4:00
5:00
6:00 Clear weather 25
6:00
7:00
8:00 Clear weather 29
9:00 Clear weather 29
10:00 Clear weather 32
11:00 Clear weather 34
12:00 Clear weather 35
13:00 Clear weather 35
14:00
15:00 Clear weather 37
15:00
16:00
17:00 Clear weather 36
18:00 Clear weather 34
19:00 Clear weather 32
20:00 Clear weather 31
21:00 Clear weather 30
22:00 Clear weather 30
23:00 Clear weather 30
3

OK, that works brilliantly from the command line - even if I'm not quite sure why at this point.

How would I run that from within a script. The complete file has 8K+ rows that need to be checked, so echo isn't an option.

Sorry for being dense. This is new to me.

the filename is testhrfile.txt

the script I tried to make is

#! /bin/awk -f-F:

{

NR>1
	{
	if($1-var1==1)
		{
		var2=$1
		}
	else
	{while (var2<$1) 
		print ++var2":00"}
	}
	{var1=$1}1
}

but it spews out:

kevin$ awk -f awktest3 testhrfile.txt
1:00
2:00
3:00
4:00
5:00
6:00
7:00
8:00
9:00
10:00
11:00
12:00
13:00
14:00
15:00
16:00
17:00
18:00
19:00
20:00
21:00
22:00
23:00
24:00
25:00
26:00
27:00
28:00
29:00
30:00
31:00
32:00
33:00
34:00
35:00
36:00

etc until CTRL-C'd
4

try this way:

#! /bin/awk
{

NR>1
{
if($1-var1==1)
{
var2=$1
}
else
{while (var2<$1) 
print ++var2":00"}
}
{var1=$1}
}1

awk -F: -f awk.script urfile

5

Thanks. It worked once I took the headers at the top of each day out. doh!

6

if you want keep the headers in the file just make a little change replacing the old code

 {var1=$1}

replaced by this code

/^[0-9]/{var1=$1}
7 
$ awk -F: -f awktest3 testhrfile.txtawk: syntax error at source line 15 source file awktest3
 context is
	 >>> /^[0-9]/{ <<< 
awk: illegal statement at source line 15 source file awktest3
awk: illegal statement at source line 15 source file awktest3
8

try:

#! /bin/awk
{
NR>1
{
if($1-var1==1)
{
var2=$1
}
else
{while (var2<$1-1) 
print ++var2":00"}
}
{if(/^[0-9]/)
var1=$1}
}1
9

Awesome. Now I'll get out the books and figure out why.

Thanks!

10

yinyuemi's code has double 6:00 and 15:00.

Here is my solution.

awk '/Hourly/ {d++;c[d]=$0;next} {a[$1 FS d]=$0}
     END {for (i=1;i<=d;i++) 
             { print c; 
               for (j=0;j<=23;j++) print (a[j ":00" FS i]=="")?j ":00":a[j ":00" FS i]
             }
         }' testhrfile.txt
11

Thanks for your reply:b:
I have modified the code

#! /bin/awk
{NR>1
{
if($1-var1==1)
{var2=$1}
else
{while (var2<$1-1) 
print ++var2":00"}
var2=$1}
{if(/^[0-9]/)
var1=$1}
}1
12

yinyuemi, I have to say your idea is wrong from the beginning.

If the record has no 0:00 or 23:00, how it can be generated by your script.

13

Hi rdcwayx,
your comments is correct, that's the limitation of my script, it's not robust enough, your code is much better.
Aagin, thanks for your correction.