Printing the output of sed using a loop

Shell Programming and Scripting
1

So I am writing a bash script that will search a file line by line for unix timestamps, store all of the timestamps into an array, then check how many of those timestamps were created within the last hour, and finally increment a counter every time it finds a timestamp created within the last hour.

whenever I try to run the script I get an error like this one

2dispoCounter.sh: line 19: [@],-1547770608: syntax error: operand expected (error token is "[@],-1547770608")

when I run the command echo $epochTimes I get thousands of epoch times that look like this

547765257 1547765347 1547765577 1547765582 1547765956 1547765962 1547765949 1547766001 1547766074 1547766089 1547766158 1547766242 1547766242 1547766527

Here is my script:

#!/bin/bash
 
a=(0 16 272 512 768 2048 2304 1024 1280)
counter=(0 0 0 0 0 0 0 0 0)
 
#grab # of results from log file and store each value in the array
j=0
for c in ${a[@]}
do
    epochTimes=()
    epochTimes=("$epochTimes[@]", $(grep 'cn2='$c' ' /opt/crowdstrike/log/output | sed -n 's/.*rt=\([0-9]\{10\}\).*/\1\n/p'))
    echo "${epochTimes[@]}"
    now=$(date +%s)
    for aTime in "${epochTimes[@]}"
    do
        echo $aTime
        echo $now
        diff=$(($aTime-$now))
        if [ $diff < 3600 ]
        then
            ((counter[$j]++))
        fi
    done
    echo ${counter[$j]}
    ((j++))
done
python testPrint.py "${counter[@]}"

So how would I go about make aTime within the script be one of these timestamps and not [@] , ?

2

You might consider changing:

    epochTimes=("$epochTimes[@]", $(grep 'cn2='$c' ' /opt/crowdstrike/log/output | sed -n 's/.*rt=\([0-9]\{10\}\).*/\1\n/p'))

to:

    epochTimes=("${epochTimes[@]}", $(grep 'cn2='$c' ' /opt/crowdstrike/log/output | sed -n 's/.*rt=\([0-9]\{10\}\).*/\1\n/p'))

and let us know if that makes any difference.

If not, change the second line of your script to:

set -xv

and see if you can figure out what is going on by watching the tracing output as your script runs. If not, show us the output you get with tracing turned on and maybe we can help you further diagnose your problem.

2 Likes
3

I figured it out.
As it turns out, for some reason the first argument returned was always [@]. So in order to solve the problem all I had to do was to start the loop at the second argument

Something like

for aTime in "${epochTimes[@]:2}"

instead of

for aTime in "${epochTimes[@]}"
4

no comma separates the array elements

   epochTimes=("${epochTimes[@]}" $(grep 'cn2='$c' ' /opt/crowdstrike/log/output | sed -n 's/.*rt=\([0-9]\{10\}\).*/\1\n/p'))

--- Post updated at 06:32 ---

so you just lose some of the data along with the error

2 Likes
5

Note that if you have made the change I suggested in post #2 in this thread, there wouldn't have been a [@], in the array (but as nezabudka noted there would have been one element in the array that ended in a comma that would not work correctly.

And, in addition to what nezabudka has already said, note that anytime you have a loop containing a repeated pipeline of grep ... | sed ... , there is a good chance that you could replace that entire loop with a single awk script that could do what you're trying to do much more efficiently. (But, of course, we'd have to know what the format of your input file ( /opt/crowdstrike/log/output ) is to help you do that.

And, if we knew the format of the output you're trying to produce, that could probably also be built into that awk script without needing to invoke python after the loop (or awk ) is done.

You might also note that the construct 'cn2='$c' ' in your grep command, would be more readable for many people if you wrote it as "cnt=$c " . But, of course, the result of using either quoting form will be the same as long as the strings assigned to $c are numeric strings as in your example.

Knowing what operating system you're using would probably also help us give you suggestions that will work correctly in your environment.

And, having a reasonably short sample input file and a corresponding sample output file showing the output you're hoping to produce would be a huge help for anyone trying to help you achieve your goal. Please help us help you.

1 Like
6

I would use awk if I knew how, but for now grep and sed is fine. I have this book on sed and awk that i am reading. I am just glad that I figured out how to use sed.

I will consider changing

'cn2='$c' '

to

"cn=$c "

As for what I am doing with the code,

  1. I am taking the results and types of detections from an antivirus program
  2. and separating them based on the type of detection represented by cn2=$c
    3.and cutting out the timestamp from each result.
  3. I then compare the timestamp on the detection to the current time to see if its under an hour old.
  4. For every detection that is under an hour old a counter within an array is incremented based on the type of virus that was detected.
7

Based on a few assumptions on the structure of your input file, like

  • the numerical "detection type" is introduced by the string cn2= , up to four digits, and terminated by a space char.
  • there's no more than one time stamp per line, introduced by the partial string rt= , 10 digits long

here's an awk proposal, working on a mock up input file, that might satisfy your request:

awk '
BEGIN                   {HRago = srand() - 3600                                                                                 # srand is seeded from the clock, and returns that value
                         for (DMX = n = split ("0 16 272 512 768 2048 2304 1024 1280", DETTYPE); n; n--) CNT[DETTYPE[n]] = 0;   # det. type, and counter (indexed by det type) arrays
                         TIMPAT = "rt=[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]"                                       # define epoch time pattern with introducing string
                        }
match ($0, TIMPAT)      {if ((substr($0, RSTART+3, 10) > HRago                  ) &&                                            # epoch time greater than one hour ago AND
                             (match ($0, /cn2=[0-9]* /)                         ) &&                                            # det.type introducer found AND
                             ((DT = substr($0, RSTART+4, RLENGTH-5)) in CNT     )       ) CNT[DT]++                             # det type known in counter array --> incr. counter
                        }
END                     {for (i=1; i<=DMX; i++) print "Detection type:", DETTYPE, ", count:", CNT[DETTYPE]                # output detection types and counters
                        }
' file
Detection type: 0 , count: 1
Detection type: 16 , count: 0
Detection type: 272 , count: 0
Detection type: 512 , count: 1
Detection type: 768 , count: 1
Detection type: 2048 , count: 0
Detection type: 2304 , count: 0
Detection type: 1024 , count: 0
Detection type: 1280 , count: 0