Remove lines having older end time

Shell Programming and Scripting
1

Hi,

In my bash script I want to add a code which remove all entries older than x days.

To simplify this problem, I have divided into 3 parts. (2 parts are done looking answer for 3rd part.)

  1. To find the latest log date - Done
    Latest_Time=`find . -name '*.tps' -exec sed '/endTime/!d; s/{//; s/,.*//' {} + | sort -r | head -1 | cut -d: -f2` #latest epoch time
  2. evaluate earliest epoch time. (All entries before this epoch time should be deleted) - Done
    Earliest_Time=`Latest_Time-(86400000*$No_OF_DAYS)` #earliest epoch time
  3. delete all log entries older than evaluated earliest time.

PS:

  1. there are multiple files and distributed in different sub folders.
  2. All files having extension as ".tps".
  3. time is in epoch format. endTime will be considered for calculations.("endTime":1488902735220)
  4. sample data
{"endTime":1488902734775,"startTime":1488902734775,"operationIdentity":"publishCacheStatistics","name":"murex.risk.control.excesses.cache.CacheStatisticsTracer","context":{"parentContext":{"id":-1,"parentContext":null},"data":[{"value":"excessCacheExcessKeysToContexts","key":"name"},{"value":"0","key":"hits"},{"value":"0","key":"misses"},{"value":"0","key":"count"},{"value":"0","key":"maxElements"},{"value":"0","key":"evictions"},{"value":"N/A","key":"policy"}],"id":0}}
{"endTime":1488902735220,"startTime":1488902735220,"operationIdentity":"publishCacheStatistics","name":"murex.risk.control.excesses.cache.CacheStatisticsTracer","context":{"parentContext":{"id":-1,"parentContext":null},"data":[{"value":"excessCacheExcessKeysToContexts","key":"name"},{"value":"0","key":"hits"},{"value":"0","key":"misses"},{"value":"0","key":"count"},{"value":"0","key":"maxElements"},{"value":"0","key":"evictions"},{"value":"N/A","key":"policy"}],"id":8}}
{"endTime":1488902735550,"startTime":1488902735550,"operationIdentity":"publishCacheStatistics","name":"murex.risk.control.excesses.cache.CacheStatisticsTracer","context":{"parentContext":{"id":-1,"parentContext":null},"data":[{"value":"excessCacheContextsToExcessIds","key":"name"},{"value":"0","key":"hits"},{"value":"0","key":"misses"},{"value":"0","key":"count"},{"value":"0","key":"maxElements"},{"value":"0","key":"evictions"},{"value":"N/A","key":"policy"}],"id":9}}

For Example:

  1. latest epoch time = 1488902735550
  2. earliest epoch time = 1488902735220

Problem: Now I am looking for command which delete all the entries which is older/lesses than earliest epoch time. In above example 1st line should be deleted.

Any help/suggestions are appreciated. Thank you :slight_smile:

2

You're not telling us how the "earliest epoch time" will be transported - in a file? A shell variable? Inside awk (etc.) already?

3

Try

awk -F, -vET="1488902735220" 'substr($1, 12, 13) >= ET' file
4 
Latest_Time=`find . -name '*.tps' -exec sed '/endTime/!d; s/{//; s/,.*//' {} + | sort -r | head -1 | cut -d: -f2`  #latest epoch time
Earliest_Time=`Latest_Time-(86400000*$No_OF_DAYS)` #earliest epoch time