Hi all, I'm looking for an awk solution for taking bins of data set.
For example, if I have two columns of data that I wish to use for a scatter plot, and it contains 5 million lines, how can I take averages of every 100 points, 1000, 10000 etc...
The idea is to take bins of the 5,000,000 points and reduce the density.
Thanks for the response, here is an example, i'll focus on just the second column
input
$ cat file.2.txt
3
2
1
10
10
10
25
30
60
The purpose is to reduce the data for an x,y scatterplot, because the file is millions of lines long. Instead of plotting every point, I want to take an average of every "n" number of points, and plot that one number. Bin might not be the correct word, perhaps a "rolling-average"? For example a bin of 3 would break the data down like so:
$ cat file.2.txt
#bin A
3
2 #average all three = 2
1
#bin B
10
10 #average all three = 10
10
#bin C
25
30 # average all 3 = 38.3
60
Output would then be:
2
10
38.3
For the case where bin is 2
$ cat file.2.txt
#bin A
3 # average = 2.5
2
#bin B
1 # average = 5.5
10
#binC
10 # average = 10
10
#bin D
25 # average = 27.5
30
#binE - ignored because only one value
60
Finally, doing this for both the x and y axis (the original file), for bin of 3: