"Smoothing" the data

cosmologist · February 7, 2012, 2:22pm

Hi!

Is there a way to use awk to "smooth" a set of data? That is, doing what the "SMOOTH" function in IDL does (SMOOTH) ?

Thanks!

Corona688 · February 7, 2012, 5:22pm

Surely there's a way to do that with awk. awk doesn't have a single function for it of course, since it doesn't make much sense to do a boxcar average on a single line...

Let me see if I have the algorithm right. For these points:

for a boxcar average of width 3, would you want:

(1+5+2)/3, (5+2+6)/3, (2+6+3)/3, (6+3+7)/3, (3+7+4)/3, (7+4+8)/3, (4+5+5)/3, (8+5+9)/3

---------- Post updated at 04:22 PM ---------- Previous update was at 04:10 PM ----------

$ cat data
1
5
2
6
3
7
4
8
5
9

$ cat smooth.awk

BEGIN { WIDTH=3 }

{       DATA[(++N)%WIDTH]=$1    }

(N>=WIDTH) {
        V=0
        for(X=(N+1); X<=(N+WIDTH); X++)
                V+=DATA[X%WIDTH];

        print V/WIDTH;
}

$ awk -f smooth.awk data
2.66667
4.33333
3.66667
5.33333
4.66667
6.33333
5.66667
7.33333
4.66667

$

cosmologist · February 20, 2012, 9:57am

Thanks for your help
I am trying to apply this idea to "degrade" the output of a model resolution.

For the sample:

$ vi model.dat

3002.000 1.2874e+32
3002.300 1.2898e+32
3002.600 1.2966e+32
3002.900 1.3000e+32
3003.200 1.2997e+32
3003.500 1.2961e+32
3003.800 1.2926e+32
3004.100 1.2909e+32
3004.400 1.2904e+32
3004.700 1.2923e+32
3005.000 1.2970e+32
3005.300 1.3027e+32
3005.600 1.3043e+32
3005.900 1.3028e+32
3006.200 1.2977e+32
3006.500 1.2894e+32
3006.800 1.2802e+32
3007.100 1.2738e+32
3007.400 1.2717e+32
3007.700 1.2728e+32
3008.000 1.2778e+32

I want to "somehow" degrade the resolution of the first column from 0.3 to 3 for the corresponding numbers in the second column something like this:

3002.000 ???
3005.000 ???
3008.000 ???

Where the "???" are the numbers I am trying to obtain using the awk script that you suggested but I am not able to get the results :wall:. Any suggestions?

cosmologist · February 21, 2012, 12:34pm

Any thoughts on this... I am still not able to solve it :(:(

Corona688 · February 21, 2012, 12:48pm

It took you a week to get back to me. It can't be urgent enough for you to break the rules by bumping.

We're not on call, if nobody answers your post immediately, wait! :wall:

What is the ??? supposed to be? How would I arrive at it? In what way does the code I gave you "not work"? My last two posts were one big question, "do I have the right algorithm", and you never answered me. :wall: :wall:

cosmologist · February 21, 2012, 1:24pm

I am sorry I didn't mean to sound rude, the reason I took a week is that I was trying to figure it out myself first
It is not urgent but I just thought that if more details are needed for my question I can give more details..
I am sorry for that... the second post wasn't only for you, it was for anyone who might have an idea... I am really sorry

Corona688 · February 21, 2012, 1:26pm

Don't be sorry, just please answer my questions They'll help anyone else trying to help you too.

cosmologist · February 21, 2012, 2:09pm

I used the code that you suggested:
For the data:

BEGIN { WIDTH=10 }

{       DATA[(++N)%WIDTH]=$1    }

(N>=WIDTH) {
        V=0
        for(X=(N+1); X<=(N+WIDTH); X++)
                V+=DATA[X%WIDTH];

        print V/WIDTH;
}

I was expecting the output:

3002.000
3005.000
3008.000

But I got:

Maybe I misunderstood the code?

drl · February 21, 2012, 2:41pm

Hi.

Yet you are expecting only 3 values back from an input vector of 22 values. Why is that?

Best wishes ... cheers, drl

Corona688 · February 21, 2012, 3:56pm

There were other parts than code, which you didn't read. I detailed clearly what I was trying to do, none of which had anything to do with stripping off decimal points -- you wanted a boxcar average...

echo "3.14159" | awk '{ printf("%d\n", $1) }'
3
$

That still doesn't clear up what you want to happen to the second column of data. You need to tell us what we're supposed to do to that. It's not going to change itself.

cosmologist · February 21, 2012, 10:37pm

I read that part and it made sense, but when I applied it to my data I didn't get what I was expecting, so now thinking about it I might have a problem with the mathematical concept itself..

I understand that this is a Unix forum not a math forum, so I'll try to figure the math part first.

Thanks

Corona688 · February 22, 2012, 12:41am

I added three numbers, and divided by three? That's liable to add more decimal points...

cosmologist · February 22, 2012, 3:08pm

Ok, I know what it is that confused me now

So in the example that you mentioned

What I was thinking was:

(1+5+2)/3, (2+6+3)/3, (3+7+4)/3, (4+5+5)/3

[If I am using the mathematical terms correctly this is convolution with a rectangle function, a special case of the boxcar function]..

Is there a way to modify your code to get that?

Thanks again for all the help!

Corona688 · February 22, 2012, 3:52pm

If you meant 4+8+5, then sure. I'll let it work on multiple columns, too.

$ cat data
1       5
5       6
2       7
6       8
3       9
7       10
4       11
8       12
5       13
9       14

$ cat smooth2.awk

BEGIN { WIDTH=3; OFS="\t" }

{
        N++;
        for(M=1; M<=NF; M++)    DATA[M,N%WIDTH]=$M
}

(N>=WIDTH) && !((N-WIDTH)%(WIDTH-1)) {

        for(M=1; M<=NF; M++)
        {
                $M=0

                for(X=(N+1); X<=(N+WIDTH); X++)
                        $M+=DATA[M,X%WIDTH];

                $M /= WIDTH;
        }

        print
}

$ awk -f smooth2.awk data

2.66667 6
3.66667 8
4.66667 10
5.66667 12

$

Note the number of decimal places in the second column is a coincidence, those sums just happened to divide nicely by three.

cosmologist · February 22, 2012, 4:19pm

Thank you for the help
I tried to apply it to a larger scale but I think I am missing something...
What I am trying to do is instead of taking the average of every 3 points, I am taking the average of every 23 points (and skipping 11 points)...
I changed this part :

BEGIN { WIDTH=3; OFS="\t" }

to

BEGIN { WIDTH=23; OFS="\t" }

but I think there is one more thing that needs to be changed (maybe in
OFS="\t" to skip 11 points not just one... any suggestions?

Thanks

Corona688 · February 22, 2012, 5:24pm

The OFS is just to make it print in nicely tabbed columns, WIDTH is all you need to change. I made everything depend on WIDTH even though that was slightly awkward just to keep things obvious...

It detects the number of columns by itself, via the NF special variable. Each column is calculated independently.

cosmologist · February 22, 2012, 5:35pm

When I use this set of numbers:

I expect the output:

But I get:

3005.6
3012.2
3018.8

Corona688 · February 22, 2012, 5:40pm

The question, really, is why you expect a different number of numbers, not the numbers you got... Just pasting it, as if it's obvious, doesn't help.

awk deals with numbers as floating point numbers, not text, so doesn't give you any particular number of decimal places unless you ask for a particular number of decimal places.

BEGIN { WIDTH=3; OFS="\t" }

{
        N++;
        for(M=1; M<=NF; M++)    DATA[M,N%WIDTH]=$M
}

(N>=WIDTH) && !((N-WIDTH)%(WIDTH-1)) {

        for(M=1; M<=NF; M++)
        {
                $M=0

                for(X=(N+1); X<=(N+WIDTH); X++)
                        $M+=DATA[M,X%WIDTH];

                # Print to 3 decimals of precision
                $M =sprintf("%.3f", $M/WIDTH);
        }

        print
}

cosmologist · February 22, 2012, 5:44pm

I didn't mean the decimals, but the actual numbers:

3005.600 3009.200 3012.800

and

3005.6 3012.2 3018.8

Corona688 · February 22, 2012, 5:52pm

Yesssssssssssssss... And the issue, again, is why you think it should be those numbers. :wall:

What calculation arrived at your numbers?

I only see three different sets of 23 numbers in that file:

01      3002.300
02      3002.600
03      3002.900
04      3003.200
05      3003.500
06      3003.800
07      3004.100
08      3004.400
09      3004.700
10      3005.000
11      3005.300
12      3005.600
13      3005.900
14      3006.200
15      3006.500
16      3006.800
17      3007.100
18      3007.400
19      3007.700
20      3008.000
21      3008.300
22      3008.600
23      3008.900

01      3008.900
02      3009.200
03      3009.500
04      3009.800
05      3010.100
06      3010.400
07      3010.700
08      3011.000
09      3011.300
10      3011.600
11      3011.900
12      3012.200
13      3012.500
14      3012.800
15      3013.100
16      3013.400
17      3013.700
18      3014.000
19      3014.300
20      3014.600
21      3014.900
22      3015.200
23      3015.500

01      3015.500
02      3015.800
03      3016.100
04      3016.400
05      3016.700
06      3017.000
07      3017.300
08      3017.600
09      3017.900
10      3018.200
11      3018.500
12      3018.800
13      3019.100
14      3019.400
15      3019.700
16      3020.000
17      3020.300
18      3020.600
19      3020.900
20      3021.200
21      3021.500
22      3021.800
23      3022.100

...and calculated them the way you showed, splitting into different sets, with the first and last numbers overlapping... Is there supposed to be more overlap in a larger set?