Unix Arithmatic operation issue , datatype issue

Hi,
I have a shell scripting. This will take 7 digit number in each line and add 7 digit number with next subsequent lines ( normal addition ).

Eg:

0000001
0000220
0001235
0000022
0000023
...........
.........
........

Like this i am having around 1500000 records. After adding , I am getting the result 2147483647 but actual result is 2156379608 . I found the root cause of this issue is that Unix temproary variable can hold only 2156379608 ( Range of integer ) as it's 32 bit. For eg:

a=2147483647
expr $a + 2

you will get the negative result. Because while expr working, the result will be stored in internal temporary variable or register then you will get the result.. but that particular temporary variable or register can accomodate only 2147483647 .. if it crosses this limit, you may get the junk value like -ve values... this is my finiding for this issue.. But I want to have resolution for this issue. how to add or do arthimatic opration if i want to have result more than limit ( 2147483647 ). I mean

a=2147483647
expr $a + 2

for this i need to get the right result rather than -ve value.

This will handle up to 15 signifcant digits:

 awk ' {total+=$0} END { print total} ' file

This will handle bigger numbers depending on your version of bc:

#/bin/ksh
total=0
while read value
do
    total=$(echo "$value + $total" | bc )
done < file
echo $total

Hi ,

Thanks. We are also using the awk for arthitmatic operation . But we are not getting exepected result. I tried with your code as well.

$ awk ' {total+=$0} END { print total} ' hash_column_wrong.txt
2.15638e+09

I got the result in expontial format. But I need the result with all the digits. How to achive this?. Then second method which you shown using the bc calculator is taking more time to complte. So I think awk is the best method. As of now , I am getting result as 2.15638e+09 . but I need to get result like 2156380000 . Please help me.

Regards,
Thambi

can you please provide the list of hash_column_wrong.txt ????

As I already mentioned, it just contains the 7 digiti number in every row. Like these, this file has around 150000 . We just need to add all the rows..that's all

0000001
0000343
0000001
0001426
0000001
0000718
0000162
0000103
0000021
0000011
0000016
0000312
0000026
0000031
0000005
0000022
0000001
0000001
0011845
0003713
0000001
0000011
0000001
0000001
0000001
..,.,
........
.......

Use awk to prepare the file for bc.

{ awk '{ print $0 " +\\" }' bigfile; echo 0; } | bc

Assuming bc is using the arbitrary precision library, you shouldn't have a problem here.

When I run the below command

awk '{ print $0 " +\\" }' hash_column_wrong.txt | bc

I got below exception..

bundling space exceeded on line 1,
bundling space exceeded on line 1,
bundling space exceeded on line 1,

is it due to \n character that occurs end of the line?.

Sounds like bc can't handle the huge line size. try this:

awk '{ print "last + " $0  }' hash_column_wrong.txt | bc  | tail -1

This prints out a sub-result for each line. The only catch is that "last" is a GNU extension.

No . .it's not working..

When I run

$ awk '{ print "last + " $0 }' hash_column_wrong.txt | bc | tail -1
syntax error on line 1,

When I run without tail..

$ awk '{ print "last + " $0 }' hash_column_wrong.txt | bc
syntax error on line 1,

When I run only , i got the result like this...

last + 0000191
last + 0000005
last + 0000001

I think format might be wrong ?.

No, it means that the bc isn't the GNU one. As I mentioned, the "last" keyword is specific to GNU. You can try downloading and installing it. Run "bc --version" afterwards to make sure you're using the GNU version. You might also be asked to install the arbitrary precision library, which I recommend using.

Is it free ware?.. bc is available in my Unix version. As you said, last keyword may not compitable with bc version. Do you mean i need to download from the site?. This is impractical for me becuase .. there are lot of things need to consider as it's client production server.. If you know any other alternate way, pls let me know.

Okay, if I say "GNU", yes, that means it's free.

Now, if you are using a Linux system, obviously the problem is something else (because that's what you would be using). If you have a Sun system, you can get install a CSW "package". Ask the site administrator if s/he knows about CSW/blastwave packages. Here's the link for gnu's bc: CSW - Details for gbc. Click here for the source which you can download and install.

Now, if that's not possible, you can take my original script and find out how many lines ending in "\" that bc can handle. Then you can use split to divide the file into that many lines:

split -l ?? bigfile

where ?? is the number of lines. You'll get lots of little files. Add them up individually with the awk-bc script (or another method). Then add the output up, again using awk-bc. You might have to recurse over the first step until you have a tree of sums.

Or, you can use perl. See next message.

Hi ,I am using HP-UX B.11.11 ( HP unix ). If i get the freeware for HP unix, I can install the GNU bc with help of Unix admin. Please provide the link for GNU bc of HP unix.

I can't spilit the file as it's a source file which comes from source sytem. I should not touch that.. and I am pasting here my awk program to add the number.. Please look at the last "else " part which shows the addition ... in this existing code only, I need to amend the changes to get the total count with full digit rather the expontial formt..

BEGIN {
totalLatestRetailPrice = 0;
}
#
{
if (substr($0,1,7)=="HCDWC01")
{
printf "%s%58s\n", substr($0,1,28), " "
}
else
if (substr($0,1,7)=="TCDWC01")
{
printf "%s%015d%44s\n", substr($0,1,27), totalLatestRetailPrice,
" "
}
else
{

            totalLatestRetailPrice \+= int\(substr\($0,14,7\)\)
            print
    \}

}

I'm on a 64-bit machine, and awk and Perl seem to work fine here. I don't have access to a 32-bit machine at the moment. But try it with perl:

#!/usr/bin/env perl -w
$bc_lines=100;

local $maxint=int(2*(2**30 - 1)+1);
my $a=0;
my @subresults;

while ($_=<>) {
  chop;
  if ($maxint - int($_) < $a) {
    push @subresults, $a;
    $a=0;
  }
  $a+=int($_);
}
while (scalar(@subresults)) {
  print join("+\\\n",splice(@subresults,0,$bc_lines),"0\n");
}

Now run this script, pipe through bc *recursively* until there is only one line left. For example:

cp bigfile.txt outfile.txt 
while [ ! -f outfile.txt -o `grep -c ^ outfile.txt` -gt 1 ]; do 
  perl test.pl  outfile.txt  |bc >outfile.$$ ; 
  mv outfile.$$ outfile.txt; 
  echo "Subcalc complete" >&2; 
done ; 
cat outfile.txt

If bc complains with the same error, lower 100 to, say, 50. Keep cutting in half till no more complaints.

the GNU bc link is in my previous post.

Sorry, I have not worked on Perl scripting.. it's new to me... I feel GNU bc method will be ok for me.. if you guide me the installation way, i can probably try with that method.....

And also I have just found that my machine is 64 bit..

$ getconf KERNEL_BITS
64

now is there anyway to get my result?

Depends on if the programs are compiled in 64-bit. Try

file `which bc` `which awk` `which perl`

Is it a 64-bit or 32-bit executable? (You might have to find the paths manually).

For installing bc, you'll have to go to the home page, whose link I provided, and follow the steps. You'll probably need the sysadmin's help, but in theory, you can do something like:

gunzip -c gbc-*tgz | tar xf - 
cd gbc????
./configure --prefix=$HOME
make install

I got the below result...

$ file `which bc` `which awk` `which perl`
/usr/bin/bc: PA-RISC1.1 shared executable dynamically linked
/usr/bin/awk: PA-RISC1.1 shared executable dynamically linked
/usr/bin/perl: PA-RISC1.1 shared executable dynamically linked -not stripped
$

Meantime, I just waiting for my Unix admin's response to install GNU bc. If i install GNU bc, current version bc will be affected. I mean GNU bc having all the features ?

Yeah, so these are the 32-bit versions. Programs that report RISC2.0 are the 64-bit versions. (See 8.33 What are the various revisions of PA-RISC?). So even though the hardware can support 64-bit integers, the programs cannot. Even installing the 64-bit version of these programs would solve your problem.

So have him install it in /usr/local/bin or in /usr/local/gnu or in your home directory ($HOME/bin) and make sure your PATH prefixes these, or use the full path, or have the program installed as "gbc".

Hi,
Thanks. I have installed GNU bc version in development which is working fine as well..

$ awk '{ print "last + " $0 }' hash_column_wrong.txt | /usr/local/bin/bc | tai
l -1
2156379608
$

But I am trying to match this code with my existing awk script.

My existing awk script ..

BEGIN {
totalLatestRetailPrice = 0;
}
#
{
if (substr($0,1,7)=="HCDWC01")
{
printf "%s%58s\n", substr($0,1,28), " "
}
else
if (substr($0,1,7)=="TCDWC01")
{
printf "%s%015d%44s\n", substr($0,1,27), totalLatestRetailPrice,
" "
}
else
{
totalLatestRetailPrice += int(substr($0,14,7))
print
}
}

I am calliing above awk script like this....

awk -f header_trailer/creation/hash_total.awk input_file > output_file

here input_file is the parameter here.. we are taking the digits from 14 - 20 ( 7 digits )
1335310 000001030000036000000092H99W710172S300

( Before, I mentioend hash_column_wong.txt file which is created from above input_file for my testing ). .. In the "else" part , logic is there to find out the total value..

else
{
totalLatestRetailPrice += int(substr($0,14,7))
print
}

We need to replace the above line ( totalLatestRetailPrice += int(substr($0,14,7)) with
our new bc caluclator line ( awk '{ print "last + " $0 }' hash_column_wrong.txt | /usr/local/bin/bc | tail -1 ).. here we don't need to pass the has_column_wrong.txt file since we are already passing input_file..