"Argument list too long" error

Shell Programming and Scripting
1

Hi everyone,

I have a problem with my shell script. As a quick overview I need to change a template file 6561 times and copy the file into a new catalogue. Thanks to your forum I have managed to write a script to do so:

#!/bin/sh

template=$1
for values in {45,165,285}\ {45,165,285}\ {45,165,285}\ {45,165,285}\ {45,165,285}\ {45,165,285}\ {45,165,285}\ {45,165,285}; do
    set $values
    A=$1; B=$2; C=$3; D=$4; E=$5; F=$6; G=$7; H=$8
    mkdir "A$A-B$B-C$C-D$D-E$E-F$F-G$G-H$H"
    cp en ./"A$A-B$B-C$C-D$D-E$E-F$F-G$G-H$H"
    cp nf ./"A$A-B$B-C$C-D$D-E$E-F$F-G$G-H$H"
    cp define.sh ./"A$A-B$B-C$C-D$D-E$E-F$F-G$G-H$H"
    cp cosmoprep.sh ./"A$A-B$B-C$C-D$D-E$E-F$F-G$G-H$H"
    sed "s/A\$/$A/;s/B\$/$B/;s/C\$/$C/;s/D\$/$D/;s/E\$/$E/;s/F\$/$F/;s/G\$/$G/;s/H\$/$H/" "template" > ./A$A-B$B-C$C-D$D-E$E-F$F-G$G-H$H/"A$A-B$B-C$C-D$D-E$E-F$F-G$G-H$H"
    cd ./A$A-B$B-C$C-D$D-E$E-F$F-G$G-H$H
    babel -igzmat "A$A-B$B-C$C-D$D-E$E-F$F-G$G-H$H" -oxyz coord.xyz
    source $USELIB/tm510
    x2t coord.xyz > coord
    sh define.sh
    sh cosmoprep.sh
    cd /yki/mount/glauber_home/mario8/Desktop/magister/arvutused/6/
done

Finishing this script takes a lot of time and it went well after I checked it after an hour, but at some point it starts to give errors:

def.sh: line 7: /bin/mkdir: Argument list too long
def.sh: line 8: /bin/cp: Argument list too long
def.sh: line 9: /bin/cp: Argument list too long
def.sh: line 10: /bin/cp: Argument list too long
def.sh: line 11: /bin/cp: Argument list too long
def.sh: line 12: ./A285-B285-C285-D285-E285-F285-G285-H285/A285-B285-C285-D285-E285-F285-G285-H285: No such file or directory
def.sh: line 13: cd: ./A285-B285-C285-D285-E285-F285-G285-H285: No such file or directory
def.sh: line 14: /usr/local/bin/babel: Argument list too long
/usr/yki/lib/use/tm510: line 30: /bin/hostname: Argument list too long
/usr/yki/science/turbomole-5.10/Config_turbo_env: line 19: /usr/yki/science/turbomole-5.10/scripts/sysname: Argument list too long
def.sh line 16: /usr/yki/science/turbomole-5.10/scripts/x2t: Argument list too long
def.sh: line 17: /bin/sh: Argument list too long
def.sh: line 18: /bin/sh: Argument list too long

I tried to name a folder with the maximum characters my script had and it worked. So I can't see the problem.

Please, help me.

Mario

2

Yes - but it is the length of the FULL pathname including your directoryname, e.g.,
/path/to/my/list/of/files/./A285-B285-C285-D285-E285-F285-G285-H285
that the system is complaining about.

grep PATH_MAX /uysr/include/limits.h

will tell you the limit of the number of characters in a fully qualified pathname. Start with that to check your filename problems.

1 Like
3

Thank you very much. I tried it, but it didn't find "PATH_MAX" in the file. So I just tried "MAX". I hoped there is an equivalent keyword there, but I didn't find anything I could relate to pathname. I copy the results here, maybe you might notice it:

mario8@slater (~) >grep PATH_MAX /usr/include/limits.h
mario8@slater (~) >grep MAX /usr/include/limits.h
#define MB_LEN_MAX      16
#  define SCHAR_MAX     127
#  define UCHAR_MAX     255
#   define CHAR_MAX     UCHAR_MAX
#   define CHAR_MAX     SCHAR_MAX
#  define SHRT_MAX      32767
#  define USHRT_MAX     65535
#  define INT_MIN       (-INT_MAX - 1)
#  define INT_MAX       2147483647
#  define UINT_MAX      4294967295U
#   define LONG_MAX     9223372036854775807L
#   define LONG_MAX     2147483647L
#  define LONG_MIN      (-LONG_MAX - 1L)
#   define ULONG_MAX    18446744073709551615UL
#   define ULONG_MAX    4294967295UL
#   define LLONG_MAX    9223372036854775807LL
#   define LLONG_MIN    (-LLONG_MAX - 1LL)
#   define ULLONG_MAX   18446744073709551615ULL
   LLONG_MAX, and ULLONG_MAX.  Instead only the values gcc defined for
#  ifndef LLONG_MAX
#   define LLONG_MAX    LONG_LONG_MAX
#  ifndef ULLONG_MAX
#   define ULLONG_MAX   ULONG_LONG_MAX

Thank you in advance,

Mario

Edit: Maybe I should shorten the script by deleting the letters and only leave the degrees (numbers) in the folder names, since I analyze the results later with another sript and due to that I don't need the letters so badly?

4

We need to know exactly what Operating System you have and the exact version (with all decimal places).

uname -a

(Blot anything confidential).

Between this post and your previous post about the same script it would appear that the "bash" Shell is keeling over with insufficient environment space and generating spurious or blank command parameters.
Btw. Googling the "mkdir" error message only comes up with Gentoo Linux.

There is excessive repetiton of parameter substitution.
Just saving the common factor for an iteration ready for the next time it is required will save over 400,000 shell parameter substitutions.
e.g.

INSTANCE="A$A-B$B-C$C-D$D-E$E-F$F-G$G-H$H"
5

I am not sure if this is helpful:

mario8@yukawa (~) >uname -a
Linux yukawa 2.6.18-6-k7 #1 SMP Fri Feb 19 23:58:00 UTC 2010 i686 GNU/Linux

But I know, that I am using Debian.

I am not sure if I understood you correct. Are you saying that the script is not working because it has to change too many parameters in one turn and all together it has too many parameters?

And also if you have the time could you provide a link on how to use the command instance exactly, because Googling for it gives me wrong shells and wrong instances :smiley: .

Thanks in advance,

Mario

Edit: I just cut the script into little pieces and it seems everything works until the shell has to use "source" command.

#!/bin/sh

template=$1
for values in {45,165,285}\ {45,165,285}\ {45,165,285}\ {45,165,285}\ {45,165,285}\ {45,165,285}\ {45,165,285}\ {45,165,285}; do
    set $values
    A=$1; B=$2; C=$3; D=$4; E=$5; F=$6; G=$7; H=$8
    mkdir "A$A-B$B-C$C-D$D-E$E-F$F-G$G-H$H"
done

These kind of small scripts worked fine.

Even this worked ok:

#!/bin/sh

template=$1
for values in {45,165,285}\ {45,165,285}\ {45,165,285}\ {45,165,285}\ {45,165,285}\ {45,165,285}\ {45,165,285}\ {45,165,285}; do
    set $values
    A=$1; B=$2; C=$3; D=$4; E=$5; F=$6; G=$7; H=$8
    cd ./A$A-B$B-C$C-D$D-E$E-F$F-G$G-H$H
    babel -igzmat "A$A-B$B-C$C-D$D-E$E-F$F-G$G-H$H" -oxyz coord.xyz
    cd /yki/mount/glauber_home/mario8/Desktop/magister/arvutused/4/
done

When it had to deal with the Turbomole program (tm510) then it worked good at first, but at some point it started to give the "argument list too long" error:

x2t.sh: line 9: /usr/yki/science/turbomole-5.10/scripts/x2t: Argument list too long
/usr/yki/lib/use/tm510: line 30: /bin/hostname: Argument list too long
/usr/yki/science/turbomole-5.10/Config_turbo_env: line 19: /usr/yki/science/turbomole-5.10/scripts/sysname: Argument list too long
Config_turbo_env: TURBOMOLE script sysname failed!

So I am really confused, because it seems to work at first, but fails at some point. And I removed most of the variables. And I can't understand why the mkdir or babel command worked well and x2t wont work.

6

You've got the right approach to diagnose the fault.

Please post x2t.sh .

When executed with "source" it becomes part of def.sh and could well be upsetting def.sh.

1 Like
7

Thank you for the compliment :slight_smile: .

The x2t.sh looks like this:

#!/bin/sh

template=$1
for values in {45,165,285}\ {45,165,285}\ {45,165,285}\ {45,165,285}\ {45,165,285}\ {45,165,285}\ {45,165,285}\ {45,165,285}; do
    set $values
    A=$1; B=$2; C=$3; D=$4; E=$5; F=$6; G=$7; H=$8
    cd ./A$A-B$B-C$C-D$D-E$E-F$F-G$G-H$H
    source $USELIB/tm510
    x2t coord.xyz > coord
    cd /yki/mount/glauber_home/mario8/Desktop/magister/arvutused/4/
done

Mario

Edit: It seems you are right, the source command screws it up. I did it like this and it worked:

mario8@xan01 (test) >use tm510
You can use Turbomole 5.10 now. Type "exit" to leave this mode.
Calculations will be carried out on a single CPU core.
[TM510] mario8@xan01 (test) >sh x2t_2.sh
[TM510] mario8@xan01 (test) >

#!/bin/sh

template=$1
for values in {45,165,285}\ {45,165,285}\ {45,165,285}\ {45,165,285}\ {45,165,285}\ {45,165,285}\ {45,165,285}\ {45,165,285}; do
    set $values
    A=$1; B=$2; C=$3; D=$4; E=$5; F=$6; G=$7; H=$8
    cd ./A$A-B$B-C$C-D$D-E$E-F$F-G$G-H$H
    x2t coord.xyz > coord
    cd /yki/mount/glauber_home/mario8/Desktop/magister/arvutused/4/
done

I am going to test the last few lines like that aswell.

Mario

Edit 2: Thank you very much, it worked. "Source was the problem".