Concatenation of a large number of files

Shell Programming and Scripting
1

Hellow i have a large number of files that i want to concatenate to one. these files start with the word 'VOICE_' for example
VOICE_0000000000
VOICE_1223o23u0
VOICE_934934927349

I use the following code:

 cat /ODS/prepaid/CDR_FLOW/MEDIATION/VOICE_* >> /ODS/prepaid/CDR_FLOW/WORK/VOICE

And i get an error that files are too long!

How can i overcome this?

Best Regards,
Christos

2

1) If you get an error message, please post the exact error message.
2) Always state what Operating System and Shell you are using.

One idea (assuming only one depth of directory tree) which avoids expanding "*" in the shell is:

find /ODS/prepaid/CDR_FLOW/MEDIATION/ -type f -name VOICE_\* -print | while read filename
do
        cat "${filename}" >> /ODS/prepaid/CDR_FLOW/WORK/VOICE
done
3

Try:

for i in /ODS/prepaid/CDR_FLOW/MEDIATION/VOICE_*
do
  cat "$i" >> /ODS/prepaid/CDR_FLOW/WORK/VOICE
done
4

:o--removed--

5

@Scrutinizer and ctsngb
When the O/P comes back with the O/S I predict that it will be AIX or something which cannot deal with long command lines after expansion of "*" in Shell.

Apologies to ctsgnb because our posts crossed.

6

No prob dude ! :slight_smile:

Solaris also has limitation to wildcard expansion :

$ uname -a
SunOS <anonymized> 5.9 Generic_122300-54 sun4u sparc SUNW,Sun-Fire-15000
$ ls | wc -l
   68616
$ ls -l *
bash: /usr/bin/ls: Arg list too long
$
7

@ctsgnb & off topic
It would not be beyond the Shell compilers to flag "" in certain constructs as a Syntax Error. If I had my way an "" would be a Syntax Error in a "for" command. Just my personal view about open-ended lists. Others may differ.

8

And do you get an error message when you do:

for i in *; do : ; done
9

@Scruti
You got me : the "for" loop runs fine .

Ok i got it : this is a limitation of the ls command (not the shell expansion)
correct ?

ls command should be coded to be able to handle huge number of argument. But I suppose it is choice of design, for performance reason.
Do you have any clue ?

$ print *
bash: print: command not found
$ sh print *
bash: /usr/bin/sh: Arg list too long
$
10

It is a limitation for external commands:

POSIX: limits.h

So the number of arguments that external commands can handle is limited by ARG_MAX. This is not the case for shell builtins.

1 Like
11

It's an operating system limitation. They can't. Ergo, you should program in ways that don't cram potentially unlimited numbers of arguments into a commandline. In some shells, you can get away with cramming billions of arguments into a builtin (and ONLY a builtin), but you really can't depend on that.

Bad: cat * > output
Good: ls | xargs cat > output xargs understands maximum arguments and can split them intelligently across several cat calls.

The same goes for things like for LINE in `cat foo` ; do ... ; done when foo exceeds 4K in size you may discover the maximum size of a shell variable on your system. Instead: while read LINE ; do ... ; done < foo

Essentially the idea's the same in all cases: Never force the shell to hold an entire anything in memory. (That actually applies to most programming languages, but shell makes it deceivingly easy to do so...) You'll hit walls at inconvenient times and it's never efficient. Process in bits.