Can't input large file to variable

Shell Programming and Scripting
1

I have a file that I'm trying to place into a variable.
I've tried the following:

DATE=`date +%Y%m%d`
filez=$(cat /tmp/Test_$DATE.txt)

DATE=`date +%Y%m%d`
filez=$(</tmp/Test_$DATE.txt)

DATE=`date +%Y%m%d`
filez=$(`cat /tmp/Test_$DATE.txt`)

None of these lines allows the file to go into the variable, unless the file is small. If it is a small file, then it works. If it is a large file, I get the following error:

builds/ul11u3sru-gate/components/bash/bash-4.4/lib/sh/strtrans.c:66: cannot allocate 1448426861 bytes (3758187128 bytes allocated)
./scripttest.bash: xmalloc: /builds/ul11u3sru-gate/components/bash/bash-4.4/lib/sh/strtrans.c:66: cannot allocate 1448426861 bytes (3758187128 bytes allocated)
./scripttest.bash: xmalloc: /builds/ul11u3sru-gate/components/bash/bash-4.4/lib/sh/strtrans.c:66: cannot allocate 1448426861 bytes (3758187128 bytes allocated)

Anyone know how to fix this? Apparently the file size causes it.

2

Now, that I'd call ambitious: Define a 5GB variable!

Not sure you exceeded any system / configuration parameter, though. Looks more like you hit a limit of the virtual memory system / manager.

What be the reason to read an entire file into a variable in the first place?

1 Like
3

There are memory limits and this depends a lot on the version of bash you operating system, etc. You should restructure your script so it doesn't need to read the whole file into a variable.

Consider reading 1 line at a time from the file, or using external utilities to manipulate the file for you.

4

You almost certainly can accomplish exactly what you want without assigning that file to a variable. Few languages but shell afford you such easy and direct methods of streaming data anywhere you want.

What exactly do you want to do? Why do you want a 5 gig variable - what will you do with it?