File size limitation of unix sort command.

Shell Programming and Scripting
1

hi ,

iam trying to sort millions of records which is delimited and i cant able to
use sort command more than 60 million..if i try to do so i got an message stating that "File size limit exceeded",Is there any file size limit for using sort command..
How can i solve this problem.

thanks

cskumar

2

what version of unix/sort exactly?
gnu sort for example supports files > 2GB,
and efficiently sorts/merges files > RAM size

3

Hi Pixelbeat,

Thanks for your quick response. We are trying this on HP UX 11. We are generating input file for sorting through c program and calling unix sort command through c only, as follows::

#include <stdio.h>
#include <string.h>
#include <stdlib.h>

FILE *fout;
int main(int argc,char *argv[])
{
int loopcnt=0;
unsigned long int TTRECS = 60000000;

	if\(\(fout = fopen\("SORTINPUT","w"\)\) == NULL\)
		\{
			printf\("\\nUnable to open the output_file file\\n"\);
			exit\(1\);		
		\}
		for \(loopcnt=TTRECS;loopcnt&gt;=1;loopcnt--\)
		\{
			fprintf\(fout,"%d|%s%d|%s|\\n",loopcnt,"Dummy",loopcnt,"Dummy Description"\);
		\}
	fflush\(fout\);
	printf\("Executing::sort -t '|' -k 1,1 SORTINPUT&gt; sorttestoutput\\n"\);

	system\("echo Sort Start Time;date"\);
	system\("sort -t '|' -k 1,1 SORTINPUT &gt; sorttestoutput"\);
	system\("echo status returned::$?"\);

    system\("echo Sort End Time;date"\);
	printf\("Total Records Configured:: %lu\\n", TTRECS\); 
	return 0;
\}

We are trying this test just to make sure that unix sort command is not having any limitations on file size. But what we observed is when file is packed with more than 60 million records [ around 2 Gb size], we are getting an error stating "File size limit exceeded" on console. Can u kindly state,

1] whether we got any maximum file size limit in HP UX while creating a file through C program.

2] Can unix sort command can handle file size > 2 GB without any failure.

3] Is there any restriction for length of each record [each line] or each field[fields are separated by a delimeter].

Thanks,

4

I think it's quite possible that you have such a file size limitation.
If I were you I'd try to edit your C code to see if the error occurs before the sort
command is called or not. I'd say it's before it, because I think the error might
begin by "sort: " if the error was caused by sort.

5

thanks for ur reply chlorine,

Yes as u said error is before sort only as we couldn't create a file of size more than 2 GB through c.
[1] Can u suggest a way to get rid of that size problem.

[2] Can u let us know whether we got any such file size limitation for sort command.

6

I have no idea if it is feasible to change the filesize limit.
For sort, the man page should tell you exactly the limitations. If the man page doesn't mention any limitations then there should be none.

7

Ah OK, you need to use interfaces
in your program that allow you to have
files > 2GB.

For linux (glibc) you can do it explicitly
in the source by defining _LARGEFILE64_SOURCE
and then calling open(,O_LARGEFILE), stat64() etc.

You can also get everything working implicitly
in 64 bit mode by just defining _FILE_OFFSET_BITS=64

8

try

ur sort with -T option

directing sort to make use of temporary directory for sorting intermediate files specified through the -T option