read() wont allow me to read files larger than 2 gig (on a 64bit)

Hi the following c-code utilizing the 'read()' man 2 read method cant read in files larger that 2gig.
Hi I've found a strange problem on ubuntu64bit, that limits the data you are allowed to allocate on a 64bit platform using the c function 'read()'

The following program wont allow to allocate more that 2.1gig memory.

#include <stdio.h>
#include <stdlib.h>
#include <err.h>
#include <fcntl.h>
#include <sysexits.h>
#include <unistd.h>
#include <sys/stat.h>

// get bytesize of file
size_t fsize(const char* fname){
  struct stat st ;
  stat(fname,&st);
  return st.st_size;
}

int main() {
  const char *infile = "bigfile.dat";
  int fd;
  size_t bytes_read, bytes_expected = fsize(infile);
  char *data;
 printf("\nLONG_MAX:%lu\n",LONG_MAX);

  if ((fd = open(infile,O_RDONLY)) < 0)
    err(EX_NOINPUT, "%s", infile);

  if ((data =(char *) malloc(bytes_expected)) == NULL)
    err(EX_OSERR, "data malloc");


  bytes_read = read(fd, data, bytes_expected);

  if (bytes_read != bytes_expected)
    err(EX_DATAERR, "Read only %lu of %lu bytes",bytes_read, bytes_expected);

  /* ... operate on data ... */

  free(data);

  exit(EX_OK);
}

./a.out
LONG_MAX:9223372036854775807
a.out: Read only 2147479552 of 2163946253 bytes: Success

According to man 2 read, the maximum is limited by SSIZE_MAX
which is defined in

/usr/include/bits/posix1_lim.h
# define SSIZE_MAX LONG_MAX

And LONG_MAX is defined in /usr/include/limits.h as
# if __WORDSIZE == 64
# define LONG_MAX 9223372036854775807L
# else
# define LONG_MAX 2147483647L
# endif
# define LONG_MIN (-LONG_MAX - 1L)

Either this is a bug in the ubuntu buildsystem,
or my build system is broken.

Can anyone with a 64 try to run the above program.

Thanks

edit:

by the way

readelf -h ./a.out
ELF Header:
  Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
  Class:                             ELF64
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              EXEC (Executable file)
  Machine:                           Advanced Micro Devices X86-64
  Version:                           0x1
  Entry point address:               0x400750
  Start of program headers:          64 (bytes into file)
  Start of section headers:          5312 (bytes into file)
  Flags:                             0x0
  Size of this header:               64 (bytes)
  Size of program headers:           56 (bytes)
  Number of program headers:         9
  Size of section headers:           64 (bytes)
  Number of section headers:         37
  Section header string table index: 34

ldd ./a.out
        linux-vdso.so.1 =>  (0x00007fff689ff000)
        libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007ffee433e000)
        libm.so.6 => /lib/libm.so.6 (0x00007ffee40ba000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00007ffee3ea3000)
        libc.so.6 => /lib/libc.so.6 (0x00007ffee3b34000)
        /lib64/ld-linux-x86-64.so.2 (0x00007ffee464e000)

thanks in advance

ssize_t largest 32 bit value is ~2.1GB, which you seem to know. I had ubuntu about several years ago, and there was a largefile option you had to enable:

cc -D_FILE_OFFSET_BITS=64 myfile.c -o myfile

Is that necessary even with a 64-bit compile in Ubuntu? The OP's "readelf" output, and the results of his LONG_MAX output show he's running a 64-bit binary.

As you can see in the code I supplied

size_t bytes_expected = fsize(infile);
  if ((data =(char *) malloc(bytes_expected)) == NULL)
    err(EX_OSERR, "data malloc");

The problem is not related to the size_t, part but solely to the 'read' function.
If it was a problem with the size_t the malloc would complain.

The max size_t on a ubuntu64 is 18446744073709551615.

The compiler flag didn't work.

Hello ,
I suppose ext3 filesystem doesnt support reading file sizes more than 2 GB
2GB Filesize Limit

Of cause ext3 support larger than 2gig files on a 64bit platform

try
dd if=/dev/zero of=bigfile bs=1024 count=5145728

Just a sentence. Probably there is an error here that is alike the one described: "Problems of 64-bit code in real programs: FreeBSD".

Two things

Read over the man page for lseek64 and make sure you are using the correct compiler flags. i do not believe you are.

Secondly, The read system call does guarantee that it will fill a buffer completely. Trying issuing a second read and see if you get your data.

Thanks for all your replies,
but it seems that its not possible to use posix read to read files larger than 2.1 gig.
This is according to the uberpenguin linus
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=e28cc715

At somepoint I guess it was possible but the kernel has regressed to only allow smaller chunks being read.

kernelsourcetree/fs/read_write.c

/*
 * rw_verify_area doesn't like huge counts. We limit
 * them to something that fits in "int" so that others
 * won't have to do range checks all the time.
 */
#define MAX_RW_COUNT (INT_MAX & PAGE_CACHE_MASK)

int rw_verify_area(int read_write, struct file *file, loff_t *ppos, size_t count);

This was rather annoying, it would have been nice if this had been documented outside of the kernelsource and kernelmailinglist.

Wait -- you're trying to read >2gb all at once?? Do you actually need it all at once? If not, this is extremely bad programming practice!

Why not just map in the file? That'll turn it into a memory area with one call, letting you use it like you'd loaded it all at once without actually having to do so, it'll autoload it as you use it(though you can force it to load it all with the MAP_POPULATE flag). See man mmap.

No, it is most definitely *not* an "extremely bad programming practice". I've worked on real-world systems that access tens if not hundreds of gigabytes at a time. In RAM. They *do* exist.

This code:

/*
 * rw_verify_area doesn't like huge counts. We limit
 * them to something that fits in "int" so that others
 * won't have to do range checks all the time.
 */
#define MAX_RW_COUNT (INT_MAX & PAGE_CACHE_MASK)

int rw_verify_area(int read_write, struct file *file, loff_t *ppos, size_t count);  

is nothing more nor less than utter abdication of 64-bit processing.

They certainly do exist, hence my asking whether he actually needed it all in RAM at once. Nine out of ten times I see someone slurping an enormous wad of data into RAM, they'd have been better off doing an mmap...

achenle - just becuase you saw somebody do something does not make it good practice.
Even in commerical software.

My company bought a package (C/shell/perl) that had non-terminating loops in it.
Does that mean it was good practice - or good software? We sent it back.

There's a very good reason to read a large amount of data with just one system call: in a contended file system, doing so allows for better scheduling of IO to data blocks that are more likely to be contiguous on disk, thus reducing overhead lost to disk head seeks. And I haven't SEEN it done, I've DONE it. And improved IO throughput on said system by about 20 or 30 percent.

Fortunately for the customer, the OS wasn't Linux, which has punted the ability to do large read and writes.

I guess 2 gig is enough for everybody, right?

Your company buying buggy software is about as relevant as the color of your car.

For the third time, I'm sure you have. I'm sure they must be working on correcting the problem that prevents two-gig reads, as well; the sticky issue is likely how to manage that on 32-bit systems.

You weren't the OP, though, and my question was what these enormous reads were for in the first place. 9 out of 10 times I see someone sucking an enormous wad of raw data into RAM, they're not doing it for performance reasons -- they just want it all in RAM to get at it without further system calls. An mmap gets them that without decimating the cache and squeezing frequently-used things into swap for no good reason, and avoids this problem entirely.