Thanks for the response. I'm using Ubuntu 10.10.
Trying out your suggestion from a shell, I am no longer getting a error message but I'm also not getting any output either. Do you have an idea why?
DESCRIPTION
The look utility displays any lines in file which contain string as a
prefix. As look performs a binary search, the lines in file must be
sorted (where sort(1) got the same options -d and/or -f that look is
invoked with).
...so look really does need to know how long the file is.
grep should work since it doesn't, but won't have the performance advantage of being able to do a binary search.
yes... 'look' doesn't do quite what I thought it did. When you give it no input file it tries to use something in /usr/share/dict!
So no, redirection won't work in this case.
Annoying that your version of look can't support large files on a 64-bit system! This may be a bug.
---------- Post updated at 01:53 PM ---------- Previous update was at 01:46 PM ----------
Looking at the code it doesn't have an explicit built-in limit, it just uses mmap. When I have access to a 64-bit machine later today I'll see what mmap does on 64-bit systems for enormous files.
Assuming your file is sorted and has some reasonablly high H value (the leading characters change) try some kind of a radix split to get small files.
For this example assume that most of the letters of the alphabet are found as the leading character of a record and they are all uppercase. This means you can have ~26 smaller files, all of which look will work on.
I tested the "look" command on a remote server through ssh. The machine has a Darwin Kernel Version 9.8.0 (32-bit). I'm getting the same error message. "File too large". Again, thank you for all the help.
Just did some testing at home. I don't think your Linux system is actually 64-bit; I just mmaped an entire 650-gigabyte file in 64-bit Linux. Granted that was a sparse file, so I went and mapped in all of /dev/sda next.
Even if you have a 64-bit processor, you get none of the benefits unless you install a 64-bit operating system -- namely, each process is limited to 4 gigabytes of virtual memory at most on a 32-bit system.
Naturally 32-bit OSX would have this limit too. Changing the OS without changing the number of bits won't give you more address space.