Perl, open multiple files with wildcards

Shell Programming and Scripting
1

I have a question regarding Perl scripting.

If I want to say open files that all look like this and assign them to a filehandle and then assign the filehandle to a variable, how do I do this?

The file names are

strand1.fa.gz.tmp
strand2.fa.gz.tmp
strand3.fa.gz.tmp
strand4.fa.gz.tmp
...
...
strand19.fa.gz.tmp

I want to assign them to file handles similar to their name
strand1.fa.gz.tmp.fh
strand2.fa.gz.tmp.fh
...
...
etc

then I want to set variables so its

$strand1=<strand1.fa.gz.tmp.fh>
...
...

So far I have this

#! /usr/bin/perl

@files=glob("*.tmp");

foreach $data (@files) {
        open($data.fh, "<$data") || die "failed $data\n";
}

I don't know if using $data.fh is right or not. Should I push the filehandles into an array (is that possible)?

I want to do this so I don't have to open the file 19 times by writing 19 lines and then writing an additional 19 lines to assign them to a variable.

Thanks.

2

Do all 19 need to be open simultaneously? Can you revisit your algorithm or what you are trying to do? Since ultimately the results of your 19 file handles are ending up in memory (with your statement of $strand1=<strand1.fa.gz.tmp.fh>), why not just loop through the 19 files, read them into memory and set up appropriate data structures in memory?

As of Perl 5.6, you can pass an uninitialized reference (a scalar value) as a file handle and Perl will return a reference to the filehandle if the file can be opened. Read the section on indirect file handles in Perldoc Open Tutorial.

In theory, you can loop through your file names, pass an uninitialized scalar to open, and push the result onto a list. There may be issues with this in terms of maximum number of handles per process or total open files on your platform.

Alternatively, if you want to go the Perl OO route, you can use IO::Handle.

A third alternative (in Perl, there is ALWAYS other ways to do it!) is to directly insert references in the symbol table that are associated with your file handle. This method was common before Perl 5.6 but is deprecated now (ie, don't do it unless you really know what you are doing and it may not work in the next version of Perl). This method is outlined by Randall Schwartz in an article here.

Several things to note on your code snippet:

  • Use strict and use warnings;
  • Use the 3 argument form of Open;
  • Use Perl failure code to see why the file did not open;
  • Use lexically scoped file handles (my $fh) and you will not colide with other parts of your program with an open file. You will also not need to use a 'close' since Perl autocloses files as the handle goes out of scope.

If you move the globbing to the command line, you may be able to avoid the complexity. Perl, by default will open a file on the command line. So:

perl -nle [your script] *.tmp

Will handle all the dirty bits for you and you just need to focus on what you want done to the files. This does not work if you want all files open simultaneously. Look at Perldoc perlrun for the details of that.

1 Like
3

thanks for the reply, sorry for not being clear, all 19 files need to be opened simultaneously =/

4

What is it you want to do with the 19 files?

5

i have a file that contains information about genes and I need to obtain the sequences upstream and downstream these genes and do analysis on that. These 19 files are the chromosome sequences. I have sorted the gene file based on chromosome location so I can essentially use 1 chromosome file at a time since I just tried to open all of them and perl ran out of memory =/

6

So why do all 19 files need to be open if you are processing 1 file at a time?

Do you have simple logic and maybe the Perl experts here can help? ie, present pseudo code of what you want to do, size of files, output desired, intermediate processing needed, platform.

7

Hey drewk,

I realized I could process them once at a time especially after I ran out of memory, before, all I wanted to do was to open it all at once instead of writing switch or if-else statements to open all the files. thanks for all the help =)