Randomly selecting sequences and generating specific output files

I have two files containing hundreds of different sequences with the same Identifiers (ID-001, ID-002, etc.,), something like this:
Infile1:

ID-001 ATGGGAGCGGGGGCGTCTGCCTTGAGGGGAGAGAAGCTAGATACA 
ID-002 ATGGGAGCGGGGGCGTCTGTTTTGAGGGGAGAGAAGCTAGATACA 
ID-003 ATGGGAGCGAAGGCGTCTGTTTTGAGGGGAGAGAAGCTAGATACA 
ID-004 ATTTGAGCGGGGGCGTCTGTTTTGAGGGGAGAGAAGCTAGATACA 
ID-005 ATGGGAGCGGGGGCGTCTGTTTTGAGGGGAGAGAAGCTAGATAGT

Infile2:

ID-001 ATGGGAGCGGGGGCGTCTGCCTTGAGGGG 
ID-002 ATGGGAGCGGGGGCGTCTGTTTTGAGGGG 
ID-003 ATGGGAGCGAAGGCGTCTGTTTTGAGGGG 
ID-004 ATTTGAGCGGGGGCGTCTGTTTTGAGGGG 
ID-005 ATGGGAGCGGGGGCGTCTGTTTTGAGGGG

I need an awk script that can randomly select a number of sequences, let say 3, with the same IDs from both input files and generate two different outfiles, something like this:
Outfile1-1:

ID-001 ATGGGAGCGGGGGCGTCTGCCTTGAGGGGAGAGAAGCTAGATACA 
ID-003 ATGGGAGCGAAGGCGTCTGTTTTGAGGGGAGAGAAGCTAGATACA 
ID-005 ATGGGAGCGGGGGCGTCTGTTTTGAGGGGAGAGAAGCTAGATAGT

Oufile2-1:

ID-001 ATGGGAGCGGGGGCGTCTGCCTTGAGGGG 
ID-003 ATGGGAGCGAAGGCGTCTGTTTTGAGGGG 
ID-005 ATGGGAGCGGGGGCGTCTGTTTTGAGGGG

I need to repeat this process a number of times, let say 10 times, and generate the corresponding Outfiles (Outfile1-2 and Outfile2-2; Outfile1-3 and Outfile2-3; Outfile1-4 and Outfile2-4; etc.,). Thus, the second pair of files will look like this:
Outfile1-2:

ID-003 ATGGGAGCGAAGGCGTCTGTTTTGAGGGGAGAGAAGCTAGATACA 
ID-004 ATTTGAGCGGGGGCGTCTGTTTTGAGGGGAGAGAAGCTAGATACA 
ID-005 ATGGGAGCGGGGGCGTCTGTTTTGAGGGGAGAGAAGCTAGATAGT

Outfile2-2:

ID-003 ATGGGAGCGAAGGCGTCTGTTTTGAGGGG 
ID-004 ATTTGAGCGGGGGCGTCTGTTTTGAGGGG 
ID-005 ATGGGAGCGGGGGCGTCTGTTTTGAGGGG

I really do not know how to do the randomization, and therefore, any help will be greatly appreciated.

In you output samples, the lines in each set of output files are all in the order in which they appeared in the input files. Is that a requirement for your output, or is it just a coincidence in the random numbers used for your example?

With truly random numbers, the output could contain more than one copy of some output lines. Is it a requirement that the output lines be unique?

Don,
The sequences should be unique.
Thanks,
X

Is your heart set on awk? Would you consider a shell script?

Here is a start in awk you could try. The first file is read twice solely to determine the number of records..

awk -v s=3 -v iter=1 '
  NR==FNR { 
    next 
  }
  FNR==1 {
    if(!set) {
      srand();
      n=NR-1
      for(i=1; i<=s; i++) {
        line=0
        while(!line || line in A) line=int(rand()*n)+1
        A[line]
      }
      set=1
    }
    close(f)
    f=FILENAME ".out" iter
  } 
  FNR in A {
    print > f
  }
' infile1 infile1 infile2

You could embed it in a shell loop that increases the iter variable ( -v iter="$loopvar" )

--

Isn't that a matter of sampling with/without replacement?

Yes. My 2nd question was whether the output is sampling with or without replacement. And, Xterra has responded that each output set is to use sampling without replacement.

My 1st question was whether the random sequences 1, 3, 5 and 1, 5, 3 (and the six other orders of those three values) are to be treated as 8 distinct output sequences or they should all be normalized to the single sequence where the output lines are in the same order as they were in the input files. That question hasn't been answered yet, so I'm assuming they are to be treated as distinct sequences. (I expect to have an awk program with a shell wrapper to set options later today unless someone else comes up with working solution 1st.)

---------- Post updated at 05:45 ---------- Previous update was at 01:24 ----------

I believe this script does what was requested with considerable (although not complete) error checking. I use the Korn shell (and tested this script using it), but this should work with any POSIX conforming shell. If you are using a Solaris/SunOS system, use /usr/xpg4/bin/awk or nawk instead of awk :

#!/bin/ksh
# SYNOPSIS
#       rso [-l line_count] [-s sequence_count] fileA fileB
# DESCRIPTION
#       The rso utility "r"andomly "s"elects "sequence_count" (default 10)
#       sequences of "line_count" (default 3) corresponding lines from "fileA"
#       and "fileB" and "o"utputs those lines to files with names:
#               Outfile1-X and Outfile2-X
#       X is a sequence number ranging from 1 through "sequence_count".
#       Leading zeros will be added to the sequence number, if needed, to make
#       all output filenames be the same length.  Lines in Outfile1-* will be
#       from "fileA" and the corresponding lines in Outfile2-* will be the
#       corresponding lines from "fileB".
#
#       Even though the output lines selected will be randomly selected, no
#       input line will be output more than once in a given sequence.
#
#       The number of lines in "fileA" and "fileB" must be the same and the
#       number of lines in the files must be greater than or equal to
#       "line_count".

# Set defaults
ec=0    # Set error code (0 -> no error)
lc=3    # Set default line_count
sc=10   # Set default sequence_count
sn=$(basename $0)       # Save script name for diagnostics

# Process command line options
while getopts l:s: opt
do      case $opt in
        (l)     lc="$OPTARG";;
        (s)     sc="$OPTARG";;
        (?)     ec=1;;
        esac
done
shift $((OPTIND - 1))

# Verify # of operands
if [ $# -ne 2 ]
then    printf "%s: 2 operands are required; %d found\n" "$sn" $#
        ec=1
fi

# If we found errors or the awk script detects an error, print a usage message
if [ $ec -ne 0 ] || ! awk -v lc="$lc" -v sc="$sc" -v sn="$sn" '
# Verify that "line_count" and "sequence_count" are positive integer values
BEGIN { if(lc !~ /^[[:digit:]]+$/ || lc < 1) {
                printf("%s: line_count (%s) must be a positive integer\n",
                        sn, lc)
                ec = 1
                exit ec
        }
        if(sc !~ /^[[:digit:]]+$/ || sc < 1) {
                printf("%s: sequence_count (%s) must be a positive integer\n",
                        sn, sc)
                ec = 2
                exit ec
        }
}
# Save input file names for diagnostics
FNR == 1 {
        fn[++fc] = FILENAME
}
# Accumulate and count input lines from both input files
{       if(FNR == NR)   f1[++c1] = $0
        else            f2[++c2] = $0
}
END {   # If we got here due to an earlier detected error, get out now
        if(ec) exit ec
        # Verify that both files contain the same numbe of lines and that
        # line_count <= # of lines in the files
        if(c1 != c2) {
                printf("%s: lines in %s (%d) must equal lines in %s (%d).\n",
                        sn, fn[1], c1, fn[2], c2)
                exit 3
        }
        if(c1 < lc) {
                printf("%s: line_count(%d) must be <= # of lines in files(%d)\n",
                        sn, lc, c1)
                exit 4
        }
        # Produce output sequences
        for(i = 1; i <= sc; i++) {
                # Set output file names
                of1 = sprintf("Outfile1-%0*d", length(sc), i)
                of2 = sprintf("Outfile2-%0*d", length(sc), i)
                # Set random list of line_count line numbers to output for this
                # output sequence
                for(j = 1; j <= lc; j++) {
                        # Find line_count distinct line numbers
                        while((k = int(rand() * c1) + 1) in list) continue
                        list[k]
                }
                # Print corresponding pairs of lines from the input files into
                # the output files and delete the printed line number from the
                # list of selected random numbers.
                for(j in list) {
                        print f1[j] > of1
                        print f2[j] > of2
                        delete list[j]
                }
                # Close the output files created for this sequence
                close(of1)
                close(of2)
        }
}' "$1" "$2" >&2
then    printf "Usage: %s [-l line_count] [-s sequence_count] fileA fileB\n" \
                "$sn" >&2
        exit 1
fi

Scrutinizer,
I am afraid I do not understand your script. I get some error when trying to run the script.
Am I missing something?
X

awk -v s=3 -v iter=1 '
  NR==FNR { 
    next 
  }
  FNR==1 {
    if(!set) {
      srand;
      n=NR-1
      for(i=1; i<=s; i++) {
        line=0
        while(!line || line in A) line=int(rand*n)+1
        A[line]
      }
      set=1
    }
    close(f)
    f=FILENAME ".out" iter
  } 
  FNR in A {
    print > f
  }
' infile1.txt infile1.txt infile2.txt
awk: cmd. line:7: srand;
awk: cmd. line:7: ^ syntax error
awk: cmd. line:11: while(!line || line in A) line=int(rand*n)+1
awk: cmd. line:11: ^ syntax error
awk: cmd. line:11: while(!line || line in A) line=int(rand*n)+1
awk: cmd. line:11:

---------- Post updated at 05:25 PM ---------- Previous update was at 05:24 PM ----------

Hanson44,
Sheel scripting will be ok too.
Thanks!
X

@xterra: What is your OS and version?

See if the following bash script works. It works fine here. If $RANDOM function causes error because not bash, there are other ways to generate random numbers under Unix.

# usage: temp.sh iterations sample_size seed

iterations=${1:-10} sample_size=${2:-3} # RANDOM=$3

seq_cnt=`cat Infile1 | wc -l`
if [ $sample_size -gt $seq_cnt ]; then echo ss too big!!; exit; fi

iteration=1
while [ $iteration -le $iterations ]; do
  rm -f prev_used_nums.txt unsorted-1.txt unsorted-2.txt
  touch prev_used_nums.txt
  seqs_written=0
  while [ $seqs_written -lt $sample_size ]; do
    let "num_to_use = 1 + ($RANDOM % seq_cnt)"
    grep -q $num_to_use prev_used_nums.txt
    if [ $? -eq 0 ]; then continue; fi # already used $num_to_use
    sed -n "$num_to_use { p; q }" Infile1 >> unsorted-1.txt
    sed -n "$num_to_use { p; q }" Infile2 >> unsorted-2.txt
    echo $num_to_use >> prev_used_nums.txt
    let "seqs_written++"
  done
  sort unsorted-1.txt > OutFile1-$iteration
  sort unsorted-2.txt > OutFile2-$iteration
  let "iteration++"
done

RedHat. I am not in front of my computer now. I will update this info later today.
Thanks!

Xterra,
Have you tried the script I provided in message #6 in this thread?

Don,
I will give it a try as soon as I get to my Linux box.
Thanks!
X

With gawk you need to include the parentheses when calling rand() and srand() . I edited it in my post...

That works very well. However, when I use it in a bash scrip, it keeps resampling the same sequences. This is the bash file I am using:

#!/bin/bash
for i in {1..6}
do
	awk -v s=3 -v iter=${i} '
	  NR==FNR { 
	    next 
	  }
	  FNR==1 {
	    if(!set) {
	      srand();
	      n=NR-1
	      for(i=1; i<=s; i++) {
	        line=0
	        while(!line || line in A) line=int(rand()*n)+1
	        A[line]
	      }
	      set=1
	    }
	    close(f)
	    f=FILENAME ".out" iter
	  } 
	  FNR in A {
	    print > f
	  }
	' infile1 infile1 infile2
done

Am I missing something?
Thanks!

Ah yes, the awk gets executed so quickly that every time it gets called it uses the same epoch time as the seed for srand() . So it will not work that way.. With a sleep 1 before the done statement it works, but that means an n seconds delay. Probably best to do the iteration within awk .

Thank you very much!

If Xterra were to change the line:

        # Produce output sequences

in the script I provided earlier to:

        # Seed the random number generator based on the current time of day
        srand()
        # Produce output sequences

the issue raised here would seem to be solved.

Did you try the bash script I submitted?

I am still testing the solutions you have kindly provided. It was just easioer for me to understand Scrutinizer script. However, I am going over your scripts now.
Thanks!