hi,
OK. I am writing a bash script, and it is almost working for me.
Problem 1: I currently have stout sent to a file (stout.miRNA.bash.$date_formatted) which I would like to have work inside my loop, but when I move it, it just prints to the screen.
Problem 2: I have a second file (summary.fastqc.$date_formatted), which is basically redirecting the same output file from multiple run output directories.
Is there a way to consolidate these output within the loop so that there is no need for the last cat statement (which is not working) and a third file? So, the script would basically print stout1 summary1; stout2 summary2;... stoutN summaryN.
Problem 3: Is there a way to add in sed/awk to print out specific lines (using search term) of the stout (stout.miRNA.bash.$date_formatted) within the loop, so that I could get the mother load and have edited_stout1 summary1; edited_stout2 summary2;... edited_stoutN summaryN?
HERE IS SOME OF THE CODE, MINUS CALLS TO PROGRAMS INSIDE THE LOOP
#!/bin/sh -f
date_formatted=$(date +%m_%d_%y)
LOCATION="/illumina/runs/Runs/Project_DefaultProject"
#move into each directory in location and do
for i in "$LOCATION"/Sample_*; do
#pass the filename to $infile
infile=`ls $i/*.fastq.gz`
####Run several programs#######
#move into each directory from one program in location and send contents of summary.txt to stout
for i in "$LOCATION"/*_fastqc; do
infile3=`ls $i/summary.txt`
echo "***testing***" $infile3
cat <$infile3 >>summary.fastqc.$date_formatted
done
done >stout.miRNA.bash.$date_formatted
#cat is not concatenating both files, seems to only be stout.miRNA.bash that is being written.
#cat summary.fastqc.$date_formatted stout.miRNA.bash.$date_formatted >new.$date_formatted
Thanks for trying to wrap your head around this one:-)
ok I spent ages on my reply and then lost it when trying to post, so here it is again. bah...
You had asked
Apologies for not being totally clear. I currently have a bash script that is run from a project directory location that contains many Sample directories. The script contains a for loop that is running three third party tools on files in each Sample Directory. I have added a loop within this loop (probably bad form!) to pull a file ('summary.txt') from one program output directory for each sample and print all summary.txt files to a single .txt. Lastly, I direct all stout from the script to a second .txt file.
I would like select specific lines from stout for each sample (or iteration of the loop), and print with each summary.txt, so that I get a report for all Samples for QA purposes. This would need to be in order of Sample processing through the loop to be readable, for Example: sample 1 stout lines, sample1 summary
sample 2 stout lines, sample2 summary.....
This output text file would look something like this (THIS IS OUTPUT FROM CODE, NOT CODE!):
Now, I can put all summary files in a single .txt and pull the whole stout to a second .txt, but have not been able to select stout lines within the loop and iterate with printing the summary.
All suggestions welcome!
Named what? They cannot all be named 'Sample' without some sort of prefix or postfix.
Generating multiple summary.txt's from each 'Sample-XXX'?
A loop inside a loop is not bad form.
What is 'stout' here? A filename, a program name, or a misspelling of 'stdout'? If it's stdout, what things stdout? What's it printing? And which bits of it do you wish to keep?
OK, great. You've given us half of what we need instead of none: You've shown the output you want.
Now show the input you have, please, and tell us how you wish to go from one to the other. Preferably enough mock data that we can actually run your test on it, so we have any idea whether what we write does anything remotely like what you want.
#!/bin/sh -f
date_formatted=$(date +%m_%d_%y)
#outputDir=$1
speciesBuild=$1
speciesAbbrev=$2
LOCATION="/illumina/runs/Runs/140513_H207_0249_AD2B9LACXX/Unaligned_Lane1/Project_DefaultProject"
#move into each directory in location and do
for i in "$LOCATION"/Sample_*; do
#test if i is a file or dir, if so change permissions
test -f "$i" && chmod 755 "$i"
#pass the filename to $infile
infile=`ls $i/*.fastq.gz`
echo "**testing**" $infile
#pass the sample name to $y to append the output directory name
x="$i"
y=${x%.*}
echo ${y}
echo ${y#*/Sample_*}
#count .fastq.gz indexes
#zcat $infile | grep '^@H' | cut -d : -f10 | sort | uniq -c | sort -nr | less > ${y#*/Sample_*}.indices.txt
#run CutAdapt to trim Wafergen adapter
/usr/local/bin/python2.7 /illumina/runs/Runs/cutadapt-1.2.1/bin/cutadapt -a AGATCGGAAGAGCACACGTCT -o $i/${y#*/Sample_*}_cutadapt_AdapterRemoved.fastq.gz $infile
#define cutadapt output as infile2
infile2=`ls $i/*cutadapt_AdapterRemoved.fastq.gz`
echo "**testing**" $infile2
#run QC with FastQC
/illumina/runs/Runs/FastQC/fastqc --outdir=$LOCATION --quiet $infile2
#run sRNAbench w/o adapter trimming
java -jar /illumina/runs/Runs/miRanalyzer/sRNAbenchDB/sRNAbench.jar input=$infile2 output=${y#*/Sample_*}.sRNAbench.$date_formatted dbPath=/illumina/runs/Runs/miRanalyzer/sRNAbenchDB/ species=$speciesBuild microRNA=$speciesAbbrev p=14 noMM=0 alignType=v minRC=2
#move into each fastqc directory in location and send contents of summary.txt to stout
for i in "$LOCATION"/*_fastqc; do
infile3=`ls $i/summary.txt`
echo "***testing***" $infile3
cat <$infile3 >>summary.fastqc.$date_formatted
done
done >stout.miRNA.bash.$date_formatted
The part that is not working is at the end of this code, where I get two outfiles:
summary.fastqc.$date_formatted
PASS Basic Statistics cutadapt_output_DEHP1_29B.fastq
PASS Per base sequence quality cutadapt_output_DEHP1_29B.fastq
PASS Per sequence quality scores cutadapt_output_DEHP1_29B.fastq
WARN Per base sequence content cutadapt_output_DEHP1_29B.fastq
WARN Per base GC content cutadapt_output_DEHP1_29B.fastq
WARN Per sequence GC content cutadapt_output_DEHP1_29B.fastq
PASS Per base N content cutadapt_output_DEHP1_29B.fastq
FAIL Sequence Length Distribution cutadapt_output_DEHP1_29B.fastq
FAIL Sequence Duplication Levels cutadapt_output_DEHP1_29B.fastq
FAIL Overrepresented sequences cutadapt_output_DEHP1_29B.fastq
FAIL Kmer Content cutadapt_output_DEHP1_29B.fastq
And, stout.miRNA.bash.$date_formatted
**testing** /illumina/runs/Runs/140513_H207_0249_AD2B9LACXX/Unaligned_Lane1/Project_DefaultProject/Sample_GRC270_DEHP2_67C/GRC270_DEHP2_67C_CGATGT_L001_R1_001.fastq.gz
/illumina/runs/Runs/140513_H207_0249_AD2B9LACXX/Unaligned_Lane1/Project_DefaultProject/Sample_GRC270_DEHP2_67C
GRC270_DEHP2_67C
cutadapt version 1.2.1
Command line parameters: -a AGATCGGAAGAGCACACGTCT -o /illumina/runs/Runs/140513_H207_0249_AD2B9LACXX/Unaligned_Lane1/Project_DefaultProject/Sample_GRC270_DEHP2_67C/GRC270_DEHP2_67C_cutadapt_AdapterRemoved.fastq.gz /illumina/runs/Runs/140513_H207_0249_AD2B9LACXX/Unaligned_Lane1/Project_DefaultProject/Sample_GRC270_DEHP2_67C/GRC270_DEHP2_67C_CGATGT_L001_R1_001.fastq.gz
Maximum error rate: 10.00%
No. of adapters: 1
Processed reads: 2139267
Processed bases: 109102617 bp (109.1 Mbp)
Trimmed reads: 2075206 (97.0%)
Trimmed bases: 53680380 bp (53.7 Mbp) (49.20% of total)
Too short reads: 0 (0.0% of processed reads)
Too long reads: 0 (0.0% of processed reads)
Total time: 155.78 s
Time per read: 0.07 ms
=== Adapter 1 ===
Adapter 'AGATCGGAAGAGCACACGTCT', length 21, was trimmed 2075206 times.
No. of allowed errors:
0-9 bp: 0; 10-19 bp: 1; 20-21 bp: 2
Lengths of removed sequences
length count expected max. errors
3 13563 33426.0 0
4 13593 8356.5 0
5 19992 2089.1 0
6 143974 522.3 0
7 48435 130.6 0
8 22001 32.6 0
Yes. I need to integrate the two output files,
summary.fastqc.$date_formatted
stout.miRNA.bash.$date_formatted
so each iteration (sample info) gets printed to file, as in my example:
PASS Basic Statistics cutadapt_output_DEHP1_29B.fastq
PASS Per base sequence quality cutadapt_output_DEHP1_29B.fastq
PASS Per sequence quality scores cutadapt_output_DEHP1_29B.fastq
WARN Per base sequence content cutadapt_output_DEHP1_29B.fastq
WARN Per base GC content cutadapt_output_DEHP1_29B.fastq
WARN Per sequence GC content cutadapt_output_DEHP1_29B.fastq
PASS Per base N content cutadapt_output_DEHP1_29B.fastq
FAIL Sequence Length Distribution cutadapt_output_DEHP1_29B.fastq
FAIL Sequence Duplication Levels cutadapt_output_DEHP1_29B.fastq
FAIL Overrepresented sequences cutadapt_output_DEHP1_29B.fastq
FAIL Kmer Content cutadapt_output_DEHP1_29B.fastq
Processed reads: 2139267
Trimmed reads: 2075206 (97.0%)
PASS Basic Statistics cutadapt_output_PB1_82B.fastq
PASS Per base sequence quality cutadapt_output_PB1_82B.fastq
PASS Per sequence quality scores cutadapt_output_PB1_82B.fastq
FAIL Per base sequence content cutadapt_output_PB1_82B.fastq
FAIL Per base GC content cutadapt_output_PB1_82B.fastq
FAIL Per sequence GC content cutadapt_output_PB1_82B.fastq
PASS Per base N content cutadapt_output_PB1_82B.fastq
FAIL Sequence Length Distribution cutadapt_output_PB1_82B.fastq
FAIL Sequence Duplication Levels cutadapt_output_PB1_82B.fastq
FAIL Overrepresented sequences cutadapt_output_PB1_82B.fastq
FAIL Kmer Content cutadapt_output_PB1_82B.fastq
Processed reads: 2159244
Trimmed reads: 254803954 (97.0%)....
---------- Post updated at 01:36 PM ---------- Previous update was at 01:34 PM ----------
this would effectively
print summary.fastqc.$date_formatted
and print / Processed reads/ for that sample
and print / Trimmed reads/ for that sample
repeat for N sample directories....
Let me change your pseudocode a little bit. Do you really mean:
for VAR in "$LOCATION"/*_fastqc/summary.txt
do
cat "$VAR"
print /Processed reads/ for that sample (which comes from where exactly? You have given no input so we cannot tell.)
print /Trimmed reads/ for that sample (which comes from what exactly? You have given no input so we cannot tell.)
done > combined.txt
I am not familiar with VAR. But the stout.miRNA.bash.$date_formatted is where /Processed reads/ and /Trimmed reads/ comes from. It was buried in my previous posting. The first few lines of the file for the first sample are:
**testing** /illumina/runs/Runs/140513_H207_0249_AD2B9LACXX/Unaligned_Lane1/Project_DefaultProject/Sample_GRC270_DEHP2_67C/GRC270_DEHP2_67C_CGATGT_L001_R1_001.fastq.gz
/illumina/runs/Runs/140513_H207_0249_AD2B9LACXX/Unaligned_Lane1/Project_DefaultProject/Sample_GRC270_DEHP2_67C
GRC270_DEHP2_67C
cutadapt version 1.2.1
Command line parameters: -a AGATCGGAAGAGCACACGTCT -o /illumina/runs/Runs/140513_H207_0249_AD2B9LACXX/Unaligned_Lane1/Project_DefaultProject/Sample_GRC270_DEHP2_67C/GRC270_DEHP2_67C_cutadapt_AdapterRemoved.fastq.gz /illumina/runs/Runs/140513_H207_0249_AD2B9LACXX/Unaligned_Lane1/Project_DefaultProject/Sample_GRC270_DEHP2_67C/GRC270_DEHP2_67C_CGATGT_L001_R1_001.fastq.gz
Maximum error rate: 10.00%
No. of adapters: 1
Processed reads: 2139267
Processed bases: 109102617 bp (109.1 Mbp)
Trimmed reads: 2075206 (97.0%)
Trimmed bases: 53680380 bp (53.7 Mbp) (49.20% of total)
Too short reads: 0 (0.0% of processed reads)
Too long reads: 0 (0.0% of processed reads)
Total time: 155.78 s
Time per read: 0.07 ms
=== Adapter 1 ===
Adapter 'AGATCGGAAGAGCACACGTCT', length 21, was trimmed 2075206 times.
No. of allowed errors:
0-9 bp: 0; 10-19 bp: 1; 20-21 bp: 2
Lengths of removed sequences
length count expected max. errors
3 13563 33426.0 0
4 13593 8356.5 0
5 19992 2089.1 0
6 143974 522.3 0
7 48435 130.6 0
8 22001 32.6 0
So, you can see I only need grab a few lines using identifiers, it is just that it needs to iterate so that we get context of sample 1, sample 2, etc. and concatenated with summary.txt for each sample.
Thank you! for you help:-)
No it is not... They only get put there because your program writes them there in the first place. So that's no more use to me than it is to you -- less, actually, because you have some idea where it's being extracted from and I don't.
Okay, so you have two seperate kinds of files -- the summaries and the samples. They come in matched pairs which you want to process together, extracting all of one and some of the other.
"$LOCATION"/Sample_* matches all sample files (not folders).
"$LOCATION"/*_fastqc matches all folders containing summaries, corresponding to the samples above.
What exactly do these paths / filenames have in common? You can't use * to match one and * to match the other when you want pairs, since they both expand to complete lists. Do you find the sample in Sample_abcde, and the summary in abcde_fastqc ?
Ah. The file stout.miRNA.bash.$date_formatted is capturing the stdout from all three programs. I am trying to extract lines from the Cutadapt stdout and the sRNAbench stdout here. The fastQC is writing the summary.txt.
#run CutAdapt to trim Wafergen adapter
/usr/local/bin/python2.7 /illumina/runs/Runs/cutadapt-1.2.1/bin/cutadapt -a AGATCGGAAGAGCACACGTCT -o $i/${y#*/Sample_*}_cutadapt_AdapterRemoved.fastq.gz $infile
#define cutadapt output as infile2
infile2=`ls $i/*cutadapt_AdapterRemoved.fastq.gz`
echo "**testing**" $infile2
#run QC with FastQC
/illumina/runs/Runs/FastQC/fastqc --outdir=$LOCATION --quiet $infile2
#run sRNAbench w/o adapter trimming
java -jar /illumina/runs/Runs/miRanalyzer/sRNAbenchDB/sRNAbench.jar input=$infile2 output=${y#*/Sample_*}.sRNAbench.$date_formatted dbPath=/illumina/runs/Runs/miRanalyzer/sRNAbenchDB/ species=$speciesBuild microRNA=$speciesAbbrev p=14 noMM=0 alignType=v minRC=2
This is actually a folder (containing the actual data being processed).
The Location is the Project (a folder with many sample data folders); The Sample_* is the data folder for each sample; the *_fastqc is the output folder for each sample from one program, fastqc, where each folder contains a summary.txt.
thank you for taking the time. I really appreciate your help.
Great. Which is which? What do they look like individually? What do you want from each individually?
You have to organize it before it gets dumped into the same giant pile, not after, so we need to alter their output, not edit the great giant file, I think.
Great. What files are in it that I need to worry about?
is it possible to organize it real time? like as it is being written to file?
Here is the code for CutAdapt, where the /Processed reads/ and /Trimmed reads/ come from:
#run CutAdapt to trim Wafergen adapter
#/usr/local/bin/python2.7 /illumina/runs/Runs/cutadapt-1.2.1/bin/cutadapt -a AGATCGGAAGAGCACACGTCT -o $i/${y#*/Sample_*}_cutadapt_AdapterRemoved.fastq.gz $infile
Here is the call to sRNAbench where the other bits come from.
I was not specific about this before because I thought it would be the same pattern search, but I need to pull lines containing "No. input reads:", "No. reads in analysis:", "mapped...reads to genomes(s)", and "Detected:" The stdout for that portion looks like this:
START WITH THE PRE-PROCESSING OF THE READS
Reading file: /illumina/runs/Runs/140513_H207_0249_AD2B9LACXX/Unaligned_Lane1/Project_DefaultProject/Sample_GRC270_DEHP2_67C/GRC270_DEHP2_67C_cutadapt_AdapterRemoved.fastq.gz
Result of pre-processing
No. input reads: 2139267
No. cleaned input reads (adapter found and trimmed): 0
No. input reads where the adapter was not found: 2139267
No. length filtered input reads (min. Length): 192482
No. length filtered input reads (max. Length): 0
No. reads in analysis: 1570818
No. unique reads in analysis: 78295
Max. read length in analysis: 51
Filtered reads (low quality or low read count): 375967
Max. read length in input file: 51
FINISHED PRE-PROCESSING
I have already shown you the summary.txt file.
The call to that program is #run QC with FastQC
#/illumina/runs/Runs/FastQC/fastqc --outdir=$LOCATION --quiet $infile2
That's your program's output. Once it's written to the file, it's too late to do anything about it inside the loop. Organize it before you put it into the file.
If you want to pile all three commands together then filter them, do it like this, before it gets to the file:
for ... in ...
do
# All commands get piped through egrep, which then prints to stdout
(
command1
command2
command3
) | egrep 'expression1|expression2|expression3'
# This file gets printed to stdout
cat restoffile
# Everything printed to stdout gets saved to onesingleoutputfile, in order
done > onesingleoutputfile
The ( ) join them together in a subshell, letting you put a pipe on the end to filter the contents of all three.
how would you integrate this with summary.txt in this example?
I am guessing I would put the
for ... in ...
do
# All commands get piped through egrep, which then prints to stdout
(
command1
command2
command3
) | egrep 'expression1|expression2|expression3'
for i in "$LOCATION"/*_fastqc; do
infile3=`ls $i/summary.txt`
echo "***testing***" $infile3
cat <$infile3 >>summary.fastqc.$date_formatted
done
# This file gets printed to stdout ADD summary.txt HERE???
cat restoffile summary.fastqc.$date_formatted
# Everything printed to stdout gets saved to onesingleoutputfile, in order
done > onesingleoutputfile
I am completely unable to give details, due to your complete inability to give details. I don't think that "$LOCATION"/*_fastqc is what you want since that will match all fastqc as I understand it, but cannot say what you do want.
I still don't know which things of your code generate what bits of your output. I still don't know where most of your files are stored... I am not even sure if these fastqc files are inside or outside your 'sample' things or not, which would have helped me answer your last question. In four days and three pages of pulling teeth, I have found one eye, one ear, and one toe of the elephant. I suppose these must be proprietary things you're not allowed to divulge.
I do note that you're blindly continuing to redirect >>summary.fastqc.$date_formatted even though you've told me you don't want to put it there. To not redirect into that file, I suggest not redirecting into that file... It will end up in stdout instead, which will be caught by the outer loop.
If that's not exactly when you wanted it, then print it sooner -- or later. If you wanted only part of it, then filter it then and there. If you print exactly what you want, in the order you want, that's what you'll get in your output.
I think I have given you all the techniques you need.
wow. communication breakdown. Part of the difficulty may be that the input is large, genomic files and the various output of each program is statistical in nature. I did not think this was of interest to the linux forum or consequence for manipulating the stdout and single outfile that I have pasted here. thanks for your help. I will post any solution.
When you have a big problem, a small model of it is sometimes sufficient to find out what you need. But not too small a model. You could also break the problem into halves, to make it simpler to deal with. Right now we either solve all of it or nothing...
You've also been confusing some terms, and I think I sense some confusion on how 'stdout' and redirection in general work, your questions may not mean what you think they do, causing confusion on both sides. Perhaps if you explained what you wanted, instead of the way you wanted to accomplish it.
Perhaps something like:
This is what I've been trying to get. This is sufficient to build a working scale-model of your problem, which we can code around and test.
Once we get that working, you can fill in the pretend bits with real bits and see if it still works for the larger problem.
Or you could keep hiding bits and dodging questions and hope we can guess the rest...