how to add duplicate lines

UNIX for Beginners Q & A UNIX for Dummies Questions & Answers
1

Hi,
I have a file that looks like this:

a_X data
a_Y data
b data
c data
d_X data
d_Y data

I **want** to duplicate the lines without the _X and _Ys. In other words, I want it to look like this

a_X data
a_Y data
b data
b data
c data
c data
d_X data
d_Y data

I have no idea how to go about this. Another detail (or complication) is that there is a header and a footer that I would prefer not to be duplicated. Is there anyway to restrict this operation to just occur between the a line that says "matrix" and ";end;". This last part is really not necessary, but would be helpful.

I have the feeling that something like sed would be good, but I can't figure it out!

Thanks!

Mikey

2

Not sed at all. Use awk. If a line has _ in it, print it once. If not, print it twice.

3

Something like this:

awk '
        /_X/ || /_Y/ { print; next; }
        { print; print; }
' input-file-name

This is simple enough that you're on your own to figure out why it works :slight_smile:

4

Wow, agama, thanks!

ok, this is my first real use of awk, more or less. here goes:

/_X/ || /_Y/
means search for exactly _X or _Y
print that line, then move onto the next line
then, there is no search term, but you print, then print (effectively printing lines without _X or _Y twice--genius!!!)

THANK YOU!!!

Is there any way to start the awk searching after the first appearance of a string, like the word "matrix"?

Best

Mikey

5

You are most welcome. Your analysis of the programme was spot on.

I'm sure some will suggest something less easy to read -- I prefer to err on the side of easy to maintain:

awk '
    /matrix/     { snarf = 1; next }    # assumes you dont want matrix lines
    snarf < 1    { next; }
    /_X/ || /_Y/ { print; next; }
                 { print; print; }
'
6

OK, here is something to horrify the unix programmers. here i am trying to analyze the datafile "fake", i am trying to do a couple of things

take the lines between matrix and end.
remove first line
remove last line

then i used your awk expression to duplicate all remaining line

then wcount up all of the lines and stick that somewhere

then grep and wc to count the occurrences of each of a number of expressions

then stick the line counts in front of the datafiles.

then remove all of the junk files...

here is another question

if i want to write

./myscript INPUTFILE

how do I code that into the script. here i just put the name of the file into the script. this is probably an easy one...i am just new to this! i don't even know the names of what to search for.

Here is the code: I would love any feedback.
And below that is a datafile

code:

awk '/matrix/,/;end;/' INPUTFILE > ZZoutput
sed '$d' ZZoutput > ZZoutfile
sed '1d' ZZoutfile > ZZoutfile1
awk '
/_X/ || /_Y/ { print; next; }
{ print; print; }
' ZZoutfile1 > ZZ_number_of_taxa
grep 'Gg' ZZ_number_of_taxa > ZAGg
wc -l ZAGg > ZQGg
grep 'Hs' ZZ_number_of_taxa > ZAHs
wc -l ZAHs > ZQHs
grep 'Panp' ZZ_number_of_taxa > ZAPanp
wc -l ZAPanp > ZQPanp
grep 'Ptro' ZZ_number_of_taxa > ZAPtro
wc -l ZAPtro > ZQPtro
grep 'Pts' ZZ_number_of_taxa > ZAPts
wc -l ZAPts > ZQPts
grep 'Ptv' ZZ_number_of_taxa > ZAPtv
wc -l ZAPtv > ZQPtv
wc -l ZZ_number_of_taxa > ZZlinecount
cat ZZlinecount ZQ* ZZ_number_of_taxa > dataset
rm ZZ*
rm ZA*
rm ZQ*

datafile:

junk stuff
matrix
Gg447874 CTTGAACATT
Gg447875 CTTGAACATT
Hs287867 CTTGAACATT
Hs287868 CTTGAACATT
Hs287869 CTTGAACATT
Hs287870 CTTGAACATT
Hs287871 CTTGAACATT
Hs287872 CTTGAACATT
;end;

---------- Post updated at 08:54 PM ---------- Previous update was at 08:53 PM ----------

whoops i meant to analyze the datafile
"INPUTFILE" at the beginning.
you probably knew what i meant.

thanks again for any advice

best

mikey

7

Parameters passed from the command line into a script can be referenced in the script using $1, $2, $3.... In your case you just need to change INPUTFILE to $1 in your script.

I prefer to assign input parameters to meaningful variable names so that it's obvious when you use them what they reference. For instance:

inputfile="$1"

And then you can use $inputfile where you have INPUTFILE in the script.

Do note that there cannot be spaces round the equal or you'll get an error.

8

and what i meant by horrify was about my horrible newbie script!!!

9

What is important is that you are trying, and learning. We all had to start somewhere and every once in a while I come across some of my old code and wonder why I ever did it that way!!

10

Thanks!
I just can't believe how much you are helping me solve a major issue!!!
I have another couple of questions.
I needed to rearrange some of the columns in the datafile.
I was able to do this using awk (i am proud of myself!)
But, there are a coiple of places where I am stumped
so for example,
if I have two files

fileA
12 test

and fileB
44 junk

and I want a line in my output that is

12 44

how do I go about that with awk?

I know that i want to do something like this

awk '{print $1}' fileA > outputA
awk '{print $1}' fileB > outputB

but then how do i get the outputA and outputB onto the same line?

you have been amazingly helpful thus far...i hope that you don't mind another couple of questions!

mikey

11

Hello. I hope you don't mind another suggestion. Try to use getline instead:

awk '
    BEGIN {
        while (getline < ARGV[1]) {
            a = $1

            if (!(getline < ARGV[2])) {
                break
            }

            b = $1

            print a " " b
        }

        exit(0)
    }
' fileA fileB
12

well i did this

cat ZZlinecount ZZcharcount > ZZline_char #make one file from two
awk '{printf $1 " "}' ZZline_char > ZZline_char1 #get the first column

and that WORKED

but when I cat this output with anther output, it puts them on the same line (I think because ZZline_char1 does not have an end of line)
I would like ZZline_char1 to have an end of line so that the think that i cat after this is on the next line

i am so close!

thanks for the help!!!

mikey

13

There are a couple of ways that you could go about this. What makes the difference is what is in the two files. A single line in each, or wanting every line processed, is pretty straight forward. If there are some lines that aren't needed it gets a bit more complicated.

Here's an example that assumes you want the first field of all lines mashed together and written to stdout. You should be able to add some extra pattern matching if you don't need all of the lines.

( sed 's/^/file1 /' file1; cat file2 ) | awk '
        /^file1/ {
                save[i++] = $2;         # we added a field, so it is $2
                next;
        }

        {
                printf( "%s %s\n", save[j++], $1 );
                if( j >= i )
                        exit( 0 );              # bail if file2 has more lines
        }
'

I think you can figure it out, but I will point out that the parenthesis round the sed and cat commands are very important. Kshell executes the commands placed the parens in a subprocess and all output from that process is piped into the awk.

If the files are largish, then there are better ways of doing this -- stuffing everything from the first file into an array isn't the best form, but is easier to understand and for a few hundred lines it is better to keep it simple.

There are other posts round this forum that do this kind of thing by putting both filenames on the awk command line, and then test FILENAME within the awk programme to determine what to do. Nothing wrong with that, but I prefer this method as it lets you dynamically supply the filenames without having to hard code them in the awk programme (they could be passed into the script and instead of file1/file2 referenced as $1/$2 or somesuch. A small amount of extra overhead in the sed processes, but a big win in terms of flexibility.

Happy to help!

---------- Post updated at 23:26 ---------- Previous update was at 23:17 ----------

Puts them on the same line because printf() is different than print. The print command automatically prints a newline while printf() does not.

A small change will solve this:

awk '{printf( "%s\n", $1 ) }' ZZline_char > ZZline_char1

And a bit of wisdom passed down by one of the original authors of awk was to always use parens with printf().

@konsolebox -- using getline() was what I referred to as the better way, but not as straight forward. You beat me to the punch :slight_smile:

14

hm
when I use
awk '{printf( "%s\n", $1 ) }' ZZline_char > ZZline_char1

it prints each number onto a separate line. the thing is that i want the first column from two rows from ZZline_char on the same line.
Then, on the next line, I want to cat another file

So ZZline_char looks like this
11 test
22 junk

i want a file that looks like this
11 22

then i have a second file that only has this
100

i want the final output to be
11 22
100

what i was getting was
11 22 100

now i am getting
11
22
100

you are all AMAZINGLY helpful!!!

mikey

---------- Post updated at 11:58 PM ---------- Previous update was at 11:36 PM ----------

OMG I did it!

awk '{printf $1 " "}' ZZline_char > ZZline_char1
echo " " > ZZnewline

then

cat ZZline_char1 ZZnewline ZZthird > dataset

that ZZnewline just put a line in there for me!!!

THANK YOU ALL SO MUCH!!!!

Now I can sleep!!!

All the best

Mikey

---------- Post updated 08-21-10 at 12:16 AM ---------- Previous update was 08-20-10 at 11:58 PM ----------

thanks so much everyone, that solved everything!!!

best

mikey