# problem with new line **URL:** \ **Category:** Shell Programming and Scripting\ **Created:** [April 3, 2006, 2:07pm UTC](https://community.unix.com/t/problem-with-new-line/160784 "2006-04-03T14:07:01Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![mskcc](https://community.unix.com/letter_avatar/mskcc/32/5_5575768a8748004e209b776fc1b2916d.png) [@mskcc](https://community.unix.com/u/mskcc)\ **Post date:** [April 3, 2006, 2:07pm UTC](https://community.unix.com/t/problem-with-new-line/160784/1 "2006-04-03T14:07:01Z")
Hi Masters, Here is the code someone posted days ago, but it didn't work well because it did't omit the newline character when $2 conbines. Can someone help me out? awk '{if($1==prev) printf(" /// "$2); else {printf("\n"$1" " $2);prev=$1}}' Basically, this code is trying to remove the redundent line in a two column spreadsheet file based on the field one, but merge field 2 if field one is redundent. Thanks in advance. ---
**Author:** ![Abhishek\_Ghose](https://community.unix.com/letter_avatar/abhishek_ghose/32/5_5575768a8748004e209b776fc1b2916d.png) [@Abhishek\_Ghose](https://community.unix.com/u/Abhishek_Ghose)\ **Post date:** [April 4, 2006, 6:32am UTC](https://community.unix.com/t/problem-with-new-line/160784/2 "2006-04-04T06:32:19Z")
Need the sample text again please. I think your thread was removed in violation of Rule Number 6. ---
**Author:** ![mskcc](https://community.unix.com/letter_avatar/mskcc/32/5_5575768a8748004e209b776fc1b2916d.png) [@mskcc](https://community.unix.com/u/mskcc)\ **Post date:** [April 4, 2006, 9:22am UTC](https://community.unix.com/t/problem-with-new-line/160784/3 "2006-04-04T09:22:32Z")
Thanks! You are right. So I am fraid of violating rules again. But the basic idea is, H82421 RDH11 H82421 DKFZP564M1462 H82435 BAGE4 H82435 MLL3 H82527 C20orf17 H82532 - H82706 ID2 H82801 - H82812 TNRC6B H82872 UACA What I need, H82421 RDH11 /// DKFZP564M1462 H82435 BAGE4 /// MLL3 H82527 C20orf17 H82532 - H82706 ID2 H82801 - H82812 TNRC6B H82872 UACA Your code gave the line break in between $2, H82421 RDH11^M /// DKFZP564M1462 H82435 BAGE4^M /// MLL3 H82527 C20orf17 H82532 - H82706 ID2 H82801 - H82812 TNRC6B H82872 UACA I could use some text editor to remove line breaks, but thought the complete code could be helpful for other people too. In addition, if the redundent lines are not sorted, what code should be? Thanks so much. ---
**Author:** ![Abhishek\_Ghose](https://community.unix.com/letter_avatar/abhishek_ghose/32/5_5575768a8748004e209b776fc1b2916d.png) [@Abhishek\_Ghose](https://community.unix.com/u/Abhishek_Ghose)\ **Post date:** [April 4, 2006, 10:34am UTC](https://community.unix.com/t/problem-with-new-line/160784/4 "2006-04-04T10:34:40Z")
Are you by any chance viewing the output file in DOS/Windows? "^M" normally appears when you do so..as these OSes do not recognize new lines(UNIX) properly ---
**Author:** ![rajeeb\_d](https://community.unix.com/letter_avatar/rajeeb_d/32/5_5575768a8748004e209b776fc1b2916d.png) [@rajeeb\_d](https://community.unix.com/u/rajeeb_d)\ **Post date:** [April 4, 2006, 2:15pm UTC](https://community.unix.com/t/problem-with-new-line/160784/5 "2006-04-04T14:15:47Z")
awk 'BEGIN{x="";i=0} NR == 1 {x=$1;y=$0} NR \> 1 && x == $1 {printf("%s /// %s\n",y,$2);i=NR+1} NR == i {x=$1;y=$0} NR \> i && x != $1 {printf("%s\n",y);x=$1;y=$0} END {printf("%s\n",y);}' ---
**Author:** ![mskcc](https://community.unix.com/letter_avatar/mskcc/32/5_5575768a8748004e209b776fc1b2916d.png) [@mskcc](https://community.unix.com/u/mskcc)\ **Post date:** [April 4, 2006, 7:21pm UTC](https://community.unix.com/t/problem-with-new-line/160784/6 "2006-04-04T19:21:41Z")
Thank you all. Both of your code generated almost same output in terms of line breaks, second code didn't work with multiple redundent lines. First output, AA001897 SPTA1 AA001918 MI-ER1^M /// ER1^M /// KIAA1610^M /// DKFZp781G0451 AA001924 Cep63^M /// FLJ13386 AA001950 KIF13A AA001952 - AA001970 - AA001976 - Second output, AA001897 SPTA1 AA001918 MI-ER1^M /// ER1 AA001918 MI-ER1^M /// KIAA1610 AA001918 MI-ER1^M /// DKFZp781G0451 AA001924 Cep63^M /// FLJ13386 AA001950 KIF13A AA001952 - AA001970 - AA001976 - AA002006 - I just wonder if this is my os problem or not. I am using Mac OS X, and bash shell. ---
**Author:** ![Abhishek\_Ghose](https://community.unix.com/letter_avatar/abhishek_ghose/32/5_5575768a8748004e209b776fc1b2916d.png) [@Abhishek\_Ghose](https://community.unix.com/u/Abhishek_Ghose)\ **Post date:** [April 5, 2006, 2:03am UTC](https://community.unix.com/t/problem-with-new-line/160784/7 "2006-04-05T02:03:21Z")
It probably could be a OS problem. You could check by looking for special charcters by using "cat -vet" on the output file. Regarding your question on what do we do if the input is unsorted wrt to the first column, we sort it first using: sort -t' ' -k 1 testfile.txt and pipe the output to the awk statement you used. ---
**Author:** ![mskcc](https://community.unix.com/letter_avatar/mskcc/32/5_5575768a8748004e209b776fc1b2916d.png) [@mskcc](https://community.unix.com/u/mskcc)\ **Post date:** [April 5, 2006, 9:29am UTC](https://community.unix.com/t/problem-with-new-line/160784/8 "2006-04-05T09:29:08Z")
Thank you so much. That will do. ---
**Author:** ![Abhishek\_Ghose](https://community.unix.com/letter_avatar/abhishek_ghose/32/5_5575768a8748004e209b776fc1b2916d.png) [@Abhishek\_Ghose](https://community.unix.com/u/Abhishek_Ghose)\ **Post date:** [April 6, 2006, 9:00am UTC](https://community.unix.com/t/problem-with-new-line/160784/9 "2006-04-06T09:00:20Z")
Heres to your "^M" error: > **[Data Extraction issue.](https://www.unix.com/shell-programming-and-scripting/25936-data-extraction-issue.html)** > > I have a small problem, I have written a following script, which extracts all the rows from source file which strats with T101 and rights it to another file mydata.dat Script my\_script #!/bin/ksh YMON | The UNIX and Linux Forums