# Spacing off when files combined using awk or cat **URL:** \ **Category:** Shell Programming and Scripting\ **Created:** [January 4, 2016, 2:06pm UTC](https://community.unix.com/t/spacing-off-when-files-combined-using-awk-or-cat/357711 "2016-01-04T14:06:53Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![cmccabe](https://community.unix.com/letter_avatar/cmccabe/32/5_5575768a8748004e209b776fc1b2916d.png) [@cmccabe](https://community.unix.com/u/cmccabe)\ **Post date:** [January 4, 2016, 2:06pm UTC](https://community.unix.com/t/spacing-off-when-files-combined-using-awk-or-cat/357711/1 "2016-01-04T14:06:53Z")
I have 133 .txt files in a directory that I am combining into 1 file. The problem is when I use `awk` or `cat` to combine the files I get out put like this: output ```nohighlight 85 138662360 KCNT1 86 138662962 KCNT1 82 138657053 KCNT1 83 138657635 KCNT1 95 138646881 KCNT1 94 138642912 KCNT1 98 138669261 KCNT1 96 138662309 KCNT1 97 138662661 KCNT1 ``` I know the input and output do not match, but the format of the input is always the same, but it seems the spacing is off when I combine the files. If I do a copy and paste (copy file 1 then 2 and paste them into a text file) I get the desired output. example input ```nohighlight name 31 Index Chromosomal Position Gene Inheritance 122 2106725 TSC2 AD 124 2115481 TSC2 AD 121 2105400 TSC2 AD 82 135782221 TSC1 AD 81 135782026 TSC1 AD 126 2138218 TSC2 AD 123 2113107 TSC2 AD 125 2126142 TSC2 AD name2 12 Index Chromosomal Position Gene Inheritance 1 43396568 SLC2A1 AD, AR name3 20 Index Chromosomal Position Gene Inheritance 188 2135240 TSC1 AD 179 2103379 TSC1 AD 191 2137899 TSC2 AD 181 2110617 TSC2 AD 190 2137857 TSC2 AD 189 2137806 TSC2 AD 186 2133798 TSC2 AD 187 2135074 TSC2 AD 180 2105400 TSC2 AD 183 2122822 TSC2 AD 192 2138218 TSC2 AD 185 2125937 TSC2 AD 184 2125788 TSC2 AD 193 2138269 TSC2 AD 182 2112981 TSC2 AD ``` desired output ```nohighlight name 31 Index Chromosomal Position Gene Inheritance 82 135782221 TSC1 AD 81 135782026 TSC1 AD name3 20 Index Chromosomal Position Gene Inheritance 188 2135240 TSC1 AD 179 2103379 TSC1 AD 191 2137899 TSC1 AD ``` ---
**Author:** ![RudiC](https://community.unix.com/letter_avatar/rudic/32/5_5575768a8748004e209b776fc1b2916d.png) [@RudiC](https://community.unix.com/u/RudiC)\ **Post date:** [January 4, 2016, 2:37pm UTC](https://community.unix.com/t/spacing-off-when-files-combined-using-awk-or-cat/357711/2 "2016-01-04T14:37:30Z")
Not the slightest idea what your problem is. Where does the combined output come into play? I can't find any of those lines in your desired output. What does "spacing is off" mean? ---
**Author:** ![cmccabe](https://community.unix.com/letter_avatar/cmccabe/32/5_5575768a8748004e209b776fc1b2916d.png) [@cmccabe](https://community.unix.com/u/cmccabe)\ **Post date:** [January 4, 2016, 2:57pm UTC](https://community.unix.com/t/spacing-off-when-files-combined-using-awk-or-cat/357711/3 "2016-01-04T14:57:32Z")
I hope this helps but I think the problem is when I combine two files there are many new lines that the new file contains that are not there when I do a copy and paste. Thank you :). ```nohighlight awk 'FNR==1{print ""}{print}' *.txt > example.txt ``` desired output (no spaces between new lines) ```nohighlight name1 1 Index Chromosomal Position Gene Inheritance 176 40757228 ADSL AR 51 1.26E+08 ALDH7A1 AR 49 1.26E+08 ALDH7A1 AR 52 1.26E+08 ALDH7A1 AR 50 1.26E+08 ALDH7A1 AR 178 62857727 ARHGEF9 AD, AR 13 1.6E+08 ATP1A2 AD name2 2 Index Chromosomal Position Gene Inheritance 102 52200340 SCN8A AD 134 61991153 CHRNA4 AD 136 62038585 KCNQ2 AD name3 3 Index Chromosomal Position Gene Inheritance 122 2106725 TSC2 AD 124 2115481 TSC2 AD 121 2105400 TSC2 AD name4 4 Index Chromosomal Position Gene Inheritance 4 43394661 SLC2A1 AD, AR 22 1.67E+08 SCN1A AD name5 5 Index Chromosomal Position Gene Inheritance 75 52319081 EFHC1 AD, AR 51 1.67E+08 SCN9A AD 103 1.31E+08 SPTAN1 AD 84 1.47E+08 CNTNAP2 AD 134 6640393 TPP1 AR ``` ---
**Author:** ![Aia](https://community.unix.com/user_avatar/community.unix.com/aia/32/917_2.png) [@Aia](https://community.unix.com/u/Aia)\ **Post date:** [January 4, 2016, 3:20pm UTC](https://community.unix.com/t/spacing-off-when-files-combined-using-awk-or-cat/357711/4 "2016-01-04T15:20:45Z")
```nohighlight awk 'FNR==1{print ""}{print}' *.txt > example.txt ``` You are introducing those undesired lines with the highlighted red part. Perhaps? ```nohighlight cat *.txt > example.txt ``` or ```nohighlight awk '1' *.txt > example.txt ``` ---
**Author:** ![cmccabe](https://community.unix.com/letter_avatar/cmccabe/32/5_5575768a8748004e209b776fc1b2916d.png) [@cmccabe](https://community.unix.com/u/cmccabe)\ **Post date:** [January 4, 2016, 3:33pm UTC](https://community.unix.com/t/spacing-off-when-files-combined-using-awk-or-cat/357711/5 "2016-01-04T15:33:47Z")
Unfortunately both the `cat` and `awk` commands produced the same output with the newlines between the name1, name2,name3. output of command ```nohighlight name1 name2 name3 ``` ```nohighlight name1 name2 name3 ``` Thank you :). ---
**Author:** ![Aia](https://community.unix.com/user_avatar/community.unix.com/aia/32/917_2.png) [@Aia](https://community.unix.com/u/Aia)\ **Post date:** [January 4, 2016, 3:40pm UTC](https://community.unix.com/t/spacing-off-when-files-combined-using-awk-or-cat/357711/6 "2016-01-04T15:40:06Z")
In that case, I have to conclude that some of the source contain those spaces. You can test what files contain empty lines: ```nohighlight grep -n '^$' *.txt ``` It will output `filename:linenumber:` for each line found empty. To eliminate empty lines: ```nohighlight awk 'NF' *.txt > example.txt ``` ---
**Author:** ![cmccabe](https://community.unix.com/letter_avatar/cmccabe/32/5_5575768a8748004e209b776fc1b2916d.png) [@cmccabe](https://community.unix.com/u/cmccabe)\ **Post date:** [January 4, 2016, 4:13pm UTC](https://community.unix.com/t/spacing-off-when-files-combined-using-awk-or-cat/357711/7 "2016-01-04T16:13:16Z")
If the filenames have spaces in them: ```nohighlight annovar analysis CAP NGS-02 2015B-R2.txt Annovar Analysis F41520 Annovar Analysis H52520 ``` Is this ok? Thank you :). The issue seems to be that in each one of the individual text files there is a bunch of whitespace at the end of each file: ```nohighlight name1 name2 name3 name4 name5 ``` I need to remove this white space from all 133 files before `cat` , maybe with a `bash` loop? Thank you 🙂 ---
**Author:** ![Aia](https://community.unix.com/user_avatar/community.unix.com/aia/32/917_2.png) [@Aia](https://community.unix.com/u/Aia)\ **Post date:** [January 4, 2016, 4:46pm UTC](https://community.unix.com/t/spacing-off-when-files-combined-using-awk-or-cat/357711/8 "2016-01-04T16:46:09Z")
The awk command I suggested in post #6 should fix it. ```nohighlight awk 'NF' *.txt > example.txt ``` ---
**Author:** ![RudiC](https://community.unix.com/letter_avatar/rudic/32/5_5575768a8748004e209b776fc1b2916d.png) [@RudiC](https://community.unix.com/u/RudiC)\ **Post date:** [January 4, 2016, 4:48pm UTC](https://community.unix.com/t/spacing-off-when-files-combined-using-awk-or-cat/357711/9 "2016-01-04T16:48:40Z")
Those lines in example.txt are NOT empty, they contain sequences of \s, and, most important, DOS line terminators: \. ---
**Author:** ![cmccabe](https://community.unix.com/letter_avatar/cmccabe/32/5_5575768a8748004e209b776fc1b2916d.png) [@cmccabe](https://community.unix.com/u/cmccabe)\ **Post date:** [January 4, 2016, 4:56pm UTC](https://community.unix.com/t/spacing-off-when-files-combined-using-awk-or-cat/357711/10 "2016-01-04T16:56:46Z")
So would the below work? Thank you 🙂 ```nohighlight for f in /home/cmccabe/Desktop/folder/*.txt ; do bname=`basename $f` pref=${bname%%.txt} sed 's/\r//' | | sed -E 's,\\t|\\r|\\n,,g' $f > /home/cmccabe/Desktop/new/${pref}_unix.txt done ``` ---
**Author:** ![Aia](https://community.unix.com/user_avatar/community.unix.com/aia/32/917_2.png) [@Aia](https://community.unix.com/u/Aia)\ **Post date:** [January 4, 2016, 5:09pm UTC](https://community.unix.com/t/spacing-off-when-files-combined-using-awk-or-cat/357711/11 "2016-01-04T17:09:39Z")
As a rule I do not download any attachment from any forum. If it is true that you have lines containing tabs and spaces, and that the character return is present and you would like to remove those, please, try the following: ```nohighlight perl -ne 's/\r//; print unless /^(\W+)?$/' *.txt > example.txt ``` That would sanitized it for both instances. ---
**Author:** ![cmccabe](https://community.unix.com/letter_avatar/cmccabe/32/5_5575768a8748004e209b776fc1b2916d.png) [@cmccabe](https://community.unix.com/u/cmccabe)\ **Post date:** [January 5, 2016, 3:24pm UTC](https://community.unix.com/t/spacing-off-when-files-combined-using-awk-or-cat/357711/12 "2016-01-05T15:24:47Z")
I apologize for the long post, I am trying to avoid attachments. If I merge file1,2,3 into one example.txt it appears that things do not copy over correctly. Then the `awk` does not result in the desired output. If I manually copy and paste the output is fine, but I have too many files to do that. `cat` doesn't seem to work either and I'm not sure what else to try. Thank you :). file1.txt ```nohighlight name 31 Index Chromosomal Position Gene Inheritance 122 2106725 TSC2 AD 124 2115481 TSC2 AD 121 2105400 TSC2 AD 82 135782221 TSC1 AD 81 135782026 TSC1 AD 126 2138218 TSC2 AD 123 2113107 TSC2 AD 125 2126142 TSC2 AD ``` file2.txt ```nohighlight name2 12 Index Chromosomal Position Gene Inheritance 1 43396568 SLC2A1 AD, AR ``` file3.txt ```nohighlight name3 20 Index Chromosomal Position Gene Inheritance 188 2135240 TSC1 AD 179 2103379 TSC1 AD 191 2137899 TSC2 AD 181 2110617 TSC2 AD 190 2137857 TSC2 AD 189 2137806 TSC2 AD 186 2133798 TSC2 AD 187 2135074 TSC2 AD 180 2105400 TSC2 AD 183 2122822 TSC2 AD 192 2138218 TSC2 AD 185 2125937 TSC2 AD 184 2125788 TSC2 AD 193 2138269 TSC2 AD 182 2112981 TSC2 AD ``` ```nohighlight perl -ne 's/\r//; print unless /^(\W+)?$/' *.txt > example.txt ``` example.txt ```nohighlight name 31 Index Chromosomal Position Gene Inheritance 122 2106725 TSC2 AD 124 2115481 TSC2 AD 121 2105400 TSC2 AD 82 135782221 TSC1 AD 81 135782026 TSC1 AD 126 2138218 TSC2 AD 123 2113107 TSC2 AD 125 2126142 TSC2 ADname2 12 Index Chromosomal Position Gene Inheritance 1 43396568 SLC2A1 AD, ARname3 20 Index Chromosomal Position Gene Inheritance 188 2135240 TSC1 AD 179 2103379 TSC1 AD 191 2137899 TSC2 AD 181 2110617 TSC2 AD 190 2137857 TSC2 AD 189 2137806 TSC2 AD 186 2133798 TSC2 AD 187 2135074 TSC2 AD 180 2105400 TSC2 AD 183 2122822 TSC2 AD 192 2138218 TSC2 AD 185 2125937 TSC2 AD 184 2125788 TSC2 AD 193 2138269 TSC2 AD 182 2112981 TSC2 AD ``` awk command to match specific name and copy header row where the match was found: ```nohighlight awk ' NF == 7 {HD = $0 RS} $3 == "TSC1" {printf "%s%s\n", HD, $0 HD = "" } ' example.txt > TSC1.txt ``` TSC1.txt (current output) ```nohighlight name 31 Index Chromosomal Position Gene Inheritance 82 135782221 TSC1 AD 81 135782026 TSC1 AD 188 2135240 TSC1 AD 179 2103379 TSC1 AD ``` Desired output ```nohighlight name 31 Index Chromosomal Position Gene Inheritance 82 135782221 TSC1 AD 81 135782026 TSC1 AD name3 20 Index Chromosomal Position Gene Inheritance 188 2135240 TSC1 AD 179 2103379 TSC1 AD ``` ---
**Author:** ![Aia](https://community.unix.com/user_avatar/community.unix.com/aia/32/917_2.png) [@Aia](https://community.unix.com/u/Aia)\ **Post date:** [January 5, 2016, 4:26pm UTC](https://community.unix.com/t/spacing-off-when-files-combined-using-awk-or-cat/357711/13 "2016-01-05T16:26:26Z")
It appears that your files `file{1,2}.txt` do not terminate in a new line. Please, give it a try at: ```nohighlight perl -nle 's/\r//; print unless /^(\W+)?$/' file{1..3}.txt > example.txt ``` The `-l` will ensure that a new line is added after each print