# Split file using awk

**URL:** <https://community.unix.com/t/split-file-using-awk/149498>\
**Category:** Shell Programming and Scripting\
**Created:** [July 20, 2004, 1:43pm UTC](https://community.unix.com/t/split-file-using-awk/149498 "2004-07-20T13:43:51Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![pvar](https://community.unix.com/letter_avatar/pvar/32/6_449c533f586939b6e5f246632a46421f.png) [@pvar](https://community.unix.com/u/pvar)\
**Post date:** [July 20, 2004, 1:43pm UTC](https://community.unix.com/t/split-file-using-awk/149498/1 "2004-07-20T13:43:51Z")

</div>

I am trying to read a file and split the file into multiple files. I need to create new files with different set of lines from the original file. ie, the first output file may contain 10 lines and the second 100 lines and so on. The criteria is to get the lines between two lines starting with some characters(variable)

eg. A sample input file

DOC100  
aaaaaa  
bbbbbb  
cccccccc  
END  
DOC200  
ddddddd  
eeeeeee  
END

I need to send the lines between DOC100 and END to out.1  
and lines between DOC200 and the next END to out.2

I thought I could do this using awk and I tried the following command but it didn't work.

k=grep DOC $filename|cut -c4-6  
awk -v cn=$k '/DOC$cn/,/END/ { print $0 }' $filename \>$outfile

Can anyone help me on this? Thanks

---

<div class="post-metadata">

**Author:** ![zazzybob](https://community.unix.com/user_avatar/community.unix.com/zazzybob/32/112_2.png) [@zazzybob](https://community.unix.com/u/zazzybob)\
**Post date:** [July 20, 2004, 4:31pm UTC](https://community.unix.com/t/split-file-using-awk/149498/2 "2004-07-20T16:31:34Z")

</div>

A script such as this will name the files out.100, out.200 where the number corresponds to whatever follows "DOC"

```nohighlight
#!/bin/sh

while read line
do
   if echo $line | grep DOC; then
       line_no=`echo $line | sed 's/DOC\([0-9]*\)/\1/g'`
   elif echo $line | grep END; then
       : #ignore
   else
       echo $line >> out.$line_no
   fi
done < file.txt

```

Where file.txt contains your data.

This is a quick script and there's room for improvement, but you get the idea.

Cheers  
ZB

---

<div class="post-metadata">

**Author:** ![Ygor](https://community.unix.com/user_avatar/community.unix.com/ygor/32/104_2.png) [@Ygor](https://community.unix.com/u/Ygor)\
**Post date:** [July 21, 2004, 4:29am UTC](https://community.unix.com/t/split-file-using-awk/149498/3 "2004-07-21T04:29:57Z")

</div>

Using awk..

```nohighlight
awk '/^DOC/{f=$1} f{print > f} /^END/{close f}' infile

```

---

<div class="post-metadata">

**Author:** ![pvar](https://community.unix.com/letter_avatar/pvar/32/6_449c533f586939b6e5f246632a46421f.png) [@pvar](https://community.unix.com/u/pvar)\
**Post date:** [July 21, 2004, 8:58am UTC](https://community.unix.com/t/split-file-using-awk/149498/4 "2004-07-21T08:58:43Z")

</div>

Thanks zazzybob and Ygor for your help. But I see some problem with each of your scripts.

zazzybob's script - First of all I have a fixed length data file that I need to split. The script creates seperate files with the intended data but it ignores the spaces in the fields, means it concatenates the fields leaving out the spaces in them.

Ygor's script - Your script creates only one file named DOC with the data between the last DOC/END. But it puts the data like what it is in the input file, that is a good sign.

---

<div class="post-metadata">

**Author:** ![zazzybob](https://community.unix.com/user_avatar/community.unix.com/zazzybob/32/112_2.png) [@zazzybob](https://community.unix.com/u/zazzybob)\
**Post date:** [July 21, 2004, 9:03am UTC](https://community.unix.com/t/split-file-using-awk/149498/5 "2004-07-21T09:03:03Z")

</div>

To solve the problem with the script I posted, just put quotes around "$line", thusly

```nohighlight
echo "$line" >> out.$line_no

```

That will preserve the whitespace

Cheers  
ZB

---

<div class="post-metadata">

**Author:** ![Ygor](https://community.unix.com/user_avatar/community.unix.com/ygor/32/104_2.png) [@Ygor](https://community.unix.com/u/Ygor)\
**Post date:** [July 21, 2004, 9:07am UTC](https://community.unix.com/t/split-file-using-awk/149498/6 "2004-07-21T09:07:01Z")

</div>

I didn't read the bit about filenames like "doc.1" before. Try this instead...

```nohighlight
awk '/^DOC/{f="doc."++d} f{print > f} /^END/{close f; f=""}' infile

```

$ head -1000000 doc.\*  
==\> doc.1 \<==  
DOC100  
aaaaaa  
bbbbbb  
cccccccc  
END

==\> doc.2 \<==  
DOC200  
ddddddd  
eeeeeee  
END

---

<div class="post-metadata">

**Author:** ![pvar](https://community.unix.com/letter_avatar/pvar/32/6_449c533f586939b6e5f246632a46421f.png) [@pvar](https://community.unix.com/u/pvar)\
**Post date:** [July 21, 2004, 9:16am UTC](https://community.unix.com/t/split-file-using-awk/149498/7 "2004-07-21T09:16:50Z")

</div>

zazzybob's script works fine now.

Ygor's script still have an issue. It creates seperate output files now but it puts lines starting with DOC & END also in the output files. I just need the data between those lines.

Thanks agin for the help.

---

<div class="post-metadata">

**Author:** ![Ygor](https://community.unix.com/user_avatar/community.unix.com/ygor/32/104_2.png) [@Ygor](https://community.unix.com/u/Ygor)\
**Post date:** [July 21, 2004, 9:30am UTC](https://community.unix.com/t/split-file-using-awk/149498/8 "2004-07-21T09:30:24Z")

</div>

Moving things around...

```nohighlight
awk '/^END/{close f; f=""} f{print > f} /^DOC/{f="doc."++d}' infile

```

$ head -1000000 doc.\*  
==\> doc.1 \<==  
aaaaaa  
bbbbbb  
cccccccc

==\> doc.2 \<==  
ddddddd  
eeeeeee

---

<div class="post-metadata">

**Author:** ![pvar](https://community.unix.com/letter_avatar/pvar/32/6_449c533f586939b6e5f246632a46421f.png) [@pvar](https://community.unix.com/u/pvar)\
**Post date:** [July 21, 2004, 9:41am UTC](https://community.unix.com/t/split-file-using-awk/149498/9 "2004-07-21T09:41:00Z")

</div>

That worked. Thanks.