# Help!!! Shell script to parse data file.

**URL:** <https://community.unix.com/t/help-shell-script-to-parse-data-file/208672>\
**Category:** Shell Programming and Scripting\
**Created:** [August 14, 2008, 10:49am UTC](https://community.unix.com/t/help-shell-script-to-parse-data-file/208672 "2008-08-14T10:49:34Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![yajaykumar](https://community.unix.com/letter_avatar/yajaykumar/32/5_5575768a8748004e209b776fc1b2916d.png) [@yajaykumar](https://community.unix.com/u/yajaykumar)\
**Post date:** [August 14, 2008, 10:49am UTC](https://community.unix.com/t/help-shell-script-to-parse-data-file/208672/1 "2008-08-14T10:49:34Z")

</div>

I am faced with a 😕 tricky problem to parse a data file ( May not be a tricky problem to the scripting guru's ).

Here is what I am faced with. I have a file with multiple rows of data and the rows are not of fixed length. "|" is used as a delimiters for individual columns and each row of data has 5 columns.

I have tried `cat data.out` and then looping through to read the lines, but since there are spaces, I am not able to read entire line of data and I am failing to read individual records.

How can I parse this data file and extract individual records? I have attached the data file to this post and also copied few records from the data file below.(at the end of this posting).

Any help/suggestions is highly appreciated.

Thank you,  
Ajay.

|test.txt

```
                         |2008-7-2.19.19. 0. 

```

162000000|/default/main/administration/STAGING/Configuration/TeamSite/local/config

```
                                  |/opt/iw-home/TeamSite/local/config                                         
                                                                                                              
                                                               |34 |

```

|test.txt

```
   |2008-7-2.19.13. 29. 529000000|/default/main/administration/STAGING/Configuration/TeamSite/local/config    
                                                                                                              
                                                              |/opt/iw-home/TeamSite/local/config             
                                                                                                              
                                                                                           |23                
|

```

|test.txt

```
                               |2008-7-2.19.23. 5. 

```

692000000|/default/main/administration/STAGING/Configuration/TeamSite/local/config

```
                                  |/opt/iw-home/TeamSite/local/config                                         
                                                                                                              
                                                               |11 |

```

|test.txt

```
   |2008-7-3.10.30. 42. 912000000|/default/main/administration/STAGING/Configuration/TeamSite/local/config    
                                                                                                              
                                                              |/opt/iw-home/TeamSite/local/config             
                                                                                                              
                                                                                           |16                
|

```

|New File.txt

```
                                   |2008-7-2.19.19. 0. 

```

162000000|/default/main/administration/STAGING/Configuration/TeamSite/local/config

```
                                  |/opt/iw-home/TeamSite/local/config                                         
                                                                                                              
                                                               |22 |

```

|New File.txt

```
       |2008-7-2.19.13. 29. 

```

529000000|/default/main/administration/STAGING/Configuration/TeamSite/local/config

```
                                  |/opt/iw-home/TeamSite/local/config                                         
                                                                                                              
                                                               |24 |
```

---

<div class="post-metadata">

**Author:** ![sudhamacs](https://community.unix.com/letter_avatar/sudhamacs/32/5_5575768a8748004e209b776fc1b2916d.png) [@sudhamacs](https://community.unix.com/u/sudhamacs)\
**Post date:** [August 14, 2008, 11:34am UTC](https://community.unix.com/t/help-shell-script-to-parse-data-file/208672/2 "2008-08-14T11:34:03Z")

</div>

I don't know whether ur data in the file is really of multiple rows (i mean 5 columns u r looking for), coz when i opened the file, it contains lot of spaces in it.  
What i did is below and got the file to parse and convert it into comma delimited. You can do what ever processing you want.

awk -F'|' 'gsub(/ /,""){print $2 ","$3 "," $4 ","$5 "," $6}' testout1.txt

---

<div class="post-metadata">

**Author:** ![Dhruva](https://community.unix.com/letter_avatar/dhruva/32/5_5575768a8748004e209b776fc1b2916d.png) [@Dhruva](https://community.unix.com/u/Dhruva)\
**Post date:** [September 24, 2008, 5:54am UTC](https://community.unix.com/t/help-shell-script-to-parse-data-file/208672/3 "2008-09-24T05:54:43Z")

</div>

One of many ways to remove spaces:

```nohighlight
 tr -d " " <testout>testout1

```

Above command will remove all spaces from testout and generate new file testout1.  
Go ahead with any kind of processing.

---

<div class="post-metadata">

**Author:** ![bakunin](https://community.unix.com/letter_avatar/bakunin/32/5_5575768a8748004e209b776fc1b2916d.png) [@bakunin](https://community.unix.com/u/bakunin)\
**Post date:** [September 24, 2008, 6:19am UTC](https://community.unix.com/t/help-shell-script-to-parse-data-file/208672/4 "2008-09-24T06:19:56Z")

</div>

> [@dhruva](#):
>
> One of many ways to remove spaces:

To remove every blank might not be desirable because some values might contain blanks. In a first step i would remove the blanks trailing and leading the delimiter characters:

```plaintext
sed 's/ *|/|/g;s/| */|/g' file > file.new

```

I hope this helps.

bakunin