how to create csv file using shell script

Shell Programming and Scripting
1

I have a file in multiple directory which has some records in the following format

File: a/latest.txt , b/latest.txt, c/latest.txt ->

Name=Jhon
Age=27
Gender=M
Street=LA Road
Occupation=Service

I want to generate a csv file from the above file as follows

File: output.csv ->

27,M,Jhon,LA Road,Service

Can you please help me out to write a shell/perl script , as I want to upload this data into a database table.

Thanks.

2

Something like this will do the csv part. You'll need a loop around your files and this code assumes one record per file - easy enough to change if that's not true

#!/usr/bin/perl

open(I, "t");
while (<I>) {
 chomp;
 @s=split /=/;
 $foo{$s[0]}=$s[1];
}
print "$foo{Age}, $foo{Gender}, $foo{Name}, etc.\n";
exit 0;
3

Assuming all files have the same format

#!/bin/bash
for F in */latest.txt
do
    {
        read Name
        read Age
        read Gender
        read Street
        read Occupation
    } < $F
    echo "$Age,$Gender,$Name,$Street,$Occupation" >> outputfile
done
4

Instead of "27,M,..." that will generate "Age=27,Gender=M,..."

A nitpick: There's nothing bash specific in that code. you can just use #!/bin/sh

Regards,
Alister

---------- Post updated at 03:04 PM ---------- Previous update was at 02:43 PM ----------

A tweaked version of frans' solution which outputs the desired format and can handle multiple records in a file (assuming each line in the file is part of a record, without any blank lines or any data not fitting the 5 line record pattern):

#!/bin/sh

IFS=''
for f in */latest.txt; do
    while read -r name; do 
        read -r age
        read -r gender
        read -r street
        read -r occupation
        echo "${age#*=},${gender#*=},${name#*=},${street#*=},${occupation#*=}" >> outputfile
    done < "$f"
done
5

Hi.

The comment by alister is correct. Slightly changing your code:

#!/usr/bin/env bash

# @(#) s1	Demonstrate split name-value pairs, write csv-style.

# Infrastructure details, environment, commands for forum posts. 
set +o nounset
LC_ALL=C ; LANG=C ; export LC_ALL LANG
echo ; echo "Environment: LC_ALL = $LC_ALL, LANG = $LANG"
echo "(Versions displayed with local utility \"version\")"
c=$( ps | grep $$ | awk '{print $NF}' )
version >/dev/null 2>&1 && s=$(_eat $0 $1) || s=""
[ "$c" = "$s" ] && p="$s" || p="$c"
version >/dev/null 2>&1 && version "=o" $p
set -o nounset
echo

FILE=${1-data1}

# replace specimen with cat if not available.
specimen $FILE

echo " Results:"
oldifs="$IFS"
IFS="="
# for F in */latest.txt
for F in $FILE
do
  {
    read junk Name
    read junk Age
    read junk Gender
    read junk Street
    read junk Occupation
  } < $F
  # echo "$Age,$Gender,$Name,$Street,$Occupation" >> outputfile
  echo "$Age,$Gender,$Name,$Street,$Occupation" 
done

exit 0

producing:

% ./s1

Environment: LC_ALL = C, LANG = C
(Versions displayed with local utility "version")
OS, ker|rel, machine: Linux, 2.6.26-2-amd64, x86_64
Distribution        : Debian GNU/Linux 5.0 
GNU bash 3.2.39

Whole: 5 of 5 lines in data1
Name=Jhon
Age=27
Gender=M
Street=LA Road
Occupation=Service

 Results:
27,M,Jhon,LA Road,Service

See man bash for description of IFS.

cheers, drl

6

Another approach:

for f in */latest.txt; do
    cut -d= -f2- "$f" | sed -n 'h;n;N;G;N;N;y/\n/,/;p' >> outputfile
done
7 
 awk -F "=" '/^Name/ {print ""; next} {printf "%s,", $2} ' */latest.txt
8

That will never print out the name associated with each record.

Also, splitting on "=" assumes that there will not be any equals signs in the value. It's probably a safe assumption but not necessarly a given (I have had to deal with data which treated the first occurrence of a character specially and any subsequent appearances literally). If there did happen to be another "=" the value of $2 would be truncated relative to the correct value.

Regards,
Alister

9

Sorry fo rthe mistake, my first idea what something else, using eval :

for F in */latest.txt
do
    for ((i=0; i<5; i++))
    do
        read L
        eval $L
    done < $F
    echo "$Age,$Gender,$Name,$Street,$Occupation" >> outputfile
done
10

Thanks to point that, just recognize the outputs are re-organized. The name part still need be exported to third part.

Here is the ugly code

awk -F "=" '{printf "%s,", $2} '  file* |sed s'/,,/\n/g' |awk -F "," '{print $2,$3,$1,$4,$5}' OFS=","