Hello,
This is pretty simple, I`m looking for a faster and better method than brute force that I`m doing.
I have a 20GB file looks like
Name1,Var1,Val1
Name1,Var2,Val2
Name2,Var1,Val3
Name2,Var2,Val4
I want 3 files.
Nameindex
1 Name1
2 Name2
...
Varindex
1 Var1
2 Var2
...
datafile
1 1 Value1
1 2 Value2
2 1 Value3
2 2 Value4
What I`m doing right now is
cut -f1 infile | sort -u | cat -n > nameindex
cut -f2 infile | sort -u | cat -n > varindex
Then sort the datafile by each column and join
awk -f sen.awk my20Gfile
where sen.awk is:
BEGIN {
FS=","
fileNames="senNames.txt"
fileVars="senVars.txt"
fileData="senData.txt"
}
{
if(!($1 in names)) { names[$1];print ++namesC, $1 >> fileNames}
if(!($2 in vars)) { vars[$2];print ++varsC, $2 >> fileVars}
print namesC, varsC, $3 >> fileData
}
the resulting files will be: senNames.txt, senVars.txt and senData.txt
The naming of the files could be easily adjusted in the script.
I think the counters should be assigned to the array items as values. For example:
awk -F, '
!($1 in A) {
print A[$1]=++c, $1>"name.idx"
}
!($2 in B) {
print B[$2]=++d, $2>"var.idx"
}
{
print A[$1], B[$2], $3>"file.dat"
}
' file
---
It can be put on a single line, but it is a bit long that way:
awk -F, '!($1 in A){print A[$1]=++c, $1>"name.idx"} !($2 in B){print B[$2]=++d, $2>"var.idx"} {print A[$1], B[$2], $3>"file.dat"}' file