Delimit file based on character length using awk

Shell Programming and Scripting
1

Hi,

I need help with one problem, I came across recently.

I have one input file which I need to delimit based on character length.

$ cat Input.txt
12345sda231453
asd760kjol62569
sdasw4g76gdf57

And, There is one comma separated file which mentions "start of the field" and "length of the field".

$ cat start_length.csv
1,2
3,3
6,3
9,

Expected output is as follows:

12|345|sda|231453
as|d76|0kj|ol62569
sd|asw|4g7|6gdf57

I have used awk to get the expected result as follows:

$ awk 'BEGIN{OFS="|"}{print substr($0,1,2),substr($0,3,3),substr($0,6,3),substr($0,9)}' Input.txt
12|345|sda|231453
as|d76|0kj|ol62569
sd|asw|4g7|6gdf57

But, the problem here is I have hardcoded "start of the field" and "length of the field" in above awk. We have bigger file containing more than 2 lacs record with more than 200 fields. So, It is not possible to hardcode "start of the field" and "length of the field" for each file.

Is there any way in which I can use start_length.csv file and somehow run it in loop to get desired output.

2

Hello Prathmesh,

Could you please try following and let me know if this helps you.

awk 'FNR==NR{A[++i]=$1;B=$2;next} {for(j=1;j<=i;j++){if(B[j]){C=C?C OFS substr($0,A[j],B[j]):substr($0,A[j],B[j])} else {C=C?C OFS substr($0,A[j]):substr($0,A[j])}};print C;C=""}' FS="," fields OFS="|" main_file

Output will be as follows.

12|345|sda|231453
as|d76|0kj|ol62569
sd|asw|4g7|6gdf57

Thanks,
R. Singh

1 Like
3

Thanks Ravinder. Your code is working fine. But, Can you please explain what it does exactly to understand it better.

4

Hello Prathmesh,

Could you please go through following and let me know if this helps you.

  
awk 'FNR==NR{                                         ####### This condition will be TRUE only when first file is being read, because FNR will be RESET for each file but NR(Number of recoreds) value will be keep on increasing till last file read.
A[++i]=$1;                                            ####### Once above condition is TRUE then I am creating an array named A whose index is a variable named i, ++i means increse value of variable i and keep it's value same as $1's(first field's) value.
B=$2;                                              ####### Creating an array named B whose index is variable i(note but not increasing the value of variable i here, to keep the same indexes for array A and B). keeping it's value to $2's value which is second field's value.
next}                                                 ####### putting next statment here to skip further all the next actions now.
{for(j=1;j<=i;j++){                                   ####### Now starting a for loop to run it till the value of variable i, which we will get variable i's final value when first file will be completly read.
if(B[j]){                                             ####### Here I am making sure array B's value is NOT NULL(because in your example at last line last field is empty so during next step doing substr I have to check this condition now.
C=C?C OFS substr($0,A[j],B[j]):substr($0,A[j],B[j])}  ####### Creating a variable named C whose value will appended each time with it's own last time value along with the current line's substring's value(Here I am using array A and array B to get the substring where obvioslu array A is for the starting position and array B denotes then length of string.
else {                                                ####### If above condition is NOT true then this else will be executed.
C=C?C OFS substr($0,A[j]):substr($0,A[j])}};          ####### create a variable named C and each time append itself with variable C with it's current line's substring's value. Here difference between the previous substring and now substring is I am not giving the till value eg--> substr(LINE, STARTING point, END Point); because we may have NO END point like your 3rd line in fields file.
print C;                                              ####### printing the variable named C.
C=""}'                                                ####### Nullyfing the variable C.
FS="," fields                                         ####### Mentioning the field seprator for fields file as comma here. NOTE it will not be for second file, awk gives us this facility to set mutiple field seprators for different files according to our requirements.
OFS="|" main_file                                     #######  Mentioning the output field seprator as | here and mentioning Input_file(main_file) here too.

Thanks,
R. Singh

1 Like
5

For awk s that can handle empty field separators, try

awk 'FNR == NR {S[NR] = $1; CNT = NR; next} {for (i=2; i<=CNT; i++) $S = "|" $S} 1' FS=, file2 FS="" OFS="" file1 
12|345|sda|231453
as|d76|0kj|ol62569
sd|asw|4g7|6gdf57
1 Like
6

Thanks. I will go through it and let you know in case of any doubt.

---------- Post updated at 08:54 PM ---------- Previous update was at 08:50 PM ----------

Thanks. Can you please explain code.

7

With FS="" , every character is a field of its own. The array S holds the char positions from file2, and file1's fields (= chars) identified by S are prefixed with | .

1 Like