How to handle new line if it is EOL or present as part of string?

UNIX for Beginners Q & A
1

Hi All

I am facing some issues while processing a file.
The file has \n as End Of Line. However for some records '\n' is also present as part of string. Hence I am not able to identify proper records from file.
File is # delimited.
Example:
2 rows spread across 4 lines.

line1: COL1#COL2#[NEWLINE]
line2:LINE WITH "\n" in it.[NEWLINE]
line3:Another line[NEWLINE]
line4:COL1#COL2#COL3[NEWLINE]

When I am reading above file, I am getting 4 rows. However it should be 2 only.
Now I want to process the file so that output looks like
2 rows spread across 2 lines only

COL1#COL2#LINE WITH "\n" in it.Another line[NEWLINE]
COL1#COL2#COL3[NEWLINE]

Just FYI: COL1,COL2,COL3 are generic and it can be anything. Also, I know in which column new lines may come.

TIA
Nitin

2

If COL1, COL2,... are anything how can you determine that they make one line?

There must be some thought process you have developed to tell when a \n character is not really the end of the line. Please specify it. Like what defines a column? columns==one word?
columns are delimited by quotes?

3

Hi Jim

Thanks for quick reply.
I am exporting data from RDBMS table into flat file with FIELDS TERMINATED BY '#' LINES TERMINATED BY '\n'.
However in the file there are few strings which have \n as the value. So a string ideally should be in a single line however it may spread to multiple lines with \n as value inside string.
Eg:

THIS IS SINGLE LINE STRING

THIS IS
MULTI-LINE
STRING SEPARATED BY
NEWLINE identifier.

Now the problem is when I m reading data in HIVE, I cannot read this string completely because it encounters \n and truncates string at that point. Remaining string is treated as another line.

is there I way I can handle this situation by processing file. I don't want to export data again from RDBMS to file.

Or in other words how do we process file, when string has delimiter as the genuine value.

Thanks
Nitin