Hi Forum.
I hope you can provide some insight as to the unexpected results that I have received running a simple script on different versions of Cygwin.
We currently have an ETL tool (Informatica) that reads data from a database and generates an output file "CLIENT_MAP_daily_trl.JOAATNGB" as follows:
CLIENT_MAP_daily_trl.JOAATNGB:
CLIENT_MAP_daily_dtl_YYYYMMDD.JOAATNGB|20230608|1939|0|N
Running cmd <file CLIENT_MAP_daily_trl.JOAATNGB>
CRM_CLIENT_MAP_daily_trl.JOAATNGB: ASCII text, with CRLF line terminators
Within our informatica tool, we call a post-task to call a script in Cygwin to modify the above file content and replace "YYYYMMDD" with today's date but after the script has been executed, the file format is different between the 2 versions of Cygwin.
cygcheck (cygwin) 1.8.6
Running cmd <file CLIENT_MAP_daily_trl.JOAATNGB>
CRM_CLIENT_MAP_daily_trl.JOAATNGB: ASCII text
cygcheck (cygwin) 2.9.0
Running cmd <file CLIENT_MAP_daily_trl.JOAATNGB>
CRM_CLIENT_MAP_daily_trl.JOAATNGB: ASCII text, with CRLF line terminators
We would like to use the new version of Cygwin but we would expect the results to be the same (File format: ASCII text) as the previous version.
Is there a settings in Cygwin that can be changed to produce the same results?
Thank you.
if the output from the informatica process is a CRLF file then why wouldn't the output of the intermediate command not be the same ( am i missing something) ?
- the post lacks cogent detail
- show the actual commands executed.
- show the format of the file produced (
file <filename> ) before and after the unnamed 'post-task'.
- repeat for both cygwin versions
the command cygcheck ... doesn't mean anything afaics, again, am
I missing something.
if you want the resultant file to be 'unix' format then run dos2unix on it - it won't change the file if its already in that format. (obviously you need to test to confirm)
Hi munkeHoller.
Thank you for the quick reply. It was just an observation that was made between the 2 versions of Cygwin. Today we are using Cygwin (older version - 1.8.6-2) in Production env and generating output files and sending to external clients, we were not expecting that using the newer version of Cygwin (2.9.0-3) - currently in Development env, it would cause a difference in the file format (even though the latter is more correct - reading a CRLF file results in an CRLF file).
We are just concerned if the external clients receive a file format different of what we are sending them today may cause them issues ingesting the files.
Yes, you are also correct that we can run the dos2unix utility to convert all the files to Unix format can be done (except we have many directories and files that would need to get applied). We were wondering if there is any global settings on Cygwin that can be set?
Informatica post-task:
sh /data/informatica/ming/Scripts/edw_Prepare_FAA_File.sh CC-CTCM
Snippet of script edw_Prepare_FAA_File.sh:
...
report_date=$(date +%Y%m%d%H%M%S)
sed -i "s/_YYYYMMDD/_${report_date}/" ${RRAP_EXTRACT_TARGET_DIR}/${src_file_prefix}.JOAATNG${src_file_suffix}
...
Informatica generated file and before running Cygwin script on both versions:
<file CLIENT_MAP_daily_trl.JOAATNGB>
CRM_CLIENT_MAP_daily_trl.JOAATNGB: ASCII text, with CRLF line terminators
Using older version of cygwin - 1.8.6 (updated file after script execution, file gets converted to ASCII text without CRLF)
<file CLIENT_MAP_daily_trl.JOAATNGB>
CRM_CLIENT_MAP_daily_trl.JOAATNGB: ASCII text
Using newer version of cygwin - 2.9.0 (updated file after script execution)
<file CLIENT_MAP_daily_trl.JOAATNGB>
CRM_CLIENT_MAP_daily_trl.JOAATNGB: ASCII text, with CRLF line terminators
Thank you for your time and let me know if you require any additional information.
i don't have access to a cygwin install, however, a quick search through the cygwin site got me to the link below, I suggest you/admins take a read through that and see if it helps.
https://cygwin.com/cygwin-ug-net/using.html#mount-table
as for applying the dos2unix command across many files ....
find <somepath> -type f -exec dos2unix {} \;
sorry, that's not a definitive answer but hope its of some help
Thank you munkeHoller - I will have a read.
1 Like
Just an update - after reading thru the document, we modified the following file /etc/fstab to make our target filesystem "/data" to be text-based instead of binary (default) as follows:
D:/EDW/data /data ntfs text,posix=0 0 0
In order to apply the fstab changes on the filesystem, we did the following command:
mount -a
As a test, I created a dummy file "a",
echo "test" > /data/a
<file /data/a>
/data/a: ASCII text, with CRLF line terminators
Shouldn't the file "/data/a" be just ASCII text since the /data directory is now defined as text-based?
Thank you for your time.
According to man mount text is the opposite.
Try binary instead!
Thank you MadeInGermany - you are correct but still was not able to resolve my issue.
Current Version 2.9.0(0.318/5/3) of Cygwin
Input File defined as "ASCII text, with CRLF line terminators"
After running sed command against input file:
Output File still defined as "ASCII text, with CRLF line terminators"
Previous Version 1.8.6-2(0.280/5/3) of Cygwin
Input File defined as "ASCII text, with CRLF line terminators"
After running sed command against input file:
Output File becomes "ASCII text"
Only way to simulate the previous version with the current version is to run the sed command with a -b option
-b, --binary (open files in binary mode (CR+LFs are not processed specially)) but I actually want the opposite. Desired results is to use new version of Cygwin but would produce results as the previous version.
Current Version 2.9.0(0.318/5/3)
Input File defined as "ASCII text, with CRLF line terminators"
After running sed command against input file:
Output File would become "ASCII text"
If I am unable to find a solution, I will just run the dos2unix utility against the files as suggested by munkeHoller.
Thank you.
Check with
rm /data/a
echo "hello world" > /data/a
file /data/a
that the new file is in Unix text format!
To change mount options, you must umount the filesystem, then mount it again.
Check the mount points with the commands
mount
df
Say the mount point is /data
Then do
cd /
umount /data
mount /data
The umount only works if the filesystem is totally released, therefore the "cd /"
The "mount /data" command takes the not given options and /dev/ path from the corresponding entry in the /etc/fstab file.
Thank you MadeInGermany - I will give it a try.