Hello,
first, I apologize if this is trivial or in the wrong place. I'm new to all of this and I don't know how everything works.
I was wondering if I could get some tips on text redirection. What I want to do is to take some characters from a file and input them in another one. More specifically, I have a large "table" file which includes all the numbers I have. These numbers need to be written at specific lines and places in other files. Each of these output files needs data from the "table", but each from a different column; I wanted to automate the process to generate these output files without having to look at the "table" and manually transcribe the numbers every time.
So far what I've come up with is a simple line of code in awk to extract the desired column. The intention is to then copy each value of the column in the appropriate place in a blank output file. The blank file is like a normal one except all the values are XXXXX (or similar string). The idea was to use sed or awk to substitute XXXXX with the appropriate value. To do this, I need to be able to direct data from one file to another, and I'm not sure how to do that.
I probably made this sound a lot more complicated than it is. I welcome all suggestions, even if it's just to say that my method is stupid and there's a better way to do it. I'm quite new to Linux, so please be patient. I know a little (VERY little) about awk and sed, but don't hesitate to point me to better functionalities if you think that they can be more useful.
Thank you for your attention.
This is probably doable with awk, which is particularly good at dealing with tabular data.
Even with a good description there's still a lot of ambiguity left -- what column separators you use if any, etc, etc. If you could post a sample of your input data, the output data you'd want from it, and explain the relation between the two, most of the ambiguity would be gone.
Ok, so here's a sample of what I have:
# Big table
Feature 100-0% 90-10% 80-20% 70-30% 60-40% 50-50% 40-60% 30-70% 20-80% 10-90% 0-100%
cdmass 12.010 10.809 9.6080 8.4070 7.2060 6.0050 4.8040 3.6030 2.4020 1.2010 0.0000
hdmass 1.0080 0.9072 0.8064 0.7056 0.6048 0.5040 0.4032 0.3024 0.2016 0.1008 0.0000
cdradius 0.8780 0.7902 0.7024 0.6146 0.5268 0.4390 0.3512 0.2634 0.1756 0.0878 0.0000
hdradius 0.1350 0.1215 0.1080 0.0945 0.0810 0.0675 0.0540 0.0405 0.0270 0.0135 0.0000
This is the first 5 lines of the master table, as a simple text file edited with vim. Each column has values for that particular percentage combination: the first one for 100-0, the second for 90-10 and so on. There are another 50 or so lines after that with more values of different features at that percentage combination.
My output looks like this:
# Modifications to force field
MASS
cd 12.01 0.878 Carbon
hd 1.008 0.135 Hydrogen
hm 1.008 0.135 Other Hydrogen
In particular, this is the 100-0 combination. The format of the output is fixed: this is the way it has to be in order to be read by the program I'm using. There are another 40 or so lines after that for the remaining features and parameters.
In the third line, starting with cd, there are two values which vary in each percentage combination. The script needs to read the value from the table and write it in the correct column. For example, to prepare the 90-10 output file, I want it to read the third column on the third line (10.809) and write that down in the second column of the third line of the output file, then take the third column on the fifth line (0.7902) and write it on the third column of the third line of the output file. I want this to be repeated for every other parameter. I don't care if I need many lines to specify where everything goes, as long as it's automated. Is there a way to do this?
My first idea was to extract each column of the table in a temporary, intermediate file, then copy across from this to a blank output file, which has XXXXX instead of numbers, by giving a command to substitute the first XXXXX it finds with the first value of the column, then the second with the second and so on. What I don't know what to do is how to redirect the substitution from a file to another file.
I would've been tempted to generate all-new files rather than modify existing ones. It'd be much easier to make sure the formatting is right that way. Fixed-width columns instead of sanely tabbed data in particularly are a bear to edit with anything. Still, though, I think you can do this with awk. 50 lines isn't a lot of data to hold in memory.
Also: as a precaution, I still suggest creating new files instead of editing the old ones, since any mistake in editing the originals could potentially ruin them.
Working on something.
Why is the value 1.0080 repeated in your output when it's not repeated in your input? I would have thought your output would look like
# Modifications to force field
MASS
cd 12.01 0.878 Carbon
hd 1.008 0.135 Hydrogen
hm 0.8780 ??? Other Hydrogen
??? since you did not give enough input data to totally create your output data.
Also, do the numbers need to be truncated to fit in the same column widths here?
I think I have something close to what you want, but explanation of this may completely change the problem.
That specific line is not supposed to be edited: the values of hm are the same across all output files. In the output file, the first column is the "mass" data, while the second is the "radius" data. So the value labeled as cdmass goes in line 3 column 2, the value labeled as cdradius in line 3 column 3, hdmass in line 4 column 2, hdradius in line 4 column 3, to use the examples shown.
What do you want to be repeated for every other parameter, then?
I need to make eleven output files, each corresponding to a different percentage combination. The format of these files has to be as in the post above.
Suppose we start with the 100-0%. The script needs to take the appropriate parameter and write it in the appropriate place. The first one is cdmass, so it needs to read the value in the table on column 2 (100-0), line 3 (cdradius), and substitute it in the output file at column 2, line 3. The next is cdradius, so it reads the value in the table at column 2 (100-0) line 5 (cdradius) and substitute it in the output file at column 3, line 3.
The same applies to all other parameters: so when making the 90-10 output file, it reads cdmass at column 3, line 3, and substitutes at column 2, line 3 of a new output file.
Essentially, I need a way to point at a specific value from a file identifying by column/line coordinates and to tell the program to substitute it at a specific column/line coordinate of another file.
I hope I've explained what I want to do: I understand that it might be a bit complicated to grasp. If there is a better way of doing it, which there almost certainly is, don't hesitate to tell me rather than chasing my probably needlessly complicated route.
I think this may do close to what you want:
# hydrogen.awk
FNR == 1 { FILECOUNT++ }
# Load master data table into memory
NR == FNR { for(N=1; N<=NF; N++)
{
DATA[NR, N]=$N;
# printf("DATA[%d,%d]=%s\n", NR, N, $N)>"/dev/stderr";
}
next }
(FNR <= 2) || (FNR > 4) { print > FILENAME ".new"; next }
# All other lines
{
print $1 " " substr(DATA[FNR,FILECOUNT],1,5) " " substr(DATA[FNR+2, FILECOUNT], 1, 5) " " $4>FILENAME ".new"
}
Use it like
awk -f hydrogen.awk mastertable.dat column1file column2file ...
It will produce output files like column1file.new and column2file.new
You must be sure to give it the files in order, as that's how it knows which one is which column.
Thanks for this, I managed to solve my problem.