Well I have something that works. What I ended up doing is loading all of the data to the objects as string. I plan to do the conversion from string to whatever as a second step. That will simplify the loading functions, since they will only have to deal with one type. I think I can do conversion as a class method that will read the string vector in the object and do the conversion to either the int vector or the float vector. In some cases it stays as string.
Here is the code,
Header file,
class col_metadata {
public:
// initialize class members
col_metadata()
: scaleMin(0.0), scaleMax(0.0),
content(""), type(""), scale_type(""),
strScaleMin(""), strScaleMax(""),
header("") { }
float scaleMin, scaleMax;
std::string content, type, scale_type,
strScaleMin, strScaleMax,
header;
};
class column_data {
public:
// initialize class members
column_data()
: header("") { }
std::string header;
std::vector<std::string> inDtaStr;
std::vector<char> inputDataChar;
std::vector<int> inputDataInt;
std::vector<float> inputDataFloat;
};
main file,
#include <string>
#include <vector>
#include <iostream>
#include <fstream>
#include <sstream>
#include "columns.h"
using namespace std;
// global declarations
// declare vector of col_metadata objects
std::vector<col_metadata> metadata;
// declare vector of column_data objects
std::vector<column_data> columns;
// functions
// creates a data object and meta data object for each column
int createColObjs( stringstream &colsRowStream ){
std::string rowCell;
// parse header row into cells, tab delimiter
while(getline(colsRowStream,rowCell,'\t')) {
// create new column data object
col_metadata newColMeta; metadata.push_back(newColMeta);
// create new column data object
column_data newCol; columns.push_back(newCol);
}
// clear the buffer
colsRowStream.clear();
}
// accepts a stringstream and parses the data in to columns
int metadata_row_toCol( stringstream &colsRowStream,
std::vector<col_metadata>& metadata,
std::string col_metadata::* var ){
int i = 0;
std::string rowCell;
// parse header row into cells, tab delimiter
while(getline(colsRowStream,rowCell,'\t')) {
metadata.*var = rowCell;
i++;
}
// clear the buffer
colsRowStream.clear();
}
// accepts a stringstream and parses the data in to columns
int header_row_toCol( stringstream &colsRowStream,
std::vector<column_data>& columns,
std::string column_data::* var ){
int i = 0;
std::string rowCell;
// parse header row into cells, tab delimiter
while(getline(colsRowStream,rowCell,'\t')) {
columns.*var = rowCell;
i++;
}
// clear the buffer
colsRowStream.clear();
}
// accepts a stringstream and parses the data in to columns
int data_row_toCol( stringstream &colsRowStream,
std::vector<column_data>& columns ){
int i = 0;
std::string rowCell;
// parse header row into cells, tab delimiter
while(getline(colsRowStream,rowCell,'\t')) {
columns.inDtaStr.push_back(rowCell);
i++;
}
// clear the buffer
colsRowStream.clear();
}
int main(int argc, char **argv) {
// declarations for args
// file names
int cmdArguments;
std::string colsInFile, splitsInFile, outputFile;
// Parse command line arguments
while ((cmdArguments = getopt (argc, argv, "c:i:o:d:h")) != -1)
switch (cmdArguments)
{
case 'c':
colsInFile = optarg;
break;
case 'i':
splitsInFile = optarg;
break;
case 'o':
outputFile = optarg;
break;
case 'd':
//set delimiter;
break;
case 'h':
std::cout << std::endl; // print help blurb
exit(0);
break;
}
// check args
if(splitsInFile.length() == 0) {
std::cout << " -> no input file was specified" << std::endl;
return(-1);
}
// declarations
int i; // loop iterator
std::string newSplitsLine, rowCell, loadvar;
std::vector<std::string> storeSplitsRows; // storage structure for input rows
stringstream colsRowStream;
// read input file
// create an input stream and open the splitsInFile file
ifstream splitsInput;
splitsInput.open( splitsInFile.c_str() );
// read innput file and store
while(getline(splitsInput,newSplitsLine)) {
// remove '\r' EOL chars
newSplitsLine.erase (remove(newSplitsLine.begin(),newSplitsLine.end(),'\r') , newSplitsLine.end());
// store row data in vector of string
storeSplitsRows.push_back(newSplitsLine);
}
// create objects to store data and meta data
//use first row to create objects for each col
colsRowStream << storeSplitsRows[0];
createColObjs ( colsRowStream );
// load metadata into metadata objects
// declare pointer to class data member, this is used to tell the loading
// function where the data goes in the object
std::string col_metadata::*var;
// load information from metadata row 0 to metadata.content
colsRowStream << storeSplitsRows[0];
//set pointer to metadata.content
var = &col_metadata::content;
// call func to load row 0 data to content
metadata_row_toCol( colsRowStream, metadata, var );
// load information from metadata row 1
colsRowStream << storeSplitsRows[1];
var = &col_metadata::type; // load type row
metadata_row_toCol( colsRowStream, metadata, var );
// load information from metadata row 2
colsRowStream << storeSplitsRows[2];
var = &col_metadata::scale_type; // load scale_type row
metadata_row_toCol( colsRowStream, metadata, var );
// load information from metadata row 3
colsRowStream << storeSplitsRows[3];
var = &col_metadata::strScaleMin; // load strScaleMin row
metadata_row_toCol( colsRowStream, metadata, var );
// load information from metadata row 4
colsRowStream << storeSplitsRows[4];
var = &col_metadata::strScaleMax; // load strScaleMax row
metadata_row_toCol( colsRowStream, metadata, var );
// load information from metadata row 5
colsRowStream << storeSplitsRows[5];
var = &col_metadata::header; // load header row
metadata_row_toCol( colsRowStream, metadata, var );
// print loaded data for checking
std::cout << std::endl;
for (i=0; i<metadata.size(); i++) {
std::cout << "metadata[" << i << "].content " << metadata.content << std::endl;
std::cout << "metadata[" << i << "].type " << metadata.type << std::endl;
std::cout << "metadata[" << i << "].scale_type " << metadata.scale_type << std::endl;
std::cout << "metadata[" << i << "].strScaleMin " << metadata.strScaleMin << std::endl;
std::cout << "metadata[" << i << "].strScaleMax " << metadata.strScaleMax << std::endl;
std::cout << "metadata[" << i << "].header " << metadata.header << std::endl;
std::cout << std::endl;
}
// load data to data objects
// load header row
int metadata_rows, j;
std::string column_data::*varCol;
std::vector<std::string> column_data::*stringVar;
// number of rows of metadata before header
metadata_rows = 5;
// load header row data
colsRowStream << storeSplitsRows[metadata_rows];
varCol = &column_data::header; // load header row
header_row_toCol( colsRowStream, columns, varCol );
for(i=metadata_rows+1; i<storeSplitsRows.size(); i++ ) {
// load first row to stringstream
colsRowStream << storeSplitsRows;
data_row_toCol( colsRowStream, columns );
}
// print loaded data for checking
std::cout << std::endl;
for (i=0; i<columns.size(); i++) {
std::cout << "columns[" << i << "].header " << columns.header << std::endl;
for (j=0; j<columns[0].inDtaStr.size(); j++) {
std::cout << "columns[" << i << "].inDtaStr[" << j << "] " << columns.inDtaStr[j] << std::endl;
}
std::cout << std::endl;
}
}
What I did to load the metadata was to declare a pointer to class data member,
std::string col_metadata::*var;
Then in the function definition, I pass a reference to the vector of objects, and also the pointer to data member,
int metadata_row_toCol( stringstream &colsRowStream,
std::vector<col_metadata>& metadata,
std::string col_metadata::* var );
As far as I can tell, the function just knows there will be a pointer, but doesn't care which data member it points to, or which object. In the loading fucntion, it's just,
metadata.*var = rowCell;
Then I select a row to process,
colsRowStream << storeSplitsRows[0];
point *var to what I want (the content string here, which is in row 0)
var = &col_metadata::content;
and call the loading function with the row, the vector of objects, and the pointer.
metadata_row_toCol( colsRowStream, metadata, var );
This lets me load each row to a different data member by changing the pointer. As far as I can tell, the only limitation is that the pointer is declared as a string, so I couldn't point to an int data member, etc. I could declare different kinds of pointer if I needed to.
For the data rows, I am loading everything as string, so the destination in the objects is always the same. I am using push_back to dynamically size the vectors.
I think that logically these functions should be class methods, but I'm not sure how to go about adding them. I seem to remember issues when referencing thing from inside a class. I also need to add methods for doing the translation from string to int and string to float.
Do you see anything grievously wrong with how I'm going about this? I am hoping to develop some nice reusable code that I can implement in other apps that use similar data. I think I can dispense with storing the entire file on input, since I am storing as string anyway, or I can just clear the storage vector when I am done with it? Is there some point at which I need to become concerned with resource allocation? I have tried the above on an input file of 250 cols and ~16,000 rows. The app uses about 135MB of RAM and takes 1min 8 seconds to complete. This includes writing the data back to an output file in an inefficient method.
LMHmedchem
The loading function will probably be a class method at some point.
awk's designed for chewing up and doing math on huge flatfiles. It's one of those things you avoid for years, finally force yourself to learn, then find yourself using every other day...
If you have to use one, you might as well use a good one...