Hi All,
I am faced with a problem. I came to know that large scale problems in MATLAB are very slow and with loops they are even slower.
I have the MATLAB script below that does some large scale computation for my school project work. I've been running this code for long now but all it did was to create more problems for me both in space and time complexities. So, I remove the loop and showing you the "non-loop" code below. (Maybe others can use it if they want to solve a similar problem 
)
str = int2str ( j );
string='/data/word_coordinates/';
str=strcat(string,str);
file_coordinate_word = strcat ( str , '.wcor' );
coordinate_vectors=dlmread(file_coordinate_word);
for coordinates=1:numlines,
W(coordinates,1:200)=term_vectors(coordinate_vectors(coordinates),:);
end
D=document_vectors( j , : );
b=sdpvar(1,numlines);
e=ones(1,numlines,1);
Constraints=[0<=b<=1,(b*e')==1];
solvesdp(Constraints,[],norm(D-b*W,2));
fprintf (fd ,'%f ',double(b));
fprintf ( fd , '%c\n' , '' );
clear Constraints
clear D
clear W
clear b
clear coordinate_vectors
clear coordinates
clear e
clear file_coordinate_word
clear j
clear str
clear string
What I used to do was previously using a "for loop" I used to generate values for j, but that was too slow. I then used to compute number of lines in each file, which again takes up some CPU time.
What I have done now is using the command created a file:
file_list.txt
find . -name "*.dat" | xargs wc -l
Generated the file names along with number of lines. (I had to remove some ./ characters though)
So, now my file_list.txt file look like this. I have even sorted the filenames (second column sorted):
1234 1
3456 2
56454 3
1233 4
543 6
In the above list, first column contains the numbers of lines in each file and second column contains the file identifier names (my files are all named as 1.dat, 2.dat, 3.dat 6.dat etc) I removed .dat from file_list.txt file.
PS: The file identifier names are not continuously named like 1 2 3 4 until end but in between one or two files are missing like you see above after 4, 6 comes in.
Now, I want to use file_list.txt to create a "very long" MATLAB script "without loop". So, now what I intend to do is to replace all 'j' variable in my MATLAB script to file identifier values (that is second column in file_list.txt) sequentially line by line (it should not ignore any line else my results will all be meaningless).
Second, the 'numlines' variable in my MATLAB script has to be replaced with the number of lines from file_list.txt (first column).
My intended output file should look something like below, so that if I run the code by pressing RUN button entire script begins to run at once.
str = int2str ( 1 );//PLEASE SEE THE CHANGE HERE
string='/data/word_coordinates/';
str=strcat(string,str);
file_coordinate_word = strcat ( str , '.wcor' );
coordinate_vectors=dlmread(file_coordinate_word);
for coordinates=1:1234,//CHANGE HERE
W(coordinates,1:200)=term_vectors(coordinate_vectors(coordinates),:);
end
D=document_vectors( 1 , : );//CHANGE HERE
b=sdpvar(1,1234);//CHANGE HERE
e=ones(1,1234,1);//CHANGE HERE
Constraints=[0<=b<=1,(b*e')==1];
solvesdp(Constraints,[],norm(D-b*W,2));
fprintf (fd ,'%f ',double(b));
fprintf ( fd , '%c\n' , '' );
clear Constraints
clear D
clear W
clear b
clear coordinate_vectors
clear coordinates
clear e
clear file_coordinate_word
clear j
clear str
clear string
str = int2str ( 2 );//CHANGE HERE
string='/data/word_coordinates/';
str=strcat(string,str);
file_coordinate_word = strcat ( str , '.wcor' );
coordinate_vectors=dlmread(file_coordinate_word);
for coordinates=1:3456,//CHANGE HERE
W(coordinates,1:200)=term_vectors(coordinate_vectors(coordinates),:);
end
D=document_vectors( 2 , : );//CHANGE HERE
b=sdpvar(1,3456);//CHANGE HERE
e=ones(1,3456,1);//CHANGE HERE
Constraints=[0<=b<=1,(b*e')==1];
solvesdp(Constraints,[],norm(D-b*W,2));
fprintf (fd ,'%f ',double(b));
fprintf ( fd , '%c\n' , '' );
clear Constraints
clear D
clear W
clear b
clear coordinate_vectors
clear coordinates
clear e
clear file_coordinate_word
clear j
clear str
clear string
[/CODE]
And this is how it continues in the entire file to create one single MATLAB script file which I can run overnight. Unfortunately, my file list file is too huge to do it manually.
I am open to suggestions too, if anyone can kindly suggest me any better way of doing what I intend to do. I am using Linux.