In a recent research experiment I was handling, I faced this task of managing huge amounts of data to the order of Terabytes and with the help of many people here, I managed to learn quite a lot of things in the whole process. I am sure that many people will keep facing these situations quite often so I am planning on writing a general purpose guide on how to go about handling large amounts of data. Please note the following before reading further:
- This guide will not intended for a specific dataset but one or two tips might be definitely of use to you.
- Some (or most... depends on what level you are) of the tips may apply to the absolute beginner
- If you have some feedback, please don't hesitate to give your suggestions because I realized that if not for the tricks I learnt in this forum, I would've wasted hundreds of man hours.
- I will try my level best to provide with some concrete examples whenever possible but if you find an error somewhere, kindly let me know.
- Lastly, as I said, now all this information is mine, some of it was collected from various sources during my work and some of it was attained with the kind help of people here and some of it was through my experience.
The following is the excerpt of the Table Of Contents that I am planning to have in the guide:
Table of Contents
- Introduction
- Meet your friends - Discover the purpose of each tool
- PuTTY
- Screen
- Bash Scripting
- Awk
- Sed
- Perl
- PHP
- Extremely Useful Commands
- Some Concepts you ought to know
- Know your enemies - Have the constrains in mind
- Downloading and Storing Huge Amounts of Data - Do it carefully or you'll be banned!
- Database or not? - Is all the effort really worth it?
- Parsing the Mammoth - The time has finally come
- Last Minute tips for a Multiprocessor Environment
- Things to Avoid - Bust the common myths
I am pretty much open and I would really love some feedback on adding/deleting some topics to the above list.