creating a webcrawler in UNIX

Please read our rules which state:

(3) Refrain from idle chatter that does not contribute to the knowledge base.

and

(6) Do not post classroom or homework problems.