Has anyone used either bteq or mload for inserting into a Teradata Database via perl. We're running Red Hat Enterprise 5.7. I've gotten the DBD::ODBC drivers installed and running. I installed the Teradata ICU, TeraGSS and ODBC pkgs. We are using perl to insert data across the network into Teradata, but wanted to know if there is a way to speed up the transfer. I am told that Fastload will not work with perl.
For speed, start at the DB and work back: Teradata FastLoad - Wikipedia, the free encyclopedia
FastLoad works with flat files and empty tables, perl can write flat files, and you can make an empty staging table. The big challenge is getting data inside the db; table to table is usually much faster and low overhead. For stremaing data, think mini-batches and be amazed how near real time it can be. You could even do simple inserts for lowest latency on a simple connection, buffering input with another thread, until a buffer high water line is passed, and then switch to mini-batch until a low water line is passed. Writing the next file while the current one is being FastLoad'd and unstaged means you can get a high peak capability with minimal latency (another thread). The higher the load, the bigger the batches and latency get, but economy of scale softens the curve. If the buffering format is FastLoad compatible, moving buffered to file is faster and easy.