Load mutliple files into Netezza table

Shell Programming and Scripting
1

Hi All,
I get ~1000 files everyday and the current unix script process one file at a time... like it unzips the file, loads into table . This process will repeat how many ever files were present in the directory. I was wondering can we process multiple files (like unzipping and loading X amount of files.. for instance load 5 files parallel) at a time.

If needed I can post the script that I have.

Thanks,
Raja

2

I dont know that program, but if you can load multiple instances of files at the same time, this might help:

As mentioned in [BASH] Script to manage background scripts (running, finished, exit code) I'd so much would like to say, install TUI then run:

Note, that the only reason i choose to use diffrent sub folders, is the keep the visual output of the tui-psm command, and i do think that 40 lines/scripts + statistics fill a full screen output, thus the limit to 40.

Within the folder those 10k files are:

BASE=/tmp/psm/
LINES=40 	# This many scripts will be generated per folder, NOT required, but for visual purpose
COUNT=0

for thisFile in *
do
	DIR=$(( $COUNT / $LINES ))
	target="$BASE/dir_$(( $COUNT / $LINES ))/script-$thisFile"
	[ -d "$(dirname $target)" ] || mkdir -p "$(dirname $target)"
	
	touch "$target"
	cat > "$target" << EOF
echo "tasks you do for each of the file $thisFile"
echo "Replace these two echo commands with your code..."
EOF
	chmod +x "$target"
done

cd "$BASE"
for thisDir in *
do
	cd "$thisDir"
	tui-psm -c *
	echo "Successfully ran: $? / $(ls|wc -l) scripts" && sleep 3
	cd ..
done

If you dont care about visual output/info, change LINES=40 to LINES=0 , and change tui-psm -c * to tui-psm -qc * for quiet mode.
Keep the -c, so echo $? will still tell you how many scripts exited with 0 (success).

Other than this self advertisement, here are some search result to handle the generated script files other than with tui-psm.

SEARCH: run scripts simultaneously

SEARCH: run scripts paralell

SEARCH: run scripts in background

  • Do yourself, there are more than 5 pages of results...

If you want more custom help, provide more custom intel. :wink:

Hope this helps