Call awk script

Shell Programming and Scripting
1

The below awk script (loop.awk) is in the cygwin home directory. I do a cd to the directory where the Sources.txt is (where I would like the data output in), but the script does not run:

 echo loop | ./loop.awk

loop.awk

 #!/bin/awk -f
BEGIN {} 
cat Sources.txt | while read a
do
//generate a uniquefilename//
wget ${a} >  //uniquefilename//
done
END {}

I am new to awk and trying to learn. Thank you in advance :).

Basically, the code connects to each website and downloads the source code for each in a separate output file.

2 
$ awk '{print "wget",$0}' Sources.txt | sh

For unique urls you can do like this

$ awk '!a[$0]++{print "wget",$0}' Sources.txt | sh

Or else try just

$ wget -i Sources.txt
3 
 
awk '!a[$0]++{print "wget",$0}' html.txt | sh
--2014-10-07 13:25:34-- https://www.genedx.com/test-catalog/available-tests/edar-gene-sequencing/
Resolving www.genedx.com (www.genedx.com)... 104.130.145.205
Connecting to www.genedx.com (www.genedx.com)|104.130.145.205|:443... connected.
ERROR: The certificate of `www.genedx.com' is not trusted.
ERROR: The certificate of `www.genedx.com' hasn't got a known issuer.

I am not sure how to add this certificate in chrome. I am going to be downloading from multiple sites does a unique certificate need to be used for each or is there a better way? Thanks.

4

Hey that's NOT an awk script, rather just an horrible mix of 1 awk line with shell script:eek:

5

@blastif.fr
Out of curiosity, what shell do you know that uses the following syntax?
//generate a uniquefilename// as a comment or external command and then as a file
//uniquefilename//

6

I'm hoping this is just pseudocode.

7

I don't know such a shell, if you need a comment sign it's just # for both awk and bash.
A awk should just include

  • comments ( # )
  • filter - action couples ( /sample/ { ...} )
8

You might fine some answers to certificate issues with wget on cygwin here

9

add argument --no-check-certificate or else install ca-certificates

10 
 awk '!a[$0]++{print "wget"--no-check-certificate,$0}' html.txt | sh
sh: line 1: wget-1: command not found
sh: line 2: wget-2: command not found
sh: line 3: wget-3: command not found
sh: line 4: wget-4: command not found
sh: line 5: wget-5: command not found
sh: line 6: wget-6: command not found
sh: line 7: wget-7: command not found
sh: line 8: wget-8: command not found
sh: line 9: wget-9: command not found
sh: line 10: wget-10: command not found
sh: line 11: wget-11: command not found

 html.txt
https://www.genedx.com/test-catalog/available-tests/edar-gene-sequencing/
https://www.genedx.com/test-catalog/available-tests/irf6-gene-sequencing/
https://www.genedx.com/test-catalog/available-tests/btd-gene-sequencing/
https://www.genedx.com/test-catalog/available-tests/crebbp-select-exons-sequencing-deldup/
https://www.genedx.com/test-catalog/available-tests/jag1-tier-1/
https://www.genedx.com/test-catalog/available-tests/jag1-tier-2/
https://www.genedx.com/test-catalog/available-tests/mmachc-gene-sequencing/
https://www.genedx.com/test-catalog/available-tests/periodic-fever-syndromes-panel-7-genes/
https://www.genedx.com/test-catalog/available-tests/sox2-gene-sequencing/
https://www.genedx.com/test-catalog/available-tests/elane-gene-sequencing/
https://www.genedx.com/test-catalog/available-tests/whole-exome-sequencing-xomedx/

Thank you.

11 
awk '!a[$0]++{print "wget --no-check-certificate",$0}' html.txt | sh