System call oddity

Shell Programming and Scripting
1

Hi all, I'm trying to use bash to create a basic parser for some text, because basic grep/sed/awk have all my needs covered.

So, I'm creating the chain of grep/sed I need and trying the execute that on the data I gather.

The problem is, it's not working as it does when hardcoded.

e.g.

#!/bin/sh
##a b c
##a b d
##a "b" x
##a p "b"

me=$0
data=`cat $0`
func1="grep 'a b'"
func2="grep ^##"
func3="grep ## | grep b"
echo "$data" | grep 'a b'
#echo "$data" | $func1

So, the last two lines should be the same.

using the 2nd to last line with grep 'a b'

beomagi@VBMagi ~ $ ./test.sh 
##a b c
##a b d
func1="grep 'a b'"
echo "$data" | grep 'a b'

using the variable for grep 'a b'

beomagi@VBMagi ~ $ ./test.sh 
grep: b': No such file or directory

But variables work in the case of the $func2 variable in my test script.

beomagi@VBMagi ~ $ ./test.sh 
##a b c
##a b d
##a "b" x
##a p "b"

So what's the deal? I have text in a variable, and want to filter it using a string of grep/sed/awk - but once there's any spaces or quotes in my filter's variable, it breaks.

Ideas?

2

Hi, welcome to the forum :slight_smile:

[s]In this case, try removing the quotes of the last executed line from "$data" .
Also, cat will produce output already, so there is no need to echo this.

And to make it even simpler try this as your last line:

$func1 $me
```[/s]


hth

EDIT:
Sry looked good at first sight.

---------- Post updated at 04:07 ---------- Previous update was at 03:40 ----------

One should not parse itself, this can cause incredible forks.
Despite said thing, i tried your code.
While i fail to understand why you cant use func1/2/3 from your original script, i found a work around for you.

```text
#!/bin/sh
##a b c
##a b d
##a "b" x
##a p "b"

me="$0"
func1="grep ^##"
str1="a b"
str2="\"b"
$func1 $me | grep "$str2"

Note that the `` changed to $() , that is because the later is easier to manage.
Also note that str2 contains an escaped quote-char for its search pattern.

Hope this helps

3

Thanks.
Yeah I got that far, but I'm hoping to be able to dynamically chain multiple greps/seds/awks etc together.

The original intent was to parse webpages - using grep/sed etc is actually simple for this.

Here's the intent - I automate getting stuff - like parsing through comics and getting them for me.

I'm looking for make it more dynamic.

e.g. here's how I'm getting one of my comics:

If I want to define all data for a comic, I specify a url, and filter rules.
rooturl:
http;//www.mangahere.co/manga/kingdom/
Chaps:
grep "a class" | grep kingdom | sed 's@<.href="@@g' | sed 's@".>@@g' | sed 's-\ --g'
Page:
grep "option value" | sed 's-^.value="--g' | sed 's-"\ .--g'
Image:
grep jpg | grep "render(this)" | sed 's-<img.src="--g' | sed 's-".--g'

Chapters (lists pages) - filter:grep "a class" | grep kingdom | sed 's@<.href="@@g' | sed 's@".>@@g' | sed 's-\ --g'

beomagi@Ganymede ~
$ curl -s http://www.mangahere.co/manga/kingdom/ | grep "a class" | grep kingdom | sed 's@<.*href="@@g' | sed 's@".*>@@g' | sed 's-\ --g'
http://www.mangahere.co/manga/kingdom/v37/c423/
http://www.mangahere.co/manga/kingdom/v37/c422/
http://www.mangahere.co/manga/kingdom/v37/c421/
http://www.mangahere.co/manga/kingdom/v37/c420/
http://www.mangahere.co/manga/kingdom/v37/c419/
http://www.mangahere.co/manga/kingdom/v37/c418/
http://www.mangahere.co/manga/kingdom/v37/c417/
http://www.mangahere.co/manga/kingdom/v37/c416/
http://www.mangahere.co/manga/kingdom/v37/c415/
http://www.mangahere.co/manga/kingdom/v37/c414/

Pages (has link to image)- filter:grep "option value" | sed 's-^.value="--g' | sed 's-"\ .--g'

beomagi@Ganymede ~
$ curl -s http://www.mangahere.co/manga/kingdom/v37/c422/ | grep "option value" | sed 's-^.*value="--g' | sed 's-"\ .*--g'
http://www.mangahere.co/manga/kingdom/v37/c422/
http://www.mangahere.co/manga/kingdom/v37/c422/2.html
http://www.mangahere.co/manga/kingdom/v37/c422/3.html
http://www.mangahere.co/manga/kingdom/v37/c422/4.html
http://www.mangahere.co/manga/kingdom/v37/c422/5.html
http://www.mangahere.co/manga/kingdom/v37/c422/6.html
http://www.mangahere.co/manga/kingdom/v37/c422/7.html
http://www.mangahere.co/manga/kingdom/v37/c422/8.html
http://www.mangahere.co/manga/kingdom/v37/c422/9.html

And lastly from here
Image filter:grep jpg | grep "render(this)" | sed 's-<img.src="--g' | sed 's-".--g'

beomagi@Ganymede ~
$ curl -s http://www.mangahere.co/manga/kingdom/v37/c422/4.html | grep jpg | grep "render(this)" | sed 's-<img.*src="--g' | sed 's-".*--g'
        http://z.mhcdn.net/store/manga/8198/37-422.0/compressed/g004.jpg?v=11425389828

So the idea is to create templates for a script to follow through for various comics. So If I want to add comics, I just have to come up with a url to initially follow, and rules to define pages. The grep/sed etc would need to be dynamic. I've done it in other languages, but bash is kinda convenient, and more than anything now it's irking me that I can't figure out why I can't make a system call echoing a variable and piping to grep.

4

Ok, here's a simpler example.

very simple script

#!/bin/sh
##apple soda
##banana split
##peach cobbler
##apple pie
##apple computer
##zuchinni casserole
##mango apple smoothie
##adam's apple
##app installer
##whatsapp
##application manager

export me=$0
datacmd="cat $me | grep ^##"
echo $datacmd
data=`$datacmd`

output:

beomagi@VBMagi ~ $ ./test.sh 
cat ./test.sh | grep ^##
cat: |: No such file or directory
cat: grep: No such file or directory
cat: ^##: No such file or directory

The first line of the output
"cat ./test.sh | grep ^##" is from the line echo $datacmd

The next 3 lines of errors all come from this : data=`$datacmd`

Yet if I directly drop this into the prompt:

beomagi@VBMagi ~ $ cat ./test.sh | grep ^##
##apple soda
##banana split
##peach cobbler
##apple pie
##apple computer
##zuchinni casserole
##mango apple smoothie
##adam's apple
##app installer
##whatsapp
##application manager

It works.

i.e. cat ./test.sh | grep ^## works,
BUT if it's a in a string type variable and I execute it as a system call it barfs.

Why?:confused:

---------- Post updated at 09:33 PM ---------- Previous update was at 09:14 PM ----------

I'm balder now.

I think that helped.

I found out that there's "eval" and I can even call bash if i want to execute my string.

backticks just don't seem to cut it for complex calls.

#!/bin/sh
##apple soda
##banana split
##peach cobbler
##apple pie
##apple computer
##zuchinni casserole
##mango apple smoothie
##adam's apple
##app installer
##whatsapp
##application manager

datacmd="cat $me | grep ^##"
echo $datacmd
data=`eval "$datacmd"`
echo "$data"

gives:

beomagi@VBMagi ~ $ ./test.sh 
cat ./test.sh | grep ^##
##apple soda
##banana split
##peach cobbler
##apple pie
##apple computer
##zuchinni casserole
##mango apple smoothie
##adam's apple
##app installer
##whatsapp
##application manager

Yaaayyy....