Extract information into large variable

WolwX · October 27, 2011, 4:12am

Hello people

That's here my first message to your forum, so I guess it would be fine ^^

I have a request about a code I want to use.

Actually, my system use a large variable, including much informations but those informations can change by more and I want to extract one of thoses informations.
For exemple I want to extract the information between

<add key="LevelName" value="

and

" type="System.String,mscorlib" />

and use it as a new variable

levelname=$(codeforextractinformationhere)

Here is an exemple of my large variable =>

<?xml version="1.0" encoding="utf-16" standalone="yes"?> <values> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="Log" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="Admin" value="true" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="LevelName" value="blabla" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="Fastdisk1" value="" type="System.String,mscorlib" /> <add key="Fastdisk2" value="" type="System.String,mscorlib" /> <add key="Fastdisk3" value="" type="System.String,mscorlib" /> <add key="backupinterval" value="300" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="true" type="System.String,mscorlib" /> <add key="backupeverything" value="false" type="System.String,mscorlib" /></values>

So if anyone have a solution for me ?

Thankssss

Skrynesaver · October 27, 2011, 4:51am

$ perl -e '
@xml=<>;
chomp @xml;
$xml=join "",@xml;
@levels=$xml=~m/<add\skey="LevelName"\svalue="#so everything after this should be captured
(.+?)# but in a non-greedy match
"\stype="System.String,mscorlib"\s\/># and only if it is followed by this text
/gx;#the x modifier allows comments in the regex and the g modifier matches all instances
print "We found the following levels in the file:\n";
for $level(@levels){
   print "\t$level\n";
}' ~/tmp.dat
We found the following levels in the file:
        blabla

WolwX · October 27, 2011, 4:54am

Skrynesaver:

 perl -e ' 
@xml=<>;
chomp @xml;
$xml=join "",@xml;
@levels=$xml=~m/<add key="LevelName" value="(.+?)" type="System.String,mscorlib" \/>/;
print "We found the following levels in the file:\n";
for $level(@levels){
   print "\t$level\n";
}' ~/tmp.dat
We found the following levels in the file:
   blabla

That's look like nice, but I wish to use it into a bash shell script

"#!/bin/bash", this will be a variable used by another command.

How I can include your code ?

Skrynesaver · October 27, 2011, 5:25am

This should work, there should however be a paragraph sized comment beside it explaining what it does for future maintainers or else write the earlier script to a file including comments and call that script from your shell script.

#!/bin/bash
levelName=$(perl -e '@x=<>;$x=join "",@x;($l)=$x=~m/<add key="LevelName" value="(.+?)" type="System.String,mscorlib" \/>/;print "$l\n";' ~/tmp.dat)
myNextCommand $levelName

jayan_jay · October 27, 2011, 5:40am

$ echo `cat infile` | tr '=' '\n' | nawk -F'"' '/type/ {print $2}'| grep '.'
true
blabla
300
true
false

WolwX · October 27, 2011, 5:41am

Ok perfect ^^

I will try to use it, but one last thing, can you tell me how I can use it by calling another variable or into a txt file ?

I mean, this variable into wich I cut this information is for exemple ${bigvariable}, and so how I include that to extract my new variable LevelName from it ?

jayan_jay · October 27, 2011, 6:04am

redirect the above ouput to a flat file and assign it to a variable ..

$ echo `cat infile` | tr '=' '\n' | nawk -F'"' '/type/ {print $2}'| grep '.' > tempfile
$ LevelName=`cat tempfile` 
$ echo $LevelName
true blabla 300 true false

WolwX · October 27, 2011, 6:22am

Ok jayan_jay, thanks
But I wanted to extract only one of those information, one specified by an very accurate delimiter for that I specified something who can only work with LevelName
You solution permit to extract all informations, by removing all other extra characters.
But you method is more friendly for me

Skrynesaver thanks too
You specified me a perfect command, but now I must include it into my script, and understand all your command since I'm not familiar with those type of perl input ^^'

Actually I used cut command, but that's not as accurate as I want, and my big variable change sometimes, so the number of delimiter I use may change and my cut command will be broked.

Here it's the code I'm using actually =>

echo ${bigvariable} >> /home/user/tempfile
cat /home/user/tempfile
levelName=$(cut -d '"' -f172 /home/user/tempfile)

So I will try to include Skrynesave's command into this portion of code =>

echo ${bigvariable} >> /home/user/tempfile
cat /home/user/tempfile
levelName=$(perl -e '@x=<>;$x=join "",@x;($l)=$x=~m/<add key="LevelName" value="(.+?)" type="System.String,mscorlib" \/>/;print "$l\n";' /home/user/tempfile)

By this way that's will work ?

jayan_jay, if you have something using awk, cut, grep or anything like that, who can replace the perl command, I will prefer that, since I will be probably more familiar to this method than the perl one.

CarloM · October 27, 2011, 6:35am

levelname=$(echo "${bigvariable}" | sed 's:.*<add key="LevelName" value="\([^"]*\)" type="System.String,mscorlib" />.*:\1:g')

WolwX · October 27, 2011, 6:42am

Oh
That's look like very fine CarloM
That's command will work with using flat file I presume ?
And the sed function remove all informations into the ${bigvariable} except the information I want that's right ?

CarloM · October 27, 2011, 6:47am

With the source (XML) in a flat file, or the output?

WolwX · October 27, 2011, 3:09pm

Ok I'm back and I did some test

So here it's the script I used to test that =>

bigvariable="<?xml version="1.0" encoding="utf-16" standalone="yes"?> <values> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="Log" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="Admin" value="true" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="LevelName" value="blabla" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="Fastdisk1" value="" type="System.String,mscorlib" /> <add key="Fastdisk2" value="" type="System.String,mscorlib" /> <add key="Fastdisk3" value="" type="System.String,mscorlib" /> <add key="backupinterval" value="300" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="true" type="System.String,mscorlib" /> <add key="backupeverything" value="false" type="System.String,mscorlib" /></values>"

levelname=$(echo "${bigvariable}" | sed 's:.*<add key="LevelName" value="\([^"]*\)" type="System.String,mscorlib" />.*:\1:g')
echo ${levelname} >> /home/+LOGS+/bla-test
cat /home/+LOGS+/bla-test

But actually there is something wrong, the output variable result by the full "bigvariable" :x

I think I have problem with the " used into bigvariable and I must to escape them or to do something ...

system · October 27, 2011, 3:32pm

perl -ne 'while(/value="(.*?)"/gi){print"$1\n" if(length$1>1)}' file.in > file.out

WolwX · October 27, 2011, 4:00pm

I'm sorry, I'm noob with that, can you explain me how include your command ?

bigvariable="<?xml version="1.0" encoding="utf-16" standalone="yes"?> <values> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="Log" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="Admin" value="true" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="LevelName" value="blabla" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="Fastdisk1" value="" type="System.String,mscorlib" /> <add key="Fastdisk2" value="" type="System.String,mscorlib" /> <add key="Fastdisk3" value="" type="System.String,mscorlib" /> <add key="backupinterval" value="300" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="true" type="System.String,mscorlib" /> <add key="backupeverything" value="false" type="System.String,mscorlib" /></values>"

levelname=$(echo "${bigvariable}" | sed 's:.*<add key="LevelName" value="\([^"]*\)" type="System.String,mscorlib" />.*:\1:g')
echo ${levelname} >> /home/+LOGS+/bla-test
cat /home/+LOGS+/bla-test

CarloM · October 27, 2011, 4:56pm

It worked when I tested it (on bash, iirc)! Is that the same input?

Also, what version of sed do you have?

EDIT:
For tip78's code, you should just be able to replace the sed command with the perl command:

levelname=$(echo "${bigvariable}" | perl -ne 'while(/value="(.*?)"/gi){print"$1\n" if(length$1>1)}')

I think, anyway - I don't use perl very often. Or use a tempfile as you did earlier.

WolwX · October 27, 2011, 5:10pm

carlom:

It worked when I tested it (on bash, iirc)! Is that the same input?

Also, what version of sed do you have?

EDIT:
For tip78's code, you should just be able to replace the sed command with the perl command:
levelname=$(echo "${bigvariable}" | perl -ne 'while(/value="(.*?)"/gi){print"$1\n" if(length$1>1)}')
I think, anyway - I don't use perl very often. Or use a tempfile as you did earlier.

Sed version 4.2.1

I tested this code =>

bigvariable="<?xml version="1.0" encoding="utf-16" standalone="yes"?> <values> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="Log" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="Admin" value="true" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="LevelName" value="blabla" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="Fastdisk1" value="" type="System.String,mscorlib" /> <add key="Fastdisk2" value="" type="System.String,mscorlib" /> <add key="Fastdisk3" value="" type="System.String,mscorlib" /> <add key="backupinterval" value="300" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="" type="System.String,mscorlib" /> <add key="" value="true" type="System.String,mscorlib" /> <add key="backupeverything" value="false" type="System.String,mscorlib" /></values>"

levelname=$(echo "${bigvariable}" | sed 's:.*<add key="LevelName" value="\([^"]*\)" type="System.String,mscorlib" />.*:\1:g')
echo ${levelname} >> /home/+LOGS+/bla-test
cat /home/+LOGS+/bla-test

That's result by the "bigvariable" without quotes =>

<?xml version=1.0 encoding=utf-16 standalone=yes?> <values> <add key= value= type=System.String,mscorlib /> <add key= value= type=System.String,mscorlib /> <add key= value= type=System.String,mscorlib /> <add key= value= type=System.String,mscorlib /> <add key= value= type=System.String,mscorlib /> <add key= value= type=System.String,mscorlib /> <add key=Log value= type=System.String,mscorlib /> <add key= value= type=System.String,mscorlib /> <add key= value= type=System.String,mscorlib /> <add key= value= type=System.String,mscorlib /> <add key= value= type=System.String,mscorlib /> <add key=Admin value=true type=System.String,mscorlib /> <add key= value= type=System.String,mscorlib /> <add key= value= type=System.String,mscorlib /> <add key= value= type=System.String,mscorlib /> <add key= value= type=System.String,mscorlib /> <add key= value= type=System.String,mscorlib /> <add key= value= type=System.String,mscorlib /> <add key= value= type=System.String,mscorlib /> <add key= value= type=System.String,mscorlib /> <add key= value= type=System.String,mscorlib /> <add key= value= type=System.String,mscorlib /> <add key= value= type=System.String,mscorlib /> <add key= value= type=System.String,mscorlib /> <add key= value= type=System.String,mscorlib /> <add key= value= type=System.String,mscorlib /> <add key= value= type=System.String,mscorlib /> <add key=LevelName value=blabla type=System.String,mscorlib /> <add key= value= type=System.String,mscorlib /> <add key= value= type=System.String,mscorlib /> <add key= value= type=System.String,mscorlib /> <add key= value= type=System.String,mscorlib /> <add key= value= type=System.String,mscorlib /> <add key= value= type=System.String,mscorlib /> <add key= value= type=System.String,mscorlib /> <add key= value= type=System.String,mscorlib /> <add key= value= type=System.String,mscorlib /> <add key= value= type=System.String,mscorlib /> <add key= value= type=System.String,mscorlib /> <add key= value= type=System.String,mscorlib /> <add key= value= type=System.String,mscorlib /> <add key= value= type=System.String,mscorlib /> <add key= value= type=System.String,mscorlib /> <add key=Fastdisk1 value= type=System.String,mscorlib /> <add key=Fastdisk2 value= type=System.String,mscorlib /> <add key=Fastdisk3 value= type=System.String,mscorlib /> <add key=backupinterval value=300 type=System.String,mscorlib /> <add key= value= type=System.String,mscorlib /> <add key= value= type=System.String,mscorlib /> <add key= value=true type=System.String,mscorlib /> <add key=backupeverything value=false type=System.String,mscorlib /></values>

CarloM · October 27, 2011, 5:39pm

Use single quotes on the bigvariable assignment:

bigvariable='<?xml version="1.0" etc etc >'

With double quotes you're effectively passing a list of strings which will be concatenated - without the double quotes - so bigvariable will be:

<?xml version=1.0 encoding=utf-16 standalone=yes?> <values> <add key= value= type=System.String,mscorlib /> etc etc

WolwX · October 28, 2011, 6:56am

levelname=$(echo "${bigvariable}" | sed 's:.*<add key="LevelName" value="\([^"]*\)" type="System.String,mscorlib" />.*:\1:g')

This one work perfectly CarloM
Thanks

Last step I guess, when I use it into my full script system that's don't work, so I will try to extract the information from a flat file.
bigvariable will be writed in a single file.
How I can adapt your command with a flat file ?

I tried with this one without sucess =>

levelname=$(echo sed 's:.*<add key="LevelName" value="\([^"]*\)" type="System.String,mscorlib" />.*:\1:g' home/+LOGS+/99)

on this one too

levelname=$(echo "home/+LOGS+/99" | sed 's:.*<add key="LevelName" value="\([^"]*\)" type="System.String,mscorlib" />.*:\1:g')

CarloM · October 28, 2011, 7:00am

levelname=$(sed 's:.*<add key="LevelName" value="\([^"]*\)" type="System.String,mscorlib" />.*:\1:g' bigvariablefile)

WolwX · October 28, 2011, 7:29am

I'm sorry but that's don't work :x

levelname=$(sed 's:.*<add key="LevelName" value="\([^"]*\)" type="System.String,mscorlib" />.*:\1:g' /home/+LOGS+/99)
echo $levelname

Print all the bigvariable informations