Shell script inputs to python

Shell Programming and Scripting
1

Hi
I am trying to pass 2 input parameters from shell script to python API end point ,but not passing what i expected when print those inputs .Please advise

data.txt

 " 7554317" ,xx5e1
 " 7554317" ,xx96
 " 7554317" ,xxd6
 " 554317" ,xde
 
cat $sites/data.txt |sort |uniq >$sites/a.txt

for i in `cat $sites/a.txt`
do
#echo "test: $i"
 id=`echo $i|awk -F "," '{print $1}'`
 tid=`echo  $i|awk -F "," '{print $2}'`

echo "id : $id"
echo "tid : $tid"

python test.py /data'{" Id":"'"${id}"'""group":"get", "item":"'"${ tid}"'"],"fields":[" Id","type"," Parameters","xx"]}'

done

Getting output
id: " 7554317"
tid: 
id:
tid: xx5e1
id: " 7554317"
2

I'm assuming you don't want the quotes around variable id but you do want to keep any leading (or internal) spaces(s).

Remove echo if python invocation is how you want.

sort $sites/data.txt | uniq | while IFS=$',\t\n' read id tid
do
    # remove quotes from around id
    id=${id#*\"}
    id=${id%\"*}

    echo "id: \"${id}\""
    echo "tid: \"${tid}\""

    echo python test.py /data'{" Id":"'"${id}"'", "group":"get", "item":"'"${tid}"'"],"fields":[" Id","type"," Parameters","xx"]}'
done

Output:

id: " 554317"
tid: "xde"
python test.py /data{" Id":" 554317""group":"get", "item":"xde"],"fields":[" Id","type"," Parameters","xx"]}
id: " 7554317"
tid: "xx5e1"
python test.py /data{" Id":" 7554317""group":"get", "item":"xx5e1"],"fields":[" Id","type"," Parameters","xx"]}
1 Like
3

Python does not need help from the shell or another utilities to do what it is shown:

Modify test.py to handle the parsing, the sorting, the processing of uniqueness and the tokenization of the data.txt.

Here's an example using a print() for visualization.

with open('data.txt') as f:
    records = set()
    for line in f:
        id, tid = line.strip().split('" ,')
        id = id[1:]
        records.add((id,tid))

for r in sorted(records):
    id, tid = r
    print("id: {}, tid: {}".format(id, tid))
2 Likes
4

Hi ,
When extracting data from API,there are 2 nested groups having same group name [group:url,title and item with different values].I want to merge these 2 nested groups to 1 or individual nested groups otherwise for each unique site more than 2 records .Please advise

 {
      "time": 1522224314,
      "customParameters": [
    
        {
          "group": "url",
          "item": "https://www.xxx.a/3990154.html"
        },
        {
          "group": "title",
          "item": "a90189275"
        }
      ],
      "site": "xx877",
      "type": "test",
      "userId": "xx1"
    },
5

Hmmm. Since the output is JSON, I advise using python JSON-Library/Functions.

Build up your data structure as needed in python and dump JSON-Format via function.

As Aia said: "Python does not need help from the shell or another utilities"

If you are able to use Python in this case, you probably won't like to use inferior Shellcode.

6

Hi ,
I have tried with python json library,but getting the following error

Traceback (most recent call last):
File "a11.py", line 4, in <module>
data = json.load(data_file)
File "/usr/lib/python2.7/json/__init__.py", line 291, in load
**kw)
File "/usr/lib/python2.7/json/__init__.py", line 339, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.7/json/decoder.py", line 367, in decode
raise ValueError(errmsg("Extra data", s, end, len(s)))

test.py

import json

with open('/home/test/a.json') as data_file:
    data = json.load(data_file)

for a in data['events']:
    a['customParameters'] = a['customParameters'][1]

new_some_json_string = json.dumps(data, indent=2)
 
print(new_some_json_string)

data file :a.json 
 "events": [
{
      "time": 1522224314,
      "customParameters": [

        {
          "group": "url",
          "item": "https://www.xxx.a/3990154.html"
        },
        {
          "group": "title",
          "item": "a90189275"
        }
      ],
      "site": "xx877",
      "type": "test",
      "userId": "xx1"
    }
]

expecting the output 
site      type   userId   group  item            group1  item1
xx877   test   xx1       url       https...        title      a90189275