This is the sample json I have pasted here. I want all the IP address strings to be converted into an array. For example "10.38.32.202" has to be converted to ["10.38.32.202"] everywhere in the JSON. There are multiple IPs in a JSON I am pasting one sample object from the JSON. But the IPs already in an Array should not be altered.
{"network_ip_v4_address": "10.38.32.202","mac_address": "A0:12:34:45","network_ip_v4_address":["10.38.61.1","10.38.32.1"]}
The clean way to do this is to read the JSON file into an array (process the JSON ) and then write back to JSON in the format you desire.
There are many JSON libs to do this in Javascript, PHP, Python and yes, even with shell utilities:
Maybe try jq ?
https://stedolan.github.io/jq/
I'm not a jq user, because I process all my JSON in either Javascript, PHP or Python , but there are some folks here who are certainly jq users.
Not a big expert on jq but with a bit of playing around I got this to work:
jq 'if .network_ip_v4_address | type == "string" then .network_ip_v4_address |= [ . ] else . end' infile
After reading your request again it appears you want to identify an IP addresses in any non-array JSON data array and convert it to an array.
input :
{"adaptor" : "ip01" , "gateway" : "192.168.0.1" , "net_mask" : "255.255.0.0" }
{"ip_v4_address": "10.38.32.202","mac_address": "A0:12:34:45"}
{"adaptor": "ip12", "network_ip_v4_address":["10.38.61.1","10.38.32.1"]}
output:
{"adaptor" : "ip01" , "gateway" : ["192.168.0.1"] , "net_mask" : ["255.255.0.0"]}
{"ip_v4_address" : ["10.38.32.202"] , "mac_address" : "A0:12:34:45"}
{"adaptor" : "ip12" , "network_ip_v4_address" : ["10.38.61.1","10.38.32.1"]}
I have written a decend(f) function that applies f recursively to every component, this is similar to the walk(f) function already in jq, except it doesn't apply f to all the array elements. I then use this to identify any string values which are IP addresses and convert them to an array:
jq -c '
def decend(f):
. as $in |
if type == "object"
then
reduce keys[] as $key
( {}; . + { ($key) : ($in[$key] | decend(f)) } ) | f
else f
end
;
decend(
if type == "string" and
test("^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}$")
then [ . ]
else .
end)' infile
Not sure why it is not working for me. I am getting the output but without the IP address in String.
Input:
{"network_ip_v4_address":"10.38.32.202","mac_address":"A0:12:34:45","network_ip_v4_address":["10.38.61.1","10.38.32.1"],"network_gateway_address":["10.38.62.1","10.38.33.1"]}
Output:
{"mac_address":"A0:12:34:45","network_gateway_address":["10.38.62.1","10.38.33.1"],"network_ip_v4_address":["10.38.61.1","10.38.32.1"]}
It's not valid to have same key multiple times in the one record. Try running your input thru a JSON validator
Its working thank you so much.
--- Post updated at 02:26 AM ---
How to permanently apply these changes to the file.
The normal practice would be to redirect to a temporary file and then move the temp file over the original.
jq -c '.....' infile > /tmp/jqtemp.$$ && mv -f /tmp/jqtemp.$$ infile
There are a couple of things I'm a little unhappy with in my proposed solution and I'd like to update to a more robust version.
Firstly if the JSON structure contains array of objects decend() stops drilling down at the array.
Second the IP address RE is inaccurate and and gets false positives eg 999.999.999.999 .
These issues are illustrated by the following example:
{
"non-ip" : "999.2.3.1",
"Adaptors" : [
{"device" : "ip01" , "gateway" : "192.168.0.1" , "net_mask" : "255.255.0.0" },
{"device": "ip12", "network_ip_v4_address":["10.38.61.1","10.38.32.1"]}
]
}
I have tightened down IP address RE and enhanced the decend() function it now takes a parent object parameter and will not apply f to string objects parented by arrays:
jq -c '
def decend(p;f):
. as $in
| if type == "object" then
reduce keys_unsorted[] as $key
( {}; . + { ($key) : ($in[$key] | decend($in;f)) } ) | f
elif type == "array" then map( decend($in;f) ) | f
elif (p | type) == "array" then .
else f
end;
decend(.;
if type == "string" and
test(
"^(" +
"(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)" +
"\\.){3}" +
"(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)" +
"$")
then [ . ] else . end)' infile
The example above is not converted correctly:
{
"non-ip" : "999.2.3.1",
"Adaptors" : [
{"device" : "ip01", "gateway" : ["192.168.0.1"], "net_mask" : ["255.255.0.0"]},
{"device" : "ip12", "network_ip_v4_address" : ["10.38.61.1", "10.38.32.1"]}
]
}
Thats Great!! I really appreciate it. Thank you so much.. you made my day as I was struggling from quite few days.
This piece of code didn't work for me. Anyways this is not required in my case I believe. --- Firstly if the JSON structure contains array of objects decend() stops drilling down at the array.
def decend(p;f):
. as $in
| if type == "object" then
reduce keys_unsorted[] as $key
( {}; . + { ($key) : ($in[$key] | decend($in;f)) } ) | f
elif type == "array" then map( decend($in;f) ) | f
elif (p | type) == "array" then .
else f
end;
This piece of code, I mean the Regex is working --- Second the IP address RE is inaccurate and and gets false positives eg 999.999.999.999.
decend(.;
if type == "string" and
test(
"^(" +
"(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)" +
"\\.){3}" +
"(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)" +
"$")
then [ . ] else . end)
Any reason why the jq code in the shell script is not executing when run from crontab. The shell code other than jq is executing properly without any issue. How to make it working? anything to do with bashprofile & how to do it? Thanks!
My first guess would be the PATH variable doesn't contain the folder where jq is stored. I'd try specifying a PATH in your script:
Find where jq is installed eg:
$ type jq
jq is hashed (/usr/bin/jq)
Now specify a sane PATH at the top of your cron script:
#!/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin