How to convert any shell command output to JSON format?

UNIX for Beginners Q & A
1

Hi All,

I am new to shell scripting, Need your help in creating a shell script which converts any unix command output to JSON format output.
example:
sample df -h command ouput :

Filesystem   size   used   avail   capacity   Mounted
/dev/dsk/c1t0d0s0   8.1G   4.0G   4.0G   50%   /
/devices   0K   0K   0K   0%   /devices
ctfs   0K   0K   0K   0%   /system/contract
proc   0K   0K   0K   0%   /proc
mnttab   0K   0K   0K   0%   /etc/mnttab
swap   3.1G   836K   3.1G   1%   /etc/svc/volatile
objfs   0K   0K   0K   0%   /system/object
sharefs   0K   0K   0K   0%   /etc/dfs/sharetab
/usr/lib/libc/libc_hwcap1.so.1   8.1G   4.0G   4.0G   50%   /lib/libc.so.1
fd   0K   0K   0K   0%   /dev/fd
swap   3.1G   84K   3.1G   1%   /tmp
swap   3.1G   32K   3.1G   1%   /var/run
/dev/dsk/c1t0d0s7   11G   11M   11G   1%   /export/home
/vol/dev/dsk/c0t0d0/sol_10_113_x86   2.1G   2.1G   0K   100%   /cdrom/sol_10_113_x86

Desired output in JSON format:

{
"Filesystem": "/dev/dsk/c1t0d0s0",
"size": "8.1G",
"used": "4.0G",
"avail": "4.0G"
"capacity": "50%",
"Mounted": "/"
}
2

Like everyone here, you are expected to show your own work, provide the name of your OS, the shell version you are using, etc.

And you are expected to search the forums before posting your question.

Welcome! And please follow forum guidelines.

Thanks.

What code have you written and tried so far?

3

Hi balu1234,

Welcom to the forum!

I'll echo Neo's comments here, what research have you done and what have you tried?

I'd also add that if you are looking for a converter that will convert any unix command line output to JSON, that is a big ask in no small measure due to the complexity and variety of outputs available in the unix cli.

If you only would like to have this output converted on an occaisional basis, then there are a number of online converters - I usually use this one. This provides a number of output options, I have added the keyed output as an example.

[
 {
   "Filesystem": "/dev/dsk/c1t0d0s0",
   "size": "8.1G",
   "used": "4.0G",
   "avail": "4.0G",
   "capacity": "50%",
   "Mounted": "/"
 },
 {
   "Filesystem": "/devices",
   "size": "0K",
   "used": "0K",
   "avail": "0K",
   "capacity": "0%",
   "Mounted": "/devices"
 },
 {
   "Filesystem": "ctfs",
   "size": "0K",
   "used": "0K",
   "avail": "0K",
   "capacity": "0%",
   "Mounted": "/system/contract"
 },
 {
   "Filesystem": "proc",
   "size": "0K",
   "used": "0K",
   "avail": "0K",
   "capacity": "0%",
   "Mounted": "/proc"
 },
 {
   "Filesystem": "mnttab",
   "size": "0K",
   "used": "0K",
   "avail": "0K",
   "capacity": "0%",
   "Mounted": "/etc/mnttab"
 },
 {
   "Filesystem": "swap",
   "size": "3.1G",
   "used": "836K",
   "avail": "3.1G",
   "capacity": "1%",
   "Mounted": "/etc/svc/volatile"
 },
 {
   "Filesystem": "objfs",
   "size": "0K",
   "used": "0K",
   "avail": "0K",
   "capacity": "0%",
   "Mounted": "/system/object"
 },
 {
   "Filesystem": "sharefs",
   "size": "0K",
   "used": "0K",
   "avail": "0K",
   "capacity": "0%",
   "Mounted": "/etc/dfs/sharetab"
 },
 {
   "Filesystem": "/usr/lib/libc/libc_hwcap1.so.1",
   "size": "8.1G",
   "used": "4.0G",
   "avail": "4.0G",
   "capacity": "50%",
   "Mounted": "/lib/libc.so.1"
 },
 {
   "Filesystem": "fd",
   "size": "0K",
   "used": "0K",
   "avail": "0K",
   "capacity": "0%",
   "Mounted": "/dev/fd"
 },
 {
   "Filesystem": "swap",
   "size": "3.1G",
   "used": "84K",
   "avail": "3.1G",
   "capacity": "1%",
   "Mounted": "/tmp"
 },
 {
   "Filesystem": "swap",
   "size": "3.1G",
   "used": "32K",
   "avail": "3.1G",
   "capacity": "1%",
   "Mounted": "/var/run"
 },
 {
   "Filesystem": "/dev/dsk/c1t0d0s7",
   "size": "11G",
   "used": "11M",
   "avail": "11G",
   "capacity": "1%",
   "Mounted": "/export/home"
 },
 {
   "Filesystem": "/vol/dev/dsk/c0t0d0/sol_10_113_x86",
   "size": "2.1G",
   "used": "2.1G",
   "avail": "0K",
   "capacity": "100%",
   "Mounted": "/cdrom/sol_10_113_x86"
 }
]

Regards

Gull04

1 Like
4

In light of Neo's comments, I'll only give you some pointers.

First, install jq . It is a tool for creating, validating and editing JSON files and can be used to extract data from a JSON file, too. If you are using Linux it will probably be available as a package in your distributions repository. If not, go to their (jq) web site and download for your system.

Second, search for CSV to JSON using jq with your favourite search engine. There is a good example (with explanations) on converting pipe-delimited values to JSON using jq on Stack Exchange.

You will probably have to tailor your solution to each command, as gull04 says (in so many words).

When you have an idea of what to do post your attempt here and I'm sure you will have lots of advice on how to fine-tune it.

Andrew

1 Like
5

Hi Andrew,

Thanks for your response.

CentOS Linux 7 is my operating system.

i installed jq pkg on my server.

i tried below command:

# jq -s -R '[[ split("\n")[] | select(length > 0) | split("  +";"") | {(.[0]): .[1]}] | add]' /tmp/df.out

i redirect my output to /tmp/df.out file.

but i am getting below output, but not the output which i desired.

[
  {
    " Filesystem Size Used Avail Use% Mounted ": null,
    " /dev/mapper/centos-root 37G 15G 22G 41% / ": null,
    " devtmpfs 1.9G 0 1.9G 0% /dev ": null,
    " tmpfs 1.9G 0 1.9G 0% /dev/shm ": null,
    " tmpfs 1.9G 9.8M 1.9G 1% /run ": null,
    " tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup ": null,
    " /dev/sda1 1014M 235M 780M 24% /boot ": null,
    " tmpfs 379M 4.0K 379M 1% /run/user/42 ": null,
    " tmpfs 379M 36K 379M 1% /run/user/1000 ": null,
    " /dev/sr0 4.3G 4.3G 0 100% /run/media/abhinav/CentOS ": null,
    " tmpfs 379M 0 379M 0% /run/user/0 ": null
  }
]
6

The trick of course if to create an array of JSON objects which are comma separated key, value pairs.

I don't think a tool like jq is designed to "think for you, as to how to create the array of JSON objects from raw data".

That's up to you to design, specific to this single application first.

Personally, I think you are jumping to tools like jq too quickly before you fully understand / design your algorithm to take your raw output and create an array of JSON objects.

If it was me, I would write this in PHP and it would only take a few lines of PHP code to take your input and create the desired JSON file.

Either way, bash or PHP or Javascript, it a simply matter:

  • Taking the output of the command as text as input into your code.
  • Parsing this data.
  • Creating a JSON object from each line of parsed data (the values), keeping the header line as the key to the JSON objects..
  • Add the JSON object above to an another array.
  • Repeat for the next line of data in your output

I have done this 100s of time (using PHP) and there is generally no short-cut to writing your algorithm to take raw text as data and parsing it into a JSON object and then creating an array of JSON objects.

However, generally speaking I create an array of arrays and then convert the final array to JSON; but it really depends on the format of the final JSON object and if it needs to be stringified to be transmitted over the net.

Also, I would caution you not to try to create "one parser to rule them all" as the output of other commands will like need a different parser.

Then when you have done this for a number of different commands, you can consider if you want to build "the mother of all UNIX commands to JSON parser".

Reference:

PHP: json_encode - Manual

1 Like
7

Did you find any examples on the web? This is the article I was alluding to in my previous post. Also the output of the df command you are using needs to be modified to turn it into something more akin to a CSV file. I would suggest:

df -h | tr -s ' ' ',' > /tmp/df.out

as I'm not sure that jq 's split function can be used to split on multiple spaces as a single delimiter.

As you have had a go, I'll post a sample based on the above link.

$ df -h . | tr -s ' ' ',' | jq -nR '[ 
( input | split(",") ) as $keys | 
( inputs | split(",") ) as $vals | 
[ [$keys, $vals] | 
transpose[] | 
{key:.[0],value:.[1]} ] | 
from_entries ]'
[
  {
    "Filesystem": "/dev/sda1",
    "Size": "150G",
    "Used": "100G",
    "Avail": "50G",
    "Use%": "60%",
    "Mounted": "/home"
  }
]
$

It's not a trivial piece of code, and when I found it I pulled it apart, line by line, until I had an understanding of what is going on.

@NEO, I don't generally do much with JSON, and find JQ is a good tool for pretty-printing, or extracting data from a JSON file. If I wanted to manipulate JSON data, I would probably use python, just as you use PHP.

Andrew

1 Like
8

I should probably start using Python for routine tasks more .... it time I got off my dependance on PHP in the shell and start doing more in Python, me thinks.

Thanks Andrew.

1 Like
9

I created a tool called jc that converts the output of many command line tools, including df, to json format:

kellyjonbrazil/jc on github.

$ df | jc --df -p
[
  {
    "filesystem": "udev",
    "1k-blocks": "977500",
    "used": "0",
    "available": "977500",
    "use_percent": "0%",
    "mounted": "/dev"
  },
  {
    "filesystem": "tmpfs",
    "1k-blocks": "201732",
    "used": "1204",
    "available": "200528",
    "use_percent": "1%",
    "mounted": "/run"
  },
  {
    "filesystem": "/dev/sda2",
    "1k-blocks": "20508240",
    "used": "5748312",
    "available": "13695124",
    "use_percent": "30%",
    "mounted": "/"
  },
  {
    "filesystem": "tmpfs",
    "1k-blocks": "1008648",
    "used": "0",
    "available": "1008648",
    "use_percent": "0%",
    "mounted": "/dev/shm"
  }
  ...
]
1 Like
10

Hi, kbrazil.
Thanks for the code. I installed it, tried it out, and it worked as expected.

For your next rev, I suggest:

jc [-p] -- "command"

So that jc -- "du" would run the du command from within python without me needing to specify du (or any other command for which you have a parser) twice

I've done that with perl a few times, but I don't know enough about python to say any more about it on that language platform ... cheers, drl

11

Hi DRL,

You can now use this style of synax with the latest version of jc!

Cheers,
kb

12

Hi, kbrazil.

Thanks for the update.
I ran this to update (as I had done earlier to do the initial install):

pip3 install --upgrade jc

but this is what I got:

No files/directories in /tmp/pip-build-wg68rqx6/ruamel.yaml.clib/pip-egg-info (from PKG-INFO)
Storing debug log for failure in /root/.pip/pip.log

On a system:

OS, ker|rel, machine: Linux, 3.16.0-7-amd64, x86_64
Distribution        : Debian 8.11 (jessie) 
Python 3.4.2
pip3 pip 1.5.6 from /usr/lib/python3/dist-packages (python 3.4)

Let me know if you see something wrong here. Do you want me to post this as an issue on github?

However, I got:

...
Successfully built ifconfig-parser
Installing collected packages: ruamel.yaml.clib, ruamel.yaml, xmltodict, ifconfig-parser, jc
Successfully installed ifconfig-parser-0.0.5 jc-1.7.4 ruamel.yaml-0.16.10 ruamel.yaml.clib-0.2.0 xmltodict-0.12.0

On a system:

OS, ker|rel, machine: Linux, 5.4.0-4-amd64, x86_64
Distribution        : Debian GNU/Linux bullseye/sid 
Python 3.7.6
pip3 pip 18.1 from /usr/lib/python3/dist-packages/pip (python 3.7)

And the command jc -p ls -l /usr/bin appeared to work correctly on that system.

Best wishes, cheers, drl

13

Hi DRL,

Sorry I just saw your post. I'm not sure why the yaml library didn't install via PIP on that system. Perhaps there was an issue with PIP or a conflict with another library? Might make sense to try installing it in a venv on that system to see if it works in a clean environment.

Ultimately it won't break anything unless you try to use the yaml parser on that system.

Actually taking a closer look, it seems the version of Python might be too old on the Debian 8.11 system. I would ensure the Python version on the system is at least 3.6.

Cheers,
kb

14

Hi, kb.

Thanks. As I noted, it did install and execute correctly on a more recent system, so I'll need to use it there. Thanks for your time and effort ... cheers, drl