Bash script - cygwin (powershell?) pull from GitHub API Parse JSON

Shell Programming and Scripting
1

All,

Have a weird issue where i need to generate a report from GitHub monthly detailing user accounts and the last time they logged in. I'm using a windows box to do this (work issued) and would like to know if anyone has any experience scripting for GitAPI using windows / cygwin / powershell?

I would normally curl for the stuff i wanted and use jsawk to parse the returned JSON, but that's not working (and i don't see an option to add jsawk to Cygwin) so i'm at a loss: Any help greatly appreciated.

Edit: thought i would add that the code block below is working except for the 'jsawk' operation. I was just including that as an example of how i would do this in a real terminal.

#!/bin/bash
IFS_bak=$IFS
IFS=$'\r\n'
uid='xxxxxxx'
GH_OAUTH='xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
curl -i --silent https://api.github.com:443/orgs/xxxxxxxxxxx/repos?access_token=$GH_OAUTH
for RepoLine in `curl --silent https://api.github.com:443/orgs/xxxxxxxxxxxxxxx/repos?access_token=$GH_OAUTH | _
  jsawk -n 'out(this.name)'`
do 
   RepoName=$(echo $RepoLine | awk -F" : " '{ print $1 }')
curl -i --silent https://api.github.com/repos/xxxxxxxxxxxx/$RepoName/teams?access_token=$GH_OAUTH
2

A bit of a hack, but without jsawk , you might try and get away with something like:

awk -F\" '$2==s{print $(NF-1)}' s=name

It may be enough for your application..

1 Like
3

Scrutinizer, thank you for the suggestion. That worked although i ran into some other issues after the fact. Long story - in the end i broke down and had the IT guys bring me an old PC destined for the recycle bin and I installed CentOS 6.5 on it so we can do this the right way :slight_smile: . Now i'm running into another issue and i'm not sure what what may be wrong here: Original code block -

#!/bin/bash
IFS_bak=$IFS
IFS=$'\r\n'
uid='xxxxxxxxxxxxxxxxxxxxxxxxxxx'
GH_OAUTH='xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
curl -i --silent https://api.github.com:443/orgs/xxxxxxxxxxxxxx/repos?access_token=$GH_OAUTH
for RepoLine in `curl --silent https://api.github.com:443/orgs/xxxxxxxxxxx/repos?access_token=$GH_OAUTH | _
  jsawk -n 'out(this.name)'`
do 
   RepoName=$(echo $RepoLine | awk -F" : " '{ print $1 }')
curl -i --silent https://api.github.com/repos/xxxxxxxxxxxxxx/$RepoName/teams?access_token=$GH_OAUTH

Getting an error in terminal"

"Line 12: Syntax error: Unexpected End of File. "

Seems like something simple, any ideas?

4

You're welcome! There seems to be a done missing. What is the underscore doing?

5

Right on both counts good sir! Thank you for your time.

---------- Post updated 08-27-14 at 12:14 PM ---------- Previous update was 08-26-14 at 04:49 PM ----------

Didn't want to start a new thread for this, sorry if this reply is too old but wanted to ask another question.

I've got the script below working, however it's not really doing what i want and i must be missing something.

these are the requirement requirements:

Initial curl ---- return list of repos for our org
"foreach" repository name in list returned (JSON) --- curl for list of users with access
"foreach" repo user list returned (JSON) ---- output usernames to a txt, csv, whatever file for auditing.

I'm struggling a little here as i'm not very experienced with bash shell scripting and jsawk in particular. Turns out there is no direct way to "for each" is bash (as far as i can tell) and i think the script may not be looping correctly. Any help appreciated, thanks for having a read.

#!/bin/bash -x
IFS_bak=$IFS
IFS=$'\r\n'
uid='xxxxxxxxxxxxxxxxxxxx'
GH_OAUTH='xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
curl -i  https://api.github.com:443/orgs/xxxxxxxxxxxxxxxxxxxxxxxxxxx/repos?access_token=$GH_OAUTH
for RepoLine in `curl --silent https://api.github.com:443/orgs/xxxxxxxxxxxxxxxxxxxxx/repos?access_token=$GH_OAUTH |
  jsawk -n 'out(this.name)'`
do
  RepoName=$(echo $RepoLine | awk -F" : " '{ print $1 }')
   curl -i https://api.github.com/repos/xxxxxxxxxxxxxxxxxxxxxxxxx/$RepoName/teams?access_token=$GH_OAUTH | 
  jsawk -n 'out(this.login)' > output.txt
done
6

Can you post a sample of what your first curl downloads and also what your first jsawk produces with this as input?

There is a second do statement in your post, that I think should not be there..

7

Scrutinizer, here are the files requested: The first shows the API curl response, the second outputs the jsawk stuff from mem, had to change the code a little to get the jsawk out, here's what i did --->

sorry not that good with awk / jsawk still learning:

#!/bin/bash
FS_bak=$IFS
IFS=$'\r\n'
uid='XXXXXXXXXXXXXXXXXX'
GH_OAUTH='XXXXXXXXXXXXXXXXXXXXXXXXX'
curl --silent  https://api.github.com:443/orgs/XXXXXXXXXXXXX/repos?access_token=$GH_OAUTH > output.txt
cat output.txt | jsawk -n 'out(this.name)' | jsawk 'return this.name' | > outputjsawk.txt
8

The outputjsawk.txt appears to be some kind of binary file. Did something go wrong?

When I process APIoutputScrub.txt with

awk -F\" '$2==s{print $(NF-1)}' s=name

instead, I get:

xxxxxxxxxxxxxx
xxxxxxxxxxx
xxxxx
xxxxxxx
xxxxxxxxxxx
xxxxxxxxxxx
xxxxxxxxxxx
xxxxx
xxxxxxxxx
9

yes, sorry not sure what i did there, the output looks like you described:

xxxxxxxxx
xxxxxx-xxxxxxxx
xxxxxx
xxxx-xxx
xxxxxxxxx-xxxx
xxxxxxxx-xxx
xxxxx-xxxxxx-xxxx-xxxxxxxx
xxxxx
xxxxx-xxxxxxxx
10

Instead of

....
  jsawk -n 'out(this.login)' > output.txt
done

Try:

....
  jsawk -n 'out(this.login)' 
done > output.txt