Python Cat

Hi,

Below is the Python Script am trying to execute:

#!/usr/bin/python

import os
import sys
import subprocess
import time

def mpg(node, sitename):
	os.system('/usr/local/bin/expect /gsn/pdp/pdp_expect ' + node + ' > /tmp/' + node + '.temp')
	date = os.popen("cat /tmp/" + node + ".temp | grep time-sampled: | awk '{print substr($0, index($0,$2))}' | tr -d '\r''").read().strip()
	totalpdp = os.popen("cat /tmp/" + node + ".temp | grep pdp-active: | head -n 1 | awk '{print $2}' | sed 's/[^0-9]*//g'").read().strip()
	total_uplink = os.popen("cat /tmp/" + node + ".temp | head -n 95 | grep bytes | grep -v ipv6 | head -n 1 | awk '{ print $2 }' | sed 's/^[ \t]*//;s/[ \t]*$//' | sed 's/[^0-9]*//g'").read().strip()
	total_downlink = os.popen("cat /tmp/" + node + ".temp | head -n 95 | grep bytes | grep -v ipv6 | tail -1 | awk '{ print $2 }' | sed 's/^[ \t]*//;s/[ \t]*$//' | sed 's/[^0-9]*//g'").read().strip()
	total_uplink_packet = os.popen("cat /tmp/" + node + ".temp | head -n 95 | /usr/sfw/bin/ggrep -A 2 uplink: | grep packets: | awk '{ print $2 }' | sed 's/^[ \t]*//;s/[ \t]*$//' | sed 's/[^0-9]*//g'").read().strip()
	total_downlink_packet = os.popen("cat /tmp/" + node + ".temp | head -n 95 | /usr/sfw/bin/ggrep -A 2 downlink: | grep packets: | awk '{ print $2 }' | sed 's/^[ \t]*//;s/[ \t]*$//' | sed 's/[^0-9]*//g'").read().strip()
	total_uplink_packet_drop = os.popen("cat /tmp/" + node + ".temp | head -n 95 | /usr/sfw/bin/ggrep -A 3 uplink: | grep dropped-packets: | awk '{ print $2 }' | sed 's/^[ \t]*//;s/[ \t]*$//' | sed 's/[^0-9]*//g'").read().strip()
	total_downlink_packet_drop = os.popen("cat /tmp/" + node + ".temp | head -n 95 | /usr/sfw/bin/ggrep -A 3 downlink: | grep dropped-packets: | awk '{ print $2 }' | sed 's/^[ \t]*//;s/[ \t]*$//' | sed 's/[^0-9]*//g'").read().strip()
	attempted_bb_act = os.popen("cat /tmp/" + node + ".temp | /usr/sfw/bin/ggrep -A 20 blackberry.net | grep pdp-attempted-activation: | awk '{print $2}' | sed 's/^[ \t]*//;s/[ \t]*$//' | sed 's/[^0-9]*//g'").read().strip()
	completed_bb_act = os.popen("cat /tmp/" + node + ".temp | /usr/sfw/bin/ggrep -A 20 blackberry.net | grep pdp-completed-activation: | awk '{print $2}' | sed 's/^[ \t]*//;s/[ \t]*$//' | sed 's/[^0-9]*//g'").read().strip()
	attempted_web_act = os.popen("cat /tmp/" + node + ".temp | /usr/sfw/bin/ggrep -A 20 web.gprs.mtnnigeria.net | grep pdp-attempted-activation: | awk '{print $2}' | sed 's/^[ \t]*//;s/[ \t]*$//' | sed 's/[^0-9]*//g'").read().strip()
	completed_web_act = os.popen("cat /tmp/" + node + ".temp | /usr/sfw/bin/ggrep -A 20 web.gprs.mtnnigeria.net | grep pdp-completed-activation: | awk '{print $2}' | sed 's/^[ \t]*//;s/[ \t]*$//' | sed 's/[^0-9]*//g'").read().strip()
	active_pdp_web = os.popen("cat /tmp/" + node + ".temp | /usr/sfw/bin/ggrep -A 3 web.gprs.mtnnigeria.net | grep pdp-active: | awk '{print $2}' | sed 's/^[ \t]*//;s/[ \t]*$//' | sed 's/[^0-9]*//g'").read().strip()
	active_pdp_bb = os.popen("cat /tmp/" + node + ".temp | /usr/sfw/bin/ggrep -A 3 blackberry.net | grep pdp-active: | awk '{print $2}' | sed 's/^[ \t]*//;s/[ \t]*$//' | sed 's/[^0-9]*//g'").read().strip()
	
	print "Active PDP WEB IS " + str(active_pdp_web)
	print "Active PDP BB IS " + str(active_pdp_bb)
	
	#os.system('echo ' + str(date) + " Total PDP " + str(totalpdp) + ' > /tmp/' + str(sitename) + '.txt')
	
mpg("mpg", "slos")
	

But when I run this code,it gives the following error:

cat: write error: Broken pipe
cat: write error: Broken pipe
cat: write error: Broken pipe
cat: write error: Broken pipe
cat: write error: Broken pipe
cat: write error: Broken pipe

Script works fine with BASH and I have found some information via google that CAT is not easy to use in python, is this true?

Or is there any way I can get rid of the errors?

I still have a lot of lines with CAT that I want to implement and I can't risk going on without solving this.

Hi,
Why use cat file ?
grep,head,... not need to read only one file.

Regards.

Your code isn't Python -- just a transparently-thin shrinkwrapping of Python around 99% pure Bourne shell. I'm not sure why you bothered.

Even for shell code this is atrocious, though. You could do this in one awk call instead of umpteen cat's, sed's, tr's, grep's, and kitchen sinks.

I tried to figure out how many extra processes you're wasting, but I lost count. This is going to be extraordinarily, noticeably slow and wasteful, even on a modern computer. Please, seriously consider rewriting this properly, either using native Python features or properly in the shell.

Hi Corona688,

Thanks for your candid opinion, much appreciated. Am still a beginner, I have no programming experience, am just learning on my own via tutorials on the internet.

Be it as it may, my challenge remains "How do I optimize this code seeing that the input file alone from EXPECT statement is over 800 lines and I need to extract about 20 to 30 variables (KPI values) from it".

And I discovered that writing equivalent code for the CAT command in python will use more lines of code?

Or if anyone else has idea, I will appreciate it.

@disedorgue --- I will also check your opinion, thanks for taking time out to suggest an idea.

Many tools including python would happily go through the file, once, and write a sort or one or many output files with the desired information.Properly formatted, sorted output can then support the desired reporting from sequential access. It's very old school.

Optimize? Not really feasible. You've written inside-out code crossing three languages. Optimizing it would leave nothing of your original code and possibly strip out the python, the shell, or the awk.

Show the input you have, and the output you want, and we will show you a better way. As DGPickett says, read the file once, do all your processing once. awk or perl would be fairly powerful languages for this.

Your cat does nothing in python or shell -- see useless use of cat.

Hi all,

Find attached for the input file. (test_expect.txt)

I want to extract the following fields (some names are identical which needs to be mapped to unique variables):

All the "pdp-active" statistics

All the "ip-address-pool-statistics"

All the "ccas-statistics"

All the "uplink and downlink statistics"

All the "apn-radius-acct-server-statistics"

Great, you have posted the input you have.

Now post the output you want. I'll work with what you posted meanwhile.

$ cat pdp-active.awk

BEGIN {
        FS="[ \t\r:]+"
}

{ sub("^" FS, ""); }

/^(ccas-statistics|uplink|downlink|apn-radius-acct-server-statistics):/ {
        while( !($0 ~ "^" FS "$") ) {
                if(/tft-filter-count/) break;
                sub("^" FS, "");
                print
                if(getline != 1) break;
        }

        print ""
}

/pdp-active:/ { print $0"\n" }

$ awk -f pdp-active.awk test_expect.txt

pdp-active: 1219453

uplink:
packets: 5581950472180
bytes: 792988409784840
dropped-packets: 322643025207
packets-ipv6: 0
bytes-ipv6: 0
dropped-packets-ipv6: 317340352

downlink:
packets: 6356546278763
bytes: 5202689233327255
dropped-packets: 224502292039
packets-ipv6: 0
bytes-ipv6: 0
dropped-packets-ipv6: 0

pdp-active: 896591

uplink:
packets: 3485891928232
bytes: 512399111046839
dropped-packets: 318241658715
packets-ipv6: 0
bytes-ipv6: 0
dropped-packets-ipv6: 313917381

downlink:
packets: 4027201598702
bytes: 3078651754149495
dropped-packets: 222837858928
packets-ipv6: 0
bytes-ipv6: 0
dropped-packets-ipv6: 0

ccas-statistics:
ccas-identifier: DirectGy
ccas-start-request: 9062722155
ccas-start-request-failed: 138124458
ccas-update-request: 34654119968
ccas-update-request-failed: 14715326
ccas-stop-request: 8909036439
ccas-stop-request-failed: 23517853
ccas-user-service-denied: 516416544
ccas-user-unknown: 23056440
ccas-authorization-failure: 0
ccas-cc-not-applicable: 0

apn-radius-acct-server-statistics:
apn-radius-acct-server-address: 10.111.254.225
apn-radius-acct-requests-sent: 6339692763
apn-radius-acct-responses-received: 6330624775
apn-radius-acct-requests-timed-out: 9067987
apn-radius-acct-requests-retransmitted: 0
apn-radius-acct-invalid-authenticators-received: 0

apn-radius-acct-server-statistics:
apn-radius-acct-server-address: 10.111.254.226
apn-radius-acct-requests-sent: 6356302334
apn-radius-acct-responses-received: 6347847956
apn-radius-acct-requests-timed-out: 8454378
apn-radius-acct-requests-retransmitted: 0
apn-radius-acct-invalid-authenticators-received: 0

apn-radius-acct-server-statistics:
apn-radius-acct-server-address: 10.109.95.1
apn-radius-acct-requests-sent: 6352763779
apn-radius-acct-responses-received: 6341544445
apn-radius-acct-requests-timed-out: 11219323
apn-radius-acct-requests-retransmitted: 0
apn-radius-acct-invalid-authenticators-received: 69

apn-radius-acct-server-statistics:
apn-radius-acct-server-address: 10.109.95.2
apn-radius-acct-requests-sent: 6313912483
apn-radius-acct-responses-received: 6296608625
apn-radius-acct-requests-timed-out: 17303850
apn-radius-acct-requests-retransmitted: 0
apn-radius-acct-invalid-authenticators-received: 31

pdp-active: 314142

uplink:
packets: 2073810339890
bytes: 274003318141745
dropped-packets: 4004657993
packets-ipv6: 0
bytes-ipv6: 0
dropped-packets-ipv6: 3358345

downlink:
packets: 2299239432854
bytes: 2112912710702597
dropped-packets: 1612649642
packets-ipv6: 0
bytes-ipv6: 0
dropped-packets-ipv6: 0

ccas-statistics:
ccas-identifier: TSCCN2
ccas-start-request: 3359
ccas-start-request-failed: 3325
ccas-update-request: 2216
ccas-update-request-failed: 2
ccas-stop-request: 32
ccas-stop-request-failed: 0
ccas-user-service-denied: 0
ccas-user-unknown: 0
ccas-authorization-failure: 0
ccas-cc-not-applicable: 0

ccas-statistics:
ccas-identifier: DirectGy
ccas-start-request: 683192600
ccas-start-request-failed: 82961991
ccas-update-request: 9731908872
ccas-update-request-failed: 3170958
ccas-stop-request: 596751377
ccas-stop-request-failed: 250921
ccas-user-service-denied: 4245574
ccas-user-unknown: 17456139
ccas-authorization-failure: 0
ccas-cc-not-applicable: 0

$

This output should be much easier to load into any program since it's organized into blocks separated by one blank line, with consistent spacing and separators.

Give us an example first line or two for each report. The rest is mechanical, unless report lines repeat headings, then give us 2 lines with a multiple child first hit. We could put each major item's elements into the environment using prefixed names in a subshell for every ':.*[^ ]' line and then pick the values for each report line to echo onto that report file.