awk use to generate report

UNIX for Beginners Q & A
1

Hi ,
In a directory list of ddl files are stored in the given format above. Above is the sample ddl file. The ddl file name is same as that of table name ie email_notifications.ddl
I want to generate below report using awk utility reading all the ddl files stored in /ddl path

Desired output:
table name, last_modified_time, totalsize
email_notifications,1559883505,8765717674
email_bounced, 1559883505,999999

Also, would like to know if the modified time in unix can be converted to actual date time instead of unix time

Note: It may happen that in other DDL files TBLPROPERTIES information may or may not available.

here is one of the file sample format:

CREATE TABLE `test.email_notifications`(
  `email_address_id` decimal(16,0), 
  `individual_id` decimal(16,0), 
  `email_address` string, 
  `last_bounce_date` string 
)
ROW FORMAT DELIMITED 
  FIELDS TERMINATED BY '|' 
STORED AS INPUTFORMAT 
  'org.apache.hadoop.mapred.TextInputFormat' 
OUTPUTFORMAT 
  'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION
  'hdfs://apps/hive/db/test.db/email_notifications'
TBLPROPERTIES (
  'last_modified_by'='ops', 
  'last_modified_time'='1559883505', 
  'numFiles'='1', 
  'numRows'='0', 
  'rawDataSize'='0', 
  'skip.header.line.count'='1', 
  'totalSize'='8765717674', 
  'transient_lastDdlTime'='1559883505')
2

You can process text with any "modern" programming language you choose.

You should pick a programming language, assuming you work in an organization, based on the skill sets and tech support policies of the organization. If your shop is mostly python , then you should write text processing scripts in python ; if you are in web shop using PHP , you should process test in PHP . The more different languages your organization must support, the more difficult (and expensive) it is. If you work in a shop with hard-core unix people who
have all their code base in shell scripts, then use these same scripts, based on the organizational culture, business model and skill sets.

So, the question you should ask yourself is "what is the programming language to process text in my organization based on the policy of the organization and the current and future skill sets of the staff?"

Most every programming language can process text. You should work within your organizations skill sets and match the programming needs with the skills of the staff, especially the younger staff, who will be around much younger than the retiring staff.

Having said that,

If you want to use awk and your organization supports this, you should be programming in awk and posting your code here for review and help.

Often the question "can I do this?" with this programming language is a veiled question to ask others to code for them.

Please show your own coding work.

Thanks.