A simpler XML tool

We've been getting a lot of XML questions lately, and I suspect it's only going to get worse better ... Normal shell utilities just can't handle it and the "proper" solutions, do-everything perl modules or things like xmlstarlet, just make my head ache.

Started coding something tonight. What the mockup can do, right now:

$ wget -q -O - "http://rss.cnn.com/rss/cnn_topstories.rss" | ./mox item.title 2> /dev/null

<title>Israeli soldier freed after 5 years</title>
<title>Expert: Long road ahead for Shalit</title>
<title>Malaria vaccine tantalizingly close</title>
<title>Clinton makes unannounced Libya trip</title>
<title>Body may be missing Maryland boy</title>
<title>Disabled captives case may grow</title>

$

I'm hoping to eventually make it a language to match and rearrange xml, with a built-in event loop sort of like awk's.

Can you provide a little bit more background and explain the objective? What do you mean by "handle": parsing, displaying, selecting data elements and/or attributes, other?

I'm trying to build a simple XML-manipulation tool. Very simple -- no DTD -- possibly not even validating.

The point is to easily match and extract and rearrange tags from inside nested context. "item.title" there extracts <title> tags and any contents in them, only when found directly inside <item> tags. You could do "^html.head.title" to extract title attributes from webpages, "table" to extract anything inside tables, and maybe "tag:key=value" to match tags where key='value' and anything inside.

I'm still thinking on how exactly to work the syntax but hope this explains the idea.

Use XPath syntax (or rather a subset)?

Perhaps. Xpath looks verbose, ugly, complicated, and redundant though. It's not really meant for streams.

Great initiative. Let us know how far you are getting and whether you are prepared to release in the public domain.

Just to add that a special GNU awk implementation XMLgawk exists.

Basic XPath syntax for specifying nodes is pretty simple.

^html.head.title - /html/head/title
table - //table
tag:key=value - tag[key=value]

Not much difference. But it was just a thought - seems easier to re-use an existing syntax than invent a new one...

Very strange... I did not edit the blue word in there, and it doesn't tell me which moderator did this or why.

I meant 'worse' in that if you know anything about awk and xml, you'll know they aren't suitable for each other, but we keep getting asked anyway.

---------- Post updated at 10:40 AM ---------- Previous update was at 10:35 AM ----------

Right now it can do this:

$ ./mox < cnn_topstories.rss /item/title 2> /dev/null

<title>Satellite expected to hit Earth this weekend</title>

<title>Libyan elections 'coming within months'</title>

<title>NATO to end mission by Oct. 31</title>

<title>What next?</title>

<title>Gadhafi's demise and the Arab Spring</title>

<title>Ahmadinejad: U.S. hated around world</title>

<title>Heir to Saudi throne dies in New York</title>

<title>At least 9 killed in Yemen clashes</title>

<title>China in mourning after toddler's death</title>

<title>Tunisia set for first Arab Spring election</title>

$

IOW, pretty much the same as my OP. But the parser is much better now.

I intend to release it into the public domain should I come up with anything useful.

I'm adopting some parts of the xpath syntax but don't think I'll implement the whole thing the same way. A lot of it does not apply.

Though I should check out this xmlgawk and see if it will do what I want.

---------- Post updated at 12:01 PM ---------- Previous update was at 10:40 AM ----------

xmlgawk refused to run properly unless actually installed, no way to test it without that.

It needed library files teased into random places by hand, make install didn't bother installing extensions -- like the xml one. The INSTALL and README's were still the useless generic ones.

Its XML parser is pointlessly strict. Feeding it a webpage inevitably causes it to die with "mismatched tag" before it bothers processing any data at all.

Their first trivial example didn't work at all, I had to modify it into something it would.

Not impressed.

It worked for me, when configured with:

./configure --disable-shared --enable-static-extensions

I don't know if it's supposed to work for html parsing though ...

HTML is a subset of XML with a few weird bits. If you can't parse HTML, you're ignoring most of the XML in the universe. The xgawk documentation claims it's supposed to be nonvalidating for the purpose of parsing less-than-ideal XML...

Building in the extensions statically is a very good idea.