Merge configuration files with similar stanzas

UNIX for Beginners Q & A
1

Greetings,
I'm attempting to create a script or command that will allow me to merge two configuration files that have the same stanzas. For example:

default/savedsearches.conf

[Denied Firewall Traffic]
disabled = 1
action.email.useNSSubject = 1
alert.track = 0
dispatch.earliest_time = -60m@m
dispatch.latest_time = now

local/savedsearches.conf

[Denied Firewall Traffic]
disabled = 0
action.email.useNSSubject = 1
alert.track = 0
dispatch.earliest_time = -60m@m
dispatch.latest_time = now
display.page.search.tab = statistics
display.visualizations.show = 0
request.ui_dispatch_app = search
request.ui_dispatch_view = search

My intent is to merge / combine these stanzas into one, but ensure that whatever is in the local folder overwrites the same line in the default folder.

merged/savedsearches.conf

[Denied Firewall Traffic]
disabled = 0
action.email.useNSSubject = 1
alert.track = 0
dispatch.earliest_time = -60m@m
dispatch.latest_time = now
display.page.search.tab = statistics
display.visualizations.show = 0
request.ui_dispatch_app = search
request.ui_dispatch_view = search

Obviously, there's more than one stanza within the file, but I'm struggling to find a way to properly append one file to the other. Thus far I've tried

cat local/savedsearches.conf default/savesearches.conf > merged/savedsearches.conf

However, that doesn't actually place the settings under one inclusive stanza. I've also played around with using sort but that doesn't actually do what I'm trying to achieve. I've also tried playing with / using the merge_config.sh script but I'm still not getting the results I require.

Any advice or assistance in resolving this would be a huge help for me, thank you!

2

Welcome!

How can you detect the end of a stanza?
Perhaps give an example with two stanzas?

3

Great question, the stanzas are separated by a space / black line as shown below.

[Denied Firewall Traffic]
disabled = 1
action.email.useNSSubject = 1
alert.track = 0
dispatch.earliest_time = -60m@m
dispatch.latest_time = now
display.general.type = statistics

[Denied Firewall Traffic]
disabled = 1
alert.track = 0
dispatch.earliest_time = -24h@h
dispatch.latest_time = now
display.general.type = statistics
4

@Unix_Nubie.
please start using markdown code tags as outlined in the forum welcome page when posting data/code samples.
I've edited your posts for now, but please start doing so going forward!

1 Like
5

@Unix_Nubie , hi, in addition to other teammate requests, please supply the following

  • expected output of a merge when -

  • files contain common stanzas - which stanza (file?) takes precedence
    we'd presume something like
    stanzmerge newStanzaFile oldStanzaFile > mergedStanzaFile

  • files contain an intersection of some stanzas - is ordering of the output pertinent ?

  • files contain no common stanzas - is this just a concatenation of files or no output?.

is the 'cat local/savedsearche ...' command shown your only attempt ?,
if others, share those please.

the more information/expectation you provide the better the chances are of the team supplying/suggesting solutions.

tks

6

This is a bit complex, a piping of simple tools like sort, uniq, cat won't work out.

Each key must be unique per stanza, but can occur in multiple stanzas.
Input files have stanzas in random order, and keys are per stanza but in random order.

This can be represented by a two-dimensional array indexed by stanza and key, storing the values.
Later occurrences of stanza,key override previous ones.
With my awk knowledge I pick awk of course:

awk '
BEGIN { FS=OFS=" = " }
/\[/ { st[stanza=$0]; next } 
NF>1 { val[stanza,$1]=$2 }
END {
  for (stanza in st) {
    print stanza
    for (comb in val) {
      split(comb, c, SUBSEP)
      if (stanza==c[1]) print c[2], val[stanza,c[2]]
    }
    print ""
  }
}
' default/savedsearches.conf local/savedsearches.conf

The standard two-dimensional arrays are bit of a hack in awk; you must know that the keys are simply joined with an embedded SUBSEP. And you can only loop through the combined index.
And finally you have no control over the order; the order of the stanzas and keys are random.
Perhaps perl or python are better suited?
Or the true multi-dimensional arrays in gawk 4.x?

1 Like
7

@MadeInGermany, good idea.
here's the "true" multi-dimentional gawk version:

awk '
BEGIN { FS=OFS=" = " }
/]/ { stanza=$0; next }
NF>1 { val[stanza][$1]=$2 }
END {
  PROCINFO["sorted_in"]="@ind_str_asc"
  for (stI in val) {
    print stI
    for (valI in val[stI])
      print valI, val[stI][valI]
    print ""
  }
}' default/savedsearches.conf local/savedsearches.conf
1 Like
10

This topic was automatically closed 300 days after the last reply. New replies are no longer allowed.