Remove pipe(|) symbol ina file, except the ones which are enclosed in double quotes

I have file with are delimited by pipe(|) symbol, I wanted those to be removed except the ones which are enclosed in double quotes.

If your quote file is:
|Life is |Beautiful"|"Indeed life |is beautiful too|"|"But unix is fun| is not"|"

It should return:
Life is Beautiful"|"Indeed life is beautiful too"|"But unix is fun is not "|"

Moderator comments were removed during original forum migration.