sed substitue whole file + 1 substitue with variables on one sed line?.

I'm trying to remove '--X' from the whole file and using variables replace $oldvar with $newvar.

I have tried with double quotes but it doesn't seem to work. $newvar is set to /usr/bin/bash. Would appreciate some guidance.

newvar=$(which bash)
oldvar=/bin/bash
  
sed "s/^--X//;1s#!$oldvar#!$newvar#" <infile >outfile

Input is:

echo $newvar
$ /usr/bin/bash

cat infile
--X#!/bin/bash

Result is:

#!/bin/bash

Escape the \! or better yet, remove them.