Bash sed search and replace question

I have several files that I need to modify using sed. I know how to do that, but now a new requirement has come up.

Now, I need to make changes to all lines that don't start with certain strings. For example, I need to change all lines except for lines that start with "a", "hello there", "hows_it_going" or "b".

Does sed have some sort of syntax that means "don't change the line under these conditions"?

to exchange all "X" by "Y" in the lines not beginning with "a".

sed '/^a/! s/X/Y/g' file.txt

Can anyone post alternative sed command to do the same thing?
Actually replacing this "/^a/!"

Thanks Elbrand! You rock!

Cola, if I understand you correctly it would be:

sed 's/^a/b/g' file.txt

That would replace all a's at the beginning of a line with a b.