Sendmail questions, SCO 5.0.6 sendmail 8.11.0

I am running SCO 5.0.6 and using sendmail 8.11.0 and having issues with smtp authentication. When trying to send mail the following message will kick back.
(reason: 530 5.7.1 Authentication required)
530 5.7.1 Authentication required

Not sure what needs to be tweeked in sendmail.cf but I suspect some settings need to be enabled for smtp authentication.

Sounds more like the message is coming from the smart relay host. Some outbound server's require that you provide authentication by receiving inbound mail before accepting outbound.
If you remove the smart relay, some mail will go, but you will need a reverse dns entry to eliminate bounce.