SMTP Flooding - How can I stop it?

Please someone help! My SMTP server has been under attack for days and I'm at my wits end. I'm by no means a UNIX security expert but I've been reading and researching for days to no avail. I'm getting unauthorized mail from external servers being relayed or redirected through my SMTP server. The IP addresses change on almost every connection. I can post log excerpts and config file info if necessary. I just don't know what would be relevant. This is a virtual dedicated server from godaddy.com running RedHat Fedora and PLESK admin control panels. I really need help so anyone who might have any ideas or suggestions, please respond! Thanks everyone.

Which MTA are you running (eg sendmail/postfix/other )?

I ran a top to see running processes and it looks like qmail. Though I thought qmail used sendmail somehow. Could be wrong though.... I'm okay with UNIX but only mostly for web server things. (Apache, MySQL, PHP...) I'm new to security and mail server testing/repair/hardening so I'm at a standstill. I made a few changes to the sendmail.cf file and one of the qmail files to throttle the connection rate and to pause 5 seconds before returning GREET. After switching Port 25 back on today the relays are no longer being passed on (I think...). My SMTP server is still being bombarded with connection requests so I'm not sure if I've completely resolved this or if I need to look elsewhere for connection restrictions. The connection attempt rate has, at times, approached 1.07 connections per second!

Do you have this directory?

/var/qmail/control

Yes I do. I've been in here before. I believe it was to add to rcpthosts when I first got this server online.

Ok, and that confirms qmail. What do you have in rcphosts, ie. is there anything other than your domain?

Below is the content of my rcpthosts file. All of the domains except gmail, yahoo, and hotmail are hosted on this server as virtual hosts and each has its own mail domain. Do you think I should remove the non-hosted domains from this file?

nestlincoln.com

kingsleymd.com

high5club.org
pkflynn.com

yes, currently you will relay mail to hotmail, gmail and yahoo accounts.

You should also consider something like spamdyke: A drop-in connection-time spam filter for qmail and/or using iptables to drop packets from the sources of the relay abuse.

Yeah...I've been reading about iptables and my programmer just told be about spamdyke. I'm still trying to get a handle on iptables but I'm sure I'll figure it out. At least I stopped the SPAM from being relayed....for now. Thanks for your help. It is much appreciated.

the best way to relieve smtp attacks are spamfilters, disabling smtp relay if it's enabled, or front it with a mail proxy type server and let it determine what's good and what's not. That will definitely take some strain off of the server. If you're trying to stop it from getting to your perimeter completely... good luck with that. Oh, and IPS with good smtp rules might also help.