Sendmail delay: 3 mins 11 secs... Every time.

Hi all,
I would like some help with a sendmail problem:
We have a new system comprising of 4 T7-1 servers, each hosting 5 LDOMs, all domains running Solaris 11.3
All emails sent from every one of these domains (including the control domains) sit in the queue for 3 mins 11 secs (sometime 3m 12s, or 3m 13s) before being delivered.
I've exhausted my googling skills investigating red herrings and blind alleys and now I'm stumped.
I have included some sanitised output below, showing the problem. I used " sendmail -v " here purely for the verbose output: The same delay occurs with mailx.
Any help would be greatly appreciated.
#########################################################

root@myldom:~# sendmail -v my.name@company.uk < test.txt
my.name@company.uk... Connecting to [127.0.0.1] via relay...
220 myldom.fqdn.uk ESMTP Sendmail 8.15.1+Sun/8.15.1; Thu, 31 Aug 2017 16:33:21 +0100 (BST)
>>> EHLO myldom.fqdn.uk
250-myldom.fqdn.uk Hello localhost.fqdn.uk [127.0.0.1], pleased to meet you
250-ENHANCEDSTATUSCODES
250-PIPELINING
250-EXPN
250-VERB
250-8BITMIME
250-SIZE
250-DSN
250-ETRN
250-DELIVERBY
250 HELP
>>> VERB
250 2.0.0 Verbose mode
>>> MAIL From:<myuser@myldom.fqdn.uk> SIZE=21
250 2.1.0 <myuser@myldom.fqdn.uk>... Sender ok
>>> RCPT To:<my.name@company.uk>
>>> DATA
250 2.1.5 <my.name@company.uk>... Recipient ok
354 Enter mail, end with "." on a line by itself
>>> .

# (Wait 3 mins 11 secs...)

050 <my.name@company.uk>... Connecting to company.uk. via esmtp...
050 <my.name@company.uk>... Connecting to mailhost.fqdn.uk via esmtp...
050 220 exchangeserver.fqdn.uk Microsoft ESMTP MAIL Service ready at Thu, 31 Aug 2017 16:36:31 +0100
050 >>> EHLO myldom.fqdn.uk
050 250-exchangeserver.fqdn.uk Hello [10.10.3.102]
050 250-SIZE 37748736
050 250-PIPELINING
050 250-DSN
050 250-ENHANCEDSTATUSCODES
050 250-STARTTLS
050 250-X-ANONYMOUSTLS
050 250-AUTH NTLM
050 250-X-EXPS GSSAPI NTLM
050 250-8BITMIME
050 250-BINARYMIME
050 250-CHUNKING
050 250 XRDST
050 >>> STARTTLS
050 220 2.0.0 SMTP server ready
050 >>> EHLO myldom.fqdn.uk
050 250-exchangeserver.fqdn.uk Hello [10.10.3.102]
050 250-SIZE 37748736
050 250-PIPELINING
050 250-DSN
050 250-ENHANCEDSTATUSCODES
050 250-AUTH NTLM LOGIN
050 250-X-EXPS GSSAPI NTLM
050 250-8BITMIME
050 250-BINARYMIME
050 250-CHUNKING
050 250 XRDST
050 >>> MAIL From:<myuser@myldom.fqdn.uk> SIZE=382
050 250 2.1.0 Sender OK
050 >>> RCPT To:<my.name@company.uk>
050 >>> DATA
050 250 2.1.5 Recipient OK
050 354 Start mail input; end with <CRLF>.<CRLF>
050 >>> .
050 250 2.6.0 <201708311533.v7VFXK30006527@myldom.fqdn.uk> [InternalId=80298708569789, Hostname=exchangeserver.fqdn.uk] Queued mail for delivery
050 <my.name@company.uk>... Sent (<201708311533.v7VFXK30006527@myldom.fqdn.uk> [InternalId=80298708569789, Hostname=exchangeserver.fqdn.uk] Queued mail for delivery)
250 2.0.0 v7VFXLXx006528 Message accepted for delivery
my.name@company.uk... Sent (v7VFXLXx006528 Message accepted for delivery)
Closing connection to [127.0.0.1]
>>> QUIT
221 2.0.0 myldom.fqdn.uk closing connection
root@myldom:~#

During the delay before sending, I ran the following in another session:

root@myldom:~# mailq;sendmail -v -q
                /var/spool/mqueue (1 request)
-----Q-ID----- --Size-- -----Q-Time----- ------------Sender/Recipient-----------
v7VFXLXx006528*      21 Thu Aug 31 16:33 <myuser@myldom.fqdn.
                                         <my.name@company.uk>
                Total requests: 1

Running /var/spool/mqueue/v7VFXLXx006528 (sequence 1 of 1)
v7VFXLXx006528: locked
root@myldom:~# 

... And this is what appears in the log:

root@myldom:~# grep v7VFXLXx006528 /var/log/syslog
Aug 31 16:33:21 myldom sendmail[6528]: [ID 801593 mail.info] v7VFXLXx006528: from=<myuser@myldom.fqdn.uk>, size=382, class=0, nrcpts=1, msgid=<201708311533.v7VFXK30006527@myldom.fqdn.uk>, proto=ESMTP, daemon=MTA-v4, 

relay=localhost.fqdn.uk [127.0.0.1]
Aug 31 16:36:34 myldom sendmail[6528]: [ID 801593 mail.info] v7VFXLXx006528: to=<my.name@company.uk>, ctladdr=<myuser@myldom.fqdn.uk> (111/100), delay=00:03:13, xdelay=00:03:13, mailer=esmtp, pri=30382, relay=mailhost.fqdn.uk 

[10.10.0.98], dsn=2.0.0, stat=Sent (<201708311533.v7VFXK30006527@myldom.fqdn.uk> [InternalId=80298708569789, Hostname=exchangeserver.fqdn.uk] Queued mail for delivery)
Aug 31 16:36:34 myldom sendmail[6527]: [ID 801593 mail.info] v7VFXK30006527: to=my.name@company.uk, ctladdr=myuser (111/100), delay=00:03:14, xdelay=00:03:13, mailer=relay, pri=30021, relay=[127.0.0.1] [127.0.0.1], dsn=2.0.0, stat=Sent 

(v7VFXLXx006528 Message accepted for delivery)
root@myldom:~#

Many thanks in advance

Welcome to the forum.

With a delay not attributable to an evident cause, I always suspect a DNS timeout, although more than 3 mins seems very long, and I don't see unresolved addresses in the log.

Hi Rudi (Oops, RE: code tags)
We do use DNS internally, but FQDNs are in the local hosts files as well.
The weird thing is, it doesn't even seem to try to send it for over 3 minutes. It's like there's a config item that says "wait this long before sending mail" but I can't find it.
Both sendmail.cf & submit.cf (and their corresponding .mc files) seem to be default unmodified.

Assuming sendmail daemon is active:

the sendmail daemon can be set up with queueing - meaning send out mail every x minutes.

There are .cf files in the mail directory tree in /etc. (you did this) Read though them and see what is set. (queueing control can be set here, too): Also review the command used to start smtp (sendmail), there are usually two sendmail processes.

I would help but I do not have a unix box here, and sendmail is beastly in how it does things. I do not remember.

You may also need to check if the box that has the problem has an active mail server or if the smtp request is forwarded to another box.

This is for solaris 10 - but I think the directories are the same most places.
And can be changed in the .cf files.
Administering the Queue Directories - Managing sendmail Services in Oracle Solaris 11.1

mqueue directory should be in /var/spool tree.

Hi Jim,
Sendmail daemon is active.

You said:

How? Presumably it can also be configured to send mail immediately... How?
I'm afraid the contents of the .cf files are hieroglyphics to me.

I came late to this project and this was already set up when I arrived, but as far as I can tell, there has been no customisation done to sendmail - it's all 'out of the box'. There is next to nothing in the sendmail.mc and submit.mc files, and they and the .cf files are all quite old, so presumably the first build.

That link says:

2. Force queue processing and display the progress of the jobs as the queue is cleared.
# /usr/lib/sendmail -q -v 

When I ran

sendmail -v -q

(I assumed the "q" meant "query") it replied:

v7VFXLXx006528: locked

To me, this seems intentional. But how?

It looks like the email is sent to the localhost over the loopback interface and is accepted, then it sits there waiting for a timer to run out before it gets sent on to the Exchange server and actually delivered.
Thanks

If you're running with 'out of the box' default configuration then in /etc/mail/sendmail.cf the "smart" relay host line probably says:

DSfoo.bar

Try changing that to

DS

That will tell it that its relay host is itself.

---------- Post updated at 04:42 PM ---------- Previous update was at 04:36 PM ----------

Although, of course, there's a delay because the mail daemon wakes up regularly to process mail, I wouldn't have thought that a problem with it would give you the exact same delay every time. So I conclude that this sounds like a configuration issue such as above.

I would suspect some issues in /etc/hosts or /etc/resolv.conf files.
/etc/resolv.conf less likely to be wrong due to being managed by smf in your release.

Check hosts file for syntax errors, post the content here is possible.

I guess the smarthost in /etc/mail/sendmail.cf (and/or submit.cf) is mailhost.fqdn.uk.
Check its DNS resolution with

nslookup -q=mx mailhost.fqdn.uk

Does it list many hosts? Are some of them not reachable on port 25?
What is the hosts line in /etc/nsswitch.conf?

Thanks hicksd8, but that has already been set as suggested:

# grep ^DS sendmail.cf submit.cf
sendmail.cf:DS
submit.cf:DS

---------- Post updated at 13:43 ---------- Previous update was at 10:20 ----------

I've found out something that may be significant, but I'm not sure what to do with it:
"mailhost" is an alias in /etc/hosts for the old mailserver which is still active, but the preferred mailserver is "outlook", which is 3 load-balanced servers:

# nslookup outlook.fqdn.uk
Server:         10.10.0.35
Address:        10.10.0.35#53

Name:   outlook.fqdn.uk
Address: 10.10.3.47
Name:   outlook.fqdn.uk
Address: 10.10.3.48
Name:   outlook.fqdn.uk
Address: 10.10.0.171

"SVP-MAIL01" is one of those load-balanced mailservers, which mail gets forwarded to from the old mailhost in the syslog snippet below:

Sep  4 11:11:56 myldom sendmail[28520]: [ID 801593 mail.info] v84ABuq7028520: from=myuser, size=108, class=0, nrcpts=1, msgid=<201709041011.v84ABuq7028520@myldom.fqdn.uk>, relay=myuser@localhost
Sep  4 11:11:56 myldom sendmail[28521]: [ID 801593 mail.info] v84ABubQ028521: from=<myuser@myldom.fqdn.uk>, size=450, class=0, nrcpts=1, msgid=<201709041011.v84ABuq7028520@myldom.fqdn.uk>, proto=ESMTP, daemon=MTA-v4, relay=localhost.fqdn.uk [127.0.0.1]
Sep  4 11:11:56 myldom sendmail[28520]: [ID 801593 mail.info] v84ABuq7028520: to=my.name@company.uk, ctladdr=myuser (111/100), delay=00:00:00, xdelay=00:00:00, mailer=relay, pri=30108, relay=[127.0.0.1] [127.0.0.1], dsn=2.0.0, stat=Sent (v84ABubQ028521 Message accepted for delivery)
Sep  4 11:15:07 myldom sendmail[28523]: [ID 702911 mail.info] STARTTLS=client, relay=mailhost.fqdn.uk, version=TLSv1.2, verify=FAIL, cipher=ECDHE-RSA-AES256-SHA384, bits=256/256
Sep  4 11:15:07 myldom sendmail[28523]: [ID 801593 mail.info] v84ABubQ028521: to=<my.name@company.uk>, ctladdr=<myuser@myldom.fqdn.uk> (111/100), delay=00:03:11, xdelay=00:03:11, mailer=esmtp, pri=120450, relay=mailhost.fqdn.uk [10.10.0.98], dsn=2.0.0, stat=Sent (<201709041011.v84ABuq7028520@myldom.fqdn.uk> [InternalId=80698140526752, Hostname=SVP-MAIL01.fqdn.uk] Queued mail for delivery)

Obviously I can't put them all in /etc/hosts with the same address, but when I took the "mailhost" line out, it failed to find any mailserver at all.
How do I tell sendmail that the mailhost is "outlook.fqdn.uk" without putting only one address in /etc/hosts?

Many thanks

---------- Post updated at 13:45 ---------- Previous update was at 13:43 ----------

P.S.

# grep hosts /etc/nsswitch.conf
hosts:  files dns

---------- Post updated at 13:46 ---------- Previous update was at 13:45 ----------

PPS: I'm not sure if this is what you meant, but...

---------- Post updated at 16:07 ---------- Previous update was at 13:46 ----------

OK, Having a better idea of what I'm looking for now, I found the solution - it's just the implementation I'm unsure of now.

I put the correct mailhost name in sendmail.cf like so:

DSoutlook.fqdn.uk

then restarted sendmail and emails get delivered in just a few seconds.

My last question on this is...
I need to add it to sendmail.cf via editing sendmail.mc and running make on it, but what's the syntax for sendmail.mc?
Do I just add the DS line as I want it to appear in sendmail.cf as it is above?

Many thanks

With the smarthost sendmail does an MX lookup first, like

nslookup -q=mx outlook.fqdn.uk

If there is no MX it falls back to gethostbyname lookup, like

getent hosts outlook.fqdn.uk

The second goes to /etc/inet/hosts first (ensure that /etc/hosts and /etc/inet/ipnodes are symlinks to it!)
I have told you to try

nslookup -q=mx mailhost.fqdn.uk

because mailhost is a more standard name for an MX, and MX (mail exchanger records) is the standard for SMTP mailers (is a list of mail servers with a weight for true load balancing). I wonder why you ignore the MX and try to emulate load balancing with a number of IP addresses.
If you want to try sendmail.mc (which is the correct way according to sendmail.org but might be more hassle than a direct change in sendmail.cf), there is a README somewhere in /usr/lib/mail/
and there is also an article in docs.oracle.com.

Thanks MadeinGermany,
I have fixed it by editing sendmail.mc and making a new sendmail.cf : Just add the DS line as you want it to appear in sendmail.cf
I didn't set this up (I'm fairly new here) and since the Exchange server is a Windows server, it's looked after by the Windows team.
I couldn't put multiple entries in /etc/hosts for outlook.fqdn.uk with multiple IP addresses, which is why I needed another way to tell sendmail where the mail server was.

# nslookup -q=mx mailhost.fqdn.uk
Server:         10.10.0.35
Address:        10.10.0.35#53

*** Can't find mailhost.fqdn.uk: No answer

# nslookup -q=mx outlook.fqdn.uk
Server:         10.10.0.35
Address:        10.10.0.35#53

*** Can't find outlook.fqdn.uk: No answer

I found the procedure on an oracle docs webpage (I'm not allowed to post URLs ) but I wouldn't have found it without the hints and tips in this thread.
Thanks to all who contributed.