Error communicating with target host

Май
15

Read these next…

• 
  

  Spark! Pro series — 9th February 2023

  Spiceworks Originals

  Today in History: America meets the Beatles on “The Ed Sullivan Show”

  At approximately 8:12 p.m. Eastern time, Sunday, February 9, 1964, The Ed Sullivan Show returned from a commercial (for Anacin pain reliever), and there was Ed Sullivan standing …

• 
  

  Green Brand Rep Wrap-Up: January 2023

  Spiceworks Originals

  Hi, y’all — Chad here. A while back, we used to feature the top posts from our brand reps (aka “Green Gals/Guys/et. al.) in a weekly or monthly wrap-up post. I can’t specifically recall which, as that was approximately eleven timelines ago. Luckily, our t…

• 
  

  Help with domain controller setup

  Windows

  I just got a new job as the only IT person for a business with around 270 employees (I would say probably less than half use computers) They don’t have any policies or procedures when it comes to IT, as they have never had an IT person. My background cons…

• 
  

  Malicious URLs

  Security

  We have firewall, we have endpoint protection, we have Safe links and Attachments for Office 365 (Microsoft Defense for Office 365 Plan 1), and still receiving links that lead to malicious web sites.It seems like security companies still didn’t develop a …

• 
  

  Snap! — Old Batteries, Lovable Bots, Quantum Breakthrough, Should We Trust AI?

  Spiceworks Originals

  Your daily dose of tech news, in brief.

  Welcome to the Snap!

  Flashback: February 8, 1996: The massive Internet collaboration “24 Hours in Cyberspace” takes place (Read more HERE.)

  Bonus Flashback: February 8, 1974: Americans end outer spa…

This article is contributed. See the original author and article here.

By now you are hopefully aware of the TLS 1.0/1.1 deprecation efforts that are underway across the industry and Microsoft 365 in particular. Head out to our documentation for more details and references if you need a refresher! Also check out this blog entry to see how you can use reporting in Exchange Online to get an overview about the TLS versions used by mails submitted to your tenant. This topic may be super-relevant to you, because as confirmed by the message center post MC229914, TLS 1.0 and TLS 1.1 deprecation started enforcing for Exchange Online mail flow endpoints beginning January 11th, 2021. The rollout will continue over the following weeks and months. This essentially means, soon this deprecation process will be over, and we will no longer accept TLS 1.0 and TLS 1.1 email connections from external sources. Also note that Exchange Online will never use TLS 1.0 or 1.1 to send outbound email.

We wanted to talk about what this means for SMTP traffic destined to Exchange Online in particular. What happens if a server on your side can only use TLS 1.0 with SMTP? Will sending fail, and if yes, how do you notice TLS 1.0 or TLS 1.1 is the root cause of your email problems? There are different variables that impact this and we will try to mention most frequent scenarios.

Before diving into further details, keep in mind that generally speaking, the TLS implementation in Exchange on-premises or Exchange Online is done opportunistically. This means:

• For receiving mail into Exchange: If the sending server does not support TLS, or if the TLS negotiation fails, Exchange Online will still accept messages unencrypted and without TLS (provided the sending server’s configuration allows that).
• For sending mail from Exchange: For outbound email, if the receiving server does not support TLS (does not advertise the STARTTLS Verb), Exchange on-premises and Exchange Online will send email without TLS (provided TLS is not forced on the send connector or outbound connector).

Another point to keep in mind is that Exchange will always attempt to initially negotiate the highest possible version of TLS which is enabled on the other server. Once this version is selected during the TLS handshake – Exchange does not attempt a lower version of TLS/SSL that might also be enabled on the server. In case there is a failure during communication, Exchange will instead re-attempt the delivery without TLS. Our previously published 3 part blog posts (Exchange Server TLS guidance part 1, Part 2 and Part 3) extensively covered how various components like Schannel, WinHTTP, .Net, etc. work together to decide the version of TLS Exchange server should use during TLS handshakes.

Other than TLS versions, another factor that we tend to overlook are the Cipher Suites supported by Office 365. While the servers or devices may use TLS 1.2, not supporting one of the ciphers suites adopted by Office 365 from the published list could also cause mail flow issues.

Let us look at the details of each scenario!

3rd party SMTP server sending to Exchange Online

The experience here will mostly depend on the sending server’s implementation. In most cases, there should be no impact. Once the TLS 1.0 attempt fails, the sender should fall back to not using TLS at all and send in an unencrypted manner. If the sender is relying solely on TLS 1.0 or TLS 1.1 and cannot send unencrypted, it is again up to the sending server’s implementation on what happens – the mail might remain queued while the sender keeps retrying. Ultimately the sending server should generate an error or an NDR after the message expiration timeout.

Exchange server (external to the organization) sending to Exchange Online

This applies to the case where your Exchange servers in contoso.com would be sending to a different organization, let’s say fabrikam.com, which is hosted in Exchange Online. For most organizations, mail flow will not break. This is because send connectors in Exchange are by default created with the setting “RequireTLS: false”, meaning they will attempt a TLS connection if the remote party supports it, but if TLS negotiation fails, they will simply fall back to not using TLS and will send anyway. The SMTP Send protocol logs will contain entries that resemble the following:

You will see that initially the mail could not be sent to Office 365 and it failed with error: TLS negotiation failed with error SocketError

#Fields: date-time,connector-id,session-id,sequence-number,local-endpoint,remote-endpoint,event,data,context
2021-01-11T16:43:14.811Z,Connector2Fabrikam,08D8B64FC6449F2A,0,,10.1.0.16:25,*,SendRoutingHeaders,Set Session Permissions
2021-01-11T16:43:14.811Z,Connector2Fabrikam,08D8B64FC6449F2A,1,,10.1.0.16:25,*,,attempting to connect
2021-01-11T16:43:14.817Z,Connector2Fabrikam,08D8B64FC6449F2A,2,10.0.0.16:6933,10.1.0.16:25,+,,
2021-01-11T16:43:14.969Z,Connector2Fabrikam,08D8B64FC6449F2A,3,10.0.0.16:6933,10.1.0.16:25,<,”220 BN3USG02FT012.mail.protection.office365.us Microsoft ESMTP MAIL Service ready at Mon, 11 Jan 2021 17:43:14 +0100″,
2021-01-11T16:43:14.969Z,Connector2Fabrikam,08D8B64FC6449F2A,4,10.0.0.16:6933,10.1.0.16:25,>,EHLO exc16.contoso.com,
2021-01-11T16:43:15.012Z,Connector2Fabrikam,08D8B64FC6449F2A,5,10.0.0.16:6933,10.1.0.16:25,<,250 BN3USG02FT012.mail.protection.office365.us Hello [10.0.0.16] SIZE 37748736 PIPELINING DSN ENHANCEDSTATUSCODES STARTTLS X-ANONYMOUSTLS AUTH NTLM X-EXPS GSSAPI NTLM 8BITMIME BINARYMIME CHUNKING XRDST,
2021-01-11T16:43:15.013Z,Connector2Fabrikam,08D8B64FC6449F2A,6,10.0.0.16:6933,10.1.0.16:25,>,STARTTLS,
2021-01-11T16:43:15.016Z,Connector2Fabrikam,08D8B64FC6449F2A,7,10.0.0.16:6933,10.1.0.16:25,<,220 2.0.0 SMTP server ready,
2021-01-11T16:43:15.016Z,Connector2Fabrikam,08D8B64FC6449F2A,8,10.0.0.16:6933,10.1.0.16:25,*,” CN=mail.contoso.com CN=R3, O=Let’s Encrypt, C=US 03C6CCE6D57C1D2DA908BF69EBD10963AE74 AF15A9798388DD9C0C03FEBC897025CD76963178 2020-12-05T09:46:36.000Z 2021-03-05T09:46:36.000Z mail.contoso.com;autodiscover.contoso.com;”,Sending certificate Subject Issuer name Serial number Thumbprint Not before Not after Subject alternate names
2021-01-11T16:43:15.043Z,Connector2Fabrikam,08D8B64FC6449F2A,9,10.0.0.16:6933,10.1.0.16:25,*,,TLS negotiation failed with error SocketError
2021-01-11T16:43:15.043Z,Connector2Fabrikam,08D8B64FC6449F2A,10,10.0.0.16:6933,10.1.0.16:25,-,,Remote

A network capture will resemble the following, which clearly explains the reason behind the failure. As you see in the following screenshot, the sending server, after the exchange of STARTTLS verb, tried to negotiate transport layer security using TLS version 1.1. The Exchange Online server instantly disconnected the session with a “FINISH” flag (FIN):

However, immediately after that, the sending server should fall back to not using TLS and will send the email anyway and it will be accepted by Exchange Online:

2021-01-11T16:43:15.047Z,Connector2Fabrikam,08D8B64FC6449F2B,0,,10.1.0.16:25,*,SendRoutingHeaders,Set Session Permissions
2021-01-11T16:43:15.047Z,Connector2Fabrikam,08D8B64FC6449F2B,1,,10.1.0.16:25,*,,attempting to connect
2021-01-11T16:43:15.050Z,Connector2Fabrikam,08D8B64FC6449F2B,2,10.0.0.16:6934,10.1.0.16:25,+,,
2021-01-11T16:43:15.053Z,Connector2Fabrikam,08D8B64FC6449F2B,3,10.0.0.16:6934,10.1.0.16:25,<,”220 BN3USG02FT012.mail.protection.office365.us Microsoft ESMTP MAIL Service ready at Mon, 11 Jan 2021 17:43:14 +0100″,
2021-01-11T16:43:15.053Z,Connector2Fabrikam,08D8B64FC6449F2B,4,10.0.0.16:6934,10.1.0.16:25,>,EHLO exc16.contoso.com,
2021-01-11T16:43:15.055Z,Connector2Fabrikam,08D8B64FC6449F2B,5,10.0.0.16:6934,10.1.0.16:25,<,250 BN3USG02FT012.mail.protection.office365.us Hello [10.0.0.16] SIZE 37748736 PIPELINING DSN ENHANCEDSTATUSCODES STARTTLS X-ANONYMOUSTLS AUTH NTLM X-EXPS GSSAPI NTLM 8BITMIME BINARYMIME CHUNKING XRDST,
2021-01-11T16:43:15.058Z,Connector2Fabrikam,08D8B64FC6449F2B,6,10.0.0.16:6934,10.1.0.16:25,*,,sending message with RecordId 40900973559810 and InternetMessageId <5149fa60b89741cfaf6e05d5767776a9@contoso.com>
2021-01-11T16:43:15.059Z,Connector2Fabrikam,08D8B64FC6449F2B,7,10.0.0.16:6934,10.1.0.16:25,>,MAIL FROM:<user@contoso.com> SIZE=9031,
2021-01-11T16:43:15.059Z,Connector2Fabrikam,08D8B64FC6449F2B,8,10.0.0.16:6934,10.1.0.16:25,>,RCPT TO:<user@fabrikam.com>,
2021-01-11T16:43:15.118Z,Connector2Fabrikam,08D8B64FC6449F2B,9,10.0.0.16:6934,10.1.0.16:25,<,250 2.1.0 Sender OK,
2021-01-11T16:43:15.120Z,Connector2Fabrikam,08D8B64FC6449F2B,10,10.0.0.16:6934,10.1.0.16:25,<,250 2.1.5 Recipient OK,
2021-01-11T16:43:15.121Z,Connector2Fabrikam,08D8B64FC6449F2B,11,10.0.0.16:6934,10.1.0.16:25,>,BDAT 2932 LAST,
2021-01-11T16:43:18.300Z,Connector2Fabrikam,08D8B64FC6449F2B,12,10.0.0.16:6934,10.1.0.16:25,<,”250 2.6.0 <5149fa60b89741cfaf6e05d5767776a9@contoso.com> [InternalId=171798691842, Hostname=BN3USG02FT012.mail.protection.office365.us] 4228 bytes in 2.816, 1.466 KB/sec Queued mail for delivery”,
2021-01-11T16:43:18.314Z,Connector2Fabrikam,08D8B64FC6449F2B,13,10.0.0.16:6934,10.1.0.16:25,>,QUIT,
2021-01-11T16:43:18.316Z,Connector2Fabrikam,08D8B64FC6449F2B,14,10.0.0.16:6934,10.1.0.16:25,<,221 2.0.0 Service closing transmission channel,
2021-01-11T16:43:18.316Z,Connector2Fabrikam,08D8B64FC6449F2B,15,10.0.0.16:6934,10.1.0.16:25,-,,Local

Note: to see where the SMTP Send protocol logs are stored on your on-premises server, run “Get-TransportServer <servername> | fl SendProtocolLog*”. Logs will be generated once you enable logging with a cmdlet like “Set-SendConnector <connectorname> -ProtocolLoggingLevel Verbose”.

If you explicitly configured your send connector with the setting “RequireTLS: True”, the fallback to non-TLS will not happen. In this case, the behavior will be similar to what is described in the next section.

On-premises Exchange server in a hybrid configuration sending to Exchange Online (internal to the organization)

In this scenario, mails are sent from your on-premises recipients to your Exchange Online recipients. When your Exchange servers are configured for hybrid, by default, the “Outbound to Office 365…” connector has “RequireTLS: True”. This means that on-premises servers won’t fall back to sending unencrypted. If the TLS 1.0/1.1 attempt fails, Exchange will keep retrying the connection using TLS several times at various intervals (the exact retry intervals and counts are described here.) The send protocol log entries will be similar to those shown above, with the difference that the “TLS negotiation failed with error SocketError” entries will just keep repeating, since there is no fallback. Unless you modified the default retry configuration, the on-premises Exchange server will keep retrying for 2 days. Throughout this time, the affected mails will stay in the queue. The queue details will look similar to this:

[PS] C:>Get-Queue <queue ID> | fl
(…)
Status : Retry
LastError : [{LED=451 4.4.397 Error communicating with target host. -> 421 4.4.2 Connection dropped due to SocketError};{MSG=};{FQDN=<servername>};{IP=<serverIP>};{LRT=1/11/2021 6:02:39 PM}]
(…)

By default, the sender will receive a delay DSN (the subject starts with “Delivery delayed”, localized) after 4 hours. Unless you do some manual intervention sooner, the sending Exchange server will normally give up after 2 days and generate an NDR. The NDR message would look like this:

Delivery has failed to these recipients or groups:
user@contoso.com
Several attempts to deliver your message were unsuccessful and we stopped trying. It could be a temporary situation. Try to send your message again later.
Diagnostic information for administrators:
Generating server: <servername>
Receiving server: <servername>
user@contoso.com
1/7/2021 7:24:14 PM – Server at <servername> returned ‘550 5.4.300 Message expired -> 451 4.4.397 Error communicating with target host. -> 421 4.4.2 Connection dropped due to SocketError’
1/7/2021 7:23:14 PM – Server at mail.contoso.com (10.0.0.16) returned ‘451 4.4.397 Error communicating with target host. -> 421 4.4.2 Connection dropped due to SocketError’

To avoid such problems, be sure to configure your on-premises Exchange servers to support TLS 1.2, as described in our three-part blog series starting here.

Exchange Online sending to Exchange server (external to the organization)

This experience will depend on how the receiving server has implemented inbound mail flow. Assuming the receiving server supports TLS (advertises STARTTLS Verb), Exchange Online will only use TLS 1.2 to send outbound email. If the receiving server does not support TLS 1.2, Exchange Online being opportunistic will try to send email without TLS. If the receiving mail server does not have TLS enforced for inbound email flow, the email will be sent without TLS. You will know if your server is enforcing TLS by querying for the RequireTLS property of the Receive Connector, e.g. ‘Get-ReceiveConnector “Default Frontend <ServerName>” | fl RequireTLS’. If TLS is enforced at the receiving end, Exchange Online will continue retrying and the email will remain queued, and eventually we will generate NDR message after 24 hours (which is default message expiration timeout for Exchange Online).

On-premises non-Exchange server, application or device relaying external emails through your Exchange Online tenant following this article

If you have an on-premises non-Exchange server, application or device that relays email through your Office 365 tenant either by SMTP AUTH client submission or by using a certificate based inbound connector, make sure these servers or devices or applications support TLS 1.2. If they do not support TLS 1.2, the TLS negotiation will fail, and a subsequent non-TLS retry might be attempted. SMTP AUTH client submission does not work without TLS. And in case relay is configured through a certificate based inbound connector, the common name (CN) or subject alternative name (SAN) verification will fail during non-TLS communication. This will cause an “550 5.7.0. Relay Access Denied” error in both scenarios. Email delivery to mailboxes hosted in your Office 365 tenant will continue to work albeit it will be treated as “anonymous” submission.

Hopefully, this clarifies what you need to look for in case mail flow starts to break with the disablement of TLS 1.0/1.1! We also want to take a moment to thank Mike Brown, Nino Bilic and Sean Stevenson for their contributions and review.

Szabolcs Vajda and Arindam Thokder

Brought to you by Dr. Ware, Microsoft Office 365 Silver Partner, Charleston SC.

Errors discussed here:

421 4.2.1 Unable to connect

451 4.4.397 Error communicating with target host

451 4.7.0 Temporary server error

Send-MailMessage : Mailbox unavailable. The server response was: 5.7.54 SMTP; Unable to relay recipient in non-accepted domain

Send-MailMessage : Error in processing. The server response was: 4.7.0 Temporary server error. Please try again later. PRX5

Send-MailMessage : Error in processing. The server response was: 4.7.0 Temporary server error. Please try again later. PRX2

421 4.2.1 Unable to connect -> SocketTimedout: Socket error code 10060

451 4.4.397 Error communicating with target host. -> 421 4.2.1 Unable to connect -> SocketTimedout: Socket error code 10060

Message or connection acked with status Retry and response 451 4.4.397 Error communicating with target host. -> 421 4.2.1 Unable to connect -> SocketTimedout: Socket error code 10060

Message or connection acked with status Fail and response 554 5.4.4 SMTPSEND.DNS.NonExistentDomain; nonexistent domain internalproxy -> DnsDomainDoesNotExist: InfoDomainNonexistent

Most of the times once we have the hybrid installed and migrated all the mailboxes to office 365. We are ready to decom the legacy server and use the 2016 for administrative purpose. But when we try to decom the legacy server we might end up with multiple issues. In the below article I have explained one of the wired mail loop issue where all external emails and emails from Line of Business(LOB) applications were looping through the legacy server and not allowing us to decom the legacy server.

Legend:

MX        –         Third party software listing to the inbound mails and passing it to hybrid server

Hybrid        –          Exchange server 2016  (All roles)

Legacy server        –     Exchange server 2013 (All roles)

Office 365        –         Office 365(that was straight forward)

Inbound from external to Office 365 mailbox

                        MX –> Hybrid –> Legacy server –> Hybrid –> Office 365

Outbound email from LOB application in on-prem

                        LOB Application –>  Hybrid –> Legacy server –> Hybrid –> Office 365/External

Things that are clear from the above.

1. Hybrid is able to send emails to Office 365. Mailflow is working
2. Sent an email from on-prem mailbox to Office 365 mailbox and found that “X-MS-Exchange-CrossTenant-AuthAs”  was showing as Internal. So the send connector in Office 365 and receive connector in Office 365 seems to be working as expected.

Use either of the below to Analyze the header to find out the mailflow behavior

https://mha.azurewebsites.net/

https://mxtoolbox.com/EmailHeaders.aspx

Wanted to do more tracing and did not want to modify the existing connector or settings. So planned to

1. Take a member server to send SMTP email from that server to Office 365 mailbox  and external
2. Create a brand new receive connector to receive emails specific to that member server IP and enable logging
3. Analyze the protocol logs

Created a Brand new receive connector on the hybrid server as shown below.

Frontend transport connector not hub transport connector

As shown above enable Verbose logging and make sure to add only the member server IP in “Remote network settings”  and leave Bindings to All and server name in the FQDN field.

Now we are ready to reproduce the issue and collect logs.

The logs are present in the below location

%ExchangeInstallPath%TransportRolesLogsFrontEndProtocolLogSmtpReceive

Used the below command on the memberserver to send email from powershell

Send-MailMessage -From <On-premuser> -To <External email> -Subject “Test” -SmtpServer <Hybrid Server FQDN>

Here on-prem user does not need to be an actual mailbox just the sending user’s domain part must be your domain.

Also I am not specifying the port or Use SSL or credentials. So it will use the specific connector that we have created for port 25.

If we get the below error its we have not configured the receive connector to relay out external emails

Send-MailMessage : Mailbox unavailable. The server response was: 5.7.54 SMTP; Unable to relay recipient in non-accepted domain

You would need to follow the below article to allow relay for external users.

Allow anonymous relay on Exchange servers

From <https://docs.microsoft.com/en-us/exchange/mail-flow/connectors/allow-anonymous-relay?view=exchserver-2019>

There are 2 methods to do this, I ran the below commands as it just grants the minimum required permissions to allow anonymous relay.

Set-ReceiveConnector “Anonymous Relay” -PermissionGroups AnonymousUsers

Get-ReceiveConnector “Anonymous Relay” | Add-ADPermission -User “NT AUTHORITYANONYMOUS LOGON” -ExtendedRights “Ms-Exch-SMTP-Accept-Any-Recipient”

NOTE:
if we store the cred of the on-prem user in $cred and run the below command it works as it was not using the frontend connector that we created. Rather it was using the default hub connector and other connector on the server.

Since the auth and permission on the connectors are different it was able to route emails

Send-MailMessage -From <On-premuser> -To <External email> -Subject “Test!” -SmtpServer <Hybrid Server FQDN> -Credential $credential -port 2525

Send-MailMessage -From <On-premuser> -To <External email>  -Subject “Test !” -SmtpServer <Hybrid Server FQDN> -Credential $credential -port 465

Post that checked and it was able to relay to external but it was looping through the legacy server.

In the logs we see the below

10,*,,Setting up client proxy session to destination(s): legacy server;hybridserver

11,*,,Setting up client proxy session failed with error: 421 4.2.1 Unable to connect -> SocketTimedout: Socket error code 10060

12,*,,Setting up client proxy session failed with error: 451 4.4.397 Error communicating with target host. -> 421 4.2.1 Unable to connect -> SocketTimedout: Socket error code 10060

13,*,SMTPSubmit SMTPAcceptAnySender SMTPAcceptAuthoritativeDomainSender AcceptRoutingHeaders,Set Session Permissions

14,>,451 4.7.0 Temporary server error. Please try again later. PRX5 ,

..

..

Finally it says Hostname: legacy server, Queued for delivery

So though both legacy server; hybrid server names were retrieved. We got error connecting to one of the server and finally it gives the message to legacy server. Which means Hybrid frontend is not able to connect to itself for hub service(More on that below)

Stopped the Frontend transport service and transport service on the legacy server and tried sending again to see if it force connects when the legacy server is not available.

In the powershell it gave the below error

Sending to Office 365 user

Send-MailMessage -From <On-premuser> -To <office 365 user> -Subject “Test” -SmtpServer <Hybrid Server FQDN>

Send-MailMessage : Error in processing. The server response was: 4.7.0 Temporary server error. Please try again later. PRX5

Sending to external we got the below error

Send-MailMessage -From <On-premuser> -To <External email> -Subject “Test” -SmtpServer <Hybrid Server FQDN>

Send-MailMessage : Error in processing. The server response was: 4.7.0 Temporary server error. Please try again later. PRX2

Internal LOB application to Office 365 mailflow stopped.

External to internal/office 365 email flow stopped. I was not able to find the messages in queue because emails were not getting submitted from MX end point to hybrid.

Though front end transport is active, it was not able to connect to its own hub connector for some reasons(we would find out soon) and legacy hub connector was down. Since front end does not queue messages locally, there was no mails in queue.

• “Front End Transport service on Mailbox servers: This service acts as a stateless proxy for all inbound and (optionally) outbound external SMTP traffic for the Exchange Server organization. The Front End Transport service doesn’t inspect message content, doesn’t communicate with the Mailbox Transport service, and doesn’t queue any messages locally.

• Transport service on Mailbox servers: This service is virtually identical to the Hub Transport server role in Exchange Server 2010. The Transport service handles all SMTP mail flow for the organization, performs message categorization, and performs message content inspection”

From <https://docs.microsoft.com/en-us/exchange/mail-flow/mail-flow?view=exchserver-2019>

<Timestamp>,<server><Receive connector name>,<Sessionguid1>,1,<Hybrid server IP>:25,<member server IP>:<outboundport1>,>,”220 <Hybrid server FQDN> Microsoft ESMTP MAIL Service ready at Mon, <DATE> 15:27:28 -0700″,

<Timestamp>,<server><Receive connector name>,<Sessionguid1>,2,<Hybrid server IP>:25,<member server IP>:<outboundport1>,<,EHLO <member server name>,

<Timestamp>,<server><Receive connector name>,<Sessionguid1>,3,<Hybrid server IP>:25,<member server IP>:<outboundport1>,>,250 <Hybrid server FQDN> Hello [<member server>] SIZE 37748736 PIPELINING DSN ENHANCEDSTATUSCODES STARTTLS 8BITMIME BINARYMIME CHUNKING,

<Timestamp>,<server><Receive connector name>,<Sessionguid1>,4,<Hybrid server IP>:25,<member server IP>:<outboundport1>,<,MAIL FROM:<<From user>>,

<Timestamp>,<server><Receive connector name>,<Sessionguid1>,5,<Hybrid server IP>:25,<member server IP>:<outboundport1>,*,<Sessionguid1>;2021-07-19T22:27:28.513Z;1,receiving message

<Timestamp>,<server><Receive connector name>,<Sessionguid1>,6,<Hybrid server IP>:25,<member server IP>:<outboundport1>,>,250 2.1.0 Sender OK,

<Timestamp>,<server><Receive connector name>,<Sessionguid1>,7,<Hybrid server IP>:25,<member server IP>:<outboundport1>,<,RCPT TO:<<Office 365 userr>>,

<Timestamp>,<server><Receive connector name>,<Sessionguid1>,8,<Hybrid server IP>:25,<member server IP>:<outboundport1>,>,250 2.1.5 Recipient OK,

<Timestamp>,<server><Receive connector name>,<Sessionguid1>,9,<Hybrid server IP>:25,<member server IP>:<outboundport1>,<,DATA,

<Timestamp>,<server><Receive connector name>,<Sessionguid1>,10,<Hybrid server IP>:25,<member server IP>:<outboundport1>,>,354 Start mail input; end with <CRLF>.<CRLF>,

<Timestamp>,<server><Receive connector name>,<Sessionguid1>,11,<Hybrid server IP>:25,<member server IP>:<outboundport1>,*,,Proxy destination(s) obtained from OnProxyInboundMessage event. Correlation Id:368b8452-81cf-48d5-b32d-a728fgdfgd3d7

<Timestamp>,<server><Receive connector name>,<Sessionguid1>,12,<Hybrid server IP>:25,<member server IP>:<outboundport1>,*,,Message or connection acked with status Retry and response 451 4.4.397 Error communicating with target host. -> 421 4.2.1 Unable to connect -> SocketTimedout: Socket error code 10060

<Timestamp>,<server><Receive connector name>,<Sessionguid1>,13,<Hybrid server IP>:25,<member server IP>:<outboundport1>,>,451 4.7.0 Temporary server error. Please try again later. PRX5 ,

<Timestamp>,<server><Receive connector name>,<Sessionguid1>,14,<Hybrid server IP>:25,<member server IP>:<outboundport1>,-,,Remote(SocketError)

<Timestamp2>,<server><Receive connector name>,<Sessionguid2>,0,<Hybrid server IP>:25,<member server IP>:<outboundport>,+,,

<Timestamp2>,<server><Receive connector name>,<Sessionguid2>,1,<Hybrid server IP>:25,<member server IP>:<outboundport>,>,”220 <Hybrid server FQDN> Microsoft ESMTP MAIL Service ready at Mon, <DATE> 15:27:48 -0700″,

<Timestamp2>,<server><Receive connector name>,<Sessionguid2>,2,<Hybrid server IP>:25,<member server IP>:<outboundport>,<,EHLO <member server name>,

<Timestamp2>,<server><Receive connector name>,<Sessionguid2>,3,<Hybrid server IP>:25,<member server IP>:<outboundport>,>,250 <Hybrid server FQDN> Hello [<member server>] SIZE 37748736 PIPELINING DSN ENHANCEDSTATUSCODES STARTTLS 8BITMIME BINARYMIME CHUNKING,

<Timestamp2>,<server><Receive connector name>,<Sessionguid2>,4,<Hybrid server IP>:25,<member server IP>:<outboundport>,<,MAIL FROM:<<From user>>,

<Timestamp2>,<server><Receive connector name>,<Sessionguid2>,5,<Hybrid server IP>:25,<member server IP>:<outboundport>,*,<Sessionguid2>;<Timestamp2>;1,receiving message

<Timestamp2>,<server><Receive connector name>,<Sessionguid2>,6,<Hybrid server IP>:25,<member server IP>:<outboundport>,>,250 2.1.0 Sender OK,

<Timestamp2>,<server><Receive connector name>,<Sessionguid2>,7,<Hybrid server IP>:25,<member server IP>:<outboundport>,<,RCPT TO:<<external user>>,

<Timestamp2>,<server><Receive connector name>,<Sessionguid2>,8,<Hybrid server IP>:25,<member server IP>:<outboundport>,>,250 2.1.5 Recipient OK,

<Timestamp2>,<server><Receive connector name>,<Sessionguid2>,9,<Hybrid server IP>:25,<member server IP>:<outboundport>,<,DATA,

<Timestamp2>,<server><Receive connector name>,<Sessionguid2>,10,<Hybrid server IP>:25,<member server IP>:<outboundport>,>,354 Start mail input; end with <CRLF>.<CRLF>,

<Timestamp2>,<server><Receive connector name>,<Sessionguid2>,11,<Hybrid server IP>:25,<member server IP>:<outboundport>,*,,Proxy destination(s) obtained from OnProxyInboundMessage event. Correlation Id:194f761b-72b4-42db-a6e0-e5f46a800165

<Timestamp2>,<server><Receive connector name>,<Sessionguid2>,12,<Hybrid server IP>:25,<member server IP>:<outboundport>,*,,Message or connection acked with status Fail and response 554 5.4.4 SMTPSEND.DNS.NonExistentDomain; nonexistent domain internalproxy -> DnsDomainDoesNotExist: InfoDomainNonexistent

<Timestamp2>,<server><Receive connector name>,<Sessionguid2>,13,<Hybrid server IP>:25,<member server IP>:<outboundport>,>,451 4.7.0 Temporary server error. Please try again later. PRX2 ,

<Timestamp2>,<server><Receive connector name>,<Sessionguid2>,14,<Hybrid server IP>:25,<member server IP>:<outboundport>,-,,Remote(SocketError)

My research pointed that

• Mails were sent to Hybrid server.
• Frontend transport service in hybrid receives the email and tries to give it to either Hybrid server hub connector or Legacy server hub connector.
• For some reasons its not able to connect to its own Hub server connector and always gives it to legacy server’s hub connector. (SEE ABOVE “Setting up client proxy session to destination”)
• Now from the legacy server hub connector mails gets routed back to Hybrid server hub connector and routes email to Office 365 or external.

Read the below to understand the transport pipeline better

Mail flow and the transport pipeline

From <https://docs.microsoft.com/en-us/exchange/mail-flow/mail-flow?view=exchserver-2019>

There could be multiple reasons why the hybrid frontend connector once it process the email not able to hand it over to its own Hub connector. Major issue is DNS.

• Either DNS is not properly configured
• The DNS where the network adapter is pointing does not have the proper info
• Pointing to wrong DNS
• Not able to resolve the server name or server FQDN.
• Etc

At first we tried to ping the server IP and it was able to ping.

1. Went to the hosts file located at c:/Windows/System32/Drivers/Etc. On the hybrid server

2. Edited the host file and add entry for Hybrid server’s IP and its host name.

3. Saved the file

4. Restarted the Transport service

Now the issue was RESOLVED and we see in the logs both the legacy server and hybrid server was used for  Outbound delivery. It randomly chooses either one.  The important point is now it does not fail when trying to connect to Hybrid hub connector from the Hybrid frontend connector

“250 2.6.0 <GUID@HYBRID FQDN> [InternalId=ID, Hostname=HYBRID FQDN] 1487 bytes in 2.801, 0.518 KB/sec Queued mail for delivery”,

And

“250 2.6.0 <GUID@HYBRID FQDN> [InternalId=ID, Hostname=Legacy FQDN] 1487 bytes in 2.801, 0.518 KB/sec Queued mail for delivery”,

Now we disabled the connectors and services on the legacy server and restarted the Transport services on the Hybrid server. Post this emails were only being routed via hybrid and we removed the mail loop.

–   Praveen Kumar E     

www.Modern365.co.in