Drupal Service Interruption: Email Delivery Problems
Incident Report for University of Guelph IT System
Resolved
This incident has been resolved.

All mail sent from Drupal websites managed by CCS Web Solutions now routes through a private SendGrid IP address. All mail sent through this new IP address has been delivered (both to @uoguelph.ca and external domains.)

CCS will continue to monitor email activity for any new delivery problems. There is no need for individual site owners to monitor their Drupal websites for missed email notifications.
Posted Sep 28, 2020 - 15:52 EDT
Update
We continue to monitor mail delivery from Drupal sites.

All mail sent from Drupal websites to @uoguelph.ca addresses in the last 72 hours has been delivered. The majority of mail sent to external (non uoguelph.ca) addresses over the same time period has also been delivered successfully. Some mail addressed to external domains continues to be blocked due to the presence of SendGrid servers on public spam blocking lists.

Please continue to monitor your Drupal website for webform submissions and confirm delivery of messages to external (non uoguelph.ca) addresses as required.
Posted Sep 22, 2020 - 11:56 EDT
Monitoring
Drupal sites managed by CCS Web Solutions have been update to use a new SendGrid account for sending mail. As of yesterday (Sep 17) at 2:40 PM, all mail sent from Drupal sites to @uoguelph.ca addresses has been delivered. Delivery to some external addresses is still affected. As our new account builds up a positive reputation with SendGrid, external delivery will improve.

Please continue to monitor Drupal web form submissions for any missed email notifications.
Posted Sep 18, 2020 - 09:33 EDT
Identified
The issue has been identified and a fix is being implemented.
Posted Sep 17, 2020 - 10:52 EDT
Update
We continue to work toward a resolution to this issue.

We will provide updates on this page until resolution of the issue. Next update will be at approximately 10:00 AM Thursday morning.

We appreciate your patience and support.
Posted Sep 16, 2020 - 17:12 EDT
Investigating
We are continuing to work through the impact of this issue, and are working on a solution.

We will provide updates on this page until resolution of the issue. Next update will be at approximately 4:00 PM.

We appreciate your patience and support as we work to resolve this issue.
Posted Sep 16, 2020 - 11:46 EDT
Update
We are currently experiencing a problem with email delivery from Drupal websites hosted by CCS Web Solutions.

Drupal sites do not generally send a lot of mail. This issue primarily affects Drupal forms (such as web forms) that send an email when the form is submitted (as a notification, or as part of a workflow process.)

If you rely on email from your Drupal site for notification of form submissions, or to trigger an external process, please take a moment to review your site for recent activity that may have failed to send an email notification. For web forms, this means visiting the form on your site (as a site manager) and clicking the Results tab.

We will continue to work with our email service provider to resolve the issue.

(Please note that this issue was originally reported as a problem with CCS Form submissions. As the underlying cause is due to problems with email delivery from Drupal sites in general, the issue description has been updated accordingly.)
Posted Sep 16, 2020 - 08:42 EDT
Identified
We have identified the root cause and are working to resolve the issue.
Posted Sep 15, 2020 - 16:09 EDT
Investigating
We are experiencing a delay in processing of CCS Forms submissions by the Footprints ticketing system. We are currently investigating the cause of this issue.

We will provide updates on this page until resolution of the issue.

We appreciate your patience and support as we work to resolve this issue.
Posted Sep 15, 2020 - 12:27 EDT