On Thursday 17th May 2012 starting at 9:00pm BST we will be reconfiguring part of our core network switching topology.

Maintenance Start: 21:00 (BST) 17th May 2012

Maintenance End: 21:30 (BST) 17th May 2012

This work requires us to switch to our redundant connection temporarily.  The reconfiguration should take no longer than 10 minutes and the redundant connection will provide normal connectivity during this time.  There will be two short outages (2-3 seconds) whilst we disable and later enable our primary connection.

We don't yet know the cause of the problem but mbox3 has been restarted and is functioning normally again.  Our apologies for the trouble.

We're investigating an issue that's affecting some mailboxes on mbox3.  Affected mailboxes are returning 502 bad gateway errors.

The mail delays on Thursday last week on the mx16 and mx26 clusters was caused by our spam scanning engine, specifically some queries to the bayes and awl databases timing out.  This caused some temporary deferrals as our mx clusters won't allow mail through until it has been properly scanned.  Any deferred mail will have later been retried and delivered successfully.

Enough mail was passing through correctly that our monitoring didn't immediately alert us.

We've since made changes to our monitoring and database configuration that should prevent this problem from reoccuring.

We have not been able to find a definitive cause for the issues with mbox1, other than that the server was under very high load at the time of the problem compared to historical monitoring. We have since significantly increased the resources available to this server (memory and CPUs) and will continue to monitor the server closely.

At 18:50 GMT our monitoring alerted us to an issue with mbox1 which had stopped accepting connections. We spent some time investigating but could not identify the cause. We restarted the services at 19:25 and the server was fully online by 19:39. Any mail sent to customers hosted on mbox1 one during this window was queued and is currently being delivered now. Customers who were trying to authenticate during this window may have received bad gateway errors even if their mailbox is not hosted on mbox1.

We will continue to monitor this server closely. Please accept our apologies for any inconvenience caused.

We're investigating an issue with our mx16 and mx26 clusters that was delaying some mail.  Mail is now flowing normally and any queued mail is being delivered. We don't yet know the exact cause.

On Thursday 1st March 2012 starting at 8:30pm GMT we will be reconfiguring part of our core network switching topology.

Maintenance Start: 20:30 (GMT) 1st March 2012

Maintenance End: 20:40 (GMT) 1st March 2012

This work requires us to switch to our redundant connection temporarily.  The reconfiguration should take no longer than 10 minutes and the redundant connection will provide normal connectivity during this time.  There will be two short outages whilst we disable and later enable our primary connection.

On Friday 16th December 2011 starting at 8pm GMT we will be restarting the Zimbra service. The restart should take no longer than 15 minutes. All incoming email will be queued, access to the Zimbra servers will be restricted and Zimbra customers will not be able to access their mailboxes.

The restart is required to apply a configuration change which will remove the X-Originating-IP header from all mail sent from Zimbra. This header is causing a couple of users internal mail to be marked as spam because they're sending mail from an IP address that's listed on an RBL.

Zimbra have the following open bug to fix this problem without requiring the header removal http://bugzilla.zimbra.com/show_bug.cgi?id=44384.

Load returned to normal at approximately 15:40 GMT.  We're still looking into the cause.