On Thursday 17th May 2012 starting at 9:00pm BST we will be reconfiguring part of our core network switching topology.

Maintenance Start: 21:00 (BST) 17th May 2012

Maintenance End: 21:30 (BST) 17th May 2012

This work requires us to switch to our redundant connection temporarily.  The reconfiguration should take no longer than 10 minutes and the redundant connection will provide normal connectivity during this time.  There will be two short outages (2-3 seconds) whilst we disable and later enable our primary connection.

Permalink

We don't yet know the cause of the problem but mbox3 has been restarted and is functioning normally again.  Our apologies for the trouble.

Permalink

We're investigating an issue that's affecting some mailboxes on mbox3.  Affected mailboxes are returning 502 bad gateway errors.

Permalink

The mail delays on Thursday last week on the mx16 and mx26 clusters was caused by our spam scanning engine, specifically some queries to the bayes and awl databases timing out.  This caused some temporary deferrals as our mx clusters won't allow mail through until it has been properly scanned.  Any deferred mail will have later been retried and delivered successfully.

Enough mail was passing through correctly that our monitoring didn't immediately alert us.

We've since made changes to our monitoring and database configuration that should prevent this problem from reoccuring.

Permalink

We have not been able to find a definitive cause for the issues with mbox1, other than that the server was under very high load at the time of the problem compared to historical monitoring. We have since significantly increased the resources available to this server (memory and CPUs) and will continue to monitor the server closely.

Permalink

At 18:50 GMT our monitoring alerted us to an issue with mbox1 which had stopped accepting connections. We spent some time investigating but could not identify the cause. We restarted the services at 19:25 and the server was fully online by 19:39. Any mail sent to customers hosted on mbox1 one during this window was queued and is currently being delivered now. Customers who were trying to authenticate during this window may have received bad gateway errors even if their mailbox is not hosted on mbox1.

We will continue to monitor this server closely. Please accept our apologies for any inconvenience caused.

Permalink

We're investigating an issue with our mx16 and mx26 clusters that was delaying some mail.  Mail is now flowing normally and any queued mail is being delivered. We don't yet know the exact cause.

Permalink

On Thursday 1st March 2012 starting at 8:30pm GMT we will be reconfiguring part of our core network switching topology.

Maintenance Start: 20:30 (GMT) 1st March 2012

Maintenance End: 20:40 (GMT) 1st March 2012

This work requires us to switch to our redundant connection temporarily.  The reconfiguration should take no longer than 10 minutes and the redundant connection will provide normal connectivity during this time.  There will be two short outages whilst we disable and later enable our primary connection.

Permalink

On Friday 16th December 2011 starting at 8pm GMT we will be restarting the Zimbra service. The restart should take no longer than 15 minutes. All incoming email will be queued, access to the Zimbra servers will be restricted and Zimbra customers will not be able to access their mailboxes.

The restart is required to apply a configuration change which will remove the X-Originating-IP header from all mail sent from Zimbra. This header is causing a couple of users internal mail to be marked as spam because they're sending mail from an IP address that's listed on an RBL.

Zimbra have the following open bug to fix this problem without requiring the header removal http://bugzilla.zimbra.com/show_bug.cgi?id=44384.

Permalink

Load returned to normal at approximately 15:40 GMT.  We're still looking into the cause.

Permalink

We're investigating a high load issue affecting zimbra-mbox3.

Permalink

I'm pleased to let you know that we finished our platform upgrade to Zimbra 7.1.3 P1 at 19:30 GMT.

Mail queued during the Zimbra upgrade is in the process of being delivered.

Permalink

On Sunday 13th November 2011 starting at 4pm GMT we will be performing an upgrade of our Zimbra platform from 7.1.0 to 7.1.3 P1. We expect the Zimbra upgrade to take approximately 3 hours, although the window is scheduled for 4 hours.

Maintenance Start: 16:00 (GMT) 13th November 2011

Maintenance End: 20:00 (GMT) 13th November 2011

Once we commence the Zimbra upgrade all incoming email will be queued, access to the Zimbra servers will be restricted and Zimbra customers will not be able to access their mailboxes.

Please read the following Zimbra release notes for information about Zimbra 7.1.3: http://files.zimbra.com/website/docs/7.0/Zimbra%20NE%20Release%20Notes%207.1.3.pdf

Details for Patch 1, which will be applied at the same time as the upgrade, are here:
http://files.zimbra.com/website/docs/7.0/ZCS%20Patch%207.1.3-1.pdf

Please monitor http://twitter.com/itnet_noc or http://status.in-tuition.net for further details.

Permalink

The full backup of Zimbra-mbox3 that runs every weekend failed to complete successfully due to the backup volume going offline.  The volume has been reattached and checked for errors.  A new full backup of Zimbra-mbox3 has been started but this will take some time to complete.  Mailboxes will become locked temporarily as they're backed up.  This should go unnoticed but there's a chance that an access attempt is made at the same time a mailbox is locked.

Unfortunately we're unable to perform any mailbox restores on zimbra-mbox3 until the backup completes.  We're also unable to restore mailboxes on zimbra-mbox3 since the last valid incremental which was on Friday 14th at 1AM.

The other Zimbra mailbox servers are unaffected.

Apologies for the trouble this may cause.

Permalink

We have spent considerable time investigating the cause of the problems with Mbox3 and narrowed it down a single, problematic mailbox which was causing sockets to be left in a hung state. Removing this mailbox and restarting the Zimbra service fixed the problem, however we have not been able to conclusively prove whether this was the cause of the problem. We have been able to introduce additional monitoring to proactively alert us in the event of similar problems and before mail is delayed.

There is a related bug open on Zimbra's Bugzilla system, but it remains unconfirmed at present because it's not possible to consistently reproduce. We have tried to reproduce the problem on our test cluster and we also tried moving the problem mailbox there, but we have not been able to glean any further useful information.

Further background is that this problem seems to stem from IMAP connections and it is interesting to note that Zimbra 7 introduced a number of backend IMAP changes, including a whole new system for handling IMAP client connections: http://bugzilla.zimbra.com/show_bug.cgi?id=9470.

Again, please accept our apologies for the inconvenience caused.

Permalink

At 16:08 BST we were alerted by our monitoring system that mails were being queued that were destined for this Zimbra mailbox server.  Mailboxes that are hosted on this server will have been affected and their emails queued.  A restart of Zimbra on this mailbox server was required to restore service.  There was a brief outage for users of this mailbox server whilst the restart was performed.  Queued mail is currently being delivered.

We believe the problem is the same one which occurred on the 5th July.  We've been able to track the problem down and will be working with Zimbra to fix.

Please accept our apologies for the inconvenience caused.

Permalink

On Sunday 24th July 2011 starting at 5pm GMT we will be adding HSM storage to the zimbra-mbox1 server.  Any mailboxes on this server will be unavailable for the 10 minute duration of this maintenance window.  Emails arriving during this maintenance window will be queued and delivered once the mailboxes are available.

Maintenance Start: 17:00 (GMT) 24th July 2011

Maintenance End: 17:10 (GMT) 24th July 2011

Permalink

Queued email is being delivered to the affected mailbox server now.

Rest assured that any legitimate emails sent to you during this incident would not have bounced or been lost - simply delayed. This is because of our multi layer architecture: External spam/virus servers handle initial receipt and if legitimate, forward to Zimbra MTA servers which in turn deliver to the appropriate host mbox server. In this case, all email was held on the Zimbra MTA servers during the problem with mbox3.

12:38 All clear update - all queued emails have been delivered. Please accept our apologies for the inconvenience caused.

Permalink

We are currently investigating why email is being queued on one of our Zimbra mailbox servers - mbox3. We will post an update shortly.

12:23 - Update.

We identified some hung connections which stopped this mailbox server accepting any new email. We had to restart the affected mailbox server and we are waiting for load to stabalise before we enable mail delivery.

Permalink

I'm pleased to let you know that we finished our platform upgrade to Zimbra 7.1.0 at 21:00 GMT.

Mail queued during the Zimbra upgrade is in the process of being delivered.  Please see the previous post for details regarding a problem queuing some mail at the start of the maintenance window.

Permalink