On Friday 16th December 2011 starting at 8pm GMT we will be restarting the Zimbra service. The restart should take no longer than 15 minutes. All incoming email will be queued, access to the Zimbra servers will be restricted and Zimbra customers will not be able to access their mailboxes.

The restart is required to apply a configuration change which will remove the X-Originating-IP header from all mail sent from Zimbra. This header is causing a couple of users internal mail to be marked as spam because they're sending mail from an IP address that's listed on an RBL.

Zimbra have the following open bug to fix this problem without requiring the header removal http://bugzilla.zimbra.com/show_bug.cgi?id=44384.

Load returned to normal at approximately 15:40 GMT.  We're still looking into the cause.

We're investigating a high load issue affecting zimbra-mbox3.

I'm pleased to let you know that we finished our platform upgrade to Zimbra 7.1.3 P1 at 19:30 GMT.

Mail queued during the Zimbra upgrade is in the process of being delivered.

On Sunday 13th November 2011 starting at 4pm GMT we will be performing an upgrade of our Zimbra platform from 7.1.0 to 7.1.3 P1. We expect the Zimbra upgrade to take approximately 3 hours, although the window is scheduled for 4 hours.

Maintenance Start: 16:00 (GMT) 13th November 2011

Maintenance End: 20:00 (GMT) 13th November 2011

Once we commence the Zimbra upgrade all incoming email will be queued, access to the Zimbra servers will be restricted and Zimbra customers will not be able to access their mailboxes.

Please read the following Zimbra release notes for information about Zimbra 7.1.3: http://files.zimbra.com/website/docs/7.0/Zimbra%20NE%20Release%20Notes%207.1.3.pdf

Details for Patch 1, which will be applied at the same time as the upgrade, are here:
http://files.zimbra.com/website/docs/7.0/ZCS%20Patch%207.1.3-1.pdf

Please monitor http://twitter.com/itnet_noc or http://status.in-tuition.net for further details.

The full backup of Zimbra-mbox3 that runs every weekend failed to complete successfully due to the backup volume going offline.  The volume has been reattached and checked for errors.  A new full backup of Zimbra-mbox3 has been started but this will take some time to complete.  Mailboxes will become locked temporarily as they're backed up.  This should go unnoticed but there's a chance that an access attempt is made at the same time a mailbox is locked.

Unfortunately we're unable to perform any mailbox restores on zimbra-mbox3 until the backup completes.  We're also unable to restore mailboxes on zimbra-mbox3 since the last valid incremental which was on Friday 14th at 1AM.

The other Zimbra mailbox servers are unaffected.

Apologies for the trouble this may cause.

We have spent considerable time investigating the cause of the problems with Mbox3 and narrowed it down a single, problematic mailbox which was causing sockets to be left in a hung state. Removing this mailbox and restarting the Zimbra service fixed the problem, however we have not been able to conclusively prove whether this was the cause of the problem. We have been able to introduce additional monitoring to proactively alert us in the event of similar problems and before mail is delayed.

There is a related bug open on Zimbra's Bugzilla system, but it remains unconfirmed at present because it's not possible to consistently reproduce. We have tried to reproduce the problem on our test cluster and we also tried moving the problem mailbox there, but we have not been able to glean any further useful information.

Further background is that this problem seems to stem from IMAP connections and it is interesting to note that Zimbra 7 introduced a number of backend IMAP changes, including a whole new system for handling IMAP client connections: http://bugzilla.zimbra.com/show_bug.cgi?id=9470.

Again, please accept our apologies for the inconvenience caused.

At 16:08 BST we were alerted by our monitoring system that mails were being queued that were destined for this Zimbra mailbox server.  Mailboxes that are hosted on this server will have been affected and their emails queued.  A restart of Zimbra on this mailbox server was required to restore service.  There was a brief outage for users of this mailbox server whilst the restart was performed.  Queued mail is currently being delivered.

We believe the problem is the same one which occurred on the 5th July.  We've been able to track the problem down and will be working with Zimbra to fix.

Please accept our apologies for the inconvenience caused.

On Sunday 24th July 2011 starting at 5pm GMT we will be adding HSM storage to the zimbra-mbox1 server.  Any mailboxes on this server will be unavailable for the 10 minute duration of this maintenance window.  Emails arriving during this maintenance window will be queued and delivered once the mailboxes are available.

Maintenance Start: 17:00 (GMT) 24th July 2011

Maintenance End: 17:10 (GMT) 24th July 2011

Queued email is being delivered to the affected mailbox server now.

Rest assured that any legitimate emails sent to you during this incident would not have bounced or been lost - simply delayed. This is because of our multi layer architecture: External spam/virus servers handle initial receipt and if legitimate, forward to Zimbra MTA servers which in turn deliver to the appropriate host mbox server. In this case, all email was held on the Zimbra MTA servers during the problem with mbox3.

12:38 All clear update - all queued emails have been delivered. Please accept our apologies for the inconvenience caused.