On Friday 16th December 2011 starting at 8pm GMT we will be restarting the Zimbra service. The restart should take no longer than 15 minutes. All incoming email will be queued, access to the Zimbra servers will be restricted and Zimbra customers will not be able to access their mailboxes.

The restart is required to apply a configuration change which will remove the X-Originating-IP header from all mail sent from Zimbra. This header is causing a couple of users internal mail to be marked as spam because they're sending mail from an IP address that's listed on an RBL.

Zimbra have the following open bug to fix this problem without requiring the header removal http://bugzilla.zimbra.com/show_bug.cgi?id=44384.

Permalink

Load returned to normal at approximately 15:40 GMT.  We're still looking into the cause.

Permalink

We're investigating a high load issue affecting zimbra-mbox3.

Permalink

I'm pleased to let you know that we finished our platform upgrade to Zimbra 7.1.3 P1 at 19:30 GMT.

Mail queued during the Zimbra upgrade is in the process of being delivered.

Permalink

On Sunday 13th November 2011 starting at 4pm GMT we will be performing an upgrade of our Zimbra platform from 7.1.0 to 7.1.3 P1. We expect the Zimbra upgrade to take approximately 3 hours, although the window is scheduled for 4 hours.

Maintenance Start: 16:00 (GMT) 13th November 2011

Maintenance End: 20:00 (GMT) 13th November 2011

Once we commence the Zimbra upgrade all incoming email will be queued, access to the Zimbra servers will be restricted and Zimbra customers will not be able to access their mailboxes.

Please read the following Zimbra release notes for information about Zimbra 7.1.3: http://files.zimbra.com/website/docs/7.0/Zimbra%20NE%20Release%20Notes%207.1.3.pdf

Details for Patch 1, which will be applied at the same time as the upgrade, are here:
http://files.zimbra.com/website/docs/7.0/ZCS%20Patch%207.1.3-1.pdf

Please monitor http://twitter.com/itnet_noc or http://status.in-tuition.net for further details.

Permalink

The full backup of Zimbra-mbox3 that runs every weekend failed to complete successfully due to the backup volume going offline.  The volume has been reattached and checked for errors.  A new full backup of Zimbra-mbox3 has been started but this will take some time to complete.  Mailboxes will become locked temporarily as they're backed up.  This should go unnoticed but there's a chance that an access attempt is made at the same time a mailbox is locked.

Unfortunately we're unable to perform any mailbox restores on zimbra-mbox3 until the backup completes.  We're also unable to restore mailboxes on zimbra-mbox3 since the last valid incremental which was on Friday 14th at 1AM.

The other Zimbra mailbox servers are unaffected.

Apologies for the trouble this may cause.

Permalink

We have spent considerable time investigating the cause of the problems with Mbox3 and narrowed it down a single, problematic mailbox which was causing sockets to be left in a hung state. Removing this mailbox and restarting the Zimbra service fixed the problem, however we have not been able to conclusively prove whether this was the cause of the problem. We have been able to introduce additional monitoring to proactively alert us in the event of similar problems and before mail is delayed.

There is a related bug open on Zimbra's Bugzilla system, but it remains unconfirmed at present because it's not possible to consistently reproduce. We have tried to reproduce the problem on our test cluster and we also tried moving the problem mailbox there, but we have not been able to glean any further useful information.

Further background is that this problem seems to stem from IMAP connections and it is interesting to note that Zimbra 7 introduced a number of backend IMAP changes, including a whole new system for handling IMAP client connections: http://bugzilla.zimbra.com/show_bug.cgi?id=9470.

Again, please accept our apologies for the inconvenience caused.

Permalink

At 16:08 BST we were alerted by our monitoring system that mails were being queued that were destined for this Zimbra mailbox server.  Mailboxes that are hosted on this server will have been affected and their emails queued.  A restart of Zimbra on this mailbox server was required to restore service.  There was a brief outage for users of this mailbox server whilst the restart was performed.  Queued mail is currently being delivered.

We believe the problem is the same one which occurred on the 5th July.  We've been able to track the problem down and will be working with Zimbra to fix.

Please accept our apologies for the inconvenience caused.

Permalink

On Sunday 24th July 2011 starting at 5pm GMT we will be adding HSM storage to the zimbra-mbox1 server.  Any mailboxes on this server will be unavailable for the 10 minute duration of this maintenance window.  Emails arriving during this maintenance window will be queued and delivered once the mailboxes are available.

Maintenance Start: 17:00 (GMT) 24th July 2011

Maintenance End: 17:10 (GMT) 24th July 2011

Permalink

Queued email is being delivered to the affected mailbox server now.

Rest assured that any legitimate emails sent to you during this incident would not have bounced or been lost - simply delayed. This is because of our multi layer architecture: External spam/virus servers handle initial receipt and if legitimate, forward to Zimbra MTA servers which in turn deliver to the appropriate host mbox server. In this case, all email was held on the Zimbra MTA servers during the problem with mbox3.

12:38 All clear update - all queued emails have been delivered. Please accept our apologies for the inconvenience caused.

Permalink

We are currently investigating why email is being queued on one of our Zimbra mailbox servers - mbox3. We will post an update shortly.

12:23 - Update.

We identified some hung connections which stopped this mailbox server accepting any new email. We had to restart the affected mailbox server and we are waiting for load to stabalise before we enable mail delivery.

Permalink

I'm pleased to let you know that we finished our platform upgrade to Zimbra 7.1.0 at 21:00 GMT.

Mail queued during the Zimbra upgrade is in the process of being delivered.  Please see the previous post for details regarding a problem queuing some mail at the start of the maintenance window.

Permalink

The Zimbra upgrade is currently taking place.  Unfortunately we found an error that caused some emails not to be queued by our MX clusters, instead bouncing incoming emails.  These bounces occurred between 17:00 and 18:30 GMT.  The cause was a broken MTA service on one of our Zimbra MTA servers.  This service has been shutdown and we'll be looking into how it got into this state.

Valid senders will receive bounce emails for any emails they sent during this time.

Gateway scanning services are unaffected.

Apologies for the trouble.

Permalink

One of our mailbox servers did not update its license last week when a new license was installed to the cluster.  The old license expired today resulting in users of this server receiving a warning.  The license has been manually updated on this server and we'll investigate why it wasn't updated automatically.

Permalink

On Sunday 19th June 2011 starting at 5pm GMT we will be performing an upgrade of our Zimbra platform from 6.0.9 to 7.1.0. We expect the Zimbra upgrade to take approximately 3 hours, although the window is scheduled for 4 hours.

Maintenance Start: 17:00 (GMT) 19th June 2011

Maintenance End: 21:00 (GMT) 19th June 2011

Once we commence the Zimbra upgrade all incoming email will be queued, access to the Zimbra servers will be restricted and Zimbra customers will not be able to access their mailboxes.

Please read the following Zimbra release notes for information about Zimbra 7.1.0: http://files.zimbra.com/website/docs/7.0/7_1_NE_GA_Release%20Notes.pdf

There are some major changes with Zimbra 7 that you may wish to inform your clients about before the upgrade, some of which are listed below:

- Webmail login theme change.

- The new version of the Outlook Connector (available after the upgrade) supports auto update.

- This is the last major Zimbra release that will include the beta IM feature.  Future versions will not support IM as Zimbra chose to remove it.

- It's now possible to brand the Outlook Connector.

- ZimbraMobileSync needs to be enabled for any BesAdmin accounts used in the ZCB profiles.  See bug #58659 for details.

Please monitor http://twitter.com/itnet_noc or http://status.in-tuition.net for further details.

Permalink

On Sunday 3rd April 2011 starting at 6pm GMT we will be replacing the SSL certificate used on our Zimbra platform.  The current certificate is due to expire on the 7th April.  We expect the installation of this certificate to take approximately 15 minutes, although the window is scheduled for 30 minutes.

Maintenance Start: 18:00 (GMT) 3rd April 2011

Maintenance End: 18:30 (GMT) 3rd April 2011

Permalink

I'm pleased to let you know that we finished our platform upgrade to Zimbra 6.0.9 at 19:40, one hour and twenty minutes ahead of schedule.

Mail queued during the Zimbra upgrade is in the process of being delivered.

Permalink

On Sunday 21st November 2010 starting at 5pm GMT we will be performing an emergency upgrade of our Zimbra platform from 6.0.8 to 6.0.9. We expect the Zimbra upgrade to take approximately 3 hours, although the window is scheduled for 4 hours.

A vulnerability resulting in a buffer overflow has been discovered in the openssl package included in Zimbra 6.0.8 that has necessitated this upgrade. Please see http://openssl.org/news/secadv_20101116.txt for more information.

Maintenance Start: 17:00 (GMT) 21st November 2010

Maintenance End: 21:00 (GMT) 21st November 2010

Once we commence the Zimbra upgrade all incoming email will be queued, access to the Zimbra servers will be restricted and Zimbra customers will not be able to access their mailboxes.

Please read the following Zimbra release notes for information about Zimbra 6.0.9: http://files.zimbra.com/website/docs/Zimbra%20NE%20Release%20Notes%206.0.9.pdf

Please monitor http://twitter.com/itnet_noc or http://status.in-tuition.net for further details.

Permalink

09:15 Our monitoring shows higher than normal load on one of the Zimbra mailbox servers, mbox3 - we are investigating the cause. This started at just after 9am this morning. We have received reports of slow response when using webmail.

10:45 UPDATE

We believe we have resolved the issues causing problems with mbox3 - webmail is now working (since 10:30) although response times are likely to be a little slower whilst usage settles down. We are monitoring the server closely and will post a further update once we are satisfied that the issue is fully resolved.

13:30 UPDATE

Shortly after 10:30 GMT server load on mbox3 returned to normal and has stayed there since.  Changes were applied to mysql and Zimbra (memory settings, cache configuration, and connection values) at 10:30 and we believe these changes resolved the problem however we're still investigating.

Please accept our apologies for this mornings trouble.

Permalink

18:40 GMT : We're currently investigating an issue with our smtp cluster used for outgoing mail.  Some emails have been queued on the cluster delaying their delivery.  Any queued emails will be delivered shortly and we'll be looking into the exact cause.

19:00 GMT: All queued emails have now been delivered.  This issue affected some local emails - remote emails were unaffected.  The exact cause is still unknown at this time.

Permalink