SAN problems - update #2

23:08 BST - we are continuing to experience problems with one of our SAN nodes - engineers from Lefthand are trying to resolve the problem. Services may be inaccessible intermittently as we have to shut down volumes whilst performing emergency maintenance. We apologise for any inconvenience this is causing and assure you that we are working to resolve this problem as an urgent priority.

Connectivity Problems

Our monitoring has alerted us to connectivity problems with some of our network which we are investigating as a matter of priority now. Mailbox servers are inaccessible. We will post an update shortly.

Update - most servers are back online - we are investigating issues with a SAN node which appears to have gone offline without explination.

Update #2 23:08 BST - we are continuing to experience problems with our SAN - engineers from Lefthand are trying to resolve the problem. Services may be inaccessible intermittently. We apologise for any inconvenience this is causing and assure you that we are working to resolve this problem as a priority.

Notification of Planned Maintenance

On Sunday the 14th of June at 8AM (BST) we will be upgrading our Zimbra platform to the latest release.  This will involve a period of downtime of approximately 4 hours where Zimbra hosted customers will not be able to access their Zimbra email.

Maintenance Start: 08:00 (BST) 14th June 2009

Maintenance End: 12:00 (BST) 14th June 2009

For details of Zimbra changes, please refer to http://files.zimbra.com/website/docs/Zimbra%20NE%20Release%20Notes.pdf

During the maintenance window all incoming email will be queued, access to the Zimbra servers will be restricted and Zimbra customers will not be able to access their mailboxes.

Notification of Planned Maintenance

On Thursday the 4th of June at 8PM (BST) we will be upgrading our SAN software.  This will involve us running in a degraded state at certain points during the upgrade.  No outage or downtime is expected, however there is increased risk during the maintenance window.

Maintenance Start: 20:00 (BST) 4th June 2009

Maintenance End: 22:00 (BST) 4th June 2009

Notification of Planned Maintenance

On Sunday the 17th of May at 8AM (BST) we will be converting our Zimbra ldap servers to 64 bit.  This will allow us to allocate more resources as required in future to keep up with demand.  Zimbra will be unavailable during the conversion.

Maintenance Start: 08:00 (BST) 17th May 2009

Maintenance End: 10:00 (BST) 17th May 2009

During the maintenance window all incoming email will be queued, access to the Zimbra servers will be restricted and Zimbra customers will not be able to access their mailboxes.

Non SSL POP3 connections to Zimbra being rejected - fixed

The problem whereby client devices trying to connect to zimbra using non SSL POP3 connections has been fixed as of 10:25 GMT this morning. The issue affected only a small number of mailboxes.

Details

We are migrating to a new 64bit Zimbra platform to allow for future expansion and to keep in line with Zimbra's latest best practice guidelines. This migration involves moving mailboxes off of their 32bit host servers onto new 64bit replacements. So far, all but one of the mailbox servers have been fully and successfully migrated and we are 50% through the final server's mailboxes (many of these 50% were migrated over this last weekend). This final server, although configured to allow non SSL connections for POP3 had an unrelated LDAP attribute set in a way which meant that non SSL connections were effectively blocked. We have reversed this setting, restarted the services and can now confirm that as of 10:25 GMT this morning, clear text POP3 connections are being accepted without issue. Our apologies for any inconvenience this may have caused.

Summary of Protocols requiring SSL

POP3 - not required, but recommended
IMAP - SSL must be enabled
HTTP - Optional (may change in future - see below)
ActiveSync - Optional (may change in future - see below)

Future SSL related changes


We are planning to set our Zimbra platform to force SSL for HTTP and ActiveSync but further testing is required to ensure complete client compatibility and to assess the impact before we go ahead. We will issue a notification well in advance of any change to allow you time to inform your users.

Notification of DNS change

On Saturday 25th April 2009 at 9am we will update the A host record of the primary unbranded Zimbra domain so that all incoming connections terminate at our Zimbra Proxy Cluster. The impact on end users should be zero, however, it is recommended that resellers read the details below to fully understand the background to this change.

Any resellers using their own URL, especially if they have created CNAME or Host records resolving to 217.154.106.217 should take note of the details below.

Details

[zimbra . protectedservice . net] will be changed to resolve to 217.154.106.232

Background Information

Currently, when users access their Zimbra mailbox, all connections are handled by Zimbra Proxy - a new fairly new feature of Zimbra which is able to route users to the actual server which hosts their mailbox. Any Zimbra mailbox server can provide proxy server duties and we have been using the Proxy feature on our production Zimbra platform in this way for nearly a month with great success. The change outlined above means that our primary domain will now resolve to our dedicated Zimbra Proxy cluster running on multiple servers configured for High Availability. This has also been in production for some weeks and we have performed various test migrations similar to what we will do on Saturday without any noticeable effect on users.

Proxied Protocols

Zimbra Proxy handles all Zimbra protocols except for XMPP (Jabber), therefore users who wish to make use of an external IM client (Adium/Pidgin etc) should work out the hostname of the server their mailbox is hosted on and use it as the server name in their IM client. It is possible to work out which server hosts a mailbox by viewing the email headers of a received email in the mailbox concerned (look for the received: header field - it will contain a server name similar to zimbra-mbox[X]....)


MX Server issue - resolved

The problem reported a few minutes ago has been resolved and was due to an LDAP issue on one of our MX server clusters. Some mail handled by mx26 may be delayed whilst the remote servers retry delivery to us. Sorry for any inconvenience this may have caused.