WhiteHat Software and Consulting Inc.

16:00EDT:  Severe weather in Southeast Michigan killed power at our main facility between 2004-5-21 14:00EDT and 2004-5-22 16:00EDT. During this period mail and web services were not available. DNS remained online through our remote secondaries. Therefore mail should have been delivered at our offsite MX until the main site returned to normal operation.

If you experienced any other anomalies during this period please contact us and describe the issue in as much detail as possible.

The first actions taken after ensuring normal operation after the blackout were to create a remote backup web server with automatic data synchronization to prevent similar failures in the future.

20:30EDT: Also to be noted, some DNS servers out there seem to be ignoring the 1 hour time to live setting of our DNS records and caching the wrong IP for www.  This should not be a problem as the wrong IP now contains a full rather than partial copy of the website, but before it was discovered that they were breaking RFCs and caching records past their TTL some sites were experiencing strange behavior as they got the wrong IP for www.  Hopefully these broken DNS servers will correct themselves soon.