16:00EDT: Severe
weather in Southeast Michigan killed power at our main
facility between
2004-5-21 14:00EDT and 2004-5-22 16:00EDT. During this period mail and
web
services were not available. DNS remained online through our remote
secondaries.
Therefore mail should have been delivered at our offsite MX until the
main site
returned to normal operation.
If you experienced any other anomalies during this period please
contact us and describe the issue in as much detail as possible.
The first actions taken after ensuring normal operation after the
blackout were to create a remote backup web server with automatic data
synchronization to prevent similar failures in the future.
20:30EDT: Also to be noted,
some DNS servers out there seem to be ignoring the 1 hour time to live
setting of our DNS records and caching the wrong IP for www.
This should not be a problem as the wrong IP now contains a full rather
than partial copy of the website, but before it was discovered that
they were breaking RFCs and caching records past their TTL some sites
were experiencing strange behavior as they got the wrong IP for
www. Hopefully these broken DNS servers will correct themselves
soon.