|
|
| Home Learn About Conferences Member Pages Mail Store Services & Help Password Join Us | |
| Home > Services & Help >System Status | |
|
|
|
| Home Learn About Conferences Member Pages Mail Store Services & Help Password Join Us | |
| Home > Services & Help >System Status | |
System Status at The WELL |
||||||||||||||||||||
|
(This page is updated only when the status changes.) Updated Mon Oct 30 07:57:11 PST 2017 All email services will be down for Amazon power maintenance starting at around 12:55am Oct 31 US Pacific Time. Service should return no later than 3am. Updated Thu Oct 26 05:23:26 PST 2017 www.well.com was down most of this evening for major site overhaul. Unfortunately, things didn't go as well as we hoped, and the updates had to be backed out. We will probably try again soon, perhaps as soon as Thursday night. Updated Mon Sep 25 15:02:00 PST 2017 www.well.com will be down briefly (5-15 minutes) starting between about 3am and 3:30a. If you need to login to Engaged, use https://user.well.com. Updated Thu Jun 15 11:58:00 PST 2017 If you make several attempts to login with the wrong password, the system may lock you out by blocking further connections from you. Unfortunately, automated brute-force password hacking bots make this lockout essential, so we cannot disable the processes that handle this. For the conferences (SSH/Picospan and Engaged), a lockout presents as an inability to reach either well.com or user.well.com at all. In most cases, the lockout lasts about 3 hours, though we have temporarily reset it to 1 hour. For email, lockouts may present in a variety of ways. Sometimes, your mail client will simply refuse to fetch or send mail, with some sort of cryptic message. Worse yet, some clients may mistakenly assume that you have the wrong password, and will ask you to enter the correct password. Don't try entering your password or creating a new password - this may make matters worse. Just be patient; the system will automatically unlock your account after about one hour. Updated Tue Jun 06 21:38:00 PST 2017 If you attempt to log in and your password does not work, that means it was reset as a result of the outage on June 6th. Please visit https://www.well.com/cgi-bin/newpass/newpass.pl to reset your password. If you do not have an off-WELL email address on file, you will need to call Helpdesk at (415) 343-5731 and we'll help you reset your password. we'll be manning that phone from 9am-3pm PDT tomorrow. Updated Tue Jun 06 12:04:00 PST 2017 As of approximately 11:30am PDT, we have taken The WELL down for unscheduled maintenance. We discovered a serious problem that requires taking the conferencing offline in order to fix. As of now, I don't have any additional information but will pass news along to you as I learn it. We are estimating ~24 hours downtime - I'll post updates here as we get a better sense of how long we will be down. We anticipate losing very few, if any, posts, and no user files, as a result of this issue. As of now, mail (iris.well.com) is unaffected, and we do not anticipate that changing. I will continue to update this page, as well as https://www.well.com/status.html, as I get more information. Thank you for your patience! Updated Wed Apr 12 17:25:00 PST 2017 All http://www.well.com URLs now redirect to https://www.well.com for security. See welltech.382, responses 1591 and up for details/problems. Updated Tue Apr 04 20:56:00 PST 2017 We have started converting www.well.com to a purely https site in accordance with the requirements of recent versions of some browsers and Google's search priorities. On Thursday, we will switch user web pages at www.well.com to use https. See welltech.382, responses 1591 and up. Updated Tue Feb 21 07:26:00 PST 2017 Email upgrade postponed yet again. Probably next weekUpdated Mon Feb 15 01:14:00 PST 2017 Email upgrade postponed until Tuesday night due to holiday.The email system (iris.well.com aka Zimbra) will be upgraded on Tuesday evening Feb 21 starting about 9pm US Pacific Time. We expect to complete the upgrade by 5am Wednesday. Incoming mail will be queued locally, and delivered as soon as possible after the system is returned to service. Updated Mon Feb 15 01:14:00 PST 2017 Email upgrade failed. We will try again next week.The email system (iris.well.com aka Zimbra) will be upgraded on Monday evening Feb 20 starting about 9pm US Pacific Time. We expect to complete the upgrade by 5am Tuesday. Incoming mail will be queued locally, and delivered as soon as possible after the system is returned to service. Updated Mon Feb 13 08:56:00 PST 2017 Email upgrade postponed until Tuesday night/Wednesday morning!The email system (iris.well.com aka Zimbra) will be upgraded on Tuesday evening Feb 14 starting about 9pm US Pacific Time. We expect to complete the upgrade by 5am Wednesday. Incoming mail will be queued locally, and delivered as soon as possible after the system is returned to service. Updated Fri Feb 10 12:47:00 PST 2017 The email system (iris.well.com aka Zimbra) will be upgraded on Monday evening Feb 13 starting about 9pm US Pacific Time. We expect to complete the upgrade by 5am Tuesday. Incoming mail will be queued locally, and delivered as soon as possible after the system is returned to service. Updated Sun Jul 24 22:09:00 PST 2016 well.com/user.well.com should be back up -- investigation proceeding Updated Sun Jul 24 21:59:00 PST 2016 well.com/user.well.com is down -- had to rebuild system, rebooting now Updated Sun Jul 24 21:35:00 PST 2016 well.com/user.well.com is down -- waiting for system to shut down so I can restart Updated Sun Jul 24 21:20:41 PST 2016 well.com/user.well.com is down -- investigating. Updated Tue Jun 27 09:29:00 PST 2016 On Monday evening, 27 Jun, at 10pm US Pacific Time, there will be a brief downtime for well.com, user.well.com, and www.well.com; this includes all conferencing, alpine email, and web pages. We anticipate that the outage will be no more than 15 minutes in length, though we are scheduling a full hour, just in case. Updated Tue May 24 09:33:26 PST 2016 The conferencing system is back and running after the upgrade. If you find you cannot reach well.com or user.well.com, try rebooting your computer and all components of your network (routers, modems, wireless servers, etc). If this does not solve the problem, your DNS server may be slow about updating its cache. It will probably update in the next few hours. In the meantime, you can use the IP number 23.22.72.90 in place of "well.com" or "user.well.com". Note that your browser may issue dire warnings when accessing via the IP number.
If you use You may answer yes to this message. If you want to confirm that you really are connecting to the new system, you can use the web conferencing system to see the necessary fingerprints. The fingerprints may be found HERE after you have logged in If you see this: Remove "well.com" from your "~/.ssh/known_hosts" file on your system. UPDATE:
Running this command from a Mac Terminal window seems to fix this:
Sun May 22 13:00:00 PST 2016 On Monday, May 23, starting at about 8pm US Pacific Time, the WELL's conferencing systems (well.com and user.well.com) will be unavailable due to system maintenance. The maintenance period is scheduled to run until about 4am Tuesday. During the maintenance period, the operating system will be upgraded from CentOS 5.6 to CentOS 6.7. The upgrade is desirable from a standpoint of staying reasonably up to date, and for security reasons. All other systems (the main web server, the email servers, and the billing server) will all remain operational during the upgrade. However, the Alpine mail program will not be available. When the system returns to operation, you may continue to have problems connecting for a short while due to DNS propagation issues. This should be rare - we have the propagation timeout set very, very low - but inevitably there are DNS systems that don't update their caches in a timely manner. We will post instructions on what to do in this case after we have turned the system back on. Until it becomes an issue, don't worry about it. If you use our web interface, you should not see any differences after connecting.
If you use You may answer yes to this message. If you want to confirm that you really are connecting to the new system, you can use the web conferencing system to see the necessary fingerprints. The fingerprints may be found HERE after you have logged in Tue Apr 5 23:30:00 PST 2016 The WELL's primary web server, www.well.com, will be unavailable from 8pm US Pacific Time on Wednesday until around 2am on Thursday as we upgrade the operating system on this server. During the upgrade, WELL web pages at www.well.com will be unavailable, as will access to your WELL WEB folders. All other services will remain available throughout most of this upgrade. Engaged - the WELL's browser-based conferencing - will be available throughout most of this upgrade. For those people who usually access Engaged by going to http://www.well.com, a very brief status page will be available - it will include a link to Engaged https://user.well.com/engaged.cgi. The web server's IP number will change as part of this upgrade. This may result in complaints from your SFTP programs if you use one to manage your personal web pages. I think most browsers will be okay - they rely on the SSL certificate more than the specifics of the system. However, there is a chance that your browser might raise a security complaint. At the very end of the upgrade, it may be necessary to reboot well.com and user.well.com - this should only take a few minutes if it happens, and will most likely be done after midnight. Thu Feb 18 16:43:00 PST 2016 All seems well now. Thu Feb 18 14:19:00 PST 2016 The system appears to have gotten stuck while rebooting. Investigating now. Thu Feb 18 13:52:00 PST 2016 Re are experiencing issues with well.com temporarily losing connectivity, and are going to try a reboot in a few minutes. During the report, WELL conferencing (both shell access and Engaged) will be unavaiable. Wed Dec 23 11:09:50 PST 2015 Mail is back, but it may take a while for your ISP to update the new DNS information. Sat Dec 22 21:21:21 PST 2015 Thanks to unscheduled outage of the mail server (from about 8pm to 9:15pm), we managed to apply the patch originally scheduled for early tomorrow morning. So, that scheduled outage is canceled. Sat Dec 22 14:23:31 PST 2015 We will apply a security patch to the Zimbra email software on Wednesday at about 1:30am US Pacific Time (immediately after the nightly backups). It shouldn't take long - we'll likely be back up around 2am. Sat Dec 19 13:37:02 PST 2015 well.com and Engaged are back up and running. It looks like we just had an NFS software failure. Sat Dec 19 13:12:28 PST 2015 well.com and Engaged have crashed. We are investigating and will have it back as soon as we can. Sun Nov 22 11:52:47 PST 2015 The email system (iris.well.com) will be down briefly at about 6pm US Pacific Time to address the repeated crashes. The system should be back up no later than 6:15pm. Tue Nov 17 11:09:45 PST 2015 All systems are up and running. Mon Nov 16 10:23:20 PST 2015 Email will be unavailable from 1:30 am to 2:30 am PST, Tuesday, November 17th. We are applying an update patch to the Zimbra software. Mail will be spooled and held for delivery or for sending out after the downtime. We are sorry for any inconvenience. Fri Jun 26 01:25:40 PDT 2015 There will be a brief outage of our primary web server, www.well.com, on Sunday morning Jun 28 between about 4am and 6am US Pacific Time. The outage should last no more than about 5-10 minutes. Thu Jun 18 13:09:02 PDT 2015 It's come to our attention that Gmail is currently delaying most email from our customers, resulting in substantial slowness and maybe even bounces of mail going to Gmail accounts. We believe that forwarding spam is the root cause of these delays, and we have been forced to disable forwarding to Gmail for several email accounts. Those emails will remain on our mailserver for direct pickup by those customers. Fortunately, there is an alternative way to get your WELL mail on Gmail if you wish to - for most purposes, it acts just like forwarding, including Gmail spam filtering, without the negative side effects on other WELL customers. To read your WELL email vial Gmail, instead of forwarding, use Gmail's mail POP fetching function using the instructions in Settings. Log into Gmail and go to https://mail.google.com/mail/u/0/?pli=1#settings/accounts Use "Check mail from other accounts (using POP3):" to set it up. This provides the benefits of forwarding without the downsides. Remember, all well.com email customers may get well.com mail directly by logging on at iris.well.com We are sorry we had to take this emergency action. Please drop by the News conference system status topic if you want to discuss it. Thu Jun 11 09:09:08 PDT 2015 All systems are up and available. The scheduled maintenance of June 10-11 is complete, and all email should be processed and delivered within a few hours. Note that the IP of the server had to change, and that if you are having trouble connecting to iris.well.com, helpdesk may be able to assist you. You will find more about the situation in the System Status topic in the News or Welltech conferences. Mon Nov 17 13:44:00 PST 2014 All systems are up and available. The conference systems will be rebooted at around 11pm US Pacific Time on Mon, Nov 17. Service should be restored 5-10 minutes later. Thu Nov 13 00:17:34 PST 2014 All systems are up and available. The scheduled maintenance on Nov 12 is complete. Wed Nov 12 15:04:04 PST 2014 All systems are up and available. Conferencing via well.com and user.well.com will be unavailable from 10pm-midnight US Pacific Time on Wednesday, Nov 12. We apologize for the inconvenience. Tue Nov 06 18:30:30 PDT 2014 Investigating system outage on well.com/user.well.com Tue Oct 28 13:13:23 PDT 2014 All systems are up and available. SECURITY UPDATE Sometime on Wednesday, Oct 29, we will update the SSL certificate associated with iris.well.com (the WELL's email server, aka Zimbra, aka Iris) to one that uses the SHA-256 (aka SHA-2) hashing algorithm. The old SHA-1 algorithm has been deprecated, and many sources, including our certificate provider, are urging that the old certificates be replaced as soon as possible. (SSL certificates are used by browsers to ensure that the site you are accessing is the site that it claims to be.) Note that this change could have incompatibility impacts on older browsers and email software. Please see welltech.374.920 for further details. Sun Oct 19 02:41:13 PDT 2014 All systems are up and available. At about 11:30pm on Oct 18, there was a configuration glitch that temporarily caused email to be unavailable due to apparent password problems. The glitch was fixed and service restored at about 2:30am. If you tried to change your password during this period, you may find that you now have two different passwords - one for mail, and one for everything else. We recommend updating your password once more to get them back in synch. Fri Oct 10 10:15:24 PDT 2014 On Monday Oct 13 (Columbus Day) at 9:00 am PDT, the server at well.com (aka user.well.com) will be shut down for maintenance by Amazon Web Services. The maintenance is scheduled to last two hours, though we expect it will be much shorter than that. During the maintenance period, the UNIX and web interfaces to our conferences, Picospan and Engaged, will not be available. The UNIX shells as well as the Alpine mail program will also be unavailable. However, email will still be available at iris.well.com. Both sending and receiving email will also continue to be available via POP and IMAP, as usual. The www.well.com website will also remain up. We apologize for any inconvenience this maintenance presents. Wed Sep 10 17:06:24 PDT 2014 A major update to our web conferencing software, commonly referred to as "Engaged," has been completed. To report any new bugs or to read the scope of the updates please see the System Status Reports topic in News or WellTech. Some highlights:
For more, including updates to parsing and formatting, please see the System Status Reports topic in News or WellTech. Questions and bug reports are welcome. All systems remain up and available. Tue Apr 15 08:06:24 PDT 2014 All systems are up and available. Tue Apr 8 13:19:51 PDT 2014 The email system received an urgent security update today. It is possible that your browser or mail program will complain about not being able to verify the identity of iris.well.com. Please see welltech.374.334- for info. Tue Oct 1 14:54:53 PDT 2013 Planned Outage: Email Maintenance and Upgrade Rescheduled for 10/2 - 10/3
There will be no outbound well.com email service -- and no access to inbound mail -- starting Wednesday 10/2 at 8pm US Pacific Time. while we upgrade the email system to the latest version, Zimbra 8. We expect to restore services at 4am Pacific Time, Thursday, 10/2. Incoming mail will safely queue on the server until we are back. During the outage, you will not be able to send, receive, or view mail, nor can it forward. We apologize for any inconvenience. For updates, please watch this page. For discussion, go to the System Status discussions in the News or WELLtech conferences. Mon Sep 30 9:20:39 PDT 2013 Access has been restored. The upgrade will be rescheduled, due to circumstances beyond our control. Log in and see the System Status topic in the News or WELLtech conferences if you want to get the details or ask questions. Mail was queued and then delivered after the system came back up. Sorry for the inconvenience. Sun Sep 29 22:52:39 PDT 2013 Update: Access will be restored ASAP, due to circumstances beyond our control. This may be before or after the original target time of 4am. We do not have a new estimated time yet. Log in and see the System Status topic in the News or WELLtech conferences for additional details. We apologize for any inconvenience. Sun Sep 29 20:31:22 PDT 2013 Planned Outage: Email Maintenance and Upgrade 9/29 - 9/30 Initiial setup for upgrade is underway. Expect an update to this page at about 10:30pm US Pacific Time. There will be no outbound well.com email service -- and no access to inbound mail -- starting Sunday 9/29 at 8pm US Pacific Time. while we upgrade the email system to the latest version, Zimbra 8. We expect to restore services at 4am Pacific Time, Monday, 9/30. Incoming mail will safely queue on the server until we are back. During the outage, you will not be able to send, receive, or view mail, nor can it forward. We apologize for any inconvenience. For updates, please watch this page. For discussion, go to the System Status discussions in the News or WELLtech conferences. Tue Apr 9 22:14:46 PST 2013 All systems are running normally. ____ Fri Apr 5 13:09:45 PDT 2013 There will be a brief outage of well.com (the shell/picospan host), user.well.com (web conferencing/Engaged), and www.well.com (the web server) Tuesday at 10pm US Pacific Time (note that this is a rescheduled date). Each system needs to be rebooted so we can enable controls to prevent disk overruns. All systems should be back and running by 10:30pm. ____ Tue Feb 26 22:54:49 PST 2013 All systems are running normally. ____ Tue Feb 26 12:26:10 PST 2013 Mail is currently unavailable as we clean up from last week's upgrade. Mail is expected to return by midnight tonight. See you then! ____ Thu Feb 21 06:09:16 PST 2013 Last night's mail system upgrade appears to have been successful. However, it took 4 hours longer than we scheduled for due to unforeseen issues with the Amazon "Cloud". See welltech.370 for details of what's new. ____ Wed Feb 20 20:00:00 PDT 2013 Mail is currently unavailable as a software upgrade is performed. Mail is expected to be available by 2am Thursday morning. See you then! Update as of 02:47:00: we're running way behind schedule. Currently, we're expecting to be back online at around 4:30am. We apologize for the inconvenience, and especially the longer than expected outage. ____ Mon Jan 7 14:48:22 PST 2013 Due to an NFS issue, user WEB directories and short-term backup recovery snapshots are unavilable on well.com. User web pages are still available at http://www.well.com, and files can be updated with scp or sftp at www.well.com, but files cannot be viewed or updated from the well.com host. In addition, you may see problems when trying to login or logout from the Web-based conferencing software (Engaged). To login, go to http://www.well.com. The error on logout can be ignored. ____ Mon Dec 3 16:32:55 PST 2012 A system reboot is scheduled for Tuesday, Dec 4, 11pm US Pacific Time. The reboot will last about 5 minutes, and will effect shell and conferencing access (both via SSH and the Web). ____ Wed Nov 21 22:31:45 PST 2012 The restart of our SMTP server is taking longer than anticipated. We have an open ticket with Amazon Web Services to investigate why the system is not shutting down properly. We don't yet have an ETA. ____ Wed Nov 21 17:10:13 PST 2012 Amazon has alerted us that our SMTP server is running on "degraded hardware". To get things back to normal, we will be rebooting the server later tonight (around 10pm). If all goes well, it should only take about 5 minutes. Alpine users are most likely to see any effects - while the server is rebooting, Alpine will complain if you try to send email. Just resend the mail a few minutes later if this happens to you. Other customers may seem some minor delays on mail being received or sent, but in almost all cases, the delayed mail should be delivered properly some time after the system comes back up. Sorry for any inconvenience this reboot will cause... ____ Tue Oct 2 22:03:27 PDT 2012 All systems are up and running. However, we have changed our DNS authoritative server, and some DNS servers are being slow to pick up this change. If you are having trouble connecting to any WELL machines, you may want to change your DNS server on your router, Mac, or PC to use 8.8.8.8 as the primary server. However, be careful to keep a record of the previous settings, since you will likely want to change these settings back at some point. Your best bet is to wait until your normal DNS server picks up the updates, which should happen within a few hours in most cases. NOTE: It looks like there have been a few cases of mail bouncing from about 7:40pm-8:50pm, and again from about 10:05pm-10:20pm US Pacific Time. Please ask senders to resend if you are missing mail from this period. We apologize for any inconvenience resulting from this necessary update. ____ Fri Jun 15 12:18:22 PDT 2012 Amazon EC2 services, the cloud where our servers reside, has called for us to shut down and relaunch well.com and user.well.com, the servers where WELL conferencing lives. Tomorrow, Saturday June 16, those services will be unavailable from approximately 9pm until 11:00pm Pacific Daylight Time while wolfy migrates services to new instances. (Last night there was significant disruption that was probably unrelated to this next action, but which also came from problems at the Amazon cloud. Sorry for the inconvenience.) Thank you for your patience and loyalty. ____ Fri Jun 15 09:38:22 PDT 2012 All systems are up and running. ____ Fri Jun 15 01:22:04 PDT 2012 Email is down due to problems in the Amazon Cloud. We do not have an ETA at this time, but email may not be available until late morning or later. ____ Wed Mar 14 15:04:28 PDT 2012 We will be rebooting well.com and user.well.com Wednesday evening (Mar 14) at about 11pm US Pacific Time to make a minor configuration change. The reboot and maintenance will take about 20 minutes, and will effect access to the conferences via both Picospan and the Web, as well as all shell services (including alpine mail). We apologize for the inconvenience. ____ Thu Mar 1 22:26:11 PST 2012 (ETA changed to midnight - we're running a bit behind schedule) On Thursday evening between about 9pm and midnight US Pacific Time, WELL conferencing via UNIX shell and the web will be down, with all access to well.com and user.well.com disabled for the duration. During this maintenance, we will be switching over to a new system kernel - we expect that this will put an end to the kernel panics we have seen over the last two months. Access to your email via https://iris.well.com/, POP, and IMAP will not be affected by this maintenance. However, please note that access to email via the "alpine" mail program will not be available since well.com itself will be unavailable. We apologize for the inconvenience of this maintenance, but random system crashes are even more inconvenient. ____ Thu Feb 23 21:15:19 PST 2012 ENGAGED IS DOWN - the database was accidentally trashed. We hope to have it back up by 10:30pm US Pacific Time. ____ Wed Jan 11 17:00:35 PST 2012
We did not reboot Tuesday, so we will be rebooting well.com tonight, Jan 11, at about 10pm US Pacific Time. The reboot will take 5 to 10 minutes, and will effect all uses of the conferences. ____ Wed Jan 11 09:37:59 PST 2012 All systems are up and running. ____ Tue Jan 10 12:19:25 PST 2012
We will be rebooting well.com tonight at about 10pm US Pacific Time in an effort to address the repeated system panics. The reboot will take 5 to 10 minutes, and will effect all uses of the conferences. ____ Fri Dec 16 15:43:25 PST 2011 After a reboot, people are back posting. All systems are up and running. ____ Fri Dec 16 14:55:25 PST 2011 We are having a problem with the well.com and user.well.com machines. While we resolve this problem, the conferences -- our forums -- are unreachable. Email at iris.well.com and the www.well.com web page remain up. We are sorry for the inconvenience. ____ Fri Dec 9 10:53:25 PST 2011 All systems are up and running. ____ Thu Dec 8 2011 16:57:24 PST 2011 Our service provider, Amazon, needs to reboot our systems to perform some sort of system update (presumably patches). Fortunately, we can prevent the automatic reboot by rebooting manually. The best time for us to do this is tonight, even though that means a short notice outage. So, tonight at about 10pm US Pacific Time, each of the various well.com systems will be brought down then rebooted. Each reboot should take less than 5 minutes, though the reboot of the Zimbra machine may take a few minutes longer (it takes several minutes just for Zimbra to shut down, and another few minutes to start up). We apologize for the short notice for this announcement. ____ Thu Oct 27 14:10:50 PDT 2011 Most remaining bugs appear to have been eradicated. If you encounter any, please report them in the cloud discussion in the welltech conference. Log in, shortcut is "g welltech" to get there. If you use the PicoSpan interface and you're getting error messages on login, the problem may be configuration in your personal log in files. If you send a message to helpdesk, we can back up and replace your current log in files with basic files so that you can test each line to find the culprit. ____ Wed Oct 26 07:45:03 PDT 2011 The WELL is now back up and running in the Amazon Elastic Computing Cloud (EC2). Please report any bugs by going to the cloud discussion in the welltech conference. Log in, shortcut is "g welltech" to get there. If there is a malformed conference mentioned in an error message on your your Conference List page when you first connect to open a page at "https://user.well.com/engaged.cgi?" you may want to remove the bad conference temporarily by editing your Settings. Reporting a bug could win you one of a bunch of interesting books, so don't hesitate. ____ Mon Oct 24 16:42:39 PDT 2011 Migration down-time planned On Tuesday night, Oct 25, we finish up our migration into the Amazon Elastic Computing Cloud (EC2). The last machines to be moved will be the main machine (well.com), the conferencing web server (user.well.com), and the web page server (www.well.com). As part of this migration, conferencing (via both the command line and web browser), as well as all access to The WELL's web pages, will be disabled. Email will remain available, except for the "alpine" mailer. We're anticipating that the migration should be complete in about 6 hours: from 9pm PDT Tuesday evening to 3am PDT Wednesday morning. Following the migration, you should be able to use conferencing and all web services as before, with only minor changes that most people will not see. If by some chance you do experience problems after the system is restored to service, don't try to change your settings please. If we are still sorting out the move that could make things harder to diagnose. Please just contact helpdesk with any bug reports or post in the System Status topic in the WELL tech or News conference, so a proper diagnosis can be made. We apologize for any inconvenience this downtime may cause. ____ Tues Aug 30 13:07:22 PDT 2011 Performance of the email system remains sluggish - we are working to restore good service. We apologize for any inconvenience this slowness may cause.____ Fri Aug 26 13:43:56 PDT 2011 On Monday night, Aug 29, we begin our migration into the Amazon Elastic Computing Cloud (EC2). The first machines to be moved will be the Zimbra mail server (iris.well.com) and a companion server. As part of this migration, email will be disabled while the migration is under way. We're anticipating that the migration should be complete in about 6 hours: from 9pm PDT Monday evening to 3am PDT Tuesday morning. During the migration, incoming email from other sites should be queued at those sites, and re-sent later after we begin accepting incoming email again. WELL customers will not be able to look at previously received email during the outage unless you have already moved that email onto your computer's storage device. Following the migration, you should be able to continue using email using the settings that have worked for the last few weeks - nothing should change. If you encounter problems after the migration, please avoid changing your mail settings and report the problems to Helpdesk or post in System Status Reports. We apologize for any inconvenience this downtime may cause.____ Thu Aug 18 21:37:03 PDT 2011 The WELL is now back up and running. Engaged is still presenting some problems, but for now, you can make your own repairs using the instructions on your Conference List page. ____ Thu Aug 18 14:54:03 PDT 2011 RAID failure knocked out some services, currently being restored Around 1:00 am on the morning of Thursday, August 18th, home directories became unreachable at The WELL, causing all services except email to fail. At 9:30 the following morning we we were able to place the following unadorned message on the well.com site: Most functions of The WELL are currently not available due to a hardware failure of the RAID where most of our data is kept. We are working on restoring full service, but do not yet have an estimated completion time. We apologize for this inconvenience. Please note that email via IMAP, POP, and the Web should be available at this time. During that time we went through some replacement hardware options, settled on one, and got it installed at our co-location facility. At about 14:45 PDT logins became possible again. RAIDs are designed to prevent such problems when a disk fails, and we are in the process of moving well.com to the cloud, so this failure was perplexing and hopefully very soon all questions will become moot. Some services are still being repaired and adjusted at this time. Comments are welcome, please email Helpdesk at the address on the footer of this page, or swing by the welltech conference. We are sorry for the disruption. ____ Mon Aug 1 16:42:55 PDT 2011 Mail upgrade complete: If you can't get mail you need to change settings All systems are up and running. We now have IMAP available for all mail customers. Please be advised that we have shut off all access to two older mail servers as of this afternoon as part of the upgrade to email started in 2009. If you are pointing to those servers in your settings, your mail will no longer work. If you are not getting or sending email, you probably need to update some settings. You no longer have access to smtp.well.com nor to mh.well.com (mail.well.com). All incoming and outgoing mail operations will now involve the iris.well.com server. To get the appropriate Ports, look up your appropriate settings. 1. You should have access to your new email via a direct web browser login at iris.well.com now and henceforth. This is the easy, quick way for most people to see mail. 2. New mail help for all kinds of programs is now up at http://www.well.com/emailhelp.html with an overview of the system, general settings and a link to a page with more settings for specific mobile devices and computer email programs. 3. You can move your old email into the new environment by use of the utilities page if you wish. 4. There is a way to use Alpine for UNIX access to your mail. See the help page first. 5. We attempted to migrate you with settings matching your former preferences. In some situations we were not able to do that! In particular, if you were forwarding elsewhere and leaving a copy you are probably set up with straight forwarding now, and you will want to adjust that with a web login to iris.well.com. If you have questions, contact helpdesk or ask fellow users who have solved the problem in the welltech conference. Thank you for your patience. ____ Mon Aug 1 09:59:46 PDT 2011 Mail upgrade is in progress. All remaining mailboxes at The WELL are being transferred to the Zimbra email system at iris.well.com over the next several hours today. Please see the mail help page for instructions on setting up once your mailbox is converted, or use your web browser to go to iris.well.com. (Note that you cannot log in there until the upgrade script converts your mail settings.) Thank you for your patience. ____ Wed Jun 22 12:55:09 PDT 2011 All systems are up and running. ____
Monday, June 20, 2011 05:36:27 PM PDT The Zimbra email system on iris.well.com will be unavailable Tuesday evening (Jun 21) between the hours of 10pm and midnight so that we can perform a required software update. The legacy email system will remain up during this period. Sorry for the inconvenience. ____ Fri May 13 13:50:40 PDT 2011 Members using the Legacy email server are finding their connections time out when they are trying to access mail from mail.well.com. We are working on the situation. ____ Mon Dec 13 2010 The big Gawker break-in yielded lots of users data including some WELL email addresses, paired with passwords from Gawker and related sites. This past weekend, the website gawker.com was the victim of a systems break-in. Over 1 million Gawker, Gizmodo, LifeHacker and related sites commenting accounts may have been compromised, with logins, email addresses and passwords grabbed. About 40 well.com email address appeared on the public list of accounts that were compromised. Since many people use the same password on multiple sites, we have taken the precaution of temporarily disabling those WELL passwords, to prevent malicious access to those accounts here. If you were locked out for this or other reasons, please generate a new WELL Password. If you were not locked out, but you used an off-WELL email address at Gawker and related sites, and you use the same password at Gawker sites and here at The WELL, please change your password. You may see some phony emails about this, if your email address was made public for all to use. Sadly, it would be trivial for somebody to make such a message look fairly official in a phishing attempt to get you to go to another site that is unsafe. For that reason we are NOT including a clickable link to change a password in that informational email. Since you are reading this announcement in the form of a post on our site, and you navigated here on your own, it's safe to put some links up here. (You may also search for the news yourself instead):
If you need help getting back in, or if you want to report a phishing attempt with a fake link, you may call us at the number on the footer section of this web page, between 9am and 4:50pm Pacific Time (Monday - Friday). We apologize for any inconvenience this action may cause, particularly if you have been careful to avoid using your WELL password on other sites. However, we must take overall security seriously, for everyone's safety and privacy. Take-aways:
____ Mon Nov 22 13:18:20 PST 2010 We are experiencing a server error for user.well.com and for PicoSpan via ssh connection, each of which serves our conferences. We are working on the situation. Sorry for the inconvenience. ____ Mon Nov 22 13:46:20 PST 2010 Conferencing is back. ____ Mon Nov 22 13:18:20 PST 2010 We are experiencing a server error for user.well.com and for PicoSpan via ssh connection, each of which serves our conferences. We are working on the situation. Sorry for the inconvenience. ____ Wed Nov 3 09:38:20 PDT 2010 And...Mail is back. Sorry for the inconvenience. ____ Fri Wed Nov 3 09:35:10 PDT 2010 Email is temporarily down. We expect it to be restored soon. No mail will be lost. ____ Wed Nov 3 09:35:10 PDT 2010 We're having a problem with our registration pages. If you're trying to sign up for membership on The WELL and you're receiving an error message, please call the number at the bottom of this page or email helpdesk, also listed at the bottom of the page, to receive a notice when sign-ups are being received again. We'll also extend some bonus time on your new account. Sorry for the inconvenience! [Note: offer was valid through 10/5/10] ____ Wed Sep 15 14:39:35 PDT 2010 On Tuesday, September 7, we shut down one more insecure method of accessing The WELL when we disabled FTP on well.com and www.well.com. You now must use SFTP (Secure FTP) to manage the files you upload to your home and Web directories on The WELL. SFTP is nearly identical to FTP, with the exception that SFTP securely encrypts the connection. The commands are mostly the same, and, if you have a GUI client, you can probably continue to use that client - most GUI FTP clients let you select SFTP or FTP in the settings area. If you use a command line FTP on your system (such as on a Mac), there's a good chance that there is also a command line SFTP installed. Using SFTP in such cases is usually as simple as just replacing the "ftp well.com" command with "sftp MYLOGIN@well.com". Once you're connected, the commands will likely be the same. Please consult the System Status topic in the News Conference for more information. ____ Mon Aug 23 17:04:01 PDT 2010 The Zimbra email system, iris.well.com, will be down Tuesday Aug 24 from 9pm for a software update. We should be back online no later than 11pm, and likely will be back earlier than that. During the downtime, iris.well.com will not be available for either sending or receiving of email. Incoming mail will be queued for delivery after the system is restored to service. We are sorry for any inconvenience while we improve our mail service. (This downtime will not affect users of the legacy email systems (well.com, mh.well.com, smtp.well.com, and webmail.well.com), since they are maintained separately, nor will it affect any other WELL services.) ____ Mon Apr 19 17:35:02 PDT 2010 well.com (picospan, Pine, Unix, etc) is currently unavailable due to a typo. We're working on restoring service as soon as possible. In the meantime, conferencing is available through Engaged, and legacy mail can be accessed through Web Mail. We apologize for the inconvenience. ____ Mon Mar 29 14:06 PDT 2010 The Picospan (well.com) and Engaged (user.well.com) systems will be rebooted Wednesday evening, between about 8pm and 9pm, to bring the system up-to-date with patches. Each system will be down for 15-30 minutes, with each system offline separately. We apologize for any inconvenience caused by these brief outages. Update as of 21:12 PDT The first reboot took longer than originally scheduled. We are now rebooting Engaged, and should be back by 22:00 PDT ____ Weds Mar 3 2:36 PST 2010 Email has been restored, and email from earlier today is now being delivered. We "downgraded" to an earlier version of VMware, which finally resolved the issues seen through the day. All systems appear to be fine again. You may see duplicate mails as a rare side-effect due to all the up and down of the servers. We are sorry for the inconvenience. ____ Weds Mar 3 2:36 PST 2010 Email problems continue. You may have had some mail delivered briefly to the legacy email system, and for a longer period to the new iris/Zimbra system, but that has been interrupted again. We are having VMware problems, and we have an expert on the case right now trying the next approach. Over the morning hours we have done a series of fixes that looked stable but have failed, and we are optimistically taking the next step. More information as soon as it is available. We are sorry for the inconvenience. ____ Weds Mar 3 1:02 PST 2010 We are seeing additional problems, and are on the case. Looks like we spoke too soon. More information as soon as it is available. ____ Weds Mar 3 12:56 PST 2010 All email systems are working properly now. There will be a little delay as all the queued email is delivered. Thank you for your patience with this unexpected interruption. ____ Weds Mar 3 12:53 PST 2010 We are hoping to wrap it up within the hour. If you are on the new iris system (powered by Zimbra software) we are now pushing that mail though successfully. If you are on the legacy email system, please hold on while we get you going again. See the welltech conference for full discussion when we get all the details. We are sorry for the inconvenience. ____ Weds Mar 3 11:35 PST 2010 We are now experiencing some difficulties with both the mh and iris email systems. An initial software reboot to mh was not sufficient, and we began working with the hardware. Now all of our email systems are involved. More information will be available soon. See the welltech conference for discussion. We are sorry for the inconvenience. ____ Tue Jan 5 16:29:01 PST 2010 The new Zimbra email system (iris.well.com) is back! You may now use the new system for new mail again, via the Web or with your preferred email application. We are now recovering from the hardware and related problems. The queue of email messages from Monday and early Tuesday is flowing in too, in addition to new mail. You may notice duplicates generated from our emergency actions, if you read some of that mail as suggested below. More information will be available soon. See the welltech conference for discussion. ____ Tues Jan 5 6:24:58 PST 2010 We have not restored access to iris.well.com yet. Work continues. Our efforts at disk repair on Monday were to no avail, and another server was pulled from service, wiped and prepared for restoration of backups. This machine will become the new iris.well.com server soon. Unfortunately, some difficulties cropped up in the backup restoration process. The good news is that we have since gotten what we feel is a credible diagnosis and theory of what that problem is, and we are working on it. Late yesterday we extended the previously announced forward/copy scheme. This means that you may not only see and respond to copies of mail received since 9:00 am PST at the Squirrelmail web page, but you will also see older mail there, from the time of the hardware crash Sunday morning. There may be some duplicates. We will temporarily keep delivering copies of new email sent to all Zimbra-enabled accounts in the older email systems, until full recovery. We are simply appalled at this series of machine failures, holiday timing and bad coincidences. As soon as the backed-up email is back, we will work on strategies to make a scenario like this one impossible. We are very sorry for the inconvenience. ____ Mon Jan 4 10:12:58 PST 2010 The partial email outage continues. This is a hardware outage on a new machine, It only impacts Zimbra users, on the new mail system only. Many WELL members have not upgraded to the modern server yet, and will see no effects at all, so if your mail is working, disregard this message. This morning at 9:00 am Pacific Time we put into effect a forward/copy scheme that will temporarily keep copies of new email sent to all Zimbra-enabled accounts in BOTH email systems. This means if you are a new mail user you may pick up copies of mail received since 9:00 am by fetching them to the older Squirrelmail web page. (Log in at webmail.well.com) ___ Sun Jan 3 12:37:20 PST 2010 The new Zimbra email system (iris.well.com) is currently down for what appears to be a hardware failure. We are working on restoring service as soon as possible, but at this time, it looks like service will not be restored until sometime on Monday. We apologize for the inconvenience caused by this outage. ___ Wed Dec 2 21:49:27 PST 2009 The scheduled maintenance is complete. All systems appear to be up and running normally. ___ Mon Nov 30 13:16:08 PST 2009 We will be applying a variety of security updates and other system patches to all well.com systems on Wednesday evening (Dec 2) between the hours of 7pm and midnight. As each system is updated, the updated system will need to be rebooted. As a result, short outages of between 5 and 15 minutes (and possibly as much as 2 hours in the case of email) can be expected on all systems at different times. While mail on both the old and new systems will be unavailable during the outages for the respective machines, mail should be delivered as usual when the system is restored to operation. We apologize for any inconvenience caused by this necessary maintenance. ___ Fri Oct 2 12:08:50 PDT 2009 A little after 11:15am PDT we had problems with our DNS that made the WELL disappear for approximately 16 minutes for anybody who was not logged in at the time. All of our machines were running smoothly, but due to the DNS problem, our services were unavailable to new external connections during that period. Most email sent to addresses in the well.com domain will be re-sent. However, retrying depends on how other services configure their email servers. There is a chance that some email could have bounced back to the sender saying it was a "fatal error". We are sorry for the inconvenience. ___ Thu Sep 10 09:07:33 PDT 2009 Mail delivery has resumed. It will take some time - several hours, at least - for the backlog of mail to be delivered. ___ Thurs Sep 10 8:16 PDT 2009 The WELL's mail servers were updated last night and rebooted. However, this morning reports from dozens of members confirmed that a further fix is needed. We are working on it right now, and delivery of mail will resume as soon as the fix is completed. We are sorry for the inconvenience. ___ Tue Sep 8 15:48 PDT 2009 The WELL's mail servers will be down for a system software update between 7:30 and 8:30pm Pacific time on Wednesday September 9th. No email will be delivered or sent during this time. The WELL will queue incoming mail, and it will be delivered when the update is completed. ___ Fri Aug 7 09:20 PDT 2009 PicoSpan is unreachable at this time. We're working on the problem. Access to the conferences is available via the Web. ___ Fri Jun 5 13:56:09 PDT 2009 The Web interface to WELL conferences (user.well.com) will be taken out of service Monday June 8th from 8pm till 1am US Pacific Time so we can move the database from one host machine to another. This should prevent certain slowdowns. Complete Plan members may use ssh for PicoSpan (command line) access to conferences. During the downtime, we will also reboot the mail server (mh.well.com). Mail service should be back up again in fewer than 30 minutes. While the mail server is down, you won't be able to get new mail. Incoming mail will be delivered after the mail server is back online. We apologize for any inconvenience these preventative moves may cause. ___ Tue Apr 7 14:10:58 PDT 2009 There will be a reboot of the mail server (mh.well.com), Thursday night at 10:00 PM Pacific Time. The system should be back up again in less than 30 minutes. During the downtime, you won't be able to get new mail. Incoming mail will queue up during that time for delivery after the mail server is back online. We apologize for the inconvenience. Tue Mar 17 16:12:01 PDT 2009 There will be a reboot of the mail server (mh.well.com), Thursday night at 9 Pacific Time. The system should be back up again in less than 30 minutes. During the downtime, you won't be able to get new mail. Incoming mail will queue up during that time for delivery after the mail server is back online. ___ Wed Jan 14 01:39:03 PST 2009 The mail server (mh.well.com) will be taken out of service briefly Sunday at 9pm US Pacific Time so we can add additional memory. This should help prevent some of the mail delays we've been seeing the past 2 weeks. The total downtime should be no more than 30 minutes. Incoming mail will be queued locally, and will be delivered after the reboot. We apologize for the inconvenience. ___ Wed Jan 14 01:39:03 PST 2009 There will be brief outage of well.com Wednesday night at 11pm Pacific Time to correct a system configuration problem. The total downtime should be no more than a few minutes. For Further Information... For information about The WELL or for WELL Customer Support use the Helpdesk contact information on the footer of this page. |
||||||||||||||||||||
|
|
||||||||||||||||||||
