I have removed all the components of that system, so we shouldn't see this issue again. If you do have problems with mail, however, please let me know. As problem with e-mail generally mean that e-mailing me won't help, you might want to ask Suzanne or DruAnn to contact me by phone.

I apologize for any inconvenience.

The new base URLs for these mirrors are

CentOS

http://mirror.hmc.edu/centos/

CTAN

http://mirror.hmc.edu/ctan/

For the time being, I have set up redirects so that attempts to retrieve data from URLs pointing to yum will automatically be redirected to mirror.hmc.edu.

At some point the machine that the old mirror server runs on may be retired or repurposed, so if you're using our CentOS mirror to update your machines or our CTAN mirror to download TeX-related material, I would encourage you to change the URLs in your YUM configuration files to point to the new location.

I will be doing some systems work this weekend, June 7--8.

Work will probably begin around 11:00 AM on Saturday, June 7, and will continue for several hours. If necessary, additional work may be done on Sunday, June 8, within a similar block of time.

What Will Be Affected

The work will disrupt most of our networked services, including e-mail, file service, interactive sessions, and the web server for periods of several minutes to an hour over the course of the work.

I also want to make sure that all of our Macs are running the latest security updates, so will be updating these machines during this time period as well.

What You Should Do

If you're using a Mac or Linux system that mounts file systems from our servers, before you leave on Friday evening,

• Save all open files;
• Close all applications;
• Log out;
• Leave your machine running.

Why

This work is necessary for us to ensure the security and improve the stability of the overall system. In particular, I am hoping that ongoing issues with our web server will be resolved as a result of this work.

I will do my best to keep as much of the system functional as possible for as much of the time as I can, but there will still be some outages.

Additional Background

Last semester we had some serious issues with interactions between the NFS support on our new file server and on our workstations and older servers, exacerbated by the HVAC failure. I was able to stabilize things, but we still see some flaky behavior (especially From the web server, which needs to be rebooted periodically).

On the Linux server side, I plan to update to the latest kernel releases and do some experimentation to see if everything will work together happily. I will need to reboot various servers and workstations an arbitrary number of times to explore all the possible interactions.

For Macs, I will install the latest updates, most of which require the machines to be rebooted. As Tiger (Mac OS X 10.4) has problems when an NFS server disappears and reappears, these machines would need to be rebooted anyway.

Comments/Problems/Other Issues

As usual, if there are problems with the scheduling of this work, requests or any other comments, please let me know.

Updates/Status Reports

As usual, updates on the status of the systems and progress reports will be posted to the ``sysblog'', on our web server at

http://www.math.hmc.edu/computing/blog/>

Thanks for your cooperation!

And esme did survive Friday night. But the rack's Ethernet switch did not. Luckily I had a spare, and I went in and replaced the dead switch. I also updated some of the configuration files on gytha, and I'm hoping that we'll see improved stability (although there may be some stuck machines that will require rebooting).

I have one more set of configuration updates to test and then implement, but I'm going to hold off on those until I'm in the office; probably around lunch on Monday. I will also probably try running a backup during the day to see whether it will trigger a crash before we return to running automated backups.

Given the failure of the switch, it's possible that the key to the problem was related to the switch handling the large amount of bandwidth consumed by the backups rather than anything connected directly to the NFS configurations or the backups (none of that configuration had been changed until last night). And if losing the switch is the only hardware failure from the HVAC failure, we will have gotten off pretty easy. I'll be keeping an eye on things in case there are additional failures.

In the meantime, I appreciate your patience while I work to restabilize the systems, and while I do some additional testing.

As a result, I have brought most everything back up. The lab systems (responding to shell.math.hmc.edu), ponder, the mail and web servers, the Amber cluster, and even our new mirror server are up and running.

hex is also running, but there's something preventing SSH connections from working (for me, too). I will look into that on Wednesday morning and will, I hope, have it working properly by the afternoon.

Once again, I apologize for the incredible inconvenience of this outage right during the crunch time at the end of the semester. I am still watching the systems fairly closely, as I have some seen some weird (that's a technical term) behavior related to NFS and RPC. I am doing my best to keep things as stable and available as I can.

I took advantage of the opportunity to add two shim classes that will catch attempts to use the older classes and tell you what you need to do to switch to the newer version.

The new version is available from the poster class page and is also installed on the math cluster (in /shared/local/share/texmf/tex/latex/hmcposter.

In the meantime, I have the file, mail, and authentication servers running. I have also brought ponder on line for general use. It's okay to (re)boot faculty, lab, and classroom workstations for use.

The Amber cluster, hex, our new mirror server (but not yum.math.hmc.edu) will remain offline until we get proper cooling back.

If you need information that is stored on hex or the Amber cluster, please send me e-mail so we can work out some way of getting you access. That won't happen before tomorrow, though, and we may have those systems back on line by then.

All computing services will be unavailable until our HVAC is back on line. Once we have cooling, I will begin restarting machines.

Apologies for the inconvenience.

I have not turned the IMAP and POP servers on, so getting mail will not be possible until I do. I have also shut down most of the Linux machines and Macs, and I won't be turning those back on until after things are more stable.

Note that tomorrow morning around 8:00 AM, we will have an HVAC engineer working to replace the dead motor in our HVAC unit. We may need to shut systems down in order to move things in the machine room to provide access to the HVAC equipment. Please check back here (the core web server is in my office and will remain operational) for more information tomorrow morning.

Please don't rush into starting up office, classroom, or lab machines unless I've said here that things are back up and reliable.

I apologize for this unforeseeable and extremely annoying outage. Please be assured that I am doing everything that I can to bring our systems back on line in a timely but safe manner.