HMC Mathematics Computing Weblog

Apparently there was a glitch in the power feed from Edison to the Claremont Colleges. The Colleges' generators did not kick in, which means that systems without UPS power -- notably the Amber cluster and the Scientific Computing Lab -- lost power and crashed.

The Scientific Computing Lab systems seem to have recovered and are running as expected. Our servers are backed by UPSs and by a local generator, so they did not lose power. The Amber cluster did lose power, and, while some machines restarted when power was restored, some did not. I have manually restarted those systems, and the cluster seems to be up and running normally.

If you're having a power-related problem, please let me know so that I can look into it.

I ended up doing some fairly significant work in the machine room Thursday afternoon and evening, which involved rewiring the entire rack. In order to be sure that some of the systems were working properly, I rebooted several of the machines in the rack, including the department's main file server (gytha) and our parallel compute server hex. As a result, some workstations -- especially Mac OS X machines -- may be confused about their NFS mounts. If you have problems logging in or if you can log in but you can't access your home directory or applications or other materials stored in /shared/local, please reboot the machine and try again.

I'm about to go to bed, but I will be reachable at home or by cell tomorrow if there are any unforeseen issues.

Earlier today, some packages that are used to deliver mail were updated, overwriting the custom packages that we had built for the system. There were two primary effects.

1. IMAP clients showed the root of your home directory instead of the contents of your mail directory; and
2. Mail delivery to your local mail directories was interrupted (mail was delivered to a mailbox on the mail server instead).

I have built new versions of our local packages and installed them. I have also redelivered any mail that was delivered to the system mail spool.

If you are still having problems with your mail (or anything else), please let me know.

I've just updated the rule set for SpamAssassin, the system that we use to check incoming mail to see if it's spam and label it appropriately.

I had thought that we were running the updates regularly, but apparently not. I'm hoping that the new rules will help catch the slew of annoying "I am tired this evening" spam that has been cluttering up my mailbox over the last couple of weeks.

If you weren't aware of our use of SpamAssassin, I would encourage you to check out our page on the tool, which will tell you a bit about how SpamAssassin works and give you some pointers on training it on your particular blend of mail.

There's also a sample .procmailrc file if you're interested in using Procmail to filter (or delete) your spam before you see it in your inbox. I've begun deleting the spammiest of my spam (messages that score more than 49.00), and I'm seriously thinking about lowering the score of the spam that I delete unseen.

Note that we take a very conservative approach to user's mail: we deliver all mail addressed to you, and only use SpamAssassin to label the mail. Filtering mail based on SpamAssassin scores is left up to you, so you can decide whether you trust the tool or not, or whether you want to use other tools to improve the handling of your mail.

While SpamAssassin seems to be fairly effective, our approach does still mean that we accept a lot of spam that we might not have to accept if we used some other spam-fighting techniques. Accepting spam has a cost in network bandwidth to transmit the messages and in CPU time to scan the messages to see if they're spam.

A technique I'm particularly interested in is greylisting, which records information about new connections and then temporarily bounces the mail, and waits for the remote mailer to try again. If the mailer tries again, then messages are accepted, but if not, they remain blacklisted. This technique takes advantage of the fact that most spammers are interested in delivering as much mail as possible in one shot, so they don't bother to retry if connections are refused. And there are some additional filtering tools (such as mailfromd) that we could use that I'll be looking into in the future.

In the meantime, I hope that SpamAssassin is still working well for everyone.