Aug  6 12:33:25 example sshd[7296]: in pam_sm_authenticate(): Failed to determine Kerberos principal name
Aug  6 12:33:29 example sshd[7294]: error: PAM: authentication error for user from example.com via 192.168.0.1
Aug  6 12:33:29 example sshd[7297]: in pam_sm_authenticate(): Failed to determine Kerberos principal name.
Aug  6 12:33:32 example sshd[7294]: error: PAM: user account has expired for user from example.com via 192.168.0.1

Then don’t do what I did. After googling to no avail, I went so far as to completely reinstall Mac OS X. No good.

[Update: Actually, what I did was clone a different system that I had recently set up and used it as the "new installation". Had I done a total, from DVD reinstallation, it would have fixed the problem but I wouldn't have discovered the cause.]

The answer? Make sure you didn’t set the Remote Login preferences in System Preferences/Sharing to “Only these users” and then forget to add the new user to the list!

D’oh.

Hopefully this will help the next person who’s looking for the answer.

It turns out to be nearly perfect. The unit is very sturdy, was easy to put together, and the shelf height is just right. There’s enough clearance for airflow but not so much that you feel space is being wasted.

I used self-stick cable tie anchors and cable ties to mount the power bricks and used double-stick mounting tape as stops to keep things in place. The old-style minis are heavy enough and are pretty non-slip, so I just put some tape at the front of the shelf to keep them from sliding off. The one new-style mini was pretty slippery so I used the tape to actually stick the base to the shelf.

The unit came with vertical rods that go in the back of each column of shelves to keep them from sliding out the back, but I decided to leave those out. That way I can slide each shelf forward to get DVDs into the mini, or back to get at the connectors.

The weak spot of the minis is the power cord (at least on the pre-2010 models) which comes out quite easily. I tied those down as well and am pretty sure they won’t jiggle their way out. I have four minis in the rack right now along with a Drobo with 10TB of disk. I’m going to be adding a 5th mini with a stackable disk drive, that’s why there’s double-high slot  still open on the mini side of the rack.

Cable management is an issue, mostly because of the power bricks long cables. I may fiddle with how I fold the cables into the shelves a bit more.

The whole thing plus a UPS and monitor/keyboard/mouse sits nicely on some steel shelves in our A/V equipment room at the museum. I still need to time how long the UPS runs. I’m only going to have the public web site minis on it.

In addition to figuring out how to deal with denyhosts, here’s how to set up Apache and Mailman log rotation.

I’m running standard Apache 2.2 that comes with Mac OS X 10.6. I like to keep all my virtual hosts in one place, so in this example they are all in /Users/web/. Each host gets a directory structure with it’s own name:

# ls -l www.example.org/
drwxrwxr-x   4 _unknown  _unknown   136 Jan 27 16:47 htdocs
drwxrwxr-x  80 _www      _www      2720 Mar 29 00:33 logs
-rw-r--r--   1 adoyle    web       1881 Feb 20 15:44 www.example.org.conf

The logs for each virtual host go into the logs directory for that host (access_log, error_log, rewrite_log). Ownership on the log files turns out to be important. I’ve found it works best for me if they are owned by the www user and group (or _www, they are essentially the same – something I need to understand the reason for someday).

In /etc/newsyslog.d/local.conf, the following lines deal with rotating logs for three virtual hosts. Using the ‘G’ flag lets you use ‘*’ and other shell wildcards in the file names. I think I could probably have collapsed these into a single line if I had used /Users/web/*/logs/*log instead. In this case, folding things up too much makes it less readable, I think.

The _www:_www takes care of preserving the file ownership after the logs are rotated. A count of 30 means keep around up to 30 old logs. $D0 means rotate daily at 0:00. The ‘B’ flag prevents the “Log file was rotated” message. Apache keeps a pidfile in /var/run/httpd.pid. If you send a kill -30 to the pid in that file, it will cause the equivalent of an ‘apachectl graceful’.

If you don’t provide the pidfile and proper signal number, the logs will rotate, new log files get created, but Apache won’t write to them because it’s still trying to write to the old ones.

# logfilename          [owner:group]            mode count  size  when   flags [/pid_file] [sig_num]
/Users/web/lists.example.org/logs/*log _www:_www 664   30     *    $D0     GBJ /var/run/httpd.pid 30
/Users/web/foo.example.org/logs/*log   _www:_www 664   30     *    $D0     GBJ /var/run/httpd.pid 30
/Users/web/www.example.org/logs/*log   _www:_www 664   30     *    $D0     GBJ /var/run/httpd.pid 30
#

With Mailman I had a slight problem. Mailman doesn’t use nice .log or _log names. It just uses names like bounce, error, post, qrunner, etc. I could have made an entry in the local.conf file for each one, but that seemed error-prone. What if later there’s a new version of Mailman that generates different log files?

My initial assumption was that I could use …/logs/* and newsyslog wouldn’t try to rotate logs it had already rotated. Guess again. After two days, my disk had nearly filled up with files ending in .bz2, .bz2.bz2, .bz2.bz2.b2z…, you get the picture. Luckily I noticed it before the disk did fill up. I got suspicious when my backups on the third day were 15GB bigger than the ones on the first day. I have no idea how many files actually got created. ‘ls’ was unable to produce a listing in the amount of time I was willing to wait. Luckily rm -rf did work. It took several hours to delete all the files.

I’m assuming newsyslog had gotten into a recursive loop right away when it ran at midnight and never stopped churning out files until I killed it.

So anyway, the moral of the story is, if you’re going to use wildcards, make sure they don’t match the rotated logs.

# logfilename          [owner:group]            mode count  size  when  flags [/pid_file] [sig_num]
/Users/mailman/logs/*[a-z]         mailman:_www 664   30     *    $D0     GBJ /Users/mailman/data/master-qrunner.pid 1
#

Mailman wants to be hit with a kill -1, so that’s what I used.

Things have been noodling along for a few weeks with my setup, so I think I have the kinks ironed out.

I had three goals – get denyhosts working, get it to start automatically at boot time, and to deal with rotating the logs.

1. Installation
Easiest first – installing denyhosts. Note that you need to be root to do this. Pretty much just follow the directions. These are the three main settings to worry about.

SECURE_LOG = /private/var/log/secure.log
LOCK_FILE = /var/run/denyhosts.pid
DAEMON_LOG = /var/log/denyhosts

Note that you also may need to create the file /etc/hosts.deny:

touch /etc/hosts.deny

Using touch will create a zero-length file if it’s not there. It won’t affect the contents if it is there.

2. Log rotation

Mac OS 10.6 uses newsyslog to rotate some log files (I’m not sure why, but apache logs don’t seem to be dealt with by newsyslog). To add your own to the mix, just put a file into /etc/newsyslog.d/ following the format for newsyslog.conf(5). I called mine local.conf

# logfilename          [owner:group]            mode count size when  flags [/pid_file] [sig_num]
/var/log/denyhosts                              640   5     *    $D0     J
#

The trouble is, this rotated the log just fine, but then denyhosts stopped logging because newsyslog essentially pulls the rug out from under denyhosts by moving the file.

One design difference between newsyslog and logrotate is the way they deal with notifying processes that logs have been rotated. Logrotate uses prerotate and postrotate scripts, which would be ideal for denyhosts. The way you start and stop it is with

daemon-control start
 
daemon-control stop

daemon-control stop actually sends a SIGTERM to the denyhosts process, but that won’t do any good in the newsyslog config file since once stopped, you need a command line to start it up again. So I decided to tweak the daemon-control script to do this. I replaced the start() function with the one here:

def start(*args):
    cmd = "%s --daemon " % DENYHOSTS_BIN
    if args: cmd += ' '.join(args)
 
    print "starting DenyHosts:   ", cmd
 
    while True:
        os.system(cmd)
        time.sleep(5)
 
        while True:
            pid = getpid()
            if pid >= 0:
                time.sleep(300)
            else:
                break

This just keeps daemon-control running rather than letting it exit after it starts denyhosts. The outer loop starts denyhosts running and later restarts it. The inner loop just waits until it sees the pid file go away. That’s a sure sign that denyhosts stopped running, most likely because of the SIGHUP it will get from newsyslog. Now all I needed to do was add the signal info to my /etc/denyhosts.d/local.conf /etc/newsyslog.d/local.conf file:

# logfilename          [owner:group]            mode count size when  flags [/pid_file] [sig_num]
/var/log/denyhosts                              640   30     *  $D0     BJ  /var/run/denyhosts.pid 15
#

I’ve also changed it to keep 30 days of logs, and added the B flag to prevent newsyslog from adding a line to the file saying it’s rotated the logs. Note that I changed the name to daemon-control2 so if I update denyhosts later, my changes don’t get clobbered.

3. Start at boot time

It turns out that modifying daemon-control to never exit is also just the ticket for running it under launchd. Launchd doesn’t work well on scripts that launch daemonized processes. It watches the script and notices that it’s exited, then tries to start it again.

I made a file called /Library/LaunchDaemons/net.hosts.deny.plist:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>Label</key>
    <string>net.denyhosts</string>
    <key>ProgramArguments</key>
    <array>
      <string>/usr/share/denyhosts/daemon-control2</string>
      <string>start</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
    <key>KeepAlive</key>
    <true/>
    <key>ServiceDescription</key>
    <string>Lauch denyhosts</string>
  </dict>
</plist>

Get it started with launchctl:

 launchctl load /Library/LaunchDaemons/net.hosts.deny.plist

My /etc/hosts.deny has about 8500 hosts in it right now. Many of those are probably from the denyhosts synchronization feature pulling in IP addresses from the central server.

Update 2010-03-26: Added some links and clarified some bits.

Update 2010-06-06: Note that /etc/hosts.deny must be present. denyhosts won’t create it.

My overall impression is that it’s slower than the beta and slower than Safari 3. But that’s gut feel, not with hard data. The “loading” bar is also far less obvious now, and I always find myself wondering whether Safari has stopped working because it takes a while to even start showing activity.

Before I ever install any updates on a server, I clone the disk with SuperDuper or CCC. That means I also have to first shut down all the services and pull the system off the net, clone the disk, boot the clone to make sure it’s ok, boot the normal disk, do the update, and test everything. Coupled with having to move the computers, 5 disk drives, and a D-Link switch, I didn’t have a fun few hours this morning.  I didn’t update the Leopard Server machine because I couldn’t get the alternate disk to boot. It turns out that the Iomega portable drive I was using (I love these little disks!) wasn’t getting enough power from the G5’s front connector and needed to be plugged in the back. By the time I figured that out, it was too late, the museum staff people were coming in and I had to have the server running again. So I’ll have to do the update another day.

Now one of the remote users on the updated Mini is having trouble getting in via ssh. So is it due to the move, the upgrade, or something completely unrelated? Having tried a bunch of things and looked at the log files, I’m leaning towards “something else”.

The one good thing that comes from this kind of thing is that you learn whether all the services are properly set up to start at boot time.

This one is hung in a box near the ceiling. It’s been running for over a year. We set them all to reboot after a power loss, so we almost never need access. When I do need to do anything to it, I grab a ladder and plug in a keyboard & mouse.

This one has also been running for about a year. Prior to that it was sitting in a server room for about two years. The enclosure gets a bit warm. We used to have a Shuttle XPC inside but it failed after about 4 months. I think it was the heat.

We have two display tables with baskets attached underneath. The Mini fits in the basket. We used to have two additional setups just like this. This photo is of the CityCar interactive exhibit. There’s another Mini at the other end of the floor in the MIT & the Sea interactive exhibit (below the Mini-in-a-box pictured above).

The latest one to be put into service is a Mini that we strapped to the top of a monster Panasonic projector. The Mini drives the smaller projector to the left in the photo. This Mini is running Vine Server so I can access it remotely. Once we’re done tweaking it, I’ll probably shut off the Vine access.

Our Admissions Desk also uses two Minis, one for the staff to access various admin tools, mail, etc. The other to drive a sign displaying admissions prices, welcoming groups, etc.

In January we’re going to be installing an exhibit developed by the Sociable Media Group at the Media Lab. So far it will have seven Mac Minis in it, as well as an iMac. There will also be a couple of Dells so I guess we’ll see how they hold up in comparison.

My wishlist for the Mini: Keep being ultra-stable, don’t get too heat sensitive. Faster graphics would be nice, but not at the expense of being more finicky.

PS – I have one in my basement at home as well, it’s running this blog, among other things. I think it’s about 3 years old. I have it on a small UPS to deal with short power outages, connected to the web via Verizon FiOS (also very reliable!).

Instead I ordered a MacBook from the Apple Education for MIT site (they are about $100 less there, and there’s a savings on AppleCare). When it gets here in a few days, I’ll clone my MacBook Pro’s hard drive to the MacBook and send the MBP to my son. They only thing I’ll miss, I suspect is the lighted keyboard, something I’ve come to like a lot. I ordered the 250GB drive and I’ll shop around for a RAM upgrade.

Then, with the savings, I’m thinking pretty seriously about an iPhone, something that would be fun to play with for the Museum Without Walls.

The iPhones I looked at at the Apple store had the new location detection firmware. The store is here but the phone indicated it was a good 1000′ southeast of the store, south of what’s labeled Holyhood Cemetery. That’s not exactly pinpoint accuracy. But the Skyhook wifi database may not have included the mall, and thus the iPhone was probably using the Google cell-tower locations to find itself. Skyhook claims 10-20m accuracy in urban areas. That sounds pretty reasonable to me.