Are you getting these messages in your console log when you make a new user on Mac OS X and then try to ssh in to the computer with that new user name?

Aug  6 12:33:25 example sshd[7296]: in pam_sm_authenticate(): Failed to determine Kerberos principal name
Aug  6 12:33:29 example sshd[7294]: error: PAM: authentication error for user from example.com via 192.168.0.1
Aug  6 12:33:29 example sshd[7297]: in pam_sm_authenticate(): Failed to determine Kerberos principal name.
Aug  6 12:33:32 example sshd[7294]: error: PAM: user account has expired for user from example.com via 192.168.0.1

Then don’t do what I did. After googling to no avail, I went so far as to completely reinstall Mac OS X. No good.

[Update: Actually, what I did was clone a different system that I had recently set up and used it as the "new installation". Had I done a total, from DVD reinstallation, it would have fixed the problem but I wouldn't have discovered the cause.]

The answer? Make sure you didn’t set the Remote Login preferences in System Preferences/Sharing to “Only these users” and then forget to add the new user to the list!

D’oh.

Hopefully this will help the next person who’s looking for the answer.

I needed to do something about the Mac minis that were accumulating on the table in my office. Digging around, I found this Rubbermaid organizer on Amazon.

It turns out to be nearly perfect. The unit is very sturdy, was easy to put together, and the shelf height is just right. There’s enough clearance for airflow but not so much that you feel space is being wasted.

I used self-stick cable tie anchors and cable ties to mount the power bricks and used double-stick mounting tape as stops to keep things in place. The old-style minis are heavy enough and are pretty non-slip, so I just put some tape at the front of the shelf to keep them from sliding off. The one new-style mini was pretty slippery so I used the tape to actually stick the base to the shelf.

The unit came with vertical rods that go in the back of each column of shelves to keep them from sliding out the back, but I decided to leave those out. That way I can slide each shelf forward to get DVDs into the mini, or back to get at the connectors.

The weak spot of the minis is the power cord (at least on the pre-2010 models) which comes out quite easily. I tied those down as well and am pretty sure they won’t jiggle their way out. I have four minis in the rack right now along with a Drobo with 10TB of disk. I’m going to be adding a 5th mini with a stackable disk drive, that’s why there’s double-high slot  still open on the mini side of the rack.

Cable management is an issue, mostly because of the power bricks long cables. I may fiddle with how I fold the cables into the shelves a bit more.

The whole thing plus a UPS and monitor/keyboard/mouse sits nicely on some steel shelves in our A/V equipment room at the museum. I still need to time how long the UPS runs. I’m only going to have the public web site minis on it.

I moved this blog from one of the Mac minis in my basement to the other (I’m trying to put everything on the newer one to free the other one up) yesterday. Originally I had been blogging using Plone (from about 2005-2007) and then moved to Wordpress. Moving the Plone part seemed like it was more work than I wanted to put in, so it’s goodbye to those posts.

I had seen this tidbit that Sean Gillies writes about in the recent OGC newsletter. My thoughts were along the lines of Sean’s. I never understood the big deal behind URNs.

EDIT: Forget the semi-rant, see the comments, and then go read about URI…

But in re-reading Sean’s post and the OGC news coming out of the June 2010 meetings, I think the terminology is a bit imprecise. Too bad the source document, 10-124r1 isn’t available on the OGC web site (promised for mid-July, I see) to see if the issue is in the document or in the news page. Here’s the news page version:

OGC Identifiers – the case for http URIs’

The OGC Members approved release of ‘OGC Identifiers – the case for http URIs’ [OGC 10- 124r1] as an OGC Whitepaper. .According to the current OGC policy either URNs or http URIs may be used in OGC standards. However, the use of http URIs (a) resolves some deployment challenges and (b) provides an opportunity for easier engagement with broader communities. So OGC should now consider taking the next step, and mandate the use of http URIs for persistent identifiers in OGC specifications. This whitepaper canvasses a number of issues around this proposal.

http URI Policy

The OGC Members approved the following as official OGC policy to be included in the OGC Policies related to OGC standards [OGC 06- 135rN]:

• OGC TC directs the OGC-NA that all new OGC identifiers issued for persistent public OGC resources shall be http URIs, instead of URNs
• New standards and new major versions of existing standards shall use http URIs for persistent public OGC resources to replace OGC URN identifiers defined in previous standards and versions, unless OGC- NA approves an exception

Operational Implications: OGC should carefully manage (maintain for the long term) the http://www.opengis.net domain and identifiers in this domain

So what’s wrong? Refer to RFC3986 (or the html version). Section 1.1.3 talks about URI, URL, and URN:

A URI can be further classified as a locator, a name, or both. The term “Uniform Resource Locator” (URL) refers to the subset of URIs that, in addition to identifying a resource, provide a means of locating the resource by describing its primary access mechanism (e.g., its network “location”). The term “Uniform Resource Name” (URN) has been used historically to refer to both URIs under the “urn” scheme [RFC2141], which are required to remain globally unique and persistent even when the resource ceases to exist or becomes unavailable, and to any other URI with the properties of a name.

A URN is a kind of URI. What is called an “http URI” is really a “just” a URL in RFC3986. And, a URN need not (or I should say “need no longer”) be something with “urn:” in the scheme. A URL could be a URN based on the last part of the definition above, “any other URI with the properties of a name”

Therefore, an “http URI” (from the OGC wording) can be either a URL or a URN, based on section 1.1.3 of RFC3986. Of course, the URN is really a URL with the additional uniqueness and persistence properties. So let’s just call OGC’s newly mandated URIs URLs.

There are two primary motivations for using RFC2141 URNs. One is as a globally unique name managed by some authority. The other is as a persistent identifier, sometimes used to map onto a URL with a resolver. The trouble with the latter is that URLs really work better in the first place, and I’m guessing that’s what 10-124r1 says.

So here’s what I think they should have said in the TC:

URI Policy

The OGC Members approved the following as official OGC policy to be included in the OGC Policies related to OGC standards [OGC 06- 135rN]:

• OGC TC directs the OGC-NA that all new OGC identifiers issued for persistent public OGC resources shall be http URLs, instead of RFC2141 URNs
• New standards and new major versions of existing standards shall use http URLs for persistent public OGC resources to replace OGC RFC2141 URN identifiers defined in previous standards and versions, unless OGC- NA approves an exception

Sorry to be so pedantic. Back in the day, there would have been half a dozen people at any given TC who would have been able to argue the finer points of this for hours….

(And I just figured out what the OGC-NA is. I guess it’s the “Naming Authority”.)

We’ve put our first QR Code up at the museum as an experiment. This seems like one way to deal with “nth level” information that might be interesting to some visitors but not to others. And it’s a way to make it easy for visitors to bookmark information for themselves.

Generating the QR Code isn’t hard. Google’s Chart api can do it. Basically, anything you put fullowing the “chl=” part of the URL here will generate a code: http://chart.apis.google.com/chart?cht=qr&chs=350×350&chl=http://think.random-stuff.org/

We’ve set aside a specific URL space for our codes to send people to. Each link will first take you to a short URL which will then redirect you to the actual URL relevant to the exhibit. The advantage of this is that we can also put up short, “human readable” links on the exhibits.  That will let us track how often the links are getting used and what mobile devices people are using. We’ve also put an explanatory page at the top level of that space.

There was a little debate about how much we should try to explain QR Codes on the exhibit itself. You can see what we wound up with in the detail photo. Basically we’ve decided that visitors will either (a) recognize the code and know what to do with it, (b) not recognize it and skip over it, or (c) ask someone. The “c” people can be given a printout of the explanatory page.

We’ve decided to introduce the codes slowly and in a way that hopefully doesn’t get in the way of people who don’t know what they are or don’t choose to use them.

Another question was whether we should format the linked pages in a phone-friendly format. The easy answer (because it requires no additional resources…) was not to do that. Phones are getting pretty good at reading full-blown web pages.

For further reading, here’s a March 2009 article titled QR codes in the museum – problems and opportunities with extended object labels

What QR Code app am I using? Right now on my iPhone 3G, I’ve got Barcodes. It’s got a huge number of one-star, negative comments but it works for me. The critical thing you need to know is that it only works on QR Codes, not regular barcodes, and at least with the 3G, you have to take the photo from about 18″ away and then use the app to zoom it to the right size.

I’m always on the lookout for low-cost, easily maintained computers to drive displays or do other odd jobs around the museum. The Mac mini is still the gold standard for me. But even with the educational discount, a Mac mini can seem like overkill. We have an exhibit coming up where there’s going to be a projection of a single, still image. Some projectors can take an image from a flash drive, but none of the ones we have laying around can do that. So it was either spend money on a projector that can, or get a computer to drive it. (We could also use a DVD player with a single frame video loop, but that wouldn’t give me full 1024×768 resolution on the projector.) So last week I bought an Eee Box B202 preloaded with Linux.

Initially it was a bust. I hooked it up to a monitor, turned it on, and it booted to a text login prompt. No nice multi-media GUI, nothing. Just login:

Googling around was no help. I couldn’t figure out how to make it start X Windows or anything. The OS that’s on there is called Red Flag Linux. It seemed to have everything it needed, but I don’t have enough Xorg fu to understand how it was supposed to figure out how to configure itself for the monitor I had.

It turns out that was a blessing in disguise. After a little more Googling, I found myself making a bootable USB stick with Ubuntu 9.10 on it. I wound up using the “From Linux” instructions from another Ubuntu box. I debated whether or not to use the netbook distro or the full desktop one. I picked the full one. It turned out to be the right choice.

I did mess with the B202’s BIOS a bit to figure out how to make it boot from the USB stick. I’m not sure whether any of that was strictly necessary, but I had changed a few things to try to get the original software running.

After that it was smooth sailing. Ubuntu booted right up, running from the USB stick. It helpfully presented the option of installing from the stick. Once I made sure it would be able to work with the ethernet and WiFi, I used the installer to reformat the drive and install Ubuntu.

Things went so well, I’m ordering another one. The first one’s going to run our projector. The next one’s going to act as a WiFi to ethernet connection sharing router.

This was my first experience with a LiveCD/LiveUSB/etc. linux. It’s probably unremarkable these days, but I’m impressed with how easy it was to do.

Update: I just got my 2nd Eee Box. This one came with Windows XP pre-installed. Here are the steps needed to install Ubuntu:

1. When it boots into ExpressGate, click the Exit icon, then hold down DEL to get into the BIOS setup
2. In BIOS->Tools: Disable ExpressGate, then hit ESC to exit
3. In BIOS->Boot->Hard Drives: hit + to make 1st drive = USB:SMI USB Disk
4. Plug in your Ubuntu USB stick
5. Hit F10 to save and exit

Now it will boot from the USB stick. At this point you can either boot into Ubuntu w/o installing (i.e. run from the stick) or install Ubuntu. If you’re unsure whether you want to go through with this before you try, then choose the first option. Note that it takes a longish time to boot from the stick.

Once it boots, you’ll see an icon in the upper left corner labeled “Install Ubuntu 9.10″. Double-click that to do an installation.

So, just to finish off what I’ve learned about newsyslog on Mac OS X…

In addition to figuring out how to deal with denyhosts, here’s how to set up Apache and Mailman log rotation.

I’m running standard Apache 2.2 that comes with Mac OS X 10.6. I like to keep all my virtual hosts in one place, so in this example they are all in /Users/web/. Each host gets a directory structure with it’s own name:

# ls -l www.example.org/
drwxrwxr-x   4 _unknown  _unknown   136 Jan 27 16:47 htdocs
drwxrwxr-x  80 _www      _www      2720 Mar 29 00:33 logs
-rw-r--r--   1 adoyle    web       1881 Feb 20 15:44 www.example.org.conf

The logs for each virtual host go into the logs directory for that host (access_log, error_log, rewrite_log). Ownership on the log files turns out to be important. I’ve found it works best for me if they are owned by the www user and group (or _www, they are essentially the same – something I need to understand the reason for someday).

In /etc/newsyslog.d/local.conf, the following lines deal with rotating logs for three virtual hosts. Using the ‘G’ flag lets you use ‘*’ and other shell wildcards in the file names. I think I could probably have collapsed these into a single line if I had used /Users/web/*/logs/*log instead. In this case, folding things up too much makes it less readable, I think.

The _www:_www takes care of preserving the file ownership after the logs are rotated. A count of 30 means keep around up to 30 old logs. $D0 means rotate daily at 0:00. The ‘B’ flag prevents the “Log file was rotated” message. Apache keeps a pidfile in /var/run/httpd.pid. If you send a kill -30 to the pid in that file, it will cause the equivalent of an ‘apachectl graceful’.

If you don’t provide the pidfile and proper signal number, the logs will rotate, new log files get created, but Apache won’t write to them because it’s still trying to write to the old ones.

# logfilename          [owner:group]            mode count  size  when   flags [/pid_file] [sig_num]
/Users/web/lists.example.org/logs/*log _www:_www 664   30     *    $D0     GBJ /var/run/httpd.pid 30
/Users/web/foo.example.org/logs/*log   _www:_www 664   30     *    $D0     GBJ /var/run/httpd.pid 30
/Users/web/www.example.org/logs/*log   _www:_www 664   30     *    $D0     GBJ /var/run/httpd.pid 30
#

With Mailman I had a slight problem. Mailman doesn’t use nice .log or _log names. It just uses names like bounce, error, post, qrunner, etc. I could have made an entry in the local.conf file for each one, but that seemed error-prone. What if later there’s a new version of Mailman that generates different log files?

My initial assumption was that I could use …/logs/* and newsyslog wouldn’t try to rotate logs it had already rotated. Guess again. After two days, my disk had nearly filled up with files ending in .bz2, .bz2.bz2, .bz2.bz2.b2z…, you get the picture. Luckily I noticed it before the disk did fill up. I got suspicious when my backups on the third day were 15GB bigger than the ones on the first day. I have no idea how many files actually got created. ‘ls’ was unable to produce a listing in the amount of time I was willing to wait. Luckily rm -rf did work. It took several hours to delete all the files.

I’m assuming newsyslog had gotten into a recursive loop right away when it ran at midnight and never stopped churning out files until I killed it.

So anyway, the moral of the story is, if you’re going to use wildcards, make sure they don’t match the rotated logs.

# logfilename          [owner:group]            mode count  size  when  flags [/pid_file] [sig_num]
/Users/mailman/logs/*[a-z]         mailman:_www 664   30     *    $D0     GBJ /Users/mailman/data/master-qrunner.pid 1
#

Mailman wants to be hit with a kill -1, so that’s what I used.

Things have been noodling along for a few weeks with my setup, so I think I have the kinks ironed out.

I just spent some time figuring out how to set up denyhosts on Snow Leopard. I’ve used denyhosts before, but never felt like I had things set up properly for Mac OS. Now I think I have it figured out, so here it is. This is for 10.6, your mileage may vary on earlier versions.

I had three goals – get denyhosts working, get it to start automatically at boot time, and to deal with rotating the logs.

1. Installation
Easiest first – installing denyhosts. Note that you need to be root to do this. Pretty much just follow the directions. These are the three main settings to worry about.

SECURE_LOG = /private/var/log/secure.log
LOCK_FILE = /var/run/denyhosts.pid
DAEMON_LOG = /var/log/denyhosts

Note that you also may need to create the file /etc/hosts.deny:

touch /etc/hosts.deny

Using touch will create a zero-length file if it’s not there. It won’t affect the contents if it is there.

2. Log rotation

Mac OS 10.6 uses newsyslog to rotate some log files (I’m not sure why, but apache logs don’t seem to be dealt with by newsyslog). To add your own to the mix, just put a file into /etc/newsyslog.d/ following the format for newsyslog.conf(5). I called mine local.conf

# logfilename          [owner:group]            mode count size when  flags [/pid_file] [sig_num]
/var/log/denyhosts                              640   5     *    $D0     J
#

The trouble is, this rotated the log just fine, but then denyhosts stopped logging because newsyslog essentially pulls the rug out from under denyhosts by moving the file.

One design difference between newsyslog and logrotate is the way they deal with notifying processes that logs have been rotated. Logrotate uses prerotate and postrotate scripts, which would be ideal for denyhosts. The way you start and stop it is with

daemon-control start
 
daemon-control stop

daemon-control stop actually sends a SIGTERM to the denyhosts process, but that won’t do any good in the newsyslog config file since once stopped, you need a command line to start it up again. So I decided to tweak the daemon-control script to do this. I replaced the start() function with the one here:

def start(*args):
    cmd = "%s --daemon " % DENYHOSTS_BIN
    if args: cmd += ' '.join(args)
 
    print "starting DenyHosts:   ", cmd
 
    while True:
        os.system(cmd)
        time.sleep(5)
 
        while True:
            pid = getpid()
            if pid >= 0:
                time.sleep(300)
            else:
                break

This just keeps daemon-control running rather than letting it exit after it starts denyhosts. The outer loop starts denyhosts running and later restarts it. The inner loop just waits until it sees the pid file go away. That’s a sure sign that denyhosts stopped running, most likely because of the SIGHUP it will get from newsyslog. Now all I needed to do was add the signal info to my /etc/denyhosts.d/local.conf /etc/newsyslog.d/local.conf file:

# logfilename          [owner:group]            mode count size when  flags [/pid_file] [sig_num]
/var/log/denyhosts                              640   30     *  $D0     BJ  /var/run/denyhosts.pid 15
#

I’ve also changed it to keep 30 days of logs, and added the B flag to prevent newsyslog from adding a line to the file saying it’s rotated the logs. Note that I changed the name to daemon-control2 so if I update denyhosts later, my changes don’t get clobbered.

3. Start at boot time

It turns out that modifying daemon-control to never exit is also just the ticket for running it under launchd. Launchd doesn’t work well on scripts that launch daemonized processes. It watches the script and notices that it’s exited, then tries to start it again.

I made a file called /Library/LaunchDaemons/net.hosts.deny.plist:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>Label</key>
    <string>net.denyhosts</string>
    <key>ProgramArguments</key>
    <array>
      <string>/usr/share/denyhosts/daemon-control2</string>
      <string>start</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
    <key>KeepAlive</key>
    <true/>
    <key>ServiceDescription</key>
    <string>Lauch denyhosts</string>
  </dict>
</plist>

Get it started with launchctl:

 launchctl load /Library/LaunchDaemons/net.hosts.deny.plist

My /etc/hosts.deny has about 8500 hosts in it right now. Many of those are probably from the denyhosts synchronization feature pulling in IP addresses from the central server.

Update 2010-03-26: Added some links and clarified some bits.

Update 2010-06-06: Note that /etc/hosts.deny must be present. denyhosts won’t create it.

Last week in Santa Barbara, during the ESIP Federation summer meeting, the NASA SPG hosted a Web Services technical session. I somehow got the job of lining up speakers and moderating the session, always a bit of a nail-biter. The session went well and we had some great speakers.

The presentations can be found on the SPG web site.

First up, I tried to set the stage a little bit, based on where I thought the audience was in terms of knowing about web services and REST. My biases probably showed through, but I really do think that anyone building web services that are meant to be generally accessible should not be using straight “SOA” (i.e. SOAP, WS-*, etc.) with no regard to REST. Furthermore, I think that REST wrappers around RPC-style interfaces is stopping short of where things could be.

Next, Josh Lieberman gave a presentation about OGC’s current mindset vis a vis REST. It looks to me like there is some critical thinking going on inside OGC on this topic. It remains to be seen whether there is also going to be any real motion towards specs that use REST the way it could be used. Look at some of Sean Gillies postings about APP for how OGC could benefit from REST. (Sean was, unfortunately just arriving in France and unavailable to come to UCSB).

Michael Burnett talked about ECHO. ECHO is a full-bore SOA implementation of a metadata clearinghouse for granule level NASA data. ECHO is actually testing out the REST waters a bit with some early experiments. Given when ECHO was initially designed, and the need to continue long-term, stable operation, I don’t think I can fault them for where they are today. There’s a lot to be said for keeping a stable API that can itself be wrapped in newer skins.

Switching gears a little, Jason Symonds from NOAA showed us how he’s been building a drought portal. His portal acts as a web services client to pull information from many other web sites into a single set of web pages. Along the way he’s had to develop a few web services of his own which he’s also making available via the portal.

Tyler Stevens demoed a new service offered by GCMD (his presentation is here, the portal itself is here), namely a web services discovery portal. GCMD has been a mainstay for dataset discovery for a number of years. More recently, they have been accepting submissions of web service descriptions that now can be found in the portal. What I really liked here is the way the services themselves can provide information about how to use them. For instance if you click on the WMS service link for this DataFed service entry, it brings you to a forms builder that helps you build WMS URLs.

Karl Benedict’s talk about the New Mexico Geographic Information System showed how they are developing a RESTful set of services to allow data upload and subsequent automatic generation of WxS services on that data. As the system has become easier to use, the amount of data being made available has been increasing by leaps and bounds.

The last talk came from the astronomy community, who seem to be a bit out in front of the earth observation community when it comes to a concerted effort to move to REST principles. Matthew Graham gave a great overview of what is going on in Virtual Observatories. I’m always a little surprised when I see how other communities really are not all that different from the ones I’ve been working in for years. When you get right down to it, I guess everyone has data ingestion, integration, storage, and service delivery problems, so I should not be surprised. The VO community seems to be tackling the problem with gusto and has been making good progress, from the look of it.

At the end, we had a discussion about the questions raised in the initial session description and also touched briefly on how NASA could work to maximize its benefit from being an OGC member (broadly speaking, not restricted to web services). Two whiteboard pictures (one, two) emerged, and there will be summary information posted soon, I think.

[Update: Summary meeting notes were just posted on the site. -- July 16]

Has anyone ever run a service discovery and orchestration scenario like the one on page 7 of this document in any setting other than a demo? How far away from being able to do this in “real life” are we?