Ubuntu on Eee Box B202

I’m always on the lookout for low-cost, easily maintained computers to drive displays or do other odd jobs around the museum. The Mac mini is still the gold standard for me. But even with the educational discount, a Mac mini can seem like overkill. We have an exhibit coming up where there’s going to be a projection of a single, still image. Some projectors can take an image from a flash drive, but none of the ones we have laying around can do that. So it was either spend money on a projector that can, or get a computer to drive it. (We could also use a DVD player with a single frame video loop, but that wouldn’t give me full 1024×768 resolution on the projector.) So last week I bought an Eee Box B202 preloaded with Linux.

Initially it was a bust. I hooked it up to a monitor, turned it on, and it booted to a text login prompt. No nice multi-media GUI, nothing. Just login:

Googling around was no help. I couldn’t figure out how to make it start X Windows or anything. The OS that’s on there is called Red Flag Linux. It seemed to have everything it needed, but I don’t have enough Xorg fu to understand how it was supposed to figure out how to configure itself for the monitor I had.

It turns out that was a blessing in disguise. After a little more Googling, I found myself making a bootable USB stick with Ubuntu 9.10 on it. I wound up using the “From Linux” instructions from another Ubuntu box. I debated whether or not to use the netbook distro or the full desktop one. I picked the full one. It turned out to be the right choice.

I did mess with the B202′s BIOS a bit to figure out how to make it boot from the USB stick. I’m not sure whether any of that was strictly necessary, but I had changed a few things to try to get the original software running.

After that it was smooth sailing. Ubuntu booted right up, running from the USB stick. It helpfully presented the option of installing from the stick. Once I made sure it would be able to work with the ethernet and WiFi, I used the installer to reformat the drive and install Ubuntu.

Things went so well, I’m ordering another one. The first one’s going to run our projector. The next one’s going to act as a WiFi to ethernet connection sharing router.

This was my first experience with a LiveCD/LiveUSB/etc. linux. It’s probably unremarkable these days, but I’m impressed with how easy it was to do.

Update: I just got my 2nd Eee Box. This one came with Windows XP pre-installed. Here are the steps needed to install Ubuntu:

  1. When it boots into ExpressGate, click the Exit icon, then hold down DEL to get into the BIOS setup
  2. In BIOS->Tools: Disable ExpressGate, then hit ESC to exit
  3. In BIOS->Boot->Hard Drives: hit + to make 1st drive = USB:SMI USB Disk
  4. Plug in your Ubuntu USB stick
  5. Hit F10 to save and exit

Now it will boot from the USB stick. At this point you can either boot into Ubuntu w/o installing (i.e. run from the stick) or install Ubuntu. If you’re unsure whether you want to go through with this before you try, then choose the first option. Note that it takes a longish time to boot from the stick.

Once it boots, you’ll see an icon in the upper left corner labeled “Install Ubuntu 9.10″. Double-click that to do an installation.

Posted in Museum, Random

newsyslog on Mac OS X

So, just to finish off what I’ve learned about newsyslog on Mac OS X…

In addition to figuring out how to deal with denyhosts, here’s how to set up Apache and Mailman log rotation.

I’m running standard Apache 2.2 that comes with Mac OS X 10.6. I like to keep all my virtual hosts in one place, so in this example they are all in /Users/web/. Each host gets a directory structure with it’s own name:

# ls -l www.example.org/
drwxrwxr-x   4 _unknown  _unknown   136 Jan 27 16:47 htdocs
drwxrwxr-x  80 _www      _www      2720 Mar 29 00:33 logs
-rw-r--r--   1 adoyle    web       1881 Feb 20 15:44 www.example.org.conf

The logs for each virtual host go into the logs directory for that host (access_log, error_log, rewrite_log). Ownership on the log files turns out to be important. I’ve found it works best for me if they are owned by the www user and group (or _www, they are essentially the same – something I need to understand the reason for someday).

In /etc/newsyslog.d/local.conf, the following lines deal with rotating logs for three virtual hosts. Using the ‘G’ flag lets you use ‘*’ and other shell wildcards in the file names. I think I could probably have collapsed these into a single line if I had used /Users/web/*/logs/*log instead. In this case, folding things up too much makes it less readable, I think.

The _www:_www takes care of preserving the file ownership after the logs are rotated. A count of 30 means keep around up to 30 old logs. $D0 means rotate daily at 0:00. The ‘B’ flag prevents the “Log file was rotated” message. Apache keeps a pidfile in /var/run/httpd.pid. If you send a kill -30 to the pid in that file, it will cause the equivalent of an ‘apachectl graceful’.

If you don’t provide the pidfile and proper signal number, the logs will rotate, new log files get created, but Apache won’t write to them because it’s still trying to write to the old ones.

# logfilename          [owner:group]            mode count  size  when   flags [/pid_file] [sig_num]
/Users/web/lists.example.org/logs/*log _www:_www 664   30     *    $D0     GBJ /var/run/httpd.pid 30
/Users/web/foo.example.org/logs/*log   _www:_www 664   30     *    $D0     GBJ /var/run/httpd.pid 30
/Users/web/www.example.org/logs/*log   _www:_www 664   30     *    $D0     GBJ /var/run/httpd.pid 30
#

With Mailman I had a slight problem. Mailman doesn’t use nice .log or _log names. It just uses names like bounce, error, post, qrunner, etc. I could have made an entry in the local.conf file for each one, but that seemed error-prone. What if later there’s a new version of Mailman that generates different log files?

My initial assumption was that I could use …/logs/* and newsyslog wouldn’t try to rotate logs it had already rotated. Guess again. After two days, my disk had nearly filled up with files ending in .bz2, .bz2.bz2, .bz2.bz2.b2z…, you get the picture. Luckily I noticed it before the disk did fill up. I got suspicious when my backups on the third day were 15GB bigger than the ones on the first day. I have no idea how many files actually got created. ‘ls’ was unable to produce a listing in the amount of time I was willing to wait. Luckily rm -rf did work. It took several hours to delete all the files.

I’m assuming newsyslog had gotten into a recursive loop right away when it ran at midnight and never stopped churning out files until I killed it.

So anyway, the moral of the story is, if you’re going to use wildcards, make sure they don’t match the rotated logs.

# logfilename          [owner:group]            mode count  size  when  flags [/pid_file] [sig_num]
/Users/mailman/logs/*[a-z]         mailman:_www 664   30     *    $D0     GBJ /Users/mailman/data/master-qrunner.pid 1
#

Mailman wants to be hit with a kill -1, so that’s what I used.

Things have been noodling along for a few weeks with my setup, so I think I have the kinks ironed out.

Posted in Mac, Random

denyhosts on Mac OS X

I just spent some time figuring out how to set up denyhosts on Snow Leopard. I’ve used denyhosts before, but never felt like I had things set up properly for Mac OS. Now I think I have it figured out, so here it is. This is for 10.6, your mileage may vary on earlier versions.

I had three goals – get denyhosts working, get it to start automatically at boot time, and to deal with rotating the logs.

1. Installation
Easiest first – installing denyhosts. Note that you need to be root to do this. Pretty much just follow the directions. These are the three main settings to worry about.

SECURE_LOG = /private/var/log/secure.log
LOCK_FILE = /var/run/denyhosts.pid
DAEMON_LOG = /var/log/denyhosts

Note that you also may need to create the file /etc/hosts.deny:

touch /etc/hosts.deny

Using touch will create a zero-length file if it’s not there. It won’t affect the contents if it is there.

2. Log rotation

Mac OS 10.6 uses newsyslog to rotate some log files (I’m not sure why, but apache logs don’t seem to be dealt with by newsyslog). To add your own to the mix, just put a file into /etc/newsyslog.d/ following the format for newsyslog.conf(5). I called mine local.conf

# logfilename          [owner:group]            mode count size when  flags [/pid_file] [sig_num]
/var/log/denyhosts                              640   5     *    $D0     J
#

The trouble is, this rotated the log just fine, but then denyhosts stopped logging because newsyslog essentially pulls the rug out from under denyhosts by moving the file.

One design difference between newsyslog and logrotate is the way they deal with notifying processes that logs have been rotated. Logrotate uses prerotate and postrotate scripts, which would be ideal for denyhosts. The way you start and stop it is with

daemon-control start

daemon-control stop

daemon-control stop actually sends a SIGTERM to the denyhosts process, but that won’t do any good in the newsyslog config file since once stopped, you need a command line to start it up again. So I decided to tweak the daemon-control script to do this. I replaced the start() function with the one here:

def start(*args):
    cmd = "%s --daemon " % DENYHOSTS_BIN
    if args: cmd += ' '.join(args)

    print "starting DenyHosts:   ", cmd

    while True:
        os.system(cmd)
        time.sleep(5)

        while True:
            pid = getpid()
            if pid >= 0:
                time.sleep(300)
            else:
                break

This just keeps daemon-control running rather than letting it exit after it starts denyhosts. The outer loop starts denyhosts running and later restarts it. The inner loop just waits until it sees the pid file go away. That’s a sure sign that denyhosts stopped running, most likely because of the SIGHUP it will get from newsyslog. Now all I needed to do was add the signal info to my /etc/denyhosts.d/local.conf /etc/newsyslog.d/local.conf file:

# logfilename          [owner:group]            mode count size when  flags [/pid_file] [sig_num]
/var/log/denyhosts                              640   30     *  $D0     BJ  /var/run/denyhosts.pid 15
#

I’ve also changed it to keep 30 days of logs, and added the B flag to prevent newsyslog from adding a line to the file saying it’s rotated the logs. Note that I changed the name to daemon-control2 so if I update denyhosts later, my changes don’t get clobbered.

3. Start at boot time

It turns out that modifying daemon-control to never exit is also just the ticket for running it under launchd. Launchd doesn’t work well on scripts that launch daemonized processes. It watches the script and notices that it’s exited, then tries to start it again.

I made a file called /Library/LaunchDaemons/net.hosts.deny.plist:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>Label</key>
    <string>net.denyhosts</string>
    <key>ProgramArguments>/key>
    <array>
      <string>/usr/share/denyhosts/daemon-control2</string>
      <string>start</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
    <key>KeepAlive</key>
    <true/>
    <key>ServiceDescription</key>
    <string>Lauch denyhosts</string>
  </dict>
</plist>

Get it started with launchctl:

launchctl load /Library/LaunchDaemons/net.hosts.deny.plist

My /etc/hosts.deny has about 8500 hosts in it right now. Many of those are probably from the denyhosts synchronization feature pulling in IP addresses from the central server.

Update 2010-03-26: Added some links and clarified some bits.

Update 2010-06-06: Note that /etc/hosts.deny must be present. denyhosts won’t create it.

Posted in Mac, Random

NASA Web Services session roundup

Last week in Santa Barbara, during the ESIP Federation summer meeting, the NASA SPG hosted a Web Services technical session. I somehow got the job of lining up speakers and moderating the session, always a bit of a nail-biter. The session went well and we had some great speakers.

The presentations can be found on the SPG web site.

First up, I tried to set the stage a little bit, based on where I thought the audience was in terms of knowing about web services and REST. My biases probably showed through, but I really do think that anyone building web services that are meant to be generally accessible should not be using straight “SOA” (i.e. SOAP, WS-*, etc.) with no regard to REST. Furthermore, I think that REST wrappers around RPC-style interfaces is stopping short of where things could be.

Next, Josh Lieberman gave a presentation about OGC’s current mindset vis a vis REST. It looks to me like there is some critical thinking going on inside OGC on this topic. It remains to be seen whether there is also going to be any real motion towards specs that use REST the way it could be used. Look at some of Sean Gillies postings about APP for how OGC could benefit from REST. (Sean was, unfortunately just arriving in France and unavailable to come to UCSB).

Michael Burnett talked about ECHO. ECHO is a full-bore SOA implementation of a metadata clearinghouse for granule level NASA data. ECHO is actually testing out the REST waters a bit with some early experiments. Given when ECHO was initially designed, and the need to continue long-term, stable operation, I don’t think I can fault them for where they are today. There’s a lot to be said for keeping a stable API that can itself be wrapped in newer skins.

Switching gears a little, Jason Symonds from NOAA showed us how he’s been building a drought portal. His portal acts as a web services client to pull information from many other web sites into a single set of web pages. Along the way he’s had to develop a few web services of his own which he’s also making available via the portal.

Tyler Stevens demoed a new service offered by GCMD (his presentation is here, the portal itself is here), namely a web services discovery portal. GCMD has been a mainstay for dataset discovery for a number of years. More recently, they have been accepting submissions of web service descriptions that now can be found in the portal. What I really liked here is the way the services themselves can provide information about how to use them. For instance if you click on the WMS service link for this DataFed service entry, it brings you to a forms builder that helps you build WMS URLs.

Karl Benedict’s talk about the New Mexico Geographic Information System showed how they are developing a RESTful set of services to allow data upload and subsequent automatic generation of WxS services on that data. As the system has become easier to use, the amount of data being made available has been increasing by leaps and bounds.

The last talk came from the astronomy community, who seem to be a bit out in front of the earth observation community when it comes to a concerted effort to move to REST principles. Matthew Graham gave a great overview of what is going on in Virtual Observatories. I’m always a little surprised when I see how other communities really are not all that different from the ones I’ve been working in for years. When you get right down to it, I guess everyone has data ingestion, integration, storage, and service delivery problems, so I should not be surprised. The VO community seems to be tackling the problem with gusto and has been making good progress, from the look of it.

At the end, we had a discussion about the questions raised in the initial session description and also touched briefly on how NASA could work to maximize its benefit from being an OGC member (broadly speaking, not restricted to web services). Two whiteboard pictures (one, two) emerged, and there will be summary information posted soon, I think.

[Update: Summary meeting notes were just posted on the site. -- July 16]

Posted in Geo

Service Discovery and Orchestration (in IEOS or elsewhere)

Has anyone ever run a service discovery and orchestration scenario like the one on page 7 of this document in any setting other than a demo? How far away from being able to do this in “real life” are we?

Posted in Geo

Dear Safari 4 developers:

My overall impression is that it’s slower than the beta and slower than Safari 3. But that’s gut feel, not with hard data. The “loading” bar is also far less obvious now, and I always find myself wondering whether Safari has stopped working because it takes a while to even start showing activity.

Posted in Mac, Random

Moving the server

Is there anything that’s more nerve-wracking than taking down a perfectly functioning server in order to do something with it? This morning I had to move two servers (a PowerMac G5 running Leopard Server, and a Mac Mini running 10.4) a whopping 6 feet in order to put them onto a dedicated power circuit. I also needed to install software updates.

Before I ever install any updates on a server, I clone the disk with SuperDuper or CCC. That means I also have to first shut down all the services and pull the system off the net, clone the disk, boot the clone to make sure it’s ok, boot the normal disk, do the update, and test everything. Coupled with having to move the computers, 5 disk drives, and a D-Link switch, I didn’t have a fun few hours this morning.  I didn’t update the Leopard Server machine because I couldn’t get the alternate disk to boot. It turns out that the Iomega portable drive I was using (I love these little disks!) wasn’t getting enough power from the G5′s front connector and needed to be plugged in the back. By the time I figured that out, it was too late, the museum staff people were coming in and I had to have the server running again. So I’ll have to do the update another day.

Now one of the remote users on the updated Mini is having trouble getting in via ssh. So is it due to the move, the upgrade, or something completely unrelated? Having tried a bunch of things and looked at the log files, I’m leaning towards “something else”.

The one good thing that comes from this kind of thing is that you learn whether all the services are properly set up to start at boot time.

Posted in Mac, Museum, Random

IFC.com in the house

The IFC Media Project came to town today, to hold a panel discussion at the MIT Museum. They brought in more equipment than we’ve ever had anyone bring in. We had to run an auxiliary 60 Amp power drop for their cameras, lights, and recording studio. 

The panel discussion was about crime reporting in various media outlets, primarily print, radio, and TV. Tucker Carlson, Juan Williams, Martin Baron, Candy Altman, and Josh Silver were on the panel. 

There’s not too much of a technology angle to this event, other than that it’s nice to have people come in who know what they are doing. They brought in all their own equipment and a huge staff. Setup started yesterday, went into the wee hours of the night, and resumed again at 5AM this morning. 

The event was held in our Innovation Gallery, where exhibits are meant to come and go. Right now there’s relatively plenty of floor space but we’re going to be putting in a few new exhibits soon so things are going to get tighter again.

Posted in Museum

Ode to the Mac Mini

Rumors about the death of the Mac Mini and now, the rebirth of the Mac Mini prompted me to post my own personal wish that the Mac Mini retain many of its current characteristics. At the MIT Museum, we use Mac Minis whenever we can. They are insanely reliable and are easy to place in just about any situation. Here are four different setups we are currently running on the first floor.

MIT & the Sea - Mac Mini in a box hung from the ceilingThis one is hung in a box near the ceiling. It’s been running for over a year. We set them all to reboot after a power loss, so we almost never need access. When I do need to do anything to it, I grab a ladder and plug in a keyboard & mouse.

CityCar - Mac Mini inside a small enclosureThis one has also been running for about a year. Prior to that it was sitting in a server room for about two years. The enclosure gets a bit warm. We used to have a Shuttle XPC inside but it failed after about 4 months. I think it was the heat.

CityCar - Mac Mini hung under table in wire basket.We have two display tables with baskets attached underneath. The Mini fits in the basket. We used to have two additional setups just like this. This photo is of the CityCar interactive exhibit. There’s another Mini at the other end of the floor in the MIT & the Sea interactive exhibit (below the Mini-in-a-box pictured above).

Holography - Mini placed on top of large projector, driving small projector.The latest one to be put into service is a Mini that we strapped to the top of a monster Panasonic projector. The Mini drives the smaller projector to the left in the photo. This Mini is running Vine Server so I can access it remotely. Once we’re done tweaking it, I’ll probably shut off the Vine access.

Our Admissions Desk also uses two Minis, one for the staff to access various admin tools, mail, etc. The other to drive a sign displaying admissions prices, welcoming groups, etc.

In January we’re going to be installing an exhibit developed by the Sociable Media Group at the Media Lab. So far it will have seven Mac Minis in it, as well as an iMac. There will also be a couple of Dells so I guess we’ll see how they hold up in comparison.

My wishlist for the Mini: Keep being ultra-stable, don’t get too heat sensitive. Faster graphics would be nice, but not at the expense of being more finicky.

PS – I have one in my basement at home as well, it’s running this blog, among other things. I think it’s about 3 years old. I have it on a small UPS to deal with short power outages, connected to the web via Verizon FiOS (also very reliable!).

Posted in Mac, Museum

Quick plug – some apps

Just a quick plug for some Mac applications that make my life a lot easier.

Namely. It’s a free application launcher for Mac OS X 10.5. I used to use MenuStrip in 10.4 but really only used it for the Quick Launcher feature. When I upgraded to 10.5, I found Namely, which does just one thing. You set a hot-key combination that lets you pop open the Namely window. Then type in a few letters of the application you want to launch, and hit enter. I find myself using the dock less and less these days.

Fluid. Another Mac OS X 10.5 only application. It lets you build a “site specific browser” i.e. a separate application out of a browser window. I’ve wrapped my Google Calendar in Fluid and just keep it running all the time.

VoodooPad Pro. I use it for two things. I have one document to keep track of stuff I’m doing. I have another that I lock with a password to keep track of all my passwords. I haven’t done any scripting with its built-in Lua script engine, but I keep meaning to…

FlySketch. I use this for screen grabs all the time. One great use is to capture those web receipts you get when you buy something or pay for something. I grab them with FlySketch and put them into VoodooPad Pro.

Li’l Snitch. A great little app that lets me know what’s happening on my net connection.

Posted in Random