July 2003 Archives

Sean Voisen over at Voisen.org has linked to 348North from his weblog. Thanks Sean!

It's great for me to hear about people who make a choice, go out on their own, and make it work. The fact that he's an Apple user and a concise writer puts him on my regular reading list.

Based on a request from one of the sites included in the cycles I've added an optional "daily" schedule to go along with the hourly schedules. For instance, Goatee is now on a daily schedule. You can check your site, or a site that you like, by going to the site list and clicking on the site's name.

At this point, I plan to leave the default schedule as hourly, but will obviously respect any requests to switch the schedule, or remove a site from the system altogether.

Edith Frost's website, one of those 348North News checks each hour, has a comprehensive list of the good and bad bots (in her opinion) that visit the site. I am relieved to see that 348North is on the good side.

When I set about to set up this aggregator, the last thing I wanted to do was to add significantly to anyone's bandwidth bill. That's why, instead of scraping, I limited the requests to just an RSS/XML summary file. I also implemented an eTag subsystem that queries for 304 Not Modified headers for those sites that support it.

I've been toying with recording bandwidth usage per site (each cycle's usage is already kept) so that site owners could decide for themselves whether they wanted 348North to continue to collect their feed. I need to work on that aspect -- I don't want to inconvienance others in my quest for selfish convienance.

I stumbled across the transcript of Conan O'Brien's 2000 speech to the graduating class at Harvard. It is quite funny, and also quite insightful. Unfortunately, it is also quite long so don't follow the link unless you have a few minutes to sit down and read it.

The most meaningful part:

I left the cocoon of Harvard, I left the cocoon of Saturday Night Live, I left the cocoon of the Simpsons. And each time it was bruising and tumultuous. And yet every failure was freeing, and today I'm as nostalgic for the bad as I am for the good. So that's what I wish for all of you--the bad as well as the good. Fall down. Make a mess. Break something occasionally. Know that your mistakes are your own unique way of getting to where you need to be. And remember that the story is never over.

I haven't had a chance to read Will Ferrell's 2003 attempt.

Let me tell you, I was excited about Gentoo Linux until this afternoon. That's just about when I started my reinstall of a Stage 1 install with a Stage 3 install that really wasn't any quicker. Perhaps that's my problem: I wanted the installation done somewhat quickly.

The background of this is that I have moved from 'nix to 'nix, looking for that which I feel comfortably up-to-date with at any given point in time. I want to be able to tell the computer, "Download expat!" and have it do all the patches, etc. for me. That's why I loved FreeBSD, and now why I am interested in Gentoo. Prior to the former and latter, RedHat graced my machines. Well, I use the word 'grace' loosely since my 9.0 installation wouldn't for the life of it work with SSLeay.

So back to the problem at hand, Gentoo is being a bear to install on my system -- and I've even RTFM! And followed it, seemingly step by step. Of course, I think I know what the problem is. I'm using software RAID 1 (mirroring) across two identical drives, the same partitions that I had with the RedHat install. There's a Boot + the Root on one disk, and Swap plus the Root mirror on the other. HDA1, HDA3 and HDA2 respectively. Having only ever heard of ReiserFS, I thought, "What the heck!" and selected that for the format of the Root partition. Boot is ext2 and Swap is, um, (linux) swap. So, bring on the troubles.

First off, it took me a moment or two to realize that I needed to manually select the gentoo doataraid from the LiveCD to even get the thing to a prompt. I figured that out on my own, thank you very much. Then, there was a problem with the DMA on the drives... I never did figure that one out. I assume the RAID/ReiserFS had something to do with it. That was the beginning of my troubles.

Next, after making it nearly up to making the kernal (from a Stage 1 install) I made the mistake of more'ing a binary which somehow changed the font mapping of the screen. Now the whole things was garbled. I switched over to another (not as pretty :) shell, logged in as root and chroot'ed, etc. into what I thought was the same shell settings as before but things didn't ever really work out after that. I'm sure I forgot something.

So, I started over. This time with Stage 3 which seemed to take just as long. Yet somehow I ended up screwing that install up too -- I installed things into the LiveCD directories instead of the Root and Boot partitions. I guess. Grub couldn't find the kernal I waited forever (on a 800Mhz PIII) to make... which was frustrating.

I guess I am used to my Mac where things 'just work', or my RedHat installation where things were pretty straight forward. Heck, Slackware wasn't that hard back in the day -- as I recall. But perhaps that isn't what Gentoo is targetting. It seems that the Gentoo project is targetting those folks who want to tinker and set every little bit and byte on the system to get the highest possible bang for their hardware buck. The reality check here is that I'm not one of those people (anymore). I just want it to work, work well, and at a speed that won't make me pull my hair out. As a reference, I'm writing this on a 500Mhz G3 iBook, which is plenty speedy for most everything I do. Of course, my Gentoo install needs to run MySQL queries quickly, and all the web jazz, but at least you know where I'm coming from.

All that being said, I've just run across this article talking about the Gentoo "Reference Platform" so perhaps I just wasn't starting in the right place. Next week I'll try again and see if my needs are met with what I think will be a great distribution for me.

I think my next job will be to make the index page static, instead of dynamic. I have the infrastructure in place to make the whole index static (i.e. html files instead of cgi files) but I decided to trade space on the server for a small fraction of load-time -- and backend load of course.

What I had in mind originally was a NetNewsWire type program that would follow me around, even when I was away from my Mac. So, a web based aggregator was the "simple" solution. The hard part was trying to figure out what options to make similar, and what to leave out. I could have done unread counts for each site for each user, but this would have increased the complexity a ton -- at least it would have increased the database size a ton. Therefore, I've left it out so far. I think the cycle idea is nearly as effective. If you haven't read anything for 6 hours, well, there you go. Read the last 6 cycles worth of info. Filter it by category, or just select the sites you want and everything will be filtered that way.

Also, the next steps will be notebooks, personal article comments, and direct entry creation for the different CMS solutions.

Of couse, I will always be open to suggestions.

I've been working on an aggregator project (like about everyone these days, it seems) for my personal use. Here it is for others to use. The system uses Perl and MySQL and I'd be more than willing to help those trying to figure out what works -- and doesn't -- with either.

I know, I know...we've been slacking on this website. Why don't YOU try to move (with a toddler and two dogs), go on vacation, live without a washer and dryer and keep up a website all at the same time! HA! I'm not complaining (I can hear Kevin saying, "Yeah right!"), but the fact remains that we have been extremely busy. Nevertheless, new pictures are up and we hope you enjoy each one of them. We're sure you'll agree that Hale Bop is just as cute as they come!

Goodbye, 348 North Line Street. You served us well.