Quoderat

Some people compare blogs (and mainstream media) to an echo chamber, constantly repeating and amplifying the same messages, but the echoes usually die out quickly. Not so, today, when I found this story on the planenews.com aviation news feed:

21 Feared Dead in Munich Crash.

About twenty one of the 44 passengers and crew of the British European Airways airliner which crashed yesterday near Munich carrying the Manchester United football team and many journalists are feared dead. About eight others are in hospital, seriously injured. Frank Swift, the former international goalkeeper, who had become a journalist, died in hospital.

I didn’t hear about any crash yesterday, but according to the Wikipedia article on Manchester United, there was a crash near Munich on 6 February 1958 that killed eight of the team’s players. In fact, when you follow the full story link in the posting, there is a story about the crash. The phrase “From the archive” is hidden in the deckline, but the dateline is “Saturday February 2, 2008″ (probably automatically updated by the site). There’s nothing else in the online version to indicate that this is an archived story from 7 February 1958, though a Brit would probably know that British European Airways ceased operations in 1974.

This is an easy mistake to make trying to keep up a blog of current events, and I don’t mean to suggest that the maintainer is stupid, or that I couldn’t do the same thing — in fact, next December, watch this spot for postings about an air attack on Perl Harbor.

Gunner William Henry Ranson (born 1843) has started a blog about his life in the ranks of Royal Artillery and as a civilian in Canada right after Confederation:

http://whranson.blogspot.com/

Gunner Ranson was my great-great-grandfather. After serving in the Royal Artillery during the 1860s, he ended up settling in Canada permanently in the 1870s. While many British officers kept diaries and wrote memoirs, very few men of the ranks did — although a good number could read and write, few had the inclination and the available time (and light) to do so — but my great-great-grandfather was an exception. While we don’t have the original diary, we do have a summary that he wrote later in life as a memoir, based on the lost diary, giving a working man’s view of both the British military and of later civilian life (often more brutal) in Victorian Canada.

My brother Tom has had the memoir for some years and has been trying to decide the best way to edit and publish it. In the end, he has decided to publish sections serially as a blog. I encourage anyone interested in British or Canadian history to read this. The blog format reminds me very strongly of the serial magazine publication common during the Victorian period.

Elliotte Rusty Harold is right that anonymity goes together with freedom, and I was happy to read his excellent posting How to Blog Anonymously. Rusty distinguishes three different kinds of anonymity — roughly “I don’t want to be embarrassed”, “I don’t want to be fired”, and “I don’t want to be hauled out of my bed by the secret police and shot” — and talks about the steps necessary to achieve each one.

Granted, anonymity has its ugly sides, like the disgusting online threats against Kathy Sierra and online abuse of Maryam Scoble, but it’s also sometimes the only conduit around the abusive authority of a government, employer, or even one’s peer group. As even Western democratic governments have become more authoritarian since 9/11, keeping these conduits open is more important than ever.

Granted, 99% or more of anonymous information is simply stupid or malicious, but if that’s the cost of freedom, it’s a relatively small cost to pay compared to the sacrifices our ancestors made to win us the freedoms in the first place.

Bob DuCharme, who is well known in the XML community, now has a weblog. Welcome.

[Update: more results]

[Update: some results left as comments; and more.]

Thanks to everyone who posted comments on my Password-Protected RSS challenge three weeks ago. It turned out that the vast majority of feed readers can handle HTTP basic authentication for feeds.

Of course, if a feed holds important, confidential information, basic authentication over HTTP won’t be enough — you need to be able to use HTTPS to encrypt your password and data in transit. I do not have an SSL/TLS certificate for megginson.com, so I moved my test over to a different site, newmatica.com. Here’s the URL for the same RSS file, this time over HTTPS (the user id and password are still “guest”):

https://www.newmatica.com/test/blog.rss

Will your feed reader allow you to subscribe to a password-protected RSS 2.0 feed when SSL/TLS is involved? Here’s what I’ve tested:

Liferea

Prompts for a username and password, and stores the password on disk as clear text.

Bloglines

Reports no feed found. However, will read the RSS file if the username and password are encoded in the URL, i.e. http://guest:guest@www.megginson.com/test/blog.rss

Straw

Fails: HTTPS not yet supported.

FeedValidator

Reports a 401 HTTP error (authorization required). Strips the username and password from the URL if they are provided.

Once again, I’ll be grateful for comments about other RSS readers.

Update: reports from readers

Once again, thanks to readers who have left reports in comments. Here are the results so far:

Sage

John Cowan and Tim Howland report success.

NewsFire

Peter Lacey reports success.

NetNewsWire

Brent Simmons reports success.

RSS Bandit

Dare Obasanjo reports success.

Opera (8.0.2)

Tony Coates reports success.

KDE Akregator

Douglas reports success.

FeedDemon

Brian R. Barker reports success.

Safari

Silas Hundt reports success, with username and password saved to the keychain.

I’ve noticed that Slashdot has started to include text ads in their RSS feed much more frequently, and I’m considering unsubscribing. I have no moral or ethical objection to text ads in RSS feeds, but this is the first time I’ve run up against that many text ads all in a row, and I find them ugly and rather annoying (on web pages, there’s more going on, and text ads don’t get in the way so much).

So, am I one of those people, often flamed in Slashdot comments, who expects something for nothing? How is Slashdot supposed to make money from providing an RSS feed to freeloaders like me? There are two reasonable answers to these challenges:

1. Slashdot strips links from their RSS feed, so if I want to follow a source or read comments, I have to visit the Slashdot site (and see their ads) anyway; and

2. in business, it’s OK to be selfish — I look for what I want, the other side looks for what they want, and if we cannot find any common ground we just wish each other luck and walk away. If I want an RSS feed without text ads, it’s OK for me to go looking for one; if Slashdot wants to put ads in their feed, it’s OK for them to do so. If there’s no common ground, then I simply stop reading Slashdot.

Of course, from an entrepreneurial point of view, that still leaves a problem: how can people make money directly from fulltext RSS feeds? Indirectly, of course, you can use them to promote yourself or your business, but what if the feeds are your business?

[More updates from comments.]

[Update: many more results, collected from comments. It looks like nearly every feed reader can handle at least HTTP basic authentication, which is good news for people planning to use RSS and Atom in government or enterprise.]

I’ve uploaded a small, static RSS file at http://www.megginson.com/test/blog.rss. The file is valid RSS 2.0 (or 2.01, if you prefer), and is served with the MIME type application/xml.

Here’s the kicker — it’s password-protected, using HTTP basic authentication. The username and password are both “guest”. Can your RSS software or webapp read this feed? Here’s what I’ve tested so far:

Liferea

Prompts for a username and password, and stores the password on disk as clear text.

Bloglines

Reports no feed found. However, will read the RSS file if the username and password are encoded in the URL, i.e. http://guest:guest@www.megginson.com/test/blog.rss

FeedValidator

Reports a 401 HTTP error (authorization required). Strips the username and password from the URL if they are provided.

Here are results gathered by others and left as comments for this posting:

CaRP

Antone Roundy reports success when the configuration line CarpConf(’basicauth’,'guest:guest’); is included. I am not certain if that line would apply to all feeds, or just to this one.

FeedDemon

Bill de hOra reports success both with this sample and with password-protected https feeds at his workplace.

RSSBandit

Dan Sholler reports success.

Sage

Tim Howland and John Cowan both report success, since Sage uses the browser’s built-in authentication support. M. David Peterson points out that anything built on top of Mozilla/Firefox or MSIE should work.

NewsFire

Peter Lacey reports success.

Thunderbird

Christof Hoeke reports success.

NetNewsWire

Ryan King reports success.

SharpReader

Roland Kaufmann reports partial success (with the username and password embedded in the URL).

iTunes

Robert Sharl reports success (though iTunes reports an error due to the lack of an audio enclosure).

Opera

Rijk van Geijtenbeek reports success.

KDE Akregator

Douglas reports success.

As weblog syndication moves into the mainstream, people are complaining more and more about how difficult it is for an average user to subscribe to a weblog (here’s a recent example, but it’s only one of many I’ve read). One problem is deciding who an average user is — a sixteen-year-old who can text message on a cellphone numberpad with her thumbs at 20 words per minute, for example, can probably find the little orange icon on a web page — but the bigger problem is that making weblog subscription too easy will destroy the biggest future benefits of RSS and Atom.

While most feeds today are general, public information (like Quoderat), more focussed, personal feeds are currently our best hope for a solution to phishing and spam. Unlike e-mail, blog feeds are highly resistant to these problems: I have to manually subscribe to a feed to start getting messages, and can unsubscribe whenever I want. Someone selling drugs claiming to lengthen part of my anatomy can write all the messages he wants, but I’ll never see them; a script kiddy in the Russian Republic can create hundreds of messages telling me to follow a link to update my bank account information, but I’ll never even know. As average users (whoever they are) deal with more and more spam, and hear about more and more phishing cases, they will start to mistrust all commercial e-mail (legit or not), and there will be an excellent opportunity for RSS and/or Atom to step in and offer government, business, and other organizations a new, trustworthy channel for communicating with the public.

But wait — what if we make subscribing to feeds too easy? What if it becomes trivially simple to add a feed to a user’s subscription list with a script or disguised link, without the user being fully aware? Users could end up with apparently legit feeds with titles like “Daily News” full of spam; even worse, a user could end up with a feed entitled “Lloyd’s Bank” with a message to — you guessed it — go and confirm her password information.

Remember that one of the big problems with e-mail came from trying to make it easier for users, say, by running attachments automatically. Let’s make sure that subscribing to RSS or Atom feeds stays at least a little bit difficult, even for grandma and grampa. A bit of work to subscribe to each feed will be our best defence.

I’ve just upgraded Quoderat to WordPress 1.5.1 and have installed the Spam Karma plugin, as recommended by Lauren Wood in a comment to my previous whining posting about comment spam. With luck, legitimate comments should now all appear immediately, while links to Online Poker should silently disappear. I’ll keep my fingers crossed.

I’ve already noticed that the default WP 1.5.1 theme is too narrow for source-code/XML listings, so the formatting on some of my older postings is messed up. I’ll work on that later. In the meantime, please let me know (somehow) if Spam Karma prevents you from leaving comments.

The idea of Cascading RSS (or aggregation aggregation) is so obvious that it has probably already been blogged to death or even implemented by well-known web sites; unfortunately, my short attention span ran out before I think up the right search words for Google, so I’ll pretend that it’s my own, original idea for now. We use RSS or Atom to tell people when a web resource has changed, but that can still involve polling dozens or hundreds of RSS files frequently. With only a few tiny tweaks, we could also use master RSS files to tell us when other RSS files have changed, cutting the polling by (potentially) orders of magnitude.

I can think of a couple of places where this approach could allow RSS into places where it hasn’t been able to go yet:

Information Management

A very enlightened company might realize that RSS gives it an excellent way to manage information from all of its divisions, branches, subsidiaries, partners, and so on. Everyone simply puts data (sales figures, inventory, projects, and so on) on the company intranet as XML data files (presumably with appropriate authentication and authorization requirements) and then uses RSS to announce when new information is available or old information has changed. If division X needs to monitor inventory data from division Y, it polls division Y’s inventory RSS file every 5 minutes to see if there’s anything new.

The problem is that the network will get messy if the company ends up with thousands of RSS files, and everyone is polling everyone else’s every 5 minutes, especially if some of them are on old, slow servers. To simplify things (and speed them up), the company could have one fast server that polls all the RSS files in the company and then produces its own RSS file with the most recent change dates for each one. Now, everyone can poll only that central server, but the divisions still own their own data. Of course, it would be possible to build this up in several cascading layers to avoid one RSS file with 1,000 entries.

Personal RSS

Sooner or later, we’ll have personal RSS for reporting information like credit card and bank transactions, as Tim Bray predicted almost two years ago. One of the biggest problems here, though, is that people might be reluctant to give personal passwords to online aggregators like Bloglines. People might, however, allow online aggregators to request the last-modified time of their credit card or bank feeds, and they could use these to build cascading RSS files, allowing users to reduce the amount of polling they have to do from home or on the road.

In other words, both advantages have to do with getting RSS into the business world (either B2B or B2C), not with improving the current blogosphere. I’ll look forward to finding out who has thought about this idea in more detail, or even implemented it.

Web pages, and especially weblogs, include apparently unnecessary links all the time. For example, is there really any need to link to Microsoft every time I mention the company’s name? Is anyone reading this posting going to follow the link (and if so, would that person have had trouble finding the site otherwise)?

Hub and Spoke

The best term I can think of to describe these links is hub URLs. They’re very much like airport hubs — connections from many smaller places feed into them, and often the only way to get from one small place to another is by passing through the hub as an intermediate point: for example, if I link to Microsoft and you link to Microsoft, someone can trace a route from my web page to your web page by changing planes, so to speak, at the Microsoft hub. One way to make the trip is to put http://www.microsoft.com/ into Technorati or a similar search engine that can supply ongoing results in an RSS or Atom feed, then read the postings that congregate around this hub URL in the blogsphere. The weblog postings are not linking to Microsoft so that you can find Microsoft; they’re linking to Microsoft so that you can find them. The nature of a hub URL is that the spoke web sites need it more than it needs any one of the spokes.

To take a less hackneyed example, here is a Technorati RSS feed of all weblog postings that link to Roy Fielding’s famous dissertation on web architecture. Granted, that’s not a very active hub URL, but still, all of the postings that link there form a community of interest, and a RESTafarian will almost certainly want to subscribe to such a feed. I expect that, more and more, the blogsphere will start grouping itself around hub URLs at least as much as it groups itself around individual personalities today.

Travel agencies

So far, so good. Search engines, the travel agencies of the web and blogsphere, already know how to take advantage of these hub URLs, as in the Technorati example I just cited above. Unofficial rumour has it that Google, for example (there I go again with a hub URL), makes great use of hub URLs for determining the relevance of search results. In fact, the whole push towards tags and folksonomies by sites like Technorati, Flickr, and del.icio.us is really an attempt to set up their own hub URLs.

In Technorati’s case, the travel agent wants not only to plan trips but to own the airport hub itself: that’s why they’re encouraging bloggers to link to the tags section of their site, making URLs like http://www.technorati.com/tags/web into hub URLs that are entirely under their control; it does not seem likely that their competitors will go along with that idea, though.

Castles and Boroughs

On problem is that the most popular URLs might end up becoming not only hubs but castles. Castles are cute tourist attractions today, associated mainly with pseudo-medieval romantic kitsch like knights and tournaments, but in the Middle Ages they were often instruments of oppression. While free landowners may originally have congregated around them for protection, they often lost their freedom (either by choice or coersion) and became feudal serfs, little more than the property of the powerful thugs who controlled the castles. If we start building our weblogs and sites in clusters around powerful hub URLs the way that free peasants built their huts around castles, are we risking the same fate?

Castles don’t show up automatically whenever people congregate together, of course. The alternative is the borough. Most of us in the developed world crowd together in suburbs, towns or cities, the ideological descendants of the boroughs, so that we can share services like water, electricity, roads, and shopping. While we have to make some compromises to live in close proximity, we do not have to give up fundamental freedoms the way that serfs around a castle did. The reason for that is that most economically-advanced countries have cities that are governed democratically rather than by a single strongman like a feudal lord; even in the Western European Middle Ages, boroughs enjoyed many freedoms and privileges, and were at least partly self-governing. So, getting back to the blogsphere, the question is this: do we want our hub URLs to be more like castles or boroughs?

This is an important question, because it is not farfetched to suggest that the owners of the most popular hub URLs could eventually start limiting the rights of the sites or blog entries linking to them. The entertainment industry has already had great success shutting down Bittorrent trackers, which simply link to files rather than actually hosting them; several courts have issued rulings against deep linking, like this one in Munich in 2002. Even when specific rulings are later overturned, it should be clear that linking is not off limits for legal action, and it is not impossible to imagine a future where someone has to agree to restrictive terms of service or even pay for the right to link to a popular hub URL like a Technorati tag or the Microsoft web site.

Wikipedia-boro

I have already suggested that Wikipedia would be a good source of subject codes, and in essence, that means using Wikipedia URLs as hub URLs. Wikipedia is not the only choice, of course, but it seems to be a particularly good one for a few reasons:

1. it is a collaborative site where anyone can add new potential hub URLs and modify the information in the pages they point to
2. our rights to use it now and in the future are guaranteed by the Gnu Free Documentation License (though to be strictly pedantic, that applies to the content rather than the URL itself)
3. linking to the Wikipedia is more likely to give you a fair description of a subject than linking the subject’s own website (think of the difference between a politician’s own web site and the Wikipedia article on the politican, and you’ll see what I mean)

If enough people start linking to Wikipedia articles in their weblog postings, topic-based RSS or Atom feeds will become very easy: for example, Feedster will happily give you an RSS feed of weblog postings linking to the current U.S. President Bush or a feed with postings that explicitly link to the country Canada: presumably, these articles are treating these topics as major subjects, rather than just mentioning them in passing, so the contents of the search feeds should be highly relevant (imagine how many false hits you’d get from mailing address, etc., just searching for the word “Canada”).

[Updated] Over in my aviation weblog, I find myself more and more linking to Wikipedia whenever I’m discussing a concept, person, place, or anything else that doesn’t have its own, canonical home page. If, as I suspect, lots of other bloggers are doing the same, then links to Wikipedia articles may soon be the blogsphere’s answer to subject codes.

News wire services like Reuters or Dow Jones put a lot of time and money into maintaining long lists of subject codes to attach to their news products. Unlike the simple categories used in blogs, subject codes tell you not just that an article is about (say) computer technology, but that it is about specific companies, industries, people, places, and concepts. News customers use the codes to classify stories automatically, routing them to the appropriate editorial sections, displaying them on trading screens, sorting them into categories on web sites, or using them to improve searches. The providers are constantly sending out updated lists, keeping their customers’ technical departments very busy.

Should weblogs be using some kind of subject code (beyond categories)? Some areas already have standard identifiers that we could use, such as ICAO codes for airports, UPCs for retail products, ISBNs for books, CUSIPs for financial instruments, or ISO codes for countries, languages, and currencies. However, each of those requires some surrounding context: you need not only the code, but some indication that it refers to a currency or an airport. They’re also managed by central authorities, making them less attractive to the weblog community.

Enter Wikipedia. If I’m posting about Washington the U.S. state, I can link to the Wikipedia article about the state; if I’m posting about Washington the U.S. president, I can link to the article about the president; if I’m posting about Washington the U.S. capital, I can link to the article about the city; and if I’m using the word Washington by metonymy to refer to the U.S. government, I can link to the article about the government.

Bingo — subject codes, just like the big newswires use, only a lot more useful and totally open. I can link to abstraction subjects like love or communism or to time periods like the middle ages just as easily as I can link to concrete people, places, or things; if there’s not already a Wikipedia article on my subject, I can always start a stub. If people keep linking to Wikipedia, search engines like Technorati and aggregators like Bloglines might start taking advantage of those links to do some automatic categorization, right down to offering links to other postings on the same subject (”Click here for other postings about Open Source“). Once people know the search engines are doing that, they’ll be bound to link to Wikipedia even more than they already are, creating a virtuous circle where both Wikipedia and the blogsphere become more valuable.

Of course, like anything that people actually do in the web (as opposed to drawing-board architectures that never get implemented), this approach is far from perfect. Once the search engines are paying attention to Wikipedia links, some people will deliberately include misleading links to have their weblog entries miscategorized, though rankings like Technorati’s should help make sure that the most relevant ones stay near the top of the list. Furthermore, Wikipedia URLs do change, especially for the sake of disambiguation, so the Wikipedia URLs will never be 100% accurate as subject codes. And finally, the Wikipedia project itself could shut down, leaving all of the subject codes orphaned. Still, since linking to Wikipedia is something many of us do anyway, it looks like a good, quick-and-dirty webby alternative to the news industry’s subject codes — it might even work better.

Update: James Tauber posted the same idea with slightly different language back in October, and has just put up a followup.