Quoderat

Let me out!!!

In other words, as the Web replaces Microsoft Windows as the world’s favorite desktop/laptop software platform (it may be there already), what good is Open Source to ordinary computer user? Even if a web site happens to be built on Open Source software (like the LAMP stack), I’m still locked in:

• How can I move my address book and archived e-mail from Hotmail to Yahoo or GMail?
• How can I move my blog (with all postings and comments) from Blogger to Bloglines or WordPress?
• How can someone move her contact list and comments from MySpace to Facebook?
• How can a buyer in Yahoo’s auction thingy verify my reputation on eBay?
• How can I move my old flight plans from Aeroplanner to FBOWeb?
• How can I move my sales contacts and data from Salesforce.com to Highrise?
• How can I move my pictures with their tags from Flickr to Smugmug?

A crack of light under the door

These are huge problems, and the solution is probably going to have a lot more to do with Open Data than with Open Source. There are already a couple of minor successes:

• Blog reading sites almost universally support OPML import and export, so that you can save the list of blogs you read from one site and move it to another.
• Online wordprocessors and spreadsheets, of course, support the Microsoft Office formats and/or the OpenDocument formats and/or RTF and CSV.

That’s not much, though. Open Source (and its predecessor buzzword, Free Software) have been very important over the past couple of decades, giving us choices beyond the Microsoft/Apple duopoly that chained our desktops (and forcing the duopoly to open up a lot) and smashing the big-iron vendor cartel that owned our servers, but as the world shifts from desktop to web-hosted software, it can’t take us much further.

Your data is the next big battle

Open Source will still matter, of course (especially for server-side work), but it won’t be an important battle ground any more. Until we can convince (or force) web sites to embrace and standardize on Open Data formats — XML, JSON, or even CSV, as appropriate — we will be in some ways even more locked in than we were in the bad old desktop days. Celebrate Open Source’s victory, by all means, but get ready for an even bigger and bloodier battle over the next 10-20 years.

The elevator pitch

With REST, every piece of information has its own URL.

If you just do that and nothing else, you’ve got 90%+ of REST’s benefits right off the bat. You can cache, bookmark, index, and link your information into a giant, well, web. It works — you’re reading this, after all, aren’t you? Betcha got here by following a link somewhere, not by parsing a WSDL to find what ports and services were available.

Real best practices

If you want to do REST well (rather than just doing REST), you can spend 2-3 minutes after your elevator ride learning a few very simple best practices to get most of the remaining 10% of REST’s benefits:

Use HTTP POST to update information. Here’s the simple rule: GET to read, POST to change. That way, no body deletes or modifies something by accident when trying to read it.

Make sure your information contains links (URLs) for retrieving related information. That’s how search engines index the web, and it can work for other kinds of information (XML, PDF, JSON, etc.) as well. Once you have one thing, you can follow links to find just about everything else (assuming that you understand the file format).

Try to avoid request parameters (the stuff after the question mark). It’s much better to have a URL like

http://www.example.org/systems/foo/components/bar/

than

http://www.example.org/get-component.asp?system=foo&component=bar

Search engines are more likely to index it, you’re less likely to end up with duplicates in caches and hash tables (e.g. if someone lists the request parameters in a different order), URLs won’t change when you refactor your code or switch to a different web framework, and you can always switch to static, pregenerated files for efficiency if you want to. Exceptions: searches (http://www.example.org/search?q=foo) and paging through long lists (http://www.example.org/systems/?start=1000&max=200) — in both of these cases, it’s really OK to use the request parameters instead of tying yourself in a knot trying to avoid them.

Avoid scripting-language file extensions. If your URLs end with “.php”, “.asp”, “.jsp”, “.pl”, “.py”, etc., (a) you’re telling every cracker in the world what exploits to use against you, and (b) the URLs will change when your code does. Use Apache mod-rewrite or equivalent to make your resources look like static files, ending in “.html”, “.xml”, etc.

Avoid cookies and URL rewriting. Well, maybe you can’t, but the idea of REST is that the state is in the thing the server has returned to you (an HTML or XML file, for example) rather than in a session object on the server. This can be tricky with authentication, so you won’t always pull it off, but HTTP authentication (which doesn’t require cookies or session IDs tacked onto URLs) will work surprisingly often. Do what you have to do to make your app work, but don’t use sessions just because your web framework tells you to (they also tie up a lot of resources on your server).

Speculative stuff (skip this)

The strength of REST is that it’s been proven through almost two decades of use on the Web, but not everything that some of the hard-core RESTafarians (and others) try to make us do has been part of that trial. Stop reading now if you just want to go ahead and do something useful with REST. Really, stop! Some of this stuff is moderately interesting, but it won’t really help you, and will probably just mess up your project, or at least make it slower and more expensive.

[maybe some day] Use HTTP PUT to create a resource, and DELETE to get rid of one. These sound like great ideas, and they add a nice symmetry to REST, but they’re just not used enough for us to know if they’d really work on a web scale, and firewalls often block them anyway. In real-life REST applications, rightly or wrongly, people just use POST for creation, modification, and deletion. It’s not as elegant, but we know it works.

[don't bother] Use URLs to point to resources rather than representations. Huh? OK, a resource is a sort-of Platonic ideal of something (e.g. “a picture of Cairo”), while a representation is the resource’s physical manifestation (e.g. “an 800×600 24-bit RGB picture of Cairo in JPEG format”). Yes, as you’d guess, it was people with or working on Ph.D.’s who thought of that. For a long time, the W3C pushed the idea of URLs like “http://www.example.org/pics/cairo” instead of “http://www.example.org/pics/cairo.jpg“, under the assumption that web clients and servers could use content negotiation to decide on the best format to deliver. I guess that people hated the fact that HTTP was so simple, and wanted to find ways to make it more complicated. Fortunately, there were very few nibbles, and this is not a common practice on the web. Screw Plato! Viva materialism! Go ahead and put “.xml” at the end of your URLs.

[blech] Use URNs instead of URLs. I think even the hard-core URN lovers have given up on this now — it’s precisely the kind of excessive abstraction that sent people running screaming from WS-* into REST’s arms in the first place (see also “content negotiation”, above), and it would be a shame to scare them away from REST as well. URLs are fine, as long as you make some minore efforts to ensure that they don’t change.

[n/a] REST needs security, reliable messaging, etc. The RESTafarians don’t say this, but I’m worried that the JSR (the Java REST group) will. We already have a secure version of HTTP TLS/SSL, and it works fine for hundreds of thousands or millions of web sites. Reliable messaging can be handled fine in the application layer, since everyone’s requirements are different anyway, or maybe we want a reliable-messaging spec for HTTP in general. In either case, please don’t pile this stuff on REST.

So to sum up, just give every piece of information its own URL, then have fun.

Browser stats

I just took a peek at my server stats for megginson.com (I’m pretty lazy about following them) and had a huge surprise. I’ve adjusted these to exclude “Unknown”, which I assume are mostly spiders and blog aggregators:

• MS Internet Explorer: 42%
• Firefox: 37%
• Mozilla: 7%
• NetNewsWire: 6%
• Opera: 3%
• Safari: 2%
• Netscape: 1%

I cut off the list at 1%. MSIE is still in the lead, but it has suffered a huge drop from a few months ago — could it be that my ISP’s version of AWStats doesn’t recognize MSIE 7 and is lumping it with “Unknown”, or is there a chance that the movement to Firefox is becoming a stampede? The last time I remember changes this fast was when MSIE was crushing Netscape in the late 1990s.

Operating system stats

My site is not primarily a Linux or Open-Source software site, and it does not seem to attract a disproportionately high share of non-Windows users. Again, excluding “Unknown”, here is the OS distribution for visitors:

• MS Windows: 74%
• Linux: 15%
• MacOS: 12%

Linux might be a touch high here, but megginson.com is no SlashDot. Something’s going on — either Firefox is doing to MSIE what MSIE did to Netscape (knocking off a stale browser sitting smugly on its assumed monopoly), or, as I mentioned, it’s just a reporting glitch.

Recipe for pickling a web site

The original site was a hand-rolled LAMP implementation, but it was designed from the start to be amenable to a static copy. To pickle it, I started by doing a recursive slurp of the live site using wget (with the -m option) — that generated permanent, static HTML copies of the dynamic, database-driven pages on the site. At that point, I had an almost, but not quite perfect static copy of the site, because there were two things that wget missed:

1. Images referred to only in CSS stylesheets (such as the banner).
2. CSS stylesheets referred to by other CSS stylesheets.

It took only a few minutes to add all of that by hand, and the site was ready to go.

Why it worked

This will be old news to a lot of people reading, but a few simple advance steps (during site design) made later static preservation easy. Here’s what I did:

• Every page has its own URL, period, end of discussion. No AJAX, no POST.
• Every page (or at least, every page that we want to archive) is reachable, directly or indirectly, from the home page.
• Script names are not shown to the public, so there are no URLs ending in “php” (hint: exposed script extensions like “php”, “asp”, or “jsp” are signs of gross incompetence in web design).
• No web pages rely on exposed GET request parameters: for example, the URLs looked like /programme/presentations/123.html, not /programme/presentation?code=123, or even worse, /show-presentation.php?code=123.

And that’s it. Of course, if the site had included live forms, I would have had to remove those as well (and any links to them), but that wouldn’t have been much extra work.

On a final note, while the live site was hosted on an Apache server (the “A” in “LAMP”), the pickled site is hosted on a Microsoft IIS server. It made no difference at all — that’s the way Web standards are supposed to work.

Keeping it simple

For PHP-driven web sites, I’m a big fan of Smarty, which uses braces (”{” and “}”) to delimit template constructions. Braces have no special meaning to XML parsers (they’re just character data), so it’s possible to put a template expression inside an attribute value (for example), while keeping the template itself as well-formed XML and not requiring the elaborate paraphrastic expressions you need to set up attribute values in XSLT:

<p id="x-{$myvalue|escape}">Hello, world!</p>

Concurrent markup resurrected

Really, Smarty adds a second set of concurrent markup on top of the XHTML. Smarty constructs don’t have to balance with XML element boundaries, and with only a little care, I’ve never ended up with a Smarty template that wasn’t well-formed. JSP’s mistake was using something that looks like XML but isn’t quite, messing up parsers. Even the old SGML CONCUR feature would not have allowed markup inside attribute values. Sometimes there’s something to be said for using two different syntaxes when you’re trying to represent two different things.

It’s not about SOAP

Forget about the SOAP vs. REST debate for a second, since most of the world doesn’t care. Google’s search API let you send a search query to Google from your web site’s backend, get the results, then do anything you want with them: show them on your web page, mash them up with data from other sites, etc. The replacement, Google AJAX API, forces you to hand over part of your web page to Google so that Google can display the search box and show the results the way they want (with a few token user configuration options), just as people do with Google AdSense ads or YouTube videos. Other than screen scraping, like in the bad old days, there’s no way for you to process the search results programmatically — you just have to let Google display them as a black box (so to speak) somewhere on your page.

A precedent for widgets instead of APIs

An AJAX interface like this is a great thing for a lot of users, from bloggers to small web site operators, because it allows them to add search to their sites with a few lines of JavaScript and markup and no real coding at all; however, the gate has slammed shut and the data is once again locked away outside the reach of anyone who wanted to do anything else.

Of course, there are alternatives still available, such as the Yahoo! Search API (also available in REST), but how long will they last? Yahoo! has its own restructuring coming up, and if Nelson Minar’s suggestion (via Forrest) is right — that Google is killing their search API for business rather than technical reasons — this could set a huge precedent for other companies in the new web, many of whom look to Google as a model. Most web developers will probably prefer the AJAX widgets anyway because they’re so much less work, so by switching from open APIs to AJAX widgets, you keep more users happy and keep your data more proprietary. What’s an investor or manager not to like?

What next?

Data APIs are not going to disappear, of course. AJAX widgets don’t allow mash-ups, and some sites have user bases including many developers who rely on being able to combine data from different sources (think CraigsList). However, the fact that Google has decided that there’s no value playing in the space will matter a lot to a lot of people. If you care about open data, this would be a good time to start thinking of credible business cases for companies to (continue) offer(ing) it.

Update: Hacking the Google AJAX API (or, back to Web ‘99)

The AJAX API is designed to allow interaction with JavaScript on the client browser, but not with the server; however, as Davanum Srinivas demonstrates, it’s possible to hack on the API to get programmatic access from the server backend. I’m not sure how this fits withThis violates Google’s terms of service, and obviously, they can make incompatible changes at any time to try to kill it, but at least there’s a back door for now. Thanks, Davanum.

Personally, I was planning to use the Yahoo (REST) search API for site search even before all this broke, because I didn’t want to waste time trying to figure out how to use SOAP in PHP. I’m glad now I didn’t waste any time on Google’s API, and I’ll just keep my fingers crossed that Yahoo’s API survives.

Anti-Goals

I don’t intend this site as a competitor to the Internet UPC Database: my main goal is to let people share their own observations and opinions about consumer products (e.g. “these diapers don’t leak”, “brand X chocolate tastes better”), and to classify and, effectively, vote for products by tagging them, and the barcodes are only secondary to that. Likewise, sites like ScanBuy and Qode, which concentrate on using cell phones for on-the-spot price comparison, are also working in a different area: I want to give people a chance to share their own information, not provide point-of-sale information to them.

Definitely not a stealth startup

In his article Stealth Startups Suck, Bloglines founder Mark Fletcher wrote that “stealth mode for a web start-up is the kiss of death.” I’m taking Mark’s advice to heart: I first sketched out the idea for Newmatica Barcode on a notebook (the paper kind) in a cabin in Perce, Quebec on the Gaspe Peninsula 10 weeks ago, and now here’s a fully-functioning site for people to try. There has been no pre-announcement, no careful dispatch of advance information to industry mavens and investors, or anything like that — you, my blog readers, are the very first people outside my immediate family to hear of this project.

So please, try out the site, create an account, enter some products, do some commenting and tagging, and be patient if you encounter any bugs (I promise to fix them as fast as possible). I’m looking forward to hearing back from you soon.

Explicit state

First, a continuation preserves the entire state of a program, including the stack, instruction counter, local variables, etc. How much of that do you really need for a hypothetical travel web app? In reality, you probably need, maybe, 1-5 variable values to restore a previous state in the travel app, so why not just save those explicitly? It would be faster, more secure (less information being saved), and much easier to performance tune and debug (since no magic is happening behind the scenes). Save those variables in a database, in a hash table, in an XML or CSV file, in memcached, or wherever happens to be most convenient. You may be looking at under 100 bytes for each saved state, so if you really want to do this, it’s not going to hurt too badly.

REST

But do you really want to do this? Most of the discussion around REST has focussed on the use of persistent URLs and how to use HTTP verbs like GET, POST, PUT, and DELETE, but there’s another, perhaps more critical idea behind REST — that the resource your retrieve (a web page, XML document, or what-have-you) contains its own transition information.

Let’s say that you load a web page into your browser, load more web pages, then use the back button to return to the original one. Now, select a link. What happens? Did you browser have to go back to the original web server, which was using continuations (or other kinds of saved state) to keep track of the links from every page you visited, so that it won’t send you to the wrong one? Of course not. The web page that you originally downloaded already included a list of all its transitions (links), and intuitive things just happen naturally when you hit the back button.

The web is stateless, but web application toolkits maintain pseudo-sessions (using cookies, URL rewriting, or what-have-you) that makes them look stateful, and that makes programmers lazy. Obviously, you don’t want to stick information like ‘isauthenticated’ on a web page, since it could be forged; likewise, you don’t want to put a credit-card number there. But it is trivially simple to make sure that forms, like links, go to the right place even when you hit the back button — just make the transitions fully independent of any session stored on the server side. For example, consider this:

<form method="post" action="/actions/book-trip">
  <button>Book this trip!</button>
</form>

Presumably, the trip the person was looking at is stored somewhere in a session variable on the browser. DON’T DO THIS! As Gilad pointed out, someone hitting the back button might end up booking the wrong trip. There are gazillions of ways to push all of the context-sensitive stuff into the web page itself, where it belongs. Here’s one example:

<form method="post" action="/actions/book-trip">
  <label>Book your economy trip to Alaska!</label>
  <input type="hidden" name="destination" value="alaska"/>
  <input type="hidden" name="package" value="economy"/>
  <button>Book it.</button>
</form>

Here’s another:

<form method="post" action="/actions/book-trip/alaska/economy">
  <label>Book your economy trip to Alaska!</label>
  <button>Book it.</button>
</form>

This is 100% backbutton-proof and it’s trivially simple to implement. It took me a while after reading Gilad’s (admittedly, strawman) example to realize that there are people who do not develop webapps this way. If they do this much damage just with a Session stack, how much pain will they be able to cause with continuations?

The REST people are right, at least on this point: there’s no need to drive a continuation bulldozer through your webapp, when a little REST garden spade will work quite nicely (and won’t tear up your lawn in the process). Don suggests that there may be other, more legitimate use cases for continuations outside of web applications, and I have no reason to disagree, but I would like to look at them pretty carefully.

Omissions

There are a couple of problems with giving thanks to inventors, though. The first is that you inevitably leave people out. David, for example, thanked Microsoft for all of AJAX and modern web development in general. AJAX doesn’t consist solely of XMLHttpRequest, however; it also needs JavaScript and a DOM (both pioneered by Netscape) to manipulate the client display, and something like XML (W3C) or JSON (Douglas Crockford) to encode the messages. Most modern web developers also want CSS (Håkon Wium Lie and Bert Bos). And then, of course, there’s HTML and HTTP (Tim Berners-Lee). To illustrate my point, I’ve certainly left out a lot more that I could have included here, and have likely misassigned at least some credit.

Death of the inventor

The second problem is that it almost never makes sense to assign credit to individual people or companies. Who should get credit for SAX? Me, because I coordinated it? James Clark, because I based many of the ideas on his earlier SGML interfaces (and he suggested many of SAX’s features)? Tim Bray, because he thought up a catchy name? The other dozens of other xml-dev members who contributed most of the core ideas? The major software vendors who actually decided to use SAX, giving it credibility outside of the xml-dev community?

The same applies to just about every other technology we use. Not only do they depend on other innovations (the Web without TCP/IP? SAX without XML?), but the successful innovations are almost always simple and obvious, so their main value comes not from any particular technical brilliance but from the brute-force fact that lots of people use them — in other words, community-building is more important than innovation. Microsoft imitated Netscape’s level-0 DOM, and then the W3C standardized it so that it would work across browsers, then browser developers agreed to follow along, then web developers decided it was safe to start using it. Microsoft initially failed to build a community for XMLHttpRequest (which was a proprietary ActiveX component), so it languished mostly unused for years, until other browsers like Mozilla/Firefox, Safari, and Opera decided to support it as well — it was only then that we started to see a real community grow, and high-profile sites like Gmail and Google Maps take off. Tim Berners-Lee’s original HTML would hardly have mattered if the early Mosaic browser hadn’t shown how to make it user-friendly. Etc., etc. While Netscape introduced some good ideas like the DOM and Javascript, they also introduced some that flopped (does anyone else remember CORBA in the browser?) — no community of users, no success.

Thank the users

The moral of the story is that technology success is not something that a person or company gives to the net, but something that comes back from it, as if you threw a stone at a tree without knowing whether an avalanche of silver or of bird dung would shower down from the branches onto your head. A complex, brilliant idea with no users is worthless; a simple, mediocre idea with lots of users is a treasure.