I leave Ottawa tomorrow to fly down to Boston and get ready for XML 2007, which begins Monday morning. If the weather holds, I plan to head down in my own Piper Warrior, but I have a refundable airline ticket just to be safe.
The planning committee has been working on this conference for a full year now, and we have ended up with a very strong list of speakers representing companies big and small, both vendors and users. Take a look!
Update: the site shopping cart is broken, and doesn’t properly remove items from the total owing — too bad.
Here’s one easy way: via TechCrunch, Deutsche Grammophon, the gold standard in renaissance/ baroque/ classical/ romantic/ orchestral/ opera/ etc. music (often confusingly referred to collectively as “classical”, roughly equivalent calling all popular music since 1890 “rap”), will start selling their catalogue as unprotected MP3s at midnight German time tonight (6:00 pm in New York City) at their new site dgwebshop.com.
As a teenager in the late 1970s, I used to visit the House of Sound in Kingston (Canada), where they had thousands of DG records — probably most of the catalogue — packed in tight on on shelves lining a wall of the store. I couldn’t always afford them, but I loved being able just to pull them out and take a look at the covers of the different famous recordings. These days, the so-called classical music section of any but a couple of specialized stores in big cities like New York or London have maybe one or two rows of worthless classical-pop compilations hidden behind the DVDs of TV series nobody watched in the 1980s — no wonder people don’t shop at record stores any more.
We tech types have been claiming for a while that music companies could make more money selling unprotected digital music, so here’s the test. I plan to give them a lot of my own money if the site actually works, though I should note a couple of caveats:
I hope the site can handle the traffic. Rock on, Deutsche Grammophon!
The XML 2007 conference (Boston, 3–5 December) now supports both personal schedules and recommendations, thanks to Edd Dumbill and Expectnation.
If you’re planning to attend the conference (and I hope you are), you can create a free account (or use your OpenID), then visit the Conference schedule page. If you’re logged in, you’ll see a star in the top right corner of each slot. If you click on the star, it will turn yellow, and the presentation will be added to your personal schedule.
The web site uses people’s schedules to set up recommendations. For example, at the time I’m writing this, you can visit the page for Stewart Taylor’s and Adam Lee’s presentation XML and XPath in the Wild (analyzing the XML and XPath actually found on the web and in Open Source projects) and get a list of these recommendations for other presentations to attend:
You don’t have to be registered for the conference to do this — feel free just to poke around and see if you can put together a personal schedule that tempts you to come to balmy Boston in December.
(Update: I left out a lot of reviewers because of a last-minute editing keystroke gone horribly wrong — with luck, everyone’s listed now.)
The speakers for XML 2007 (Boston, 3–5 December) are listed on this web page — it’s well worth a quick skim, to see the range of people and companies represented this year.
However, beyond the IDEAlliance staff and and contractors, and the speakers, there are a lot of other people (all volunteers) who help to make a conference like this happen. I’d like to use this posting to thank them publicly, and hope desperately that I haven’t left anyone out. The order of names in each section is not significant.
For North America, it’s 1-800-816-2237.
The phone number is not available anywhere on Acer’s support sites (supposedly, it’s hidden somewhere in the Windows XP control panel, but you can’t get that with a broken computer), so I thought a nice, Google-friendly posting might be in order to help anyone else looking at a broken Acer notebook.
Probably because the number’s so well hidden, I got through instantly to a human being, who (a) seemed to be based in North America rather than overseas, and (b) knew what he was talking about, or at least was able to get answers.
I’ve recently been highlighting some of the presentations from the upcoming XML 2007 conference (Boston, 3–5 December). For the others, I’ve written a little bit about what I (and other members of the planning committee) found interesting and relevant about the proposal. In this case, though, I’m comfortable letting the title speak for itself:
A Lightweight Approach to Building the Department of Defense’s Semantic Web: Can Mashups Bring the “Wild, Wild Web” to the Warfighter? (Mary Ann Malloy and Rosamaria Morales, MITRE Corporation)
The track at XML 2007 (Boston, 3–5 December) that I often refer to simply as the “Document track” is actually called “Documents and Publishing.” That’s an important distinction, because publishing — even using a text-y format like XML — doesn’t have to involve producing a document of any kind, print or online.
An excellent illustration of that point is Matthew Browning’s and Robin Doran’s presentation, BBC iPlayer Content production: The Evolution of an XML Tool-Chain. Matthew and Robin talk about how, once the team stopped thinking about their information in relational database terms and started thinking in XML terms (RelaxNG, to be specific), it became a lot easier to manage changes and improvements to the project, which lets viewers watch shows they’ve missed from the last seven days of BBC television. It worked so well, in fact, that two other projects became redundant.
That’s not to say that it was necessarily an easy transition, but in the BBC’s case, at least, the rewards far outstripped the cost. That’s precisely the point of our 2007 conference them, “XML in Practice” — we’ve selected many papers that emphasize what did and didn’t work in real, large-scale projects, rather than focusing on new specs and prototypes, to help people make the same kinds of decisions in their own projects. XML won’t always make sense — in some cases, it will never make sense — but learning from others’ experience is a much better guide than speculation.
Earlier postings:
I didn’t have time to look at the OpenSocial API yesterday, so I’m continuing today looking at the data format for the last major area, persistence data.
My first impression of the persistence data API is that it doesn’t belong in v.1 of OpenSocial — unlike the member/friends and activities APIs, it doesn’t seem to be solving a core problem for social-site app writers (I have no way to get at a friends list except through the site’s API, but I can store my own data, thanks). I can see only two reasons that it’s here, neither of them very admirable:
I’ll give Google the benefit of a doubt and assume that it’s a vision thing, but that’s still very unhealthy — specs should solve the real problems of the present, not the speculative problems of the future, especially bare-bones v.1 specs like this.
Now that that’s out of my system, let’s take a look at what you get back from a URL like http://{DOMAIN}/feeds/apps/{appId}/persistence/global (and its many variants). From the spec, here’s what you get when you request a single piece of information from the API:
<entry xmlns='http://www.w3.org/2005/Atom'> <title type="text">somekey</title> <content type="text">somevalue</content> </entry>
Or, in non-XML terms,
$globals{'somekey'} = 'somevalue'
That comes from a URL like http://{DOMAIN}/feeds/apps/{appId}/persistence/global/somekey which requests a single value. Using the first URL mentioned gets you a feed of name=value pairs, sort-of like an associative array:
<?xml version='1.0' encoding='UTF-8'?>
<feed xmlns='http://www.w3.org/2005/Atom'>
<id>http://sandbox.orkut.com:80/feeds/apps/02864641990088926753/persistence/global</id>
<updated>2007-10-30T20:53:20.086Z</updated>
<title>Persistence</title>
<link rel='http://schemas.google.com/g/2005#feed' type='application/atom+xml'
href='http://sandbox.orkut.com:80/feeds/apps/02864641990088926753/persistence/global'/>
<link rel='http://schemas.google.com/g/2005#post' type='application/atom+xml'
href='http://sandbox.orkut.com:80/feeds/apps/02864641990088926753/persistence/global'/>
<link rel='self' type='application/atom+xml'
href='http://sandbox.orkut.com:80/feeds/apps/02864641990088926753/persistence/global'/>
<generator version='1.0' uri='/feeds'>Orkut</generator>
<entry>
<id>http://sandbox.orkut.com:80/feeds/apps/02864641990088926753/persistence/global/somekey</id>
<title>somekey</title>
<content>somevalue</content>
<link rel='self' type='application/atom+xml'
href='http://sandbox.orkut.com:80/feeds/apps/02864641990088926753/persistence/global/somekey'/>
<link rel='edit' type='application/atom+xml'
href='http://sandbox.orkut.com:80/feeds/apps/02864641990088926753/persistence/global/somekey'/>
</entry>
</feed>
There’s only one entry in the spec’s example, but there could be a lot more. Basically, this is the equivalent of something like
$globals = { 'somekey' => 'somevalue' }
The comparison isn’t quite fair, because there are also some links explaining what you can do to modify this information, etc., but it still seems like a lot of markup for not much value (pun intended). I wonder if this would be a good place to use JSON instead of Atom+XML? After all, the serious apps will be doing their own data storage anyway, and the client-only apps will probably use a JavaScript API that hides the Atom from the developer.
As hinted at, at least, in my URL posting, there are several different data scopes:
It seems like a reasonable division of scope, especially since the app can’t get anything out that it didn’t put in.
I do believe that, eventually, many web apps will be about to outsource storage as a service instead of having to maintain their own databases and database clusters — in fact, Amazon’s S3 and its competitors already provide precisely this service, though they might not be optimized for a lot of name=value look ups. I’m surprised though, that this could be considered a key feature of a social app spec, when so much else was left out.
As I mentioned in an earlier posting, for the XML 2007 conference this year (Boston, 3–5 December) the number of submissions on WS-* topics was way down, while topics like REST, microformats, and mashups are starting to sneak their way from the web into the enterprise world. However, the WS-* submissions we did get were very good, and one that especially grabbed our attention was Mark O’Neill’s Case Notes from a Vulnerability Assessment of a Bank’s Web Services.
Mark had the enviable opportunity to be part of a team of white-hat crackers, trying to break the security of web services at an unnamed bank — fortunately, he’s been allowed to share his findings with us. After listing vulnerabilities ranging from data smuggling in CDATA sections to SQL injection, he mentions that “the bank’s attempt to apply preventative security measures, such as SSL and XML Schema validation, actually proved to provide a false sense of security, and in fact introduced a number of security vulnerabilities of their own.”
And lest the Restafarians get too smug, he found lots of vulnerabilities in REST as well as WS-*. Beyond Mark’s talk, it’s also worth noting that apps using OpenSocial, which is REST-based, have already been cracked twice, so no one in charge of protecting a net-based API (REST or WS-*) should breathe too easy.
See you in Boston.
Earlier postings:
This is the third part of a series where I’m working through the OpenSocial specs as I write — that means that I haven’t preread and predigested this stuff, but am creating a record of how I approach a new set of specifications and try to understand them. First, I looked at the basic URLs for data access, since they provide the best high-level description of the OpenSocial capabilities (read-only info on members and their friends, read/write info on a member’s activity notifications, and a simple data-storage API). Next, I looked at the data format for the most important content, the member profile and friends lists. This time, I’ll look at the format for activity notifications, which is also based on the Atom syndication format.
To get a list of a member’s recent activities (uploaded a photo, poked a friend, got a new job, or stuff like that, I guess) an OpenSocial application uses the URL pattern http://{DOMAIN}/activities/feeds/activities/user/{userId} according to the specs, though I suspect that might be intended to be http://{DOMAIN}/feeds/activities/user/{userId} for consistency with the other data-access URLs — it’s hard to be certain. The host should return an Atom feed of activities, like this template example lifted from the spec:
<atom:feed xmlns:atom='http://www.w3.org/2005/Atom'
xmlns:openSearch='http://a9.com/-/spec/opensearchrss/1.0/'
xmlns:gact='http://schemas.google.com/activities/2007'>
<atom:id>http://www.google.com/activities/feeds/activities/user/userID/source/sourceID</atom:id>
<atom:updated>1970-01-01T00:00:00.000Z</atom:updated>
<atom:category scheme='http://schemas.google.com/g/2005#kind'
term='http://schemas.google.com/activities/2007#activity'/>
<atom:title>Feed title</atom:title>
<atom:link rel='alternate' type='text/html' href='http://sourceID.com/123'/>
<atom:link rel='http://schemas.google.com/g/2005#feed'
type='application/atom+xml'
href='http://www.google.com/activities/feeds/activities/user/userID/source/sourceID'/>
<atom:link rel='http://schemas.google.com/g/2005#post'
type='application/atom+xml'
href='http://www.google.com/activities/feeds/activities/user/userID/source/sourceID'/>
<atom:link rel='self' type='application/atom+xml'
href='http://www.google.com/activities/feeds/activities/user/userID/source/sourceID'/>
<atom:author>
<atom:name>unknown</atom:name>
</atom:author>
<openSearch:totalResults>1</openSearch:totalResults>
<openSearch:startIndex>1</openSearch:startIndex>
<openSearch:itemsPerPage>25</openSearch:itemsPerPage>
<atom:entry>
<atom:id>http://www.google.com/activities/feeds/activities/user/userID/source/sourceID/a1</atom:id>
<atom:updated>2007-10-27T19:41:51.574Z</atom:updated>
<atom:category scheme='http://schemas.google.com/g/2005#kind'
term='http://schemas.google.com/activities/2007#activity'/>
<atom:title>Activity title</atom:title>
<atom:link rel='self' type='application/atom+xml'
href='http://www.google.com/activities/feeds/activities/user/userID/source/sourceID/a1'/>
<atom:link rel='edit' type='application/atom+xml'
href='http://www.google.com/activities/feeds/activities/user/userID/source/sourceID/a1'/>
<gact:received>2007-10-27T19:41:51.478Z</gact:received>
</atom:entry>
</atom:feed>
There’s a lot of front-matter in this, so it’s hard to realize at first glance that it lists only a single activity (in the atom:entry element near the bottom). The entry itself uses mostly standard Atom elements, except for one extension element from the Google activities namespace, giving the date that the notification was received (received date is also important in the news industry, so maybe this is something Atom needs to add to its core). Other than that, the activity itself is easy enough to understand: it has a unique id, a couple of dates, a title (which seems also to serve as the sole description), and web links for viewing and editing.
Unlike the member and friends info, which was read-only, OpenSocial allows apps to post new activities and edit or delete existing ones, but only in what is called a “source-level feed” — that’s a list of a user’s activities limited to a single source (which, I assume, is the application), using the URL pattern http://{DOMAIN}/activities/feeds/activities/user/{userId}/source/{sourceId} (which, again, may be a typo with an extra “activities” path element at the beginning). In other words, an application can read activities from any source, but it can mess around only with the ones it created. I’m not sure yet how the application knows its source id, or how the host verifies the app’s identity, but I’ll be looking at those issues in a later posting.
For members and friends, I noted that the spec’s example included the OpenSearch namespace but didn’t use it. This time, the namespace is used for the totalResults, startIndex, and itemsPerPage elements. These suggest that it’s possible to page through long lists of activities, though I could find no mention of that in the spec. Again, I don’t know much about Atom, but I think that Atom-blessed way to handle paging would involve using “first”, “next”, and “last” links.
I’m not deeply into social networking myself — with my adolescent children using Facebook, my joining that site would be like showing up in a leather jacket at their highschool dance, and 99% of the time I spend on the more grown-up sites like Plaxo, LinkedIn, and Dopplr is used approving connection requests. As a result, I wasn’t aware of how important activity notifications were for a social-networking site.
Whatever happens with OpenSocial, I have found it to be a good architectural introduction to social networking in 2007, though I suspect that the next thing I’m going to look at — the persistence data API — has more to do with Google’s business requirements than with social networking itself.
This is the second part of a series of postings describing how I’m trying to understand the technical specs for the new Google-led OpenSocial initiative. In the first part, I cut down through all the text in the specs to get at the basic URLs, which represent the raw skeleton of services defined by the spec. This time, I’m going to look at the data formats, starting with the real bread and butter of social networking, people and their friends.
The content format for OpenSocial is always the Atom syndication format, a competitor to RSS for syndicating blogs and other similar information. I haven’t spent very much time with Atom yet — I appreciate that it’s more fully-specified than RSS 2.0, but I already know RSS and have run into no practical problems with it (through I’m aware of the potential ones) — so I’m probably not going to notice if or where the OpenSocial specs are violating the spirit or even letter of the Atom specs. I’ve occasionally seen complaints from Atom-heads about Atom-compliance in Google’s GData, and assume those apply to OpenSocial as well.
When you ask an OpenSocial provider for information about a member (using the URL pattern http://{DOMAIN}/feeds/people/{userId}), the spec says you get back something like this, assuming you’re authorized to make the request (lifted straight from the spec, and not namespace-compliant):
<entry xmlns='http://www.w3.org/2005/Atom'
xmlns:georss='http://www.georss.org/georss'
xmlns:gd='http://schemas.google.com/g/2005'>
<id>http://sandbox.orkut.com:80/feeds/people/14358878523263729569</id>
<updated>2007-10-28T14:01:29.948-07:00</updated>
<title>Elizabeth Bennet</title>
<link rel='thumbnail' type='image/*'
href='http://img1.orkut.com/images/small/1193601584/115566312.jpg'/>
<link rel='alternate' type='text/html'
href='http://orkut.com/Profile.aspx?uid=17583631990196664929'/>
<link rel='self' type='application/atom+xml'
href='http://sandbox.orkut.com:80/feeds/people/14358878523263729569'/>
<georss:where>
<gml:Point xmlns:gml='http://www.opengis.net/gml'>
<gml:pos>51.668674 -0.066235</gml:pos>
</gml:Point>
</georss:where>
<gd:extendedProperty name='lang' value='en-US'/>
<gd:postalAddress/>
</entry>
Aside from the fact that the tech writer is a Jane Austen fan, a couple of other points jump out:
Of course, in reality, the most important information about a member is the member’s friends list, but that information comes through a separate URL, http://{DOMAIN}/feeds/people/{userId}/friends.
This example is also lifted from the spec (and is still missing the declaration for the GML namespace):
<feed xmlns='http://www.w3.org/2005/Atom'
xmlns:openSearch='http://a9.com/-/spec/opensearchrss/1.0/'
xmlns:georss='http://www.georss.org/georss'
xmlns:gd='http://schemas.google.com/g/2005'>
<id>http://sandbox.orkut.com:80/feeds/people/14358878523263729569/friends</id>
<updated>2007-10-28T21:01:03.690Z</updated>
<title>Friends</title>
<link rel='http://schemas.google.com/g/2005#feed' type='application/atom+xml'
href='http://sandbox.orkut.com:80/feeds/people/14358878523263729569/friends'/>
<link rel='self' type='application/atom+xml' href='http://sandbox.orkut.com:80/feeds/people/14358878523263729569/friends'/>
<author><name>Elizabeth Bennet</name></author>
<entry>
<id>http://sandbox.orkut.com:80/feeds/people/02938391851054991972</id>
<updated>2007-10-28T14:01:03.690-07:00</updated>
<title>Jane Bennet</title>
<link rel='thumbnail' type='image/*' href='http://img1.orkut.com/images/small/null'/>
<link rel='alternate' type='text/html' href='http://sandbox.orkut.com:80/Profile.aspx?uid=574036770800045389'/>
<link rel='self' type='application/atom+xml' href='http://sandbox.orkut.com:80/feeds/people/02938391851054991972'/>
<georss:where>
<gml:Point xmlns:gml='http://www.opengis.net/gml'>
<gml:pos>51.668674 -0.066235</gml:pos></gml:Point></georss:where>
<gd:extendedProperty name='lang' value='en-US'/>
<gd:postalAddress/>
</entry>
<entry>
<id>http://sandbox.orkut.com:80/feeds/people/12490088926525765025</id>
<updated>2007-10-28T14:01:03.691-07:00</updated>
<title>Charlotte Lucas</title>
<link rel='thumbnail' type='image/*' href='http://img2.orkut.com/images/small/null'/>
<link rel='alternate' type='text/html' href='http://sandbox.orkut.com:80/Profile.aspx?uid=5799256900854924919'/>
<link rel='self' type='application/atom+xml' href='http://sandbox.orkut.com:80/feeds/people/12490088926525765025'/>
<georss:where>
<gml:Point xmlns:gml='http://www.opengis.net/gml'>
<gml:pos>0.0 0.0</gml:pos></gml:Point></georss:where>
<gd:extendedProperty name='lang' value='en-US'/>
<gd:postalAddress/>
</entry>
<entry>
<id>http://sandbox.orkut.com:80/feeds/people/15827776984733875930</id>
<updated>2007-10-28T14:01:03.692-07:00</updated>
<title>Fitzwilliam Darcy</title>
<link rel='thumbnail' type='image/*' href='http://img3.orkut.com/images/small/1193603277/115555466.jpg'/>
<link rel='alternate' type='text/html' href='http://sandbox.orkut.com:80/Profile.aspx?uid=14256507824223085777'/>
<link rel='self' type='application/atom+xml' href='http://sandbox.orkut.com:80/feeds/people/15827776984733875930'/>
<georss:where>
<gml:Point xmlns:gml='http://www.opengis.net/gml'>
<gml:pos>53.017016 -1.424363</gml:pos></gml:Point>
</georss:where>
<gd:extendedProperty name='lang' value='en-US'/>
<gd:postalAddress/>
</entry>
</feed>
Again, very straight-forward, if not namespace-compliant (due to the missing GML namespace declaration). There’s also a declaration of an OpenSearch namespace URI that’s never used, suggesting a feature that was removed in haste just before release. The friends list is simply a feed of person entries, just like the single entry returned for the member query, with a title, date, etc. at the top. Note that you always get the full friends list — there’s no support for filtering — so this might not be fun for someone who has 10,000+ friends.
What I don’t see, either in the example or the spec, is a way to provide typed relationships, like “spouse”, “colleague”, “classmate”, etc. I don’t know how important that is to application developers — simply getting the list of friends is probably the most important thing.
In a year or two, we’ll know whether the Google-lead OpenSocial initiative was a turning point in the social web or just a weak shot fired across Facebook’s bow. In the meantime, I think it’s worth taking some time to digest the API docs, which are still pretty rough.
Instead of reading and understanding everything first and then posting from a (virtual) podium, I’m going to try to work out my own understanding of the APIs right here on the web. That means that I’ll be asking questions that I’ll find the answers for later, that I’ll be making incorrect assumptions, and that I’ll be deferring hard stuff (like authorization/authentication) until I understand the basics. This is not, then, an OpenSocial primer by any stretch, since I don’t actually know what I’m talking about, but it might be useful as a snapshot of how a developer approaches a new API.
OpenSocial is designed so that any app can get information from any site as long as it has permission (I’ll figure out how that works later) — to accomplish that, it uses standard URL patterns on every site returning Atom entries and feeds. So after digging through a lot of crackerjack, I finally found the prize buried in the docs. Here are the URL patterns:
Listing all the URLs together like this, instead of spreading them out over pages and pages of docs, is the best way to start with a REST API. For example, you can tell at a glance what what kind of information is available and what is and isn’t writable (you can’t add new friends for a user, but you can add a new activity). Sure, Javascript libraries, etc. are nice, but the class hierarchies can obscure how simple the underlying data actually is (or “are”, if you’ve studied Latin). You can also spot possible typos in the docs — for example, what are the odds that the activity URLs are really supposed to start with “/activities/feeds/” when everything else starts with “/feeds/”? It could be poor, inconsistent design, but I suspect cut-and-paste errors.
The next time I get around to looking at OpenSocial, I’ll try to figure out the formats — it shouldn’t be too hard, since they’re all Atom entries or feeds. Then I’ll get into messier stuff like auth/auth, and I may eventually try adding OpenSocial support to my OurAirports hobby site, though it doesn’t even support friends yet.
What kinds of problems in XML can be solved using hardware acceleration? Personally, I don’t think that (software-based) XML parsing is usually a bottleneck in typical applications, but it might be for networking applications that deal with huge volumes of XML messages, especially at the large enterprise or carrier level.
For the XML 2007 conference (Boston, 3–5 December) Rich Salz, formerly with Datapower and now with IBM, has put together a 90-minute panel to go in depth into an area that few XML specialists know about.