Beginning of the end for open web data APIs?
December 18th, 2006[Update: hacking the Google Search AJAX API — see below.]
[Update #2: Don Box is thinking along the same lines as I am.]
[Update #3: Rob Sayre points out that there is, in fact, a published browser-side JavaScript API underlying the AJAX widget.]
Over on O’Reilly Radar, Brady Forrest mentioned that Google is shutting down its SOAP-based search API. Another victory for REST over WS-*? Nope — Google doesn’t have a REST API to replace it. Instead, something much more important is happening, and it could be that REST, WS-*, and the whole of open web data and mash-ups all end up on the losing side.
It’s not about SOAP
Forget about the SOAP vs. REST debate for a second, since most of the world doesn’t care. Google’s search API let you send a search query to Google from your web site’s backend, get the results, then do anything you want with them: show them on your web page, mash them up with data from other sites, etc. The replacement, Google AJAX API, forces you to hand over part of your web page to Google so that Google can display the search box and show the results the way they want (with a few token user configuration options), just as people do with Google AdSense ads or YouTube videos. Other than screen scraping, like in the bad old days, there’s no way for you to process the search results programmatically — you just have to let Google display them as a black box (so to speak) somewhere on your page.
A precedent for widgets instead of APIs
An AJAX interface like this is a great thing for a lot of users, from bloggers to small web site operators, because it allows them to add search to their sites with a few lines of JavaScript and markup and no real coding at all; however, the gate has slammed shut and the data is once again locked away outside the reach of anyone who wanted to do anything else.
Of course, there are alternatives still available, such as the Yahoo! Search API (also available in REST), but how long will they last? Yahoo! has its own restructuring coming up, and if Nelson Minar’s suggestion (via Forrest) is right — that Google is killing their search API for business rather than technical reasons — this could set a huge precedent for other companies in the new web, many of whom look to Google as a model. Most web developers will probably prefer the AJAX widgets anyway because they’re so much less work, so by switching from open APIs to AJAX widgets, you keep more users happy and keep your data more proprietary. What’s an investor or manager not to like?
What next?
Data APIs are not going to disappear, of course. AJAX widgets don’t allow mash-ups, and some sites have user bases including many developers who rely on being able to combine data from different sources (think CraigsList). However, the fact that Google has decided that there’s no value playing in the space will matter a lot to a lot of people. If you care about open data, this would be a good time to start thinking of credible business cases for companies to (continue) offer(ing) it.
Update: Hacking the Google AJAX API (or, back to Web ‘99)
The AJAX API is designed to allow interaction with JavaScript on the client browser, but not with the server; however, as Davanum Srinivas demonstrates, it’s possible to hack on the API to get programmatic access from the server backend. I’m not sure how this fits withThis violates Google’s terms of service, and obviously, they can make incompatible changes at any time to try to kill it, but at least there’s a back door for now. Thanks, Davanum.
Personally, I was planning to use the Yahoo (REST) search API for site search even before all this broke, because I didn’t want to waste time trying to figure out how to use SOAP in PHP. I’m glad now I didn’t waste any time on Google’s API, and I’ll just keep my fingers crossed that Yahoo’s API survives.
December 19th, 2006 at 09:29:09
[...] Dave Megginson comments. Is this the beginning of the end for mashups and all the “Web 2.0″ excitement? Or just the end of SOAP? (Steve Loughran). [...]
December 19th, 2006 at 04:10:25
[...] David Megginson: “… it could be that REST, WS-*, and the whole of open web data and mash-ups all end up on the losing side.” [...]
December 19th, 2006 at 04:27:46
hahaha. the sky is falling, indeed chicken little.
i think the very phrase buisness case elludes the growing sense of the network thats being built. google just took a step away from the network, just as other companies will wax and wane in this same decision to participate or not to participate in the network. these are decisions of individuals, and i’m confident that no sum of individuals will ever stop the networked access of information from being the most pertinent and defining characteristics of this era. the defining characteristic of the network is its unpredictability, its infinite recombinant forms, the shapes we can never think of. i wish the best of luck to all the fools who think they can continue to exist in stable static forms of their own choosing, not seeing, hearing or speaking with the rest of the network, not allowing themselves to be shaped by it. these are people seeking existance in a vaccuum.
as it happens, the web, as a brutally client server architecture, is a particularly poor at creating the open information spaces that the current young bloods dream deeply and wildly about. so far the web has only been a reflection of the available open source computations available at any given time, and i see no reason to think that our capacity to wield information and the value of our network is determined by anything else besides this open computation.
screen scraping is the ultiamte freedom. if fools continue pressing pre-formed buisness cases on the network, mashups will simply revert to elaborate grease monkey scripts that allow users to rewrite and re-edit the trash they’re being fed. screen scraping is pure, its the users action, your own manipulation of the content you’ve been handed, and entails no nasty redistribution clauses. screen scraping is the only thing decentralized about the entire so called “web”.
December 19th, 2006 at 04:50:13
[...] End to open data? Dave Megginson points out that Google dropped their SOAP API for a business reason: to keep tight control on how their data is accessed and used. I have a strong suspicion he’s right. Google is cutting down on the non-essentials, and guess what the first non-essential is? [...]
December 19th, 2006 at 06:35:10
[...] Follow any comments here with the RSS feed for this post. Post a comment or leave a trackback: Trackback URL. « links for2006-12-18 [...]
December 19th, 2006 at 06:44:43
If this is accurate, then it will be interesting to see how Google itself reacts when sites that it indexes for its own search engine business begin to “hide” (make it difficult to get at) their data behind widgets and libraries.
December 19th, 2006 at 07:54:22
[...] From the comments to megginson.com’s post about Google’s API’s: [...]
December 19th, 2006 at 11:17:02
I would agree to a small extent with rektide, that this is a rather dramatic reaction to a singular event. However, it’s saddening that it’s taken this long, or something of this nature to enlighten the community to a certain truth that has been there all along. The web does not guarantee openness, openness is merely the standard that has been adopted.
In truth the desktop area is far more open because users have vastly more control over their desktops.
It’s not exactly news to me, as I predicted about 6 months ago that interoperability in the web would start to become less pervasive (see http://ryan-technorabble.blogspot.com/2006/07/web-interoperability-and-identity.html). I doubt it’s a 180 degree turn, there’s still a lot of momentum to sustain the standard and a lot of industrious people committed to it. But I do think that it will become hard to ignore that interoperability is not simply something you can take for granted, and that openness is not a feature of the web, it is a product of the community.
December 20th, 2006 at 12:33:45
[...] Quoderat » Beginning of the end for open web data APIs? everyone and their mother is pointing here; i’d be willing to buy the sky is falling bit more if i saw hard numbers proving that making open data available was a losing proposition economically - plus, we need to ask who’s using this? (tags: ajax api google mashups opensearch search service soap rest) [...]
December 20th, 2006 at 04:45:52
[...] De beslissing van Google om hun open zoek API af te sluiten voor nieuwe gebruikers heeft een interessante discussie tot gevolg. Eentje die best bepalend kan zijn voor de ontwikkeling van het web, o.a. wanneer het gaat om de vele mashups die wel allemaal zo leuk vinden… [...]
December 20th, 2006 at 05:49:05
[...] Google has replaced it’s full API with dumbed down AJAX only widget. The web a little too open for them? Is this as some people suggest already the “Beginning of the end for open web data APIs?“, i hope not… we were just getting started. [...]
December 20th, 2006 at 06:42:05
[...] Es gibt einige Unruhe über die Einstellung der SOAP Such-API durch Google. Ich will hier nicht auf die Details eingehen - das hat u.a. der Google Watch Blog schon hinreichend getan -, das Thema liegt ja auch nicht direkt im Beritt dieses Blogs. Aber es ist natürlich ein deutliches Signal, dass Google seine APIs nicht als Selbstverständlichkeit betrachten möchte. Die SOAP-API ist immerhin recht populär unter Entwicklern, nicht zuletzt deshalb, weil sie einige gestalterische Freiheiten einräumte, mehr jedenfalls als die AJAX-API, die Google als Ersatz anbietet. Was das für andere APIs bedeutet, und ob das Ganze nun wirklich den Anfang vom Ende für offene Daten-APIs oder gar für Mashups überhaupt bedeutet, muss man abwarten. Aber es wäre naiv, anzunehmen, dass ähnliche Angebote nicht auch irgendwann mal zur Disposition stehen könnten. [...]
December 20th, 2006 at 09:27:28
First they protect the resources. Then they protect the sources for the resources.
Ever signed an employment agreement that makes anything you invent while in their employ the IP of the employer? No? Then you never worked for a professional company of any significance.
It is not surprising they took this long to get around to this. Dope dealers get you hooked first. It is only surprising that you are so angst-ridden about that.
“As the twig is bent…, “, David. The web changes nothing in the human equation except short cycle behaviors. The long cycle behaviors reemerge dominant as always. Over a longer time, you may see some subtle variations change features in certain environment but these will come as side effects, not direct effects. Beware pleioptropy.
len
December 20th, 2006 at 02:29:10
[...] Dare doesn’t think I was clear enough when I called bullshit on David Megginson’s suggestion that open APIs may be on their way out. So let me try again… [...]
December 20th, 2006 at 03:26:17
[...] Update: Here is another interesting post, Beginning of the end of for open web APIs?, that is worth checking out on this issue. [...]
December 20th, 2006 at 04:54:12
David,
it’s possible to programatically access the AJAX Search API. Please see here for proof (calling from plain ol’ Java):
http://blogs.cocoondev.org/dims/archives/004722.html
thanks,
dims
December 20th, 2006 at 09:17:21
Hi Dave. The paragraph under “It’s Not About SOAP” seems completely incorrect. Google does provide readymade widgets, but the public, documented API is rather extensive underneath.
http://www.franklinmint.fm/blog/archives/000951.html
Also, the server-side processing argument seems to miss the point. The canonical “mashup” is a web page that pulls in JS data from Google Maps and some other source. If you want people to use Google Search results on their web pages, the wrong answer is “here is the SOAP/XML-REST/HTTP/Whatever API”. That is for the (vocal) priesthood, not web authors. Your main point seems to be that it doesn’t allow bots. *shrug
December 21st, 2006 at 08:03:21
Thanks for the Javascript API clarification, Robert. I haven’t had time to look in detail to see if/how Google is working around the same-site XMLHttpRequestObject restriction (invisible iframes?), but client-side mashups of the cute-overlay-over-Google-maps variety, while fun, have their limitations — eventually, for many genuinely-useful things, you’re going to have to do data crunching on the server side.
December 22nd, 2006 at 01:59:46
Well, I think the limitation is basically the same one Google has always had: “no bots”.
Regarding the API traffic, it looks like a variation on JSON-P. Basically, you do this:
—————————
/* write a callback function */
function myJSONCallback(aJSON) { alert(aJSON) }
/* append a script element to the document */
var myScript = document.createElement(”script”);
myScript.src = “http://api.example.com/makeArray?elements=3&callback=myJSONCallback”;
—————————
The browser sends a GET to request the script src, and this can be any host. The body of the request looks like this:
myJSONCallback(["","",""]);
The contents of the parens are a JSON object literal.
December 22nd, 2006 at 08:13:45
[...] (Editor’s note: Some people, it seems, are steamed about changes Google has made to the way it lets outside developers tap into its search and ad platforms. We asked Rob Drapkin to write about the changes in Google’s ad platform. These changes are different Google’s separate changes to its search platform, where you can see an example of a complaint here.) [...]
December 23rd, 2006 at 10:24:50
[...] Early in the week, I posted about the end of the Google search API, and speculated that — since everyone else tends to copy Google — it might be the start of a general trend away from open data APIs and in favour of server-side AJAX widgets. In response, Amit Kumar of Yahoo sent me an e-mail message (after failing to get past Spam Karma in the comment system for my blog): [...]
December 25th, 2006 at 11:15:18
Our response to this is the EvilAPI and EvilRSS
January 1st, 2007 at 09:04:11
[...] Google a “déprécié” son API de recherche SOAP et c’est Dave Megginson qui nous livre la meilleur analyse: Beginning of the end for open web data APIs? Ce n’est pas une victoire de REST sur SOAP/WS-* mais le début de la fin du web as a platform, une composante importante du Web 2.0. [...]
January 10th, 2007 at 02:53:26
[...] Funny, but hardly the point as one of the comments on his post points out, referring to David Megginson who asks, rather more importantly, if this is the Beginning of the end for open web data APIs? Pointing out that, in this case, the alternative to the SOAP API is a set of Ajax widgets, David’s argument is that this could become a common business approach - the data is not directly exposed, and deploying Ajax components is popular to many because it is so easy to do. [...]
February 15th, 2007 at 11:40:29
[...] Of course there are many other ways of presenting search results. Many of the more interesting.ways to present the data reorder the results according to other criteria or third-party data (like other search results), or even dispense with a linear order. Now if you want to pull in search data and manipulate it in this way, you’re going to have to use the Yahoo! REST API’s. But will Yahoo! follow suit and get rid of their general API’s in favor of brand-preserving Ajax widgets? Dave Megginson doesn’t think so, but sees some clouds on the horizon. [...]
March 1st, 2007 at 09:16:35
[...] Of course there are many other ways of presenting search results. Many of the more interesting.ways to present the data reorder the results according to other criteria or third-party data (like other search results), or even dispense with a linear order. Now if you want to pull in search data and manipulate it in this way, you’re going to have to use the Yahoo! REST API’s. But will Yahoo! follow suit and get rid of their general API’s in favor of brand-preserving Ajax widgets? Dave Megginson doesn’t think so, but sees some clouds on the horizon. [...]
March 17th, 2007 at 12:37:39
I’m really surprised there hasn’t been a bigger backlash against this. Where is the outrage?
March 19th, 2007 at 03:06:38
Google doesn’t have a REST API to replace it. Instead, something more important is happening, and it could be that REST, WS-*, and the whole of open web data and mash-ups all end up on the losing side.