A funny thing happened as I was noodling on this ZapFlash. I was all set to put the nail in the Application Programming Interface (API) coffin, continuing the discussion of just how awful Remote Procedure Call (RPC) interfaces are, and how we should avoid them at all costs. But then I ran across the “explosion of APIs” meme at ProgrammableWeb and elsewhere. After all, your head must be planted deeply in the sand not to notice the plethora of APIs at all our favorite Social Web sites like Twitter, Facebook, and hundreds of others. How can the API be dead when APIs are more important and plentiful than ever?

Maybe I’m just nitpicking. As APIs become less RPC-centric and more RESTful, the inflexible tight coupling of the past is giving way to a new golden age of API-ness, where anybody can connect any piece of software seamlessly and automatically to any Web site or app we might care to use. But upon closer reflection, my concern is not all nitpickiness. There are API best practices and API worst practices, just as there are good APIs and bad ones. Perhaps more to the point, most of the APIs out there, well, are among the bad ones. So, what makes for a good API today, with all the power and capabilities that modern technology approaches afford us? Why are so many people getting them wrong? And what can we do to steer everyone in the right direction?

Programmable Interfaces: The Never-ending Story

The real reason that APIs get under my skin is the “P” – “programmable,” as in programmatic. The core notion of an API, after all, is that you want one application to be programmable by something external to it, in the sense that you have an imperative control flow that goes from here to there and back to here.

The problem with such imperative programming in any distributed environment, of course, is that there is no end of problems that can arise between here and there. Network issues, incompatibilities, timing and sequence issues, the list goes on and on, leading computer scientists to the notion of functional programming. In the functional world, functions (which also go by the names procedures or methods) encapsulate the software at the remote location, forming black boxes with fixed inputs and outputs. Any local piece of software, therefore, can call a remote procedure and get the desired response, without having to worry about whether multiple calls to that procedure might interfere with each other or lead to other nasty surprises that a purely imperative approach might exhibit.

Here’s the rub: functional programming is nothing more than RPC, with all the tight coupling issues that come along for the ride. However, functional programming is also considered to be a type of declarative programming. Declarative programming has always been the ugly duckling of the programming language world, because with a declarative approach, you don’t specify the control flow. Instead, you describe the desired behavior you want the software to exhibit. And somehow, via some kind of behind-the-scenes hand waving, the software miraculously does what you want it to.

Functional programming is declarative in the sense that a remote procedure acts as a black box. Specify the operations, inputs, and outputs (its signature, in API-speak), and the inner workings of the procedure aren’t your problem. That is, unless you don’t understand what it’s supposed to do for you, or heaven forbid, its behavior changes.

REST to the Rescue?

Building abstracted interfaces that enable organizations to deal better with change is what SOA is all about, of course. Unfortunately, the tools we had at our disposal – namely, Web Services – didn’t go far enough. Yes, we moved from RPC-style Web Services to document-style, and that improved our loose coupling substantially. But even document-style Web Services still have operations, a holdout from the bad old functional programming days.

Enter REST. ZapThink has discussed the move away from RPC to RESTful interfaces before: in REST-Based SOA: an Iconoclastic Approach, but of course, we’ve been discussing the advantages of document style interfaces over RPC for several years now. But people still aren’t getting it. Not only are techies missing the point of REST, even when they build supposedly RESTful APIs, they’re failing to follow the REST constraints.

For example, many of these ostensible RESTful APIs don’t provide self-descriptive messages. With REST, every message from a resource should contain all the metadata necessary for any client to understand what it can do with representations of that resource. In the RPC days, procedural responses may have contained only data with no metadata. With Web Service interactions, Service responses contained some metadata in the form of the SOAP envelope, header, and the tagging structure in the body. But even with Web Services, the WSDL contract and policy metadata are external to the Service. You don’t expect to get the contract when you query a Service.

With a RESTful interaction, on the other hand, contract metadata may be returned in particular representations, whether it be media type metadata, schemas, or even less structured metadata. And of course, representations also include hyperlinks, which also provide contract metadata to the client.

Separation of resources from representations is another essential REST constraint – one that developers also frequently violate. If there are any instructions to the resource in a request other than one of the four operations of the uniform interface (GET, POST, PUT, and DELETE), then you are violating this constraint. For example, if your request is instructing a resource to update a record, you’re not being RESTful. It’s up to the resource to determine whether a POST or PUT should update a record, not the representation.

Finally, hypermedia as the engine of application state – the dreaded HATEOAS – is perhaps the most overlooked of the REST constraints, even though it’s the most important. The big picture of HATEOAS are the hypermedia applications that we’re trying to build, but even at the API level, HATEOAS is critical, in conjunction with the self-descriptive message constraint. In any truly RESTful interaction, the resource returns a representation that contains all necessary metadata, including hyperlinks that instruct the client what it can do next. Likewise, a RESTful client has no idea what a resource can do for it unless it follows such links.

“RESTful” Equals “Web-friendly”

What so many techies lose sight of is the fact that REST isn’t supposed to make the Web more like system integration; it’s meant to make system integration more like the Web. When you click a link in a Web page, you expect to go to another Web page, or perhaps a video or sound file or some other media representation. REST takes that simple interaction and abstracts it. Now you have an abstracted client (instead of a browser) supporting a request of a resource (the Web server capability, for example, the php script that generated the response) on an abstracted server (the Web server) that returns a representation (the resulting media file or Web page) as per the uniform interface (what the link you clicked was supposed to do). These abstractions let us apply simple Web behavior to all manner of system-to-system interactions. But never, ever lose sight of the fact that you want simple Web behavior from your application.

I’m sure many readers who made it this far are muttering to themselves that their RESTful APIs are truly RESTful. Well, maybe, but probably not, and here’s proof. Let’s take a popular, supposedly RESTful API as an example: the Yahoo! Maps PlaceFinder Geocoding API. You’ll notice on the documentation page a “BASE URI,” which it expresses as http://where.yahooapis.com/geocode?[parameters]. There are two problems with this expression. First, it’s not a hyperlink (it just looks like one). Second, the URL has the question mark in it, requiring the user to know what parameters might be valid.

Let’s say instead we turn this expression into a true hyperlink, leaving off the parameters since we don’t know what they are: http://where.yahooapis.com/geocode. Go ahead,  click on that link, I dare ya. Sure enough, you get an error message. True, it’s XML formatted, but it’s missing something absolutely essential: a hyperlink.

In fact, Yahoo! made several serious mistakes: first, the base URI isn’t a hyperlink, violating HATEOAS. Second, the metadata that tell you how to use the interface are separate from the interface, rather than  returned in the response from hyperlinking to the base URI, violating the self-descriptive messages constraint. Third, the URI requires parameters as part of the search query (the characters after the question mark), violating the separation of resource and representation. And finally, the error response it did return didn’t include a hyperlink, once again violating HATEOAS. Bottom line: there’s really very little that’s RESTful about this API.

While it may be amusing to pick on Yahoo!, it’s important to point out that they aren’t alone. In fact, they’re typical. It’s quite rare, in fact, for a supposed RESTful interface to be truly Web-friendly. After all, most techies don’t expect Web friendliness from APIs. APIs are, well, programming interfaces, while the Web consists of user interfaces. But in the REST world, programming interfaces are simply abstracted user interfaces. In other words, truly RESTful APIs aren’t really like the APIs we know and love, and have been working with for years. They’re fundamentally different.

The ZapThink Take

I know this ZapFlash is deeply iconoclastic, but that’s what you’ve come to expect from ZapThink, after all. However, the point of this article isn’t to scold everybody for doing REST wrong. The point is to help you think differently about what it means to program in a distributed environment. The point to the “programmable Web” isn’t to make the Web more programmable, it’s to make software more Web-like. If we can finally free ourselves from the last vestiges of imperative, RPC-style programming, even going so far as to steer clear of functional programming, and move to a fully declarative, document-centric paradigm, only then will we be able to achieve the resilience and power of the Web when we tackle system integration challenges.

This mind shift is at the core of REST. When Roy Fielding wrote his dissertation, he didn’t come up with REST and then build the Web following its constraints. On the contrary: he deeply understood what made the Web itself so special, and he sought to express that specialness as an architectural style. True, tackling system integration following true RESTful principles is difficult. But remember, building the Web was easy. Take TCP/IP, add a few simple standard protocols, code an HTML rendering app we called a browser, and the Web more or less took it from there. Almost twenty years later, the Web is unimaginably immense, and yet it still works just fine, thank you very much. Is it too much to ask for all of our IT systems to behave the same way?

Image source: cloudzilla

Today is a wonderful time for anyone interested in Cloud Computing to be working with the US government. On the one hand, the government considers Cloud to be strategically important, and they already have a track record as an early adopter of Cloud Computing on a grand scale. On the other hand, the government is also in the unique position of being able to drive standards for the approach—and in fact, they are even responsible for establishing the most widely adopted definition of Cloud Computing.

The federal agency who has taken this leadership position is the National Institute for Standards and Technology (NIST), an agency of the US Department of Commerce. NIST’s formal definition of Cloud Computing is already well known—“a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction.” Concise as that definition is, it only marks the beginning of the work NIST is doing to formalize and standardize the full breadth of Cloud Computing approaches, both within the government as well as for the world at large.

I learned about the breadth of NIST’s work on Cloud last week, when I had the pleasure of attending the NIST Cloud Computing Forum & Workshop. They are leading a cross-industry effort to “provide thought leadership and guidance around the cloud computing paradigm to catalyze its use within industry and government. NIST aims to shorten the adoption cycle, which will enable near-term cost savings and increased ability to quickly create and deploy enterprise applications. NIST aims to foster Cloud Computing systems and practices that support interoperability, portability, and security requirements that are appropriate and achievable for important usage scenarios.” To this end, they have followed up their formal definition with a Cloud Computing Technology Roadmap, which consists of three volumes: requirements to further US government adoption, information for Cloud adopters, and technical considerations for government Cloud Computing deployment decisions. They have also published a Cloud Computing reference architecture and standards roadmap.

To be sure, NIST has generated a daunting quantity of information here—but ignore these documents at your peril. If you work for a US government agency, then you likely have a mandate to move toward Cloud Computing, and NIST spells out many of the details. But even if you have nothing to do with the government,  it’s important to remember that NIST is also a standards body in its own right, as well as a coordinating agency for other standards bodies. No other group or agency anywhere else in the world has achieved the same leadership position with respect to today’s nascent Cloud standards efforts.

One of the main reasons NIST is able to maintain this position is because they have an inclusive approach. Want to contribute? You’re welcome to. Have an issue with something in one of the documents? Then let them know. After all, one of the reasons they’ve generated so much content is because they have so many contributors, not just from the government, but from people around the world.

Even ZapThink isn’t above joining the fray. We’ve reviewed NIST’s documents in the context of ZapThink’s eye for agile enterprise architecture, and we’ve identified a missing link. Of course, looking at something and trying to identify what’s missing is always difficult, especially when so many contributors have already pored over the material so carefully. The trick is to break out of “horseless carriage” patterns of thinking: instead of considering the Cloud to be little more than an outsourced, virtualized data center, put on your architect’s hat and consider how Cloud Computing’s unique characteristics will change how you do architecture.

NIST’s Cloud Deployment Scenarios

I found this missing link when I reviewed the Cloud deployment scenarios in the NIST Standards Roadmap document. Here is their diagram illustrating the eight generic deployment scenarios that they identified:

They sort the various deployment scenarios into three categories:

Single Cloud

• Scenario 1: Deployment on a single Cloud
• Scenario 2: Manage resources on a single Cloud
• Scenario 3: Interface enterprise systems to a single Cloud
• Scenario 4: Enterprise systems migrated or replaced on a single Cloud

Multiple Clouds (serially, one at a time)

• Scenario 5: Migration between Clouds
• Scenario 6: Interface across multiple Clouds
• Scenario 7: Work with a selected Clouds

Multiple Clouds (simultaneously, more than one at a time)

• Scenario 8: Operate across multiple Clouds

From ZapThink’s perspective, the most interesting of these are scenarios 1, 3, and 4, because they consider the relationships between enterprise systems and the Cloud. ZapThink has written about these relationships before, most recently in The Keys to Enterprise Public Cloud, but also back in mid-2010, when we discussed Cloud Architecture’s Missing Link.

The missing link we pointed out in that ZapFlash was the ability to compose Cloud-based Services with on-premise Services as part of an enterprise SOA effort. It could be argued, however, that composing Cloud-based Services falls under Scenario 3, since Services are a type of interface. But there’s more to this story—and to understand how the NIST folks missed it, it’s important to follow their line of reasoning.

NIST’s Blind Spot

NIST has divided their Cloud standards efforts into three categories: interoperability, portability, and security. Interoperability standards are the most straightforward, especially for anyone who has worked with Web Services, which of course are little more than standards-based interfaces intended to promote interoperability and loose coupling.

Portability standards are more complicated, because NIST considers both application portability and data portability. In the Cloud context, application portability centers on the ability to move virtual machine (VM) instances from one Cloud to another. Data portability, however, is more difficult, because applications process different kinds of data, and those data flow throughout an entire system. For one organization, data portability might mean moving a single database from one Cloud to another, but for a different organization, the requirement might be for the portability of an entire SaaS application, along with all of its distributed data.

NIST’s focus on interoperability and portability (and security, of course, which is an entire conversation in its own right) makes perfect sense in light of their focus on standards, since the standardization of these three capabilities will go a long way in furthering NIST’s core mission. So it’s no wonder that their three Cloud deployment scenarios that involve enterprise systems consist of deploying or migrating to a Cloud (facilitated by portability standards), or interfacing with a Cloud (facilitated by interoperability standards).

It should come as no surprise, therefore, that NIST missed another deployment scenario: building applications that leverage both on-premise and Cloud-based capabilities, where those applications rely upon more than interoperability, portability, and the ubiquitous security.

Building applications that are compositions of Cloud-based and on-premise Services is a simple example, but doesn’t go far enough, because even this scenario falls into the “horseless carriage” trap of considering the Cloud to be nothing more than a virtualized data center. Factor elasticity into the equation, however, and we must consider new approaches to architecting such applications that go beyond considerations of interoperability and portability.

Building the Cloud’s Inherent Elasticity into Hybrid Applications

More than any other characteristic, elasticity distinguishes true Clouds from simple virtualized data centers. If your app requires more resources, the Cloud will provision those resources automatically, and then release them when you’re done with them—until you need them again. Furthermore, those elastic resources may be among any of the different types of Cloud resources (networks, servers, storage, applications and Services,  as per the NIST definition), or any combination thereof.

As a result, when you architect your app, you don’t know how many of each of these resources you will be using at any point in time, since the number can change with no warning. You must take this change into account when architecting your data, your middleware, your execution environments, your application logic, and your presentation tier—in other words, your entire distributed application.

Cloud providers do their best to hide the underlying complexity inherent in delivering elastic infrastructure. But when you’re building a hybrid app—that is, one that includes Cloud-based as well as on-premise capabilities—your architects must have deeper knowledge of the underlying capabilities of the Cloud environment than Cloud providers are typically comfortable revealing. In other words, even once Cloud interoperability and portability standards mature, architects will still require additional information about the underlying capabilities of their Cloud environments that such standards won’t cover.

The ZapThink Take

This ZapFlash may leave you wanting more—namely, how precisely do you architect with the Cloud in mind? Unfortunately, there isn’t enough room for the answer to that question in this ZapFlash, but fear not, we’ll be laying out more details in the weeks and months to come.

Can’t wait? Then come to our Licensed ZapThink Architect SOA & Cloud course in San Diego January 16-19. We’ll dive deep into architecting with the Cloud, as we enjoy warm breezes off the Pacific from our lush venue by Seaforth Marina on Quivira Basin. Or better yet, take advantage of ZapThink’s new Cloud Competency Center by dropping us a line at info@zapthink.com. We’re here to help!

Many of you know me as one half of the ZapThink team – an advisor, analyst, sometimes-trainer, and pundit that has been focused on XML, Web Services, SOA, and now Cloud Computing over the past decade or so. Some you may also know that immediately prior to starting ZapThink I was one of the original members of the UDDI Advisory Group back in 2000 when I was with ChannelWave, and I also sat on a number of standards bodies including RosettaNet, ebXML, and CPExchange initiatives. Furthermore, as part of the ZapThink team, I tracked the various WS-* standards from their inception to their current “mature” standing. I’ve closely followed the ups and downs of the Web Service Interoperability (WS-I) organization and more than a few efforts to standardize such things as business process. Why do I mention all this? To let you know that I’m no slouch when it comes to understanding the full scope and depth of the Web Services family of standards. And yet, when push came to shove and I was tasked with implementing SOA as a developer, what did I choose? REST.

Representational State Transfer, commonly known as REST, is a style of distributed software architecture that offers an alternative to the commonly accepted XML-based Web Services as a means for system-to-system interaction. ZapThink has written numerous times about REST and its relationship to SOA and Web Services. Of course, this has nothing to do with Service-Oriented Architecture, as we’ve discussed in numerous ZapFlashes in the past. The power of SOA is in loose coupling, composition, and how it enables approaches like Cloud Computing. It is for these reasons that I chose to adopt SOA for a project I’m currently working on. But when I needed to implement the Services I had already determined were necessary, I faced a choice: use Web Services or REST-based styles as the means to interact with the Services. For the reasons I outline below, REST was a clear winner for my particular use case.

Web Services in Theory and in Practice

The main concepts behind Web Services were established in 1999 and 2000 during the height of the dot-com boom. SOAP, then known as the Simple Object Access Protocol and later just “SOAP,” is the standardized, XML-based method for interacting with a third-party service. Simple in concept, but in practice, there’s many ways to utilize SOAP. RPC style (we think not) or Document style? How do you identify end points? And what about naming operations and methods? Clearly SOAP on its own leaves too much to interpretation.

So, this is the role that the Web Services Description Language (WSDL) is supposed to fill. But writing and reading (and understanding) WSDL is a cumbersome affair. Data type matching can be a pain. Versioning is a bear. Minor server-side changes often result in different WSDL and a resulting different Service interface, and on the client-side, XSD descriptions of the service are often similarly tied to a particular version of the SOAP endpoint and can break all too easily. And you still have all the problems associated with SOAP. In my attempts to simply get a Service up and running, I found myself fighting more with SOAP and WSDL than doing actual work to get Services built and systems communicating.

The third “leg” of the Web Services concept, Universal Description, Discovery and Integration (UDDI), conceptually makes a lot of sense, but in practice, hardly anyone uses it. As a developer, I couldn’t even think of a scenario where UDDI would help me in my particular project. Sure, I could artificially insert UDDI into my use case, but in the scenario where I needed loose coupling, I could get that by simply abstracting my end points and data schema. To the extent I needed run-time and design-time discoverability or visibility into Services at various different states of versioning, I could make use of a registry / repository without having to involve UDDI at all. I think UDDI’s time has come and gone, and the market has proven its lack of necessity. Bye, bye UDDI.

As for the rest of the WS-* stack, these standards are far too undeveloped, under implemented, under-standardized, inefficient, and obscure to make any use of whatever value they might bring to the SOA equation, with a few select exceptions. I have found the security-related specifications, specifically OAuth, Service Provisioning Markup Language (SPML), Security Assertion Markup Language (SAML), eXtensible Access Control Markup Language (XACML), are particularly useful, especially in a Cloud environment. These specifications are not Web Services dependent, and indeed, many of the largest Web-based applications use OAuth and the other specs to make their REST-based environments more secure.

Why REST is Ruling, and Applying SOA Principles

I ended up using REST for a number of reasons, but the primary one is simplicity. As most advocates of REST will tell you, REST is simpler to use and understand than Web Services. Development with REST is easier and quicker than building WSDL files and getting SOAP to work and this is the reason why many of the most-used web APIs are REST-based. You can easily test HTTP-based REST requests with a simply browser call. It can also be more efficient as a protocol since it doesn’t require a SOAP envelope for every call and can leverage the JavaScript Object Notation (JSON) as a data representation format instead of the more verbose and complex to process XML.

But even more than the simplicity, I appreciated the elegance of the REST approach. The basic operation and scalability of the Web has proven the underlying premise of the fundamental REST approach. HTTP operations are standardized, widely accepted, well understood, and operate consistently. There’s no need for a REST version of the WS-I. There’s no need to communicate company-specific SOAP actions or methods – the basic GET, POST, PUT, and DELETE operations are standardized across all Service calls.

Even more appealing is the fact that the vendors have not polluted REST with their own interests. The primary driver for Web Services adoption has been the vendors. Say what you might about the standard’s applicability outside a vendor environment, one would be very hard pressed to utilize Web Services in any robust way without first choosing a vendor platform. And once you’ve chosen that platform, you’ve pretty much committed to a specific Web Services implementation approach, forcing third-parties and others to comply with the quirks of your particular platform.

Not so with REST. Not only does the simplicity and purity of the approach eschew vendor meddling, it actually negates much of the value that vendor offerings provide. Indeed, it’s much easier (and not to mention lower cost) to utilize open source offerings in REST-based SOA approaches than more expensive and cumbersome vendor offerings. Furthermore, you can leverage existing technologies that have already proven themselves in high-scale, high-performance environments.

REST: Focus on Architecture, Not on HTTP

So, how did I meld the fundamental tenets of SOA with a REST-based implementation approach? In our Web-Oriented SOA ZapFlash, we recommended using the following approach to RESTafarian styles of SOA:

• Make sure your Services are properly abstracted, loosely coupled, composable, and contracted
• Every Web-Oriented Service should have an unambiguous and unique URI to locate the Service on the network
• Use the URI as a means to locate as well as taxonomically define the Service in relation to other Services.
• Use well-established actions (such as POST, GET, PUT, and DELETE for HTTP) for interacting with Services
• Lessen the dependence on proprietary middleware to coordinate Service interaction and shift to common Web infrastructure to handle SOA infrastructure needs

Much of the criticism of REST comes not from the interaction approach, but rather from the use of HTTP. Roy Fielding, the progenitor of REST, states in his dissertation that REST was initially described in the context of HTTP, but is not limited to that protocol. He states that REST is an architectural style, not an implementation, and that the Web and the use of the HTTP protocol happens to be designed under such style. I chose to implement REST using eXtensible Messaging and Presence Protocol (XMPP) as a way of doing distributed, asynchronous messaging styles of REST-based Service interaction. XMPP, also known as the Jabber Protocol, has already proven itself as a widely-used, highly-scalable messaging protocol for secure and distributed near-realtime messaging protocol. XMPP-based software is deployed widely across the Internet, and forms the basis of many high-scale messaging systems, including those used by Facebook and Google.

Am I bending the rules or the intent of REST by using XMPP instead of HTTP? Perhaps. If HTTP suits you, then you have a wide array of options to choose from in optimizing your implementation. Steven Tilkov does a good job of describing how to best apply HTTP for REST use. But you don’t have to choose XMPP for your implementation if HTTP doesn’t meet your needs. There are a number of other open-source approaches to alternative transports for REST existing including RabbitMQ (based on the AMQP standard), ZeroMQ, and Redis.

The ZapThink Take

The title of this ZapFlash is a bit of a misnomer. In order to be a convert to something you first need to be indoctrinated into another religion, and I don’t believe that REST or Web Services is something upon which to take a religious stance. That being said, for the past decade or so, dogmatic vendors, developers, and enterprise architects have reinforced the notion that to do SOA properly, you must use Web Services. ZapThink never believed that this was the case, and my own experiences now shows that SOA can be done well in practice without using Web Services in any significant manner. Indeed, my experience shows that it is actually easier, less costly, and potentially more scalable to not use Web Services unless there’s an otherwise compelling reason.

The conversation about SOA is a conversation about architecture – everything that we’ve talked about over the past decade applies just as equally when the Services are implemented using REST or Web Services on top of any protocol, infrastructure, or data schema. While good enterprise architects do their work at the architecture level of abstraction, the implementation details are left to those who are most concerned with putting the principles of SOA into practice.

Ever since we called out coarse granularity as a Web Services Idée Forte (powerful idea) way back in 2002, SOA architects have continued to struggle with the concept of granularity. Unlike the other two Idées Fortes—asynchrony and loose coupling—the principle of granularity hasn’t gotten the respect that it deserves over the years. In spite of this oversight, ZapThink considers granularity to be a core SOA design principle, and we treat it as such in our Licensed ZapThink Architect course.

From a technical perspective, the granularity of an interface impacts performance. Excessively fine-grained interfaces lead to increased traffic on the network, because the consumer must re-query the provider multiple times to complete a task. However, if the performance impact of excessive fine granularity were the whole story, granularity would be an important consideration, but not important enough to rate as a core SOA design principle.

The reason we consider granularity to be so significant is because there is a general correlation between coarse granularity and business context. Generally speaking, when the business makes a request of a Service, then either the request or the response (or both) have some structure to it. For example, if you submit an order, you are sending a number of details that have business context—namely, the details of the order. Likewise, if you request customer information from a Service, for example, you typically want a number of details.

On the other hand, fine-grained Services tend to have a technical context but lack a business context. Submit a number, get a number in response, for example. There are exceptions, naturally: a credit score Service is the one we talk about in the LZA course. Submit a Social Security number, get back a three-digit credit score. In spite of examples like this one, however, coarse granularity generally correlates with business context.

The tradeoff with coarse granularity is that while it correlates with business context, coarse grained Services also tend to be less reusable, because the interface becomes increasingly specific to a particular purpose. Architects must therefore balance reusability and business context when they consider granularity in their interface designs.

Distinguishing Functional Granularity

Up to this point we’ve been discussing the granularity of the interface. After all, Services are interfaces or abstractions of interfaces, so among an architect’s first considerations when designing a Service is the granularity of the interface. However, there is more to granularity than the interface: the granularity of the underlying implementation, what ZapThink calls functional granularity.

For example, let’s consider an “add product to order” Service. Send this Service a product number, and it adds the product to an existing order and returns a confirmation. Clearly, this Service has a fine-grained interface.

Now, what if the product you were adding was actually a set of products? For example, you add a laptop to an order and the laptop, battery and several questionable pieces of software come along for the ride. You could call add product Service many times, of course, but that might lead to a performance bottleneck, and you may not know all the individual product numbers in the set anyway. Instead, rework the add product to order Service so that you can send an ID that represents a set of products as an input. Now you can add any number of products with a single Service call. The granularity of the interface doesn’t change, but the new Service’s functional granularity is now much coarser.

The next design question is whether the add (single) product to order Service and the add multiple products to order Service should be two separate Services or a single Service. You could argue that you should implement two Services with different levels of functional granularity as two distinct Services. However, this design decision introduces unnecessary consumer-side coupling, because the consumer has to know which Service to call in order to elicit a specific response. Instead, you should define the functional granularity of a Service in the Service contract, as part of the semantic definition of the Service, so that you can combine both situations into the same Service.

Because the functional granularity of a Service would otherwise be hidden from the consumer, it’s important to provide the information about this behavioral aspect of the Service in the contract, so that you can expose a single abstracted business Service that represents whatever level of granularity consumers require from the Service. This approach also increases the reusability of the Service. In fact, separating functional granularity from interface granularity allows architects to have it both ways, because the interface may be fine-grained, improving reusability, while the functionality is coarse-grained, increasing the business context for the Service.

Functional Granularity of Composite Services

Adding products to an order is a simple example of coarse functional granularity, since a developer would likely be able to implement such a Service with a single well-crafted database call. In other instances, however, it’s more feasible to implement the underlying functionality of the Service as a composition.

For example, let’s say you wanted to implement an approve mortgage Service. Send the service an ID number for a mortgage application, and the Service executes a multiple-step mortgage approval process and returns a confirmation. The approve mortgage Service is another example of a Service with coarse functional granularity and fine interface granularity. Once again, the Service has a clear business context, which correlates with the functional more so than the interface granularity.

The ZapThink Take

Our 2002 Idées Fortes article focused on Web Services, first, because 2002 was the heyday of SOAP-based Services, and second, because SOAP messages lend themselves better to coarse-grained interactions, because of the substantial overhead that SOAP  requires.

Our discussion today, however, refers to Services much more generally. The entire discussion applies equally well to REST-based interactions as it does to SOAP-based Web Service interactions. In fact, the use of REST-based Services should emphasize functional granularity in their design. The use of the GET operation (one of only four possible operations as part of the HTTP-centric REST approach) uses the URL as the mechanism to specify the resource-based Service the consumer is querying. While it is possible to use query strings to add more data to the GET call, this approach is problematic if you want to query a range of resources rather than a single resource. Alternatively, an architect can use a URL hierarchy or a different Service to refer to a group of resources rather than a single one, but these alternatives face the same tight coupling issue mentioned earlier.

Instead, the architect should leverage the concept of coarse functional granularity. Create a GET Service in which the resource in the query can refer to either a single resource or multiple resources. In this way, the Service can increase its functional granularity without any change to the interface granularity. Indeed, the existing Service won’t fail even as you increase its functional granularity.

This REST-specific angle highlights an important point. From the architect’s perspective, the choice of message and transport protocols is separate from the granularity consideration. The architect’s focus is instead on balancing reusability and business context while maintaining loose coupling: a focus core to proper Service-Oriented thinking, regardless of whether the Services are SOAP or REST-based.

We at ZapThink were encouraged by the fact that our recent ZapFlash on Deep Interoperability generated some intriguing responses. Deep Interoperability is one of the Supertrends in the ZapThink 2020 vision for enterprise IT (now available as a poster for free download or purchase). In essence the Deep Interoperability Supertrend is the move toward software products that truly interoperate, even over time as standards and products mature and requirements evolve. ZapThink’s prediction is that customers will increasingly demand Deep Interoperability from vendors, and eventually vendors will have to figure out how to deliver it.

One of the key points in the recent ZapFlash was that the Web Services standards don’t even guarantee interoperability, let alone Deep Interoperability. We had a few responses from vendors who picked up on this point. They had a few different angles, but the common thread was that hey, we support REST, so we have Deep Interoperability out of the box! So buy our gear, forget the Web Services standards, and your interoperability issues will be a thing of the past!

Not so fast. Such a perspective misses the entire point to Deep Interoperability. For two products to be deeply interoperable, they should be able to interoperate even if their primary interface protocols are incompatible. Remember the modem negotiation on steroids illustration: a 56K modem would still be able to communicate with an older 2400-baud modem because it knew how to negotiate with older modems, and could support the slower protocol. Similarly, a REST-based software product would have to be able to interoperate with another product that didn’t support REST by negotiating some other set of protocols that both products did support.

But this “least common denominator” negotiation model is still not the whole Deep Interoperability story. Even if all interfaces were REST interfaces we still wouldn’t have Deep Interoperability. If REST alone guaranteed Deep Interoperability, then there could be no such thing as a bad link.

Bad links on Web pages are ubiquitous, of course. Put a perfectly good link in a Web page that connects to a valid resource. Wait a few years. Click the link again. Chances are, the original resource was deleted or moved or had its name changed. 404 not found.

OK, all you RESTafarians out there, how do we solve this problem? What can we do when we create a link to prevent it from ever going bad? How do we keep existing links from going bad? And what do we do about all the bad links that are already out there? The answers to these questions are all part of the Deep Interoperability Supertrend.

One important point is that the modem negotiation example is only a part of the story, since in that case, you already have the two modems, and the initiating one can find the other one. But Deep Interoperability also requires discoverability and location independence. You can’t interoperate with a piece of software you can’t find.

But we still don’t have the whole story yet, because we must still deal with the problem of change. What if we were able to interoperate at one point in time, but then one of our endpoints changed. How do we ensure continued interoperability? The traditional answer is to put something in the middle: either a broker in a middleware-centric model or a registry or other discovery agency that can resolve abstract endpoint references in a lightweight model (either REST or non-middleware SOA). The problem with such intermediary-based approaches, however, is that they relieve the vendors from the need to build products with Deep Interoperability built in. Instead, they simply offer one more excuse to sell middleware.

The ZapThink Take

At its core Deep Interoperability is a peer-to-peer model, in that we’re requiring two products to be deeply interoperable with each other. But peer-to-peer Deep Interoperability is just the price of admission. If we have two products that are deeply interoperating, and we add a third product to the mix, it should be able to negotiate with the other two, not just to establish the three pairwise relationships, but to form the most efficient way for all three products to work together. Add a fourth product, then a fifth, and so on, and the same process should take place.

The end result will be IT environments of arbitrary size and complexity supporting Deep Interoperability across the entire architecture. Add a product, remove a product, or change a product, and the entire ecosystem adjusts accordingly. And if you’re wondering whether this ecosystem-level adjustment is an emergent property of our system of systems, you’ve hit the nail on the head. That’s why Deep Interoperability and Complex Systems Engineering are adjacent on our ZapThink 2020 poster.

There are a number of factors at play here, as Ron Schmelzer, also a managing partner at ZapThink, says in an IDG article on the WS-I announcement. Web services have lost ground to the alternative REST Web services and mergers, and acquisitions have “also diminished issues of interoperability,” Schmelzer points out.

Read the entire post at http://www.itbusinessedge.com/cm/blogs/lawson/is-the-web-losing-ground-on-freedom-interoperability/?cs=44432.

I was surprised to learn that the Web Services Interoperability Organization (WS-I) completed its work this week, transferring its intellectual property to OASIS and calling it a day. Hooray! All Web Services implementations can now be fully interoperable! Simply make sure your vendors follow the now-final WS-I practices and conflicts between products from different vendors are now a thing of the past!

Not so fast.

At ZapThink we like to say that open standards are the bumps on the Legos. The whole point to standards like those in the Web Services family is to provide seamless interoperability between products that support those standards. After all, what fun would Legos be if the bumps were all different sizes?

Unfortunately, standards alone never guaranteed interoperability, which is why WS-I existed in the first place. The WS-I committee hammered out specific ways to implement specific standards, offering guidance to vendors on how to implement the standards to guarantee interoperability.

So now, the WS-I committee says that they’re all done. But wait! Where’s our seamless interoperability?

While the Web Services standards were an important step in the long road to true interoperability, the static interfaces that these standards specify are only part of the story. Contracted interfaces alone can never guarantee interoperability long-term.

The story, of course, is one of change. The Lego metaphor breaks down, because the bumps have never needed to change. Today’s Legos interoperate with Legos from fifty years ago, because our requirements for the plastic blocks are essentially the same as they were back then. But in the world of IT, change is constant, as it is for the business requirements that drive IT. Any fixed set of standards, no matter how robust and detailed, will never solve the interoperability problem for long.

Lest we forget, this story is not a new one. Remember the analog modem market from the 1980s and 1990s? 300 baud acoustic couplers gave way to 1200 baud then 2400, on up to 56K before newer technologies like ADSL took over. But in the heyday of the analog modem, each box supported multiple standard protocols, and would attempt to negotiate with the modem on the other end of the line using the fastest protocol it knew. If that didn’t work, each modem tried the next slower one, and the next, until they were able to establish a connection.

We need the same kind of process now with general product-to-product interoperability, only while the modem interaction story was single-dimensional, now we’re working in multiple dimensions. As a result, the “modem negotiation” model of interoperability is now orders of magnitude more complex.

Eventually, however, the vendors will figure it out. At that point we will have what ZapThink calls Deep Interoperability. Deep Interoperability – one of the ZapThink 2020 Supertrends – suggests that products actively change how they seek to interoperate until they’re successful. Modem negotiation on steroids.

Imagine: you buy a new enterprise IT product. You hook it to the network and turn it on. It automatically negotiates with everything else in your environment until it’s fully integrated. And when something changes, it negotiates again as needed.

Are vendors working on Deep Interoperability now? It’s not clear. Unfortunately, they’re not very motivated to get interoperability right. After all, proprietary interfaces lead to customer lock-in, and all vendors want that.

So maybe that’s why WS-I has thrown in the towel. Vendors were driving the effort all along, of course. They love paying lip service to standard interfaces, because customers demand them. But if such standards really worked – if products truly interoperated, in a way that stood the test of time – then you could easily replace a poorer product with a better one.

And what vendor today wants that?

Read the entire article at http://news.idg.no/cw/art.cfm?id=36A5E21F-1A64-67EA-E415D22D9DB80589.