Conventional wisdom would have you believe that Public Clouds are inherently insecure, and that the only way to meet your organization’s stringent security requirements in the Cloud is to implement your own Private Cloud. Conventional wisdom, you say? Unfortunately, there is precious little wisdom available of any kind when it comes to Cloud Computing, let alone the conventional type!

In fact, large software and hardware vendors are largely responsible for the whole “Public Cloud is insecure” canard, introducing fear, uncertainty, and doubt (FUD) into the marketplace. After all, building a Private Cloud means buying a lot of new gear. The last thing the big vendors want is for their customers to move to Public Clouds—unless, of course, they belong to the vendor in question. Don’t be fooled. Public Clouds are typically more secure than Private Clouds, for a number of reasons. Here’s why.

Why Public Clouds are More Secure…

• Hardened thru continual hacking attempts – Public Cloud providers are a juicy target. Hackers know how to find them, realize there’s good stuff inside, and would be the envy of all their hacker pals if they were able to breach the Public Cloud’s defenses. As a result, h4x0r types have been hammering on Amazon Web Services, Microsoft Azure, and all the others. Thousands of them. For years now.

• Attract the best security people available – Public Cloud providers not only attract hackers, they attract talent. If you’re a top Cloud security expert, where would you rather work: Amazon? Or some big insurance company or manufacturer or government agency? I thought so.

• Get the latest security gear due to economies of scale – How many Cloud data centers do the big Public Cloud providers own? And how fast are they building new ones? You don’t need to know the specifics to realize the answers are boatloads and wicked fast. And they’re buying gear for them. New gear. Boatloads of it. Wicked fast.

Why Private Clouds are Less Secure…

• Suffer from “perimeter complacency” – it’s amazing how many enterprises think that their DMZs and firewalls give them adequate security. If it’s on the internal network, it must be secure! As though they completely missed the Internet. And email. Not to mention viruses. What about twenty-somethings downloading malware to the corporate network through their phones? Now the enterprise wants a Private Cloud, so they can put the whole kit and caboodle on their internal network for security purposes. Good luck with that.

• Unknown staff competence – sure, your organization has a lot of great security people. They all know their stuff. Try this: have a big party for them. Two hours in, take a look around the room. See that guy with the lampshade on his head? He’s responsible for Private Cloud security.

• Insufficient penetration testing – how do you test to make sure your Private Cloud is secure—or any other part of your IT infrastructure, for that matter? Simple: have your testers run a series of security tests. Or maybe hire a third party to run them for you. If all the tests pass, you’re secure, right? Maybe for like a minute, until the hackers figure out new attacks that didn’t make it into your security tests. Whoops.

• May have older gear in use – you spent hundreds of thousands of dollars on security hardware. In 2009. Now you’re putting the final touches on your Private Cloud. Try this: ask your CIO for hundreds of thousands of dollars more to replace that three-year-old gear. The response? Maybe next year. Try updating the patches. I’m sure you can make do with what we have. And maybe you can—but don’t expect it to compare with the brand new shiny stuff going into Public Cloud data centers every day.

Virtual Private Clouds to the Rescue?

With a Virtual Private Cloud (VPC), a Public Cloud provider gives you a dedicated, secure connection (usually via a VPN) to your Public Cloud instances. In some cases, those instances are physically separated from other customers, so that your stuff can’t end up on the same box as somebody else’s stuff.

VPCs may actually be the most secure option available today, as you have the best of both worlds. Furthermore, they may address specific regulatory or other governance issues that may prevent your organization from using a multitenant Public Cloud. If you read the first section of this ZapFlash and think that neither Public nor Private sounds secure enough, then a VPC may be the way to go.

However, VPCs aren’t for everyone. They may only be marginally more secure than Public Cloud, as Public Cloud providers have generally done a bang-up job securing their multitenant architectures. And keep in mind, a single-tenant VPC will typically be substantially more expensive than a regular Public Cloud equivalent. The bottom line: VPCs are more about peace of mind than actually increasing security.

The ZapThink Take

You’ll have to excuse me, I’m in a particularly snarky mood today. I must admit that the title of this ZapFlash is actually an overgeneralization. It’s certainly possible that your Private Cloud is more secure than some Public Clouds out there. The true message of this article is that building a truly secure Private Cloud is much harder than it sounds, and the extra work necessary has largely already been taken care of by the Public Cloud providers. And it should now be obvious that Private Clouds are by no means inherently more secure than Public ones.

But there’s a bigger lesson here. Security is all about risk mitigation, and it’s simply impossible to reduce your risk to zero. There’s no such thing as perfect security, which is another way of saying that perfect security is infinitely expensive. Risk mitigation involves weighing acceptable risks, given the nature of those risks and the cost involved in mitigating them. When you deliberate on the question of Public vs. Private Clouds, keep in mind that both approaches are inherently risky—but then again, choosing neither is also risky. Your job is to get the necessary facts in order to make the best decision you can about which risks you are willing to accept. Confuse FUD with facts at your peril.

Your data center, sometime in the mid 1990’s: the server you ordered finally arrives. Could be Windows, Linux, some flavor of Unix, doesn’t matter. You unpack it. Boot it up. Patch the OS. Configure the OS. Install software off of CDs. Patch the software. Configure the software. Move data to the box. Test. Tweak. Test again. Finally, the box goes live.

Cut to 2012. You’re working in the Cloud now. You provision a virtual machine (VM) instance in the Cloud. Or three. Or maybe a few dozen. Only you’re not just provisioning VMs. You also provision some dynamic storage. Maybe some Cloud-based queues. You also want some SaaS-based services.

And your software release cycles? Weekly. No, daily. How about hourly?

Now what?

Clearly, it’s impractical to set up your Cloud instances manually, the way we used to set up servers in the good old days. So you go through the process once and create an image file that represents your Platonic ideal of what a fully configured VM instance should look like. Now, every time you need to provision a new VM instance, simply reconstitute the image. Right?

Not so fast! There are numerous gotchas to this scenario. Every time you need to patch anything, you would need to create a new image. If different VM instances are meant to differ in any way – say, contain different application data – you would need to configure those differences manually. But most significantly, there is far more to your Cloud environment than single VM instances. What about the storage? Databases? Network configuration? What about the architecture?

The Basics of Automated Provisioning

Remember, “automated” means “not manual,” in the sense that hands are not allowed. You want the ability to deploy, update, and repair your entire application infrastructure using nothing but pre-defined, automated procedures. Ideally, you want to automatically provision your entire environment from bare-metal (hardware with no operating systems – or anything else – installed on them) all the way up to running business services completely from a pre-defined specification, including the network configuration. Furthermore, there should be no direct management of individual boxes. You want to manage the entire Cloud deployment as a single unit.

Deploying sophisticated provisioning tools, of course, is a large part of the secret. And the more sophisticated the tools, the less skilled your staff has to be. Ideally, any people familiar with a few basic commands and appropriate permissions should be able to deploy any release to any integrated development, test, or production environment. They only require minimal domain specific knowledge. You don’t need a senior sysadmin. You don’t even need a systems developer. Any junior techie should be able to handle the task.

If something goes wrong, you should be able to revert to a “previously known good” state at any time. In a mature Cloud environment, it’s always easier to reprovision than it is to repair. Reprovisioning could mean an automated cycle of validating and regenerating application and system configurations, or even rerunning the full provisioning cycle from the base OS up to running business applications.

In many cases, of course, the previously known good state isn’t good enough, typically because there are live data in the real time state that would be lost with this kind of rollback. As a result, such rollbacks must be handled carefully, as they really aren’t rollbacks in the sense of a two-phase commit. Instead, with fully automated provisioning, the provisioning system should be able to “roll forward to a previous version,” where the provisioning tools will automatically return your applications to a functionally acceptable state, with all your data intact.

Automated provisioning depends upon the environment specification. This spec is essentially a declarative representation of how you want your entire deployment. Your provisioning tools will then essentially execute the spec, starting with bare metal and possibly stock virtual machine images, and then they will automatically deploy, configure, and start up the entire system or the application stack (or both), with no runtime decisions or tweaking by an operator. The spec should also contain sufficient detail to direct the appropriate tools to test whether the automation is implemented correctly, and if it isn’t, to take the appropriate action.

This specification can be as sophisticated as your tools and your architecture allow it to be. It may vary from release to release, and you should be able to break it down for specific tools that handle different parts of the configuration. The spec may also have conditional logic, and can also specify deployment or configuration changes over time, for example, the instruction to provision additional instances when traffic numbers cross a threshold.

You may also want to handle the automatic configuration of the application stack separately from the configuration of the system stack, as your applications may change more frequently than the systems. The goal is to make the spec sufficiently sophisticated so that the automation itself doesn’t vary from release to release. It will only require updates when your requirements call for a significant architectural change.

The View from Above and Below the Clouds

There are fundamentally two sides to this story: the view from the perspective of the Cloud service provider (including the internal providers of private Clouds), vs. the view from the consumer of Cloud-based resources. Clearly, Amazon, Microsoft, and the other public Cloud providers have figured out how to automate the configuration of their public Cloud environments. For organizations building their own private Clouds, the challenge is to take a page out of the public service providers’ playbooks on how to run a Cloud environment. Bottom line: if you don’t get automated configuration management down pat, you’re not running a private Cloud at all. You simply have a traditional data center with some Cloud-like features – and furthermore, you have a data center that is more expensive to run than necessary.

If you’re in a position to consume Cloud resources, regardless of the Cloud deployment model, then automated provisioning is every bit as important as it is for Cloud service providers, only now it impacts your existing IT processes and policies. As organizations adopt the Cloud, they increasingly transform the role of operations. No longer does your ops team actually take care of servers, networks, and applications. Instead, you’re automating that work, shifting the expertise required to the development team who must now create and manage the automation scripts that form the specification. Or perhaps the ops team moves their cubicles to the dev area, working hand-in-hand with developers to handle those scripts. Either way, Cloud changes everything in the IT department.

The Realization of the DevOps Vision

Reworking the relationship between dev and ops, or DevOps, is nothing new, of course. According to Wikipedia, “DevOps is an emerging set of principles, methods and practices for communication, collaboration and integration between software development (application/software engineering) and IT operations (systems administration/infrastructure) professionals. It has developed in response to the emerging understanding of the interdependence and importance of both the development and operations disciplines in meeting an organization’s goal of rapidly producing software products and services.” While ZapThink hasn’t discussed DevOps by name up to this point, we have been calling for iterative, full-lifecycle governance for several years now – an essential enabler of success with SOA in particular and agile architectures in general.

With the rise of Cloud Computing, DevOps is entering what might be its “golden age.” As Cloud provisioning specifications become more sophisticated, creating them becomes more of a development task than an operational one. Ops doesn’t go away, of course, but it moves to the other side of the Cloud: supporting Cloud data centers. In other words, if you have a private Cloud, your ops team is responsible for managing the private Cloud infrastructure. And yes, if you use a public Cloud, well, you have the luxury of outsourcing operations to your Cloud provider. Good sysadmins need not worry, of course. If anything, demand for your skills is only increasing with the move to the Cloud.

The ZapThink Take

First there was software development. Write a bunch of code and run it on a computer – “the computer is the computer.”

Then there was systems development. Write a bunch of code and put it on a bunch of computers, and have them serve up bits of it to many more computers – “the network is the computer.”

Now we’re at the dawn of Cloud development: create sophisticated Cloud provisioning/deployment/management specifications, and run those in the Cloud. Yes, the Cloud itself becomes the computer. We’re not talking IaaS, PaaS, or SaaS here. Even those oh-so-2011 Cloud service models are only elements of the spec, for automated provisioning tools to provision and configure dynamically.

We’re not there yet, of course – but there are a number of increasingly sophisticated automated provisioning tools on the market today, and an increasing number of organizations are leveraging them to take full advantage of the Cloud. Want to learn more? ZapThink covers automated provisioning, including a broad discussion of available tools, in our Enterprise Cloud Computing and Architecting for the Cloud courses. We’re running them in Singapore February 23-24, Sydney February 27-28, Melbourne Australia February 29-March 1, Delhi March 5, Mumbai March 6, Hyderabad March 7, Bengaluru March 8, and London March 15-16. Be prepared for a comprehensive, vendor-independent, architecture-focused fire hose of everything Cloud. There is no way to get this material from anyone but ZapThink. Classes are filling up so register now. We hope to see you at one of the courses soon!

Image source: Horia Varlan

A funny thing happened as I was noodling on this ZapFlash. I was all set to put the nail in the Application Programming Interface (API) coffin, continuing the discussion of just how awful Remote Procedure Call (RPC) interfaces are, and how we should avoid them at all costs. But then I ran across the “explosion of APIs” meme at ProgrammableWeb and elsewhere. After all, your head must be planted deeply in the sand not to notice the plethora of APIs at all our favorite Social Web sites like Twitter, Facebook, and hundreds of others. How can the API be dead when APIs are more important and plentiful than ever?

Maybe I’m just nitpicking. As APIs become less RPC-centric and more RESTful, the inflexible tight coupling of the past is giving way to a new golden age of API-ness, where anybody can connect any piece of software seamlessly and automatically to any Web site or app we might care to use. But upon closer reflection, my concern is not all nitpickiness. There are API best practices and API worst practices, just as there are good APIs and bad ones. Perhaps more to the point, most of the APIs out there, well, are among the bad ones. So, what makes for a good API today, with all the power and capabilities that modern technology approaches afford us? Why are so many people getting them wrong? And what can we do to steer everyone in the right direction?

Programmable Interfaces: The Never-ending Story

The real reason that APIs get under my skin is the “P” – “programmable,” as in programmatic. The core notion of an API, after all, is that you want one application to be programmable by something external to it, in the sense that you have an imperative control flow that goes from here to there and back to here.

The problem with such imperative programming in any distributed environment, of course, is that there is no end of problems that can arise between here and there. Network issues, incompatibilities, timing and sequence issues, the list goes on and on, leading computer scientists to the notion of functional programming. In the functional world, functions (which also go by the names procedures or methods) encapsulate the software at the remote location, forming black boxes with fixed inputs and outputs. Any local piece of software, therefore, can call a remote procedure and get the desired response, without having to worry about whether multiple calls to that procedure might interfere with each other or lead to other nasty surprises that a purely imperative approach might exhibit.

Here’s the rub: functional programming is nothing more than RPC, with all the tight coupling issues that come along for the ride. However, functional programming is also considered to be a type of declarative programming. Declarative programming has always been the ugly duckling of the programming language world, because with a declarative approach, you don’t specify the control flow. Instead, you describe the desired behavior you want the software to exhibit. And somehow, via some kind of behind-the-scenes hand waving, the software miraculously does what you want it to.

Functional programming is declarative in the sense that a remote procedure acts as a black box. Specify the operations, inputs, and outputs (its signature, in API-speak), and the inner workings of the procedure aren’t your problem. That is, unless you don’t understand what it’s supposed to do for you, or heaven forbid, its behavior changes.

REST to the Rescue?

Building abstracted interfaces that enable organizations to deal better with change is what SOA is all about, of course. Unfortunately, the tools we had at our disposal – namely, Web Services – didn’t go far enough. Yes, we moved from RPC-style Web Services to document-style, and that improved our loose coupling substantially. But even document-style Web Services still have operations, a holdout from the bad old functional programming days.

Enter REST. ZapThink has discussed the move away from RPC to RESTful interfaces before: in REST-Based SOA: an Iconoclastic Approach, but of course, we’ve been discussing the advantages of document style interfaces over RPC for several years now. But people still aren’t getting it. Not only are techies missing the point of REST, even when they build supposedly RESTful APIs, they’re failing to follow the REST constraints.

For example, many of these ostensible RESTful APIs don’t provide self-descriptive messages. With REST, every message from a resource should contain all the metadata necessary for any client to understand what it can do with representations of that resource. In the RPC days, procedural responses may have contained only data with no metadata. With Web Service interactions, Service responses contained some metadata in the form of the SOAP envelope, header, and the tagging structure in the body. But even with Web Services, the WSDL contract and policy metadata are external to the Service. You don’t expect to get the contract when you query a Service.

With a RESTful interaction, on the other hand, contract metadata may be returned in particular representations, whether it be media type metadata, schemas, or even less structured metadata. And of course, representations also include hyperlinks, which also provide contract metadata to the client.

Separation of resources from representations is another essential REST constraint – one that developers also frequently violate. If there are any instructions to the resource in a request other than one of the four operations of the uniform interface (GET, POST, PUT, and DELETE), then you are violating this constraint. For example, if your request is instructing a resource to update a record, you’re not being RESTful. It’s up to the resource to determine whether a POST or PUT should update a record, not the representation.

Finally, hypermedia as the engine of application state – the dreaded HATEOAS – is perhaps the most overlooked of the REST constraints, even though it’s the most important. The big picture of HATEOAS are the hypermedia applications that we’re trying to build, but even at the API level, HATEOAS is critical, in conjunction with the self-descriptive message constraint. In any truly RESTful interaction, the resource returns a representation that contains all necessary metadata, including hyperlinks that instruct the client what it can do next. Likewise, a RESTful client has no idea what a resource can do for it unless it follows such links.

“RESTful” Equals “Web-friendly”

What so many techies lose sight of is the fact that REST isn’t supposed to make the Web more like system integration; it’s meant to make system integration more like the Web. When you click a link in a Web page, you expect to go to another Web page, or perhaps a video or sound file or some other media representation. REST takes that simple interaction and abstracts it. Now you have an abstracted client (instead of a browser) supporting a request of a resource (the Web server capability, for example, the php script that generated the response) on an abstracted server (the Web server) that returns a representation (the resulting media file or Web page) as per the uniform interface (what the link you clicked was supposed to do). These abstractions let us apply simple Web behavior to all manner of system-to-system interactions. But never, ever lose sight of the fact that you want simple Web behavior from your application.

I’m sure many readers who made it this far are muttering to themselves that their RESTful APIs are truly RESTful. Well, maybe, but probably not, and here’s proof. Let’s take a popular, supposedly RESTful API as an example: the Yahoo! Maps PlaceFinder Geocoding API. You’ll notice on the documentation page a “BASE URI,” which it expresses as http://where.yahooapis.com/geocode?[parameters]. There are two problems with this expression. First, it’s not a hyperlink (it just looks like one). Second, the URL has the question mark in it, requiring the user to know what parameters might be valid.

Let’s say instead we turn this expression into a true hyperlink, leaving off the parameters since we don’t know what they are: http://where.yahooapis.com/geocode. Go ahead,  click on that link, I dare ya. Sure enough, you get an error message. True, it’s XML formatted, but it’s missing something absolutely essential: a hyperlink.

In fact, Yahoo! made several serious mistakes: first, the base URI isn’t a hyperlink, violating HATEOAS. Second, the metadata that tell you how to use the interface are separate from the interface, rather than  returned in the response from hyperlinking to the base URI, violating the self-descriptive messages constraint. Third, the URI requires parameters as part of the search query (the characters after the question mark), violating the separation of resource and representation. And finally, the error response it did return didn’t include a hyperlink, once again violating HATEOAS. Bottom line: there’s really very little that’s RESTful about this API.

While it may be amusing to pick on Yahoo!, it’s important to point out that they aren’t alone. In fact, they’re typical. It’s quite rare, in fact, for a supposed RESTful interface to be truly Web-friendly. After all, most techies don’t expect Web friendliness from APIs. APIs are, well, programming interfaces, while the Web consists of user interfaces. But in the REST world, programming interfaces are simply abstracted user interfaces. In other words, truly RESTful APIs aren’t really like the APIs we know and love, and have been working with for years. They’re fundamentally different.

The ZapThink Take

I know this ZapFlash is deeply iconoclastic, but that’s what you’ve come to expect from ZapThink, after all. However, the point of this article isn’t to scold everybody for doing REST wrong. The point is to help you think differently about what it means to program in a distributed environment. The point to the “programmable Web” isn’t to make the Web more programmable, it’s to make software more Web-like. If we can finally free ourselves from the last vestiges of imperative, RPC-style programming, even going so far as to steer clear of functional programming, and move to a fully declarative, document-centric paradigm, only then will we be able to achieve the resilience and power of the Web when we tackle system integration challenges.

This mind shift is at the core of REST. When Roy Fielding wrote his dissertation, he didn’t come up with REST and then build the Web following its constraints. On the contrary: he deeply understood what made the Web itself so special, and he sought to express that specialness as an architectural style. True, tackling system integration following true RESTful principles is difficult. But remember, building the Web was easy. Take TCP/IP, add a few simple standard protocols, code an HTML rendering app we called a browser, and the Web more or less took it from there. Almost twenty years later, the Web is unimaginably immense, and yet it still works just fine, thank you very much. Is it too much to ask for all of our IT systems to behave the same way?

Image source: cloudzilla

The presents are unwrapped, family has headed home, and the leftovers are almost gone. Once again, it’s time for ZapThink’s annual retrospective and predictions for the coming year. While it’s true that many pundits and prognosticators churn out their guesses for the year to come around this time, ZapThink takes the practice one step further: we actually review the previous year’s predictions and score ourselves on how well we did. Here then are our results from last year and our predictions for the year to come.

2011 Predictions: 2 ½ out of 3

I’m proudest of how well we did on our first prediction: one spectacular Cloud flameout. We so nailed this one, with Amazon Web Services well-publicized crash in April. I can’t help but say we told you so. We even went so far as to say “if you believe your Cloud provider is invulnerable, you’re fooling yourself.” Bottom line: expect for and plan for failure, especially in the Cloud.

We did well on our second prediction as well: that we’d run out of IPv4 addresses in 2011, and that the resulting shortage would lead to the growth of a secondary market in IP addresses. In fact, a black market for IP addresses has sprung up, in addition to the more above-board purchases, like the addresses Nortel sold to Microsoft. If the IPv4 shortage hasn’t impacted your organization yet, then 2012 might be the year you feel the pinch.

Our third prediction, however, is more of a work in progress. We predicted an “Enterprise Architecture is dead” story that would parallel the “SOA is dead” meme from 2009, as organizations move away from checklist architectures to more agile approaches that show better results. We’re giving ourselves 50% on this one, as no “EA is dead” meme has cropped up, or at least, no more in 2011 than in years previous. But speaking from personal experience, an increasing number of organizations are looking to move toward more agile, cost-effective architectural approaches, notably in the US Government with CIO Steven VanRoekel’s “Future First” initiative. As checklist architectures were endemic across the government, this shift is a good sign that our tax dollars are finally being spent wisely on architecture, at least here in the US.

Before we move on to our predictions for 2012, there’s one more prognostication to mention, from ZapThink’s predictions for 2008. We predicted that ZapThink would be acquired that year. We were off by three years, but we’re  now a happy part of Dovèl Technologies!

Predictions for 2012

Future First steals thunder from the Cloud – Our first prediction centers on the US Government, who is moving headlong into the Cloud. Dozens of agencies are taking the Cloud First initiative to heart, moving email, content management, and other key capabilities to the Cloud. Now, with FedRAMP ready to take off, agencies will be able to reduce the cost and complexity of moving to the Cloud.

But perhaps, not so much in 2012. We do predict FedRAMP will save billions of dollars of IT costs eventually, but there is simply too much politics involved to expect it to pay off substantially this coming year. In fact, we expect government adoption of the Cloud in 2012 to be more about politics than achieving real value. Instead, it will be a year of expanding the focus to Future First, as various federal departments and agencies, in conjunction with industry and academia, flesh out the combination of Agile, Lean, open source, and Cloud that VanRoekel has in mind. Cloud won’t go away, of course, as Cloud is an integral part of Future First. But the focus will gradually shift to Future First.

HTML 5 becomes the glue of the ZapThink 2020 vision – As the next major version of the Web’s central markup language, HTML 5 has been a long time in coming. In 2012, we predict it will hit a critical tipping point, as the next generation of Web markup, the replacement for Adobe Flash, and the common technology framework for Mobile app development. In addition, HTML 5’s Web Sockets technology that allows for full duplex tunneling via HTTP will open up new approaches for integrating capabilities in the context of rich, cross-platform user interface experiences.

As a result, HTML 5 will provide technology underpinnings, not only for mobile computing, but for ongoing improvements in technology across the board. After all, the user interface is where the value of technology manifests itself. We predict, therefore, that HTML 5 will impact multiple facets of enterprise IT, well beyond the user interface. Stay tuned for more thought leadership from ZapThink on this topic.

The rise of RESTful BPM – Just as SOA was misconstrued as an ESB-centric technology approach, people have confused REST with a way to construct APIs. However, that’s not what REST is about. REST is an architectural style for building hypermedia applications. And while you can think of a hypermedia application as a glorified Web site, it’s actually more than that: it’s a runtime workflow.

The idea of RESTful applications as runtime workflows is both simple and deeply powerful, as it entirely changes the way we might look at Business Process Management (BPM) tools. Traditional BPM tools are heavyweight, integration-centric tools that typically rely upon layers of infrastructure. Even lightweight BPM tools still rely upon centralized state engines. With REST, however, hypermedia become the engine of application state, freeing us from relying upon centralized state engines, instead allowing us to take full advantage of Cloud elasticity, as well as the inherent simplicity of REST.

Will people understand this shift in BPM approaches in 2012? Largely, no. But what will happen is that RESTful BPM will at least appear on the radar of organizations looking to approach process automation and optimization that leverages the Cloud without falling into the big vendor trap. What we do predict is more buzz about RESTful BPM, and at least one vendor bold enough to offer a truly RESTful BPM tool, instead of the non-RESTful attempts that we currently see from the vendors today. Are you a vendor who believes you offer a RESTful BPM tool that actually follows the REST constraints? We want to hear from you.

The ZapThink Take

ZapThink is pleased with the progress enterprise IT is making toward our ZapThink 2020 vision. The point to this vision was that individual trends in IT interrelate, and thus you shouldn’t separate any of them from the broader context of change within the organization. Whether it be SOA, Cloud Computing, mobile computing, or any of the five Supertrends, the only way to understand the message of ZapThink 2020 is to take them all together.

Our predictions for 2012 follow this pattern. The point to Future First is that it brings together Agile approaches with the Cloud and more. Our expectations for HTML 5 focus not on the user interface, but on how this standard will facilitate the ZapThink 2020 vision across enterprise IT. Likewise, RESTful BPM turns traditional integration—and even traditional SOA—on their heads, circling back to the World Wide Web as the central organizer for business at large.

Our predictions, therefore, are somewhat convoluted, not by accident, but by necessity. Pulling single strands out of the complex knot we call enterprise IT wouldn’t tell a useful story. Only by leaving the strands together and exploring how they interrelate can we gather useful insight into the change that faces us every day.

On two occasions I have been asked,—”Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?” … I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question.

—Charles Babbage, 1864

The long-standing computer science principle of “garbage in, garbage out” (GIGO) is so fundamental to IT that it predates digital computing by almost a century. And yet here we are in the twenty-first century, moving to the Cloud, and Babbage’s exasperated response is no truer or more on point. For not only is the Cloud a magnet for all sorts of garbage, it is also generating new garbage at a brisk clip.

Uploading Garbage to the Cloud

In today’s frantic rush to “move to the Cloud,” too many organizations are failing to ask what they should move to the Cloud. Instead, they envision the Cloud as some kind of huge, nebulous server in the sky, a perfect receptacle for  whatever they have on-premise. Got email? Put it in the Cloud! Got data? Put your data in the Cloud, the bigger the better! Running business processes on-premise? Move them to the Cloud!

Not so fast. Let’s slow down a bit and consider the ramifications of moving too quickly—and haphazardly—to the Cloud.

• Unclean data – This is the obvious example, pure GIGO. If your current on-premise data are unclean, say you have inconsistent customer demographic information, or obsolete product information, or any other data quality challenge, it goes without saying that moving such information to the Cloud won’t do your data, or your business, any good.  Instead, think of moving your data to the Cloud as though you were moving your elderly parents to a condo. It’s a wonderful excuse to finally dig through the layers of detritus so that you only move data that are clean, accurate, and valuable to the business.

• Spaghetti code – you may be eyeing that old custom-coded legacy app as a prime Cloud candidate. It’s too slow, it doesn’t scale well, and it’s a bear to integrate now, so won’t the Cloud automatically make it fast, scalable, and easy to integrate? Sorry to burst your bubble. If you’re focusing on an IaaS approach, what you’ll find is that spaghetti code is every bit as intractable in the Cloud as it is on-premise. What about PaaS? Chances are that old code won’t run at all. Today’s PaaS environments expect and enforce a certain level of code quality.

• Obsolete and Cloud-unfriendly business processes  –  Does this sound familiar? The business asks IT to automate a set of processes, but states unequivocally that “the processes are fine the way they are. Automate them but don’t change them. After all, we’ve been doing things the same way for years. Why change now?” Yes, the business often says that, but seasoned IT veterans have long realized that the business never actually means it. When the business asks IT to touch a process, there is always at least an implied requirement to try to make it better: faster, more streamlined, better aligned with the underlying business need.

Moving business process implementations to the Cloud raises the stakes in this complex dance between business and technology, because the Cloud offers a wealth of new opportunities for improving processes. Furthermore, how users interact with Cloud-based assets is often fundamentally different from how users interact with traditional enterprise apps. Any organization that has moved from an older CRM app (or no CRM app at all) to Salesforce.com has learned this lesson first hand. But Salesforce is merely a harbinger of greater change to come. One of the main reasons Salesforce has been so successful is because they offer their clients new ways of conducting business—in other words, better processes. Any SaaS solution should build on their example.

Generating New Garbage in the Cloud

The garbage problem doesn’t end with garbage you might put in the Cloud. The Cloud also presents numerous opportunities to generate new kinds of garbage.

• Zombie instances – it’s so easy and cheap now for anyone in your organization to spawn their own Cloud instances, including virtual machines, storage instances, and more. Furthermore, such instances are elastic: need more of them? The Cloud is only too happy to oblige. But what happens when you’re done with them? You’re supposed to delete them. After all, elasticity works in both directions. All too often, however, instances that have served their purpose are left around like so much space junk. After a while, nobody remembers what they’re for or if they still have something important in them. The last thing you want to do is delete an instance with valuable data or code on it. So to play it safe, you leave it around. Forever. Your Cloud provider is only too happy to keep billing you for these Zombie instances.

• Data with no provenance – any Antiques Roadshow aficionado knows that antiques with provenance are more valuable than those without. The same goes for your data. Do you know if the data you’re working with are the latest version? Do you know they haven’t been tampered with? If not, then those data are worse than useless, since they may be incorrect, or even worse, keeping them around may violate any number of regulations. Here again the elasticity of the Cloud works against you.

• Manual or poorly abstracted configurations – Let’s say you’ve built a sophisticated Cloud app based on elastic VM instances. If you need more, simply provision more. But then let’s say some admin somewhere in your IT shop goes into one of these instances and changes a config file in order to get an app to run on that instance. Now you have no way to update your instances without breaking your app—and if that admin didn’t tell anybody about the reconfiguration, then tracking down the problem will present a time-consuming challenge.

Simply creating a static image file to generate new VM instances—and keeping rogue admins from monkeying with them—won’t solve the problem, because there is more to your app than the instances. Instead, you need a next generation configuration management approach that automates configuration for the Cloud. See Chef or Puppet for an indication where this market is going. (You can expect a ZapFlash on Cloud configuration management in the near future.)

• Cloud-unfriendly architecture choices – We covered one example of this problem in our ZapFlash The Secret to a RESTful Cloud: stateful Cloud instances. Essentially, inappropriate state information is just more garbage in the Cloud. Another example would be inappropriate transactionality in the Cloud. Cloud Computing lends itself to particular ways of architecting applications, and attempting to shoehorn the wrong architectural approach into the Cloud is about as effective as Cinderella’s stepsisters’ efforts with the glass slipper.

The ZapThink Take

How do you avoid garbage in the Cloud? Architecture is a large part of the answer, of course, but governance is equally important. Organizations should establish and enforce Cloud-centric policies as well as extending current IT governance to the Cloud. With great power comes great responsibility, and the Cloud offers enormous new power to many different roles within the IT organization. The Cloud is fraught with pitfalls. Without sufficient governance, you’re bound to fall in one.

It is also important to note that the issues in this ZapFlash apply equally to private as well as public Clouds. Organizations generally realize that public Clouds present numerous governance challenges, and look to private Clouds because they are ostensibly less risky. But such a stance offers little more than a false sense of security—one that may backfire, if organizations assume that in the absence of proper architecture and governance, a private Cloud is the better choice. Don’t wait to implement adequate Cloud governance until after you’ve run into these problems. Governance should be an integral part of any Cloud strategy, before you move to the Cloud.

If you’ve been following ZapThink for the last few years, you know we’re talking less about SOA and more about REST and Cloud. Not that there’s anything wrong with SOA—we’re simply focusing on the current challenges organizations face when building agile architectures. So, it should come as no surprise that we finally write a ZapFlash on RESTful Clouds.

You might think the story we have to tell about RESTful Clouds has to do with RESTful APIs to the Cloud. Sure, we want to be able to access Cloud resources as well as Cloud management capabilities via RESTful interfaces. Makes so much sense that the Apache Foundation has already been working on it in their Deltacloud initiative, to name one of many such efforts.

Deltacloud and its brethren are promising, to be sure, but that’s not what we’re talking about here. RESTful APIs to the Cloud miss the point to REST, and don’t address the core challenge of architecting for the Cloud, two trends ZapThink has been exploring over the last few months. In this ZapFlash, it’s finally time to tie them together into a neat holiday bow:

Trend #1: We explored how the elasticity property of Clouds impacts how we architect applications for the Cloud. We even called this aspect of Cloud architecture a “blind spot” in the ZapFlash Architecting Beyond Cloud Computing’s Horseless Carriage.

Trend #2: We discussed how REST really isn’t about interfaces and APIs, it’s about distributed hypermedia applications in the ZapFlash The Right End of REST.

The story of RESTful Clouds, therefore, isn’t about RESTful APIs to the Cloud at all, useful though they may be. Instead, the fact that the REST architectural style focuses on hypermedia applications addresses one of the knottiest challenges of architecting for the Cloud: how we deal with application state.

The Challenge of Application State in the Cloud

The traditional approach to maintaining application state for any distributed application is to use some kind of stateful object on the server. A stateful object contains data that maintain the context of that application for a particular user across conversations consisting of multiple calls to a specific instance of that object. In essence, stateful objects keep track of what individual users are doing when they use an application.

A familiar example of a stateful object is the traditional shopping cart. Every shopping cart instance belongs to an individual customer. Therefore, if you had ten thousand customers shopping at the same time, you would have ten thousand shopping cart instances, which can cause problems for your application. Not only do all these carts present an obvious scalability challenge, but if you need to update a given customer’s shopping cart, you must first locate the correct shopping cart instance for that customer. No other shopping cart will do.

Furthermore, remember that in the Cloud, you must plan for and expect failure. If you are dependent on a single instance of a shopping cart, and the resources that support that cart crash, then you are faced with another challenge. Hopefully you’ve been saving the cart’s state somewhere (and of course, the state for every other cart for every other customer), so that you can reconstitute the failed cart elsewhere. Perhaps the worst that would happen would be that the customer would have to start their shopping over, but in other situations, the failure of a stateful instance can leave the customer in an inconsistent state. That’s a surefire recipe for losing customers.

Stateless objects, on the other hand, don’t contain any information or context between calls of that object. In other words, each such call stands alone and doesn’t rely upon prior calls as part of an ongoing conversation. You could call one instance of a stateless object, and then make a call on a different instance of the same object, and you wouldn’t able to tell the difference.

If an object is stateless, therefore, the Cloud is free to use any instance of that object to get its work done. You no longer have to worry about contention for a single instance of an object, a situation that could lead to a variety of distributed computing challenges including race conditions, deadlocks, and starvation. Instead, you can simply rely upon the elasticity of the Cloud to add more instances as needed.

It is an important best practice, therefore, that all application logic in the Cloud should be stateless. No object instances, no session Beans, no server cookies. If the load on your application spikes, the Cloud should respond elastically by provisioning adequate resources to meet the need. The more stateless your application is, the better able the Cloud will be to achieve this seamless elasticity.

HATEOAS to the Rescue

Elastic, stateless shopping carts are all well and good, but what about my shopping cart? I just put my holiday shopping in there. Of course it has to be stateful!

Here’s where our discussion of state gets a bit murky. There are actually two different types of state at play here: application state and resource state. In the case of the shopping cart, the application state consists of what happens to be in individual carts as customers work through the purchasing process, and where in the process they are at any point in time. The resource state includes information the application must access or update, including the customer mailing address, credit card information, and the actual purchase transaction. In other words, the application state is dynamic and the resource state is persistent.

When you deal with resource state in the Cloud, therefore, you’re working at the persistence tier, where traditional approaches to scalability like database sharding and traditional approaches to reliability like replication and caching work reasonably well. There are limitations on Cloud persistence, however; don’t expect to achieve two-phase commit levels of reliability, because the Cloud’s inherent partition tolerance and availability prevent it from exhibiting immediate data  consistency. Data within a particular Cloud instance, however, are still internally consistent.

Application state is a different matter. Treating application state as though it were resource state—writing your application state to your database every time a customer does anything with their shopping cart—limits your scalability, elasticity, and reliability. Don’t go there if you can avoid it. Instead, you want hypermedia to be the engine of application state. In other words, your stateless application instance must give the client everything it needs to know in order to work its way through the purchasing process, and the client maintains the application state for the entire process. You don’t need to spawn a stateful shopping cart instance on the server every time a customer hits your application, since after all, the more users you have, the more clients you have. Why not let the client do the work?

A RESTful Shopping Cart

To explain how RESTful shopping carts might work in the Cloud, we must set up two separate RESTful interactions. The first is between the application tier and the persistence tier. The application tier serves as the client in this case, requesting a representation of the customer’s shopping cart instance from the resource on the persistence tier. This application tier client is stateless; the representation has all the necessary information about the customer as well as the process logic for the purchasing processes that you want the customer to follow.

The second interaction is between the customer’s client and the application tier, which now serves as the server for this interaction. The customer follows links in the representations that the resources on the server return, and thus the client executes the purchasing process as per the customer’s requirements. But the code on the application tier is still completely stateless; it is in charge of following the instructions provided by the persistence tier in a declarative manner.

If the application instance on the application tier crashes, the Cloud environment automatically spawns a replacement. When the customer clicks a link, that replacement knows to repopulate the customer’s shopping cart representation based upon the information in the GET, POST or other operation on that link’s URI, and furthermore, knows where the customer was in their purchasing process based on the information in the operation on the URI as well. In other words, the client runs the shopping cart application by enabling the customer to follow links, and those requests tell the Cloud environment everything it needs to know to meet the customer’s needs, without maintaining any application state of its own. That’s HATEOAS in action.

A RESTful Cloud in Action

A great application of RESTful Cloud principles is the Amazon.com shopping cart. Try this experiment: log into your Amazon.com account simultaneously from two different browsers. Add an item to your cart from one browser. Reload the Amazon home page on the other: note that the number of items in your cart went up by one. Why? Because you haven’t actually begun the purchasing process yet. The shopping cart count on the Amazon home page is part of your resource state.

Next, proceed on one browser as though you were purchasing the item. In the middle of the process, change the quantity of the item you’re trying to purchase from one to two. Again, reload the Amazon home page from the other browser: this time the cart count doesn’t change. You have two items in your cart according to one browser and one item according to the other, even though you only have one shopping cart.

What’s going on here? Amazon’s persistence tier handed your purchase process off to a Cloud instance, and your first browser is maintaining application state for that instance. The Cloud instance can therefore be entirely stateless, which enables Amazon to maximize the elasticity of their environment. If Amazon does it that way, then so should you.

The ZapThink Take

Architecting your Cloud-based app so that all Cloud-based code is stateless is essential for implementing rapid elasticity—and furthermore, such Cloud-based code may be in virtual machine instances, in application packages running on PaaS environments, or in SaaS applications. Furthermore, HATEOAS is essential for handling application state in such a way that enables server resources to be stateless. Therefore, REST is essential for rapid elasticity in the Cloud.

I fear, however, that many RESTafarians won’t fully understand this important conclusion, because so many of them focus on RESTful interfaces—the proverbial forest for the trees. They take a developer’s perspective rather than an architect’s. From the architect’s perspective, REST is no more or less than an architectural style for building distributed hypermedia applications. And you’re not doing REST unless you follow HATEOAS.

Today is a wonderful time for anyone interested in Cloud Computing to be working with the US government. On the one hand, the government considers Cloud to be strategically important, and they already have a track record as an early adopter of Cloud Computing on a grand scale. On the other hand, the government is also in the unique position of being able to drive standards for the approach—and in fact, they are even responsible for establishing the most widely adopted definition of Cloud Computing.

The federal agency who has taken this leadership position is the National Institute for Standards and Technology (NIST), an agency of the US Department of Commerce. NIST’s formal definition of Cloud Computing is already well known—“a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction.” Concise as that definition is, it only marks the beginning of the work NIST is doing to formalize and standardize the full breadth of Cloud Computing approaches, both within the government as well as for the world at large.

I learned about the breadth of NIST’s work on Cloud last week, when I had the pleasure of attending the NIST Cloud Computing Forum & Workshop. They are leading a cross-industry effort to “provide thought leadership and guidance around the cloud computing paradigm to catalyze its use within industry and government. NIST aims to shorten the adoption cycle, which will enable near-term cost savings and increased ability to quickly create and deploy enterprise applications. NIST aims to foster Cloud Computing systems and practices that support interoperability, portability, and security requirements that are appropriate and achievable for important usage scenarios.” To this end, they have followed up their formal definition with a Cloud Computing Technology Roadmap, which consists of three volumes: requirements to further US government adoption, information for Cloud adopters, and technical considerations for government Cloud Computing deployment decisions. They have also published a Cloud Computing reference architecture and standards roadmap.

To be sure, NIST has generated a daunting quantity of information here—but ignore these documents at your peril. If you work for a US government agency, then you likely have a mandate to move toward Cloud Computing, and NIST spells out many of the details. But even if you have nothing to do with the government,  it’s important to remember that NIST is also a standards body in its own right, as well as a coordinating agency for other standards bodies. No other group or agency anywhere else in the world has achieved the same leadership position with respect to today’s nascent Cloud standards efforts.

One of the main reasons NIST is able to maintain this position is because they have an inclusive approach. Want to contribute? You’re welcome to. Have an issue with something in one of the documents? Then let them know. After all, one of the reasons they’ve generated so much content is because they have so many contributors, not just from the government, but from people around the world.

Even ZapThink isn’t above joining the fray. We’ve reviewed NIST’s documents in the context of ZapThink’s eye for agile enterprise architecture, and we’ve identified a missing link. Of course, looking at something and trying to identify what’s missing is always difficult, especially when so many contributors have already pored over the material so carefully. The trick is to break out of “horseless carriage” patterns of thinking: instead of considering the Cloud to be little more than an outsourced, virtualized data center, put on your architect’s hat and consider how Cloud Computing’s unique characteristics will change how you do architecture.

NIST’s Cloud Deployment Scenarios

I found this missing link when I reviewed the Cloud deployment scenarios in the NIST Standards Roadmap document. Here is their diagram illustrating the eight generic deployment scenarios that they identified:

They sort the various deployment scenarios into three categories:

Single Cloud

• Scenario 1: Deployment on a single Cloud
• Scenario 2: Manage resources on a single Cloud
• Scenario 3: Interface enterprise systems to a single Cloud
• Scenario 4: Enterprise systems migrated or replaced on a single Cloud

Multiple Clouds (serially, one at a time)

• Scenario 5: Migration between Clouds
• Scenario 6: Interface across multiple Clouds
• Scenario 7: Work with a selected Clouds

Multiple Clouds (simultaneously, more than one at a time)

• Scenario 8: Operate across multiple Clouds

From ZapThink’s perspective, the most interesting of these are scenarios 1, 3, and 4, because they consider the relationships between enterprise systems and the Cloud. ZapThink has written about these relationships before, most recently in The Keys to Enterprise Public Cloud, but also back in mid-2010, when we discussed Cloud Architecture’s Missing Link.

The missing link we pointed out in that ZapFlash was the ability to compose Cloud-based Services with on-premise Services as part of an enterprise SOA effort. It could be argued, however, that composing Cloud-based Services falls under Scenario 3, since Services are a type of interface. But there’s more to this story—and to understand how the NIST folks missed it, it’s important to follow their line of reasoning.

NIST’s Blind Spot

NIST has divided their Cloud standards efforts into three categories: interoperability, portability, and security. Interoperability standards are the most straightforward, especially for anyone who has worked with Web Services, which of course are little more than standards-based interfaces intended to promote interoperability and loose coupling.

Portability standards are more complicated, because NIST considers both application portability and data portability. In the Cloud context, application portability centers on the ability to move virtual machine (VM) instances from one Cloud to another. Data portability, however, is more difficult, because applications process different kinds of data, and those data flow throughout an entire system. For one organization, data portability might mean moving a single database from one Cloud to another, but for a different organization, the requirement might be for the portability of an entire SaaS application, along with all of its distributed data.

NIST’s focus on interoperability and portability (and security, of course, which is an entire conversation in its own right) makes perfect sense in light of their focus on standards, since the standardization of these three capabilities will go a long way in furthering NIST’s core mission. So it’s no wonder that their three Cloud deployment scenarios that involve enterprise systems consist of deploying or migrating to a Cloud (facilitated by portability standards), or interfacing with a Cloud (facilitated by interoperability standards).

It should come as no surprise, therefore, that NIST missed another deployment scenario: building applications that leverage both on-premise and Cloud-based capabilities, where those applications rely upon more than interoperability, portability, and the ubiquitous security.

Building applications that are compositions of Cloud-based and on-premise Services is a simple example, but doesn’t go far enough, because even this scenario falls into the “horseless carriage” trap of considering the Cloud to be nothing more than a virtualized data center. Factor elasticity into the equation, however, and we must consider new approaches to architecting such applications that go beyond considerations of interoperability and portability.

Building the Cloud’s Inherent Elasticity into Hybrid Applications

More than any other characteristic, elasticity distinguishes true Clouds from simple virtualized data centers. If your app requires more resources, the Cloud will provision those resources automatically, and then release them when you’re done with them—until you need them again. Furthermore, those elastic resources may be among any of the different types of Cloud resources (networks, servers, storage, applications and Services,  as per the NIST definition), or any combination thereof.

As a result, when you architect your app, you don’t know how many of each of these resources you will be using at any point in time, since the number can change with no warning. You must take this change into account when architecting your data, your middleware, your execution environments, your application logic, and your presentation tier—in other words, your entire distributed application.

Cloud providers do their best to hide the underlying complexity inherent in delivering elastic infrastructure. But when you’re building a hybrid app—that is, one that includes Cloud-based as well as on-premise capabilities—your architects must have deeper knowledge of the underlying capabilities of the Cloud environment than Cloud providers are typically comfortable revealing. In other words, even once Cloud interoperability and portability standards mature, architects will still require additional information about the underlying capabilities of their Cloud environments that such standards won’t cover.

The ZapThink Take

This ZapFlash may leave you wanting more—namely, how precisely do you architect with the Cloud in mind? Unfortunately, there isn’t enough room for the answer to that question in this ZapFlash, but fear not, we’ll be laying out more details in the weeks and months to come.

Can’t wait? Then come to our Licensed ZapThink Architect SOA & Cloud course in San Diego January 16-19. We’ll dive deep into architecting with the Cloud, as we enjoy warm breezes off the Pacific from our lush venue by Seaforth Marina on Quivira Basin. Or better yet, take advantage of ZapThink’s new Cloud Competency Center by dropping us a line at info@zapthink.com. We’re here to help!

With contributions from Jason Bloomberg.

Nowhere is semantic precision as important as it is in the healthcare industry, where a single misunderstood utterance or code might put a person’s life in jeopardy. In fact, precision has historically been so important in health IT that it has always trumped agility as a priority for integrating distributed systems. We’d better hardwire our integrations, the conventional wisdom goes, because that’s the only way to guarantee a high standard of patient care.

Tragically, this conventional wisdom has turned out to be entirely false. Generations of inflexible technology have disillusioned healthcare providers, who either shun the latest shiny widget from the IT department, or even worse, misuse it, for example, by cramming custom data into “comment” fields because the structured input on the screen doesn’t meet clinical requirements.

The primary reason technology has been unable to solve this problem is basically because at its core, semantic integration isn’t really about technology. It’s a human communication problem. Even today’s fastest supercomputers can’t understand the meaning of words. Computers think in zeroes and ones, while human meaning is context sensitive, variable, and often ambiguous—even in healthcare.

Traditional integration approaches cannot deal with such vagaries, but agile architectural approaches like Service-Oriented Architecture (SOA) can—but only in conjunction with robust standard semantic models and frameworks that codify the human meaning behind the data. These models and frameworks must bring the precision, while SOA must bring the agility.

Electronic Health Records: Semantic Battleground

With the steady increase in Electronic Health Record (EHR) adoption in the healthcare industry, finding a reliable, secure, cost-effective method to share information between EHRs is a significant challenge. An ideal solution should be standards-based and comply with federal mandates such as the Health Insurance Portability and Accountability Act (HIPAA). The central consortium responsible for establishing the semantic models for sharing such information is Health Level Seven International (HL7), a standards organization that provides a comprehensive framework and related standards for the exchange, integration, sharing, and retrieval of electronic health information to support clinical practice and the management, delivery, and evaluation of health services. The HL7 standards are intended to provide interoperability among IT systems, in order to improve the delivery of care, optimize workflow, reduce ambiguity, and enhance knowledge transfer among all stakeholders, including healthcare providers, government agencies, vendors, and patients.

HL7 standards cover numerous healthcare-related scenarios. For example, most laboratory and diagnostic systems now deliver results via HL7 messages to hospitals, health insurance companies, and other authorized parties. Such messages carry one record for each test observation, and each of these records contains one field that identifies the test, for example, serum sodium, and another field that reports its value. Such observation records carry other fields for reporting the units of measure and other information. The field that carries the observation identifier is OBX-3, and the field that carries the observation value is called OBX-5.

This message structure alone, however, doesn’t provide sufficient semantic information. Many labs send their own idiosyncratic codes in OBX-3 to identify the observation: one laboratory would identify serum sodium with the one code and another lab would use a different code. Such an extreme degree of variation would be a huge barrier to the development of clinical repositories and research databases in the absence of a universal code system for tests that every stakeholder would agree to use.

Furthermore, with each laboratory having its own unique code for every test observation, point-to-point integration becomes an expensive challenge. To address this issue, we implemented a terminology Service that normalizes all of these idiosyncratic codes to codes that follow a set of open standards, including the standard known as Logical Observation Identifier Names and Codes (LOINC). LOINC offers a universal code system for reporting laboratory and other clinical observations. With LOINC, it’s possible to identify observations in electronic messages such as HL7 observation messages, so that when stakeholders receive such messages from multiple sources, they can automatically file the results in the right fields of the EHRs and other forms, thus resolving semantic ambiguities.

However, HL7 message semantics and LOINC are still only a part of the answer. Interoperability is also essential. In a clinical setting, reliable, timely, and accurate information sharing has a direct impact on patient safety. One of the big challenges when integrating heterogeneous healthcare systems is that patient demographics and clinical data reside in a variety of legacy systems. The challenge now becomes how to provide semantically correct data to each system that participates in the data exchange in a way that doesn’t limit flexibility.

The Service-Oriented Approach to Health Information Integration

In order to address this agility challenge, the first order of business is to build an abstraction layer by creating detailed interface control documents that document the input/output requirements and mechanisms for each system participating in the integration. Second, wrap all legacy data interfaces with Web Services. The primary function of these Services is to loosely couple the legacy APIs from Service interactions at the Service intermediary. This loose coupling reduces the cost of changes over time, including revisions in the semantic models, the addition of new use cases, and the ability to phase out the underlying legacy systems.

Third, the heart of the integration is to implement a translation layer between the legacy interfaces with the canonical data defined in the Web Service using medical data standards such as LOINC, which serves as an additional abstraction layer, further increasing the ability of the system to respond to change over time.To simplify the integration, we took a message-oriented approach centered on the Enterprise Service Bus (ESB), which served as the intermediary for our project.

As part of this approach, we leveraged the integration profiles defined by the Integrating the Healthcare Enterprise (IHE) Technical Framework. IHE simplifies and standardizes the integration of healthcare systems by defining a technical framework for implementing established messaging standards to achieve specific clinical goals. IHE doesn’t actually define new integration standards, but rather supports the use of existing standards including HL7 as appropriate, defining configuration choices when necessary. Implementing the translation layer using IHE, HL7, LOINC, and other standards provided a level of abstraction that was critical to the success of the project.

As with all ESBs on the market, our ESB supported XML natively. However, HL7 messages are in a compact, non-XML format. Since we were following a message-oriented pattern based on HL7, The ESB also had to support XML Style Sheet Transformations (XSLT) and XPath processing for flexible data transformations. We also needed an HL7 validating parser to validate the message and extract the relevant information.

Furthermore, to support content-based routing, we achieved a performance increase by querying the messages natively using XPath. The ESB must then route messages from each clinical application correctly to the target system. We leveraged HL7 message headers to apply the necessary content-based routing patterns, allowing the routing of messages to any system (either internal or external) based upon either the message content or message headers.

We used both XPath and XQuery to process the headers to achieve both point-to-point and one-to-many routing scenarios within the ESB. The ESB also stored message transactions in message queues using different messaging patterns (asynchronous and synchronous messaging, publish and subscribe, store and forward, for example) and supported message sequencing in order to deliver the HL7 messages to the clinical applications in the order they arrived at the ESB.

The ZapThink Take

The traditional integrator thinks about connecting things, while the SOA-based integrator thinks about changing things. In this example, the goal of each abstraction layer is not only to hide complexity, but also to support change over time. For example, the terminology Service not only translates between standards like LOINC and underlying proprietary coding schemes, it also supports the ability for such standards to evolve. They don’t change very often, but change they will. Change is unavoidable, especially in the context of human understanding of meaning.

The ESB itself provides an additional layer of abstraction via transformations and content-based routing—two types of operations that are both metadata-driven. As a result, as the HL7 semantic models improve, the integration behavior on the ESB can dynamically change via straightforward changes in configuration. The ESB is unable to replace the human activity of maintaining the semantic models, but it is able to dynamically respond to the changes as necessary to provide agility, while the semantic models continue the never-ending battle to be increasingly precise.

Photo by tiarescott.

One of our Licensed ZapThink Architects, Michael Poulin, struggled with our recent ZapFlash, Where is the SOA in REST-Based SOA? In a forum post, Poulin asked:

If we have a UI that works with the middle- and back-end resources, do we care if … REST or Web Services are used behind the UI? If [the] UI all of a sudden becomes the orchestrator on its own (which contradicts the essence of Service with interfaces where one of [them is the] UI), how [may the] UI journey/workflow … be attributed to the communication channels – REST calls – running behind it? What constitute[s] “A REST application” … and how [does it differ] from non-REST application[s], especially from the architecture perspective? Where [is] the Architecture and where [is] the architecture Implementation?

Poulin is not alone in his confusion. The topic of REST is a minefield: it seems that most people don’t understand it, even when they write authoritatively on the subject. And the true authoritative source, Dr. Roy Fielding, speaks in the precise yet obscure prose of an academic, often sowing as much confusion as he does clarity. And yet, the answers lie in Fielding’s explanations, even though studying them feels a bit like sitting at the feet of the Buddha, hoping for enlightenment.

Unraveling REST

Let’s begin with Wikipedia, a source Fielding bemoans as being a poor substitute for truth. Be that as it may, it’s an expedient starting point. Wikipedia defines REST as “a style of software architecture for distributed hypermedia systems such as the World Wide Web.” I always thought this definition was rather silly: after all, how many distributed hypermedia systems can you think of other than the Web? But in fact, this definition leads us to the answers to both of Poulin’s core questions: first, what constitutes a REST application, and second, where is the architecture and where is its implementation?

Of course, Fielding’s precise definition of REST differs from the Wikipedia simplification. According to Fielding, “the Representational State Transfer (REST) style is an abstraction of the architectural elements within a distributed hypermedia system.” And therein lies the key to Poulin’s second question: REST is an abstraction.

Of course, abstractions are nothing new—they are the basic tool of any architect, and a topic ZapThink has spoken about numerous times. But most people don’t quite grok just how abstract REST is. For example, do you think that REST is HTTP-specific? No, it’s not. It deals with abstracted hypertext protocols, of which HTTP is the most familiar. Does REST call for uniform interfaces consisting of GET, POST, PUT, and DELETE operations? No again. Those are examples of uniform interface operations, but REST itself doesn’t specify them, other than an abstracted GET operation. And of course, resources are abstractions of server capabilities or entities, not the capabilities or entities themselves, while representations are abstractions of media type instances, not the instances themselves. Even the terms “client” and “server” are abstractions: a REST client may actually run on a physical server, and a REST server may serve as a client. In fact, there’s nothing “client/server” about REST.

So when Wikipedia uses the phrase “a style of software architecture for distributed hypermedia systems such as the World Wide Web,” it’s pointing out that REST applies constraints to abstracted distributed hypermedia systems, of which the Web is only the most familiar example.

Confused yet? You’re not alone, and that’s the point. Architects look at the level of abstraction behind REST and wonder, “how can this mess be useful?” while developers look at it and think, “OK whatever, let’s build a RESTful API and go from there.” At that point, the architects take off their architect hats and dive into the technical details alongside the coders. The result? A lot of noise and confusion, ostensible RESTful APIs that are not really RESTful at all, and the mistaken belief that you can comply with some of the REST architectural constraints but not all of them, and still be doing some kind of REST.

And that brings us to Poulin’s first question: what is a REST application anyway?

REST Applications: More than Bricks & Mortar

A REST application is a distributed hypermedia application, of course – something you apply a distributed hypermedia system to in order to accomplish some goal. Easy to say, but deceptively difficult to understand, until you realize that REST is all about distributed hypermedia applications. They’re the point of REST. If you think you’re “doing” REST without the goal of building a distributed hypermedia application, you’re missing the point of REST entirely.

Unfortunately, most people who are trying to do REST, simply put, are missing the point of REST entirely!

It’s as though we’re trying to build a brick building, so we hire an architect who comes up with plans for a brick building. But we look at the plans and we don’t really understand them, but we know we want to build with bricks, so we spend all our time figuring out the best way to mix mortar. Once we’ve come up with really good mortar, we convince ourselves we know how to build the building, even though we still don’t understand the plan.

That’s the state of REST today. People look at REST’s four architectural constraints, and say, hmm, I don’t understand the fourth one – Hypermedia as the Engine of Application State (HATEOAS). But I think I can figure out the bits about resources and representations. What they’re failing to realize is the hypermedia bit is the point of what they’re doing, just as actually building the building according to plan is the point of the mortar.

The mortar-first approach to REST has been institutionalized in the form of the Richardson Maturity Model, which suggests that HATEOAS is at the highest level of maturity, and it’s perfectly fine to start at the lower levels and work your way up. In other words, it’s fine to start construction once you have good mortar; you’ll figure out the plan eventually. Clearly, that’s no way to build a house!

The Right End of REST

What is it about HATEOAS that’s so difficult? State engines? Computer Science 101, puhleeze. Hypermedia? Not really that confusing, either. After all, hypertext is just a document with links in it to other documents, and by generalizing hypertext to the term hypermedia, we’re simply allowing for media other than HTML documents and the like. After all, a video or sound file could have a link in it, right? So, where’s the confusion?

The confusion lies in the second letter A – application. As in distributed hypermedia application. The point of REST, remember? The architectural starting point for any implementation that REST constraints are appropriate for is the design of the hypermedia application that meets the given business requirements. If you look at the business requirements, and your expertise as an architect tells you that a distributed hypermedia application isn’t the right thing to build, then don’t use REST.

On the other hand, a distributed hypermedia application might be just the ticket. The next question you should ask is: how dynamic should this application be? In other words, what agility does the business require? Put the analysis that provides the answer to that question in your Agility Model. Then let the Agility Model – not the Richardson Maturity Model – guide your design.

RESTful Agility Levels

To illustrate the agility levels for distributed hypermedia applications that should frame your Agility Model, let’s work through some concrete examples. As you go through these levels, compare them to the Richardson model.

• Level 1: Static hypermedia application, consisting of a set of static Web pages containing nothing but HTML, interconnected by links. Not particularly agile. Is it REST? Yes, but in a very simplistic way. Does it meet your requirements? Probably not. (Level 1 is good enough for our ZapFlashes, however, in case you hadn’t noticed.)
• Level 2: Hypermedia application consisting of static Web pages that contain HTML and client-side JavaScript (no funky Ajax or the like). Pages link to each other, and the links may be dynamic, based upon client-side logic in the JavaScript.
• Level 3: Hypermedia application consisting of dynamic Web pages built on the fly on the Web server, using php or Java Server Pages or whatever server scripting environment floats your boat. Pages link to each other, but the links may be dynamic, based upon server-side logic.
• Level 4: Hypermedia application consisting of a set of dynamic representations that conform to a variety of media types (HTML documents, XML documents, images, video, you name it), where those representations have links to other representations, and furthermore, the links may be dynamic, based upon client-side or server-side logic or both, as appropriate.

Once you have determined the agility requirements for your application, now you may think about how best to implement its components – the hyperlinked representations. Answering that question finally leads you to a consideration of the resources that generate those representations. But you don’t start with the resources, you start with the hypermedia application. You finish with the resources.

The ZapThink Take

The examples in the agility levels above are just that: examples. Particular implementation decisions that may be appropriate for a given situation. The architecture, however, is the overall, best practice-driven approach for beginning with the business challenge and working through the levels of abstraction to come up with a working implementation at the end. As an architectural style, REST is simply a set of constraints on the architecture: one way of doing things that makes it easier to solve certain problems. The architect must decide whether REST or any other style is appropriate for the problem at hand, but if you choose REST, then you must begin with the distributed hypermedia application you wish to build.

Hopefully I’ve answered Poulin’s questions (I’m sure he’ll let us know if I haven’t). But there’s still a part of his post left to address. If you read to the end, you’ll see he wonders what happened to ZapThink. Well, Michael, we’re a part of a larger company now, to be sure, but I’m still here, and I continue to do my best to push the envelope of architectural understanding. That’s why we continue to update our Licensed ZapThink Architect course, which is now at version 9, and includes expanded content on REST-Based SOA, as you might expect. We hope to see you in a future class – and if you’re in the Washington DC area, please come to our ZapForum networking event on October 12^th and meet the “new” ZapThink. RSVP to zapforum@zapthink.com if you’d like to join us!

Over the years, updating our Licensed ZapThink Architect (LZA) course has given us ample time to explore advances in SOA and Cloud Computing. Now that we’re working on version 9 of the course, we’re taking a closer look at REST-based SOA. Of course, ZapThink has discussed REST for several years now, but recently we’ve seen some fascinating REST-based SOA case studies, both at startups as well as within the US Federal Government.

Nevertheless, while many architects believe that as an architectural style, REST is simpler and more straightforward that Web Services-based SOA, our research is turning up continued confusion over the principles of REST and how best to implement them. Everybody seems to get the basics—operate on resources at URIs with the four HTTP-centric operations GET, POST, PUT, and DELETE—but most people seem to miss the subtleties. Combine that confusion with the fact that you can do REST without SOA, the specifics of REST-based SOA are even more elusive, as we must pare down the essentials of both REST and SOA to understand the true nature of the combined approach. How, therefore, should we handle Service abstractions, contracts, and compositions – arguably, the essence of SOA – in a REST-based SOA world?

Where is the Service Abstraction?

At the center of the SOA approach is the notion of a Service abstraction. REST resources are abstractions as well, but resources are abstractions of capabilities or entities on the server, which is not quite the same thing as a Service abstraction. In SOA, the Service abstraction supports Business Services, which represent flexible, business-centric capabilities. A Business Service may abstract multiple Service interfaces, where routing and transformation operations on intermediaries present a loosely coupled façade.

Most RESTafarians, however, don’t think at this level. They are thinking of clients (e.g. browsers) accessing resources at URIs which return representations. A representation is an HTML page, an XML file, a video, etc. The business context is lost in a sea of URI formats and Internet media types.

What RESTafarians often overlook is that the intermediary pattern is actually one of the core architectural constraints of REST. URIs need not point directly to resources; it is perfectly OK for an intermediary to resolve the URI into a physical endpoint. After all, that’s what DNS servers do!

From the SOA perspective, we can rely upon the intermediary to execute routing rules and transformations as necessary to support the business abstraction. Furthermore, we can establish and enforce the policy (as a part of our SOA governance framework) that the only allowed way to access resources is via endpoints on an intermediary. From the REST perspective, think DNS server on steroids: instead of simply resolving URLs to IP addresses, resolve any formal URI structure to physical resource endpoints by following a rich set of transformation and routing metadata.

Where is the Contract?

At the technical level, a Service is a contracted interface or an abstraction of contracted interfaces. Web Services have contracts that comply with WSDL, but there’s no equivalent of WSDL for REST resources. True, resources have uniform interfaces that the four HTTP operations define, but simply knowing you can GET a resource or POST to a resource doesn’t tell you anything about what that resource is supposed to do. Accessing a resource does give you a representation of that resource, however. Representations can comply with standard Internet media types (formerly known as MIME types), but even the media type specification is insufficient to qualify as a contract.

Sun Microsystems tried to promote the Web Application Description Language (WADL) as a RESTafarian alternative to WSDL, but work on WADL has largely petered out now that Sun is part of Oracle. The point to WADL was more to stub out REST resources in Java than to provide an implementation-neutral contract language in any case.

Where, then, is the contract? Let’s look at a simple REST example: the simplest, of course, being the Web itself. Let’s say you are filling in a form on a Web page and then hit submit. Where is the contract?

The form method is POST, and the POST data are the information that you filled into the form. The resource is identified by the form action URL. So far so good. Have you found the contract yet?

In this example, the contract is the Web form itself. The form specifies and constrains the POST data you may input, and specifies the form action, which is a hyperlink to the next resource. You browsed to the page with the Web form by following an earlier link or loading a URL for a resource that returned that Web page as a representation of that resource.

Remember, a REST application is a set of resources that return representations that link to other resources – in other words, hypermedia. One resource returns one or more representations (Web pages, XML files, etc.) that contain links to other resources, and it is those hyperlinks (and their associated metadata) that specify the application behavior.

While a Web page with a form is the simplest and most common example of how to contract POST data, we can generalize that form however we like, depending on what type of client we want to support. For machine-to-machine interactions, for example (that is, when the client is not simply a browser), the first resource may return an XML representation that provides a contracted interface to the client for POSTing to the linked resource. How your resource builds that representation is up to you.

In Web Services-based SOA we store the contract metadata in a centralized registry/repository. In REST-based SOA each resource is responsible for returning contract metadata either for itself or for any resource it hyperlinks to. As a result, we may not able to obtain contracts for resources we’re not (yet) able to access, but on the other hand, we can code our resources to dynamically generate contracts if we wish. In REST-based SOA, therefore, contract changes can be automated, where in Web Services-based SOA, contract change is a complex, manual process that requires rigorous governance.

Where is the Composition?

The third core characteristic of SOA we look for is the ability to compose Services into applications. Such compositions might be orchestrations, when they have a pre-defined flow, or choreographies, when the order of steps in the composition is not determined ahead of time.

A REST application, of course, is an example of a composition of resources. From the SOA perspective, furthermore, a REST application is a workflow – that is, a composition with human steps. We can also consider such compositions to be choreographies, because the order of steps depends upon which links the user clicks. Users may click links in a different order every time they work their way through the application.

The question still remains: how do we create automated orchestrations in the REST world? The answer is simpler than it looks. In REST, everything can be a resource. Therefore, orchestrations can be resources as well. An orchestration resource might return a BPEL representation or a BPMN representation or perhaps a simplified representation of an orchestration that doesn’t have the baggage of either BPEL or BPMN. If anything, establishing a pre-defined orchestration is simpler than a hypermedia composition, because the orchestration logic is static, while with a hypermedia composition, the underlying resource logic may change the composition logic on the fly. Just because we don’t have to fix our application state transitions ahead of time doesn’t mean we’re not allowed to.

The ZapThink Take

Over the more than ten years ZapThink has been writing about SOA, we’ve fought the battle to explain what SOA really was, fighting the deluge of misinformation from profit-seeking vendors and ill-informed industry analysts. Fortunately, this time vendors aren’t trying to coopt REST to sell software the way they did SOA, to be sure, but the fact still remains that there is extensive confusion and misinformation about REST, just as there still is for SOA.

Mix the two together, therefore, and you’re just asking for trouble. But the effort is worth the trouble, for one simple fact: done right, REST-based SOA actually works. It paves a path to the agile architecture that we’ve been seeking since we first dipped our toe into the ocean of distributed computing. Of course, there’s a catch: the “done right” bit. The devil is in the details.

ZapThink will be providing far more detail on this topic over the next year, in our ZapFlash newsletters, in the next version of our LZA course, and in our Podcasts and at our events. Our new REST-based SOA module in the LZA course in particular breaks new ground and lays the groundwork for simpler, more successful approaches to SOA. Hope to see you at one of our events or classes soon!