Monthly Archives: February 2009

REST in a Nutshell Part 1

REST is probably the most powerful architectural styles to appear on the scene in the last few years.

OK, its not new. Roy wrote his thesis in 2000, but REST is the architectural style that underpins the web so its been around for a while. The thing about Dr. Fielding’s thesis is that its not written for the normal person. As a result, a number of naive implementations have become very popular, the most notable being Rails. Rails’ implementation of REST misses a key feature of REST: hypertext

In this article, we will look at some initial considerations and cover some basic terminology.

The first thing you need to ask yourself is, “Do I need a REST API?” Roy Fielding gives this comment on his blog:

REST is software design on the scale of decades: every detail is intended to promote software longevity and independent evolution. Many of the constraints are directly opposed to short-term efficiency.

So here we have three things to consider before we continue building our REST API.

First, can we reasonably expect what we’re building to last for decades? A REST API for tracking the latest goings on of Jennifer Lopez, you might not have a good candidate for a REST API.
Second, would serendipitous use of your application be valuable? If you know how your application will be used through out its life time, then you don’t need REST.
Third, can you forgo short-term efficiency? This is important to remember. REST architectures are frequently not simple to build or maintain so you have to be ready for that.

Unless you really are building something to track the latest antics your favourite celebrity, you really should be answering in the affirmative to each of those. Whether you are talking about some internal enterprise system or a public web application, I am convinced that the promiscuous publication of APIs is the best way to realize plenty of value in the future. Now you don’t have to use REST for this to happens, but I think that REST is the best way to to do things.

So now what?

Well, we’re not ready to start designing anything. We first need to know some basic terminology.

Resource — Its all the things in our application that we care about and its interactions. Your resources may map to your database entities but you probably won’t have a 1:1 relationship. There will be database entities for which you expose multiple resources. You will have resources for which there are no database entities, but perhaps interact with multiple database tables. It may also be completely transient, like search.

Representation — Resources are never transmitted over the network; representations of your resources are transmitted. You may, and frequently will, have multiple representations of a resource. An example: let’s say you have a resource for the sales for the month. You may have an XML or JSON representation that is the raw data; you might also have an HTML table or a PNG chart. Representations are identified by their media types.

URL — Every resource is identified by one URL. Actually you can use any URI to identify a resource, but unless you can retrieve the resource it is not much value. (I suppose that truly static resources could be identified by a URN provided all the relevant representations were known ahead of time but its not a very likely situation). All representations with the same URI represent the same resource and only representations with the same URI represent the same resource.

This poses a problem. In general, it is hard to have different media types served from the same URI. If a request comes in for /Reports/May07 how do you know if the end user needs the JSON or the PNG representation? More importantly, if you want both the JSON and the PNG representations how do you distinguish between them.

There are two ways to resolve this. The naive way is to modify the URI, this is the approach that most frameworks (like Rails) take, primarily because its easy and its the most reliable method if you’re user agent is a web browser. Typically this means that you have URIs like /Reports/May07.xml and /Reports/May07.json and /Reports/May07.png. This causes a further problem of how do you know that those three URIs all represent the same resource? This can be made to work but it causes problems elsewhere in your design. The correct solution is to use content negotiation. The HTTP RFC defines the Accepts header which allows the user agent to specify which media types it will accept and what its preferences are. We will get into content negotiation in depth in a later article.

One of the most important thing to know about URIs is that they must be meaningless in and of themselves. Any information you transmit in the URI must be intended only for human consumption, your API must work equally well if all the URIs were GUIDs like http://api.example.com/3B405B56-07B9-11DE-A4BB-6CCD55D89593.

Media Type — The relationship between media type, resource and representation is this: you construct a representation of a resource using a media type. Media types are identified by their mime type. Thus the HTML media type is identified by the mime type “text/html” and the JPEG image media type is identified by the mime type “image/jpeg”. If you are constructing representations that are meant to be used directly by end users, you will want to use one of the generic media types like HTML or JPEG or Rich Text. If, however, you are creating representations that are to be used by other systems, you should create new mime types. For example, if you have an XML representation of your resource. You might be tempted to simply use the “application/xml” or “text/xml” mime types to identify it and rely on schema parsing or namespaces for the client to know how to handle the mark-up. However, if you have some compelling reason to have multiple XML representations, you suddenly have a problem. Now HTTP cannot do its thing; it is not aware of schema types or name spaces so it has no way of knowing what is being sending. The correct approach is to create a new mime type. So instead of using “application/xml” you use “application/my-cool-mimetype+xml”. This way you get the benefits of XML parsing and your client code knows what to do with it.

Closing Remarks

I have a few opinions that will colour these articles that should be made clear up front.

  1. REST APIs are not for end users — You really should only use REST for interactions between systems. It is frequently the case that other styles are better suited for end users. User interfaces should be driven by user centred design and information architecture. At some point you start to compromise the REST API. For example, if /Widget/001 is the URI for some widget resource, making a GET request will give you a static representation of the widget. Now lets say you want to edit it, your browser needs a form to fill out. What URL do you request now? You might request something like /Widget/001/edit. But now it breaks down. Technically you are requesting a different representation of the same resource but you’ve identified it as a new resource. The site will work but you’ve compromised the architectural style. For this reason, I say always keep your user facing interface separate from your system facing interface. You life will be much easier.
    • Browsers are not REST clients — To be a REST client (using HTTP), you must be capable of performing GET, POST, PUT and DELETE requests. All web browsers are only capable of doing GET and POST requests. This is because, at this point in time, HTML does not allow you to do anything else. If you can’t do PUT and DELETE, its not a REST client.
    • XHR is a REST client — The XML HTTP Request object is a REST client because it is capable of making PUT and DELETE requests. Thus you can create a user interface in a modern browser that has JavaScript enabled which uses a REST API to communicate with back end systems
  2. Developing a REST interface for your application is about creating and documenting media types — You are probably going to write an application that is used on the world wide web, which means that you’ll be using HTTP. This means that a large part of the hard work is already done for you. What you need to do (once you’ve memorized HTTP) is model your resources and create media types to build representations from. Those who consume your API will need to know what your media types are so that when they receive a representation using that media type, that it knows what to do with it.
    • There is an important point buried in that thought: The ONLY things you should be communicating outside your API are (1) the starting URI from which application being interacting with your site (this should simply be http://api.example.com) and (2) the meaning/usage of the media types you supply.

Next time we’ll look at modelling resources.

Where are the REST frameworks?

I like REST. I’ll admit that I didn’t understand all of Dr. Roy Fielding’s Dissertation (he is, after all writing to experts in the field of API design) but I think that I’ve managed to understand enough of it that Rails’ implementation of REST leaves a lot to be desired. Rails is not alone, many services are releasing APIs that they are calling REST. Dr. Fielding recently addressed this in a post on his site. Like his thesis, he is talking to API designers, not to Joe and Jane Developer who needs to get the latest web site off the ground. Perhaps more practically, Subbu Allamaraju gave the issue of REST and hyperlinking a thorough exfoliation over at InfoQ.

If you read Subbu’s article and compare it to the standard Rails 2.x web application you will quickly see that difference. In a rails application, if you GET a JSON representation of our quintessential blog all you are going to be told is that the owner_id is 1232; your code has to know that Owners are Users and that they can be retrieved from http://api.example.com/users/{user_id}. This is like putting an <img img_id=123> in your html and expect the browser to know to retrieve the file from http://www.example/resources/images/123.

There is the comparison that most people who think they are writing REST APIs are forgetting; HTTP. If you are writing a REST API then you are writing something VERY much like HTTP. None of us are likely to need to create a REST API. I work for a financial institution and we might have a need to create a REST API like HTTP that is geared towards exchanging financial transactions between internal systems, our partners and our clients, but I suspect that would be unnecessary.

No most of us have no need for creating a new REST API. What we should be talking about is publishing RESTful media types. If I’m going to build the archetypical Blog application, then I need media types for a Blog, a Post, a Comment, a Tag, an Author and what ever else.

A Note: I’m making a distinction that I don’t think Dr. Fielding would make. Dr. Fielding makes it clear that the linking mechanism is central to a REST API, and since HTTP leaves much of the linking mechanism (but certainly not all) up to the user agent, a true REST API requires a media types. However, the vast majority of the work that we do is over HTTP so we have no need, in general, to worry about that half of a REST API. I think it greatly clarifies things if we simply talk about creating REST Media Types rather than APIs. I just noticed that I’m not the first one to realize this (after Dr. Fielding of course).

What about the Frameworks

I’ve been trying to find a Ruby framework that can handle REST Media Types right out of the gate. Waves, which its resource oriented architecture seems like a good place to start. They are right at the start of putting together what they call foundations for REST services, I have high hopes for what we’ll see from that work.

REST ≠ MVC

One of the biggest problems is that people are trying to bolt REST onto MVC frameworks. The problem is it is probably the wrong programming model to use. The controller for a REST service is absolutely trivial; you can only call a maximum of 4 methods on any given resource, and they are always handled the same, so the controller is completely anemic and should fall into the framework itself. Views are also the wrong paradigm since they only deal with out going representations; you really need some mechanism for specifying the transformation of a resources into a representation (that is its media type), and back. All that you really have left are the resources (assuming they’re like models) and the transformations.

REST and Your Web Application

I’m probably going to annoy a lot of people with this. A web browser (as they are currently implemented) is NOT a RESTful client. You can use XHR object supplied by a browser’s JavaScript engine as a REST client, but the browser itself interacting with plain old HTML cannot act RESTfully. In part, its because your browser won’t make PUT or DELETE requests (you have to fake it with extra parameters on a POST). Mostly its a matter of usability.

For a non-trivial use case, you are going to want to something wizard like. That means you are going to have to create a number of intermediate, transient representations to maintain that state. Its an affectation in this case. The transient representations serve no purpose but to maintain this fiction that its REST.

I honestly think that you should really simply leave the your REST API as an API and build a separate application to act as your User Interface. You’ll make it easier on yourself in the end

REST and MVC

Like I said earlier, REST is not MVC. But that doesn’t mean that its not related to MVC. MVC is very well suited to building a User Interface. As I said earlier, you should build a separate application for your User Interface. Well, there is no reason why you can’t use a Resource as the Model in your MVC application. Rails already does this; ActiveResource allows you to use a REST Resource instead of a database table for your model.

This is where things get really interesting.

If you are already using resource representations for the models in your MVC user interface, why limit yourself to a single domain? There is no reason why you couldn’t use multiple domains. This allows for what I call DMVC: Distributed Model View Controller. By distributing your models over a number of applications (via REST media types) you can get get some great long term efficiencies by the serendipitous, and unplanned, reuse of REST APIs.

For example, if you have a non-trivial content management system, you are going to want to move your content changes from the author to an editor to perhaps a legal reviewer and maybe finally to a marketing manager before those changes are “live”. At the same time, you may have defect tracking system where defects move from reported, through any number of stages to completed. Traditionally, you would include something like acts_as_statemachine in each of your rails apps to get this functionality.

But if you apply the REST architectural style, you can create 3 independent applications: Content Management, Defect Management, Workflow. The Workflow applciation would have something like a State resource which would be associated with some other resource without knowing anything about that resource. If the business logic was sufficiently complex (for instance, a change in the State resource would change the associated resource) you could create another application that combined a State resource with a (let’s say) Defect resource to expose StatefulDefect resources.

Also, because you have decoupled your the workflow from both your CMS and your defect tracking, you can reuse both to create a project management application where the content is not work flowed but managed wiki style and defects are not workflowed rather raised and closed since you have hourly builds to the project server (or what ever).

We already know how to optimize and scale web servers, and the same can be done here. Or, if this is a purely internal set of services (or even largely) you could create a more efficient connection mechanisms/schemes than HTTP (I’m not sure what those would be). Or you might try more efficient media types (for example using JSON or even a binary format).

REST, Associations & ORM

One of the more valuable parts of Rails is the ActiveRecord ORM. It allows you to say that a Blog belongs_to a User and a User has_many blogs. Associations between these objects are by ID as they refer to other records in the database.

But REST blows that apart. Your Blog and your User are no longer stored in the same database. We refer to every unique resource by a URL so it may live somewhere totally unaccessible to us. If I have blogs and you have users, I can say that my blog belongs to a user, but I cannot expect your users will know anything about my blogs. Now perhaps the representation of the user that you give me includes a list of blogs that the user owns, in which case I can PUT back a representation that includes the URI of my Blog, but my code cannot rely on that information being known.

You also get some interesting behaviours.

All you have is a URI. You have NO idea what that resource is. As a result, its conceivable that I might try to create a blog that is “owned” by an image. Or the URI point to a resource that is a legitimate owner of the blog but its media types are not those that the blog application understands. For the most part, that’s OK because the blog application is really just an API, its not our User Interface. The User Interface application will need to have coping mechanisms for this but I’m not sure that’s a problem. If all you’re producing is an API, then its not even your problem.

More importantly, you get all sorts of strangely compelling polymorphism. Its liberating not to know what you’re associated with since it guarantees that you’ll become associated with all sorts of unexpected things.