Rails 2.3 Template for Dreamhost

I’ve started playing with Rails 2.3 after a bit of a break (Ruby hasn’t been my main programming language until very recently) and I have to say that Templates ROCK!

Working from Casper Fabricus’ script to setup a new git repository on Dreamhost and John Nunemaker’s tutorial on getting your rails app running on dreamhost with passenger and combined them into one template:

#Get the applicaiton name from the current working directory
app_name = File.basename( FileUtils.pwd )

#Ask some leading questions
ssh_domain = ask( "What is the domain of you git repository (ie. example.com)?")
ssh_user = ask( "What is the shell user name to access your git repository?" )
ssh_location = "#{ssh_user}@#{ssh_domain}"
#There is probably a better way to write these two lines
project_domain = app_name
project_domain = ask( "What is the domain to which your application will be deployed?" ) if no?( "Use #{app_name} as the deployment domain for your application?") 

#Setup Capistrano
capify!

#Need an odd mixture of variable substitution.
deploy_file = "default_run_options[:pty] = true
# be sure to change these
set :user, '#{ssh_user}'
set :ssh_domain, '#{ssh_domain}'
set :application, '#{app_name}'
set :domain, '#{project_domain}'
"
deploy_file <<
%q{ # the rest should be good
set :repository,  "#{user}@#{ssh_domain}:git/#{application}.git" 
set :deploy_to, "/home/#{user}/#{domain}" 
set :deploy_via, :remote_cache
set :scm, 'git'
set :branch, 'master'
set :git_shallow_clone, 1
set :scm_verbose, true
set :use_sudo, false

server domain, :app, :web
role :db, domain, :primary => true

namespace :deploy do
  task :restart do
    run "touch #{current_path}/tmp/restart.txt" 
  end
end
}
file 'config/deploy.rb', deploy_file

# rails:rm_tmp_dirs
["./tmp/pids", "./tmp/sessions", "./tmp/sockets", "./tmp/cache"].each do |f|
  run("rmdir ./#{f}")
end

run "ssh #{ssh_location} 'mkdir -p ~/git/'#{app_name}'.git && cd ~/git/'#{app_name}'.git && git --bare init'"
git :init
git :remote => "add origin ssh://#{ssh_location}/~/git/#{app_name}.git"

initializer '.gitignore', <<-CODE
log/\\*.log
log/\\*.pid
db/\\*.db
db/\\*.sqlite3
db/schema.rb
tmp/\\*\\*/\\*
.DS_Store
doc/api
doc/app
config/database.yml
CODE

git :add => "."
git :commit => "-a -m 'Created #{app_name}'" 
git :push => "origin master"

initializer '.git/config', <<-CODE
[branch "master"]
  remote = origin
  merge = refs/heads/master
CODE

Why REST Frameworks?

Many people are simply slapping an interface on to every web application they build and don’t give it much thought. Tim is right in asking why try to coerce your website into a REST paradigm since the browser is a bad (or rather not a general) REST client. We shouldn’t.

However, this doesn’t dispose of the need for a truly REST framework (that is one which uses real hypertext for is representations).

Lets say I have a compelling web application for which I publish a separate API. Assuming that I have a business model associated with that application, then someone is going to take my API and use it in some way I hadn’t anticipated. Now I’ve added value to my application without doing any work. That ability has a multiplicative effect as there are no limits to how many people can use my API. That applies to any API but REST is more likely to succeed since it is intended to promote long term (on the scale of decades) serendipitous evolution.

So there is value in publishing APIs for my applications (independent of the user interface), the potential for added value in the future.

But I think that there is a far greater benefit for our own development processes. Two principals can be maximized using REST APIs: Loose Coupling and High Cohesion.

What I am suggesting is that you should architect your systems so that you produce one (internal) application for each concern in your system. That is to say, if I was building an on-line storefront, I’d create three REST applications: one application to handle user management, one to handle product Management and one to handle order fulfillment. These all would be completely internal to my business.

I would then create a publicly accessible application which would consume the other three applications. This would be a User Interface so it would be designed around User Centered Design principals.

So what are the advantages to the approach? Here are but a few.

Scalability — Let’s say that the sale of widgets has gone through the roof and I’m seeing 10,000 people every hour come to my site. I’m going to have to look at load balancing. I now need to distribute my Web UI onto 4 machines, two of which I’ve put in Ottawa, and two are here in Vancouver. What else do I need to change? Well it turns out that once you’ve logged in (session mgmnt is handled by the Web UI), all you need is the User’s profile information. Since that is seldom changed in a session, the Web UI will be able to use its cached version; so long as the user management can respond with 304 NOT MODIFIED efficiently, one server can handle its load handily. Far more people browse than order, lets say 1 in 10 people place an order. That means that we’re expecting about 1000 orders per hour which our order application can handle nicely so we won’t scale it for now. But we are going to have to do something about the product information, that needs to scale with the Web UI… or does it? From the point of view of the Web UI, the product info is static and most of it changes infrequently. I don’t need to put up multiple instances of my product management app, I simply need a good web cache at each of my data centers to sit between the web UI and the product management system.
Maintainability — With four independent systems making up or storefront, I can now push changes to any one without affecting the others. This has some great operational benefits as any change to a system affects the whole system so when you’re only changing small systems that reduces the risk. Its also more testable. It becomes very easy to write mock applications to consume the resources of any of your internal systems or to produce mock APIs to test your web interface (this makes it very easy to write resilient selenium tests since the data won’t change).
Low(er) Risk Evolution — Next week the VP of Business Development is going to announce that you have a strategic partnership with Acme DooDads. They’re going to sell your widgets. Now we need to expose our product and order systems to them. We also want to make sure that only they have access to these services so we’re not going to put them outside the firewall. Instead we’re going to write a new public application that will interact with our product and order systems. Since ACME isn’t as modern as we are, we have to build a SOAP interface for them. Since they’re sending all the shipping particulars in each order request, we’ll create a new “AcmeShippingDestination” application to hold the representations that would normally be in our User management application. Here’s the value: We’ve added a significant number of features without editing any of the existing code. It also applies internally; I may need to create an application that consumes one or more other REST applications to expose other REST resources to internal applications adding some value along the way.
Fully Aware Deputies — Maybe its because I spent 7 years developing online banking applications, but I always expect the worst. If you look at your average open source CMS (like Word Press or Radiant) you’ll notice that one application handles both the distribution and creation of your content. This is an instance of “The Confused Deputy Problem”http://en.wikipedia.org/wiki/Confused_deputy_problem. The reason is that all the database access is through one user who generally has universal read/write privileges and any attacker who can exploit flaws in the system (and they always exist) has full reign over the database. A better architectural approach is a publicly accessible application that has only the permissions it needs and a private application that has full access. With REST, I would argue that our “Read Only” product product resources that are used by the Web UI are different than the “Read/Write” product resources that our business users are going to interact with. That is if we have separate applications for both the read only and read/write resources, I avoid the confused deputy and I can use normal network access controls to keep everything safe and secure. This is especially true in companies where information is classified in security terms as “Public”, “Private”, “Confidential”, etc. There you usually have strict controls on who can see the different types of information. Further any single entity may have information that can be disclosed at each security level; it would be most appropriate to have different resources for each security level served from each from their own application (again so that network level access controls can be used).

Some of these things are not unique to REST, to be sure. It also puts a little bit of strain on certain parts of our application; because we don’t know what is at the end of a URI, or if I even have permission to access it, we have to be a lot more flexible in what we expect. Actually flexibility is one of the hardest things here. Think of a web browser (which is a model for how we develop applications that consume REST resources); all it knows is the starting URI you provide, HTTP and how to render a handful of media types. This is all that your REST client application may know as well.

REST in a Nutshell Part 1

REST is probably the most powerful architectural styles to appear on the scene in the last few years.

OK, its not new. Roy wrote his thesis in 2000, but REST is the architectural style that underpins the web so its been around for a while. The thing about Dr. Fielding’s thesis is that its not written for the normal person. As a result, a number of naive implementations have become very popular, the most notable being Rails. Rails’ implementation of REST misses a key feature of REST: hypertext

In this article, we will look at some initial considerations and cover some basic terminology.

The first thing you need to ask yourself is, “Do I need a REST API?” Roy Fielding gives this comment on his blog:

REST is software design on the scale of decades: every detail is intended to promote software longevity and independent evolution. Many of the constraints are directly opposed to short-term efficiency.

So here we have three things to consider before we continue building our REST API.

First, can we reasonably expect what we’re building to last for decades? A REST API for tracking the latest goings on of Jennifer Lopez, you might not have a good candidate for a REST API.
Second, would serendipitous use of your application be valuable? If you know how your application will be used through out its life time, then you don’t need REST.
Third, can you forgo short-term efficiency? This is important to remember. REST architectures are frequently not simple to build or maintain so you have to be ready for that.

Unless you really are building something to track the latest antics your favourite celebrity, you really should be answering in the affirmative to each of those. Whether you are talking about some internal enterprise system or a public web application, I am convinced that the promiscuous publication of APIs is the best way to realize plenty of value in the future. Now you don’t have to use REST for this to happens, but I think that REST is the best way to to do things.

So now what?

Well, we’re not ready to start designing anything. We first need to know some basic terminology.

Resource — Its all the things in our application that we care about and its interactions. Your resources may map to your database entities but you probably won’t have a 1:1 relationship. There will be database entities for which you expose multiple resources. You will have resources for which there are no database entities, but perhaps interact with multiple database tables. It may also be completely transient, like search.

Representation — Resources are never transmitted over the network; representations of your resources are transmitted. You may, and frequently will, have multiple representations of a resource. An example: let’s say you have a resource for the sales for the month. You may have an XML or JSON representation that is the raw data; you might also have an HTML table or a PNG chart. Representations are identified by their media types.

URL — Every resource is identified by one URL. Actually you can use any URI to identify a resource, but unless you can retrieve the resource it is not much value. (I suppose that truly static resources could be identified by a URN provided all the relevant representations were known ahead of time but its not a very likely situation). All representations with the same URI represent the same resource and only representations with the same URI represent the same resource.

This poses a problem. In general, it is hard to have different media types served from the same URI. If a request comes in for /Reports/May07 how do you know if the end user needs the JSON or the PNG representation? More importantly, if you want both the JSON and the PNG representations how do you distinguish between them.

There are two ways to resolve this. The naive way is to modify the URI, this is the approach that most frameworks (like Rails) take, primarily because its easy and its the most reliable method if you’re user agent is a web browser. Typically this means that you have URIs like /Reports/May07.xml and /Reports/May07.json and /Reports/May07.png. This causes a further problem of how do you know that those three URIs all represent the same resource? This can be made to work but it causes problems elsewhere in your design. The correct solution is to use content negotiation. The HTTP RFC defines the Accepts header which allows the user agent to specify which media types it will accept and what its preferences are. We will get into content negotiation in depth in a later article.

One of the most important thing to know about URIs is that they must be meaningless in and of themselves. Any information you transmit in the URI must be intended only for human consumption, your API must work equally well if all the URIs were GUIDs like http://api.example.com/3B405B56-07B9-11DE-A4BB-6CCD55D89593.

Media Type — The relationship between media type, resource and representation is this: you construct a representation of a resource using a media type. Media types are identified by their mime type. Thus the HTML media type is identified by the mime type “text/html” and the JPEG image media type is identified by the mime type “image/jpeg”. If you are constructing representations that are meant to be used directly by end users, you will want to use one of the generic media types like HTML or JPEG or Rich Text. If, however, you are creating representations that are to be used by other systems, you should create new mime types. For example, if you have an XML representation of your resource. You might be tempted to simply use the “application/xml” or “text/xml” mime types to identify it and rely on schema parsing or namespaces for the client to know how to handle the mark-up. However, if you have some compelling reason to have multiple XML representations, you suddenly have a problem. Now HTTP cannot do its thing; it is not aware of schema types or name spaces so it has no way of knowing what is being sending. The correct approach is to create a new mime type. So instead of using “application/xml” you use “application/my-cool-mimetype+xml”. This way you get the benefits of XML parsing and your client code knows what to do with it.

Closing Remarks

I have a few opinions that will colour these articles that should be made clear up front.

REST APIs are not for end users — You really should only use REST for interactions between systems. It is frequently the case that other styles are better suited for end users. User interfaces should be driven by user centred design and information architecture. At some point you start to compromise the REST API. For example, if /Widget/001 is the URI for some widget resource, making a GET request will give you a static representation of the widget. Now lets say you want to edit it, your browser needs a form to fill out. What URL do you request now? You might request something like /Widget/001/edit. But now it breaks down. Technically you are requesting a different representation of the same resource but you’ve identified it as a new resource. The site will work but you’ve compromised the architectural style. For this reason, I say always keep your user facing interface separate from your system facing interface. You life will be much easier.
- Browsers are not REST clients — To be a REST client (using HTTP), you must be capable of performing GET, POST, PUT and DELETE requests. All web browsers are only capable of doing GET and POST requests. This is because, at this point in time, HTML does not allow you to do anything else. If you can’t do PUT and DELETE, its not a REST client.
- XHR is a REST client — The XML HTTP Request object is a REST client because it is capable of making PUT and DELETE requests. Thus you can create a user interface in a modern browser that has JavaScript enabled which uses a REST API to communicate with back end systems
Developing a REST interface for your application is about creating and documenting media types — You are probably going to write an application that is used on the world wide web, which means that you’ll be using HTTP. This means that a large part of the hard work is already done for you. What you need to do (once you’ve memorized HTTP) is model your resources and create media types to build representations from. Those who consume your API will need to know what your media types are so that when they receive a representation using that media type, that it knows what to do with it.
- There is an important point buried in that thought: The ONLY things you should be communicating outside your API are (1) the starting URI from which application being interacting with your site (this should simply be http://api.example.com) and (2) the meaning/usage of the media types you supply.

Next time we’ll look at modelling resources.

Where are the REST frameworks?

I like REST. I’ll admit that I didn’t understand all of Dr. Roy Fielding’s Dissertation (he is, after all writing to experts in the field of API design) but I think that I’ve managed to understand enough of it that Rails’ implementation of REST leaves a lot to be desired. Rails is not alone, many services are releasing APIs that they are calling REST. Dr. Fielding recently addressed this in a post on his site. Like his thesis, he is talking to API designers, not to Joe and Jane Developer who needs to get the latest web site off the ground. Perhaps more practically, Subbu Allamaraju gave the issue of REST and hyperlinking a thorough exfoliation over at InfoQ.

If you read Subbu’s article and compare it to the standard Rails 2.x web application you will quickly see that difference. In a rails application, if you GET a JSON representation of our quintessential blog all you are going to be told is that the owner_id is 1232; your code has to know that Owners are Users and that they can be retrieved from http://api.example.com/users/{user_id}. This is like putting an <img img_id=123> in your html and expect the browser to know to retrieve the file from http://www.example/resources/images/123.

There is the comparison that most people who think they are writing REST APIs are forgetting; HTTP. If you are writing a REST API then you are writing something VERY much like HTTP. None of us are likely to need to create a REST API. I work for a financial institution and we might have a need to create a REST API like HTTP that is geared towards exchanging financial transactions between internal systems, our partners and our clients, but I suspect that would be unnecessary.

No most of us have no need for creating a new REST API. What we should be talking about is publishing RESTful media types. If I’m going to build the archetypical Blog application, then I need media types for a Blog, a Post, a Comment, a Tag, an Author and what ever else.

A Note: I’m making a distinction that I don’t think Dr. Fielding would make. Dr. Fielding makes it clear that the linking mechanism is central to a REST API, and since HTTP leaves much of the linking mechanism (but certainly not all) up to the user agent, a true REST API requires a media types. However, the vast majority of the work that we do is over HTTP so we have no need, in general, to worry about that half of a REST API. I think it greatly clarifies things if we simply talk about creating REST Media Types rather than APIs. I just noticed that I’m not the first one to realize this (after Dr. Fielding of course).

What about the Frameworks

I’ve been trying to find a Ruby framework that can handle REST Media Types right out of the gate. Waves, which its resource oriented architecture seems like a good place to start. They are right at the start of putting together what they call foundations for REST services, I have high hopes for what we’ll see from that work.

REST ≠ MVC

One of the biggest problems is that people are trying to bolt REST onto MVC frameworks. The problem is it is probably the wrong programming model to use. The controller for a REST service is absolutely trivial; you can only call a maximum of 4 methods on any given resource, and they are always handled the same, so the controller is completely anemic and should fall into the framework itself. Views are also the wrong paradigm since they only deal with out going representations; you really need some mechanism for specifying the transformation of a resources into a representation (that is its media type), and back. All that you really have left are the resources (assuming they’re like models) and the transformations.

REST and Your Web Application

I’m probably going to annoy a lot of people with this. A web browser (as they are currently implemented) is NOT a RESTful client. You can use XHR object supplied by a browser’s JavaScript engine as a REST client, but the browser itself interacting with plain old HTML cannot act RESTfully. In part, its because your browser won’t make PUT or DELETE requests (you have to fake it with extra parameters on a POST). Mostly its a matter of usability.

For a non-trivial use case, you are going to want to something wizard like. That means you are going to have to create a number of intermediate, transient representations to maintain that state. Its an affectation in this case. The transient representations serve no purpose but to maintain this fiction that its REST.

I honestly think that you should really simply leave the your REST API as an API and build a separate application to act as your User Interface. You’ll make it easier on yourself in the end

REST and MVC

Like I said earlier, REST is not MVC. But that doesn’t mean that its not related to MVC. MVC is very well suited to building a User Interface. As I said earlier, you should build a separate application for your User Interface. Well, there is no reason why you can’t use a Resource as the Model in your MVC application. Rails already does this; ActiveResource allows you to use a REST Resource instead of a database table for your model.

This is where things get really interesting.

If you are already using resource representations for the models in your MVC user interface, why limit yourself to a single domain? There is no reason why you couldn’t use multiple domains. This allows for what I call DMVC: Distributed Model View Controller. By distributing your models over a number of applications (via REST media types) you can get get some great long term efficiencies by the serendipitous, and unplanned, reuse of REST APIs.

For example, if you have a non-trivial content management system, you are going to want to move your content changes from the author to an editor to perhaps a legal reviewer and maybe finally to a marketing manager before those changes are “live”. At the same time, you may have defect tracking system where defects move from reported, through any number of stages to completed. Traditionally, you would include something like acts_as_statemachine in each of your rails apps to get this functionality.

But if you apply the REST architectural style, you can create 3 independent applications: Content Management, Defect Management, Workflow. The Workflow applciation would have something like a State resource which would be associated with some other resource without knowing anything about that resource. If the business logic was sufficiently complex (for instance, a change in the State resource would change the associated resource) you could create another application that combined a State resource with a (let’s say) Defect resource to expose StatefulDefect resources.

Also, because you have decoupled your the workflow from both your CMS and your defect tracking, you can reuse both to create a project management application where the content is not work flowed but managed wiki style and defects are not workflowed rather raised and closed since you have hourly builds to the project server (or what ever).

We already know how to optimize and scale web servers, and the same can be done here. Or, if this is a purely internal set of services (or even largely) you could create a more efficient connection mechanisms/schemes than HTTP (I’m not sure what those would be). Or you might try more efficient media types (for example using JSON or even a binary format).

REST, Associations & ORM

One of the more valuable parts of Rails is the ActiveRecord ORM. It allows you to say that a Blog belongs_to a User and a User has_many blogs. Associations between these objects are by ID as they refer to other records in the database.

But REST blows that apart. Your Blog and your User are no longer stored in the same database. We refer to every unique resource by a URL so it may live somewhere totally unaccessible to us. If I have blogs and you have users, I can say that my blog belongs to a user, but I cannot expect your users will know anything about my blogs. Now perhaps the representation of the user that you give me includes a list of blogs that the user owns, in which case I can PUT back a representation that includes the URI of my Blog, but my code cannot rely on that information being known.

You also get some interesting behaviours.

All you have is a URI. You have NO idea what that resource is. As a result, its conceivable that I might try to create a blog that is “owned” by an image. Or the URI point to a resource that is a legitimate owner of the blog but its media types are not those that the blog application understands. For the most part, that’s OK because the blog application is really just an API, its not our User Interface. The User Interface application will need to have coping mechanisms for this but I’m not sure that’s a problem. If all you’re producing is an API, then its not even your problem.

More importantly, you get all sorts of strangely compelling polymorphism. Its liberating not to know what you’re associated with since it guarantees that you’ll become associated with all sorts of unexpected things.