rest/opacity

(Adapted from a post by RESTfather Roy Fielding on rest-discuss; used by permission. The context was a discussion about what Tim Berners-Lee meant by his so-called 'Opacity Axiom', specifically how it applies to REST.)

Roy on Opacity in REST

Historical Context

...In order to understand TimBL's design notes, you have to know about the context in which he is writing a response. In this case, various thoughts were recorded as the "Opacity Axiom" in response to a discussion about client behavior and the perceived need for URNs. It has long since been taken out of context and abused in various ways.

Also, keep in mind that TimBL's design note was written two years after Henrik and I [Roy Fielding] worked out the important bits of what would become the HTTP object model, later re-named REST to get away from OOM terms, about 18 months after we had similar discussions at MIT with TimBL and the rest of the W3C team, and more than a year after HTTP/1.0 was finished and HTTP/1.1 proposed. So, saying it was something that I "didn't really care to integrate so much" is missing the mark by quite a bit -- he is trying to describe our model to people who did not understand it.

Original Intent

The opacity principle, as actually used on the Web, refers only to the machine interpretation of request processing as being dependent on control data (e.g., hypertext anchors and message field names) rather than on metadata appearing within the URI. It is the same reason why we distinguish media types from data formats -- the fact that a string of bytes looks like angle tags doesn't mean we want to process it as HTML. Ignoring any semantically significant data in a URI allows operations on a resource to be orthogonal to identification of the resource.

REST does include the opacity axiom in the original sense of that phrase. I did not use it by name in REST because it isn't a principle at all -- opacity is just a name TimBL used for the set of constraints around URI processing by clients (a byproduct of the constraints that you will find in REST). The principle involved is orthogonal design.

Applicability to Clients

"Opacity of URI" only applies to clients and, even then, only to those parts of the URI that are not defined by relevant standards. Origin servers, for example, have the choice of interpreting a URI as being opaque or as a structure that defines how the server maps the URI to a representation of the resource. Cool URIs will often make a transition from being originally interpreted as structure by the server and then later treated as an opaque string (perhaps because the server implementation has changed and the owner wants the old URI to persist). The server can make that transition because clients are required to act like they are ignorant of the server-private structure.

Clients are allowed to treat a URI as being structured if that structure is defined by standard (e.g., scheme and authority in "http") or if the server tells the client how its URI is structured. For example, both GET-based FORM actions and server-side image map processing compose the URI from a server-provided base and a user-supplied suffix constructed according to an algorithm defined by a standard media type.

The Bottom Line

Note, however, that some people have taken the mere title of "opacity" and assumed that it meant URIs should not have meaningful construction at all. TimBL's axiom doesn't say that and neither does REST.

Summary by Jon Hanna

(extracted from the microformats-rest mailing list; headers added later)

Opacity Option

I think it's important that URIs *can* be treated opaquely - you can link to them, you can cache their resources without processing them beyond naive string-matching, process RDF graphs again without processing them beyond naive string-matching, and so on.

Non-Opacity Option

This does not preclude the fact that agents *may* construct or parse URIs in more sophisticated ways, it just means that agents don't have to, especially in the case of agents that are not aware of the specific purpose of the application.

Further Clarification by Roy

Not Quite

Er, not quite. The key is that the server is free to tell the client that there does exist structure in a given URI-space, and then the client is free to make use of that knowledge in future requests. That is how server-side imagemaps worked -- HTML says that the src URI is structured such that appending "?X,Y" to that URI, where X and Y are non-negative integers, corresponds to points on a map that can respond to future GET requests.

Thus, one way for the server to tell the client that a given URI is structured is to provide the URI in a standard element of a standard media type that has been defined as such. Another is to include the URI in a response header field.

Caveat to the Caveat

Note, however, that I am also one of the creators of the robots.txt standard. That particular scenario requires that the spider find out the constraints on a site prior to the very first access of a real hypertext link. It is too late to obtain that information by looking at header fields after a normal request is made. Adding a new method to HTTP was out of the question because it would have to be deployed before it could be used [OPTIONS could fulfill that purpose today, 12 years after robots.txt was defined]. Adding fields to DNS wasn't an option back then, either. Reserving /robots.txt remains the best option given the alternatives available.

Note, however, that no other "reserved URI" features have the same requirements as spiders, so it is still true that favicon and pics chose the wrong solution.

Final Comment By Dr. Ernie Prabhakar

This implies to me that the server could use a specific microformat to indicate that a given URL can be treated in a particular way. In the absence of a such a microformat, clients should not make any such assumptions; further, servers should avoid requiring clients to understand that microformat in order to receive service.

See Also

• The treatment of URI opacity on the RestWiki.
• Sean McGrath's parable on ITworld.

Issues

Questions by Mike Schinkel

1.) Ernie, you state the following:

Cool URIs will often make a transition from being originally interpreted as structure by the server and then later treated as an opaque string (perhaps because the server implementation has changed and the owner wants the old URI to persist). The server can make that transition because clients are required to act like they are ignorant of the server-private structure.

This doesn't quite sit right with me for some reason. Doesn't "opacity" refer to how a client sees the URI, not how a server sees it? To the server, nothing is/needs to be opaque, right? As such the change from a file/directory to a virtual reference inside the server configuration should have absolutely no relevence to the client and no relevence to the fact that the server/authority might have previously defined the URI structure for specific processing by a client. In the later case a change in server implementation should be totally invisible to any given client. Right?

Sure, "the opacity axiom" is specifically about client-side implementations, but I was referring to the more general concept of an "opaque string." My point was merely to illustrate that the server could choose whether or not to treat the URI as opaque (contrast how Apache and Squid satisfy URI requests) because the client (absent information to the contrary for the server) is' required to treat the URI as opaque.Dr. Ernie 13:48, 23 Jan 2007 (PST)

2.) Additionally, you state:

This implies to me that the server could use a specific microformat to indicate that a given URL can be treated in a particular way. In the absence of a such a microformat, clients should not make any such assumptions; further, servers should avoid requiring clients to understand that microformat in order to receive service.

Can you give an example of such a Microformat? Does one exist, or is it hypothetical on your part? Also, wouldn't such a "Microformat" be, almost by definition, non-visible and thus according to community concensus not a Microformat? (more of a "Microdirective", if you will. :)

Actually, it might be more helpful to think of this in the context of an HTML form, which is a type of semantic markup that provides guidance for humans (and potentially machines) about how to contruct appropriate URLs. A hypothetical microformat (no, I don't know of any so far) would most likely be some sort of "rel" relation on an anchor or form tag, and thus still "visible" in the usual sense.Dr. Ernie 13:48, 23 Jan 2007 (PST)

Thanks in advance for the clafifications.

MikeSchinkel 14:44, 19 Nov 2006 (PST)