Data-Sharing Networks

Recently I’ve been involved in designing a software system (dsn-chassis) to support data-sharing for the World-Wide Antimalarial Resistance Network (WWARN). We’re also involved in developing and maintaining software for other data-sharing networks, such as MalariaGEN. Thus we have a vested interest in figuring out what requirements these data-sharing networks have in common, so we can identify common standards and software components that can be adapted and re-used. Naturally, we want to minimise the overall effort of building and maintaining supporting infrastructure for each new data-sharing community.

Applications and Services

One shift in perspective that I’ve encouraged is a move from viewing the software required to support a data-sharing network as a single, monolithic, web application, to viewing the software as a collection of web services and applications that are loosely coupled via open protocols and formats. The hope is that this view will make it easier to identify the most generic components of the required infrastructure, such as metadata persistence and query, or authentication and authorisation. We could then adopt service protocols and formats that give us the broadest possible choise of re-using and building on existing open-source software, the smallest possible dependency or lock-in to any one vendor or product, and the flexibility to extend and customise to cope with those requirements that are unique to a given data-sharing network.

That’s the ideal, anyway.

Metadata Persistence and Query

One capability that is common to the data-sharing networks we’re involved in is management of metadata relating to scientific data being shared within the network. This metadata includes, among other things, information about the study from which the data originates, such as the scientific protocols (i.e., procedures) that were used in generating the data. This metadata can be complicated, and varies substantially between different types of experiment. It is, however, needed to evaluate the quality and comparability of data from different studies, which is a prerequisite for aggregating those data in a sensible way. To get a sense for the type of metadata that needs to be captured, check out an early prototype of the study questionnaire being developed for WWARN.

Although the exact nature of this metadata will differ from one data-sharing network to the next, the basic capabilities to persist (CRUD) and query those metadata is common. Hence, one of the first things we’ve done is to take a look at various APIs and protocols for persisting and querying arbitrary packages of metadata, and their available implementations.

Atom Publishing Protocol

One protocol that stood out was the Atom Publishing Protocol and the related Atom Format, for a number of reasons. First, it follows the REST architectural style, which implies a number of helpful constraints. Second, there are at least a few reasonably mature open-source implementations (e.g., eXist, AtomServer) that are geared not just towards blog publishing but to arbitrary data and metadata management. Third, big players like Google and Microsoft are standardisting on Atom as a basis for their Web service APIs – not that I like to flock, but reading the Google Data APIs especially gave me some confidence that many of the difficult practical issues have already been encountered and solutions found.

REST-easy?

One of the most compelling reasons I’ve found to use the REST style in designing a Web service API, and to follow the approach of mapping the four basic persistence-related operations (Create, Retrieve, Update, Delete) on to the appropriate HTTP verbs (POST, GET, PUT, DELETE), is simplicity and ease of implementation. A second reason is the ability to decouple the persistence protocol from the data model, at least to some extent. In my experience, these two factors in particular contribute to being able to rapidly develop prototypes, and to carry as little baggage as possible forward as those prototypes (and the underlying data models) inevitably evolve towards a production system.

We’ve also been working with the eXist open-source XML database, which implements AtomPub out of the box, as our metadata persistence service implementation. One of the nice things about eXist is that it will store whatever XML you throw at it, so you can fiddle with your Atom content extensions to your heart’s content without ever needing to touch the server-side code or configuration. Another nice feature is that it supports Web service endpoints implemented as XQuery scripts, which is a very convenient way to add a wide range of query service capabilities to the data and metadata you’ve stored via the Atom service.

Our current, early, prototype of a data management system for WWARN uses eXist as the Atom service implementation, and a GWT application that we’ve rolled ourselves as the user-interface.

REST-not-so-easy?

While this approach has proved excellent during the development of early prototypes, there are a number of challenges approaching in moving the system towards production strength that are causing me some consternation. My main reason in writing this article is to highlight those issues, in the hope that others can provide some helpful ideas and advice.

I (somewhat pithily) entitled this article “REST-not-so-easy” because, while some of these issues are specific to AtomPub, others seem common to any Web service API based on the REST style. It could very well be that I’m missing a few rather obvious and simple solutions, I rather hope that’s the case.

So, on to the issues…

Link Expansion

When building our user interface, we commonly found that a single view required us to retrieve data not just from a single Atom entry, but rather from a graph of linked Atom entries. For example, in our data model, Dataset entries are linked to Study entries, which represent the study from which the dataset originates. When the user is viewing a dataset, they also want to view some basic information about the linked study, such as its title and summary, in addition to a hyperlink that allows them to navigate to further information about the study.

The client application can, of course, perform two HTTP GET operations to retrieve this simple graph of two linked Atom entries, which is easy enough. However, we have other examples where the graph to be retrieved has a depth of 2 or more. In these cases, implementing the client application becomes far simpler if the whole graph can be retrieved with a single HTTP GET operation, rather than many. A single GET is also better for latency, which is an important concern for us where users may be located in parts of the world with poor network bandwidth.

A workaround for this that we cooked up early-on was to develop query services that returned a single, root Atom entry as the result, with the required links outbound from and inbound to the target Atom entries expanded inline, within the atom:link element. I was initially unsure of the sanity of this approach, especially given that including an atom:entry directly within an atom:link element does seem to break the Atom Format Spec. However, I found some discussion from folks at Microsoft and Google on doing inline link expansion, which gave me some confidence that this idea isn’t so crazy. I also found a suggestion that returning the graph of linked Atom entries as a feed, rather than a single Atom entry with links expanded inline, was a more elegant way to go, which I have some sympathy for. However, we’ve gone with the inline expansion approach for now, which is working well so far. For example implementations using XQuery, see chassis dataset query service and the supporting function library.

There is an obvious gotcha here, which is that if you retrieve an Atom entry with links expanded inline, then PUT that entry back to the edit link URL, you will end up storing the linked entries inline too, which leads to all sorts of interesting errors. So, any view that needs to do a PUT must first retrieve a fresh, unexpanded representation of the entry, before it can be modified and PUT back to the server.

I’ve also glanced at an article on AtomServer which describes the idea of aggregate feeds created by joining separate collections. It looks like this might provide a similar capability, but in a quite different way. However, if I have understood the idea correctly, it looks like you could only ever fetch a graph on entries 1 deep.

Referential Integrity; Broken Links

Our simple data model currently comprises entities such as Study, Dataset, Data File, Submission and Review. There are, of course, associations between these entities, such as an assocation between a Dataset and the Study from which it originated, as mentioned above. When we map our logical data model onto Atom, each type of entity maps onto a distinct collection of Atom entries. Each type of entity also gets its own Atom content extension, which means that we stick the entity data into some XML nested in the atom:content element of the Atom entry. Associations between entities are mapped onto links between Atom entries. I.e., we use the atom:link element to represent associations between entities, where the value of the rel attribute represents the type of the association.

To create a new association, a client retrieves a representation of the entry that is the source (i.e., subject) of the assocation, adds a new atom:link element with the appropriate rel attribute (describing the association type) and href attribute (pointing to the target/object of the association), and then PUTs the entry representation back to the entry’s edit link URL.

By default, eXist’s Atom implementation does not do any referential integrity checks on the links in an Atom entry. This is perfectly sensible, because no integrity constraints have been declared, and because the link could equally point to an Atom entry anywhere in the Web, not just other Atom entries located on the same service.

However, if an Atom entry that is the target of one or more links is deleted, then all of those links will be broken. How should the service deal with this, if at all? Is it OK to do as the Web does, and leave it up to the client (and the user) to deal with broken links? Or should the service be a bit smarter, e.g. by noisily preventing deletion of entries that are link targets, or by silently deleting links to deleted targets, or by some other mechanism? What about new links that are created to non-existent targets? Should those be prevented somehow? And should the service differentiate between links whose targets are entries hosted by the same Atom service and links whose targets are elsewhere in the Web? If so, how? I.e., are there different kinds of referential integrity that need to be considered?

Our current, very tentative, position is to do as the Web does, and leave it to the client to handle broken or bad links. The client will typically do this by simply notifying the user of a broken link. The client will provide the facility to allow the user to fix broken links, but what the user does next is up to them. However, we haven’t implemented any of this functionality in the client application yet, so I don’t know what the consequences will be.

Transactions

Sometimes, what is a single operation from the user workflow point of view, such as creating a study, or creating a dataset, or updating a study with new information, maps onto a single Atom Protocol request, such as an HTTP POST or PUT. However, often a single user operation maps onto multiple Atom Protocol requests. For example, creating a new revision of a data file involves four HTTP requests, which are a POST (create a new media entry that represents the data file revision), a PUT (add some metadata about the media entry, e.g., the original file name), a GET (retrieve a fresh representation of the data file entry), and finally a PUT (link the data file entry to the new revision).

Any of these requests could fail, leaving the data in an inconsistent state. How should we handle this possibility?

The obvious answer is to provide some sort of transactional capability, such that the client can invoke these requests within the context of a single transaction that will either completely succeed or fail with no change to the data. But that begs two further questions: how do you add transactions to the Atom Protocol, and how do you implement them?

In my not-very-extensive searching of the Web, I have not found much in the way of discussion and/or implementation of transactional capabilities for REST-style web service APIs, except for some work on transactional support for JAX RS based applications. An alternative to a protocol extension is simply to design the Web service API to expose only those operations that are atomic from the user workflow point of view, then handle transaction-type issues behind the scenes. But that would mean leaving AtomPub behind and layering another API on top, which seams to completely defeat the purpose of going for REST/AtomPub in the first place.

The other option is, of course, to leave it up to the client to deal with inconsistent data, which might include trying cleaning it up automatically or simply notifying the user of problems and leaving the rest up to them, as with the discussion of broken links above. But that would leave open all sorts of weird and wonderfully wrong possible states of the system.

If someone were to point me to an extension to the Atom Protocol that allows multiple Atom requests to be carried out in the context of a single transaction, and an existing, mature, open-source implementation, I would be very happy .. I think.

AtomServer has a capability to combine several Atom requests into a “batch”, via a custom extension to the Atom Protocol, based on the batch processing capability in the Google Data APIs. Although I couldn’t find any mention of adding transactional processing of an entire batch in the AtomServer documentation, it does mention that a batch is processed as a single database operation, and so it shouldn’t be too difficult to wrap that with a transaction. I’m not sure how that would impact the AtomServer batch response model, however.

Granularity

The Atom Publishing Protocol, as with any CRUD-style API, is relatively coarse-grained, in the sense that if you want to update any field of an entry, you have to update the whole thing at once. This coarse-grained nature is good from one point of view, because it means that the details of the data model are kept separate from the design of the protocol. The data model can then evolve without needing any change to the protocol specification or implementation. However, it does mean that the protocol can be quite inefficient and wasteful in terms of network bandwidth. I.e., if you’ve stored a reasonable amount of data in a single Atom entry, but all you want to do is change the title, you’ll have to PUT the whole thing. This can mean noticeable latency in the client application, especially where network bandwidth is poor.

On the face of it, this issue is less serious than dealing with broken links or inconsistent data, but it does make other protocols that provide the ability to submit a fine-grained change set or change request for a particular data or metadata record (such as the Talis Platform API) start to look quite appealing.

When you delve a bit deeper, this issue also interacts with the issue of authorisation and access control, which I’ll come to below.

Authentication

I’ll just mention this briefly to say that, for us, authentication is not an issue. We use a Spring Security filter to implement authentication using HTTP Basic in the development environment, and webauth in production.

I have seen articles on inventing new authentication protocols for Atom to work around restrictions imposed by their server environment, but those restrictions don’t apply to us.

As a side note, I suspect there is a bug in the authentication code in the eXist Atom servlet, as we had some pain trying to turn it off and use Spring Security for authentication instead (for HTTP Basic authentication, we had to add a filter that removes the Authorisation header before it gets to the eXist Atom servlet, otherwise the servlet’s own authentication process was activated), but I haven’t pinpointed it yet. I’d like to see eXist factor out and consolidate all authentication code from its Atom and XQuery servlets to dedicated filters, which would make for easier integration with Spring Security or other authentication frameworks .. but that’s another story.

Security, Authorisation and Access Control

I’ve saved the best for last.

Within a data-sharing network, typically different roles are defined, with different privileges/permissions/authorities/rights/acls/…

For example, for WWARN we have defined a submitter role, which is a person who wants to share original research data with WWARN; a gatekeeper role, who performs an initial review of submitted data and decides whether or not the data should be accepted for curation; a curator role who’s job is to clean up and standardise data submitted from different studies so that it can be sensibly aggregated; a coordinator who oversees the operations of the data-sharing network; and an administrator who installs, configures and maintains the systems.

There are things people should be able to do. For example, a submitter creates studies, datasets, and data files, and can submit a dataset to the network. A gatekeeper reviews submissions and assigns a curator if the submission is accepted. A curator creates derived data files and reviews curated data for validity and conformance with standard data dictionaries.

There are, of course, things people should not be able to do. For example, a submitter cannot review their own submission or decide that their submission should be accepted by the network. Neither can a submitter assign a curator to their submission. These capabilities are allowed for the gatekeeper only.

These permissions can be implemented at the user-interface level. I.e., the user-interface can expose only those functionalities that are permitted for the user’s role(s). However, if the client application is using a Web service to implement some or all of these operations, then the Web service API must also have appropriate constraints, otherwise users would be able to hack the API and do things they shouldn’t. If the Web service API is based on Atom, then the problem we have is how to implement the appropriate authorisation constraints in a way that works with Atom and existing implementations.

Some constraints are simple to implement. For example, the constraint that only a submitter can create a study can be implemented by allowing only POST requests to the Studies Atom collection URL if the user has the submitter role. I.e., constraints that can be mapped onto a specific class of CRUD operation on a specific Atom collection with a specific role are straightforward, and could, for example, be implemented using a Spring Security filter and URL patterns.

Others are not so simple. For example, a submitter who creates a new study becomes the owner of that study. The original owner may also grant ownership to other submitters; the owner(s) of a study are the only people who can update information about that study. This begs the questions, how do we represent entry-specific access control constraints in the Atom Protocol and/or Format, and what implementations (if any) are available?

The eXist database does have its own solution for Atom security. You can declare access control constraints for a specific Atom entry by including an exist:permissions element within the Atom entry. However, this presents three problems for us. First, it requires that we use the eXist database of users, but we have to integrate with another, external database of users, so we would have to keep those two synchronised. Second, it is based on the Unix/Linux model of file system permissions, which is too inflexible. For example, we want to enable a user to grant arbitrary permissions for a given entry to arbitrary collections of specified users. Having a single owner/group for each entry means you cannot do this sort of thing. Third, the format is specific to eXist. If we buy into their permissions format, we will have a job to port it to another atom implementation such as AtomServer, if we need to do that at some point in the future.

eXist also has an XACML sub-system, but the capabilities do not obviously map onto our requirements for access control to Atom collections and member entries.

The Google Data APIs have a different approach. See for example the Google Calendar API section on sharing calendars. Each access control rule is an Atom entry. There is one collection of access control rule entries for each calendar. Each access control rule has a “scope”, which typically specifies a user, and a “role”, which is typically a permission such as “read” or “editor”. See also the gAcl namespace reference.

The Google approach has the flexibility we need, but I’m baulking at implementing something like this ourselves on top of eXist or another Atom implementation.

And there is another issue here. It’s not hard to imagine situations where you might want different users to be able to update only specific parts of an Atom entry, but not the whole thing. I.e., you might want to have fine-grained, within-entry access control rules for different users. This sort of thing can be handled in a purely Java environment, for example, using Spring method security and Spring domain object security. However, it’s not obvious how to implement this sort of thing in a general way for REST/Atom, and so we’ve deliberately designed our data model to avoid needing this kind of rule. I.e., rules are only ever defined at the granularity of a single entry, nothing finer. This means that authorisation considerations have a significant impact on the design of the data model. Not necessarily an issue, but interesting nonetheless.

We need a way forward here, which means minimum coding for us (we want to focus our effort on the applications, rather than the underlying services), and maximum simplicity and portability.

Any Thoughts?

If you’ve read this and have any thoughts, ideas or suggestions about any of the issues above, really anything at all, no matter how trivial, please do drop me an email, or add a comment to this article, or join the dsn-chassis group and post a message there. Thanks for reading.

First, let me say that it has been my pleasure to attend Dublin Core conferences since 2005 in Madrid.

Thanks to the organisers here for putting on a great conference, and for inviting me to give this talk, it is a real honour.

I will post notes from this talk on my blog at purl.org/net/aliman – so don’t worry if you miss anything.

It is also excellent timing, as after at least 5 years of talking about it, I can finally say that, on 18 August 2009 SKOS – the Simple Knowledge Organisation System – was published as a W3C Recommendation, thanks in large part to the experience and support of this community.

I’d like to talk more about SKOS later in this presentation.

Before I get into the main body of my talk, let me say first that, one of the lessons I’ve learned in working on Semantic Web Deployment, especially in the last 2 years, is that big ideas can look great when viewed from a distance, but the reality on the ground will always be far more complex and surprising than you anticipate, as I’m sure many of you can testify to.

In particular, the cost/benefit trade-offs for investing in a particular technology or approach can vary wildly from one situation to the next. Nothing is more important than keeping an open mind.

So, although I love talking about this big ideas, I’m going to try to put them to one side for this morning (although, I’m sure I’ll still be tempted by one or two). Instead, I’d like to simply share a few experiences, and hopefully give you at least a flavour of where some of the opportunities and the challenges might lie.

Now, given that we’ve been talking about metadata for at least two days now, I thought I’d spend a few minutes talking about something completely different.

Fruit Flies

These are fruit flies, of the species Drosophila melanogaster.

On the left is a male, on the right a female. Females are about 2.5 millimeters long, males are slightly smaller.

When I first saw this image, I thought it was a photograph. But, when I looked closer, I noticed it was in fact a drawing, and it was signed “E Wallace”.

Edith Wallace, I found out, was curator of stocks and artist for Thomas Hunt Morgan, who c. 1908 began using fruit flies in experimental studies of heredity at Columbia University.

Heredity simply means the passing of biological traits, such as green eyes or brown hair, from parents to offspring.

I’d like to talk more about Thomas Hunt Morgan, but before I do, it’s worth noting that when Charles Darwin published his book On The Origin of Species by Means of Natural Selection in 1859, which of course depends in its central argument on the inheritance of traits from one generation to the next, the underlying mechanisms of that inheritance, i.e., of heredity, were completely unknown.

It was, of course, Gregor Mendel’s work on pea plants, published first in 1865, but not widely known until the turn of the 20th century, which first suggested that organisms inherit traits in a discrete, distinct way. It is these discrete units or particles of inheritance that we now call genes.

However, at the turn of the century, although the theory of genes was accepted, it still was not known which molecules in the cell carried these genes.

It was T. H. Morgan’s aim to determine what these molecules were.

He needed a suitable animal to study, and chose Drosophila, primarily because of cost and convenience – they are cheap and easy to culture, have a short life cycle and lots of offspring.

Morgan was looking for heritable mutations to study, and spent a long time looking before he discovered a few flies with white instead of the usual red eyes. The white-eyed mutation only occurs in male flies, i.e., it is a sex-linked trait. The fact that the trait depends on the sex of the individual suggested to Morgan that the genes responsible for the mutation reside on the sex chromosome … and that the chromosomes generally are the carriers of genetic information.

Morgan and his students also discovered that some traits, such as wing length and eye shape, do not get mixed up randomly from one generation to the next, but rather tend to be inherited together.

They demonstrated that the reason why these traits tend go together is because the genes responsible are on the same chromosome, and are in fact quite close together on the same chromosome, and hence tend to stay together when chromosomes recombine during formation of sperm and egg.

They then used these observations to construct the very first genetic maps, that is, maps of where different genes are in relation to each other on the chromosomes. This was the first crucial step in understanding how a genome is organised.

Morgan started something of a trend when he chose Drosophila, because for the last century Drosophila has been one of the most intensively studied model organisms.

It is fair to say that much of what we know today about genetics, and the molecular mechanisms underlying development, behaviour, aging, and many other biological processes common to flies and humans, we know from research on fruit flies.

In the last decade, that research has entered an entirely new phase.

In 2000, the complete genome sequence of Drosophila melanogaster was published. For the first time, we gained a complete picture of the location and DNA sequence of every gene in the genome. Publication of those data has unlocked entirely new methods and avenues of scientific investigation.

However, many questions remain unanswered.

What do all those genes actually do?

Where and when do they play a part in various biological processes?

And, especially, how do genetic differences between individuals relate to different biological outcomes?

E.g., translating that question from flies to humans, why do individuals with one genotype tend to be resistant to a disease such as malaria, whereas others don’t?

Answering these types of questions is the domain of functional genomics.

A functional genomic study typically asks, in relation to some biological process like development of sperm or the eye, …

What genes are active, where, when and how much?

Which genes are interacting with each other?

What happens if you stop a gene from working (knock it out), then restore it?

For example, this image shows that a particular gene called schumacher-levy is active during the developing Drosophila embryo, and is localised to a specific organ, in this case the developing gonad.

Other advances in biotechnology over the last decade have revolutionised this type of research.

For example, we now have a number of tools for carrying out high-throughput experiments. These are experiments where, rather than looking at just a handful of genes, we can look at 100s or even 1000s of genes at a time.

The bottom line: these high-throughput technologies, in addition to rapid advances in genome sequencing technology, generate a very large amount of highly heterogeneous data.

And our ability to generate new data on unprecedented scales is accelerating. Really, the pace of change is staggering.

To give you an idea of the rate and scale of new advances, consider that, in July 2008 the Wellcome Trust Sanger Institute, the primary institute for DNA sequencing in the UK, announced that, every two minutes, they produce as much DNA sequence as was deposited in the whole first five years of the international DNA sequence databases, from 1982 to 1987.

Finding, comparing, and integrating these data is a critical challenge for ongoing biological research in all organisms, and is especially critical in translating findings from model organisms such as Drosophila to human health.

Because this is such a critical problem, much excellent work has already gone into making Drosophila data publicly available and accessible.

For example, FlyBase provides access to primary genome sequence data on all sequenced Drosophila species. It is the primary reference point for all Drosophila genome-related data.

FlyBase also establishes a controlled vocabulary for all Drosophila genes, which is a vital tool in integrating data from multiple sources, because genes are often the point of intersection between data sources.

BDGP embryo in situ database is an example of a database holding the output of high-throughput functional genomic studies. They publish images that depict the expression of a gene within the developing fly embryo, at various stages during the course of embryo development, for thousands of genes.

FlyAtlas also publishes data from high-throughput experiments, but using a different technique. They use DNA microarrays to get quantitative data on gene expression in different tissues, so the data tell you not only whether a gene is active, but also how much it is being expressed.

Much of the focus has, to date, been on providing a direct user-interface to each source of data. So each data provider has a set of web-based tools for a human researcher to query and visualise the data.

However, the focus is shifting towards enabling data to be harvested and integrated across databases in an automated way, because it is recognised that this could save much time and effort, and because some questions just couldn’t be answered any other way.

And there are projects making headway here, for example, FlyMine uses a conventional data warehouse approach to integrate data on Drosophila. However, it is no small challenge, and there is a pressing need to make the end products much more usable, and because public funding will always be scarce, to make the whole process as cost-effective, scalable and sustainable as possible.

FlyWeb Project

With this in mind, in January 2008, I moved to the Zoology Department at the University of Oxford, to work with a team there led by Dr David Shotton on a small project called FlyWeb.

FlyWeb asked two questions…

1. Can we build tools for Drosophila biologists that cut down the effort required to search across different sources of gene expression data?

2. Under the hood, what (semantic) web tools and design patterns help us to build cross-database information systems, and ensure that they are robust, performant and quick to build.

To answer these question, we set about building a proof of concept, which is deployed at openflydata.org.

How does openflydata.org work?

Each of the cross-database search applications I’ve demonstrated is, simply, a mashup.

It is a light weight, HTML+JavaScript application that runs in the browser, that fetches data in real time from several different web service endpoints.

Why did we use the mashup approach?

Simplest thing we could think to do. Also gave us flexibility to experiment with semantic web technology, without totally committing to it. I.e., we could mix semantic web and other solutions, if it proved easier to do so.

The bottom line was, we wanted to produce useful, compelling functionality for a biological researcher, in a reasonable time frame and with a reasonable level of performance and reliability. If semantic web helped, great, if not, fine, we’ll try something else.

Of course, it would have been great if each data source had provided a web service endpoint for their data with the necessary query functionality … but they didn’t, so we made some ourselves.

The approach we took was, for each data source, we first converted the data to RDF (the Resource Description Framework, one of the key Semantic Web Standards), then loaded the data into an off-the-shelf open-source RDF storage system (Jena TDB written by Andy Seaborne, now at Talis), then mounted each RDF store as a SPARQL endpoint.

What is a SPARQL endpoint?

It is, simply, a web service endpoint that uses the SPARQL protocol as its interface (API).

What is the SPARQL protocol? It is a simple HTTP-based protocol for sending SPARQL queries to be evaluated against a given data store.

What are SPARQL queries? SPARQL is a query language, roughly analogous to SQL (read only), but built for the Web. Basically, it gives you a way to ask pretty much any question you want to ask of a given set of data.

Why did we use RDF & SPARQL?

1. Rapid prototyping. Leverage off-the-shelf, open source, software, such as Jena, or Mulgara (see David Wood). In principle, we only had to write the software to convert the data from each source to RDF. We could then use OTS software to deploy an RDF database and SPARQL endpoint. (Caveat…)

2. SPARQL is a simple protocol with an expressive query language. That means its easy to write code for, but gives the client application (in our case, the mashups) the power to ask any question it wants. (Caveat…)

Point 2. also means we can offer these SPARQL endpoints as a service to the biological community, so others with a bit of savvy can ask their own questions. In particular, they can ask questions that we (the service provider) hasn’t thought of, which (in theory) promotes innovative re-use and exploitation of the data.

You’ll noticed I haven’t yet talked about either of the two key themes of this conference, semantic interoperability or linked data.

I haven’t mentioned these yet because, the point I’d like to make is that, depending on the context, there *may* be compelling, practical, short-term reasons to evaluate Semantic Web-based technology for a data integration project …

1. The (relative) ease of deploying a web service endpoint for querying a data source; i.e., of making the data accessible, lowering the barriers to re-use.

2. The (relative) simplicity of exploiting those web services to prototype light weight cross-database search and on-the-fly data integration applications. (pardon the pun)

Caveats…

Of course, the best choice of technology depends greatly on the existing technological context, and on the expertise available to you. As with relational databases, XML, or any family of related technologies, becoming productive with a new technology requires an investment in terms of people, training and time.

A second caveat is that, the more open and expressive a query protocol like SPARQL is, the harder it becomes to guarantee the performance and availability of a service using that protocol. It is a denial-of-service type problem. If anyone can ask any question they like, some people will ask hard questions, whether intentionally or by accident, which could degrade service performance for others.

SPARQL is thus a double edged sword. On the one hand, its open nature and expressivity is a major advantage, but that openness creates challenges when it comes to providing reliable and performant services to others.

I see resolving this tension as one of the key challenges for the semantic web community in making the technology widely applicable. We explored some strategies for mitigating these issues in FlyWeb, but we certainly did not find all of the answers.

Now, let’s talk about semantic interoperability and linked data.

The two classic problems you encounter when integrating data from different sources are…

1. schema alignment … each data source has a different data model … they structure their data in a different way … using different names for similar types of entities and relationships … or using similar names for what are actually very different types of entities and relationships …

2. coreference resolution … each data source may use different identifiers for the same thing (e.g., genes) … or (more rarely) identifiers may clash …

Our approach to schema alignment was not to try to completely align all our data sources in a single step.

Rather, we tried to pick the low-hanging fruit, and take an incremental approach, making use of existing data models as much as possible.

So, for example, the data from FlyBase come from a relational database, which is structured according to a relational schema developed over a number of years by the Generic Model Organism Database community, a schema they call Chado.

When we transformed FlyBase’s data to RDF, we used the Chado schema to help design the RDF data structures we were generating. In fact, we went one step further than that, and we semi-automatically generated an OWL ontology from the Chado relational schema. This ensured that we took a systematic approach to the data transformation, and that the definitions for the data structures in the output RDF could be grounded in the definitions already established by Chado.

Our approach to coreference resolution was, similarly, to make use of existing controlled vocabularies.

Our biggest problem was identifying genes. A single gene might be known in the scientific literature by many different names. This has been a perennial problem for Drosophila biologists, and a big part of what FlyBase has done has been to establish a definitive controlled vocabulary for Drosophila gene names, and curate a list of known synonyms for each gene.

So we used FlyBase’s unique gene identifier system as a foundation, constructed a set of URIs for Drosophila genes based on the FlyBase identifier. We then used these URIs to link data from each of the various sources.

You’ll notice I said “link data” just there. What do I mean by that?

Well, I’d like to make a distinction between two types of linking.

1. “semantically linked” – data from different sources use a common set of URIs to identify data entities, e.g., people, places, genes, diseases, (… or any two URIs that identify the same entity have been explictly mapped)

2. “web linked” – data are semantically linked, and URIs resolve to data so links can be followed by a crawler … this is what most people mean when they talk about “linked data”

We might also describe a third notion, …

3. “semantically aligned” – data from different sources use a common schema, that is, they share a common data model, (… or their respective data models have been explicitly mapped)

To build our cross-search applications, we went for the low-hanging fruit. I.e., we did just enough, and no more, to get them working.

This meant a very small amount of semantic alignment. In fact, it was quite possible to work around differences between data models, as long as those data models were understood. We certainly did not need to accomplish a complete and perfect alignment of all data models, before we could start building prototypes.

We also did not make the data web linked, i.e., we did not publish true linked data. Why not? Because it didn’t serve our immediate needs. We needed performant and queryable web services to data for each source, so we could build a mashup that selected the data it needed. Whether the data were actually linked in the web was, for this project, not relevant.

We did, however, work on semantically linking the data, i.e., mapping differences in the identifiers used, especially for genes, and this was absolutely critical to getting a reasonable level of recall and precision.

Which is not to say that I think true, web linked data is a bad idea. There may be other reasons for deploying web linked data, which would have been relevant especially if we had wanted to go beyond a proof-of-concept system.

But the point is, you can get a quick win if you make data available via a queryable web service. You save a lot of time and effort if data from multiple sources are semantically linked. But your data certainly don’t need to be perfectly and completely semantically aligned before you can start using them.

SKOS

I’d like to leave flies now, and return to SKOS.

As you all know better than I do, one of the cornerstones of information retrieval for many years has been the development of controlled structured vocabularies, such as the Library of Congress Subject Headings, or the Dewey Decimal Classification, or the Agrovoc Thesaurus.

Ever since the advent of the Web, there has been a desire to make better use of these valuable tools, to help organise and connect information as it emerges from closed silos and is shared via the Web.

Hopefully, the Simple Knowledge Organisation System (SKOS) will go some way towards enabling that to happen.

SKOS provides a common, standard, data model, for controlled structured vocabularies like thesauri, taxonomies and classification schemes.

This means that, if you own or have developed a controlled vocabulary, and would like to make it available for others to use, you can use SKOS to publish your vocabulary as linked data in the Web.

Because SKOS is now a standard, your data will be linkable with other vocabularies published in a similar way, and (hopefully) compatible with a variety of different software systems.

If you were at DC2008, you would have heard Ed Summers talk about his work to deploy the Library of Congress Subject Headings as linked data, using SKOS.

His initial work was deployed at an experimental site, but since then, based on Ed’s work, the Library of Congress has deployed their new Authorities and Vocabularies Service at id.loc.gov.

The first service deployed there is, of course, the LCSH.

To explain what LOC have done, for each heading in the LCSH, LOC have minted a URI for that heading.

For example, the URI http://id.loc.gov/authorities/sh95000541#concept identifies the LCSH heading for the World Wide Web.

If you plug that URI into the location bar of your browser, you’ll get a conventional web page providing a summary of that heading.

With a small change in the way you make that request, you can also retrieve a machine-readable representation of that heading. I.e., you can get data. Those data are structured using SKOS.

Each heading is, of course, linked to other headings, so you could, if you wanted to, follow the links from one heading to the next, collecting data along the way.

Alternatively, if you want to re-use the entire LCSH in your own application, you can download the whole thing in bulk, again as data, structured using SKOS.

I hope what the Library of Congress have done with LCSH achieves three things.

First, I hope it means more people use the LCSH. For all it’s quirks, the LCSH, like many other vocabularies, is an invaluable resource, and I hope we will see it turned to wild and wonderful new uses.

Second, I hope it encourages other projects to re-use the LCSH URIs, to link their metadata records to LCSH via the Web. That would make it much easier to make use of links between metadata records across existing collections.

Third, I hope LCSH serves as a hub for linking other vocabularies as they emerge into the Web.

I think that, in a short time, it is not unrealistic to imagine that, we could see LCSH as a hub in a web of linked vocabularies, with that web of vocabularies itself serving as a hub for a much larger and broader web of linked metadata.

That this is possible is demonstrated by the fact that the LCSH has already been linked to another vocabulary, the French RAMEAU vocabulary, used by the Bibliotheque nationale de France.

A second piece of work I’d like to highlight is Michael Panzer’s work on dewey.info.

One other project I would like to highlight from within the DCMI community is the work of Jon Phipps and Diane Hillman on the NSDL metadata registry.

In fact, to call it a registry doesn’t, in my mind, do it justice, because it is a complete vocabulary development, maintenance and publication platform, built using SKOS.

And it doesn’t only cover vocabularies, it covers metadata schemas (a.k.a., element sets) too.

And not only does the registry make all of the underlying data accessible via normal HTTP requests, (which you could use to implement linked data), it provides a SPARQL endpoint too, so you can query the schemas and vocabularies however you like.

The LCSH, dewey.info, and the NSDL metadata registry, are just three examples of recent uses of SKOS, a good source for more is the SKOS Implementation Report.

Let me be the first to say that SKOS isn’t perfect. Neither does it cover every eventuality. While thesauri, classification schemes and subject heading systems do have something in common, they also exhibit diversity. In many cases, that diversity is not just a historical artifact, but exists for good reason, because the vocabulary is adapted for a specialised purpose.

Our goal with SKOS was to capture enough of this commonality to enable some interoperability, but to provide an extensible foundation from which different communities could innovate, and explore solutions to their own particular problems.

We’ve seen already, at this conference, for example, in Michael Panzer and Marcia Zeng’s presentation, how work is well underway to develop extensions to SKOS for classification schemes.

This photo was taken during a discussion of how to extend SKOS for the Japanese National Diet Library Subject Headings.

On this note, I’d like to share a small insight, which I gained while I was working on SKOS, thanks to my colleagues in the W3C Semantic Web Deployment Working Group.

When I started working on SKOS, I thought that developing a standard was about getting everyone to do the same thing. I.e., it was about uniformity.

Now, I have a different perspective.

Consider that, if the developers of the original Web standards had tried to think of every possible way the Web might be used, then tried to design a complete system to accommodate all those possibilities they could think of, the Web would probably not exist today, for two reasons.

First, they would still be here today, imagining new possibilities, and arguing about how to deal with conflicting requirements.

Second, they would have built a system that was too complicated.

Whether by intention, inspiration, or accident, the original Web standards have not led to uniformity, but have rather led to an explosion of innovation and diversity.

Thus, my insight is that, a good standard, at least for the Web, is one that provides a platform for innovation. It musn’t try to do too much. Of course, it must be clear about everything that is within its scope, and so provide a sound basis for interoperability. But it should be aggressive about limiting its scope. And it must be flexible and extensible, to accommodate differences, and to enable unexpected ideas to be realised.

Striking this balance is, of course, far from easy, and I have no way of knowing whether SKOS has found the right balance. However, the people I have met through my work on SKOS continue to be an inspiration, and I hope at the very least it will provide a stepping stone to the future.

RDA, FRBR and RDF

Of course, when it comes to sharing and linking metadata, controlled vocabularies and SKOS are only a small part of the picture.

We also need standards for sharing and linking the metadata itself, standards that provide a basis for interoperability but that also can, as Michael Crandall beautifully illustrated on tuesday, accommodate the richness and complexity of of our descriptions, and of the artifacts they describe, be they literary works, works of art, or the results of scientific inquiry.

Here, too, there are opportunities to build on a significant body of previous work.

Two such bodies of work are the Functional Requirements for Bibliographic Records (FRBR), and the Anglo-American Cataloging Rules (AACR), which is the precursor to the Resource Description and Access (RDA) specification.

Now, I am not an expert on bibliographic metadata, so I cannot comment on the details of these standards.

However, I can tell you that, it is possible to take the data models underlying FRBR and RDA, to publish those data models using Semantic Web standards, and then to use those models as a framework for transforming existing metadata records to RDF and publishing them as linked data in the Web.

Earlier this year, I did a very modest amount of work with the DCMI-RDA task group, proving the concept. Using a set of cataloging examples, and using the RDA elements schema developed by the task group, I developed some patterns for representing bibliographic metadata as RDF.

I then tested these on a larger scale, using a dump of just under 7 million MARC records from LOC. I showed that at least some of the metadata from the MARC records could be transformed to RDF using the FRBR schema and RDA elements schema and vocabularies.

Next steps for this work would be to increase the coverage of the converted data, and to publish it not only as linked data but also via queryable (e.g., SPARQL) web services, which would drastically reduce the barrier to re-use of this fantastic resource.

Data-Sharing Networks for Malaria Research

I’d like to conclude my talk by returning to scientific research.

In June this year, after the FlyWeb Project finished, I moved up the hill in Oxford to join the Centre for Genomics and Global Health, which is a joint research programme of Oxford University and the Wellcome Trust Sanger Institute, directed by Prof. Dominic Kwiatkowski.

The main focus of our research is to assist the global campaign to eliminate malaria.

According to the WHO’s World Malaria Report 2008, half the world’s population is at risk of malaria, and an estimated 247 million cases led to nearly 881,000 deaths in 2006. Small children remain by far the most likely to die of the disease.

The recent advances in biotechnology that I mentioned earlier, in particular the rapid advances in DNA sequencing and genotyping technology are, of course, being brought to bear on the problem.

One of the most promising approaches is called genomic epidemiology, which combines genetic data from the lab with clinical data from the field, to understand why, for example, some people are less susceptible to serious infection than others. It is hoped that, by understanding the natural mechanisms of protective immunity against malaria work, this may contribute to the development of an effective malaria vaccine.

The crunch is that this type of research cannot be done on a small scale. Because genomic epidemiology involves analysing hundreds of thousands of points of variation in the human genome, and searching for associations between these genetic differences and different disease outcomes, a large number of samples (i.e., patients) need to be included, to gain the necessary statistical power to find genuine associations.

Thus genomic epidemiology requires research collaboration on an unprecedented scale. And the key to enabling this type of research is data-sharing.

Will the Semantic Web and linked data help? I hope so. The Web, in all its chaos and diversity, will certainly play a pivotal role. But the challenges are too broad to be solved by one family of technologies alone.

Many of the key challenges are social, rather than technological.

For example, enabling a scientific community (i.e, one not trained in data modeling) to quickly reach working agreements on data standards, and enabling scientists to translate between their own view of their data, and a standardised view of their data, is vital. Here, bridging the gap between the technology and the people has never been more important.

Similarly, reaching agreements on when data may be shared, and on how it may be used, is key. Because, of course, in addition to ensuring credit is received for individual scientific research, personal genetic and clinical data is highly sensitive, and there are strict ethical rules about data use and privacy.

In spite of these challenges, I remain hopeful. I am especially encouraged by the openness of communities like this one, and on the willingness of those communities to share their experience and expertise.

With that, I hope you enjoy the rest of the conference, and thank you for listening.

Just a short post to say that the Simple Knowledge Organization System (SKOS) Reference is now a W3C Recommendation.

W3C issued the following press release: From Chaos, Order: W3C Standard Helps Organize Knowledge

I’m proud to have been a part of this work, and extremely grateful to all those who have supported and contributed over the last 5 years.

A little tidbit, if you want to run GWT unit tests in manual mode from Eclipse, right click the test case class and select Run As > GWT JUnit Test as you would normally, which will create a run configuration for you. The first time round this will run the test in hosted mode. To get the test to run in manual mode, go to Run Configurations, select the run configuration for your test, then in the VM arguments box under the Arguments tab enter the following …

-Dgwt.args="-manual"

Almost two months ago now, the Semantic Web Deployment Working Group published the SKOS Reference Candidate Recommendation. Since then, we’ve had a good number of high quality implementations (see also Sean’s SKOS implementations spreadsheet), which is excellent news.

Over the last year or so, my main priority has been the FlyWeb Project. Unfortunately, FlyWeb was supported by short-term funding (18 months), and is coming to an end soon. Here are a few belated notes on what we did and why we did it…

The main goal of FlyWeb was to minimize the time required for a researcher in the domain of Drosophila (fruit fly) functional genomics, with no informatics training, to find and compare gene expression data from different databases on a large number of genes. With this in mind, we developed openflydata.org, which hosts the following cross-database gene expression data search applications:

• openflydata.org/search/gene-expression – search for a single gene of interest, and then retrieve and display expression data for that gene, including tissue-specific mRNA levels from FlyAtlas, embryo in situ hybridization images and ontology annotations from BDGP, and testis in situ hybridization images from FlyTED. Also retrieved are literature references relevant to the selected gene, provided by FlyBase.
• openflydata.org/search/gene-batch-expression – search for a batch of genes, then retrieve and compare expression data from FlyAtlas, BDGP and FlyTED, for all matching genes.
• openflydata.org/search/by-expression-profile – search for genes matching a given tissue-specific mRNA expression profile, based on data from FlyAtlas, and then retrieve further expression data for each gene found.

The applications are all pure JavaScript, built using a custom library called FlyUI. They fetch data AJAX-style directly from four SPARQL endpoints, one for each of the four sources of genomic data. On the server side, we use Jena TDB as the underlying RDF storage and query engine, and SPARQLite as the SPARQL protocol server. The whole thing runs on a small EC2 instance.

Further details on our work to convert the four data sources to RDF, in addition to bulk RDF downloads, SPARQL endpoints and more, can be found at the links below:

• FlyBase
• BDGP
• FlyAtlas
• FlyTED

This is just a short post to say that I’m moving to a new role shortly, and so my contact details are changing also.

To reach me via email, use: alimanfoo at gmail dot com

I will be on leave from 15-31 May.

I gave a presentation on SKOS and Semantic Web Deployment last week at the Library of Congress. Here’s the blurb..

Links are valuable. Links between documents, between people, between ideas, between data. Data is now a first class Web citizen, and the Web is expanding as more of these valuable networks are deployed within its fabric. Well-established knowledge organization systems like the Library of Congress Subject Headings will play a major role within these networks, as hubs, connecting people with information and providing a firm foundation for network growth as many new routes to the discovery of information emerge through the collective action of individuals. Or will they?

This talk introduces the Simple Knowledge Organization System (SKOS), a soon-to-be-completed W3C standard for publishing thesauri, classification schemes and subject headings as linked data in the Web. This talk also presents SKOS in the context of the W3C’s Semantic Web Activity, and in particular the work of the W3C’s Semantic Web Deployment Working Group where other specifications are being developed for publishing linked data in the Web, for embedding linked data in Web pages, and for managing Semantic Web vocabularies. Finally, this talk takes a mildly inquisitive look at the value propositions for linked data in the Web, and how LCSH might be deployed in the Web for better information discovery.

The W3C Semantic Web Deployment Working Group is kicking off it’s final face-to-face meeting at the Library of Congress in Washington, D.C. The main purpose of the meeting is to resolve outstanding issues for the Simple Knowledge Organization System (SKOS), which are summarised on the meeting agenda.

As an aside, I heard recently about the deployment of the Library of Congress Subject Headings (LCSH) as linked data in the Web, using SKOS. This nice work provides a great backdrop to our meeting.

I had a few troubles installing RDFLIB, the Python RDF library, on my Windows Vista laptop, and getting everything to work with PyDev in Eclipse.

I have Python 2.5 installed from the MSI. When I ran python setup.py install from the RDFLIB download directory, I got the message:

error: Python was built with Visual Studio 2003;
extensions must be built with a compiler than can generate compatible binaries.
Visual Studio 2003 was not found on this system. If you have Cygwin installled,
you can try compiling with MingW32, by passing "-c mingw32" to setup.py.

I have Cygwin installed, so I installed Cygwin’s Python 2.5, then used that to run python setup.py install, which worked fine.

However, when I tried to use “Run As … Python unit-test” from within Eclipse (with PyDev installed), it didn’t work. Apparently, there are compatibility problems between PyDev and Cygwin, mostly related to windows path names.

So I went back to trying to install RDFLIB using the Windows Python. I could run python setup.py build -c mingw32 (with gcc-mingw32 installed and cygwin’s binaries directory on my path), but I still couldn’t run python setup.py install because the ‘install’ command doesn’t accept the ‘-c’ argument.

Eventually, I made it work by creating a cfg file for distutils (distutils.cfg) eg: /c/Python2x/Lib/distutils/distutils.cfg containing:

[build]
compiler=mingw32

as described here at the end of the page, under “One Last Step”.

I.e. once I had created the cfg file, and the cygwin binaries were on my path, I could run python setup.py install using the Windows Python, which also works with PyDev in Eclipse.

The W3C Semantic Web Deployment Working Group has announced the publication of the SKOS Reference as a W3C First Public Working Draft:

• http://www.w3.org/TR/2008/WD-skos-reference-20080125/

This is a substantial update to and replacement for the previous SKOS Core Vocabulary Specification W3C Working Draft dated 2 November 2005. The publication has been announced in the W3C news, and a request for comments has been sent to various mailing lists.

The abstract from this new specification:

This document defines the Simple Knowledge Organization System (SKOS), a common data model for sharing and linking knowledge organization systems via the Semantic Web.

Many knowledge organization systems, such as thesauri, taxonomies, classification schemes and subject heading systems, share a similar structure, and are used in similar applications. SKOS captures much of this similarity and makes it explicit, to enable data and technology sharing across diverse applications.

The SKOS data model provides a standard, low-cost migration path for porting existing knowledge organization systems to the Semantic Web. SKOS also provides a light weight, intuitive language for developing and sharing new knowledge organization systems. It may be used on its own, or in combination with formal knowledge representation languages such as the Web Ontology language (OWL).

This document is the normative specification of the Simple Knowledge Organization System. It is intended for readers who are involved in the design and implementation of information systems, and who already have a good understanding of Semantic Web technology, especially RDF and OWL.

For an informative guide to using SKOS, see the upcoming SKOS Primer.

Synopsis

Using SKOS, conceptual resources can be identified using URIs, labeled with lexical strings in one or more natural languages, documented with various types of note, linked to each other and organized into informal hierarchies and association networks, aggregated into concept schemes, and mapped to conceptual resources in other schemes. In addition, labels can be related to each other, and conceptual resources can be grouped into labeled and/or ordered collections.

Abstract

This short paper summarises some work done on the possibility of using OAIS information model as a basis for the model-driven design and implementation of components within a digital preservation software architecture. Two model transformations were defined using the Enterprise Architect template language. The first model-transformation transforms a platform-independent UML class model (PIM) into a set of UML interfaces specific to the Java 1.5 platform (here called a Java API model). The second model-transformation transforms a platform-independent UML class model (PIM) into set of UML classes specific to the Java 1.5 platform, implementing the interfaces generated by the first model-transformation (here called a Java implementation model). Both were applied to the OAIS information model as PIM, and the generated models are presented here with discussion.

Introduction

The CASPAR project aims to design and implement components of a distributed digital preservation architecture, taking the Open Archival Information System (OAIS) Reference Model as the basis for this architecture. Parts of the OAIS reference model are defined using UML class diagrams. Can these UML class diagrams be used in the design of CASPAR software components? Can they be used directly, or do they need to be refined in some way? What design patterns, principles and methodologies should be used to guide this refinement?

A standard pattern for the design of software systems using UML is to define a platform-independent model (PIM), and then to use model-transformations to generate other models, each specific to a given software platform (e.g. Java or C++). These platform-specific models (PSMs) are then used to generate code in the given language. These models and the resulting software system constitute a model-driven architecture. @@REF

This paper presents some work exploring the feasibility of using the OAIS information model as a platform-independent model in a model-driven design and implementation process, with Java 1.5 as one of the target platforms.

Methods

Enterprise Architect 7 was used as the primary tool for developing UML models and model-transformations. EA7 supports a bespoke template language, which can be used to define and then execute arbitrary model-transformations. EA7 also includes a number of pre-defined model-transformations for specific platforms, including Java. The pre-defined Java transformation preserves most aspects of a UML class model intact, for example it transforms a class into a class and an interface into an interface; it does, however, convert platform-independent data types to Java primitive types, and it adds public getter and setter methods for each attribute and navigable association of a class or interface, following the Java beans pattern.

Although EA7’s pre-defined Java transform provides some useful functionality, it is not sufficient for the CASPAR design process, for several reasons. Firstly, it handles multiplicities of greater than one poorly. Secondly, it does not make use of Java generics. Thirdly, the approach taken so far in CASPAR has been to define two models: an interface model (an API) and an implementation model (a set of classes implementing that API), each with their own distinct package paths. EA7’s pre-defined Java transformation simply converts package to package, class to class and interface to interface, and so cannot be used to generate CASPAR’s two separate models (API and implementation).

To overcome these limitations, two new model-transformations were defined using EA7’s template language. These transformations were based on the pre-defined Java transformation, with a number of modifications. The first transformation converts classes in the PIM into interfaces in a Java API model, in a fixed package (info.preserveddigital.infomodel). The second transformation converts classes in the PIM into classes in a Java implementatio model, in a fixed package (eu.casparpreserves.infomodel). Both transformations include modifications to handle all multiplicities appropriately, including the use of Java generics with Java collections where attributes or associations have multiplicities greater than one.

All class diagrams from section 4.2.1 of the OAIS reference model were then added to Enterprise Architect as a single package within a platform-independent model (PIM). The text of section 4.2.1 was completely ignored in the construction of the PIM, i.e. none of the informal statements made in the text were added as features of the model, so the PIM was a faithful transcription of the diagrams in section 4.2.1 only.

The pre-defined Java transformation, the custom Java API transformation, and the custom Java implementation transformation were then all applied to the PIM, and the output stored in EA.

Results

To give a flavour for the outputs of the two model-transformations, consider the class diagram illustrated in figure 4-10 of the OAIS reference model. Figure 4-10 defines the relationships between data objects, information objects and representation information. This class diagram, as the PIM used here, is given below.

The diagram below was composed from the output of the custom Java API model-transformation. Notice that each interface in the transformation output has getter and setter methods for each of the associations in which the original class was involved. Notice also that the interface RepresentationInformation has getter and setter methods for associations given in the PIM in another diagram (from figure 4-11 in OAIS).

The diagram below was composed from the output of the custom Java implementation model-transformation. Notice that all of the getter and setter methods implementing the corresponding interfaces are present. Notice also that private members for each of these properties are also now present, allowing generation of Java source code with initial code for getter and setter method bodies.

Finally, compare these with the output from EA7’s pre-defined Java transformation.

Discussion

In UML, each connector (link) can have a name, and both source and target roles for that connector can also have a name. However, in the OAIS information model, some links have a name, some don’t, and none of the links have role names. If role names are present, these can be used to give more sensible names to getter and setter methods and private members. However, without role names, the class name of the target type has to be used (e.g. getRepresentationInformation). This is fine when there is only one connector between any two classes, but where there is more than one connector between any two classes, role names are needed to disambiguate between them. Therefore, it is suggested that all connectors in the OAIS information model be given a link name, a source role name (if source is navigable) and a target role name (if target is navigable).

Notice that the class InformationObject has a method getRepresentationInformation():RepresentationInformation, due to the composition association between InformationObject and RepresentationInformation. Notice also that the class RepresentationInformation has a method getRepresentationInformations():java.util.Collection due to the recursive association on that class. The class diagrams in the OAIS information model do not specify that RepresentationInformation is a sub-class of InformationObject, hence this generalization is not present in the PIM used here. However, the text suggests such a generalization. If RepresentationInformation were a sub-class of InformationObject, then the class RepresentationInformation would end up with two methods, both getRepresentationInformation():RepresentationInformation (inherited) and getRepresentationInformations():java.util.Collection. This is due to the fact that the two “interpreted using” connectors have different multiplicities in OAIS. This creates an ambiguity, which would have to be resolved.

This is an example of the type of issue that can be revealed by an analysis of the output of a model-transformation on the OAIS information model as a PIM. It is expected that further analysis of other aspects of the transformation output will reveal similar issues with other classes. Therefore, it is recommended that further study be given to the model-transformation output generated using the transformations described here, and that this be fed back into further iterations of the OAIS information model.

Conclusions

There are potential issues with using the OAIS information model as-is as a platform-independent model in a model-driven design process. The output of model-transformations require inspection and refinement before being suitable for implementation in preservation systems. This is not necessarily a shortcoming of the OAIS information model, but does raise an important question: is the OAIS information model intended to serve as a genuine basis for the implementation of software, or is it merely intended as a guide to understanding?

Further study of the outputs of the model-transformations generated here is recommended. This study is likely to complement the findings of any attempt to re-engineer the OAIS information model as a a formal ontology, which has elsewhere revealed potential ambiguities and inconsistencies therein.

I’ve been looking around for something to help me manage my burgeoning photo collection. I’ve got a Sony Ericsson K800 and a Nikon D40, and between the two of them I’m generating quite a few images. Adobe Photoshop Album Starter Edition came with my mobile phone software, so I tried that to start with. The tagging interface worked well for me — a quick once through tagging with who, where, when and occasionally what is all I ever have time for, and is usually enough to allow me to find an image again. However, the two things that bugged me about Photoshop Album were (1) that there was no integration with Flickr, so if I uploaded photos I’d have to retag them completely, and (2) I couldn’t export my photo catalog or move it between computers easily.

After a not too exhaustive search on the Web, I found PicaJet, and downloaded the free edition. I was encouraged because the tagging interface is great (very similar to Photoshop Album), and because PicaJet has an integrated Flickr uploader which preserves all of your tagging. I also discovered that the photo catalog can be easily exported, so in a nutshell, PicaJet ticks my boxes. You can do quite a lot with the free edition — tag photos, upload to flickr, some basic editing. I’ll be upgrading to PicaJet FX (the full version, around £30) mainly because I want to be able to do more with the tag categories — in the free version you can only have a two-level hierarchy, and you can’t add new top-level categories.

I tried Picasa2, but that doesn’t have any tagging support or Flickr integration.

I also downloaded Microsoft Photo Gallery, which advertises Flickr integration. The installation process was painfully slow, then the application crashed when I tried to launch it on my bog-standard Windows XP machine.

The W3C’s Semantic Web Deployment Working Group is developing two new technologies which may be relevant to e-learning technology. These are the Simple Knowledge Organisation System (SKOS), and RDFa.

SKOS is a lightweight language for representing intuitive, semi-formal conceptual structures. So, for example, the figure below (taken from the SKOS Core Guide) depicts concepts with intuitive hierarchical and associative relationships to other concepts, and with preferred and alternative labels in one (or more) languages — these are the kinds of structures that can be expressed using SKOS. Once expressed in this form, conceptual structures can easily be published on the Web, shared between applications, linked/mapped to other conceptual structures and so on. Typically, these conceptual structures are used as tools for navigating around complex or unfamiliar subject areas, for retrieving information across languages, and for bringing together related information from different sources.

RDFa is a language for embedding richly structured data and metadata within Web pages. This allows a Web page to expose much of its underlying meaning to applications, enabling a range of new functionalities within Web clients, exchanging data between Web sites, services, and the users’ desktop applications. For example, a Web page about a new music album can use RDFa to embed structured data expressing facts about that album, such as the track listing, artist, links to sample media files etc. A Web browser with a suitable plugin or extension can use this data to offer new functions to the user, such as download the tracklisting with available samples to my music library, or compare prices from online vendors.

Both of these technologies are on the W3C Recommendation track, and are scheduled for completion in April 2008.

SKOS Foundations and Motivations

SKOS inherits much from the development of knowledge organisation systems (KOS) within the library and information sciences. Thesauri, classification schemes, subject heading systems and taxonomies are all examples of KOS widely used in information systems today.

The original motivation for SKOS was to provide a standard, low-cost way of migrating or “porting” existing KOS, especially thesauri, to the Semantic Web, so that they could be used as-is for the development of lightweight Semantic Web applications such as search/browse Web portals. This remains one of the central requirements for current development of SKOS. However, it’s worth noting that SKOS is also increasingly seen as a “bridging” technology, providing the missing link between the rigorous logical formalism of ontology languages such as OWL and the chaotic, informal and weakly-structured world of social approaches to information management, as exemplified by social tagging applications.

As such, SKOS is a very interesting technology to work on, because in the same thought you have to consider the model-theoretic semantics of RDF and OWL (and their consequences for formal reasoning), the ways in which people naturally express and organise their own conceptualisations (especially when working as a collaboration), and the potential for computational processes to “spot” and analyse emergent patterns in networks of unstructured information. The future for knowledge organisation is undoubtedly in highly collaborative, intuitive, and computer-aided environments, where people interact in a natural but structured way, being guided (perhaps unwittingly) towards the creation of emergent structures, supported by and feeding back into a range of analytic systems working behind the scenes to mine, discover and exploit patterns in information. SKOS is a small but important part of this bigger picture, in as much for the work it will lead to in the future as for the applications it can enable today.

SKOS Basics

The basic building block in SKOS is the notion of a conceptual resource, or often simply “concept”. Concepts can be labelled in one or more languages, can be annotated with various types of documentation, can be arranged into intuitive hierarchies and association networks, and can be aggregated into concept schemes and linked/mapped to concepts in other schemes.

All of these SKOS primitive features can be extended or refined to support more detailed, fine-grained conceptual models. SKOS can also be used in part or as a whole in a “mix-and-match” with other RDF vocabularies and OWL ontologies. A concrete example of this is the Semantically Interlinked Online Communities (SIOC) ontology, where SKOS can be “plugged in” to describe the topics or tags defined on a community Web site.

SKOS Design

SKOS is built on the Resource Description Framework (RDF) and the Web Ontology Language (OWL). However, SKOS deliberately hides much of the compexity of these two languages. It provides an interface between the formal underpinnings of the Semantic Web, and the more informal, intuitive ways in which people naturally express and organise knowledge. Thus, informal and semi-formal conceptual structures or knowledge organisation systems can be expressed directly in SKOS and used immediately in Semantic Web applications, without requiring any formal re-engineering.

While these topics are still under debate, consensus is emerging that the formal semantics of SKOS are, by design, very limited. Therefore, a small number of logical consequences follow from using only SKOS, compared with the larger number of logical consequences that follow from using RDFS or especially OWL directly. Whereas in some situations, a powerful set of logical entailments is very valuable, in others this can be inappropriate and/or unecessary. This is the typically the case where a formal language such as OWL is abused to express what is at best a semi-formal conceptualisation (e.g. a “concept map” or thesaurus), and a number of surprising and inappropriate inferences then follow. SKOS provides the option to model at a simpler, less formal level, which is then a starting point for more formalisation as required.

SKOS is itself an RDF vocabulary (i.e. a set of URIs), whose semantics is defined using the RDF Vocabulary Description Language (RDF Schema) and OWL. SKOS can therefore be used as a lightweight conceptual modeling language in its own right, or can be used as an adjunct to the primitives provided by RDF, RDFS and OWL in a “mixed mode” modeling environment.

An example of where SKOS, RDFS and OWL are used in “mixed mode” is the Semantic Web Environmental Directory (SWED), a prototype Web portal with “faceted browsing” functionality for finding UK organisations and projects in in the environment sector. Here, RDFS and OWL are used to model projects, organisations and their properties, such as topic of interest, geographical coverage and so on. SKOS is then used to model the semi-formal taxonomies which provide the descriptive vocabulary for these properties, e.g. animal welfare, welfare of captive animals, biodiversity etc. Of course, this is a Semantic Web application, so information can be drawn together and integrated from many different sources.

SKOS Development

SKOS is formally developed and maintained by the W3C Semantic Web Deployment Working Group (SWDWG). It is a work item on the W3C Recommendation track, which means it is subject to the full W3C Web standardisation process. However, whilst formal responsibility for SKOS rests with the SWDWG, the working group carries out all development in an open, consensus-led environment, and is ably supported by an extended community of interest. Informal participation in the SKOS development process is warmly welcomed — to join in, subscribe to the public-esw-thes@w3.org mailing list (you can also browse the mailing lists’ online archives). To participate formally in the development of SKOS, contact your W3C Advisory Committee representative about joining the SWDWG.

RDFa

From the abstract to the latest version of the RDFa Primer:

Current Web pages, written in XHTML, contain inherent structured data: calendar events, contact information, photo captions, song titles, copyright licensing information, etc. When authors and publishers can express this data precisely, and when tools can read it robustly, a new world of user functionality becomes available, letting users transfer structured data between applications and Web sites. An event on a Web page can be directly imported into a desktop calendar. A license on a document can be detected to inform the user of his rights automatically. A photo’s creator, camera setting information, resolution, and topic can be published as easily as the original photo itself.

RDFa is simply a collection of XML attributes that can be used within an XHTML document to embed structured data within that document. This data can then be extracted by an RDFa parser as a set of RDF triples.

So, for example, the following snippet of XHTML has embedded data describing a person’s contact information. The data has been embedded using the special RDFa attributes @instanceof, @property, @about, @rel and @content.

  <p class="contactinfo" about="http://example.org/staff/jo">
    <span property="contact:fn">Jo Smith</span>.
    <span property="contact:title">Web hacker</span>
    at
    <a rel="contact:org" href="http://example.org">
      Example.org
    </a>.
    You can contact me
    <a rel="contact:email" href="mailto:jo@example.org">
      via email
    </a>.
  </p>

This snippet, when parsed, yields the following RDF triples:

   contact:fn "Jo Smith";
   contact:title "Web Hacker";
   contact:org ;
   contact:email .

Given this data, a Web client could, for example, offer functions to import this contact information into a desktop contact management system.

This is a very simple example, but hopefully it illustrates the general principle that, once data is available in Web pages, new functionality becomes possible. In the e-learning sector, we might imagine Web pages which not only describe historical events, but encode data about the time, place and people involved in those events. A history student might then “cut and paste” these data from many different Web pages into their own virtual learning space, allowing them to discover and explore the many-dimensional relationships between people, places and events and build up their own structured “mini-history” which is specific to a particular learning objective or research question.

RDFa and Microformats

There’s a lively debate ongoing on the Web today about the relationship between RDFa and an analogous technology called “microformats”. Microformats have the same objective as RDFa, of embedding data within Web pages. While I’m not in a position to comment in any depth on the comparisons and relative merits of these two approaches, RDFa developers argue that the RDFa approach provides a more scalable, general purpose approach, which requires only a single implementation, and which allows different types of data to “play well” with each other. On the other hand, a bespoke transformation is required for each different microformat, and microformats could easily “clash” with each other under certain circumstances. Having said that, microformats are used on the Web today, whereas RDFa has only a number of prototype implementations. A blog post by Evan Prodromou discusses the issue in more detail, although some of the information there may be out of date (see e.g. the comments at the end).

Conclusions

Both SKOS and RDFa have interesting consequences for e-learning technology, and especially for leveraging the Web as a platform for delivering learning.

SKOS provides a lightweight technology for overlaying distributed learning content with intuitive conceptual structures, which could for example be used to aid discovery and navigation of learning resources. Conceptual structures are also themselves learning resources in their own right, and although SKOS is oriented towards information retrieval applications, the use of SKOS to express, evolve, exchange and publish “knowledge” as part of a learning process remains an intriguing avenue for exploration.

RDFa provides a general purpose technology for embedding data in Web pages. This in turn can enable a richer experience when interacting with the Web, moving beyond Web pages as unbreakable chunks of information towards Web pages as highly flexible containers of data which can be extracted, adapted, re-used and re-purposed. This technology has the potential to completely change out interaction with the Web as a learning experience.

Both technologies would greatly benefit from active participation from the e-learning community, to ensure that the requirements of near-future learning technologies are met by SKOS and RDFa within the timeline set for standardisation.

This post looks at some of the problems of identifying, decribing and linking “versions” of “digital objects”, from the point of view of the Web, drawing especially on the Architecture of the Web published by W3C. These thoughts were stimulated by the recent kickoff meeting of the new Version Information Framework (VIF) project, at which versioning was discussed in the context of adding value to digital repositories — I hope this post provides some useful input to the VIF project team.

One of the difficulties of talking about information is that it is so intangible. A book on a shelf or a file on a computer hard drive are tangible enough, but when we come to talk about a book in abstract (e.g. the complete works of Shakespeare) or a Web page, things become less tangible. One of the problems is that our normal vocabulary for talking about things at these intangible/virtual/abstract levels is not very well developed — e.g. we use “book” to refer both to a physical book on a shelf and the abstract notion of a book (which may have many copies on many shelves).

The Functional Requirements for Bibliographic Records (FRBR) specification goes a long way towards improving this situation, giving us vocabulary for four separate levels of abstraction — Work, Expression, Manifestation and Item. However, these concepts can be ambiguous and hard to apply in some situations. Nevertheless, they are an important reference point.

Web Architecture

On the Web, we face a similar situation, in that the vocabulary for talking about different levels of abstraction and the relationships between them is not well developed. However, the Web is by nature a virtual information space, and the Architecture of the Web provides some very interesting mechanisms for talking about information as a virtual commodity. The Web has also had to deal with the conceptual relationships between an information resource as a virtual entity (i.e. a Web page), and the many different media types (i.e. data formats, e.g. HTML, XML, PDF) and languages (i.e. English, Japanese etc.) which can be used to transmit or carry information.

From the Architecture of the Web, Volume One:

The World Wide Web uses relatively simple technologies with sufficient scalability, efficiency and utility that they have resulted in a remarkable information space of interrelated resources, growing across languages, cultures, and media. In an effort to preserve these properties of the information space as the technologies evolve, this architecture document discusses the core design components of the Web. They are identification of resources, representation of resource state, and the protocols that support the interaction between agents and resources in the space.

According to the Architecture of the Web, the Web is built from “resources”, and in particular, “information resources”. In practice, the notion of an “information resource” is hard to pin down, but an “information resource” is roughly defined as a resource whose essential characteristics can be conveyed in a message. Resources are identified by URIs, and you can use a URI to access a resource. The most common type of access involves requesting a “representation” of a resource, which is also known as “dereferencing a URI”. This is what your Web browser does when you type a URI in the address bar and click “Go”.

Content Negotiation

One of the interesting properties of the Web is the ability to provide several alternative representations in different media types from the same URI, which is supported in the HTTP protocol. This is also known as “content negotiation“. When a user agent (e.g. a Web browser) requests a representation of a resource, it specifies which media types it can “accept” in response. The server then sends a response in the media type best matching the request.

To take a very simple example, the URI “http://www.w3.org/Icons/w3c_main” identifies the main W3C logo icon. This image is available in two different media types — GIF and PNG. You can vary which representation you receive by changing the “Accept” header in the HTTP request.

So the URI “http://www.w3.org/Icons/w3c_main” clearly identifies something virtual, which is “above” (at a higher level of abstraction than) the notion of media type or format. Let’s call this a content negotiable resource.

Media type is not the only axis along which representations of a resource may vary. A multilingual Web site can provide representations in different languages from the same URI. For example, the Debian Web site “http://www.debian.org” provides alternative representations in English, French, Spanish and many other languages — i.e. this is also a content negotiable resource.

So it’s clear that an information resource might be a content negotiable resource — something for which representations might be provided in one or more media types and one or more languages.

However, many information resources in the Web only provide a single representation, in some media type and language. For example, “http://www.w3.org/Icons/w3c_main.png” only provides a PNG representation of the main W3C logo. Let’s call this a content invariant resource.

The interesting thing for our discussion of “versioning” is that we might express a relationship between a content negotiable resource A and a content invariant resource B, where the single representation provided by B is equivalent to one of the representations provided by A.

If we had some appropriate RDF vocabulary, we might even state this formally, e.g.

# links between resources
 rdf:type vif:ContentNegotiable.
 rdf:type vif:ContentInvariant.
 rdf:type vif:ContentInvariant.
 vif:contentVariant .
 vif:contentVariant .
 vif:contentAlternate .

# definition of the vocabulary we used above
vif:ContentNegotiable rdfs:subClassOf vif:InformationResource.
vif:ContentInvariant rdfs:subClassOf vif:InformationResource.
vif:contentVariant rdfs:domain vif:ContentNegotiable; rdfs:range vif:ContentInvariant.
vif:contentAlternate rdfs:domain vif:ContentInvariant; rdfs:range vif:ContentInvariant; rdf:type owl:SymmetricProperty.

If this metadata were made available to an application (user agent), the application could then make a user aware that various alternative representations of some Web resource were available, for example.

Changes Over Time

The other interesting property of the Web architecture is that things can change over time. For example, if you go to “http://www.bbc.co.uk/news” today, you’ll see something different from what was there yesterday. Yet, the URI “http://www.bbc.co.uk/news” identifies an information resource, so this resource must be changing over time. Let’s call this a changeable resource.

On the other hand, some Web resource haven’t ever changed, and are promised to never change. For example, “http://www.w3.org/TR/2004/REC-webarch-20041215/” identifies a time-specific “edition” or “version” of the Architecture of the World Wide Web technical report. Let’s call this an unchangeable resource. However, note that W3C also provides the URI “http://www.w3.org/TR/webarch/“, which always corresponds to the latest version in the report series.

The interesting thing for our discussion of “versioning” is that we might express a relationship between a changeable resource A and an unchanging resource B, where the representation provided B is equivalent to the representation provided by A at some specific point in time. If we had some appropriate RDF vocabulary, we might state this formally, e.g.

# links between resources
 rdf:type vif:Changeable.
 rdf:type vif:Unchanging.
 rdf:type vif:Unchanging.
 vif:snapshot .
 vif:snapshot .
 vif:priorState .

# definition of vocabulary used above
vif:Changeable rdfs:subClassOf vif:InformationResource.
vif:Unchanging rdfs:subClassOf vif:InformationResource.
vif:snapshot rdfs:domain vif:Changeable; rdfs:range vif:Unchanging.
vif:priorState rdfs:domain vif:Unchanging; rdfs:range vif:Unchanging.

If this metadata were made available to a user agent, the application could then make the user aware that a more recent representation of some Web resource was available, or that a history of changes to that resource was available, for example.

Design Patterns

So on the Web, there are content-negotiable resources and content-invariant resources; and there are changeable resources, and unchanging resources. The Web Architecture itself is entirely ambivalent to which is which. However, we might make use of these different classes, to provide some useful functionality to users of digital repositories, accessible through the Web.

Given that any information resource in the Web could be either changeable or unchanging, content-negotiable or content-invariant, there are four different possibilities to consider for every resource. This makes for a fairly complicated set of possible interrelationships. However, we might define some design patterns, which could be useful in particular situations — for example, in exposing information from digital repositories as part of the Web.

We might, for example, describe a three-level pattern.

At the top level is a changeable, content-negotiable information resource.

At the second level are a set of unchanging, content-negotiable information resources. These are less abstract, corresponding to a specific snapshot or “revision” of a resource.

At the bottom level, are a set of unchanging, content-invariant information resources. These are the most contrete entities, corresponding to a specific content variant of a specific snapshot of a resource.

The diagram below in an attempt to illustrate this pattern.

The interesting thing about this pattern is that we might say very concretely how this pattern should be implemented in the Web — i.e. how Web servers could be set up to help user agents understand better the differences and interrelationships between the various information resources. For example, the URI denoting the top level information resource could be set up to redirect to the URI of the most recent snapshot at the second level of information resources; these URIs at the second level could then be set up to content-negotiate directly as per the HTTP protocol, but offering information in the “Content-Location” HTTP header which gives the user agent a clue as to how to link to a specific content variant.

We could also cook up some RDF vocabulary to formally express all of these interrelationships, as shown in the examples above, then publish this in the Web or embed it in web pages. This would allow user agents to be fully aware of the structure of the information space, and do even more intelligent things like make the user aware that he/she is viewing an out-of-date version and that a more recent version is available, or that content variants are available in a range of languages.

Of course, this is only one of a number of possible design patterns, which need further exploration. But hopefully, this at least gives a few ideas as to how the architecture of the Web — a virtual information space — might shed light on a discussion of identifying, describing and linking “versions” of “digital objects”.

In this note, I begin to explore the use of the Eriksson-Penker UML extensions for business process modeling, as a tool for modeling the processes or work flows required to successfully interpret or virtualise a digital object.

Previously, in part 1 of this series, I explored the abstract notions of data, information, representation information and interpretation, as defined by the OAIS Information Model. In part 2, I tried to apply these notions to a simple example of a Web page. I found that we need to go beyond the OAIS Information Model if we want to capture and represent the “recipes” that take you from a sequence of bits to something more useful, in the general case where there may be multiple steps or stages required to process, virtualise or render a digital object.

Recipes and Dependencies

Take again the example from part 2 of a simple Web page, encoded as an XHTML 1.0 Transitional document using the UTF-8 character set, and stored as a single sequence of bits.

I’m interested in modeling the “recipe” that tells me how to turn the encoded sequence of bits back into a Web page, because this recipe will define the “dependencies” for the preserved object. By “dependency” I mean those items of information and/or software that are required to execute the recipe — the ingredients and utensils, to use the cooking analogy. Note that by “execution” I do not necessarily mean execution by a computer — steps in a recipe might well be entirely manual.

If I knew what these dependencies were, I could then compare them with the knowledge and software currently held by the designated community (DC), and decide which of the dependencies also need to be preserved.

I could also design a system which computes any “gaps” that arise between the knowledge and software held by the designated community and those required for execution of the recipe. This is one of the goals of the CASPAR project.

Modeling the Recipe as a Process

Take the first step in decoding and rendering the Web page — turning a sequence of bits into a sequence of UNICODE characters.

The diagram below models this step in a general way, using the Eriksson-Penker UML extensions for business process modeling. The model defines an atomic process “Decode UTF-8″, which takes as input a “Bit Sequence” and generates as output a “UNICODE Character Sequence”.

Ideally, this step would be automated. I.e. a piece of software is used to execute the atomic process. If I understand Eriksson-Penker correctly, we could model this piece of software as a resource, acting as a supply object in the process.

From this model, we can deduce that execution of the recipe depends on a piece of software which is capable of decoding UTF-8. As long as the designated community has a piece of software which can fulfill this contract, then the recipe can be executed, and nothing else need be preserved.

However, what if at some point in the future, the designated community no longer has such a piece of software? The process to decode UTF-8 would have to be done manually, or a new piece of software would have to be written, to execute the recipe. In either case, some information would be required — the UTF-8 encoding standard. We could model this standard as information, acting as a supply object in the process.

Complications — The Roles of Information and Software

A complication now arises, when we want to say, “if you have a UTF-8 decoder, use that, otherwise you’ll need the UTF-8 encoding standard.” In other words, the recipe so far has a dependency on a UTF-8 decoder, and on the UTF-8 encoding standard, but not on both at the same time — either/or will do.

If we don’t capture this either/or clause, then we would end up in difficulty. To realise why, consider the case where the designated community is in possession of a software component that can decode UTF-8, but does not have knowledge of the UTF-8 encoding standard itself. In this case, we would spuriously believe that there was a “gap” in our preservation system, because the UTF-8 encoding standard dependency is not met by the designated community’s knowledge base, when in fact it isn’t needed.

Also, note that there are two possible responses to the situation where the designated community no longer has software to decode UTF-8. We could preserve some software, which can be compiled and/or executed on the DC’s current software platform; or we could preserve the UTF-8 encoding standard.

If we model instead a separate process to construct a UTF-8 decoder, then we can at least compute the priority of the dependencies. We can realise when a vital resource in the network of dependencies is not available, and then can decide between falling back on supplying the information required to construct the missing component, or archiving the component itself.

It’s worth noting that there are at least three types of “dependency” in the discussion so far. There is the dependency of a process on software resources required to execute the process. There is the dependency of a process on information resources which may be required to execute the process. And there is the dependency of one software component or system on another.

Options and Responsibilities — Extending Recipes into the Archive

Finally, say the UNICODE standard falls out of use, and is replaced by XYZ character encoding. If the DC has an XYZ decoder, another option may become available — convert UTF-8 to XYZ. We can model this choice, however note that what we are modeling is no longer strictly a recipe we expect to pass entirely to the recipient — rather, the archive could take responsibility for the conversion step, which is a migration activity — so this model might be a bit confusing, but is interesting to consider.

Previously, in The O.A.I.S. Information Model Revisited – Part 1, I explored the abstract notions of data, information, representation information and interpretation.

I found that the O.A.I.S. notion of interpretation makes most sense when viewed as an act or operation, taking data and representation information as input, yielding new information as output.

In this note, I’d like to explore these ideas further, and see how they related to some real world examples of digital preservation.

“Recipes” for Interpreting Archived Data

In particular, I’m interested in the “recipes” that tell you how to convert a sequence of bits into something more useful.

This is a fundamental requirement for any preservation archive – when retrieving an archived information item, you need the bits that encode that information, but you also need to know how to turn those bits into something else, something you can use.

The O.A.I.S. Information Model acknowledges this, by highlighting the need for representation information, but does it go far enough? Does the model really help us to understand the problems of reconstructing a useful artefact from an archived sequence of bits?

Automation and Virtualisation

Finally, it should be possible, at least to a limited extent, to automate the reconstruction of useful artefacts from sequences of bits. This connects us with the notion of virtualisation, which is not explored in O.A.I.S.

Can the O.A.I.S. Information Model help us to build more automation into the interpretation of archived data?

What does virtualisation mean, in the context of the O.A.I.S. Information Model?

Example: An XHTML Document

Take, for example, a simple Web page, written in English, and encoded as an XHTML 1.0 document using the UTF-8 character set.

Given the encoded sequence of bits, how do we reconstruct the Web page?

One possible recipe is as follows:

1. Obtain a sequence of UNICODE characters from the sequence of bits, using the UTF-8 encoding standard.

2. Obtain an XML document (technically speaking, an XML infoset) from the sequence of UNICODE characters, by parsing according to the XML 1.0 standard.

3. Obtain an XHTML document from the XML infoset, by processing according to the XHTML 1.0 standard and the Document Object Model (level 2).

4. Render the XHTML document to screen, print, voice or other media, following the rendering rules given in the XHTML 1.0 standard.

This isn’t the only recipe we could follow, but it illustrates the point that, however we do it, a number of steps will typically be involved.

What does this mean for my analysis so far?

Blurring the Line Between Data and Information

The first question this example raises is, where does the data end and the information begin?

It’s easy to see that the original sequence of bits is a data object, but what about the sequence of UNICODE characters? The XML infoset? The XHTML document? The Web page rendered on screen? At what point does the data become information?

Until now, we’ve stuck with the O.A.I.S. notion that data interpreted using representation information yields information. But can we say that a sequence of bits can be interpreted as a sequence of UNICODE characters, or that a sequence of UNICODE characters can be interpreted as an XML infoset, or that an XML infoset can be interpreted as an XHTML document?

If we can use “interpretation” in this way, then clearly we have to relax our distinction between data and information – the dividing line between data and information becomes blurred, and doesn’t help us to understand what we mean by interpretation.

Cooking with Representation Information

In the example above, we have a data object – the encoded Web page – and some items of representation information – the UTF-8, XML 1.0, XHTML 1.0 and DOM standards.

However, if an archive were to supply someone, say 20 years in the future, with the data object and these four items of representation information, this wouldn’t be very useful. The person might eventually figure out what order to apply the standards in and what to do with them, from reading and understanding the standards themselves. However, it would be much more useful if the person were also given some sort of a recipe along with the representation information, which tells them what to do with it all – at least in outline.

It would be even better if the person were also given some help in finding and composing some software components to implement the recipe.

Structure and Semantics

The O.A.I.S. Information Model makes a distinction between three types of representation information – structure, semantics, and other.

From section 4.2.1.3.1., the structure information of the representation information describes

“the format, or data structure concepts, which are to be applied to the bit sequences and that in turn result in more meaningful values such as characters, numbers, pixels, arrays, tables, etc.”

From the same section, the semantic information will include

“special meanings associated with all the elements of the Structural Information, operations that may be performed on each data type, and their interrelationships.”

Finally,

“Representation Information contains both Structure Information and Semantic Information, although in some implementations the distinction is subjective .” (My emphasis.)

In our example above, knowing that the Web page is written in English is clearly semantic information, and the UTF-8 encoding standard is clearly structure information; but the categorisation as either structure or semantics is not so clear for the others.

Conclusions?

So, to draw some tentative conclusions, based on a very limited analysis so far …

Given a sequence of bits, some representation information is clearly needed to turn those bits into something useful. But simply providing items of representation information, without any instructions about what to do with them, is not enough.

Indicating that some items of representation information contain structure information, and others contain semantic information, has limited usefulness. It may be useful, because the structure information is typically used before the semantic information, however where there may be several layers of structure and semantics, and we still won’t have all the instructions we need.

Providing a recipe, which specifies a work flow within which various items of representation information are applied, is much more useful. Such a work flow is useful without needing to make any distinction between data and information. Such a work flow is also useful without needing to know whether some item of representation information is about structure or semantics.

This is not to say that the notions of data, information, representation information, structure information and semantic information aren’t useful. They are all useful when it comes to understanding the problem, and planning solutions at a high level.

However, when it comes to designing concrete systems at lower levels of detail, grounded in the practicalities of handling and working with archived data, these notions tend to break down. They don’t stand up to a formal analysis, and are probably not suitable for direct translation into formal specifications or software designs.

Moving Forward – Virtualisation Work Flows

My tentative conclusions suggest that, if we want to provide the kinds of “recipe” for reconstructing archived information I’ve hinted at above, then we need to go beyond the O.A.I.S. Information Model.

Clearly the notion of work flows is important, as is virtualisation. These I hope to explore in further notes.

Agile Development and Rapid Prototyping

I’m doing some rapid prototyping of Web applications and Web services in Java, and because I need to redesign often, I want the design and implementation processes to be as agile and lightweight as possible. I.e. each change I make to the design should take little or no effort to implement & deploy.

Persistent Data

The biggest problem I’ve encountered so far is persistence. I need to implement applications with persistent data. However, I’m coding (at least on the server side) in Java. Implementing persistence for Java applications typically requires an object-relational mapping, which if coded by hand, costs effort. There are alternatives for handling persistence more-or-less automatically, like J2EE’s annotations. However, I’m also very much in favour of coding with POJOs, injecting dependencies wherever possible, and decoupling application code from any implementation-specific considerations to do with persistence.

Data Access Objects (DAOs)

A popular design pattern for persistence is the use of Data Access Objects (DAOs) which abstract the data access and manipulation operations from the implementation-specific details, allowing the persistence platform to be swapped out without affecting application logic.

UML 2 & MDA Transforms

I’ve also been working recently with UML 2 and Model Driven Architecture (MDA) transforms. The tool I’ve been using is Enterprise Architect, which encourages the development of a Platform Independent Model (PIM), which can then be transformed into class models tailored to various programming languages (Java, C# etc.). A PIM can also be transformed into database models tailored to various database platforms (Oracle, Postgres, MySQL etc.). From these platform-specific models, you can then generate actual code or DDL scripts.

The Dream

So the dream is this: I design the data model for my application in a totally platform-independent way, using UML; I then use MDA transforms to generate a database model, a Java class model, and at least the interfaces for a set of Java DAOs (ideally the implementations as well); I then use code generation to automatically generate the DDL scripts and Java class/interface definitions.

I.e. I want to go from a PIM to a fully-implemented persistence layer at the press of one or two buttons.

Oh, and I don’t want to pay any money for any of this

This where I’ve got to so far…

Designing the PIM

Let’s start with a really simple application, for storing and manipulating data about people, organisations, and employment. The PIM class model is below:

Transforming the PIM to a DDL Model

EA has a built-in model transformation for generating a Database Definition (DDL) model. I used this transformation to generate the following DDL model:

Note that I’ve set Postgres as the default database in EA, so this DDL model is tailored for Postgres. The DDL model can easily be converted in EA for other databases e.g. Oracle or MySQL.

Generating DDL Scripts

From the DDL Model I can generate a DDL script for my database of choice.

There is a minor issue here, which is that the DDL script doesn’t contain any useful triggers. I.e. using the DDL script generated from the model above, I cannot delete a person if I haven’t previously deleted any employment associations involving that person (because of the foreign key constraint). Typically I would want a database trigger which would handle these dependencies for me — i.e. if I delete a person, then all of that person’s associations are also deleted. Currently, I have to write these triggers by hand. I haven’t figured out how to get inside EA’s DDL generation code to solve this yet.

Transforming the PIM to a Java Model

Anyway, returning to the PIM, I’m also going to want some Java classes to represent data objects in my application. Using EA’s built-in Java model transform, I generated the following class model from the PIM:

Basically, the transform creates the get/set methods for the attributes and associations in the PIM.

Java Collections and Generics

You’ll notice that the parameter and return types for some of the methods are the class java.util.Set. The class that EA should use for handling collections is configurable, and I’ve set this to the java.util.Set class; however, I’m using Java 5, and I’d like to use generics, but I couldn’t find a way to insert the right generic parameters — so this has to be done by hand.

Also, I’d like to add a field in each of these classes to store the value of the primary key in the underlying database, for use in multiple queries — that *should* be possible by tweaking EA’s built-in Java transform, although I haven’t tried it yet.

Transforming the PIM to a Java DAO Model

Finally, I want to transform the PIM to a model defining at least some DAO interfaces for my data model. For this I had to define an entirely new model transform in EA. Here’s the result of my Java DAO model transform:

As you can see, the typical select, insert, update and delete operations required for each type of entity and association have been generated.

Writing a Custom Model Transform in EA

EA uses a sort of template language for defining model transforms. When you create a new transformation type in EA, it gives you a default template for each type of artefact in the input (File, Namespace, Class, Attribute, Operation etc.) which simply copies everything over from input to output. You can then tweak these to get what you want. For my Java DAO transform, I modified the File template to the following:

Package
{
  name="Java DAO Model"
  namespaceroot="true"
%list="Namespace" @separator="\n\n" @indent="  "%
}

… I modified the Namespace template to:

%if packagePath == ""%
%list="Namespace" @separator="\n"%
%list="Class" @separator="\n"%
%list="Connector" @separator="\n"%
%endTemplate%

Package
{
  %TRANSFORM_CURRENT()%
  %list="Namespace" @separator="\n" @indent="  "%
  %list="Class" @separator="\n" @indent="  "%
  %list="Connector" @separator="\n"%
}

… I modified the Class template to:

%if elemType != "Class"%
%endTemplate%

Interface
{
  %TRANSFORM_REFERENCE()%
  name=%qt%%className%Dao%qt%
  language="Java"

  %if elemType != "Association"%

  $COMMENT="handle class"

  $COMMENT="count operation"
  Operation
  {
    name=%qt%count%className%s%qt%
    scope="public"
    type="int"
  }

  $COMMENT="select operation"
  Operation
  {
    name=%qt%select%className%%qt%
    scope="public"
    type=%qt%%className%%qt%
    Parameter
    {
      type="int"
      name=%qt%%CONVERT_NAME(className, "Pascal Case","Camel Case")%ID%qt%
    }
  }

  $COMMENT="select all operation"
  Operation
  {
    name=%qt%selectAll%className%s%qt%
    scope="public"
    type=%qt%java.util.Set%qt%
  }

  $COMMENT="insert operation"
  Operation
  {
    name=%qt%insert%className%%qt%
    scope="public"
    type="void"
    Parameter
    {
      type=%qt%%className%%qt%
      name=%qt%%CONVERT_NAME(className, "Pascal Case","Camel Case")%%qt%
    }
  }

  $COMMENT="update operation"
  Operation
  {
  	name=%qt%update%className%%qt%
  	scope="public"
  	type="void"
    Parameter
    {
      type="int"
      name=%qt%%CONVERT_NAME(className, "Pascal Case","Camel Case")%ID%qt%
    }
    Parameter
    {
      type=%qt%%className%%qt%
      name=%qt%%CONVERT_NAME(className, "Pascal Case","Camel Case")%%qt%
    }
  }

  $COMMENT="delete operation"
  Operation
  {
  	name=%qt%delete%className%%qt%
  	scope="public"
  	type="void"
    Parameter
    {
      type="int"
      name=%qt%%CONVERT_NAME(className, "Pascal Case","Camel Case")%ID%qt%
    }
  }

  %endIf%

}

… and finally, I modified the Connector template to:

Interface
{
  %TRANSFORM_REFERENCE()%
  name=%qt%%connectorName%Dao%qt%
  language="Java"

  $COMMENT="insert operation"
  Operation
  {
    name=%qt%insert%connectorName%%qt%
    scope="public"
    type="void"
    Parameter
    {
      type="int"
      name=%qt%%CONVERT_NAME(connectorSourceRole, "Pascal Case","Camel Case")%ID%qt%
    }
    Parameter
    {
      type="int"
      name=%qt%%CONVERT_NAME(connectorDestRole, "Pascal Case","Camel Case")%ID%qt%
    }
  }

  $COMMENT="delete operation"
  Operation
  {
    name=%qt%delete%connectorName%%qt%
    scope="public"
    type="void"
    Parameter
    {
      type="int"
      name=%qt%%CONVERT_NAME(connectorSourceRole, "Pascal Case","Camel Case")%ID%qt%
    }
    Parameter
    {
      type="int"
      name=%qt%%CONVERT_NAME(connectorDestRole, "Pascal Case","Camel Case")%ID%qt%
    }
  }

  $COMMENT="select source role"
  Operation
  {
    name=%qt%select%connectorSourceElemName%s%qt%
    scope="public"
    type=%qt%java.util.Set%qt%
    Parameter
    {
      type="int"
      name=%qt%%CONVERT_NAME(connectorDestRole, "Pascal Case","Camel Case")%ID%qt%
    }
  }

  $COMMENT="count source role"
  Operation
  {
    name=%qt%count%connectorSourceElemName%s%qt%
    scope="public"
    type="int"
    Parameter
    {
      type="int"
      name=%qt%%CONVERT_NAME(connectorDestRole, "Pascal Case","Camel Case")%ID%qt%
    }
  }

  $COMMENT="select dest role"
  Operation
  {
    name=%qt%select%connectorDestElemName%s%qt%
    scope="public"
    type=%qt%java.util.Set%qt%
    Parameter
    {
      type="int"
      name=%qt%%CONVERT_NAME(connectorSourceRole, "Pascal Case","Camel Case")%ID%qt%
    }
  }

  $COMMENT="count dest role"
  Operation
  {
    name=%qt%count%connectorDestElemName%s%qt%
    scope="public"
    type="int"
    Parameter
    {
      type="int"
      name=%qt%%CONVERT_NAME(connectorSourceRole, "Pascal Case","Camel Case")%ID%qt%
    }
  }
}

Everything else I left the same.

Living the Dream?

So as you’ve probably realised, I’m not quite living the dream, but I’m getting there

There are still places where I have to do some work to get from PIM to implementation. These include defining triggers to handle deletion dependencies in the database, and handling generics in the Java classes.

Of course, I still have to write implementations of my DAO interfaces by hand. However, it *should* be possible to either create another UML model transform which will do that automatically for a specific persitence platform, using the initial code sections for each DAO class, or create some code generation templates (or both). I’ve got a plan to do this for Spring’s JDBC classes, but I haven’t got round to it yet …

Agile Development and Rapid Prototyping

I’m doing some rapid prototyping of Web applications and Web services in Java, and because I need to redesign often, I want the design and implementation processes to be as agile and lightweight as possible. I.e. each change I make to the design should take little or no effort to implement & deploy.

Persistent Data

The biggest problem I’ve encountered so far is persistence. I need to implement applications with persistent data. However, I’m coding (at least on the server side) in Java. Implementing persistence for Java applications typically requires an object-relational mapping, which if coded by hand, costs effort. There are alternatives for handling persistence more-or-less automatically, like J2EE’s annotations. However, I’m also very much in favour of coding with POJOs, injecting dependencies wherever possible, and decoupling application code from any implementation-specific considerations to do with persistence.

Data Access Objects (DAOs)

A popular design pattern for persistence is the use of Data Access Objects (DAOs) which abstract the data access and manipulation operations from the implementation-specific details, allowing the persistence platform to be swapped out without affecting application logic.

UML 2 & MDA Transforms

I’ve also been working recently with UML 2 and Model Driven Architecture (MDA) transforms. The tool I’ve been using is Enterprise Architect, which encourages the development of a Platform Independent Model (PIM), which can then be transformed into class models tailored to various programming languages (Java, C# etc.). A PIM can also be transformed into database models tailored to various database platforms (Oracle, Postgres, MySQL etc.). From these platform-specific models, you can then generate actual code or DDL scripts.

The Dream

So the dream is this: I design the data model for my application in a totally platform-independent way, using UML; I then use MDA transforms to generate a database model, a Java class model, and at least the interfaces for a set of Java DAOs (ideally the implementations as well); I then use code generation to automatically generate the DDL scripts and Java class/interface definitions.

I.e. I want to go from a PIM to a fully-implemented persistence layer at the press of one or two buttons.

Oh, and I don’t want to pay any money for any of this

This where I’ve got to so far…

Designing the PIM

Let’s start with a really simple application, for storing and manipulating data about people, organisations, and employment. The PIM class model is below:

Transforming the PIM to a DDL Model

EA has a built-in model transformation for generating a Database Definition (DDL) model. I used this transformation to generate the following DDL model:

Note that I’ve set Postgres as the default database in EA, so this DDL model is tailored for Postgres. The DDL model can easily be converted in EA for other databases e.g. Oracle or MySQL.

Generating DDL Scripts

From the DDL Model I can generate a DDL script for my database of choice.

There is a minor issue here, which is that the DDL script doesn’t contain any useful triggers. I.e. using the DDL script generated from the model above, I cannot delete a person if I haven’t previously deleted any employment associations involving that person (because of the foreign key constraint). Typically I would want a database trigger which would handle these dependencies for me — i.e. if I delete a person, then all of that person’s associations are also deleted. Currently, I have to write these triggers by hand. I haven’t figured out how to get inside EA’s DDL generation code to solve this yet.

Transforming the PIM to a Java Model

Anyway, returning to the PIM, I’m also going to want some Java classes to represent data objects in my application. Using EA’s built-in Java model transform, I generated the following class model from the PIM:

Basically, the transform creates the get/set methods for the attributes and associations in the PIM.

Java Collections and Generics

You’ll notice that the parameter and return types for some of the methods are the class java.util.Set. The class that EA should use for handling collections is configurable, and I’ve set this to the java.util.Set class; however, I’m using Java 5, and I’d like to use generics, but I couldn’t find a way to insert the right generic parameters — so this has to be done by hand.

Also, I’d like to add a field in each of these classes to store the value of the primary key in the underlying database, for use in multiple queries — that *should* be possible by tweaking EA’s built-in Java transform, although I haven’t tried it yet.

Transforming the PIM to a Java DAO Model

Finally, I want to transform the PIM to a model defining at least some DAO interfaces for my data model. For this I had to define an entirely new model transform in EA. Here’s the result of my Java DAO model transform:

As you can see, the typical select, insert, update and delete operations required for each type of entity and association have been generated.

Writing a Custom Model Transform in EA

EA uses a sort of template language for defining model transforms. When you create a new transformation type in EA, it gives you a default template for each type of artefact in the input (File, Namespace, Class, Attribute, Operation etc.) which simply copies everything over from input to output. You can then tweak these to get what you want. For my Java DAO transform, I modified the File template to the following:

Package
{
  name="Java DAO Model"
  namespaceroot="true"
%list="Namespace" @separator="\n\n" @indent="  "%
}

… I modified the Namespace template to:

%if packagePath == ""%
%list="Namespace" @separator="\n"%
%list="Class" @separator="\n"%
%list="Connector" @separator="\n"%
%endTemplate%

Package
{
  %TRANSFORM_CURRENT()%
  %list="Namespace" @separator="\n" @indent="  "%
  %list="Class" @separator="\n" @indent="  "%
  %list="Connector" @separator="\n"%
}

… I modified the Class template to:

%if elemType != "Class"%
%endTemplate%

Interface
{
  %TRANSFORM_REFERENCE()%
  name=%qt%%className%Dao%qt%
  language="Java"

  %if elemType != "Association"%

  $COMMENT="handle class"

  $COMMENT="count operation"
  Operation
  {
    name=%qt%count%className%s%qt%
    scope="public"
    type="int"
  }

  $COMMENT="select operation"
  Operation
  {
    name=%qt%select%className%%qt%
    scope="public"
    type=%qt%%className%%qt%
    Parameter
    {
      type="int"
      name=%qt%%CONVERT_NAME(className, "Pascal Case","Camel Case")%ID%qt%
    }
  }

  $COMMENT="select all operation"
  Operation
  {
    name=%qt%selectAll%className%s%qt%
    scope="public"
    type=%qt%java.util.Set%qt%
  }

  $COMMENT="insert operation"
  Operation
  {
    name=%qt%insert%className%%qt%
    scope="public"
    type="void"
    Parameter
    {
      type=%qt%%className%%qt%
      name=%qt%%CONVERT_NAME(className, "Pascal Case","Camel Case")%%qt%
    }
  }

  $COMMENT="update operation"
  Operation
  {
  	name=%qt%update%className%%qt%
  	scope="public"
  	type="void"
    Parameter
    {
      type="int"
      name=%qt%%CONVERT_NAME(className, "Pascal Case","Camel Case")%ID%qt%
    }
    Parameter
    {
      type=%qt%%className%%qt%
      name=%qt%%CONVERT_NAME(className, "Pascal Case","Camel Case")%%qt%
    }
  }

  $COMMENT="delete operation"
  Operation
  {
  	name=%qt%delete%className%%qt%
  	scope="public"
  	type="void"
    Parameter
    {
      type="int"
      name=%qt%%CONVERT_NAME(className, "Pascal Case","Camel Case")%ID%qt%
    }
  }

  %endIf%

}

… and finally, I modified the Connector template to:

Interface
{
  %TRANSFORM_REFERENCE()%
  name=%qt%%connectorName%Dao%qt%
  language="Java"

  $COMMENT="insert operation"
  Operation
  {
    name=%qt%insert%connectorName%%qt%
    scope="public"
    type="void"
    Parameter
    {
      type="int"
      name=%qt%%CONVERT_NAME(connectorSourceRole, "Pascal Case","Camel Case")%ID%qt%
    }
    Parameter
    {
      type="int"
      name=%qt%%CONVERT_NAME(connectorDestRole, "Pascal Case","Camel Case")%ID%qt%
    }
  }

  $COMMENT="delete operation"
  Operation
  {
    name=%qt%delete%connectorName%%qt%
    scope="public"
    type="void"
    Parameter
    {
      type="int"
      name=%qt%%CONVERT_NAME(connectorSourceRole, "Pascal Case","Camel Case")%ID%qt%
    }
    Parameter
    {
      type="int"
      name=%qt%%CONVERT_NAME(connectorDestRole, "Pascal Case","Camel Case")%ID%qt%
    }
  }

  $COMMENT="select source role"
  Operation
  {
    name=%qt%select%connectorSourceElemName%s%qt%
    scope="public"
    type=%qt%java.util.Set%qt%
    Parameter
    {
      type="int"
      name=%qt%%CONVERT_NAME(connectorDestRole, "Pascal Case","Camel Case")%ID%qt%
    }
  }

  $COMMENT="count source role"
  Operation
  {
    name=%qt%count%connectorSourceElemName%s%qt%
    scope="public"
    type="int"
    Parameter
    {
      type="int"
      name=%qt%%CONVERT_NAME(connectorDestRole, "Pascal Case","Camel Case")%ID%qt%
    }
  }

  $COMMENT="select dest role"
  Operation
  {
    name=%qt%select%connectorDestElemName%s%qt%
    scope="public"
    type=%qt%java.util.Set%qt%
    Parameter
    {
      type="int"
      name=%qt%%CONVERT_NAME(connectorSourceRole, "Pascal Case","Camel Case")%ID%qt%
    }
  }

  $COMMENT="count dest role"
  Operation
  {
    name=%qt%count%connectorDestElemName%s%qt%
    scope="public"
    type="int"
    Parameter
    {
      type="int"
      name=%qt%%CONVERT_NAME(connectorSourceRole, "Pascal Case","Camel Case")%ID%qt%
    }
  }
}

Everything else I left the same.

Living the Dream?

So as you’ve probably realised, I’m not quite living the dream, but I’m getting there

There are still places where I have to do some work to get from PIM to implementation. These include defining triggers to handle deletion dependencies in the database, and handling generics in the Java classes.

Of course, I still have to write implementations of my DAO interfaces by hand. However, it *should* be possible to either create another UML model transform which will do that automatically for a specific persitence platform, using the initial code sections for each DAO class, or create some code generation templates (or both). I’ve got a plan to do this for Spring’s JDBC classes, but I haven’t got round to it yet …

Introduction & Motivation

The Reference Model for an Open Archival Information System (OAIS) is an influential standard in the digital preservation domain. It contains an information model, which lays out some basic ideas about digital information, how it is encoded, interpreted and packaged. It also contains a functional model, which lays out the main functional components that should be present in a digital preservation system.

The CASPAR Project is currently designing and implementing software components for a distributed infrastructure to support digital preservation. The starting point for the design of these components is the OAIS reference model, and in particular, the OAIS information model.

This note captures some initial thoughts on the OAIS information model, working towards answers to the following questions:

1. Does the OAIS Information Model make sense?
2. Can it be used as the basis for designing software components, within a UML model-driven software engineering process?

This is work in progress, so don’t expect a faultless analysis!

The OAIS Information Model

The OAIS information model introduces the notion of representation information, and its key role in providing the additional information needed to yield usable information from data.

Section 2.2 of the OAIS reference model provides a preliminary definition of information, and of information packages. Section 4.2.1 then provides a more formal, “logical model” for archival information, using UML class diagrams.

Section 2.2 states that, “Data interpreted using its Representation Information yields Information,” which is illustrated informally in figure 2-2.

The distinction between data and information is made to emphasise the point that the preservation of information content is the primary goal of a preservation system. I.e. it is not good enough to simply preserve sequences of bits — it must be possible to derive at least some usable information from those bits at some point in the future. To be able to derive information from data, some representation information is required, which specifies how that data should be interpreted.

The relationships between data, information and representation information are then expressed in more detail in section 4.2.1, and in particular in the UML class diagram given in figure 4-10. An equivalent class diagram is shown below.

Data, Information and Interpretation

There is clearly a need to distinguish representation information as an essential component in the preservation of digitally encoded information, and this need is clearly expressed in OAIS sections 2.2 and 4.2.1.

However, there are a number of ambiguities and inconsistencies both within and between these two sections of the OAIS reference model, which could hinder the use of the OAIS information model as a robust starting point for a software design process.

Interpretation as an Act

Let us start again from the informal statement that, “Data interpreted using its Representation Information yields Information.”

This statement does not seem to suggest that, crudely stated, data + representation information = information. Rather, it suggests that, if you have some data and some representation information, then you will be capable of interpreting that data to yield some information. I.e. an act of interpretation is required in order to yield information from data using representation information.

To put this another way, it suggests that representation information is involved in some sort of transformation or function which maps data onto information.

We could illustrate this intuition using a UML activity diagram, for example:

Representation Information as Information

A question that naturally arises at this stage is, what is representation information? Is it information, or is it data? The name suggests that it is information — which could therefore have been obtained by the interpretation of some other data. In other words, OAIS section 2.2 suggests that there is no fundamental distinction between information and representation information — one person’s information could be another person’s representation information.

We could extend the UML activity diagram above to illustrate this notion of representation information as information, being itself derived from the interpretation of some other data:

This diagram could obviously be extended upwards or downwards in the same way forever — clearly illustrating the basic idea from the OAIS reference model that representation information itself depends on other representation information, leading to a network of dependencies between representation information.

The Components of Information

Now consider the statement made in section 4.2.1.1 that, “the Information Object is composed of a Data Object [...] and the Representation Information that allows for the full interpretation of the data into meaningful information,” (my emphasis). This statement is reflected in figure 4-10 by the aggregation associations between the classes Information Object, Data Object and Representation Information.

From Enterprise Architect’s UML help:

An aggregation relationship is a type of association that shows that an element contains or is composed of other elements. Used in Class models to show how more complex elements (aggregates) are built from a collection of simpler elements (component parts; eg. a car from wheels, tires, motor and so on).

Thus, it seems that section 4.2.1 makes a fairly vague statement that information is composed of data and information, whereas section 2.2 makes the stronger statement that information can be derived from data using representation information (via an act of interpretation). While these two statements are not necessarily inconsistent, it is clear that the UML class diagram in figure 4-10 does not fully capture our intuitions about data and information from section 2.2.

The aggregation association is typically used to indicate a part-whole relationship between two classes. However, the idea that data and representation information are the parts of information seems quite different from the idea that information is obtained (derived) from the interpretation of data (by some act).

The idea that information contains data seems even less intuitive.

Explicitly Modeling the Interpretation of Data

There is another ambiguity and potential inconsistency between the informal definitions in section 2.2 and the more detailed information model in section 4.2.1.

Let us assume that representation information is information (and not data). According to figure 4-10, data is interpreted using representation information. However, representation information is also interpreted using representation information. There are clearly two completely different notions of interpretation here. In the first, data can be interpreted using representation information; in the second, representation information is itself the result of interpretation of some other data using some other representation information.

On a closely related point, the cardinality of representation information in the interpretation of data is also ambiguous. In figure 4-10, data is interpreted using exactly one item of representation information; however representation information is interpreted using any number of items of other representation information. Either there are two completely separate notions of interpretation here, or there is an inconsistency.

We can resolve these issues by replacing the UML class diagram of figure 4-10 with a different class model, in which the notion of an interpretation (of some data, using some representation information) is made explicit.

There are a number of alternatives here. One way of making the notion of an interpretation explicit is to take a purely static view, and define a class (Interpretation):

Note the following points about this class diagram:

• An instance of Interpretation involves exactly one instance of Data, one or more instances of Information playing the role of representation information, and yields exactly one instance of Information.
• An instance of Information is associated with exactly one instance of Interpretation (by which it was obtained).
• An instance of Data is associated with any number of Interpretations.
• An instance of Information can play the role of representation information in any number of Interpretations.

A second alternative is to take an entirely behavioural view of interpretation, and define an interface (Interpreter) which exposes an interpret operation:

Note that, informally speaking, the interpret operation takes as input an item of data and a number of items of information (as representation information), and returns a new item of information.

A third alternative is to take a hybrid view, and view an Interpretation as the result of an interpret operation:

Note in this model, the interpret operation returns an instance of Interpretation, which has an association back to the Interpreter which created it.

This third option is probably the most useful, from the point of view of defining software components that can e.g. automatically interpret some data according to machine-processable representation information (e.g. EAST specifications, DEDSL data dictionaries etc.) … however, I’ll try to explore that in more detail in further notes.