If you want to know more about the technical side of the work, see the paper “OpenFlyData: An exemplar data web integrating gene expression data on the fruit fly Drosophila melanogaster” in the Journal of Biomedical Informatics.

Integrating Gene Expression Data

We wanted to help reduce the amount of time spent by a Drosophila functional genetics research group on experimental design and on validating experimental results. Experimental design includes selecting genes that might be relevant to the biological function of interest (in this case, male fertility). Validating results includes checking your gene expression data against other published data for the same gene – a discrepancy suggests an artifact or problem in your data, or in the published data … either way it’s likely to be important.

The problem here is a common refrain – the relevant data are not all found in the same place. Trawling and querying multiple sites and manually compiling the results takes a lot of time. Could we build some tools that help bring these data together?

Technology Hypothesis – Data Webs, Semantic Web & SPARQL

We wanted to be as user-driven as possible, i.e., to stay focused on what the researchers needed, and to be open-minded about technology, using whatever tools made us most productive.

But we did have a technology hypothesis, which was part the reason why JISC funded the FlyWeb project. Our hypothesis was that building a data integration solution for our Drosophila researchers using Semantic Web standards and open source software would be (a) feasible, and (b) reasonably cost-efficient. David Shotton and Graham Klyne, the original proposal authors, had also previously developed a vision for “data webs“, an architectural pattern for integrating a set of biological data within a specific domain or for a specific purpose. Would the data webs pattern help us to build a solution?

So how did we marry these two forces: on the one hand being user-driven, on the other having a technology hypothesis that we wanted to test?

Well, in the spirit of agile, we tried to make our development iterations as short as possible. I.e., we tried to work in a way that meant we had something to put in front of users in the shortest possible time frame. When we were discussing architectural patterns and technologies, and several alternatives looked to be of similar complexity or difficulty, we favoured approaches involving RDF, OWL or SPARQL, and that were closer to the original data webs vision.

However, our goal was not to prove that a solution based on semweb standards and tech was any cheaper or better than a non-semweb alternative; just that it was possible and not prohibitively expensive. This is interesting because, if data integration solutions in related problem domains were all based on semweb standards, then they also might play together, as well as solving their own particular problems … or so the argument goes. I.e., there would be some re-use or re-purposing benefit from each individual data integration solution, and maybe some network effect, if everyone used SPARQL, for example. Of course there would be work involved in linking two data web solutions, but it might be less because at least we’d bottom out at the same standards – RDF, SPARQL, and maybe even some shared ontologies.

But you can’t even begin to talk about network effects if you can’t first show that you can solve specific problems effectively and cheaply. I.e., solutions need to make sense locally.

SPARQL Mashups

An architectural pattern that we adopted early in the project was the “SPARQL mashups” pattern. A SPARQL mashup is an HTML+JavaScript application that runs entirely in the browser, and that retrieves data directly from two or more SPARQL endpoints via the SPARQL protocol.

To see a SPARQL mashup in action, go to http://openflydata.org/search/gene-expression, click the “show logger” link at the bottom of the page (or open firebug if you’re in firefox), then type “schuy” into the search box. You should see SPARQL queries being sent to various SPARQL endpoints and result sets being returned in the JSON SPARQL results format.

For example, here’s the query that finds genes from FlyBase matching the query term “aly”:

# Select feature short name, unique name, annotation ID, and official full name, given
# any label and where feature is D. melanogaster gene.

PREFIX xsd:
PREFIX chado:
PREFIX skos:
PREFIX so:
PREFIX syntype: 

SELECT DISTINCT ?uniquename ?name ?accession ?fullname WHERE {

  ?feature skos:altLabel "aly" ;
    a so:SO_0000704 ;
    chado:organism  ;
    chado:uniquename ?uniquename ;
    chado:name ?name ;
    chado:feature_dbxref [
      chado:accession ?accession ;
      chado:db <http://openflydata.org/id/flybase/db/FlyBase_Annotation_IDs>
    ] .

  OPTIONAL {
    ?fs
      chado:feature ?feature ;
      chado:is_current "true"^^xsd:boolean ;
      chado:synonym [
        a syntype:FullName ;
        chado:name ?fullname ;
      ] ;
      a chado:Feature_Synonym .
  }

}

Each of the panels in the UI corresponds (more-or-less) to a data source. The search term is first used in a SPARQL query to the FlyBase endpoint to find matching genes. If there is only a single gene matching the query, the gene is automatically selected, and further SPARQL queries are then sent to other data sources (e.g., FlyAtlas, BDGP, Fly-TED) to retrieve gene expression data relevant to that gene. If more than one gene matches the query (e.g., try “aly”) the user has to select a gene before the next set of queries are dispatched.

Why did we use the SPARQL mashup pattern?

Well, it allowed us to use some off-the-shelf open source software. All we had to do was code a transformation from the data in its published format to RDF. Once we had an RDF dump for each data source, we loaded the data into a triple store (we used Jena TDB) then deployed the store as a SPARQL endpoint via a SPARQL protocol server (we used Joseki initially, then SPARQLite).

Once we had a SPARQL endpoint for each data source, we could develop a simple HTML+JavaScript application in the spirit of a conventional mashup, using the SPARQL protocol as the API to the data.

A nice feature of using SPARQL here is that you don’t have to think about the API to the data, at least not from a web service point of view. The SPARQL protocol and query language basically give you an API for free. All you have to figure out is what query you need to get the right data. And you don’t need to write any code on the server side, other than that required to transform your data to RDF.

Also, because your API supports a query language (SPARQL), you don’t need to know up-front exactly what data you need or what questions you’re going to ask (although obviously it helps to have a rough idea). I.e., if you get half-way through coding your mashup and realise you need to query the data in a different way, or retrieve more or less data, you just tweak the SPARQL query you’re sending. I.e., there are no consequences for your server-side code, your API can already handle it.

This also means your API can handle unanticipated use cases. I.e., if someone else wants to query the data for a completely different purpose, chances are they can already do it – the expressiveness of SPARQL means that the chances others will be able to use your data are high. Although this wasn’t a motivation in our project, we liked the idea.

Dealing With Big(-ish) RDF Data

As we scaled up from initial prototypes, we hit a few snags. The biggest challenge was dealing with the FlyBase data, which amounted to about 180 million triples in our final dump. Also, queries had to be quick, because users of the mashup apps are waiting for SPARQL queries to evaluate in real time. Here’s a few tricks we found for working with RDF data at this scale.

• Fast data loading – For data loading, we found we could get between 15,000 and 30,000 triples per second from Jena TDB on a 64-bit platform. That meant the FlyBase dataset loaded in somewhere between 1.5 and 3 hours. To load the data, we fired up a large EC2 instance, and loaded the data onto an EBS volume. When the load was done, we detached the volume and attached it to a small instance which hosted the query endpoint, and shut down the large instance to keep running costs down. We didn’t try this, but using a RAID 0 array and striping your data across multiple EBS volumes might increase load performance even further (there’s a nice article by Eric Hammond on using RAID 0 on EC2).
• Everything has to be streaming – The transformation from source format (e.g., relational database) to RDF has to be streaming. The SPARQL query engine has to be streaming. And the SPARQL protocol implementation has to be streaming. That’s part of why we rolled our own SPARQL protocol implementation in the end (SPARQLite) – Joseki at the time did not write result sets in a streaming fashion, for valid reasons, but which limits scalability.
• To get good query performance we pre-calculated some data. E.g., when we wanted to do a case-insensitive match against RDF literals in a query pattern, we computed the lower-case version of the literal and added it to the data as extra triples, then wrote queries with literals in lower case too – rather than, say, using a regex filter. SPARQL queries go much faster when they have a concrete literal or URI node to work from early in the query; queries with loose patterns and FILTERs can be very slow, because you’re pushing a lot of triples through the filters. We did also try using the SPARQL-lucene integration (LARQ) for text matches, but couldn’t get this to quite fast enough (sub 3s) for the FlyBase gene name queries, although it was used heavily in some other projects (CLAROS and MILARQ). You can also make queries go faster by shortening query paths. E.g., if you have a pattern you want to query like { ?x :p ?y . ?y :q “foo”. } your query may go faster if you first invent a new predicate :r and compute some new triples via a rule or query like CONSTRUCT { ?x :r ?z } WHERE { ?x :p ?y . ?y :q ?z. }, then add these triples to your dataset and query using the pattern { ?x :r “foo” } instead.
• Beware that how you write your query may make a difference. Depending on which optimiser you use, TDB will do some re-ordering of the query to make it go faster (I believe to put more selective bits earlier), but if you know your data well (statistics are helpful) then writing the query with this in mind can help the query engine. E.g., if you have a triple pattern with a specific predicate and a specific subject or object that you know should only have a few matches, put this right at the top of the query. Basically, put the most discriminating parts of the query as early as possible. This also means that often triple patterns with rdf:type are not that helpful early on, because they don’t narrow down the results much, although this is what you tend to put first for readability.
• Test-driven data – When you generate a large RDF dataset, you need to be sure you got the transformation right, and the data is as you expect it to be, otherwise you can waste a lot of time. I.e., you need to be able to test your triples. We designed some simple test harnesses for our data, where a set of test SPARQL queries were run against the data. Each SPARQL query was an ASK or SELECT and the test case defined an expectation for each query result. For very large datasets, you may also want to code some sanity checks on the n-triples dump before trying to load into a triplestore and test with SPARQL, e.g., scanning with grep and/or awk to find triples with predicates you expect to be there.

Open SPARQL Endpoints – Mitigating Denial of Service

Above I mildly extolled the virtues of SPARQL as an API to data – anyone can write the query they need to extract the data they want, and you don’t need to anticipate all requirements a priori.

The obvious downside to the expressiveness of SPARQL and openness of SPARQL endpoints is that they are vulnerable to accidental or intential denial of service attacks. I.e., someone can write a hard query and tie up your query engine’s compute and/or memory resources, if not crash your box.

Although deploying a production service or guaranteeing service levels wasn’t part of our remit, we were concerned that unless we could mitigate this vulnerability, SPARQL outside the firewall would never really be useful beyond a proof-of-concept. I.e., we would never be able to advertise our endpoints as a production web service, so that others could write mashups or other applications that query the data and depend on the service.

We spent a bit of time working on this, and this may be a solved problem now in newer query engines, but at the time we thought to place some limits on the queries that open endpoints would accept. For example, SPARQLite endpoints could be configured to disallow queries with triple patterns with variable predicates, or FILTER or OPTIONAL clauses, or to enforce a LIMIT on all queries’ result sets. This is not a complete solution, because you could still write hard queries, but at least it removed some of the obvious attacks. A better solution would probably involve monitoring queries’ resource usage and killing any that take too long or consume too much resources – a bit like how Amazon’s SimpleDB places limits on service usage, including a 5 second maximum query execution time.

Mapping to RDF

The elephant in the room here is mapping the data to RDF, and that’s where a lot of the work went. All of our data sources came in some non-RDF format, either as CSV files or a relational database. For the CSV sources we hand-coded RDF transformations as Python scripts. For the relational databases, we made heavy use of D2RQ, although we did not use D2R server to transform SPARQL queries to SQL on-the-fly due to performance and scalability issues, rather we used the D2R dump utility to generate a complete RDF dump of an SQL datasource in n-triples format, then loaded that into a Jena TDB triplestore which backed our SPARQL endpoints.

The main issue was the time it takes to design a mapping from a fairly complex relational schema like Chado to RDF. Rather than trying to find one or more existing, published, ontologies to use in the RDF outputs of the mapping, and designing the mappings by hand, we tried a different approach. Inspired by model-driven engineering, we developed a Python utility which, driven by some simple annotations on the source SQL schema definition, generated both a suitable OWL ontology and a complete D2RQ mapping file. This worked well with a schema like Chado which has consistent structural patterns and naming conventions. There’s a worked example in the supplementary information (S4) to the OpenFlyData paper in the Journal of Biomedical Informatics.

The problem with this approach is, of course, that you end up with one schema/ontology per data source. Initially we thought this would force us to do some ontology alignment and to map everything to a common ontology, but we quickly realised this just wasn’t necessary. The mashup applications quite happily query each source according to its own ontology, and have just enough knowledge of what each ontology means to integrate the results in a sensible way. I.e., you can develop applications that work with multiple data sources without perfect (or even partial) ontology alignment. Obviously, aligning ontologies is desirable, but that can be a long-term ambition – using ontologies derived from the source data at least gets you started, and gets you talking about data semantics rather than getting bogged down by differences in syntax, formats or protocol (because RDF and SPARQL are the interlingua for these).

Lasting Impressions

The message I took away from this project is that, if you already have some data, and you want to make the data available to web application developers and other hackers in a useful way, then SPARQL can be a good option. It’s fairly straightforward (even, dare I say, fun) to code simple HTML+JavaScript mashups that bring data from different SPARQL endpoints together on-the-fly (pardon the pun). SPARQL won’t be a panacea, and you may find some queries just aren’t quick enough to evaluate in real time, so you may have to find ways to optimise these queries when moving to production, but it’s worth doing some benchmarking, as triplestores like Jena TDB are quick for certain types of query.

The pain comes when you need to convert data to RDF. But you don’t need to get hung up on finding the right ontologies or designing a perfect or even complete mapping. Convert what you need, using a custom ontology that is designed for your application or generated from the source data, and just get going – you’ll have plenty of iterations to refactor the data.

Would I use SPARQL again? Yes, for read-only data services and data integration webapps, I’d definitely consider it. And there are some new features coming in SPARQL 1.1 which look very useful. If someone solves the denial-of-service problem for open SPARQL endpoints (and they may already have) then the case for SPARQL as a data-sharing standard is compelling. Certainly an area to watch.

For reference, I’m using Ubuntu 10.04 with all software installed via APT (apache2, mysql-server, libapache2-mod-auth-mysql).

To install mod_auth_mysql…

$ sudo apt-get install libapache2-mod-auth-mysql
$ sudo a2enmod auth_mysql

The biggest gotcha is that the configuration documentation for mod_auth_mysql is badly out of date. There have been some substantial changes to the configuration parameter names since that was written, although I could not find any definitive documentation of the new configuration parameters. There are a couple of other gotchas in there too, I’ll come to those in a minute.

Before configuring Apache, I set up a test database of usernames and passwords. This is what I did…

$ mysql -uroot -p
mysql> grant all on auth.* to auth_user@localhost identified by 'XXX';
mysql> flush privileges;
mysql> create database auth;
mysql> use auth;
CREATE TABLE user_info ( user_name CHAR(100) NOT NULL, user_passwd CHAR(100) NOT NULL, PRIMARY KEY (user_name) );
INSERT INTO `user_info` VALUES ('test', MD5('test'));
CREATE TABLE user_group ( user_name char(100) NOT NULL, user_group char(100) NOT NULL, PRIMARY KEY (user_name,user_group) );
INSERT INTO `user_group` VALUES ('test', 'test-group');

Note the length of the user_password field. 100 characters is probably more than needed, but you will definitely need more than the 20 characters suggested in some documentation if you want to use a password hash like MD5. (If the field is too short, then password hashes will get truncated when they’re inserted into the database.)

Then I configured mod_auth_mysql to authenticate users for my whole domain. In the appropriate virtual host configuration file (e.g., /etc/apache2/sites-enabled/000-default) I added the following…

<Location />

# these lines force authentication to fall through to mod_auth_mysql
AuthBasicAuthoritative Off
AuthUserFile /dev/null

# begin auth_mysql configuration
AuthMySQL On
AuthMySQL_Host localhost
AuthMySQL_User auth_user
AuthMySQL_Password XXXX
AuthMySQL_DB auth
AuthMySQL_Password_Table user_info
AuthMySQL_Username_Field user_name
AuthMySQL_Password_Field user_passwd
AuthMySQL_Empty_Passwords Off
AuthMySQL_Encryption_Types PHP_MD5
AuthMySQL_Authoritative On
#AuthMySQL_Non_Persistent Off
#AuthMySQL_Group_Table user_group
#AuthMySQL_Group_Field user_group

# generic auth configuration
AuthType Basic
AuthName "auth_mysql test"
Require valid-user

</Location>

Note the “PHP_MD5″ encryption type. (Some of the documented encryption types don’t seem to be available, e.g., “MD5″.)

Then…

$ sudo apache2ctl -t # check syntax
$ sudo apache2ctl restart

Then when browsing to the host, I get an authentication challenge, and can log in with username “test” and password “test”.

Using mod_authn_dbd Instead

There is another way to get Apache to use a relational database to look up usernames and passwords when authenticating – mod_authn_dbd. That module seems more current and has up-to-date documentation, see e.g., the Apache 2.2 mod_authn_dbd module docs and the Apache 2.2 docs on password encryption.

Note however that you cannot use normal MD5 encryption to store passwords in the database with this module. If you want to use MD5 you have to use the special Apache MD5 algorithm.

Also note that to get this working with MySQL you will need to install the MySQL driver for DBD, which you can do via APT:

$ sudo apt-get install libaprutil1-dbd-mysql

If you get a message like “DBD: Can’t load driver file apr_dbd_mysql.so” then this is what you need to do – don’t believe the articles that tell you you need to recompile APR

Centre for Genomics and Global Health

The Centre for Genomics and Global Health (CGGH), a joint research programme of Oxford University and the Wellcome Trust Sanger Institute, is tasked with providing support for data-sharing networks that enable clinicians and researchers around the world to collaborate effectively on large-scale research projects.

MalariaGEN

The largest data-sharing network that CGGH currently supports is MalariaGEN, the Malaria Genomic Epidemiology Network. MalariaGEN is a partnership of researchers in 21 countries who are using genomic epidemiology to understand how protective immunity against malaria works, which is a fundamental problem in malaria vaccine development. MalariaGEN has been operational since 2008.

CGGH acts as the MalariaGEN Resource Centre, providing scientific and operational support for MalariaGEN’s research and training activities. A key aspect of this operational support is the design, development and hosting of Web-based information systems that are used by MalariaGEN to manage data shared by MalariaGEN’s research partners. CGGH previously developed and currently hosts a system called Topheno, which is the system used by MalariaGEN to manage data sharing. Many lessons have been learned in the development and use of Topheno, and much my current work is building on that experience.

WWARN

A second data-sharing network that CGGH supports is WWARN, the World-Wide Antimalarial Resistance Network. WWARN is a global collaboration working to ensure that anyone affected by malaria receives effective and safe drug treatment. WWARN’s aim is to provide quality-assured intelligence, based on the balance of currently-available scientific data, to track the emergence of malarial drug resistance. WWARN is due to begin operations in the first half of 2010.

CGGH has responsibility for WWARN’s scientific informatics module, which includes in its scope the design, development and hosting of Web-based information systems to support WWARN’s data-sharing operations. These systems are currently under development.

Common Features of WWARN and MalariaGEN

There are some key similarities between WWARN and MalariaGEN.

In both cases, the operational workflow begins with the submission of original research data, usually by a researcher who is/was involved in the study from which the data originates, acting from their host institution (usually a university).

In both cases, data are submitted to the network from a distributed community of researchers. In the case of MalariaGEN, the set of researchers submitting data to MalariaGEN is delimited by the set of partners who have signed up to one of MalariaGEN’s Consortial Projects. For WWARN, the set of researchers submitting data is envisaged to be slightly more open-ended, with researchers submitting any original data that is relevant to one of WWARN’s four scientific modules.

In both cases, data are not primarily captured for submission to MalariaGEN or WWARN, but are captured as part of an independently funded original research study. Each study from which data originates has its own scientific objectives, which may be related to the objectives of the data-sharing network, but if so are usually more specific and finer-grained. The subjects for each original study are usually drawn from at most a handful of locations within a single country. The data-sharing networks then work to aggregate the data from many independent studies, in a reliable and scientifically valid manner, to conduct coarser-grained analyses across larger scales of time, space and biology than are considered in any one original study.

This last point has a number of important consequences. For example, because data are being primarily captured for an original research study, and not for the data-sharing network’s secondary analyses, the network is not in a position to mandate the manner or format of data collection and representation. Data may be collected for a range of purposes using different means and a diversity of representations. The data-sharing network must learn to deal with this heterogeneity, and this forms a large part of the network’s data management operations.

Also, because the data-sharing network is not the primary-endpoint for the data, those involved in the secondary analysis of shared data typically have to cajole researchers into submitting their data, because doing so means time out from their primary research activities. Therefore, the data-sharing network wants to minimise the obstacles it presents to those submitting data, and to find ways in which they can add value for the submitters’ primary research, even though that research will not be perfectly aligned with the secondary research activities and goals of the network as a whole.

Other Data-Sharing Networks

In addition to MalariaGEN and WWARN, CGGH is also involved in supporting an informal network of researchers working on the malaria parasite (Plasmodium) genomes. Here the main focus is on generating and analysing detailed genome sequence data using next-generation sequencing technology, although there may also be a need to share and aggregate other, related data. Finally, CGGH is involved in the UKCRC Modernising Medical Microbiology project, which may involve management of data from a number of different sources, although some of these sources will have been collected for health reasons and not for research.

Thus, involvement in data-sharing networks is a fundamental feature of CGGH’s activities. Although CGGH’s involvement is far from limited to informatics, and also encompasses sample and data management, statistics, ethics and programme management, nevertheless a key responsibility is the development of Web-based information systems that support the operational activities of a data-sharing network. It is also worth noting that the development or extension of those systems is often the rate-limiting step in establishing a data-sharing network or enabling it to adapt to a new type of data or analysis.

Generic Information Systems for Data-Sharing Networks

It is thus of urgent strategic importance to CGGH to identify those requirements for information systems that are common across these data-sharing networks. Once these requirements are understood, we need to identify a set of existing software and services that can be adopted and deployed to fulfil those common requirements. The underlying driver is to minimise the amount of time and effort spent on designing, developing and running common infrastructure, and thus make available as much effort as possible to deal with those requirements that are unique to the scientific activities of a particular data-sharing network.

What Next?

We’re working to identify and document many of the key requirements that are known to be common at least between MalariaGEN and WWARN. This work should hopefully then provide a basis for finding and evaluating existing software and services, and for designing a reference architecture which provides the highest possible point of departure for developing information systems for each data-sharing network. We’re going to need lots of help with that, so please feel free to contact me if you think you might be able to help.

Recently I’ve been involved in designing a software system (dsn-chassis) to support data-sharing for the World-Wide Antimalarial Resistance Network (WWARN). We’re also involved in developing and maintaining software for other data-sharing networks, such as MalariaGEN. Thus we have a vested interest in figuring out what requirements these data-sharing networks have in common, so we can identify common standards and software components that can be adapted and re-used. Naturally, we want to minimise the overall effort of building and maintaining supporting infrastructure for each new data-sharing community.

Applications and Services

One shift in perspective that I’ve encouraged is a move from viewing the software required to support a data-sharing network as a single, monolithic, web application, to viewing the software as a collection of web services and applications that are loosely coupled via open protocols and formats. The hope is that this view will make it easier to identify the most generic components of the required infrastructure, such as metadata persistence and query, or authentication and authorisation. We could then adopt service protocols and formats that give us the broadest possible choise of re-using and building on existing open-source software, the smallest possible dependency or lock-in to any one vendor or product, and the flexibility to extend and customise to cope with those requirements that are unique to a given data-sharing network.

That’s the ideal, anyway.

Metadata Persistence and Query

One capability that is common to the data-sharing networks we’re involved in is management of metadata relating to scientific data being shared within the network. This metadata includes, among other things, information about the study from which the data originates, such as the scientific protocols (i.e., procedures) that were used in generating the data. This metadata can be complicated, and varies substantially between different types of experiment. It is, however, needed to evaluate the quality and comparability of data from different studies, which is a prerequisite for aggregating those data in a sensible way. To get a sense for the type of metadata that needs to be captured, check out an early prototype of the study questionnaire being developed for WWARN.

Although the exact nature of this metadata will differ from one data-sharing network to the next, the basic capabilities to persist (CRUD) and query those metadata is common. Hence, one of the first things we’ve done is to take a look at various APIs and protocols for persisting and querying arbitrary packages of metadata, and their available implementations.

Atom Publishing Protocol

One protocol that stood out was the Atom Publishing Protocol and the related Atom Format, for a number of reasons. First, it follows the REST architectural style, which implies a number of helpful constraints. Second, there are at least a few reasonably mature open-source implementations (e.g., eXist, AtomServer) that are geared not just towards blog publishing but to arbitrary data and metadata management. Third, big players like Google and Microsoft are standardisting on Atom as a basis for their Web service APIs – not that I like to flock, but reading the Google Data APIs especially gave me some confidence that many of the difficult practical issues have already been encountered and solutions found.

REST-easy?

One of the most compelling reasons I’ve found to use the REST style in designing a Web service API, and to follow the approach of mapping the four basic persistence-related operations (Create, Retrieve, Update, Delete) on to the appropriate HTTP verbs (POST, GET, PUT, DELETE), is simplicity and ease of implementation. A second reason is the ability to decouple the persistence protocol from the data model, at least to some extent. In my experience, these two factors in particular contribute to being able to rapidly develop prototypes, and to carry as little baggage as possible forward as those prototypes (and the underlying data models) inevitably evolve towards a production system.

We’ve also been working with the eXist open-source XML database, which implements AtomPub out of the box, as our metadata persistence service implementation. One of the nice things about eXist is that it will store whatever XML you throw at it, so you can fiddle with your Atom content extensions to your heart’s content without ever needing to touch the server-side code or configuration. Another nice feature is that it supports Web service endpoints implemented as XQuery scripts, which is a very convenient way to add a wide range of query service capabilities to the data and metadata you’ve stored via the Atom service.

Our current, early, prototype of a data management system for WWARN uses eXist as the Atom service implementation, and a GWT application that we’ve rolled ourselves as the user-interface.

REST-not-so-easy?

While this approach has proved excellent during the development of early prototypes, there are a number of challenges approaching in moving the system towards production strength that are causing me some consternation. My main reason in writing this article is to highlight those issues, in the hope that others can provide some helpful ideas and advice.

I (somewhat pithily) entitled this article “REST-not-so-easy” because, while some of these issues are specific to AtomPub, others seem common to any Web service API based on the REST style. It could very well be that I’m missing a few rather obvious and simple solutions, I rather hope that’s the case.

So, on to the issues…

Link Expansion

When building our user interface, we commonly found that a single view required us to retrieve data not just from a single Atom entry, but rather from a graph of linked Atom entries. For example, in our data model, Dataset entries are linked to Study entries, which represent the study from which the dataset originates. When the user is viewing a dataset, they also want to view some basic information about the linked study, such as its title and summary, in addition to a hyperlink that allows them to navigate to further information about the study.

The client application can, of course, perform two HTTP GET operations to retrieve this simple graph of two linked Atom entries, which is easy enough. However, we have other examples where the graph to be retrieved has a depth of 2 or more. In these cases, implementing the client application becomes far simpler if the whole graph can be retrieved with a single HTTP GET operation, rather than many. A single GET is also better for latency, which is an important concern for us where users may be located in parts of the world with poor network bandwidth.

A workaround for this that we cooked up early-on was to develop query services that returned a single, root Atom entry as the result, with the required links outbound from and inbound to the target Atom entries expanded inline, within the atom:link element. I was initially unsure of the sanity of this approach, especially given that including an atom:entry directly within an atom:link element does seem to break the Atom Format Spec. However, I found some discussion from folks at Microsoft and Google on doing inline link expansion, which gave me some confidence that this idea isn’t so crazy. I also found a suggestion that returning the graph of linked Atom entries as a feed, rather than a single Atom entry with links expanded inline, was a more elegant way to go, which I have some sympathy for. However, we’ve gone with the inline expansion approach for now, which is working well so far. For example implementations using XQuery, see chassis dataset query service and the supporting function library.

There is an obvious gotcha here, which is that if you retrieve an Atom entry with links expanded inline, then PUT that entry back to the edit link URL, you will end up storing the linked entries inline too, which leads to all sorts of interesting errors. So, any view that needs to do a PUT must first retrieve a fresh, unexpanded representation of the entry, before it can be modified and PUT back to the server.

I’ve also glanced at an article on AtomServer which describes the idea of aggregate feeds created by joining separate collections. It looks like this might provide a similar capability, but in a quite different way. However, if I have understood the idea correctly, it looks like you could only ever fetch a graph on entries 1 deep.

Referential Integrity; Broken Links

Our simple data model currently comprises entities such as Study, Dataset, Data File, Submission and Review. There are, of course, associations between these entities, such as an assocation between a Dataset and the Study from which it originated, as mentioned above. When we map our logical data model onto Atom, each type of entity maps onto a distinct collection of Atom entries. Each type of entity also gets its own Atom content extension, which means that we stick the entity data into some XML nested in the atom:content element of the Atom entry. Associations between entities are mapped onto links between Atom entries. I.e., we use the atom:link element to represent associations between entities, where the value of the rel attribute represents the type of the association.

To create a new association, a client retrieves a representation of the entry that is the source (i.e., subject) of the assocation, adds a new atom:link element with the appropriate rel attribute (describing the association type) and href attribute (pointing to the target/object of the association), and then PUTs the entry representation back to the entry’s edit link URL.

By default, eXist’s Atom implementation does not do any referential integrity checks on the links in an Atom entry. This is perfectly sensible, because no integrity constraints have been declared, and because the link could equally point to an Atom entry anywhere in the Web, not just other Atom entries located on the same service.

However, if an Atom entry that is the target of one or more links is deleted, then all of those links will be broken. How should the service deal with this, if at all? Is it OK to do as the Web does, and leave it up to the client (and the user) to deal with broken links? Or should the service be a bit smarter, e.g. by noisily preventing deletion of entries that are link targets, or by silently deleting links to deleted targets, or by some other mechanism? What about new links that are created to non-existent targets? Should those be prevented somehow? And should the service differentiate between links whose targets are entries hosted by the same Atom service and links whose targets are elsewhere in the Web? If so, how? I.e., are there different kinds of referential integrity that need to be considered?

Our current, very tentative, position is to do as the Web does, and leave it to the client to handle broken or bad links. The client will typically do this by simply notifying the user of a broken link. The client will provide the facility to allow the user to fix broken links, but what the user does next is up to them. However, we haven’t implemented any of this functionality in the client application yet, so I don’t know what the consequences will be.

Transactions

Sometimes, what is a single operation from the user workflow point of view, such as creating a study, or creating a dataset, or updating a study with new information, maps onto a single Atom Protocol request, such as an HTTP POST or PUT. However, often a single user operation maps onto multiple Atom Protocol requests. For example, creating a new revision of a data file involves four HTTP requests, which are a POST (create a new media entry that represents the data file revision), a PUT (add some metadata about the media entry, e.g., the original file name), a GET (retrieve a fresh representation of the data file entry), and finally a PUT (link the data file entry to the new revision).

Any of these requests could fail, leaving the data in an inconsistent state. How should we handle this possibility?

The obvious answer is to provide some sort of transactional capability, such that the client can invoke these requests within the context of a single transaction that will either completely succeed or fail with no change to the data. But that begs two further questions: how do you add transactions to the Atom Protocol, and how do you implement them?

In my not-very-extensive searching of the Web, I have not found much in the way of discussion and/or implementation of transactional capabilities for REST-style web service APIs, except for some work on transactional support for JAX RS based applications. An alternative to a protocol extension is simply to design the Web service API to expose only those operations that are atomic from the user workflow point of view, then handle transaction-type issues behind the scenes. But that would mean leaving AtomPub behind and layering another API on top, which seams to completely defeat the purpose of going for REST/AtomPub in the first place.

The other option is, of course, to leave it up to the client to deal with inconsistent data, which might include trying cleaning it up automatically or simply notifying the user of problems and leaving the rest up to them, as with the discussion of broken links above. But that would leave open all sorts of weird and wonderfully wrong possible states of the system.

If someone were to point me to an extension to the Atom Protocol that allows multiple Atom requests to be carried out in the context of a single transaction, and an existing, mature, open-source implementation, I would be very happy .. I think.

AtomServer has a capability to combine several Atom requests into a “batch”, via a custom extension to the Atom Protocol, based on the batch processing capability in the Google Data APIs. Although I couldn’t find any mention of adding transactional processing of an entire batch in the AtomServer documentation, it does mention that a batch is processed as a single database operation, and so it shouldn’t be too difficult to wrap that with a transaction. I’m not sure how that would impact the AtomServer batch response model, however.

Granularity

The Atom Publishing Protocol, as with any CRUD-style API, is relatively coarse-grained, in the sense that if you want to update any field of an entry, you have to update the whole thing at once. This coarse-grained nature is good from one point of view, because it means that the details of the data model are kept separate from the design of the protocol. The data model can then evolve without needing any change to the protocol specification or implementation. However, it does mean that the protocol can be quite inefficient and wasteful in terms of network bandwidth. I.e., if you’ve stored a reasonable amount of data in a single Atom entry, but all you want to do is change the title, you’ll have to PUT the whole thing. This can mean noticeable latency in the client application, especially where network bandwidth is poor.

On the face of it, this issue is less serious than dealing with broken links or inconsistent data, but it does make other protocols that provide the ability to submit a fine-grained change set or change request for a particular data or metadata record (such as the Talis Platform API) start to look quite appealing.

When you delve a bit deeper, this issue also interacts with the issue of authorisation and access control, which I’ll come to below.

Authentication

I’ll just mention this briefly to say that, for us, authentication is not an issue. We use a Spring Security filter to implement authentication using HTTP Basic in the development environment, and webauth in production.

I have seen articles on inventing new authentication protocols for Atom to work around restrictions imposed by their server environment, but those restrictions don’t apply to us.

As a side note, I suspect there is a bug in the authentication code in the eXist Atom servlet, as we had some pain trying to turn it off and use Spring Security for authentication instead (for HTTP Basic authentication, we had to add a filter that removes the Authorisation header before it gets to the eXist Atom servlet, otherwise the servlet’s own authentication process was activated), but I haven’t pinpointed it yet. I’d like to see eXist factor out and consolidate all authentication code from its Atom and XQuery servlets to dedicated filters, which would make for easier integration with Spring Security or other authentication frameworks .. but that’s another story.

Security, Authorisation and Access Control

I’ve saved the best for last.

Within a data-sharing network, typically different roles are defined, with different privileges/permissions/authorities/rights/acls/…

For example, for WWARN we have defined a submitter role, which is a person who wants to share original research data with WWARN; a gatekeeper role, who performs an initial review of submitted data and decides whether or not the data should be accepted for curation; a curator role who’s job is to clean up and standardise data submitted from different studies so that it can be sensibly aggregated; a coordinator who oversees the operations of the data-sharing network; and an administrator who installs, configures and maintains the systems.

There are things people should be able to do. For example, a submitter creates studies, datasets, and data files, and can submit a dataset to the network. A gatekeeper reviews submissions and assigns a curator if the submission is accepted. A curator creates derived data files and reviews curated data for validity and conformance with standard data dictionaries.

There are, of course, things people should not be able to do. For example, a submitter cannot review their own submission or decide that their submission should be accepted by the network. Neither can a submitter assign a curator to their submission. These capabilities are allowed for the gatekeeper only.

These permissions can be implemented at the user-interface level. I.e., the user-interface can expose only those functionalities that are permitted for the user’s role(s). However, if the client application is using a Web service to implement some or all of these operations, then the Web service API must also have appropriate constraints, otherwise users would be able to hack the API and do things they shouldn’t. If the Web service API is based on Atom, then the problem we have is how to implement the appropriate authorisation constraints in a way that works with Atom and existing implementations.

Some constraints are simple to implement. For example, the constraint that only a submitter can create a study can be implemented by allowing only POST requests to the Studies Atom collection URL if the user has the submitter role. I.e., constraints that can be mapped onto a specific class of CRUD operation on a specific Atom collection with a specific role are straightforward, and could, for example, be implemented using a Spring Security filter and URL patterns.

Others are not so simple. For example, a submitter who creates a new study becomes the owner of that study. The original owner may also grant ownership to other submitters; the owner(s) of a study are the only people who can update information about that study. This begs the questions, how do we represent entry-specific access control constraints in the Atom Protocol and/or Format, and what implementations (if any) are available?

The eXist database does have its own solution for Atom security. You can declare access control constraints for a specific Atom entry by including an exist:permissions element within the Atom entry. However, this presents three problems for us. First, it requires that we use the eXist database of users, but we have to integrate with another, external database of users, so we would have to keep those two synchronised. Second, it is based on the Unix/Linux model of file system permissions, which is too inflexible. For example, we want to enable a user to grant arbitrary permissions for a given entry to arbitrary collections of specified users. Having a single owner/group for each entry means you cannot do this sort of thing. Third, the format is specific to eXist. If we buy into their permissions format, we will have a job to port it to another atom implementation such as AtomServer, if we need to do that at some point in the future.

eXist also has an XACML sub-system, but the capabilities do not obviously map onto our requirements for access control to Atom collections and member entries.

The Google Data APIs have a different approach. See for example the Google Calendar API section on sharing calendars. Each access control rule is an Atom entry. There is one collection of access control rule entries for each calendar. Each access control rule has a “scope”, which typically specifies a user, and a “role”, which is typically a permission such as “read” or “editor”. See also the gAcl namespace reference.

The Google approach has the flexibility we need, but I’m baulking at implementing something like this ourselves on top of eXist or another Atom implementation.

And there is another issue here. It’s not hard to imagine situations where you might want different users to be able to update only specific parts of an Atom entry, but not the whole thing. I.e., you might want to have fine-grained, within-entry access control rules for different users. This sort of thing can be handled in a purely Java environment, for example, using Spring method security and Spring domain object security. However, it’s not obvious how to implement this sort of thing in a general way for REST/Atom, and so we’ve deliberately designed our data model to avoid needing this kind of rule. I.e., rules are only ever defined at the granularity of a single entry, nothing finer. This means that authorisation considerations have a significant impact on the design of the data model. Not necessarily an issue, but interesting nonetheless.

We need a way forward here, which means minimum coding for us (we want to focus our effort on the applications, rather than the underlying services), and maximum simplicity and portability.

Any Thoughts?

If you’ve read this and have any thoughts, ideas or suggestions about any of the issues above, really anything at all, no matter how trivial, please do drop me an email, or add a comment to this article, or join the dsn-chassis group and post a message there. Thanks for reading.

Thanks to the organisers here for putting on a great conference, and for inviting me to give this talk, it is a real honour.

I will post notes from this talk on my blog at purl.org/net/aliman – so don’t worry if you miss anything.

It is also excellent timing, as after at least 5 years of talking about it, I can finally say that, on 18 August 2009 SKOS – the Simple Knowledge Organisation System – was published as a W3C Recommendation, thanks in large part to the experience and support of this community.

I’d like to talk more about SKOS later in this presentation.

Before I get into the main body of my talk, let me say first that, one of the lessons I’ve learned in working on Semantic Web Deployment, especially in the last 2 years, is that big ideas can look great when viewed from a distance, but the reality on the ground will always be far more complex and surprising than you anticipate, as I’m sure many of you can testify to.

In particular, the cost/benefit trade-offs for investing in a particular technology or approach can vary wildly from one situation to the next. Nothing is more important than keeping an open mind.

So, although I love talking about this big ideas, I’m going to try to put them to one side for this morning (although, I’m sure I’ll still be tempted by one or two). Instead, I’d like to simply share a few experiences, and hopefully give you at least a flavour of where some of the opportunities and the challenges might lie.

Now, given that we’ve been talking about metadata for at least two days now, I thought I’d spend a few minutes talking about something completely different.

Fruit Flies

These are fruit flies, of the species Drosophila melanogaster.

On the left is a male, on the right a female. Females are about 2.5 millimeters long, males are slightly smaller.

When I first saw this image, I thought it was a photograph. But, when I looked closer, I noticed it was in fact a drawing, and it was signed “E Wallace”.

Edith Wallace, I found out, was curator of stocks and artist for Thomas Hunt Morgan, who c. 1908 began using fruit flies in experimental studies of heredity at Columbia University.

Heredity simply means the passing of biological traits, such as green eyes or brown hair, from parents to offspring.

I’d like to talk more about Thomas Hunt Morgan, but before I do, it’s worth noting that when Charles Darwin published his book On The Origin of Species by Means of Natural Selection in 1859, which of course depends in its central argument on the inheritance of traits from one generation to the next, the underlying mechanisms of that inheritance, i.e., of heredity, were completely unknown.

It was, of course, Gregor Mendel’s work on pea plants, published first in 1865, but not widely known until the turn of the 20th century, which first suggested that organisms inherit traits in a discrete, distinct way. It is these discrete units or particles of inheritance that we now call genes.

However, at the turn of the century, although the theory of genes was accepted, it still was not known which molecules in the cell carried these genes.

It was T. H. Morgan’s aim to determine what these molecules were.

He needed a suitable animal to study, and chose Drosophila, primarily because of cost and convenience – they are cheap and easy to culture, have a short life cycle and lots of offspring.

Morgan was looking for heritable mutations to study, and spent a long time looking before he discovered a few flies with white instead of the usual red eyes. The white-eyed mutation only occurs in male flies, i.e., it is a sex-linked trait. The fact that the trait depends on the sex of the individual suggested to Morgan that the genes responsible for the mutation reside on the sex chromosome … and that the chromosomes generally are the carriers of genetic information.

Morgan and his students also discovered that some traits, such as wing length and eye shape, do not get mixed up randomly from one generation to the next, but rather tend to be inherited together.

They demonstrated that the reason why these traits tend go together is because the genes responsible are on the same chromosome, and are in fact quite close together on the same chromosome, and hence tend to stay together when chromosomes recombine during formation of sperm and egg.

They then used these observations to construct the very first genetic maps, that is, maps of where different genes are in relation to each other on the chromosomes. This was the first crucial step in understanding how a genome is organised.

Morgan started something of a trend when he chose Drosophila, because for the last century Drosophila has been one of the most intensively studied model organisms.

It is fair to say that much of what we know today about genetics, and the molecular mechanisms underlying development, behaviour, aging, and many other biological processes common to flies and humans, we know from research on fruit flies.

In the last decade, that research has entered an entirely new phase.

In 2000, the complete genome sequence of Drosophila melanogaster was published. For the first time, we gained a complete picture of the location and DNA sequence of every gene in the genome. Publication of those data has unlocked entirely new methods and avenues of scientific investigation.

However, many questions remain unanswered.

What do all those genes actually do?

Where and when do they play a part in various biological processes?

And, especially, how do genetic differences between individuals relate to different biological outcomes?

E.g., translating that question from flies to humans, why do individuals with one genotype tend to be resistant to a disease such as malaria, whereas others don’t?

Answering these types of questions is the domain of functional genomics.

A functional genomic study typically asks, in relation to some biological process like development of sperm or the eye, …

What genes are active, where, when and how much?

Which genes are interacting with each other?

What happens if you stop a gene from working (knock it out), then restore it?

For example, this image shows that a particular gene called schumacher-levy is active during the developing Drosophila embryo, and is localised to a specific organ, in this case the developing gonad.

Other advances in biotechnology over the last decade have revolutionised this type of research.

For example, we now have a number of tools for carrying out high-throughput experiments. These are experiments where, rather than looking at just a handful of genes, we can look at 100s or even 1000s of genes at a time.

The bottom line: these high-throughput technologies, in addition to rapid advances in genome sequencing technology, generate a very large amount of highly heterogeneous data.

And our ability to generate new data on unprecedented scales is accelerating. Really, the pace of change is staggering.

To give you an idea of the rate and scale of new advances, consider that, in July 2008 the Wellcome Trust Sanger Institute, the primary institute for DNA sequencing in the UK, announced that, every two minutes, they produce as much DNA sequence as was deposited in the whole first five years of the international DNA sequence databases, from 1982 to 1987.

Finding, comparing, and integrating these data is a critical challenge for ongoing biological research in all organisms, and is especially critical in translating findings from model organisms such as Drosophila to human health.

Because this is such a critical problem, much excellent work has already gone into making Drosophila data publicly available and accessible.

For example, FlyBase provides access to primary genome sequence data on all sequenced Drosophila species. It is the primary reference point for all Drosophila genome-related data.

FlyBase also establishes a controlled vocabulary for all Drosophila genes, which is a vital tool in integrating data from multiple sources, because genes are often the point of intersection between data sources.

BDGP embryo in situ database is an example of a database holding the output of high-throughput functional genomic studies. They publish images that depict the expression of a gene within the developing fly embryo, at various stages during the course of embryo development, for thousands of genes.

FlyAtlas also publishes data from high-throughput experiments, but using a different technique. They use DNA microarrays to get quantitative data on gene expression in different tissues, so the data tell you not only whether a gene is active, but also how much it is being expressed.

Much of the focus has, to date, been on providing a direct user-interface to each source of data. So each data provider has a set of web-based tools for a human researcher to query and visualise the data.

However, the focus is shifting towards enabling data to be harvested and integrated across databases in an automated way, because it is recognised that this could save much time and effort, and because some questions just couldn’t be answered any other way.

And there are projects making headway here, for example, FlyMine uses a conventional data warehouse approach to integrate data on Drosophila. However, it is no small challenge, and there is a pressing need to make the end products much more usable, and because public funding will always be scarce, to make the whole process as cost-effective, scalable and sustainable as possible.

FlyWeb Project

With this in mind, in January 2008, I moved to the Zoology Department at the University of Oxford, to work with a team there led by Dr David Shotton on a small project called FlyWeb.

FlyWeb asked two questions…

1. Can we build tools for Drosophila biologists that cut down the effort required to search across different sources of gene expression data?

2. Under the hood, what (semantic) web tools and design patterns help us to build cross-database information systems, and ensure that they are robust, performant and quick to build.

To answer these question, we set about building a proof of concept, which is deployed at openflydata.org.

How does openflydata.org work?

Each of the cross-database search applications I’ve demonstrated is, simply, a mashup.

It is a light weight, HTML+JavaScript application that runs in the browser, that fetches data in real time from several different web service endpoints.

Why did we use the mashup approach?

Simplest thing we could think to do. Also gave us flexibility to experiment with semantic web technology, without totally committing to it. I.e., we could mix semantic web and other solutions, if it proved easier to do so.

The bottom line was, we wanted to produce useful, compelling functionality for a biological researcher, in a reasonable time frame and with a reasonable level of performance and reliability. If semantic web helped, great, if not, fine, we’ll try something else.

Of course, it would have been great if each data source had provided a web service endpoint for their data with the necessary query functionality … but they didn’t, so we made some ourselves.

The approach we took was, for each data source, we first converted the data to RDF (the Resource Description Framework, one of the key Semantic Web Standards), then loaded the data into an off-the-shelf open-source RDF storage system (Jena TDB written by Andy Seaborne, now at Talis), then mounted each RDF store as a SPARQL endpoint.

What is a SPARQL endpoint?

It is, simply, a web service endpoint that uses the SPARQL protocol as its interface (API).

What is the SPARQL protocol? It is a simple HTTP-based protocol for sending SPARQL queries to be evaluated against a given data store.

What are SPARQL queries? SPARQL is a query language, roughly analogous to SQL (read only), but built for the Web. Basically, it gives you a way to ask pretty much any question you want to ask of a given set of data.

Why did we use RDF & SPARQL?

1. Rapid prototyping. Leverage off-the-shelf, open source, software, such as Jena, or Mulgara (see David Wood). In principle, we only had to write the software to convert the data from each source to RDF. We could then use OTS software to deploy an RDF database and SPARQL endpoint. (Caveat…)

2. SPARQL is a simple protocol with an expressive query language. That means its easy to write code for, but gives the client application (in our case, the mashups) the power to ask any question it wants. (Caveat…)

Point 2. also means we can offer these SPARQL endpoints as a service to the biological community, so others with a bit of savvy can ask their own questions. In particular, they can ask questions that we (the service provider) hasn’t thought of, which (in theory) promotes innovative re-use and exploitation of the data.

You’ll noticed I haven’t yet talked about either of the two key themes of this conference, semantic interoperability or linked data.

I haven’t mentioned these yet because, the point I’d like to make is that, depending on the context, there *may* be compelling, practical, short-term reasons to evaluate Semantic Web-based technology for a data integration project …

1. The (relative) ease of deploying a web service endpoint for querying a data source; i.e., of making the data accessible, lowering the barriers to re-use.

2. The (relative) simplicity of exploiting those web services to prototype light weight cross-database search and on-the-fly data integration applications. (pardon the pun)

Caveats…

Of course, the best choice of technology depends greatly on the existing technological context, and on the expertise available to you. As with relational databases, XML, or any family of related technologies, becoming productive with a new technology requires an investment in terms of people, training and time.

A second caveat is that, the more open and expressive a query protocol like SPARQL is, the harder it becomes to guarantee the performance and availability of a service using that protocol. It is a denial-of-service type problem. If anyone can ask any question they like, some people will ask hard questions, whether intentionally or by accident, which could degrade service performance for others.

SPARQL is thus a double edged sword. On the one hand, its open nature and expressivity is a major advantage, but that openness creates challenges when it comes to providing reliable and performant services to others.

I see resolving this tension as one of the key challenges for the semantic web community in making the technology widely applicable. We explored some strategies for mitigating these issues in FlyWeb, but we certainly did not find all of the answers.

Now, let’s talk about semantic interoperability and linked data.

The two classic problems you encounter when integrating data from different sources are…

1. schema alignment … each data source has a different data model … they structure their data in a different way … using different names for similar types of entities and relationships … or using similar names for what are actually very different types of entities and relationships …

2. coreference resolution … each data source may use different identifiers for the same thing (e.g., genes) … or (more rarely) identifiers may clash …

Our approach to schema alignment was not to try to completely align all our data sources in a single step.

Rather, we tried to pick the low-hanging fruit, and take an incremental approach, making use of existing data models as much as possible.

So, for example, the data from FlyBase come from a relational database, which is structured according to a relational schema developed over a number of years by the Generic Model Organism Database community, a schema they call Chado.

When we transformed FlyBase’s data to RDF, we used the Chado schema to help design the RDF data structures we were generating. In fact, we went one step further than that, and we semi-automatically generated an OWL ontology from the Chado relational schema. This ensured that we took a systematic approach to the data transformation, and that the definitions for the data structures in the output RDF could be grounded in the definitions already established by Chado.

Our approach to coreference resolution was, similarly, to make use of existing controlled vocabularies.

Our biggest problem was identifying genes. A single gene might be known in the scientific literature by many different names. This has been a perennial problem for Drosophila biologists, and a big part of what FlyBase has done has been to establish a definitive controlled vocabulary for Drosophila gene names, and curate a list of known synonyms for each gene.

So we used FlyBase’s unique gene identifier system as a foundation, constructed a set of URIs for Drosophila genes based on the FlyBase identifier. We then used these URIs to link data from each of the various sources.

You’ll notice I said “link data” just there. What do I mean by that?

Well, I’d like to make a distinction between two types of linking.

1. “semantically linked” – data from different sources use a common set of URIs to identify data entities, e.g., people, places, genes, diseases, (… or any two URIs that identify the same entity have been explictly mapped)

2. “web linked” – data are semantically linked, and URIs resolve to data so links can be followed by a crawler … this is what most people mean when they talk about “linked data”

We might also describe a third notion, …

3. “semantically aligned” – data from different sources use a common schema, that is, they share a common data model, (… or their respective data models have been explicitly mapped)

To build our cross-search applications, we went for the low-hanging fruit. I.e., we did just enough, and no more, to get them working.

This meant a very small amount of semantic alignment. In fact, it was quite possible to work around differences between data models, as long as those data models were understood. We certainly did not need to accomplish a complete and perfect alignment of all data models, before we could start building prototypes.

We also did not make the data web linked, i.e., we did not publish true linked data. Why not? Because it didn’t serve our immediate needs. We needed performant and queryable web services to data for each source, so we could build a mashup that selected the data it needed. Whether the data were actually linked in the web was, for this project, not relevant.

We did, however, work on semantically linking the data, i.e., mapping differences in the identifiers used, especially for genes, and this was absolutely critical to getting a reasonable level of recall and precision.

Which is not to say that I think true, web linked data is a bad idea. There may be other reasons for deploying web linked data, which would have been relevant especially if we had wanted to go beyond a proof-of-concept system.

But the point is, you can get a quick win if you make data available via a queryable web service. You save a lot of time and effort if data from multiple sources are semantically linked. But your data certainly don’t need to be perfectly and completely semantically aligned before you can start using them.

SKOS

I’d like to leave flies now, and return to SKOS.

As you all know better than I do, one of the cornerstones of information retrieval for many years has been the development of controlled structured vocabularies, such as the Library of Congress Subject Headings, or the Dewey Decimal Classification, or the Agrovoc Thesaurus.

Ever since the advent of the Web, there has been a desire to make better use of these valuable tools, to help organise and connect information as it emerges from closed silos and is shared via the Web.

Hopefully, the Simple Knowledge Organisation System (SKOS) will go some way towards enabling that to happen.

SKOS provides a common, standard, data model, for controlled structured vocabularies like thesauri, taxonomies and classification schemes.

This means that, if you own or have developed a controlled vocabulary, and would like to make it available for others to use, you can use SKOS to publish your vocabulary as linked data in the Web.

Because SKOS is now a standard, your data will be linkable with other vocabularies published in a similar way, and (hopefully) compatible with a variety of different software systems.

If you were at DC2008, you would have heard Ed Summers talk about his work to deploy the Library of Congress Subject Headings as linked data, using SKOS.

His initial work was deployed at an experimental site, but since then, based on Ed’s work, the Library of Congress has deployed their new Authorities and Vocabularies Service at id.loc.gov.

The first service deployed there is, of course, the LCSH.

To explain what LOC have done, for each heading in the LCSH, LOC have minted a URI for that heading.

For example, the URI http://id.loc.gov/authorities/sh95000541#concept identifies the LCSH heading for the World Wide Web.

If you plug that URI into the location bar of your browser, you’ll get a conventional web page providing a summary of that heading.

With a small change in the way you make that request, you can also retrieve a machine-readable representation of that heading. I.e., you can get data. Those data are structured using SKOS.

Each heading is, of course, linked to other headings, so you could, if you wanted to, follow the links from one heading to the next, collecting data along the way.

Alternatively, if you want to re-use the entire LCSH in your own application, you can download the whole thing in bulk, again as data, structured using SKOS.

I hope what the Library of Congress have done with LCSH achieves three things.

First, I hope it means more people use the LCSH. For all it’s quirks, the LCSH, like many other vocabularies, is an invaluable resource, and I hope we will see it turned to wild and wonderful new uses.

Second, I hope it encourages other projects to re-use the LCSH URIs, to link their metadata records to LCSH via the Web. That would make it much easier to make use of links between metadata records across existing collections.

Third, I hope LCSH serves as a hub for linking other vocabularies as they emerge into the Web.

I think that, in a short time, it is not unrealistic to imagine that, we could see LCSH as a hub in a web of linked vocabularies, with that web of vocabularies itself serving as a hub for a much larger and broader web of linked metadata.

That this is possible is demonstrated by the fact that the LCSH has already been linked to another vocabulary, the French RAMEAU vocabulary, used by the Bibliotheque nationale de France.

A second piece of work I’d like to highlight is Michael Panzer’s work on dewey.info.

One other project I would like to highlight from within the DCMI community is the work of Jon Phipps and Diane Hillman on the NSDL metadata registry.

In fact, to call it a registry doesn’t, in my mind, do it justice, because it is a complete vocabulary development, maintenance and publication platform, built using SKOS.

And it doesn’t only cover vocabularies, it covers metadata schemas (a.k.a., element sets) too.

And not only does the registry make all of the underlying data accessible via normal HTTP requests, (which you could use to implement linked data), it provides a SPARQL endpoint too, so you can query the schemas and vocabularies however you like.

The LCSH, dewey.info, and the NSDL metadata registry, are just three examples of recent uses of SKOS, a good source for more is the SKOS Implementation Report.

Let me be the first to say that SKOS isn’t perfect. Neither does it cover every eventuality. While thesauri, classification schemes and subject heading systems do have something in common, they also exhibit diversity. In many cases, that diversity is not just a historical artifact, but exists for good reason, because the vocabulary is adapted for a specialised purpose.

Our goal with SKOS was to capture enough of this commonality to enable some interoperability, but to provide an extensible foundation from which different communities could innovate, and explore solutions to their own particular problems.

We’ve seen already, at this conference, for example, in Michael Panzer and Marcia Zeng’s presentation, how work is well underway to develop extensions to SKOS for classification schemes.

This photo was taken during a discussion of how to extend SKOS for the Japanese National Diet Library Subject Headings.

On this note, I’d like to share a small insight, which I gained while I was working on SKOS, thanks to my colleagues in the W3C Semantic Web Deployment Working Group.

When I started working on SKOS, I thought that developing a standard was about getting everyone to do the same thing. I.e., it was about uniformity.

Now, I have a different perspective.

Consider that, if the developers of the original Web standards had tried to think of every possible way the Web might be used, then tried to design a complete system to accommodate all those possibilities they could think of, the Web would probably not exist today, for two reasons.

First, they would still be here today, imagining new possibilities, and arguing about how to deal with conflicting requirements.

Second, they would have built a system that was too complicated.

Whether by intention, inspiration, or accident, the original Web standards have not led to uniformity, but have rather led to an explosion of innovation and diversity.

Thus, my insight is that, a good standard, at least for the Web, is one that provides a platform for innovation. It musn’t try to do too much. Of course, it must be clear about everything that is within its scope, and so provide a sound basis for interoperability. But it should be aggressive about limiting its scope. And it must be flexible and extensible, to accommodate differences, and to enable unexpected ideas to be realised.

Striking this balance is, of course, far from easy, and I have no way of knowing whether SKOS has found the right balance. However, the people I have met through my work on SKOS continue to be an inspiration, and I hope at the very least it will provide a stepping stone to the future.

RDA, FRBR and RDF

Of course, when it comes to sharing and linking metadata, controlled vocabularies and SKOS are only a small part of the picture.

We also need standards for sharing and linking the metadata itself, standards that provide a basis for interoperability but that also can, as Michael Crandall beautifully illustrated on tuesday, accommodate the richness and complexity of of our descriptions, and of the artifacts they describe, be they literary works, works of art, or the results of scientific inquiry.

Here, too, there are opportunities to build on a significant body of previous work.

Two such bodies of work are the Functional Requirements for Bibliographic Records (FRBR), and the Anglo-American Cataloging Rules (AACR), which is the precursor to the Resource Description and Access (RDA) specification.

Now, I am not an expert on bibliographic metadata, so I cannot comment on the details of these standards.

However, I can tell you that, it is possible to take the data models underlying FRBR and RDA, to publish those data models using Semantic Web standards, and then to use those models as a framework for transforming existing metadata records to RDF and publishing them as linked data in the Web.

Earlier this year, I did a very modest amount of work with the DCMI-RDA task group, proving the concept. Using a set of cataloging examples, and using the RDA elements schema developed by the task group, I developed some patterns for representing bibliographic metadata as RDF.

I then tested these on a larger scale, using a dump of just under 7 million MARC records from LOC. I showed that at least some of the metadata from the MARC records could be transformed to RDF using the FRBR schema and RDA elements schema and vocabularies.

Next steps for this work would be to increase the coverage of the converted data, and to publish it not only as linked data but also via queryable (e.g., SPARQL) web services, which would drastically reduce the barrier to re-use of this fantastic resource.

Data-Sharing Networks for Malaria Research

I’d like to conclude my talk by returning to scientific research.

In June this year, after the FlyWeb Project finished, I moved up the hill in Oxford to join the Centre for Genomics and Global Health, which is a joint research programme of Oxford University and the Wellcome Trust Sanger Institute, directed by Prof. Dominic Kwiatkowski.

The main focus of our research is to assist the global campaign to eliminate malaria.

According to the WHO’s World Malaria Report 2008, half the world’s population is at risk of malaria, and an estimated 247 million cases led to nearly 881,000 deaths in 2006. Small children remain by far the most likely to die of the disease.

The recent advances in biotechnology that I mentioned earlier, in particular the rapid advances in DNA sequencing and genotyping technology are, of course, being brought to bear on the problem.

One of the most promising approaches is called genomic epidemiology, which combines genetic data from the lab with clinical data from the field, to understand why, for example, some people are less susceptible to serious infection than others. It is hoped that, by understanding the natural mechanisms of protective immunity against malaria work, this may contribute to the development of an effective malaria vaccine.

The crunch is that this type of research cannot be done on a small scale. Because genomic epidemiology involves analysing hundreds of thousands of points of variation in the human genome, and searching for associations between these genetic differences and different disease outcomes, a large number of samples (i.e., patients) need to be included, to gain the necessary statistical power to find genuine associations.

Thus genomic epidemiology requires research collaboration on an unprecedented scale. And the key to enabling this type of research is data-sharing.

Will the Semantic Web and linked data help? I hope so. The Web, in all its chaos and diversity, will certainly play a pivotal role. But the challenges are too broad to be solved by one family of technologies alone.

Many of the key challenges are social, rather than technological.

For example, enabling a scientific community (i.e, one not trained in data modeling) to quickly reach working agreements on data standards, and enabling scientists to translate between their own view of their data, and a standardised view of their data, is vital. Here, bridging the gap between the technology and the people has never been more important.

Similarly, reaching agreements on when data may be shared, and on how it may be used, is key. Because, of course, in addition to ensuring credit is received for individual scientific research, personal genetic and clinical data is highly sensitive, and there are strict ethical rules about data use and privacy.

In spite of these challenges, I remain hopeful. I am especially encouraged by the openness of communities like this one, and on the willingness of those communities to share their experience and expertise.

With that, I hope you enjoy the rest of the conference, and thank you for listening.

W3C issued the following press release: From Chaos, Order: W3C Standard Helps Organize Knowledge

I’m proud to have been a part of this work, and extremely grateful to all those who have supported and contributed over the last 5 years.

-Dgwt.args="-manual"

The main goal of FlyWeb was to minimize the time required for a researcher in the domain of Drosophila (fruit fly) functional genomics, with no informatics training, to find and compare gene expression data from different databases on a large number of genes. With this in mind, we developed openflydata.org, which hosts the following cross-database gene expression data search applications:

• openflydata.org/search/gene-expression – search for a single gene of interest, and then retrieve and display expression data for that gene, including tissue-specific mRNA levels from FlyAtlas, embryo in situ hybridization images and ontology annotations from BDGP, and testis in situ hybridization images from FlyTED. Also retrieved are literature references relevant to the selected gene, provided by FlyBase.
• openflydata.org/search/gene-batch-expression – search for a batch of genes, then retrieve and compare expression data from FlyAtlas, BDGP and FlyTED, for all matching genes.
• openflydata.org/search/by-expression-profile – search for genes matching a given tissue-specific mRNA expression profile, based on data from FlyAtlas, and then retrieve further expression data for each gene found.

The applications are all pure JavaScript, built using a custom library called FlyUI. They fetch data AJAX-style directly from four SPARQL endpoints, one for each of the four sources of genomic data. On the server side, we use Jena TDB as the underlying RDF storage and query engine, and SPARQLite as the SPARQL protocol server. The whole thing runs on a small EC2 instance.

Further details on our work to convert the four data sources to RDF, in addition to bulk RDF downloads, SPARQL endpoints and more, can be found at the links below:

• FlyBase
• BDGP
• FlyAtlas
• FlyTED

To reach me via email, use: alimanfoo at gmail dot com

I will be on leave from 15-31 May.

Links are valuable. Links between documents, between people, between ideas, between data. Data is now a first class Web citizen, and the Web is expanding as more of these valuable networks are deployed within its fabric. Well-established knowledge organization systems like the Library of Congress Subject Headings will play a major role within these networks, as hubs, connecting people with information and providing a firm foundation for network growth as many new routes to the discovery of information emerge through the collective action of individuals. Or will they?

This talk introduces the Simple Knowledge Organization System (SKOS), a soon-to-be-completed W3C standard for publishing thesauri, classification schemes and subject headings as linked data in the Web. This talk also presents SKOS in the context of the W3C’s Semantic Web Activity, and in particular the work of the W3C’s Semantic Web Deployment Working Group where other specifications are being developed for publishing linked data in the Web, for embedding linked data in Web pages, and for managing Semantic Web vocabularies. Finally, this talk takes a mildly inquisitive look at the value propositions for linked data in the Web, and how LCSH might be deployed in the Web for better information discovery.

As an aside, I heard recently about the deployment of the Library of Congress Subject Headings (LCSH) as linked data in the Web, using SKOS. This nice work provides a great backdrop to our meeting.

I have Python 2.5 installed from the MSI. When I ran python setup.py install from the RDFLIB download directory, I got the message:

error: Python was built with Visual Studio 2003;
extensions must be built with a compiler than can generate compatible binaries.
Visual Studio 2003 was not found on this system. If you have Cygwin installled,
you can try compiling with MingW32, by passing "-c mingw32" to setup.py.

I have Cygwin installed, so I installed Cygwin’s Python 2.5, then used that to run python setup.py install, which worked fine.

However, when I tried to use “Run As … Python unit-test” from within Eclipse (with PyDev installed), it didn’t work. Apparently, there are compatibility problems between PyDev and Cygwin, mostly related to windows path names.

So I went back to trying to install RDFLIB using the Windows Python. I could run python setup.py build -c mingw32 (with gcc-mingw32 installed and cygwin’s binaries directory on my path), but I still couldn’t run python setup.py install because the ‘install’ command doesn’t accept the ‘-c’ argument.

Eventually, I made it work by creating a cfg file for distutils (distutils.cfg) eg: /c/Python2x/Lib/distutils/distutils.cfg containing:

[build]
compiler=mingw32

as described here at the end of the page, under “One Last Step”.

I.e. once I had created the cfg file, and the cygwin binaries were on my path, I could run python setup.py install using the Windows Python, which also works with PyDev in Eclipse.

• http://www.w3.org/TR/2008/WD-skos-reference-20080125/

This is a substantial update to and replacement for the previous SKOS Core Vocabulary Specification W3C Working Draft dated 2 November 2005. The publication has been announced in the W3C news, and a request for comments has been sent to various mailing lists.

The abstract from this new specification:

This document defines the Simple Knowledge Organization System (SKOS), a common data model for sharing and linking knowledge organization systems via the Semantic Web.

Many knowledge organization systems, such as thesauri, taxonomies, classification schemes and subject heading systems, share a similar structure, and are used in similar applications. SKOS captures much of this similarity and makes it explicit, to enable data and technology sharing across diverse applications.

The SKOS data model provides a standard, low-cost migration path for porting existing knowledge organization systems to the Semantic Web. SKOS also provides a light weight, intuitive language for developing and sharing new knowledge organization systems. It may be used on its own, or in combination with formal knowledge representation languages such as the Web Ontology language (OWL).

This document is the normative specification of the Simple Knowledge Organization System. It is intended for readers who are involved in the design and implementation of information systems, and who already have a good understanding of Semantic Web technology, especially RDF and OWL.

For an informative guide to using SKOS, see the upcoming SKOS Primer.

Synopsis

Using SKOS, conceptual resources can be identified using URIs, labeled with lexical strings in one or more natural languages, documented with various types of note, linked to each other and organized into informal hierarchies and association networks, aggregated into concept schemes, and mapped to conceptual resources in other schemes. In addition, labels can be related to each other, and conceptual resources can be grouped into labeled and/or ordered collections.

This short paper summarises some work done on the possibility of using OAIS information model as a basis for the model-driven design and implementation of components within a digital preservation software architecture. Two model transformations were defined using the Enterprise Architect template language. The first model-transformation transforms a platform-independent UML class model (PIM) into a set of UML interfaces specific to the Java 1.5 platform (here called a Java API model). The second model-transformation transforms a platform-independent UML class model (PIM) into set of UML classes specific to the Java 1.5 platform, implementing the interfaces generated by the first model-transformation (here called a Java implementation model). Both were applied to the OAIS information model as PIM, and the generated models are presented here with discussion.

Introduction

The CASPAR project aims to design and implement components of a distributed digital preservation architecture, taking the Open Archival Information System (OAIS) Reference Model as the basis for this architecture. Parts of the OAIS reference model are defined using UML class diagrams. Can these UML class diagrams be used in the design of CASPAR software components? Can they be used directly, or do they need to be refined in some way? What design patterns, principles and methodologies should be used to guide this refinement?

A standard pattern for the design of software systems using UML is to define a platform-independent model (PIM), and then to use model-transformations to generate other models, each specific to a given software platform (e.g. Java or C++). These platform-specific models (PSMs) are then used to generate code in the given language. These models and the resulting software system constitute a model-driven architecture. @@REF

This paper presents some work exploring the feasibility of using the OAIS information model as a platform-independent model in a model-driven design and implementation process, with Java 1.5 as one of the target platforms.

Methods

Enterprise Architect 7 was used as the primary tool for developing UML models and model-transformations. EA7 supports a bespoke template language, which can be used to define and then execute arbitrary model-transformations. EA7 also includes a number of pre-defined model-transformations for specific platforms, including Java. The pre-defined Java transformation preserves most aspects of a UML class model intact, for example it transforms a class into a class and an interface into an interface; it does, however, convert platform-independent data types to Java primitive types, and it adds public getter and setter methods for each attribute and navigable association of a class or interface, following the Java beans pattern.

Although EA7′s pre-defined Java transform provides some useful functionality, it is not sufficient for the CASPAR design process, for several reasons. Firstly, it handles multiplicities of greater than one poorly. Secondly, it does not make use of Java generics. Thirdly, the approach taken so far in CASPAR has been to define two models: an interface model (an API) and an implementation model (a set of classes implementing that API), each with their own distinct package paths. EA7′s pre-defined Java transformation simply converts package to package, class to class and interface to interface, and so cannot be used to generate CASPAR’s two separate models (API and implementation).

To overcome these limitations, two new model-transformations were defined using EA7′s template language. These transformations were based on the pre-defined Java transformation, with a number of modifications. The first transformation converts classes in the PIM into interfaces in a Java API model, in a fixed package (info.preserveddigital.infomodel). The second transformation converts classes in the PIM into classes in a Java implementatio model, in a fixed package (eu.casparpreserves.infomodel). Both transformations include modifications to handle all multiplicities appropriately, including the use of Java generics with Java collections where attributes or associations have multiplicities greater than one.

All class diagrams from section 4.2.1 of the OAIS reference model were then added to Enterprise Architect as a single package within a platform-independent model (PIM). The text of section 4.2.1 was completely ignored in the construction of the PIM, i.e. none of the informal statements made in the text were added as features of the model, so the PIM was a faithful transcription of the diagrams in section 4.2.1 only.

The pre-defined Java transformation, the custom Java API transformation, and the custom Java implementation transformation were then all applied to the PIM, and the output stored in EA.

Results

To give a flavour for the outputs of the two model-transformations, consider the class diagram illustrated in figure 4-10 of the OAIS reference model. Figure 4-10 defines the relationships between data objects, information objects and representation information. This class diagram, as the PIM used here, is given below.

The diagram below was composed from the output of the custom Java API model-transformation. Notice that each interface in the transformation output has getter and setter methods for each of the associations in which the original class was involved. Notice also that the interface RepresentationInformation has getter and setter methods for associations given in the PIM in another diagram (from figure 4-11 in OAIS).

The diagram below was composed from the output of the custom Java implementation model-transformation. Notice that all of the getter and setter methods implementing the corresponding interfaces are present. Notice also that private members for each of these properties are also now present, allowing generation of Java source code with initial code for getter and setter method bodies.

Finally, compare these with the output from EA7′s pre-defined Java transformation.

Discussion

In UML, each connector (link) can have a name, and both source and target roles for that connector can also have a name. However, in the OAIS information model, some links have a name, some don’t, and none of the links have role names. If role names are present, these can be used to give more sensible names to getter and setter methods and private members. However, without role names, the class name of the target type has to be used (e.g. getRepresentationInformation). This is fine when there is only one connector between any two classes, but where there is more than one connector between any two classes, role names are needed to disambiguate between them. Therefore, it is suggested that all connectors in the OAIS information model be given a link name, a source role name (if source is navigable) and a target role name (if target is navigable).

Notice that the class InformationObject has a method getRepresentationInformation():RepresentationInformation, due to the composition association between InformationObject and RepresentationInformation. Notice also that the class RepresentationInformation has a method getRepresentationInformations():java.util.Collection due to the recursive association on that class. The class diagrams in the OAIS information model do not specify that RepresentationInformation is a sub-class of InformationObject, hence this generalization is not present in the PIM used here. However, the text suggests such a generalization. If RepresentationInformation were a sub-class of InformationObject, then the class RepresentationInformation would end up with two methods, both getRepresentationInformation():RepresentationInformation (inherited) and getRepresentationInformations():java.util.Collection. This is due to the fact that the two “interpreted using” connectors have different multiplicities in OAIS. This creates an ambiguity, which would have to be resolved.

This is an example of the type of issue that can be revealed by an analysis of the output of a model-transformation on the OAIS information model as a PIM. It is expected that further analysis of other aspects of the transformation output will reveal similar issues with other classes. Therefore, it is recommended that further study be given to the model-transformation output generated using the transformations described here, and that this be fed back into further iterations of the OAIS information model.

Conclusions

There are potential issues with using the OAIS information model as-is as a platform-independent model in a model-driven design process. The output of model-transformations require inspection and refinement before being suitable for implementation in preservation systems. This is not necessarily a shortcoming of the OAIS information model, but does raise an important question: is the OAIS information model intended to serve as a genuine basis for the implementation of software, or is it merely intended as a guide to understanding?

Further study of the outputs of the model-transformations generated here is recommended. This study is likely to complement the findings of any attempt to re-engineer the OAIS information model as a a formal ontology, which has elsewhere revealed potential ambiguities and inconsistencies therein.

After a not too exhaustive search on the Web, I found PicaJet, and downloaded the free edition. I was encouraged because the tagging interface is great (very similar to Photoshop Album), and because PicaJet has an integrated Flickr uploader which preserves all of your tagging. I also discovered that the photo catalog can be easily exported, so in a nutshell, PicaJet ticks my boxes. You can do quite a lot with the free edition — tag photos, upload to flickr, some basic editing. I’ll be upgrading to PicaJet FX (the full version, around £30) mainly because I want to be able to do more with the tag categories — in the free version you can only have a two-level hierarchy, and you can’t add new top-level categories.

I tried Picasa2, but that doesn’t have any tagging support or Flickr integration.

I also downloaded Microsoft Photo Gallery, which advertises Flickr integration. The installation process was painfully slow, then the application crashed when I tried to launch it on my bog-standard Windows XP machine.

SKOS is a lightweight language for representing intuitive, semi-formal conceptual structures. So, for example, the figure below (taken from the SKOS Core Guide) depicts concepts with intuitive hierarchical and associative relationships to other concepts, and with preferred and alternative labels in one (or more) languages — these are the kinds of structures that can be expressed using SKOS. Once expressed in this form, conceptual structures can easily be published on the Web, shared between applications, linked/mapped to other conceptual structures and so on. Typically, these conceptual structures are used as tools for navigating around complex or unfamiliar subject areas, for retrieving information across languages, and for bringing together related information from different sources.

RDFa is a language for embedding richly structured data and metadata within Web pages. This allows a Web page to expose much of its underlying meaning to applications, enabling a range of new functionalities within Web clients, exchanging data between Web sites, services, and the users’ desktop applications. For example, a Web page about a new music album can use RDFa to embed structured data expressing facts about that album, such as the track listing, artist, links to sample media files etc. A Web browser with a suitable plugin or extension can use this data to offer new functions to the user, such as download the tracklisting with available samples to my music library, or compare prices from online vendors.

Both of these technologies are on the W3C Recommendation track, and are scheduled for completion in April 2008.

SKOS Foundations and Motivations

SKOS inherits much from the development of knowledge organisation systems (KOS) within the library and information sciences. Thesauri, classification schemes, subject heading systems and taxonomies are all examples of KOS widely used in information systems today.

The original motivation for SKOS was to provide a standard, low-cost way of migrating or “porting” existing KOS, especially thesauri, to the Semantic Web, so that they could be used as-is for the development of lightweight Semantic Web applications such as search/browse Web portals. This remains one of the central requirements for current development of SKOS. However, it’s worth noting that SKOS is also increasingly seen as a “bridging” technology, providing the missing link between the rigorous logical formalism of ontology languages such as OWL and the chaotic, informal and weakly-structured world of social approaches to information management, as exemplified by social tagging applications.

As such, SKOS is a very interesting technology to work on, because in the same thought you have to consider the model-theoretic semantics of RDF and OWL (and their consequences for formal reasoning), the ways in which people naturally express and organise their own conceptualisations (especially when working as a collaboration), and the potential for computational processes to “spot” and analyse emergent patterns in networks of unstructured information. The future for knowledge organisation is undoubtedly in highly collaborative, intuitive, and computer-aided environments, where people interact in a natural but structured way, being guided (perhaps unwittingly) towards the creation of emergent structures, supported by and feeding back into a range of analytic systems working behind the scenes to mine, discover and exploit patterns in information. SKOS is a small but important part of this bigger picture, in as much for the work it will lead to in the future as for the applications it can enable today.

SKOS Basics

The basic building block in SKOS is the notion of a conceptual resource, or often simply “concept”. Concepts can be labelled in one or more languages, can be annotated with various types of documentation, can be arranged into intuitive hierarchies and association networks, and can be aggregated into concept schemes and linked/mapped to concepts in other schemes.

All of these SKOS primitive features can be extended or refined to support more detailed, fine-grained conceptual models. SKOS can also be used in part or as a whole in a “mix-and-match” with other RDF vocabularies and OWL ontologies. A concrete example of this is the Semantically Interlinked Online Communities (SIOC) ontology, where SKOS can be “plugged in” to describe the topics or tags defined on a community Web site.

SKOS Design

SKOS is built on the Resource Description Framework (RDF) and the Web Ontology Language (OWL). However, SKOS deliberately hides much of the compexity of these two languages. It provides an interface between the formal underpinnings of the Semantic Web, and the more informal, intuitive ways in which people naturally express and organise knowledge. Thus, informal and semi-formal conceptual structures or knowledge organisation systems can be expressed directly in SKOS and used immediately in Semantic Web applications, without requiring any formal re-engineering.

While these topics are still under debate, consensus is emerging that the formal semantics of SKOS are, by design, very limited. Therefore, a small number of logical consequences follow from using only SKOS, compared with the larger number of logical consequences that follow from using RDFS or especially OWL directly. Whereas in some situations, a powerful set of logical entailments is very valuable, in others this can be inappropriate and/or unecessary. This is the typically the case where a formal language such as OWL is abused to express what is at best a semi-formal conceptualisation (e.g. a “concept map” or thesaurus), and a number of surprising and inappropriate inferences then follow. SKOS provides the option to model at a simpler, less formal level, which is then a starting point for more formalisation as required.

SKOS is itself an RDF vocabulary (i.e. a set of URIs), whose semantics is defined using the RDF Vocabulary Description Language (RDF Schema) and OWL. SKOS can therefore be used as a lightweight conceptual modeling language in its own right, or can be used as an adjunct to the primitives provided by RDF, RDFS and OWL in a “mixed mode” modeling environment.

An example of where SKOS, RDFS and OWL are used in “mixed mode” is the Semantic Web Environmental Directory (SWED), a prototype Web portal with “faceted browsing” functionality for finding UK organisations and projects in in the environment sector. Here, RDFS and OWL are used to model projects, organisations and their properties, such as topic of interest, geographical coverage and so on. SKOS is then used to model the semi-formal taxonomies which provide the descriptive vocabulary for these properties, e.g. animal welfare, welfare of captive animals, biodiversity etc. Of course, this is a Semantic Web application, so information can be drawn together and integrated from many different sources.

SKOS Development

SKOS is formally developed and maintained by the W3C Semantic Web Deployment Working Group (SWDWG). It is a work item on the W3C Recommendation track, which means it is subject to the full W3C Web standardisation process. However, whilst formal responsibility for SKOS rests with the SWDWG, the working group carries out all development in an open, consensus-led environment, and is ably supported by an extended community of interest. Informal participation in the SKOS development process is warmly welcomed — to join in, subscribe to the public-esw-thes@w3.org mailing list (you can also browse the mailing lists’ online archives). To participate formally in the development of SKOS, contact your W3C Advisory Committee representative about joining the SWDWG.

RDFa

From the abstract to the latest version of the RDFa Primer:

Current Web pages, written in XHTML, contain inherent structured data: calendar events, contact information, photo captions, song titles, copyright licensing information, etc. When authors and publishers can express this data precisely, and when tools can read it robustly, a new world of user functionality becomes available, letting users transfer structured data between applications and Web sites. An event on a Web page can be directly imported into a desktop calendar. A license on a document can be detected to inform the user of his rights automatically. A photo’s creator, camera setting information, resolution, and topic can be published as easily as the original photo itself.

RDFa is simply a collection of XML attributes that can be used within an XHTML document to embed structured data within that document. This data can then be extracted by an RDFa parser as a set of RDF triples.

So, for example, the following snippet of XHTML has embedded data describing a person’s contact information. The data has been embedded using the special RDFa attributes @instanceof, @property, @about, @rel and @content.

  <p class="contactinfo" about="http://example.org/staff/jo">
    <span property="contact:fn">Jo Smith</span>.
    <span property="contact:title">Web hacker</span>
    at
    <a rel="contact:org" href="http://example.org">
      Example.org
    </a>.
    You can contact me
    <a rel="contact:email" href="mailto:jo@example.org">
      via email
    </a>.
  </p>

This snippet, when parsed, yields the following RDF triples:

   contact:fn "Jo Smith";
   contact:title "Web Hacker";
   contact:org ;
   contact:email .

Given this data, a Web client could, for example, offer functions to import this contact information into a desktop contact management system.

This is a very simple example, but hopefully it illustrates the general principle that, once data is available in Web pages, new functionality becomes possible. In the e-learning sector, we might imagine Web pages which not only describe historical events, but encode data about the time, place and people involved in those events. A history student might then “cut and paste” these data from many different Web pages into their own virtual learning space, allowing them to discover and explore the many-dimensional relationships between people, places and events and build up their own structured “mini-history” which is specific to a particular learning objective or research question.

RDFa and Microformats

There’s a lively debate ongoing on the Web today about the relationship between RDFa and an analogous technology called “microformats”. Microformats have the same objective as RDFa, of embedding data within Web pages. While I’m not in a position to comment in any depth on the comparisons and relative merits of these two approaches, RDFa developers argue that the RDFa approach provides a more scalable, general purpose approach, which requires only a single implementation, and which allows different types of data to “play well” with each other. On the other hand, a bespoke transformation is required for each different microformat, and microformats could easily “clash” with each other under certain circumstances. Having said that, microformats are used on the Web today, whereas RDFa has only a number of prototype implementations. A blog post by Evan Prodromou discusses the issue in more detail, although some of the information there may be out of date (see e.g. the comments at the end).

Conclusions

Both SKOS and RDFa have interesting consequences for e-learning technology, and especially for leveraging the Web as a platform for delivering learning.

SKOS provides a lightweight technology for overlaying distributed learning content with intuitive conceptual structures, which could for example be used to aid discovery and navigation of learning resources. Conceptual structures are also themselves learning resources in their own right, and although SKOS is oriented towards information retrieval applications, the use of SKOS to express, evolve, exchange and publish “knowledge” as part of a learning process remains an intriguing avenue for exploration.

RDFa provides a general purpose technology for embedding data in Web pages. This in turn can enable a richer experience when interacting with the Web, moving beyond Web pages as unbreakable chunks of information towards Web pages as highly flexible containers of data which can be extracted, adapted, re-used and re-purposed. This technology has the potential to completely change out interaction with the Web as a learning experience.

Both technologies would greatly benefit from active participation from the e-learning community, to ensure that the requirements of near-future learning technologies are met by SKOS and RDFa within the timeline set for standardisation.

One of the difficulties of talking about information is that it is so intangible. A book on a shelf or a file on a computer hard drive are tangible enough, but when we come to talk about a book in abstract (e.g. the complete works of Shakespeare) or a Web page, things become less tangible. One of the problems is that our normal vocabulary for talking about things at these intangible/virtual/abstract levels is not very well developed — e.g. we use “book” to refer both to a physical book on a shelf and the abstract notion of a book (which may have many copies on many shelves).

The Functional Requirements for Bibliographic Records (FRBR) specification goes a long way towards improving this situation, giving us vocabulary for four separate levels of abstraction — Work, Expression, Manifestation and Item. However, these concepts can be ambiguous and hard to apply in some situations. Nevertheless, they are an important reference point.

Web Architecture

On the Web, we face a similar situation, in that the vocabulary for talking about different levels of abstraction and the relationships between them is not well developed. However, the Web is by nature a virtual information space, and the Architecture of the Web provides some very interesting mechanisms for talking about information as a virtual commodity. The Web has also had to deal with the conceptual relationships between an information resource as a virtual entity (i.e. a Web page), and the many different media types (i.e. data formats, e.g. HTML, XML, PDF) and languages (i.e. English, Japanese etc.) which can be used to transmit or carry information.

From the Architecture of the Web, Volume One:

The World Wide Web uses relatively simple technologies with sufficient scalability, efficiency and utility that they have resulted in a remarkable information space of interrelated resources, growing across languages, cultures, and media. In an effort to preserve these properties of the information space as the technologies evolve, this architecture document discusses the core design components of the Web. They are identification of resources, representation of resource state, and the protocols that support the interaction between agents and resources in the space.

According to the Architecture of the Web, the Web is built from “resources”, and in particular, “information resources”. In practice, the notion of an “information resource” is hard to pin down, but an “information resource” is roughly defined as a resource whose essential characteristics can be conveyed in a message. Resources are identified by URIs, and you can use a URI to access a resource. The most common type of access involves requesting a “representation” of a resource, which is also known as “dereferencing a URI”. This is what your Web browser does when you type a URI in the address bar and click “Go”.

Content Negotiation

One of the interesting properties of the Web is the ability to provide several alternative representations in different media types from the same URI, which is supported in the HTTP protocol. This is also known as “content negotiation“. When a user agent (e.g. a Web browser) requests a representation of a resource, it specifies which media types it can “accept” in response. The server then sends a response in the media type best matching the request.

To take a very simple example, the URI “http://www.w3.org/Icons/w3c_main” identifies the main W3C logo icon. This image is available in two different media types — GIF and PNG. You can vary which representation you receive by changing the “Accept” header in the HTTP request.

So the URI “http://www.w3.org/Icons/w3c_main” clearly identifies something virtual, which is “above” (at a higher level of abstraction than) the notion of media type or format. Let’s call this a content negotiable resource.

Media type is not the only axis along which representations of a resource may vary. A multilingual Web site can provide representations in different languages from the same URI. For example, the Debian Web site “http://www.debian.org” provides alternative representations in English, French, Spanish and many other languages — i.e. this is also a content negotiable resource.

So it’s clear that an information resource might be a content negotiable resource — something for which representations might be provided in one or more media types and one or more languages.

However, many information resources in the Web only provide a single representation, in some media type and language. For example, “http://www.w3.org/Icons/w3c_main.png” only provides a PNG representation of the main W3C logo. Let’s call this a content invariant resource.

The interesting thing for our discussion of “versioning” is that we might express a relationship between a content negotiable resource A and a content invariant resource B, where the single representation provided by B is equivalent to one of the representations provided by A.

If we had some appropriate RDF vocabulary, we might even state this formally, e.g.

# links between resources
 rdf:type vif:ContentNegotiable.
 rdf:type vif:ContentInvariant.
 rdf:type vif:ContentInvariant.
 vif:contentVariant .
 vif:contentVariant .
 vif:contentAlternate .

# definition of the vocabulary we used above
vif:ContentNegotiable rdfs:subClassOf vif:InformationResource.
vif:ContentInvariant rdfs:subClassOf vif:InformationResource.
vif:contentVariant rdfs:domain vif:ContentNegotiable; rdfs:range vif:ContentInvariant.
vif:contentAlternate rdfs:domain vif:ContentInvariant; rdfs:range vif:ContentInvariant; rdf:type owl:SymmetricProperty.

If this metadata were made available to an application (user agent), the application could then make a user aware that various alternative representations of some Web resource were available, for example.

Changes Over Time

The other interesting property of the Web architecture is that things can change over time. For example, if you go to “http://www.bbc.co.uk/news” today, you’ll see something different from what was there yesterday. Yet, the URI “http://www.bbc.co.uk/news” identifies an information resource, so this resource must be changing over time. Let’s call this a changeable resource.

On the other hand, some Web resource haven’t ever changed, and are promised to never change. For example, “http://www.w3.org/TR/2004/REC-webarch-20041215/” identifies a time-specific “edition” or “version” of the Architecture of the World Wide Web technical report. Let’s call this an unchangeable resource. However, note that W3C also provides the URI “http://www.w3.org/TR/webarch/“, which always corresponds to the latest version in the report series.

The interesting thing for our discussion of “versioning” is that we might express a relationship between a changeable resource A and an unchanging resource B, where the representation provided B is equivalent to the representation provided by A at some specific point in time. If we had some appropriate RDF vocabulary, we might state this formally, e.g.

# links between resources
 rdf:type vif:Changeable.
 rdf:type vif:Unchanging.
 rdf:type vif:Unchanging.
 vif:snapshot .
 vif:snapshot .
 vif:priorState .

# definition of vocabulary used above
vif:Changeable rdfs:subClassOf vif:InformationResource.
vif:Unchanging rdfs:subClassOf vif:InformationResource.
vif:snapshot rdfs:domain vif:Changeable; rdfs:range vif:Unchanging.
vif:priorState rdfs:domain vif:Unchanging; rdfs:range vif:Unchanging.

If this metadata were made available to a user agent, the application could then make the user aware that a more recent representation of some Web resource was available, or that a history of changes to that resource was available, for example.

Design Patterns

So on the Web, there are content-negotiable resources and content-invariant resources; and there are changeable resources, and unchanging resources. The Web Architecture itself is entirely ambivalent to which is which. However, we might make use of these different classes, to provide some useful functionality to users of digital repositories, accessible through the Web.

Given that any information resource in the Web could be either changeable or unchanging, content-negotiable or content-invariant, there are four different possibilities to consider for every resource. This makes for a fairly complicated set of possible interrelationships. However, we might define some design patterns, which could be useful in particular situations — for example, in exposing information from digital repositories as part of the Web.

We might, for example, describe a three-level pattern.

At the top level is a changeable, content-negotiable information resource.

At the second level are a set of unchanging, content-negotiable information resources. These are less abstract, corresponding to a specific snapshot or “revision” of a resource.

At the bottom level, are a set of unchanging, content-invariant information resources. These are the most contrete entities, corresponding to a specific content variant of a specific snapshot of a resource.

The diagram below in an attempt to illustrate this pattern.

The interesting thing about this pattern is that we might say very concretely how this pattern should be implemented in the Web — i.e. how Web servers could be set up to help user agents understand better the differences and interrelationships between the various information resources. For example, the URI denoting the top level information resource could be set up to redirect to the URI of the most recent snapshot at the second level of information resources; these URIs at the second level could then be set up to content-negotiate directly as per the HTTP protocol, but offering information in the “Content-Location” HTTP header which gives the user agent a clue as to how to link to a specific content variant.

We could also cook up some RDF vocabulary to formally express all of these interrelationships, as shown in the examples above, then publish this in the Web or embed it in web pages. This would allow user agents to be fully aware of the structure of the information space, and do even more intelligent things like make the user aware that he/she is viewing an out-of-date version and that a more recent version is available, or that content variants are available in a range of languages.

Of course, this is only one of a number of possible design patterns, which need further exploration. But hopefully, this at least gives a few ideas as to how the architecture of the Web — a virtual information space — might shed light on a discussion of identifying, describing and linking “versions” of “digital objects”.