More thoughts on linked data and the future of cataloging

I really enjoyed Slistopher's fun post and article link about the venom library.  I wondered about the metadata too, and his suggestion that it could be the project for next year's class cracked me up. But on a more serious note, it prompted me to look up metadata schemas for real, organic things, and it turns out there's a ton of them, this list of schemas being, I am sure, just one of many.  And that got me to thinking about linked data and cataloging in world of linked data.

Suppose you had this book on lemurs:  Behavioral variation : case study of a Malagasy lemur. Current cataloging techniques give it a several LC subject headings (this is from WorldCat; the older record in my library's catalog uses only the first four):

Verreaux's sifaka -- Behavior.
Social behavior in animals.
Mammals -- Behavior.
Mammals -- Madagascar.
Behavior, Animal.
Lemuridae.
Social Behavior.
Mammals.
Madagascar.
Lemuren.
Verhalten.
Madagaskar.
Lemurs

In a linked data world, we would reasonably expect that the basic descriptive metadata (author, publisher, identifiers such as ISBN, etc.) would use some kind of permanent identifiers that would allow them to be meaningfully linked to related content.  But what about those subjects?

In both WorldCat and my library's catalog, the subjects are clickable links, but clicking them results in a search for that subject only within the catalog.  In a more linked environment, would siloized search still be appropriate?  Maybe the subject links could bring in lots more things that are linked to that heading using LC's linked data service (e.g. Lemuridae).  But is that enough?  Will other curators of desirable content use LCSH?  Maybe instead of, or at least in addition to, expecting outside entities to conform to library-centric aboutness terminology, cataloging records should link to data hubs appropriate to the content.  Just one example in this case is the ITIS taxonomy (e.g. Verreaux's sifaka). That would potentially link this book on the social behavior of a specific type of lemur to scientific articles, datasets, images, and perhaps even its genome.  It's not hard to imagine other elements that might link in similar ways:  Madagascar, for example, or perhaps even primate social behavior.

Do you think this kind of outside vocabulary will eventually replace LCSH or will it still be important for broad, general categories like "Social Behavior"?  Will we still need it for faceted, siloized library searches?  Will we still even have the silos?

Perma.cc: Addressing link rot in legal scholarship

I've written several posts on the subject of persistent identifiers, especially DOI, and their importance in maintaining access and enabling permanent cross-referencing and citation.  However, DOI is intended for stable objects, primarily version-of-record scholarly publications (articles, chapters, etc.).  Blog posts, websites, wikis, social media, and similar Internet content can also be useful for research, but authors are forced to fall back on including ordinary URLs when citing these sources. This problem has been specifically documented in legal scholarship--when Harvard Law School researchers recently surveyed the state of legal citations, they made the disturbing finding that "more than 70% of the URLs within the Harvard Law Review and other journals, and 50% of the URLs found within United States Supreme Court opinions, do not link to the originally cited information."

In response, an online preservation service called Perma.cc was "developed by the Harvard Law School Library in conjunction with university law libraries."  Perma.cc works like this:  if an author wants to cite a website or other online source that lacks a permanent identifier, he or she can go to the Perma.cc site and input the URL. Perma.cc downloads and archives the content at that URL and returns a perma.cc link to the researcher, who then uses it in the citation. When the article is submitted for publication to a participating journal, the journal staff check the perma.cc link for accuracy and then "vest" it for permanent archiving. Readers who click the link are taken to a page that offers a choice of the current live page, the archived page (which may not include linked content such as graphics), and an archived screen shot (which will not include live links). Sites that do not want Perma.cc to make their content publicly available can opt out through a metatag or robots.txt file; in these cases, Perma.cc will place the page content in a dark archive, accessible only to the citing author and vesting organization.  Interestingly, Perma.cc still harvests the content, even though it doesn't make it publicly available.

It will be interesting to see if this service (which is still in beta as of this writing) is successful, and if it expands beyond the legal field.  It seems to me to be an excellent companion to DOI, as DOI gets around the problem of copyrighted, toll access content by maintaining only links, rather than archiving content, while Perma.cc provides the same level of stability for freely-accessible but less stable web content.

Identifier persistence

I agree with Madam Librarian that the article A Policy Checklist for Enabling Persistence of Identifiers was convoluted and difficult to follow in places.  For the sake of my own understanding, I'll try to summarize here the basic points I got out of it.  Maybe this will help others in the class as well.

The article presents a set of numbered questions, but then proceeds to address them in a completely different order while attempting to map them to a checklist, which is numbered in yet another style. For the sake of sanity, I will just briefly summarize the article's main points in the order presented, and present examples where it seems useful.

What should I identify persistently?  Analyze your resources, decide which ones can be consistently identified in some way, and then prioritize these identifiable resources.  There will probably be many items that have identifiers, but only a subset of these will require persistence; typically these would represent key access points for your user community.

What steps should I take to guarantee persistence?  This is best handled through policies, supported by automation of processes. Information management should be decoupled from identifier management. In practice, this means that information within a system is identified and managed using local keys--e.g. this could be the URL for a journal article. However, the identifiers for this same information that are shared with outside entities--indexing services and library databases, e.g.--should be based on indirect identifiers, which can be updated when necessary in a way that is invisible to users.

An example of this is an article DOI.  The hypothetical article "36 Tips for Awesomeness" (local identifier) is published in the Spring 2006 issue of Fabulous Journal, given a URL of www.fabulousjournal.com/36_Tips_for_Awesomeness (local identifier), and assigned a DOI of 10.8992/fj.1234 (persistent identifier).  Over the next few years, the journal is bought by another publisher and all content is moved to www.awesomepublisher.com/journal/(ISSN)1000-0001. A year or two after that, the new publisher merges Fabulous Journal with Really Cool Journal, requests a new ISSN, and moves all content to www.awesomepublisher.com/journal/(ISSN)2002-200X. Awesome Publisher has good policies for persistence and updates DOI with each change.  This is the result:

10.8992/fj.1234 initially points to www.fabulousjournal.com/36_Tips_for_Awesomeness

10.8992/fj.1234 then points to www.awesomepublisher.com/journal/(ISSN)1000-0001/36_Tips_for_Awesomeness

10.8992/fj.1234 currently points  to www.awesomepublisher.com/journal/(ISSN)2002-200X/36_Tips_for_Awesomeness

As long as the services that refer to this article use the DOI instead of the article URL, it will remain accessible despite the changes going on in the background.

What technologies should I use to guarantee persistence?  Whichever ones work best with your existing technology and workflow. It's more important that the process works seamlessly and with minimum effort than it is to commit to one specific technology, no matter what That One IT Guy in your division says.

How long should identifiers persist?  The answer to this is, as long as is appropriate, but make sure that you (1) don't promise what you can't deliver (no one can actually guarantee "forever") and (2) are up-front about it ("provisions are in place to guarantee persistence for a minimum of 30 years beyond the online publication date" or "this link will expire in 7 days").

What do you mean by "persistent"?  The article explains that there are degrees of persistence, and breaks them down into a list (I'll use the same article example to explain).

Persistence of Name or Association:
  (1) The title "36 Tips for Awesomeness" will always be associated with that specific article on awesome tips--it won't suddenly be associated with an article on cattle diseases.
 (2)  The article may continue to be referred to in various places as www.fabulousjournal.com/36_Tips_for_Awesomeness even though that URL no longer works. In other words, the association persists in unmaintained places outside the control of the resource owner.
 (3) The article will always be associated with DOI 10.8992/fj.1234, whether or not the publisher updates the DOI information when the article changes location.

Persistence of Service:
 (1) Retrieval:  Can the item still be obtained over the guaranteed time period?  In the case of our article, two of the three listed URLs would eventually fail to retrieve the article, but the DOI should continue to work, resulting in retrieval of the article no matter where it is hosted.
 (2) Resolution:  A URL may resolve without resulting in a successful retrieval. For example, the original author of our 36 tips might get into a copyright dispute with the new publisher, resulting in the article being taken down.  In this case, the publisher might arrange for the URL to resolve to a page with the basic metadata for the article and a brief note about the missing content.  If the URL instead results in a "page not found" error, then it lacks persistent resolution.

Whether a service guarantees retrieval or resolution is an important distinction and should be clearly stated.  Both retrieval and resolution are essential but different.

Persistence of Accountability:
This is mostly for archival purposes. Is some kind of metadata maintained that gives the history of who has created and edited a specific record?

TL;DR:  Persistent identifier policies in an information management environment should clearly outline the following:  which identifiers will be persistent, how persistence will be maintained, how long the user can expect persistence to last, and whether persistence guarantees access to a specific item (retrieval) or guarantees access to (at minimum) information about that item (resolution).

DOI and CrossRef explained

"DOI: The 'Big Brother' in the dissemination of scientific documentation," by Miquel Termens, does an excellent job of explaining the reasons and organizations behind DOI (Digital Object Identifiers). The article also gives a clear, concise overview of the inner workings of CrossRef.  (CrossRef, as their site explains, "... allows the user to move from one article to another at the citation level, regardless of journal or publisher.") I would recommend this article for anyone interested in cross linking or the mechanics of access to scholarly publishing.

Although the article is now 8 years old, not too much appears to have changed, other than how DOI is used in the e-book world. DOI is now typically used for scholarly e-books in much the same way it is used for e-journals:  assigned at both title and chapter level by those scholarly publishers who host DRM-free PDF e-book versions on their sites (e.g. Springer, Elsevier ScienceDirect, Wiley, JSTOR, and others).  In adding book chapter DOIs, publishers can take advantage of reference linking (CrossRef et al.), facilitate user access through index and discovery systems, and allow for limited resource sharing (ILL and scholarly sharing at the chapter level).

This quote from the article illustrates the importance of DOI in the scholarly publishing chain and explains the dismay of everyone involved when administrative error temporarily brought the dx.doi.org domain down earlier this week:

The Handle System offers an additional improvement—the working URL is always operative and does not require maintenance. URL obsolescence, and the consequently large number of broken links, is one of the major problems in present day internet structure and it slows down the allocation of links between websites [10]. Handle System provides a solution because, for each, document, it generates a persistent address, a URN, which can be used to locate it. DOI codes can operate as URNs if they are preceded by an address resolution server, e.g., [http://dx.doi.org/10.1126/science.1088234]; hence, the reader does not need to be concerned about a link’s operating state as this is attended to by the publisher of the referenced journal by maintaining an up-to-date URL in the DOI databases.

Breaking the chain: the fragile web of metadata

I can still hear you sayin'
You would never break the chain
(Never break the chain)

Fleetwood Mac - The Chain Lyrics | MetroLyrics 

One of the amazing things about working with electronic resources in the library environment is how easy we can make things for our users.  Back when I was an undergrad, in what now seems like the Jurassic Period, researching a topic meant going to the library and using the card catalog to find books, then venturing into the reference room to confront the vast shelves of periodical indexes.  It was a time-consuming process that involved checking for the same topics in volume after volume of the same index, writing down citations, and then hoping that (1) your library had the title and (2) someone hadn't stolen or defaced the issue you needed (the dreaded microfilm format had its own problems).

Now, thanks to the Internet and incredible advancement in online indexes and publishing, today's researcher can go to a library's databases, research a topic, and be presented with appropriate citations.  Many of these citations will link directly to the article; the rest will have a link to the library's link resolver, which will offer links to the article, or to the library's catalog (to look for a print copy), ILL system, or other appropriate services.

This link resolver software (explained in detail here) is, of course!, metadata-driven, and relies on specific information to form the link that is presented to the user.  This metadata may include journal title, ISSN, article title, volume/issue/page number, or other information, but frequently will instead be based on the Digital Object Identifier (DOI) assigned to the article. In theory, this is a highly stable way of creating links, as the DOI should remain the same even if the article moves to another location.  Let me reiterate that:  in the unstable world of electronic journal  publishing, DOI is a persistent link that solves many of the problems that happen when journals change publishers, platforms, or URL.

This is an example of a DOI-based link:

http://ezproxy.lib.calpoly.edu/login?url=http://dx.doi.org/10.1017/S1368980008002152?nosfx=y

This takes our users to this article:

 Yeh-Chung Chien, Ya-Jing Huang, Chun-Sen Hsu, Jane C-J Chao and Jen-Fang Liu (2009). Maternal lactation characteristics after consumption of an alcoholic soup during the postpartum ‘doing-the-month’ ritual. Public Health Nutrition, 12, pp 382-388. doi:10.1017/S1368980008002152. 

So, here's the part that isn't so widely advertised, but should surprise no one:  library link resolver technology isn't perfect.  In fact, I get dozens of emails every day from users reporting broken links.  Sometimes this is user error or confusion, especially when we can't link to the article level (not every publisher has the necessary infrastructure ... or provides the necessary metadata), but sometimes it's obvious that something is seriously wrong.

This last Tuesday, January 20, I suddenly saw a huge upsurge of broken link reports.  It didn't take much detective work to see that they were all DOI-based ... and every link went to a "not found" page for doi.org.  There was nothing we could do (reconfiguring our entire link resolver was not an option), and no way our users could reach any of that content except by manually browsing to the citation they needed using our journals A-Z list and drilling down through the publisher's site.

So what happened?  It turns out that the dx.doi.org domain name didn't get renewed.  The entire site went dark, and broke all of the services that rely on it, including CrossRef, FigShare, and many others, among them library link resolvers.  The mistake was discovered within a few hours, and the service was brought back up, but it took several more hours before the restored domain had fully propagated through the Internet.  There is a good write-up about the incident on the CrossRef blog, but this is the most important point from it:

For all the redundancy built into our systems (multiple servers, multiple hosting sites, Raid drives, redundant power), we were undone by a simple administrative task. 

Conclusion:  metadata is necessary, and wonderful, and accuracy is essential, but remember that the systems that rely on it are fragile, and never forget the potential for human error.