It is much to the credit of René van Horik of the Dutch Research Data Archive DANS that he did not just prepare a one-way presentation about the work the European project APARSEN is doing on trust, audit and certification (which would have been the easy way out), but that he brought together a jury of his peers to discuss the project from different points of view. Such an approach takes a lot more work, but it leads to lively and informative confrontations for the audience – and the readers of this blog. From iPRES2012 - by Inge Angevaare
For those new to the APARSEN work on Audit and Certification, you can find a summary (from Screening the Future 2012) here. The EU is building a three-tiered framework for audit and certification of trusted digital repositories in Europe and the APARSEN work on trust feeds into that. The first level is a self-assessment against very broad, general criteria (Data Seal of Approval), then comes a self-assessment against a real ISO standard, ISO 16363, and the highest level is a third-party audit against ISO 16363. The work of APARSEN carries on from the TRAC project by the US Center for Research Libraries.
After René’s introduction, the first speaker from the panel, our host in Toronto Seamus Ross, wasted no time at all to get his point across. “There is no such thing as trustworthiness,” he said. “There is so much involved in running a digital repository, that a) we cannot measure trustworthiness, and b) we cannot maintain it over time.”
So much for a friendly kick-off …
“Instead,” Ross continued, “I believe in a risk management approach. I can measure risks to the security of digital objects – the risk load. And I can measure the risk resilience of an organization. Typically, a small, young organization can adapt easily; an old, large organization, with many more actors in play, will be much slower to react.” Seamus did not mention it, but a good example of such an approach is the Drambora toolkit.
Helen Tibbo, of the U. of North Carolina School of Library and Information Sciences, agreed with much that Seamus Ross said, but, she added, “that is not the whole story.”
Her main argument was with the metrics involved in ISO16363. “For instance, ISO 16363 prescribes that you have to have a mission. But it says nothing about the quality of that mission” (again, see previous post). Helen Tibbo also questioned the role of the auditors. “If auditing becomes a career, what will happen to objectivity?” Tibbo’s comments come from a “trusted” source, because she was part of the team that carried out test audits for the APARSEN project.
The panel went on to discuss research data quality and authenticity. Silvio Salza presented the recommendations of the APARSEN project that, really, the entire process of data creation and all subsequent actions which could possibly harm the data should be monitored. Helen Tibbo was quick to react to that: “Yes, that would be an ideal world. But it is not going to happen. Nobody has the money and staff to do that.” From the audience Bjarne Anderson (Denmark) agreed with Helen. “Counting the rows and the columns is about the best we can do.”
Which prompted session chair Peter Doorn (Director of DANS) to conclude: “The world is messy, the digital world is messier, and the research world is the messiest,” – no doubt thinking of a recent scandal in the Netherlands whereby a psychologist faked all of his survey data and got away with that for years before being caught out.
At the end of the session, Seamus Ross concluded that it had been “fun”, with those different points of view. All agreed, however, that the goal of audit and certification must be to make digital archives perform better. And that, somehow, sometime, we must come up with more objective metrics to benchmark digital archives, whether those be trust-based or risk-based. But preferably automated, embedded in the systems, because if we have to do all of this manually, we will never find enough funding.