Take a look through ISO-15489 and see how many of its requirements are directed at meeting the needs of ‘the user’. Now I’d hesitate to say “none” as its quite possible that I’ve overlooked one or two, but I think I’m pretty safe in suggesting “precious few” as an accurate answer. Reading through it soon becomes clear who records management’s primary stakeholder is assumed to be – and its clearly not the user with virtually every recommendation defined in terms of what “the organisation” requires and what is in “the organisations” best interests.
This probably doesn’t come as a great surprise, after all RM has long strived to be acknowledged as an established ‘corporate function’ with an enterprise-wide remit. Indeed all the benefits that RM can offer as stated in Part 1 of 15489 are described by the way in which they “enable organisations to…”.
Most RM technologies follow this lead and seek to deliver benefits to ‘the organisation’. But where does this leave the individual users of which the organisation is comprised? Can we automatically assume that what is in the best interests of the organisation will also be so for its staff? Taken to its logical conclusion the answer must inevitably be yes; after all the organisation that fails to make a profit or continually finds itself in the law courts will soon find itself unable to pay the salaries of its present staff or the pensions of its former. But how strong a connection is this really seen by many? Might it be that the vast majority of staff simply wants to turn up in the morning, complete their allotted tasks as quickly and easily as possible and leave at a reasonable time in the evening with as few complications and hurdles as possible?
Few record managers have the authority to compel users to adopt the procedures or systems they introduce. Instead we rely on a mixture of inspiration and perspiration to encourage adoption – with varying degrees of success. My review of the existing literature is still in progress but from what I have read so far a recurring theme when it comes to RM system implementation failures is lack of staff engagement. As Rachel Maguire stated in her ‘Lessons learnt from implementing an electronic records management system’ (Records Management Journal, Vol 15 No.3, pp.150-7)
“In spite of extensive training, most staff never got to grips with the system”
From the literature and anecdotal evidence this seems a common occurrence, but why should it be so? Is it that the solutions we offer simply don’t meet the requirements of our users? Do we even know what their requirements are? Or have we been guilty of paying lip-service to such considerations whilst instead focusing our attention of trying to deliver solutions which benefit the organisation entire but at the users’ expense?
Outside of the RM sphere disciplines such as HCI (Human-computer interaction) and User Centered Design are well established and important aspects of the design and implementation of IT systems and technologies and help to deliver devices and applications that users actively want to use, partially because they have had a very real and active role in the design process and because this has led to solutions that make their lives a little bit easier. Myself and Jay Vidyarthi (a Human-Computer Interaction specialist from Canada) wrote a paper for the Records Management Journal (Human-computer interaction: the missing piece of the records management puzzle?, RMJ, Vol. 20 No.3, 2010) looking into some of these very issues but inevitably that only scratched the surface.
Now as part of my role at JISC infoNet I am hoping to build on this work by producing a resource which seeks to redress this balance and to find ways of integrating aspects of HCI and User Centred Design into the design and implementation of information management technologies. As such I would be really grateful to hear of any examples that you know of or have been involved with of information or records management projects which may have explored any of these issues and sought to apply these principles; or (perhaps more likely) to hear of projects which have stumbled or failed due at least in part to failings in this area.
Of course none of this is meant to suggest that ‘the organisation’ doesn’t matter, or that we should lose sight of the bigger picture whilst attempting to solve the specific problems of every member of staff – but merely to recognise that without the genuine and positive engagement of the latter we will never truly be able to serve the interests of the former.
Monday, 23 May 2011
Monday, 18 April 2011
Calculating the impact of the Impact Calculator
Back in October 2009 I posted on the launch of our new Impact Calculator, a tool designed to address the worrying dearth of reliable evidence supporting the various claims that are often made in favour of investment in records management (‘increasing productivity’, decreasing overheads etc’). Well, here we are, 18 months, 270 registered downloads (and lots more unregistered) of the Impact Calculator, 6 completed pilot projects and 1 published research paper later.
Hopefully, the paper in question – based on the findings from 6 UK universities all of whom used the Impact Calculator to measure the return on investment (or not) realised through improvements to records management – within the Records Management Journal (Vol. 21 No.1, 2011) will finally drive a stake in the heart of that oft-quoted but seemingly completely mythical previous ‘source’ of ‘data’: “the Coopers & Lybrand study” which has been doing the rounds of lazy records management presenters and authors for about the past decade!
Of course data based solely on the experience of 6 universities is always going to be limited in terms of its broader applicability and the evidential weight we can place upon it, but at least their methodology and the workings of the calculator are transparent and consistent and therefore a considerable step forward on what has gone before.
And the conclusions from this work? Well, there are several and its difficult to summarise them adequately in a blog post, but the following statements all feature in my RMJ paper’s conclusion:
- There is no single absolute threshold of data purity and evidential rigour appropriate when measuring the impact of records management
- the cost effectiveness of records management is closely linked to questions of volume
- measuring impact represents a challenging new role for records professionals
- There are pros and cons to judging the impact of records management in terms of the tangible benefits realised
And the following statements, also based on the outputs from the same pilot projects have also featured in presentations on the Impact Calculator that I have given to post-graduate students at both University of Aberystwyth and San Jose University, California:
- retrospective appraisal projects rarely deliver a financial return on investment
- implementing a retention schedule ‘from this point on’ can deliver cost savings
- RM is only cost effective above a certain scale of operation
- investment in better processes and systems is more cost effective than increased reliance on cheaper labour
Although the full text of my RMJ paper is only available via the Records Management Journal from Emerald Publishing, the full results of the 6 pilot projects with commentary is publicly available from the JISC infoNet website.
As noted above, the jury is still out as to whether seeking to measure the impact of records management purely in statistical terms is a good thing or not and certainly there are lots of other reasons (legal, regulatory, historical etc) as to why it makes sense to appraise legacy records even if the evidence suggests that it will never recoup the money you spend on doing it. But at least now there is a tool and the beginnings of an evidence base out there for those who do feel that the business case for investment in records management within their organisation would be strengthened through the addition of some relevant and reliable data regarding its financial worth.
Hopefully, the paper in question – based on the findings from 6 UK universities all of whom used the Impact Calculator to measure the return on investment (or not) realised through improvements to records management – within the Records Management Journal (Vol. 21 No.1, 2011) will finally drive a stake in the heart of that oft-quoted but seemingly completely mythical previous ‘source’ of ‘data’: “the Coopers & Lybrand study” which has been doing the rounds of lazy records management presenters and authors for about the past decade!
Of course data based solely on the experience of 6 universities is always going to be limited in terms of its broader applicability and the evidential weight we can place upon it, but at least their methodology and the workings of the calculator are transparent and consistent and therefore a considerable step forward on what has gone before.
And the conclusions from this work? Well, there are several and its difficult to summarise them adequately in a blog post, but the following statements all feature in my RMJ paper’s conclusion:
- There is no single absolute threshold of data purity and evidential rigour appropriate when measuring the impact of records management
- the cost effectiveness of records management is closely linked to questions of volume
- measuring impact represents a challenging new role for records professionals
- There are pros and cons to judging the impact of records management in terms of the tangible benefits realised
And the following statements, also based on the outputs from the same pilot projects have also featured in presentations on the Impact Calculator that I have given to post-graduate students at both University of Aberystwyth and San Jose University, California:
- retrospective appraisal projects rarely deliver a financial return on investment
- implementing a retention schedule ‘from this point on’ can deliver cost savings
- RM is only cost effective above a certain scale of operation
- investment in better processes and systems is more cost effective than increased reliance on cheaper labour
Although the full text of my RMJ paper is only available via the Records Management Journal from Emerald Publishing, the full results of the 6 pilot projects with commentary is publicly available from the JISC infoNet website.
As noted above, the jury is still out as to whether seeking to measure the impact of records management purely in statistical terms is a good thing or not and certainly there are lots of other reasons (legal, regulatory, historical etc) as to why it makes sense to appraise legacy records even if the evidence suggests that it will never recoup the money you spend on doing it. But at least now there is a tool and the beginnings of an evidence base out there for those who do feel that the business case for investment in records management within their organisation would be strengthened through the addition of some relevant and reliable data regarding its financial worth.
Monday, 6 December 2010
Records management: the plasterer's hammer?
“For a field largely reliant on the active participation of the individual users responsible for creating, using and managing records to achieve its aims, much of records management appears sorely lacking in the depth and sophistication of its knowledge about those same user s, their needs and objectives”.
So begins the conclusion of the paper written by myself and Jay Vidyarthi and published in the latest volume of the Records Management Journal. (Vol 20, No 3)
The paper discusses the way in which records management has focused almost exclusively and to the exclusion of virtually all other considerations on the needs of ‘the organisation’ often to the detriment of the users we are so reliant upon. Records management is a discipline which strives for standardisation, consistency and uniformity; for example in the form of functional classification schemes attempting to map activities across the entire organisation with a view to constructing a ‘corporate file plan’ or shared metadata schemas. This drive to standardisation isn’t just evident within organisations, but across them – be it in the guise of ISO 15489 or any of the specification standards for an EDRMS – all of which have at their heart the desired goal of uniformity of approach.
Read any section of 15489 and it’s abundantly clear who the main beneficiary of records management is intended to be – and its not the user. Virtually every section defines its objectives in terms of the benefits it will provide to ‘the organization’ with the user(s) getting barely a mention. Now none of this may strike the user as particularly surprising, nor in any way negative. After all, records management has long strived to be acknowledged as a ‘corporate function’ alongside HR, finance etc and clearly many of the drivers for it (accountability, governance, regulation etc) tend to apply at the organisational, rather than the individual level.
None of this is intended to criticise, but to shed some light on why it is that records management often struggles to satisfy the requirements of the individual users it relies on for success and why it could be argued that it has given up even trying. At its most extreme this disparity between the design of many records management systems and the needs of the individual user is most succinctly summed up in a quote made by one EDRMS user to me once that ‘making me use an EDRMS is like asking a plasterer to use a hammer’!
This clearly puts records management and the technology we rely on to implement it (whatever that technology may be) in something of a quandary. Is it really possible for it to successfully serve two equally demanding masters? Can we really hope to find ways of meeting the myriad, highly specific, highly personal demands of our user community in a way which not only pleases each individual user but also in a way which continues to meet the obligations and interests of the organisation as a whole?
Carry on as we are and I fear the answer will continue to be ‘no’; but open our eyes and ears to some radical new perspectives and it could yet be a ‘yes’. Human-Computer Interaction, or HCI is a combination of computer science, cognitive psychology, sociology, information science and design which might just represent the ‘missing piece of the puzzle’. A blog post doesn’t provide the space to explore the detail – that’ what Jay and I start to do in the RMJ paper. Here it suffices to describe it as a structured approach which puts the users first to ensure that they can interact with the system in ways which meet their needs whilst also continuing to meet the needs of the organisation. By shining a light on the behaviour, needs, opinions, tendencies and motivations of end-users it’s the first step towards achieving truly effective records management systems. After all, give somebody a tool that patently saves them time, energy and frustration and they would be foolish not to embrace it; but so too we must acknowledge that the reverse is true and that to try to make somebody use a tool that promises to only help someone (or something) else but at their own personal expense and surely we must concede that they would be a fool to use it.
The implications of such a shift in emphasis are profound, for records management as traditionally conceived is a house built from the top down determined by the needs of the organisation, and not one built from the bottom up based on the needs of its users. But it also offers some tantalising prospects: not just RM systems that users actively want to engage with, but also the possibility that we could start to use this new found knowledge of user behaviour to design and create records management systems that can actually manage records ‘automatically’ (at least in part) based on this behaviour – in a way similar to that used by Amazon et al to organise their content to aid the user experience. Desirable? Definitely. Possible? Who knows, but what this space…
So begins the conclusion of the paper written by myself and Jay Vidyarthi and published in the latest volume of the Records Management Journal. (Vol 20, No 3)
The paper discusses the way in which records management has focused almost exclusively and to the exclusion of virtually all other considerations on the needs of ‘the organisation’ often to the detriment of the users we are so reliant upon. Records management is a discipline which strives for standardisation, consistency and uniformity; for example in the form of functional classification schemes attempting to map activities across the entire organisation with a view to constructing a ‘corporate file plan’ or shared metadata schemas. This drive to standardisation isn’t just evident within organisations, but across them – be it in the guise of ISO 15489 or any of the specification standards for an EDRMS – all of which have at their heart the desired goal of uniformity of approach.
Read any section of 15489 and it’s abundantly clear who the main beneficiary of records management is intended to be – and its not the user. Virtually every section defines its objectives in terms of the benefits it will provide to ‘the organization’ with the user(s) getting barely a mention. Now none of this may strike the user as particularly surprising, nor in any way negative. After all, records management has long strived to be acknowledged as a ‘corporate function’ alongside HR, finance etc and clearly many of the drivers for it (accountability, governance, regulation etc) tend to apply at the organisational, rather than the individual level.
None of this is intended to criticise, but to shed some light on why it is that records management often struggles to satisfy the requirements of the individual users it relies on for success and why it could be argued that it has given up even trying. At its most extreme this disparity between the design of many records management systems and the needs of the individual user is most succinctly summed up in a quote made by one EDRMS user to me once that ‘making me use an EDRMS is like asking a plasterer to use a hammer’!
This clearly puts records management and the technology we rely on to implement it (whatever that technology may be) in something of a quandary. Is it really possible for it to successfully serve two equally demanding masters? Can we really hope to find ways of meeting the myriad, highly specific, highly personal demands of our user community in a way which not only pleases each individual user but also in a way which continues to meet the obligations and interests of the organisation as a whole?
Carry on as we are and I fear the answer will continue to be ‘no’; but open our eyes and ears to some radical new perspectives and it could yet be a ‘yes’. Human-Computer Interaction, or HCI is a combination of computer science, cognitive psychology, sociology, information science and design which might just represent the ‘missing piece of the puzzle’. A blog post doesn’t provide the space to explore the detail – that’ what Jay and I start to do in the RMJ paper. Here it suffices to describe it as a structured approach which puts the users first to ensure that they can interact with the system in ways which meet their needs whilst also continuing to meet the needs of the organisation. By shining a light on the behaviour, needs, opinions, tendencies and motivations of end-users it’s the first step towards achieving truly effective records management systems. After all, give somebody a tool that patently saves them time, energy and frustration and they would be foolish not to embrace it; but so too we must acknowledge that the reverse is true and that to try to make somebody use a tool that promises to only help someone (or something) else but at their own personal expense and surely we must concede that they would be a fool to use it.
The implications of such a shift in emphasis are profound, for records management as traditionally conceived is a house built from the top down determined by the needs of the organisation, and not one built from the bottom up based on the needs of its users. But it also offers some tantalising prospects: not just RM systems that users actively want to engage with, but also the possibility that we could start to use this new found knowledge of user behaviour to design and create records management systems that can actually manage records ‘automatically’ (at least in part) based on this behaviour – in a way similar to that used by Amazon et al to organise their content to aid the user experience. Desirable? Definitely. Possible? Who knows, but what this space…
Wednesday, 25 August 2010
Is the Cloud aware that it has 'the future of digital archiving in its hands'?
As anyone in the audience at the ECA conference in Geneva earlier this year will be aware, one of issues which I’ve been mulling over in recent months relates to roles and responsibilities in ‘the cloud’. The question I was asked to address in Switzerland was ‘in whose hands does the future of digital preservation lie?’ and my succinct response was: ‘Google's’. This was (for reasons evident in the paper I gave) meant both literally – given their increasing dominance of the cloud space but also metaphorically, as an encapsulation of all cloud service providers.
And certainly when my colleague, Doug Belshaw, pointed me in the direction of this post regarding Facebook’s archiving policy it became clear that I’m not the only one thinking about the (unintended?) consequences for all parties of where this might lead us.
Its tempting to see things only from our (by that I mean the archival community) side of the fence – to lament the inevitable decline in our future professional role that the handing over of content to commercial external service providers for its long term preservation will entail and to worry about what it may mean for the archives (and their users) of the future.
But maybe we should also pause to reflect on what it may mean for these service providers themselves and whether they actually have as much concern about the implications of this new found responsibility on their side as we do on ours.
For as I concluded my paper in Geneva:
"Perhaps we should actually stop to ask Google and their peers whether they are indeed aware of the fact that the future of digital preservation lies in their hands and the responsibilities which comes with it and whether this is a role they are happy to fulfil. For perhaps just as we are in danger of sleepwalking our way into a situation where we have let this responsibility slip through our fingers, so they might be equally guilty of unwittingly finding it has landed in theirs.
If so, might this provide the opportunity for dialogue between the archival professions and cloud based service providers and in doing so, the opportunity for us to influence (and perhaps even still directly manage) the preservation of digital archives long into the future".
To again quote from the conclusion of my paper:
"Maybe the interconnection of content creation and use and its long term preservation need not be as indivisible within the cloud as it might first appear. Yes Google’s appetite for content might appear insatiable, but that does not necessarily mean that they wish to hold it all themselves – after all, their core business of search does not require them to hold themselves every web page they index, merely to have the means to crawl it and to return the results to the user. Might we be able to persuade them that the same logic should also apply to the contents of Google Apps, Blogger, YouTube and the like? If so, might the door be open for us, the archival community through the publicly funded purse to create and maintain our own meta-repository within which online content can be transferred, or just copied, for controlled, managed long term storage whilst continuing to provide access to it to the services and companies from which it originated?
That way they get to continue to accrue the benefit of allowing their users to access and manipulate digital content in ways which benefit their bottom line, the user continues to enjoy the services they have grown accustomed to and the archival community can sleep soundly, safe in the knowledge that whilst service providers are free to do what they want with live content, its long term preservation and safety continues to lie in our own experienced and trusted hands".
I wonder if such dialogue is already occurring between Google, Facebook et al and the likes of NARA, NAA and TNA. Lets hope so…
And certainly when my colleague, Doug Belshaw, pointed me in the direction of this post regarding Facebook’s archiving policy it became clear that I’m not the only one thinking about the (unintended?) consequences for all parties of where this might lead us.
Its tempting to see things only from our (by that I mean the archival community) side of the fence – to lament the inevitable decline in our future professional role that the handing over of content to commercial external service providers for its long term preservation will entail and to worry about what it may mean for the archives (and their users) of the future.
But maybe we should also pause to reflect on what it may mean for these service providers themselves and whether they actually have as much concern about the implications of this new found responsibility on their side as we do on ours.
For as I concluded my paper in Geneva:
"Perhaps we should actually stop to ask Google and their peers whether they are indeed aware of the fact that the future of digital preservation lies in their hands and the responsibilities which comes with it and whether this is a role they are happy to fulfil. For perhaps just as we are in danger of sleepwalking our way into a situation where we have let this responsibility slip through our fingers, so they might be equally guilty of unwittingly finding it has landed in theirs.
If so, might this provide the opportunity for dialogue between the archival professions and cloud based service providers and in doing so, the opportunity for us to influence (and perhaps even still directly manage) the preservation of digital archives long into the future".
To again quote from the conclusion of my paper:
"Maybe the interconnection of content creation and use and its long term preservation need not be as indivisible within the cloud as it might first appear. Yes Google’s appetite for content might appear insatiable, but that does not necessarily mean that they wish to hold it all themselves – after all, their core business of search does not require them to hold themselves every web page they index, merely to have the means to crawl it and to return the results to the user. Might we be able to persuade them that the same logic should also apply to the contents of Google Apps, Blogger, YouTube and the like? If so, might the door be open for us, the archival community through the publicly funded purse to create and maintain our own meta-repository within which online content can be transferred, or just copied, for controlled, managed long term storage whilst continuing to provide access to it to the services and companies from which it originated?
That way they get to continue to accrue the benefit of allowing their users to access and manipulate digital content in ways which benefit their bottom line, the user continues to enjoy the services they have grown accustomed to and the archival community can sleep soundly, safe in the knowledge that whilst service providers are free to do what they want with live content, its long term preservation and safety continues to lie in our own experienced and trusted hands".
I wonder if such dialogue is already occurring between Google, Facebook et al and the likes of NARA, NAA and TNA. Lets hope so…
Thursday, 1 July 2010
Wisdom of the citizen?
Very interesting to see the new government initiative launched today by Nick Clegg which calls on the public to help decide which laws they want scrapped.
Now I should point out from the beginning that I'm not passing any political judgement here on its relative merit. What interested me was how the proposal reflects the whole 'wisdom of the crowd' ethos which as many of you will know I have long been advocating as a model for ensuring records management.
The idea is explained more fully in the Daily Telegraph and elsewhere, but in summary (and to quote from Nick Clegg)
"Today we are taking an unprecedented step. Based on the belief that it is people, not policy makers, who know best, we are asking the people of Britain to tell us how you want to see your freedom restored"
Certainly echoes there of my belief that the creators and users of records are often far better placed than Records Managers to understand their records and how we should be looking for innovative ways of extracting this knowledge and using it for management purposes.
And how is this to be done? To quote from Nick Clegg again:
"We are calling on you for your ideas on how to protect our hard-won liberties and repeal unnecessary laws... we're hoping for virtual mailbags full of suggestions. Every single one will be read, with the best put to Parliament"
Again, interesting to see an example of how technology can now be employed to gather and quantify information from a large cross-section of interested people and then used to inform the deliberations of those whose formal role it is to make such decisions.
Now whether this is just political gimmick or a genuine attempt at change is not for me to comment on. But as a high profile example of how technology now has the potential to empower individuals (be they 'citizens' or 'users') and how decision makers can and should now make use of such decision does seem worthy of comment.
Now I should point out from the beginning that I'm not passing any political judgement here on its relative merit. What interested me was how the proposal reflects the whole 'wisdom of the crowd' ethos which as many of you will know I have long been advocating as a model for ensuring records management.
The idea is explained more fully in the Daily Telegraph and elsewhere, but in summary (and to quote from Nick Clegg)
"Today we are taking an unprecedented step. Based on the belief that it is people, not policy makers, who know best, we are asking the people of Britain to tell us how you want to see your freedom restored"
Certainly echoes there of my belief that the creators and users of records are often far better placed than Records Managers to understand their records and how we should be looking for innovative ways of extracting this knowledge and using it for management purposes.
And how is this to be done? To quote from Nick Clegg again:
"We are calling on you for your ideas on how to protect our hard-won liberties and repeal unnecessary laws... we're hoping for virtual mailbags full of suggestions. Every single one will be read, with the best put to Parliament"
Again, interesting to see an example of how technology can now be employed to gather and quantify information from a large cross-section of interested people and then used to inform the deliberations of those whose formal role it is to make such decisions.
Now whether this is just political gimmick or a genuine attempt at change is not for me to comment on. But as a high profile example of how technology now has the potential to empower individuals (be they 'citizens' or 'users') and how decision makers can and should now make use of such decision does seem worthy of comment.
Sunday, 2 May 2010
Is digital preservation now routine?
It’s been a while since I attended a conference specifically themed around digital preservation / electronic archiving and having spent a few days last week in Geneva at the excellent European Conference on Archiving I was struck by the change. Not many years ago such conferences were dominated by debate about the technical complexities it posed, about the relative merits of competing theoretical approaches such as emulation and migration and the risks we faced if and when we got it wrong. The fragility of digital media was stressed and compared unfavourably to the durability of their traditional counterparts (encapsulated by seemingly endless comparisons between the original Domesday Book and its 1980s electronic equivalent).
I heard none of this at ECA, at least not in the sessions I attended or the conversations I was party to. Instead there were plenty of case studies from around Europe of organisations who are quietly and successfully getting on with it. On the evidence of the past few days we seem to have found ourselves in the situation where our ability to actually preserve this stuff indefinitely and to continue to provide access to it seems, without much triumph or fanfare to now be taken as read. This is not, of course, the same as saying that no more problems or challenges exist, but they seem to be of a more prosaic, ‘routine’ nature revolving around the need to secure budgets and improve the user experience etc.
More interestingly still if a single concern dominating the conference can be identified it seemed to be one related to the volume of information being created and stored today and estimated to be created tomorrow. I lost count of the number of presentations which contained jaw-dropping predictions of the amount of data soon to be at our fingertips and the challenges this will pose in terms of resource discovery, legal discovery and overall management. But interestingly virtually never its preservation. So, based on the evidence of this conference alone, it seems as though within a few short years we have jumped from a situation where we used to worry obsessively that we were in danger of losing everything to one where we now stress about how to manage a world where we will lose nothing, with barely a pause for reflection on this change.
My other observation was that (my own meagre contribution to one side) there was little or no discussion about the growing impact of the web as a storage ‘repository’ as heralded by the rise of ‘the cloud’. The unspoken assumption behind most of the debate and the projects and initiatives they represented seemed to be that these organisations will always have physical control of the electronic information they wish to preserve both prior to and after its ingest into their electronic archive, but as I tried to stress in my paper, I wonder how safe an assumption that will prove to be?
I heard none of this at ECA, at least not in the sessions I attended or the conversations I was party to. Instead there were plenty of case studies from around Europe of organisations who are quietly and successfully getting on with it. On the evidence of the past few days we seem to have found ourselves in the situation where our ability to actually preserve this stuff indefinitely and to continue to provide access to it seems, without much triumph or fanfare to now be taken as read. This is not, of course, the same as saying that no more problems or challenges exist, but they seem to be of a more prosaic, ‘routine’ nature revolving around the need to secure budgets and improve the user experience etc.
More interestingly still if a single concern dominating the conference can be identified it seemed to be one related to the volume of information being created and stored today and estimated to be created tomorrow. I lost count of the number of presentations which contained jaw-dropping predictions of the amount of data soon to be at our fingertips and the challenges this will pose in terms of resource discovery, legal discovery and overall management. But interestingly virtually never its preservation. So, based on the evidence of this conference alone, it seems as though within a few short years we have jumped from a situation where we used to worry obsessively that we were in danger of losing everything to one where we now stress about how to manage a world where we will lose nothing, with barely a pause for reflection on this change.
My other observation was that (my own meagre contribution to one side) there was little or no discussion about the growing impact of the web as a storage ‘repository’ as heralded by the rise of ‘the cloud’. The unspoken assumption behind most of the debate and the projects and initiatives they represented seemed to be that these organisations will always have physical control of the electronic information they wish to preserve both prior to and after its ingest into their electronic archive, but as I tried to stress in my paper, I wonder how safe an assumption that will prove to be?
Friday, 5 March 2010
'Big data': big potential, big challenges
This week’s Economist has an excellent special report on managing information entitled ‘Data, Data everywhere’. It looks at the changes, opportunities and challenges posed by our new found ability to create and manipulate vast quantities of data – big data. There are lots of impressive/daunting (depending on your point of view) statistics about just how much data we are now talking about (40 billion photos on Facebook for example) during this “industrial revolution of data”. It also explores the concept of ‘data exhaust’ the trail of clicks which users leave behind them and which Google and others have been able to put to such incredible use: from search to speech recognition and from spell checking to language translation. All made possible not by attempting to training computers the rules which determine how these concepts work, but instead by tracking the activities of billions of user transactions which do the work of refining, correcting and adding relative value to words. Those who have heard my ‘Meet the future of Records Management: Amazon.com’ conference paper will know that I have long suspected that we could and should be making use of this exact same ‘exhaust’ to help us manage information, as well as profit from it - what I describe as 'Automated Records Management' (See also Records Management Journal Vol 19 No.2 2009 for a paper I wrote on this entitled 'Forget Electronic Records Management its Automated Records management that we desperately need')
There’s also interesting stuff in the Economist supplement on the problems of how to make sense of all this data, including new ways of visualising it and the prediction that statistics will soon be one of the coolest jobs around(!). It also makes some interesting points about the need for management to be trained in how to make sense of all this data. This chimes with a conversation I had with a Chief Exec a few weeks ago who also made the case for ensuring that senior management were aware of good old fashioned archival concepts such as provenance and context to give them a better appreciation of what the data they are looking at is actually telling them or how much it can be relied upon (rather than what they wish it was telling them and how much faith they may wish to place in it).
To give the Economist its due it does also look beyond the potential and address some of the challenges (and not just in relation to security – see previous post). Admittedly it does appear a little confused about the subject of data retention stating that ‘current roles on digital records state that data should never be stored for longer than necessary because they might be misused or inadvertently released’. It then goes on to state that ‘in future it is more likely that companies will be required retain all digital files, and ensure their accuracy, rather than to delete them’ – a vision of the future likely to strike fear into every records managers heart. There are some immediate flaws obvious in this logic (in the EU at least) where current data protection laws prevent this in relation to personal data, and elsewhere the Economist itself draws attention to the problems that storing such massive amounts of data is causing to the existing technical and resource infra-structure that Google et al rely on which would seem to favour a more selective approach to data retention on pragmatic grounds if nothing else. But whether such concerns are considered enough to stop the ‘lets keep and exploit everything’ bandwagon which lies behind much of this report is at best debatable and at worst, I suspect, distinctly unlikely.
There’s also interesting stuff in the Economist supplement on the problems of how to make sense of all this data, including new ways of visualising it and the prediction that statistics will soon be one of the coolest jobs around(!). It also makes some interesting points about the need for management to be trained in how to make sense of all this data. This chimes with a conversation I had with a Chief Exec a few weeks ago who also made the case for ensuring that senior management were aware of good old fashioned archival concepts such as provenance and context to give them a better appreciation of what the data they are looking at is actually telling them or how much it can be relied upon (rather than what they wish it was telling them and how much faith they may wish to place in it).
To give the Economist its due it does also look beyond the potential and address some of the challenges (and not just in relation to security – see previous post). Admittedly it does appear a little confused about the subject of data retention stating that ‘current roles on digital records state that data should never be stored for longer than necessary because they might be misused or inadvertently released’. It then goes on to state that ‘in future it is more likely that companies will be required retain all digital files, and ensure their accuracy, rather than to delete them’ – a vision of the future likely to strike fear into every records managers heart. There are some immediate flaws obvious in this logic (in the EU at least) where current data protection laws prevent this in relation to personal data, and elsewhere the Economist itself draws attention to the problems that storing such massive amounts of data is causing to the existing technical and resource infra-structure that Google et al rely on which would seem to favour a more selective approach to data retention on pragmatic grounds if nothing else. But whether such concerns are considered enough to stop the ‘lets keep and exploit everything’ bandwagon which lies behind much of this report is at best debatable and at worst, I suspect, distinctly unlikely.
Subscribe to:
Posts (Atom)