Very interesting to see that the folks in charge of the UK Web Archive at the British Library are planning to adopt a crowd-sourcing approach to informing their selection of websites to archive
When I wrote Managing the Crowd back in 2008 I put forward the view that agreeing appraisal decisions largely on the basis of user opinion of the worth of a record was likely to be the only way to go. Here we not only see an example of this being attempted in practice, but an interesting explanation for the decision:
“we recognise that this manual selection process can sometimes be time consuming for frequent selectors. It’s also inevitably subjective, reflecting the interests of a relatively small number of selectors.”
This reflects two of the major tensions informing my own thinking on this back then (and now). That is that the kind of manual approaches to selection and appraisal traditionally adopted, ie manual processes undertaken by a relatively small number of trained professionals simply isn’t sustainable in the face of the ever growing onslaught of information being created. This will come as no surprise as an issue to any information professional even if solutions to it – this initiative aside - still seem rather thinner on the ground…
The admission of subjectivity within existing approaches to appraisal and the implication of ‘selection bias’ that underlies it is less widely discussed, perhaps because until now we’ve had no alternative. It stands to reason that despite our best efforts any selective appraisal process must inevitably be biased in some way, whether the appraiser is conscious of it or not. But having appraisal decisions based at least in part on user behaviour promises to go along way to resolving some of these issues. What would of course be fascinating would be a comparative study which compares the websites which would have been selected for capture by existing manual methods with those that are captured by the Web Archive according to ‘the crowd’ to go some way to seeing how closely (or not) the two are aligned.
A counter argument to all this may run along the lines of ‘but what if the sites the public are viewing most are not the most important ones.’ In short: ‘do the public really know best?’ Perhaps wisely the British Library are also incorporating “curatorial input to this approach, so we’ll be asking curators from the Library to assess the quality and relevance of resulting selections”. But it does pose and interesting question: should we be seeking to capture as accurately as possible the sites which the public believed to be of interest/use to a particular topic or those that we as information professionals believe they should have been interested in? The former of these may lead to the capture of some surprising, perhaps even ‘unsuitable’ sites, whilst the latter would perhaps provide a more informed, maybe a more ‘official’ version of events. But which would be the most accurate?
Its also interesting to note that this approach to crowd-sourcing isn’t just relying on user opinion but on the results of actual user behaviour. They aren’t just asking people to collectively vote for sites they wish to see included in the archive, but are analysing data from twitter regarding which sites were linked to at the time. Using user behaviour to inform appraisal wasn’t something I considered back in 2008 but have done quite of thinking about since, notably in a paper in the Records Management Journal in 2009 (Vol 19 No.2) titled ‘Forget electronic records management, its automated records management that we desperately need’. That is that we use the data about user behaviour generated by business systems (which records they have opened, whether they edited or just read it, what they looked at next etc) as the means to help inform our records management policies based not on what we believe to be organisational need, but on actual patterns of behaviour. This is something we are all familiar with through sites such as Amazon and their ‘users who looked at this item also looked at these items…’ functionality. Use of such ‘behavioural analytics’ is also gathering momentum within academia with institutions for example using library usage patterns to identify at an early stage which students may be disengaging from their studies. To my mind the ability to closely monitor and analyse user behaviour in this way has the potential to not only increase the scalability of much of records management but also to increase the level of sophistication in which it can operate.
Maybe there is hope yet.
Thursday, 8 December 2011
Thursday, 1 September 2011
The storage addiction
The link between a story about people's apparent addiction to paying to store old sofas, records, magazines that they no longer have room for at home and a critique of email management strategies over the past 15 years or so is not, perhaps, immediately obvious. And yet I was certainly struck by many of the same underlying trends that I alluded to in a recent paper I gave to the Digital Preservation Coalition's Email Preservation Workshop entitled: Email management: Fifteen wasted years and counting and the the piece featured on today's BBC website about 'The self storage craze'
If you can't be bothered to read the full text of my paper then Chris Prom has neatly summed up the main thrust of my argument in his own blog post. Basically, it is that our users are now strongly influenced in their approach to information management by the external tide of technology which leads them to expect near-infinite storage at their finger tips as a given. Our users like to keep 'stuff'. They may not always know why, or for what purpose it is being kept but want to keep it nonetheless and this apparently applies just as much to battered old bits of furniture and betamax videos we can no longer play as it does to emails and other records.
What I argued in the paper to the DPC was that we as records professionals seem to have either ignored or dismissed this impetus and have spent the best part of fifteen years trying, virtually always unsuccessfully, to fight against it and to impose rules regarding the retention and disposal of emails and other records which not only runs counter to the overall direction of technological development but to human nature. As i say in the paper:
"Trying to sell the concept of manual disposal of emails is a bit like telling the driver of a Porsche that he still has to have a man with a red flag walk in front of him."
This appears to me to be another example, perhaps the classic example of how we have failed to understand how our users think and act and have instead tried to impose management solutions on them which may suit the corporate agenda but which singularly fails to meet user requirements. And then we wonder why users fail to engage with the solutions we provide...
All this is very dear to my heart at the moment as I am currently working on producing a new infoKit for the JISC infoNet website: "Implementing information management technologies" which aims to fuse records management and Human-Computer Interaction approaches to create a methodology for implementing information managment techologies which gives equal weight to meeting the needs of the users as it does to the organisation. I doubt it will be the complete answer, but at least it will hopefully mark a step in the right direction. It won't be released until early next year so watch this space...
If you can't be bothered to read the full text of my paper then Chris Prom has neatly summed up the main thrust of my argument in his own blog post. Basically, it is that our users are now strongly influenced in their approach to information management by the external tide of technology which leads them to expect near-infinite storage at their finger tips as a given. Our users like to keep 'stuff'. They may not always know why, or for what purpose it is being kept but want to keep it nonetheless and this apparently applies just as much to battered old bits of furniture and betamax videos we can no longer play as it does to emails and other records.
What I argued in the paper to the DPC was that we as records professionals seem to have either ignored or dismissed this impetus and have spent the best part of fifteen years trying, virtually always unsuccessfully, to fight against it and to impose rules regarding the retention and disposal of emails and other records which not only runs counter to the overall direction of technological development but to human nature. As i say in the paper:
"Trying to sell the concept of manual disposal of emails is a bit like telling the driver of a Porsche that he still has to have a man with a red flag walk in front of him."
This appears to me to be another example, perhaps the classic example of how we have failed to understand how our users think and act and have instead tried to impose management solutions on them which may suit the corporate agenda but which singularly fails to meet user requirements. And then we wonder why users fail to engage with the solutions we provide...
All this is very dear to my heart at the moment as I am currently working on producing a new infoKit for the JISC infoNet website: "Implementing information management technologies" which aims to fuse records management and Human-Computer Interaction approaches to create a methodology for implementing information managment techologies which gives equal weight to meeting the needs of the users as it does to the organisation. I doubt it will be the complete answer, but at least it will hopefully mark a step in the right direction. It won't be released until early next year so watch this space...
Thursday, 9 June 2011
Paying lip service to the user
Since my last post on the need to rebalance our approach to records management around the needs of the user I have been doing a little research of the RM literature on the subject. Admittedly this has been confined to the Records Management Journal (as a member of the Editorial Advisory Board I have easy access to it!) but looking through the back issues of 20 years for articles relating to RM system implementation has still proved an interesting exercise. I guess I should say at the outset (if not already obvious) that this was no formal literature review, but it at least helped contextualise some of my own thoughts, observations and anecdotatal evidence.
I found 7 papers relating to RM system implementation; most, if not all, it has to be said, relating specifically to EDRMS and I guess this focus has to be borne in mind when making any generalised assumptions. But that aside they nearly all make the point that ‘user buy-in’ is critical to the success of the project and that projects are largely doomed to failure without it.
There is also a remarkable consistency regarding how this can best be achieved: ‘Involve users early’, ‘’invite user representatives onto the project board’, ‘invite user representatives to try it out in a model office’, ‘give users plenty of training and support’ etc etc. All perfectly sound advice you might think and indeed it’s difficult to argue against any of them but at the same time I can’t shake the feeling that despite all these good intentions they are all missing the real point.
It’s a little like we are saying ‘when forcing a pedestrian to learn to drive a car its important to include them in the decision making process from the beginning; invite them to help choose what make and model; let them decide the colour; be sure to offer them plenty of training first and then be prepared to sit with them during their first few trips’. Great. But what if they didn’t actually want to drive a car in the first place? What if walking or taking the bus really suits their lifestyle? In short rather than simply asking them what colour car they would like, why not ask them how we could best improve their journey to work, however they decide to make it?
Biagio and Ibricu (RMJ Vol 18 No.3) inadvertently made an interesting point when they stated:
“users often fail to understand the corporate perspective of an EDRMS implementation and tend to remain focused on how the system will help them perform their jobs more efficiently and which tangible benefits and improvements it will bring”
Is it me, or do I detect a note of criticism of the user’s values here? Almost as if being censured for daring to put how they perform their own roles above what is of benefit to the organisation and its records managers. Well, welcome to the real world. Of course users are going to put their own working practice and working lives first. You’d have to be a pretty strange beast to volunteer to make your own work less efficient and less enjoyable just so someone else can benefit.
This question of just who the EDRMS is for also rears its head in another article by Ganesh Vednere (RMJ Vol 19, No.2) when he states:
“some people have been known to say ‘oh we are not that familiar with the technologies, so we let the technology team do the selection for us’ – well, that’s fine, but remember that ultimately it is the records management team that has to live with the platform”.
Now Ganesh was making a different point here about the need for records managers to get involved in the technical selection process rather than just delegating it to IT but what is striking here is the overt assumption that is the ‘records management team’ who have to live with the consequences of a poorly chosen system. A fair enough point, but what about the poor users? They don’t even get a mention. Doesn’t the same logic apply to them as well? Aren’t we asking them to ‘live with a platform’ that in all likelihood they had little or no input into choosing and configuring?
So my point? That the literature is absolutely right to highlight the importance of user engagement but that they seem to be fundamentally misjudging when in the process this needs to occur and the weight that needs to be placed upon their feelings. And that this process can probably best be started by replacing the question ‘how can we configure this system to try to meet your requirements? With ‘how can we help you do your job better?’
I found 7 papers relating to RM system implementation; most, if not all, it has to be said, relating specifically to EDRMS and I guess this focus has to be borne in mind when making any generalised assumptions. But that aside they nearly all make the point that ‘user buy-in’ is critical to the success of the project and that projects are largely doomed to failure without it.
There is also a remarkable consistency regarding how this can best be achieved: ‘Involve users early’, ‘’invite user representatives onto the project board’, ‘invite user representatives to try it out in a model office’, ‘give users plenty of training and support’ etc etc. All perfectly sound advice you might think and indeed it’s difficult to argue against any of them but at the same time I can’t shake the feeling that despite all these good intentions they are all missing the real point.
It’s a little like we are saying ‘when forcing a pedestrian to learn to drive a car its important to include them in the decision making process from the beginning; invite them to help choose what make and model; let them decide the colour; be sure to offer them plenty of training first and then be prepared to sit with them during their first few trips’. Great. But what if they didn’t actually want to drive a car in the first place? What if walking or taking the bus really suits their lifestyle? In short rather than simply asking them what colour car they would like, why not ask them how we could best improve their journey to work, however they decide to make it?
Biagio and Ibricu (RMJ Vol 18 No.3) inadvertently made an interesting point when they stated:
“users often fail to understand the corporate perspective of an EDRMS implementation and tend to remain focused on how the system will help them perform their jobs more efficiently and which tangible benefits and improvements it will bring”
Is it me, or do I detect a note of criticism of the user’s values here? Almost as if being censured for daring to put how they perform their own roles above what is of benefit to the organisation and its records managers. Well, welcome to the real world. Of course users are going to put their own working practice and working lives first. You’d have to be a pretty strange beast to volunteer to make your own work less efficient and less enjoyable just so someone else can benefit.
This question of just who the EDRMS is for also rears its head in another article by Ganesh Vednere (RMJ Vol 19, No.2) when he states:
“some people have been known to say ‘oh we are not that familiar with the technologies, so we let the technology team do the selection for us’ – well, that’s fine, but remember that ultimately it is the records management team that has to live with the platform”.
Now Ganesh was making a different point here about the need for records managers to get involved in the technical selection process rather than just delegating it to IT but what is striking here is the overt assumption that is the ‘records management team’ who have to live with the consequences of a poorly chosen system. A fair enough point, but what about the poor users? They don’t even get a mention. Doesn’t the same logic apply to them as well? Aren’t we asking them to ‘live with a platform’ that in all likelihood they had little or no input into choosing and configuring?
So my point? That the literature is absolutely right to highlight the importance of user engagement but that they seem to be fundamentally misjudging when in the process this needs to occur and the weight that needs to be placed upon their feelings. And that this process can probably best be started by replacing the question ‘how can we configure this system to try to meet your requirements? With ‘how can we help you do your job better?’
Monday, 23 May 2011
Its the user, stupid
Take a look through ISO-15489 and see how many of its requirements are directed at meeting the needs of ‘the user’. Now I’d hesitate to say “none” as its quite possible that I’ve overlooked one or two, but I think I’m pretty safe in suggesting “precious few” as an accurate answer. Reading through it soon becomes clear who records management’s primary stakeholder is assumed to be – and its clearly not the user with virtually every recommendation defined in terms of what “the organisation” requires and what is in “the organisations” best interests.
This probably doesn’t come as a great surprise, after all RM has long strived to be acknowledged as an established ‘corporate function’ with an enterprise-wide remit. Indeed all the benefits that RM can offer as stated in Part 1 of 15489 are described by the way in which they “enable organisations to…”.
Most RM technologies follow this lead and seek to deliver benefits to ‘the organisation’. But where does this leave the individual users of which the organisation is comprised? Can we automatically assume that what is in the best interests of the organisation will also be so for its staff? Taken to its logical conclusion the answer must inevitably be yes; after all the organisation that fails to make a profit or continually finds itself in the law courts will soon find itself unable to pay the salaries of its present staff or the pensions of its former. But how strong a connection is this really seen by many? Might it be that the vast majority of staff simply wants to turn up in the morning, complete their allotted tasks as quickly and easily as possible and leave at a reasonable time in the evening with as few complications and hurdles as possible?
Few record managers have the authority to compel users to adopt the procedures or systems they introduce. Instead we rely on a mixture of inspiration and perspiration to encourage adoption – with varying degrees of success. My review of the existing literature is still in progress but from what I have read so far a recurring theme when it comes to RM system implementation failures is lack of staff engagement. As Rachel Maguire stated in her ‘Lessons learnt from implementing an electronic records management system’ (Records Management Journal, Vol 15 No.3, pp.150-7)
“In spite of extensive training, most staff never got to grips with the system”
From the literature and anecdotal evidence this seems a common occurrence, but why should it be so? Is it that the solutions we offer simply don’t meet the requirements of our users? Do we even know what their requirements are? Or have we been guilty of paying lip-service to such considerations whilst instead focusing our attention of trying to deliver solutions which benefit the organisation entire but at the users’ expense?
Outside of the RM sphere disciplines such as HCI (Human-computer interaction) and User Centered Design are well established and important aspects of the design and implementation of IT systems and technologies and help to deliver devices and applications that users actively want to use, partially because they have had a very real and active role in the design process and because this has led to solutions that make their lives a little bit easier. Myself and Jay Vidyarthi (a Human-Computer Interaction specialist from Canada) wrote a paper for the Records Management Journal (Human-computer interaction: the missing piece of the records management puzzle?, RMJ, Vol. 20 No.3, 2010) looking into some of these very issues but inevitably that only scratched the surface.
Now as part of my role at JISC infoNet I am hoping to build on this work by producing a resource which seeks to redress this balance and to find ways of integrating aspects of HCI and User Centred Design into the design and implementation of information management technologies. As such I would be really grateful to hear of any examples that you know of or have been involved with of information or records management projects which may have explored any of these issues and sought to apply these principles; or (perhaps more likely) to hear of projects which have stumbled or failed due at least in part to failings in this area.
Of course none of this is meant to suggest that ‘the organisation’ doesn’t matter, or that we should lose sight of the bigger picture whilst attempting to solve the specific problems of every member of staff – but merely to recognise that without the genuine and positive engagement of the latter we will never truly be able to serve the interests of the former.
This probably doesn’t come as a great surprise, after all RM has long strived to be acknowledged as an established ‘corporate function’ with an enterprise-wide remit. Indeed all the benefits that RM can offer as stated in Part 1 of 15489 are described by the way in which they “enable organisations to…”.
Most RM technologies follow this lead and seek to deliver benefits to ‘the organisation’. But where does this leave the individual users of which the organisation is comprised? Can we automatically assume that what is in the best interests of the organisation will also be so for its staff? Taken to its logical conclusion the answer must inevitably be yes; after all the organisation that fails to make a profit or continually finds itself in the law courts will soon find itself unable to pay the salaries of its present staff or the pensions of its former. But how strong a connection is this really seen by many? Might it be that the vast majority of staff simply wants to turn up in the morning, complete their allotted tasks as quickly and easily as possible and leave at a reasonable time in the evening with as few complications and hurdles as possible?
Few record managers have the authority to compel users to adopt the procedures or systems they introduce. Instead we rely on a mixture of inspiration and perspiration to encourage adoption – with varying degrees of success. My review of the existing literature is still in progress but from what I have read so far a recurring theme when it comes to RM system implementation failures is lack of staff engagement. As Rachel Maguire stated in her ‘Lessons learnt from implementing an electronic records management system’ (Records Management Journal, Vol 15 No.3, pp.150-7)
“In spite of extensive training, most staff never got to grips with the system”
From the literature and anecdotal evidence this seems a common occurrence, but why should it be so? Is it that the solutions we offer simply don’t meet the requirements of our users? Do we even know what their requirements are? Or have we been guilty of paying lip-service to such considerations whilst instead focusing our attention of trying to deliver solutions which benefit the organisation entire but at the users’ expense?
Outside of the RM sphere disciplines such as HCI (Human-computer interaction) and User Centered Design are well established and important aspects of the design and implementation of IT systems and technologies and help to deliver devices and applications that users actively want to use, partially because they have had a very real and active role in the design process and because this has led to solutions that make their lives a little bit easier. Myself and Jay Vidyarthi (a Human-Computer Interaction specialist from Canada) wrote a paper for the Records Management Journal (Human-computer interaction: the missing piece of the records management puzzle?, RMJ, Vol. 20 No.3, 2010) looking into some of these very issues but inevitably that only scratched the surface.
Now as part of my role at JISC infoNet I am hoping to build on this work by producing a resource which seeks to redress this balance and to find ways of integrating aspects of HCI and User Centred Design into the design and implementation of information management technologies. As such I would be really grateful to hear of any examples that you know of or have been involved with of information or records management projects which may have explored any of these issues and sought to apply these principles; or (perhaps more likely) to hear of projects which have stumbled or failed due at least in part to failings in this area.
Of course none of this is meant to suggest that ‘the organisation’ doesn’t matter, or that we should lose sight of the bigger picture whilst attempting to solve the specific problems of every member of staff – but merely to recognise that without the genuine and positive engagement of the latter we will never truly be able to serve the interests of the former.
Monday, 18 April 2011
Calculating the impact of the Impact Calculator
Back in October 2009 I posted on the launch of our new Impact Calculator, a tool designed to address the worrying dearth of reliable evidence supporting the various claims that are often made in favour of investment in records management (‘increasing productivity’, decreasing overheads etc’). Well, here we are, 18 months, 270 registered downloads (and lots more unregistered) of the Impact Calculator, 6 completed pilot projects and 1 published research paper later.
Hopefully, the paper in question – based on the findings from 6 UK universities all of whom used the Impact Calculator to measure the return on investment (or not) realised through improvements to records management – within the Records Management Journal (Vol. 21 No.1, 2011) will finally drive a stake in the heart of that oft-quoted but seemingly completely mythical previous ‘source’ of ‘data’: “the Coopers & Lybrand study” which has been doing the rounds of lazy records management presenters and authors for about the past decade!
Of course data based solely on the experience of 6 universities is always going to be limited in terms of its broader applicability and the evidential weight we can place upon it, but at least their methodology and the workings of the calculator are transparent and consistent and therefore a considerable step forward on what has gone before.
And the conclusions from this work? Well, there are several and its difficult to summarise them adequately in a blog post, but the following statements all feature in my RMJ paper’s conclusion:
- There is no single absolute threshold of data purity and evidential rigour appropriate when measuring the impact of records management
- the cost effectiveness of records management is closely linked to questions of volume
- measuring impact represents a challenging new role for records professionals
- There are pros and cons to judging the impact of records management in terms of the tangible benefits realised
And the following statements, also based on the outputs from the same pilot projects have also featured in presentations on the Impact Calculator that I have given to post-graduate students at both University of Aberystwyth and San Jose University, California:
- retrospective appraisal projects rarely deliver a financial return on investment
- implementing a retention schedule ‘from this point on’ can deliver cost savings
- RM is only cost effective above a certain scale of operation
- investment in better processes and systems is more cost effective than increased reliance on cheaper labour
Although the full text of my RMJ paper is only available via the Records Management Journal from Emerald Publishing, the full results of the 6 pilot projects with commentary is publicly available from the JISC infoNet website.
As noted above, the jury is still out as to whether seeking to measure the impact of records management purely in statistical terms is a good thing or not and certainly there are lots of other reasons (legal, regulatory, historical etc) as to why it makes sense to appraise legacy records even if the evidence suggests that it will never recoup the money you spend on doing it. But at least now there is a tool and the beginnings of an evidence base out there for those who do feel that the business case for investment in records management within their organisation would be strengthened through the addition of some relevant and reliable data regarding its financial worth.
Hopefully, the paper in question – based on the findings from 6 UK universities all of whom used the Impact Calculator to measure the return on investment (or not) realised through improvements to records management – within the Records Management Journal (Vol. 21 No.1, 2011) will finally drive a stake in the heart of that oft-quoted but seemingly completely mythical previous ‘source’ of ‘data’: “the Coopers & Lybrand study” which has been doing the rounds of lazy records management presenters and authors for about the past decade!
Of course data based solely on the experience of 6 universities is always going to be limited in terms of its broader applicability and the evidential weight we can place upon it, but at least their methodology and the workings of the calculator are transparent and consistent and therefore a considerable step forward on what has gone before.
And the conclusions from this work? Well, there are several and its difficult to summarise them adequately in a blog post, but the following statements all feature in my RMJ paper’s conclusion:
- There is no single absolute threshold of data purity and evidential rigour appropriate when measuring the impact of records management
- the cost effectiveness of records management is closely linked to questions of volume
- measuring impact represents a challenging new role for records professionals
- There are pros and cons to judging the impact of records management in terms of the tangible benefits realised
And the following statements, also based on the outputs from the same pilot projects have also featured in presentations on the Impact Calculator that I have given to post-graduate students at both University of Aberystwyth and San Jose University, California:
- retrospective appraisal projects rarely deliver a financial return on investment
- implementing a retention schedule ‘from this point on’ can deliver cost savings
- RM is only cost effective above a certain scale of operation
- investment in better processes and systems is more cost effective than increased reliance on cheaper labour
Although the full text of my RMJ paper is only available via the Records Management Journal from Emerald Publishing, the full results of the 6 pilot projects with commentary is publicly available from the JISC infoNet website.
As noted above, the jury is still out as to whether seeking to measure the impact of records management purely in statistical terms is a good thing or not and certainly there are lots of other reasons (legal, regulatory, historical etc) as to why it makes sense to appraise legacy records even if the evidence suggests that it will never recoup the money you spend on doing it. But at least now there is a tool and the beginnings of an evidence base out there for those who do feel that the business case for investment in records management within their organisation would be strengthened through the addition of some relevant and reliable data regarding its financial worth.
Subscribe to:
Posts (Atom)