The link between a story about people's apparent addiction to paying to store old sofas, records, magazines that they no longer have room for at home and a critique of email management strategies over the past 15 years or so is not, perhaps, immediately obvious. And yet I was certainly struck by many of the same underlying trends that I alluded to in a recent paper I gave to the Digital Preservation Coalition's Email Preservation Workshop entitled: Email management: Fifteen wasted years and counting and the the piece featured on today's BBC website about 'The self storage craze'
If you can't be bothered to read the full text of my paper then Chris Prom has neatly summed up the main thrust of my argument in his own blog post. Basically, it is that our users are now strongly influenced in their approach to information management by the external tide of technology which leads them to expect near-infinite storage at their finger tips as a given. Our users like to keep 'stuff'. They may not always know why, or for what purpose it is being kept but want to keep it nonetheless and this apparently applies just as much to battered old bits of furniture and betamax videos we can no longer play as it does to emails and other records.
What I argued in the paper to the DPC was that we as records professionals seem to have either ignored or dismissed this impetus and have spent the best part of fifteen years trying, virtually always unsuccessfully, to fight against it and to impose rules regarding the retention and disposal of emails and other records which not only runs counter to the overall direction of technological development but to human nature. As i say in the paper:
"Trying to sell the concept of manual disposal of emails is a bit like telling the driver of a Porsche that he still has to have a man with a red flag walk in front of him."
This appears to me to be another example, perhaps the classic example of how we have failed to understand how our users think and act and have instead tried to impose management solutions on them which may suit the corporate agenda but which singularly fails to meet user requirements. And then we wonder why users fail to engage with the solutions we provide...
All this is very dear to my heart at the moment as I am currently working on producing a new infoKit for the JISC infoNet website: "Implementing information management technologies" which aims to fuse records management and Human-Computer Interaction approaches to create a methodology for implementing information managment techologies which gives equal weight to meeting the needs of the users as it does to the organisation. I doubt it will be the complete answer, but at least it will hopefully mark a step in the right direction. It won't be released until early next year so watch this space...
Showing posts with label data storage. Show all posts
Showing posts with label data storage. Show all posts
Thursday, 1 September 2011
Friday, 5 March 2010
'Big data': big potential, big challenges
This week’s Economist has an excellent special report on managing information entitled ‘Data, Data everywhere’. It looks at the changes, opportunities and challenges posed by our new found ability to create and manipulate vast quantities of data – big data. There are lots of impressive/daunting (depending on your point of view) statistics about just how much data we are now talking about (40 billion photos on Facebook for example) during this “industrial revolution of data”. It also explores the concept of ‘data exhaust’ the trail of clicks which users leave behind them and which Google and others have been able to put to such incredible use: from search to speech recognition and from spell checking to language translation. All made possible not by attempting to training computers the rules which determine how these concepts work, but instead by tracking the activities of billions of user transactions which do the work of refining, correcting and adding relative value to words. Those who have heard my ‘Meet the future of Records Management: Amazon.com’ conference paper will know that I have long suspected that we could and should be making use of this exact same ‘exhaust’ to help us manage information, as well as profit from it - what I describe as 'Automated Records Management' (See also Records Management Journal Vol 19 No.2 2009 for a paper I wrote on this entitled 'Forget Electronic Records Management its Automated Records management that we desperately need')
There’s also interesting stuff in the Economist supplement on the problems of how to make sense of all this data, including new ways of visualising it and the prediction that statistics will soon be one of the coolest jobs around(!). It also makes some interesting points about the need for management to be trained in how to make sense of all this data. This chimes with a conversation I had with a Chief Exec a few weeks ago who also made the case for ensuring that senior management were aware of good old fashioned archival concepts such as provenance and context to give them a better appreciation of what the data they are looking at is actually telling them or how much it can be relied upon (rather than what they wish it was telling them and how much faith they may wish to place in it).
To give the Economist its due it does also look beyond the potential and address some of the challenges (and not just in relation to security – see previous post). Admittedly it does appear a little confused about the subject of data retention stating that ‘current roles on digital records state that data should never be stored for longer than necessary because they might be misused or inadvertently released’. It then goes on to state that ‘in future it is more likely that companies will be required retain all digital files, and ensure their accuracy, rather than to delete them’ – a vision of the future likely to strike fear into every records managers heart. There are some immediate flaws obvious in this logic (in the EU at least) where current data protection laws prevent this in relation to personal data, and elsewhere the Economist itself draws attention to the problems that storing such massive amounts of data is causing to the existing technical and resource infra-structure that Google et al rely on which would seem to favour a more selective approach to data retention on pragmatic grounds if nothing else. But whether such concerns are considered enough to stop the ‘lets keep and exploit everything’ bandwagon which lies behind much of this report is at best debatable and at worst, I suspect, distinctly unlikely.
There’s also interesting stuff in the Economist supplement on the problems of how to make sense of all this data, including new ways of visualising it and the prediction that statistics will soon be one of the coolest jobs around(!). It also makes some interesting points about the need for management to be trained in how to make sense of all this data. This chimes with a conversation I had with a Chief Exec a few weeks ago who also made the case for ensuring that senior management were aware of good old fashioned archival concepts such as provenance and context to give them a better appreciation of what the data they are looking at is actually telling them or how much it can be relied upon (rather than what they wish it was telling them and how much faith they may wish to place in it).
To give the Economist its due it does also look beyond the potential and address some of the challenges (and not just in relation to security – see previous post). Admittedly it does appear a little confused about the subject of data retention stating that ‘current roles on digital records state that data should never be stored for longer than necessary because they might be misused or inadvertently released’. It then goes on to state that ‘in future it is more likely that companies will be required retain all digital files, and ensure their accuracy, rather than to delete them’ – a vision of the future likely to strike fear into every records managers heart. There are some immediate flaws obvious in this logic (in the EU at least) where current data protection laws prevent this in relation to personal data, and elsewhere the Economist itself draws attention to the problems that storing such massive amounts of data is causing to the existing technical and resource infra-structure that Google et al rely on which would seem to favour a more selective approach to data retention on pragmatic grounds if nothing else. But whether such concerns are considered enough to stop the ‘lets keep and exploit everything’ bandwagon which lies behind much of this report is at best debatable and at worst, I suspect, distinctly unlikely.
Wednesday, 30 January 2008
How to keep those servers cool!
There’s an interesting article in the latest edition of Government Computing magazine (unfortunately there is no e-version of the article to link to). The article in question is entitled ‘Greening the data centre’ and refers to the problems that organisations are encountering in terms of rising energy costs resulting from the ‘hot and hungry’ new blade servers that organisations are cramming into their data centres. According to the piece it is predicted that between 2000 and 2010 we will have "installed six times the amount of servers in our data centres and 69 times the amount of storage".
The problem is apparently that our data centre buildings are not designed to cope with the power required and heat generated by such machines, plus of course energy-consumption is now a political and ethical hot potato.
Now the interesting thing is the range of possible solutions outlined in the article. These vary from ‘better power management in data centres’, through to ‘taking your servers… and virtualising them’ or simply replacing old technology with new.
Nowhere does the rather obvious suggestion of ‘keeping less information’ get a look in. It would be interesting to know what percentage of the content of these steaming servers is actually still useful and still required? Of course the volume of information organisations create and need to retain is always increasing – but I bet there is still a huge percentage that could safely be destroyed if only anyone knew what it was, and whether it was still actually required…
But given that the main contributor to the piece is a Vice President at IBM perhaps its not that that surprising that the suggestion is to buy more kit, rather than to make better use of what exists already…
The problem is apparently that our data centre buildings are not designed to cope with the power required and heat generated by such machines, plus of course energy-consumption is now a political and ethical hot potato.
Now the interesting thing is the range of possible solutions outlined in the article. These vary from ‘better power management in data centres’, through to ‘taking your servers… and virtualising them’ or simply replacing old technology with new.
Nowhere does the rather obvious suggestion of ‘keeping less information’ get a look in. It would be interesting to know what percentage of the content of these steaming servers is actually still useful and still required? Of course the volume of information organisations create and need to retain is always increasing – but I bet there is still a huge percentage that could safely be destroyed if only anyone knew what it was, and whether it was still actually required…
But given that the main contributor to the piece is a Vice President at IBM perhaps its not that that surprising that the suggestion is to buy more kit, rather than to make better use of what exists already…
Wednesday, 11 July 2007
Archivists as "Data Storage Specialists"
I was interested to read a letter from my old colleague, David Ryan, published in that 'Bible' of the IT industry Computer Weekly a couple of weeks ago. David's letter serves to remind both the IT industry, and perhaps equally importantly we as professional archivists and records managers, that our professional skills in data storage are greatly - an increasingly - needed in the digital world. The question is, why aren't we better at promoting ourselves in this way and what can we as a profession do to address this?
David's letter is reproduced in full below.
---------------------------------------------------------------------------------
Dear Sir,
Re: Storage Reports in Computer Weekly, 22 May 2007
Having been a qualified Archivist, and therefore, a Data Storage Specialist, for over twenty years, I was bemused at your recent reports on the alleged lack of career definition in data storage and the legal issues surrounding storage. I have been within the last decade Head of Information Management for a major pharmaceutical company and more recently, Head of Digital Preservation at the UK National Archives, where I designed the PRONOM file format recognition system and, with colleagues, built TNA's Digital Archive.
I, like many other archivists, have written on the topic on data retention. In 1997, ten years ago, the Records Management Journal published an article by Steve Bailey and myself on the uses of a retention schedule database. This was based on practical experience. It may be useful for your readers who are dealing with the ever growing volume of data their organisations produce, to consult not only their in-house counsel but their local Records Manager or alternatively, seek advice from the Records Management Society or Society of Archivists. The members of these two bodies are dealing with the selection of data for long term storage in all formats on a rational basis every day of their working lives.
David Ryan
David's letter is reproduced in full below.
---------------------------------------------------------------------------------
Dear Sir,
Re: Storage Reports in Computer Weekly, 22 May 2007
Having been a qualified Archivist, and therefore, a Data Storage Specialist, for over twenty years, I was bemused at your recent reports on the alleged lack of career definition in data storage and the legal issues surrounding storage. I have been within the last decade Head of Information Management for a major pharmaceutical company and more recently, Head of Digital Preservation at the UK National Archives, where I designed the PRONOM file format recognition system and, with colleagues, built TNA's Digital Archive.
I, like many other archivists, have written on the topic on data retention. In 1997, ten years ago, the Records Management Journal published an article by Steve Bailey and myself on the uses of a retention schedule database. This was based on practical experience. It may be useful for your readers who are dealing with the ever growing volume of data their organisations produce, to consult not only their in-house counsel but their local Records Manager or alternatively, seek advice from the Records Management Society or Society of Archivists. The members of these two bodies are dealing with the selection of data for long term storage in all formats on a rational basis every day of their working lives.
David Ryan
Subscribe to:
Posts (Atom)