Finding the SharePoint sweet spot

I have written in the past about the importance of information professionals placing a higher priority on user needs, and even perhaps preferences, than has traditionally been the case.  Citing the failure of many an EDRMS implementation as primarily being down to these having been a technology designed around meeting the needs of the information manager rather than (and arguably even at the expense of) the end user.

At the same time there are, of course, limits as to how far any system designed to serve the needs of the many – perhaps even an entire organisation – can be successful in meeting the needs of each and every user.  Creating a hundred different systems, all based on individual user whim and with no sense of underlying unity or consistency will obviously and inevitably just create its own set of problems: not least the failure to create something which can in any way be classed as an ‘enterprise wide’ solution.

One of the reasons I like SharePoint is that it does give you the potential to try to span these competing demands.  The concept of Team Sites allows you to create solutions that can be bespoke to each function or team: upping the structure and controls where they are required, but easing them off where they are not; whilst doing so within the context of a shared enterprise wide ‘tenancy’ and within a general underlying framework that is yours to define.  This ability to customise and to push hard or to ease off the information management controls as local circumstances dictate seems to me to be one of the main distinguishing features between SharePoint and some of the EDRMS of old.

Of course the challenge for the information manager is still in achieving the correct balance between local freedom and required central management: wherever on the spectrum the particular function in question may exist.  To be able to find the ‘sweet spot’ that makes your system at least as attractive as the personal solutions they have devised for themselves, whilst not compromising further than you know is wise in terms of corporate information governance and management.  Push too far in one direction and you risk creating an elegant and robust corporate solution on paper but which no one actually uses; push too far in the other and you risk just replicating existing problems in a new environment. 

I’ve had cause to consider these issues in recent weeks as we have made the decision to reverse our previous system governance rule that did not allow the use of folders within SharePoint Document Libraries.  There is plenty of literature out there confirming that our original ban is the sound way to go.  Some of the reasons cited have a logical technical basis (the risk of creating lengthy URL’s that break in built system limits, for example).  But beyond this one of the main reasons repeatedly cited is that use of folders for resource discovery and content organisation purposes is not as good, as flexible and as sophisticated as using metadata, columns and views to achieve the same results.  This may well be true.  Use of metadata etc does open up a host of possibilities that folders do not and for those prepared to take the leap of faith and immerse themselves in this way of working it is undoubtedly beneficial.  The problem is that (in my experience at least)  only a small minority of users are prepared to abandon the folders based approach to working that they have spent the past decade or so working perfectly happy with.  If I had a pound for every time a potential or new user has asked me how they create a folder in their team site I would be a happy man.  The same would also be true if applied to every user who was left looking disappointed and (to varying degrees, sceptical) when told that the combination of metadata and views make folders redundant. 

The actual catalyst for us making the decision to allow folders (with caveats) within Team Sites was not, actually, just the weight of user opinion, but a growing requirement for site owners to be able to restrict permissions to certain content within their site.  Yes, there are other ways this can be done and we do prefer this to be done at a Document Library level but this does seem like overkill for what is often just a handful of 'restricted' items.  But we eventually came to the conclusion that in most cases the use of folders with unique permissions within a document library was the most straight forward and proportionate approach.  It also had the effect of giving us the opportunity to re-evaluate our previous position.

In short, the conclusion we came to was that for us and our users the pros of using folders in certain circumstances outweigh the cons.  We are cautioning restraint and still pushing the importance of metadata and views as the primary route but concede that folders do have a valuable role to play in our SharePoint implementation: not least in terms of user engagement and the perception of SharePoint as a familiar and useful environment. 

Here's a list of the caveats and cautions we have communicated to our Site Owners:

• Don’t create sub-folders (folders within folders). This creates long URLs which will break SharePoint’s in built maximum URL length and cause errors
• Whilst you can drag and drop items into folders it is then not easy to move items between folders
• Moving an item will change its URL, breaking any direct links to it that you may have created and shared
• Once contained within a folder it is harder to incorporate the individual items within any Document Library-level retrieval or sorting by metadata
• SharePoint will not prevent the same item being uploaded into multiple folders, increasing the risk of duplicate information existing within the system.

Is there a risk that users will forget our caveats, abandon metadata wholesale and replicate entire filing systems within SharePoint?  Possibly, though (we think) unlikely.  It is certainly something we shall keep an eye on and work with Site owners in the months ahead to nip in the bud if it looks a risk.  Will it help increase the general usability of SharePoint to the end user and its attractiveness as an information management environment? Definitely.  And, for now at least, that makes it a sweet spot worth seizing.

SharePoint and the 'continuum of control'

So, for the past year or so I have been heading up a SharePoint implementation project within Jisc.  It has proved a very interesting task and has got me thinking in all sorts of directions.  It has also been an extremely useful opportunity to refresh whatever professional skills and knowledge I may have and to reacquaint myself with life at the sharp end of records and information management.    

As some of you will know, a recurring area of interest of mine for some time has been the shifting foci of power that we have witnessed in recent years when it comes to information creation and control: away from the organisation and towards the user and what, if anything, our professional response should be. 

SharePoint, it appears to me, sits at an interesting point along this continuum.  Or, more interestingly still, can potentially be made to sit at pretty much any point along it from ‘bolted down corporate repository’ (mandatory metadata, records declaration and retention management etc) to ‘free and easy user-focused solution’ (create a Team site, store content, share it with who you like).  It is this range of potential management, control and usage options that interests me as I suspect it is pretty unique within this landscape.  After all, if you let all your users loose with Dropbox or Googledocs, you are always going to be limited in the range of centrally defined management controls that you can put in place across it.  Alternatively it would be difficult to try to turn a full blown EDRMS into a collaborative tool which is entirely at the user’s discretion when it comes to information creation and management. 

So the benefit, in theory at least, of SharePoint is that it does at least give you as the information manager the potential to find your own ‘sweet spot’ on this ‘continuum of control’: to decide which elements you want to enforce and which you wish to leave to the user’s discretion; to determine how much information management policy and rigour you wish to implement, and how much you want to leave to individual whim.  Finding exactly where this sweet spot is for your organisation is, it seems to me, half the key to success when it comes to implementing SharePoint: enforce too much control and you may find it a near impossible feat to convince your users that they should abandon their GoogleDocs accounts for it; but include too little and you may find you are simply replacing one unmanaged and ungoverned mess of information with another. 

And, of course, if our experience is anything to go by, you are likely to discover different sweet spots exist from department and department and function and function within your organisation with some users demanding a level of rigour and governance that others would find totally off putting.  Trying to be ‘all things to all men’ may seem like a tall order, and perhaps it is from a practical perspective but at least it is theoretically possible within SharePoint.

Yesterday I took part in a UCISA Webinar that explored some of these themes and which explained a little about the decisions that we have taken here at Jisc with regards to managing information and records within SharePoint.  I started with a brief(ish)overview of some of the history that has shaped the interplay between user andorganisation with regards to the management of information and a copy of the script for this is available.  I also gave a presentation which explored how we have approached SharePoint withinJisc, and in particular how we have implemented the types of records management controls that we consider appropriate for our needs.  The whole Webinar is also available for downloading (1.5 hours) if interested and which not only includes the above with narration, but also a presentation on University of Highlands and Island’s experience of using SharePoint and all the questions and discussions surrounding the presentations. 

Now that I have got the blogging bit between my teeth again I’ll perhaps elaborate a little more on the approach we have taken in future posts in the near future but think this is probably enough to be getting on with for now…

Encouraging user participation in RM projects

Its been a recurring theme of mine for the last couple of years now that records management needs to find better ways of connecting with users; of thinking a little less about ‘the organisation’ and a little more about ‘the individual’.  For without the support and enthusiasm of those on the ground even the most ambitious and robust of RM implementations is pretty well doomed to failure. 

Human Computer Interaction and User Centred Design based approaches certainly have a great deal of potential in this regard and the more I have researched these areas the more convinced I become that we need to find ways to bring such techniques more fully into the RM canon.  It would certainly be interesting to know just how many EDRM/ERM systems have ever undergone any robust, independent usability testing.  The change in tack from some vendors away from the “our product integrates ‘seamlessly’ with your users desktop” message of the past to the “our product integrates seamlessly with Sharepoint so your users never even know its there” message of today suggests a certain recognition that this was largely a battle lost.

I suspect many of those involved in RM projects might protest that user consultation and engagement has been an important facet of their projects.  Focus groups, ‘model offices’, and user representatives on project teams are all well established mechanisms for ensuring the user voice isn’t lost.  But how effective are they?  Are these channels which really proactively encourage free thought and honest reflection, or are they (perhaps subconsciously) designed to only produce a narrow range of responses with most of the important decisions already made: more a question of ‘Which user interface do you like best: A, B, or C’ than ‘What could we do to improve your access to the information you need to do your job more easily’?  Whereas the former already assumes a new interface is needed, whether the user likes it or not; the latter makes no such assumptions and could elicit a broad spectrum of ideas that go far beyond changes to a system interface.

As part of my role at JISC infoNet I’ve been increasingly engaged with participatory techniques and stringing these together to run participatory workshops.  These are simple, creative exercises that are designed to get groups of people working together, raising issues, sharing ideas and forming a consensus.  They are the polar opposite of most of the workshops I’ve ever encountered in terms of the energy and enthusiasm they generate and the results they generate.  Just this week we’ve released an online guide to how to conduct such an event through our Planning a participatory workshop infoKit.

We’ve been using these techniques extensively with project teams, College leadership teams and internally within our own team for over a year now and I increasingly think they have an important role to play as part of any information or records related project.  Of course, just getting running a participatory workshop or two or livening up a meeting with some of these exercises alone isn’t going to guarantee a successful RM project, but it might just represent one of the pieces of the records management puzzle that I’m increasingly convinced is missing at the moment.

Process Modelling – A missed opportunity?

“Staff working on the project did not have experience of defining business processes or of translating metrics into monetary terms. It was difficult at first to think of the management of personnel files and employee records as a business process."

 So reported one of the projects piloting our Impact Calculator back in 2010. I stumbled on this by chance just the other day whilst preparing for a workshop on the Impact Calculator but the reason for my interest actually relates to some thoughts I have been having recently about process mapping and in particular the tools and standards that records managers adopt when carrying out process mapping for their purposes and how these compare with the ‘industry standards’ employed by those for whom business mapping is core to what they do.

To this end I sent an email to the UK records management jiscmail list, acknowledging that:

 “functional analysis has long been considered an important aspect of the records management canon. We understand the importance of taking a ‘functional approach’ to record keeping and for having classification schemes, retention schedules and other RM controls determined by a function-based structure.


What I would be very interested to know, however, is what tools and/or standards do people tend to use when it comes to undertaking this analysis and capturing the outputs?”

To be fair I received a fairly limited response, but what I did receive struck me as interesting. In essence a pattern seemed to emerge where by the records manager reported that they use Powerpoint or Mindmap or something at what could be described as the ‘basic’ or ‘non-specialist’ end of the spectrum whist at the same time pointing out that ‘the organisation’ uses something else (usually something more specialised). Obviously it’s all very anecdotal and based on a small sample but I wonder whether it suggests some potentially missed opportunities?

What do I mean? Well, use of different tools and individualised (non standard) approaches to modelling may make it harder for the records manager to add their outputs to whatever models are being compiled throughout the rest of the organisation. If so, is there a risk that by not ‘talking the same language’ as our business analyst and IT colleagues that we are making it harder for ourselves to add RM controls and services to the enterprise’s architecture? Does a lack of adoption of accepted standards and tools also mean we are limiting our capacity as records managers to start sharing and joining up our process models between organisations working in the same sectors, limiting the opportunity for their reuse and for collaborative work? And finally, are the Mindmaps, Powerpoint slides and non-standard Visio notations really fit for our own purposes, or have we just grown accustomed to accepting the limitations of what we can achieve within these tools, rather than exploring the benefits that greater knowledge and use of more sophisticated tools could bring to our own daily work?

I’m not really sure where, if anywhere, any of this is leading or whether its not really an issue at all, but cant seem to shake the nagging feeling that there is some trick being missed by a lot of us at the moment.

The behaviour of the crowd: a new era for appraisal?

Very interesting to see that the folks in charge of the UK Web Archive at the British Library are planning to adopt a crowd-sourcing approach to informing their selection of websites to archive


When I wrote Managing the Crowd back in 2008 I put forward the view that agreeing appraisal decisions largely on the basis of user opinion of the worth of a record was likely to be the only way to go. Here we not only see an example of this being attempted in practice, but an interesting explanation for the decision:

“we recognise that this manual selection process can sometimes be time consuming for frequent selectors. It’s also inevitably subjective, reflecting the interests of a relatively small number of selectors.”

This reflects two of the major tensions informing my own thinking on this back then (and now). That is that the kind of manual approaches to selection and appraisal traditionally adopted, ie manual processes undertaken by a relatively small number of trained professionals simply isn’t sustainable in the face of the ever growing onslaught of information being created. This will come as no surprise as an issue to any information professional even if solutions to it – this initiative aside - still seem rather thinner on the ground…

The admission of subjectivity within existing approaches to appraisal and the implication of ‘selection bias’ that underlies it is less widely discussed, perhaps because until now we’ve had no alternative. It stands to reason that despite our best efforts any selective appraisal process must inevitably be biased in some way, whether the appraiser is conscious of it or not. But having appraisal decisions based at least in part on user behaviour promises to go along way to resolving some of these issues. What would of course be fascinating would be a comparative study which compares the websites which would have been selected for capture by existing manual methods with those that are captured by the Web Archive according to ‘the crowd’ to go some way to seeing how closely (or not) the two are aligned.

A counter argument to all this may run along the lines of ‘but what if the sites the public are viewing most are not the most important ones.’ In short: ‘do the public really know best?’ Perhaps wisely the British Library are also incorporating “curatorial input to this approach, so we’ll be asking curators from the Library to assess the quality and relevance of resulting selections”. But it does pose and interesting question: should we be seeking to capture as accurately as possible the sites which the public believed to be of interest/use to a particular topic or those that we as information professionals believe they should have been interested in? The former of these may lead to the capture of some surprising, perhaps even ‘unsuitable’ sites, whilst the latter would perhaps provide a more informed, maybe a more ‘official’ version of events. But which would be the most accurate?

Its also interesting to note that this approach to crowd-sourcing isn’t just relying on user opinion but on the results of actual user behaviour. They aren’t just asking people to collectively vote for sites they wish to see included in the archive, but are analysing data from twitter regarding which sites were linked to at the time. Using user behaviour to inform appraisal wasn’t something I considered back in 2008 but have done quite of thinking about since, notably in a paper in the Records Management Journal in 2009 (Vol 19 No.2) titled ‘Forget electronic records management, its automated records management that we desperately need’. That is that we use the data about user behaviour generated by business systems (which records they have opened, whether they edited or just read it, what they looked at next etc) as the means to help inform our records management policies based not on what we believe to be organisational need, but on actual patterns of behaviour. This is something we are all familiar with through sites such as Amazon and their ‘users who looked at this item also looked at these items…’ functionality. Use of such ‘behavioural analytics’ is also gathering momentum within academia with institutions for example using library usage patterns to identify at an early stage which students may be disengaging from their studies. To my mind the ability to closely monitor and analyse user behaviour in this way has the potential to not only increase the scalability of much of records management but also to increase the level of sophistication in which it can operate.

Maybe there is hope yet.

The storage addiction

The link between a story about people's apparent addiction to paying to store old sofas, records, magazines that they no longer have room for at home and a critique of email management strategies over the past 15 years or so is not, perhaps, immediately obvious. And yet I was certainly struck by many of the same underlying trends that I alluded to in a recent paper I gave to the Digital Preservation Coalition's Email Preservation Workshop entitled: Email management: Fifteen wasted years and counting and the the piece featured on today's BBC website about 'The self storage craze'

If you can't be bothered to read the full text of my paper then Chris Prom has neatly summed up the main thrust of my argument in his own blog post. Basically, it is that our users are now strongly influenced in their approach to information management by the external tide of technology which leads them to expect near-infinite storage at their finger tips as a given. Our users like to keep 'stuff'. They may not always know why, or for what purpose it is being kept but want to keep it nonetheless and this apparently applies just as much to battered old bits of furniture and betamax videos we can no longer play as it does to emails and other records.

What I argued in the paper to the DPC was that we as records professionals seem to have either ignored or dismissed this impetus and have spent the best part of fifteen years trying, virtually always unsuccessfully, to fight against it and to impose rules regarding the retention and disposal of emails and other records which not only runs counter to the overall direction of technological development but to human nature. As i say in the paper:

"Trying to sell the concept of manual disposal of emails is a bit like telling the driver of a Porsche that he still has to have a man with a red flag walk in front of him."

This appears to me to be another example, perhaps the classic example of how we have failed to understand how our users think and act and have instead tried to impose management solutions on them which may suit the corporate agenda but which singularly fails to meet user requirements. And then we wonder why users fail to engage with the solutions we provide...

All this is very dear to my heart at the moment as I am currently working on producing a new infoKit for the JISC infoNet website: "Implementing information management technologies" which aims to fuse records management and Human-Computer Interaction approaches to create a methodology for implementing information managment techologies which gives equal weight to meeting the needs of the users as it does to the organisation. I doubt it will be the complete answer, but at least it will hopefully mark a step in the right direction. It won't be released until early next year so watch this space...

Paying lip service to the user

Since my last post on the need to rebalance our approach to records management around the needs of the user I have been doing a little research of the RM literature on the subject. Admittedly this has been confined to the Records Management Journal (as a member of the Editorial Advisory Board I have easy access to it!) but looking through the back issues of 20 years for articles relating to RM system implementation has still proved an interesting exercise. I guess I should say at the outset (if not already obvious) that this was no formal literature review, but it at least helped contextualise some of my own thoughts, observations and anecdotatal evidence.

I found 7 papers relating to RM system implementation; most, if not all, it has to be said, relating specifically to EDRMS and I guess this focus has to be borne in mind when making any generalised assumptions. But that aside they nearly all make the point that ‘user buy-in’ is critical to the success of the project and that projects are largely doomed to failure without it.

There is also a remarkable consistency regarding how this can best be achieved: ‘Involve users early’, ‘’invite user representatives onto the project board’, ‘invite user representatives to try it out in a model office’, ‘give users plenty of training and support’ etc etc. All perfectly sound advice you might think and indeed it’s difficult to argue against any of them but at the same time I can’t shake the feeling that despite all these good intentions they are all missing the real point.

It’s a little like we are saying ‘when forcing a pedestrian to learn to drive a car its important to include them in the decision making process from the beginning; invite them to help choose what make and model; let them decide the colour; be sure to offer them plenty of training first and then be prepared to sit with them during their first few trips’. Great. But what if they didn’t actually want to drive a car in the first place? What if walking or taking the bus really suits their lifestyle? In short rather than simply asking them what colour car they would like, why not ask them how we could best improve their journey to work, however they decide to make it?

Biagio and Ibricu (RMJ Vol 18 No.3) inadvertently made an interesting point when they stated:

“users often fail to understand the corporate perspective of an EDRMS implementation and tend to remain focused on how the system will help them perform their jobs more efficiently and which tangible benefits and improvements it will bring”

Is it me, or do I detect a note of criticism of the user’s values here? Almost as if being censured for daring to put how they perform their own roles above what is of benefit to the organisation and its records managers. Well, welcome to the real world. Of course users are going to put their own working practice and working lives first. You’d have to be a pretty strange beast to volunteer to make your own work less efficient and less enjoyable just so someone else can benefit.

This question of just who the EDRMS is for also rears its head in another article by Ganesh Vednere (RMJ Vol 19, No.2) when he states:

“some people have been known to say ‘oh we are not that familiar with the technologies, so we let the technology team do the selection for us’ – well, that’s fine, but remember that ultimately it is the records management team that has to live with the platform”.

Now Ganesh was making a different point here about the need for records managers to get involved in the technical selection process rather than just delegating it to IT but what is striking here is the overt assumption that is the ‘records management team’ who have to live with the consequences of a poorly chosen system. A fair enough point, but what about the poor users? They don’t even get a mention. Doesn’t the same logic apply to them as well? Aren’t we asking them to ‘live with a platform’ that in all likelihood they had little or no input into choosing and configuring?

So my point? That the literature is absolutely right to highlight the importance of user engagement but that they seem to be fundamentally misjudging when in the process this needs to occur and the weight that needs to be placed upon their feelings. And that this process can probably best be started by replacing the question ‘how can we configure this system to try to meet your requirements? With ‘how can we help you do your job better?’