Taskforce workshop: assessments

In Friday's workshop a few of us got talking about the idea of "voluntary assessments" raised in the Ambitions document. These weren't really defined so it was difficult to know quite which was the best approach for this topic.

There was some concern that any assessment would just be another stick to beat libraries with, like issue figures and visitor counts have become. Equally, would they become another set of targets to be gamed? Both are extremely valid concerns.

I think it's essential that public libraries have a solid suite of KPIs — not for comparison with so-called "peers" in a way that takes no heed of communities or contexts but for comparison with past performance and identifying strengths and weaknesses in the operation. But the "voluntary assessments" *shouldn't* be about KPIs: there are a couple of other useful functions they could perform.

One way would be to direct the stick upwards, towards the DCMS. It's vanishingly unlikely to happen but it could be that one of the Department's own performance narratives, published in its annual report, could include an assessment of the health of the national public library service derived from local returns. 

• I don't like expenditure as a measure of performance (any bloody fool can spend money) but given that the audience for such a thing would be a political one that largely measures achievement by expenditure one of the assessment measures could be expenditure per capita population, which could be broken down into: investment in buildings; investment in skilled staff; investment in stock; and investment in community activities. (Did you spot the gear-shift there?) 
• DCMS would — finally! — have to be able to report a definitive number of public libraries in the country, and any changes and trends. 
• It would be interesting to see a national picture of:
• The number of staffed library open hours
• The number of library open hours manned by volunteers
• The number of unsupervised library open hours
• The number of "daytime" (9am — 5pm) hours libraries are closed and left fallow
• And so on. I won't go into more detail because it's so unlikely to happen it's hardly worth the wishing for.

Another, possibly more plausible, function would use the assessment as a feedback mechanism for continuous service improvement (absolutely not a set of targets!). They would be used to evaluate rather than monitor the performance of the service and help direct local decision-making. 

• The assessment criteria could be deliberately aspirational and impossible to achieve: the assessment would evaluate the direction of travel of the service and the impact on resources, staff and the needs of the communities that the service serves. 
• Comparisons would be with past performance rather than against "peers," which would remove any unwelcome competitive friction between organisations that should be working collaboratively, 
• This would also mean that it would be harder to game the figures and it would be harder to coast on past glories as assessments would be reporting the direction of travel towards the impossible goal rather than the successful negotiation of an arbitrary obstacle.
• For example, one of the "impossible goals" could be 100% of the local population's being active members of the library (however that would be defined). Last year Library Service A could have attained 56% active membership and this year 58% while Library Service B managed 69% and 60%. Crude peer-to-peer analysis would suggest that Library Service B is performing better but A is actually making a better fist of continuous service improvement — it's the rate and direction of performance, not the absolute figures, that are the measure of how the service is being managed and resourced; they'll all have different baseline starting points.

There is a problem with this idea: while it works well in many organisations and is pretty standard performance management fare I don't think our political environment is adult enough not to try and turn this into a badly-fitting set of targets and league tables. It would be nice to be shown to be wrong, though.

In the end there is no "In the end"

Projects and events are finite but, unless you're a journeyman project manager or an events manager, they have to live in the context of the organisation or service that they serve. As such they are episodes in a longer serial narrative rather than stand-alone short stories.

There are a number of ways in which these episodes are embedded in the narrative. The past is part of the backdrop and rationale for the activity. The activity itself is — for good or ill — a landmark in the narrative but this is static. The resources employed and delivered are identified in the business case and in the lessons learned process but these only provide the potential for the future narrative, they don't impel it.

So what does get the narrative moving?

"What happens next?"

• What happens next? = continuous service improvement
• What happens next? = business case for future projects
• What happens next? = sustainable service delivery

If your organisational reaction to a project or event is to push it into the drawer marked "History" you're losing the benefit of experience and you have to wonder why on earth you bothered in the first place. Similarly, if it doesn't evoke a "what happens next?" response perhaps you shouldn't have done it in the first place.

The latest DCMS review

I admit it, I missed the boat. By a long chalk. And so I didn't submit my views to the latest public consultation on public libraries. A combination of too many ideas, too little time and self-discipline and worrying overly-much as to how I'd make any of it fit to the three questions asked by the commissioned group.

I've no illusions that I would have made a ha'penny's difference but here are my workings out, in case anyone can use any of it:

At the outset I would like to wish you the best of luck with this latest review of the public library service in England and the hope that whatever your conclusions they are operationally practicable and support at least a decent-quality library service for our communities. You start with a serious handicap: DCMS announcements of reviews of the English public library service are a seasonal thing like the first cuckoo of Spring or the first M&S advert of Christmas. Each comes and goes and together their total operational impact in the real world has been the square root of jack all. This appalling legacy is going to colour too many of the views you are likely to hear. Including mine, unfortunately. I would love it if you could confound my cynicism.

• There is a crying need for national leadership.
  • Now the Olympic Games are over and done with, what is DCMS for? Over the past decade — aside from the occasional launch of an enquiry into the public library service — the department’s engagement with the service has been not so much arm’s length as running a mile from.
  • The public library service in England is undefined, at best weakly supported and subject to no performance management.
  • Regulatory guidance on the delivery of the service is virtually non-existent.
  • The potential for improving the efficiency and effectiveness of the public library service by pooling resources and delivery channels across local geopolitical boundaries is being driven patchily at the local level at the same time as long-standing sharing mechanisms are being abandoned or left to wither on the vine.
  • The English public library service is not an integral part of a national literacy programme, a national digital literacy programme or a national information literacy programme despite the huge amount of good work being locally done in these areas by many, if not most library authorities.
    • DCMS is not demonstrating that it knows or much cares about:
      • How many public library buildings are currently still in use;
      • What other delivery channels are being made available for library services;
      • What services are being delivered by these delivery channels;
      • Whether or not these services adequately reflect the needs of the communities they serve;
      • What resources should be employed to provide these services.
• Nobody knows what the public library service is. Everybody has an opinion, nobody has an empirical measure and there is no bottom-line base level of service that can be expected nationwide.
  • The sad fig-leaf that is the 1964 Act provides a fine-sounding but practically-useless sound bite. The sole practical impact of the Act is that public libraries used to get listed under “Statutory” rather than “Discretionary” when the auditors came round to see how well the local council was doing.
  • There is a view that if a building has had the word “library” stuck on it some time in its lifetime and the doors are still open then all is well in the world.
  • There is another view that so long as a building is open to the public and has some books for loan that it is a public library.
  • There is yet another view that wonders why, so long after Erasmus talked about “libraries without walls” and after nearly two decades of public libraries’ beginning to deliver their services online, English public library services are so often defined by the buildings with the word “library” stuck on them not the services being provided and delivered, often outwith those library walls.
  • Ironically, while there is a long-standing UK standard specifying the base common denominator functions for a library management system there isn’t a similar baseline specification for the service such a system would be supporting.
  • There are no baseline metrics for the public library service. The old public library standards were limited in scope and flawed in definition but they at least required that some attempt at performance management and the accumulation of business intelligence was being made. One would not want the public library service to be defined only by what could be measured (worse still only what could be measured forty years ago!) but any credible argument that the service being delivered is anything more than “the doors are open, end of story” must be supported by robust data. CIPFA returns provide some useful data but this is limited, not always freely available and not at all concerned with outcomes. Benchmark data Should include:
    
    • Traditional transactional and visitor throughputs.
    • Outcomes of programmes of library activity.
    • Demographic engagement and outcomes — a demonstration that the service is serving its communities and not just providing services “for people like us by people like us.”
    • Stock analyses, including data on special collections, reserve stock and specifically-local elements (not just “local studies” collections). This would also include contextual age-related data — a collection of Victorian books in a special collection is a matter of interest, a collection of fifteen-year-old children’s picture books is a matter of concern.
    • Performance at each service point, including buildings, outreach and digital channels. Transactional data at library buildings normalised to numbers per staff hour so that variations from the norm can be readily identified; while there should be some variation in response to the needs of the local community other variations may be cause for concern.
    • Analyses of delivery channels both within and without the library buildings managed by the service.
  • Once benchmark data had been established, openly-reported trend analyses should include:
    • Patterns of change of use;
    • Patterns of replacement of use — this might be as simple as 78’s being replaced in stock by mp3’s or as complex as a community of use migrating from one library to another;
    • Contextual commentary — for instance a note of the impact of the school next door closing; a new motorway cutting off a community from its library; or the involvement in a new programme of activities.

There is a need for the availability and application of librarianship skills at a community level. (This is not a call for a quota of “professionals” in each library authority: this has been tried before and too many of us have experience of working alongside librarians who were doing nothing that the “unqualified” library assistants were doing at least as well.) The librarian is a means to an end, not an end unto itself.

• The creation of local, parochial bibliographic metadata is culturally- and economically-beneficial to our communities. This is not limited to the traditional form of local studies collections — though these may be seen as an important component of the Arts Council’s commitment to the accessibility of the nation’s heritage.
  • Small-scale publication — especially self-publication — is easier than ever, particularly in e-book formats. There is a very real danger that much of this material will be permanently excluded from the national bibliography. Librarians, working with local authors and publishers should be tasked with the creation and publication of the appropriate metadata.
  • Many titles have a geolocational context that is not recorded or reflected in the commercially-available metadata. Making this local context available provides a hook for the recreational reader; resources for researchers and for teachers creating reading and learning materials; and support for literature-based community activities and tourism programmes.
• A national audit is urgently needed of those special collections not already dispersed, dissolved or disposed of as a result of austerity measures. In particular it is important to find out how much — or little — of these have been catalogued and published electronically so that a programme of work can be set up to address the oversights.
• Community knowledge bases.
• Grey literature.
• Information literacy.
• Local Freedom of Information libraries.

Engagement with the digital world

• Digital inclusion/digital literacy
• Digital libraries
• Integrating the virtual and physical worlds
• Crowdsourcing literary engagement
• Curating user-created content
• [All that stuff you’ve been arguing for fruitlessly for the past decade]

Staff development and continuous service improvement

• Essential — needs to be resourced and needs a proper framework for all staff
• Need to avoid replicating the errors and missed opportunities of the NOF-funded training programme for supporting the People’s Network — no “magic bullets” like ECDL
• Training needs dovetailing with service development needs
• Anticipating the support needs of communities and customers

Use and management of volunteers

• Complementary to paid staff
• Needs to be fair to the volunteer — what’s in it for them?
• Needs to be fair to the service — what’s in it for them?
• Needs to be fair to the community — what’s in it for them?
• Not an easy management win
  • Greater churn that paid staff — constant need for recruitment and training support
  • Too little good supervision of remote front-line staff at the moment — how would the same managers add supervision of volunteers to their portfolio?
  • Discipline and behaviour (this is true of all staff — not just volunteers — but fewer available sticks and carrots)
  • How to manage reputational damage when things go wrong?

Lessons Learned

It being the end of year and it being a time for reflection and review and that I thought I'd put down a few thoughts on a process that's sadly neglected by many library projects: Lessons Learned.

In my experience, too often the lesson learned is; "We seem to have managed that in the end, so it's OK to fly by the seats of our pants next time, too." This is an opportunity missed: experience is not what happens to you, it's what you do with it. If what you do with it is nothing then the experience is lost. So it's important to build the Lessons Learned process into any project.

The purpose of a Lessons Learned Document is to capture the experience accrued by the project in a formal document for use on similar future projects, including:

• Problems that occurred, how they were handled and how they may be avoided in the future.
• What went well with the project and why, so that other project managers may capitalize on this experience.

It is not the purpose of a Lessons Learned Document to apportion praise or blame.

This document should be used to support the continuous service improvement processes within the organisation.


Just to put my money where my mouth is, these are the recommendations from the lessons learned process from our project migrating from Dynix to Spydus:

1. A specification of operational functions is essential for the Statement of User Requirements. The more explicitly practical and measurable the more robust the selection process in procurement.
• Actively encourage staff input in the specification process to get ideas for the specification and buy-in for the project.
• Actively investigate other solutions and technologies so that you aren’t just doing a like-for-like replacement and limiting yourself to established business delivery models.


2. Before a procurement process that you own begins you need the following:
• What is the process? What are the critical paths?
• Who are the stakeholders — customer/ project/ procurement/ legal/ partners/ whoever
• Who is/are responsible for doing each step of the process?
• What information/ documentation is required for each step?
• When do you know each step has been completed?
• Has this all been agreed by all the stakeholders?


3. Agree a Project Initiation Document and work from it.
   • Make sure that you know who is doing what and in what order.
   • Make sure you know what isn’t to be done.


4. Work to the project:
   • Make sure that you’ve agreed who is doing what and in what order.
   • Make sure everyone knows what isn’t to be done.
   • Have clear lines of communication.
   • Get together regularly to review progress and, where necessary, revise action plans.
   • Allow at least three weeks between Subject Expert Training and Train the Trainers to allow options to be explored, modelled and tested for use (particularly with new functionality) adequately.
   • Train the Trainers is an opportunity to test the commissioning to date. Allow at least a week between Train the Trainers and the first batch of Cascade Training to test the safety of any changes.
   • Have cut-off points for commissioning changes:
     • No changes to codes and data structures after data migration.
     • Severely limit the number of system parameter changes after Train the Trainers.
     • Admit no changes to any part of the system (except in emergency) on the day you go live.

Make sure that the technical infrastructure requirements are included in the Statement of User Requirements and agreed with the supplier before commencing the installation.

The OPAC is an integral part of the system, not an add-on, so it needs to be treated as part of the whole.

• The training for the management of the OPAC needs to be included in the Subject Expert programme.
• Make sure that all the people having input to the commissioning of the OPAC understand its purpose and function.

Spending time cleaning up the data using familiar tools in the system you know saves a considerable amount of time, effort and problems with both the data migration and the operation of the new LMS.

Prepare for the MARC21 environment by making sure the existing catalogue data maps at least adequately and by making sure that there is sufficient MARC21 expertise within the organisation to verify that it does.

It’s useful and important to see how a reference site uses a process.

• It’s important to make sure that the ‘right’ experience of a site visit is realized: be clear about what the experience needs to be beforehand and proactively manage distractions.

Any library service that is not already used to MARC cataloguing should make sure well before the Subject Expert Training that:

• There is sufficient expertise for the catalogue data mapping process.
• Staff who will be using catalogue processes (including acquisition via EDI) need to understand at least the basics of the format.