Developing A Framework for Evaluating the Utilization of Reference Sources in The Federal Polytechnic Bali Library

Sources in The Federal Polytechnic Bali Library Usman Shehu Buba Main Library Federal Polytechnic Bali, Taraba State [email protected] 08086298583 Hassan Mohammad Main Library Federal Polytechnic Bali, Taraba State [email protected] 09030763886 https://doi.org/10.56201/ijee.v12.no.3.2026.pg506.514 Abstract Reference sources including dictionaries, encyclopaedias, directories, bibliographies, handbooks, atlases, indexes, abstracts, and their digital equivalents support quick fact-finding, topic exploration, and access to further information. Yet a library’s ownership of these materials does not, by itself, demonstrate that they are being used effectively or that they meet users’ needs. This article proposes a practical framework for evaluating the utilization of reference sources in the Federal Polytechnic Bali Library. The framework treats utilization as a multidimensional concept involving awareness, access, frequency and purpose of use, user competence, relevance, satisfaction, and contribution to learning and research. It combines usage evidence, user feedback, collection analysis, and staff perspectives, while accounting for barriers such as limited awareness, outdated content, inadequate connectivity, and insufficient user training. The article also presents implementation guidelines covering governance, baseline assessment, instrument design, data collection, analysis, reporting, and periodic review. The proposed framework is intended to support evidence-informed decisions about collection development, reference services, information-literacy instruction, and resource allocation. Because the framework is a proposal rather than a report of completed fieldwork, its indicators and procedures should be adapted and tested in the local institutional context. Keywords: reference sources; resource utilization; academic libraries; evaluation framework; information literacy; Federal Polytechnic Bali Introduction Polytechnic libraries contribute to teaching, learning, research, and professional development by organizing and providing access to relevant information. Among their resources, reference sources have a distinctive role. They are generally consulted to answer specific questions, establish definitions, identify facts, locate people or organizations, trace literature, and gain an initial understanding of a subject. They may be available in print, online, or in both formats. The presence of reference materials in a library is not sufficient evidence of their value to users. A reference collection can be large yet poorly matched to the curriculum, difficult to discover, outdated, or underused. Conversely, a modest collection may be highly valuable if it is current, accessible, well publicized, and closely aligned with users’ information needs. Evaluating www.ijee.io utilization therefore requires more than counting titles or recording loans. It requires examining whether intended users know about the sources, can access them, use them for appropriate purposes, and find them useful (International Organization for Standardization [ISO], 2022; Oakleaf, 2010). This article proposes a framework for evaluating the utilization of reference sources in the Federal Polytechnic Bali Library. It is designed to help the library identify strengths and gaps, improve services, and make defensible decisions about collection development and user support. The framework is presented as a locally adaptable model, not as a description of findings already obtained at the institution. Conceptualizing the Utilization of Reference Sources For this framework, “utilization” refers to the extent to which library users are aware of, can access, and use reference sources to meet academic, research, administrative, or personal information needs. Utilization includes both the act of consulting a source and the value users derive from that consultation (ISO, 2022; ISO, 2014). This definition distinguishes utilization from several related concepts: Availability concerns whether a source is provided by the library. Accessibility concerns whether users can locate and consult it, including whether they face physical, technological, financial, or authentication barriers. Use concerns whether and how often users consult it. Effective use concerns whether the source is appropriate to the task and helps the user obtain reliable, relevant information. Impact concerns the contribution of the source to a learning, teaching, research, or decision- making outcome (ISO, 2014; Oakleaf, 2010). These concepts should not be treated as interchangeable. For example, a low recorded loan count does not necessarily mean a print reference work is unused, since users may consult it within the library. Similarly, a high number of database sessions does not establish that users found relevant material or used it appropriately. Reference sources also vary in format and function. A sound evaluation should include, where applicable: 1. General reference works: dictionaries, encyclopaedias, directories, yearbooks, and almanacs. 2. Subject reference works: specialized handbooks, manuals, technical standards, and subject encyclopaedias. 3. Bibliographic tools: indexes, abstracts, bibliographies, catalogues, and citation databases. 4. Geographical and statistical sources: atlases, maps, statistical compilations, and official data portals. 5. Digital reference resources: online dictionaries, encyclopaedias, databases, institutional repositories, and other curated electronic sources. The boundaries of the category should be agreed upon before data collection. This prevents different evaluators from counting different types of resources under the same label (ISO, 2022). Rationale for an Evaluation Framework A structured framework can help the Federal Polytechnic Bali Library answer practical questions such as: 1. Which reference sources are known and used by students, lecturers, and researchers? 2. What information tasks do users rely on them to complete? 3. Are the available print and electronic resources relevant to the institution’s programmes? 4. Can users find and access the sources without unnecessary difficulty? 5. Which user groups face the greatest barriers? www.ijee.io 6. Do reference services and information-literacy instruction improve users’ ability to select and use appropriate sources? 7. What changes to the collection, technology, staffing, or promotion are most justified? Without a systematic approach, decisions may depend on anecdotal impressions, isolated complaints, or incomplete circulation records. A framework provides common definitions, indicators, and procedures, making it easier to compare results over time and to connect evaluation findings with library planning (Oakleaf, 2010). Proposed Framework The proposed framework is organized into seven connected dimensions: collection relevance and quality; awareness and discoverability; access and availability; patterns and purposes of use; user competence and support; user experience and perceived value; and institutional contribution and improvement. These dimensions should be assessed together rather than treated as independent measures (ISO, 2022; ISO, 2014). 1. Collection relevance and quality This dimension examines whether the reference collection is suitable for the institution’s academic and research activities. Relevant questions include: - Are the resources aligned with the programmes and subjects taught at the polytechnic? - Are editions and online content sufficiently current for their intended use? - Do resources cover major subject areas equitably? - Are authoritative sources available for technical, vocational, scientific, social-science, and general information needs? - Are there significant gaps, duplication, or materials with limited continuing value? Possible indicators include the proportion of resources mapped to academic programmes, the age or edition profile of selected materials, the number of identified subject gaps, and user ratings of relevance. Currency should be judged according to the subject: a historical reference work may remain useful for longer than a technical standard or statistical publication (ISO, 2022; ISO, 2014). 2. Awareness and discoverability A resource cannot be used if intended users do not know it exists or cannot find it. This dimension evaluates whether users are aware of relevant sources and whether the library’s discovery arrangements help them locate those sources. Indicators may include: - The proportion of surveyed users who recognize selected reference sources. - The proportion who can identify where or how to access them. - The ease with which users locate resources through the catalogue, library website, signage, subject guides, or staff assistance. - The visibility and frequency of orientation, demonstrations, and promotional activities. Awareness should be measured separately from use. A user may know about a database but lack access, while another may use a source without knowing that the library formally provides it (Association of College and Research Libraries [ACRL], 2016). 3. Access and availability This dimension concerns the practical conditions under which users consult reference sources. It includes physical access to print materials and technical access to electronic resources. Relevant indicators include: - Whether sources are available when users need them. - The ease of locating print materials on shelves. www.ijee.io - Opening hours and access arrangements. - Availability of computers, electricity, network connectivity, and suitable study space. - Reliability of authentication and remote-access procedures, where provided. - Accessibility for users with disabilities. - Frequency of access failures or service interruptions. Access should be assessed from both administrative records and user experience. For example, a digital resource may be licensed but effectively inaccessible if connectivity is unreliable or users are uncertain how to log in (ISO, 2022). 4. Patterns and purposes of use This dimension describes how frequently, by whom, and for what purposes reference sources are used. It should distinguish among user groups—such as students, academic staff, researchers, and other authorized users—and, where feasible, among programmes or departments. Possible indicators include: - Frequency of consultation over a defined period. - Types of resources consulted and formats preferred. - Main purposes of use, such as assignment preparation, project work, lesson preparation, literature searching, fact-checking, or personal learning. - Use by academic programme or user category. - Repeat use and preferred channels for obtaining assistance. Multiple sources of evidence are important. Circulation records can help measure the borrowing of eligible materials, but they do not capture in-library consultation. For electronic resources, vendor statistics may show sessions, searches, or downloads, but these measures should be interpreted cautiously and consistently. Brief user surveys, observation, and reference-service records can help fill gaps (ISO, 2022; ISO, 2014). 5. User competence and support Effective use depends partly on users’ ability to select suitable sources, formulate search queries, assess credibility, and interpret information. This dimension evaluates the relationship between user skills and the support provided by the library. Indicators may include: - Users’ self-reported confidence in locating and using reference sources. - Performance on short, practical information-search tasks. - Participation in library orientation or information-literacy sessions. - The availability and use of reference assistance, tutorials, guides, and demonstrations. - Staff capacity to support users with print and electronic resources. Self-reported confidence should not be treated as a substitute for demonstrated skill. Where appropriate, the library can use short task-based exercises—for example, asking participants to locate a relevant standard, identify a reliable definition, or find a statistical source—and assess completion using a simple rubric (ACRL, 2016). 6. User experience and perceived value This dimension examines whether users find reference sources and related services relevant, usable, and satisfactory. It may cover: - Perceived usefulness for coursework, teaching, or research. - Ease of use and clarity of instructions. - Satisfaction with the range, currency, and reliability of resources. - Satisfaction with staff assistance and response times. - Reasons for non-use or discontinued use. www.ijee.io Open-ended questions are particularly useful here because they allow users to explain problems not anticipated by the evaluation team. Responses should be analysed systematically rather than relying only on selected quotations (Oakleaf, 2010; ISO, 2014). 7. Institutional contribution and continuous improvement The final dimension connects utilization evidence to decisions and outcomes. It asks whether evaluation results lead to changes in the collection or services and whether those changes make a measurable difference. Possible indicators include: - Collection or service changes made in response to evaluation findings. - Changes in awareness, access, or use after an intervention. - User feedback on the effect of training or resource improvements. - Evidence that reference sources support teaching, learning, or research tasks. - Completion of planned actions within agreed timelines. It may be difficult to attribute academic outcomes solely to reference sources. The framework therefore recommends measuring contribution and perceived support rather than claiming direct causation unless a suitable research design supports such a claim (ISO, 2014; Oakleaf, 2010). Evaluation Matrix The following matrix provides a starting point for operationalizing the framework. The library should refine the indicators according to available records, staffing, and institutional priorities. Dimension Example indicators Suggested evidence Collection relevance and quality Programme coverage; currency; identified gaps; perceived relevance Collection audit; curriculum mapping; user survey Awareness and discoverability Recognition of resources; success in locating them Survey; search task; website and catalogue review Access and availability Availability; access failures; ease of shelf or online access Observation; access logs; user feedback Patterns and purposes of use Frequency, user group, format, and purpose of use Circulation data; electronic usage reports; survey User competence and support Search-task performance; training participation; assistance requests Practical assessment; attendance records; reference logs User experience and value Satisfaction, usability, perceived usefulness, reasons for non-use Survey; interviews; focus groups Institutional contribution and improvement Actions taken; post-intervention changes; user-reported support Action-plan review; repeat measures; interviews Methodological Approach A mixed-methods approach is recommended because no single data source captures the full range of utilization. Quantitative data can show patterns, while qualitative evidence can explain why those patterns occur (ISO, 2022; ISO, 2014). 1. Population and sampling The evaluation population should include the library’s principal user groups: students, academic staff, and other relevant institutional users. Sampling should reflect differences in programme, level of study, and user role where these are important to the research questions. www.ijee.io If the user population is manageable, a census may be practical; otherwise, a transparent sampling strategy should be used. The article does not prescribe a sample size because that depends on the size and structure of the actual population. The library should document its sampling method, response rate, and any limitations on representativeness (ISO, 2022). 2. Data-collection methods A practical evaluation may combine the following: 1. Collection audit: Record titles, formats, subject areas, editions or publication dates, access conditions, and apparent relevance to programmes. 2. User questionnaire: Gather information on awareness, frequency and purpose of use, access barriers, satisfaction, and training needs. 3. Interviews or focus groups: Explore experiences in greater depth, including reasons for non-use and differences among user groups. 4. Usage and service records: Review circulation, in-library consultation where observable, database reports, reference enquiries, and training attendance. 5. Task-based assessment: Test whether users can locate and use selected sources for realistic information needs. 6. Environmental observation: Review signage, shelf arrangement, workstations, connectivity, accessibility, and the visibility of online access instructions. Questionnaire items should be clear, neutral, and tied to the framework’s dimensions. Questions should specify a recall period for example, use during the current semester so that respondents interpret frequency consistently. 3. Data analysis Quantitative results may be summarized using counts, percentages, averages, and comparisons across user groups. Where sample size and data quality permit, the library may examine associations for example, whether training participation is related to reported confidence or successful task completion. Such associations should not automatically be interpreted as causal effects. Qualitative responses should be coded into themes such as relevance, awareness, access, technical barriers, staff support, and training needs. The evaluation team should retain an audit trail describing how categories were developed and how differing interpretations were resolved. The findings should be triangulated. For example, low reported use combined with low awareness may suggest a need for promotion, while low use despite high awareness may point to relevance, access, or usability problems. Usage statistics should be interpreted in context, not treated as a complete measure of value (ISO, 2022). Guidelines for Implementation Step 1: Establish governance and scope The library should appoint a small evaluation team comprising, where possible, library staff, a representative of academic departments, an information-technology representative, and a student or user representative. The team should define the evaluation’s purpose, the resource types included, the user groups covered, and the period under review. Step 2: Agree on definitions and indicators Before collecting data, the team should define what counts as a reference source and how each indicator will be measured. For example, it should distinguish in-library consultation from borrowing and database sessions from successful retrieval of relevant information. Clear definitions improve consistency and make future comparisons more meaningful (ISO, 2022). www.ijee.io Step 3: Conduct a baseline assessment The initial evaluation should establish the current position of the collection, access arrangements, user awareness, and patterns of use. The baseline provides a point of comparison for later interventions. The library should record contextual factors such as examination periods, network interruptions, or changes in enrolment that may affect the results. Step 4: Develop and pilot instruments Questionnaires, interview guides, observation checklists, and task-based assessments should be reviewed for clarity and relevance. A small pilot with users who are not included in the main analysis can identify confusing wording, excessive length, or technical problems. Instruments should be revised before full deployment. Step 5: Collect data ethically and consistently Participation in surveys, interviews, and assessments should be voluntary and based on clear information about the purpose of the evaluation. Personal data should be minimized, stored securely, and reported in a way that does not identify individual respondents. Where institutional procedures require ethical review or administrative approval, the team should obtain it before data collection. Step 6: Analyse and interpret evidence The team should analyse each data source according to the agreed indicators and then compare findings across sources. Results should be disaggregated only where doing so is useful and does not compromise confidentiality. Missing data, low response rates, and limitations in vendor statistics should be reported transparently. Step 7: Translate findings into an action plan Evaluation should lead to specific, prioritized actions. Depending on the evidence, these might include: - Updating or replacing outdated reference materials. - Addressing gaps in subjects linked to teaching programmes. - Improving catalogue records, signage, subject guides, or website instructions. - Strengthening network access or electronic-resource authentication. - Providing targeted information-literacy sessions. - Revising staff procedures for reference assistance. - Promoting underused but relevant sources to particular user groups. Each action should have a responsible person or unit, a completion date, required resources, and a measurable indicator of progress (Oakleaf, 2010). Step 8: Review outcomes and repeat the cycle The library should repeat selected measures after implementing changes. A short-term follow- up may assess whether users are more aware of a resource or whether access problems have declined; a broader review may be conducted periodically according to institutional planning cycles. The purpose is not simply to produce a report but to create a continuing cycle of assessment, action, and review ISO, 2014; ISO, 2022). Using Findings for Library Decision-Making The framework can inform several areas of library management. Collection development: Evidence about curricular relevance, currency, and user demand can support acquisition, renewal, replacement, or withdrawal decisions. Low use should be www.ijee.io considered alongside the resource’s academic importance, accessibility, and potential value; it should not automatically trigger cancellation or disposal (ISO, 2022). Reference services: Enquiry records and user feedback can show which questions are difficult to answer and where staff guidance is most needed. The library can use this evidence to improve service points, referral procedures, or subject support. Information-literacy instruction: Task-based results can identify specific skill gaps, such as difficulty distinguishing authoritative sources, using indexes, or navigating databases. Instruction can then be tailored to actual needs rather than delivered as a generic orientation alone (ACRL, 2016). Technology and access: Reports of failed logins, slow connectivity, or inadequate workstations can help the library and institution prioritize infrastructure improvements. Resource advocacy: A clear account of who uses reference sources, for what purposes, and with what barriers can support funding requests and discussions with academic departments and institutional management (Oakleaf, 2010; ISO, 2014). Challenges and Limitations Several limitations should be anticipated. First, usage statistics are imperfect. Print consultation may not be recorded, while electronic reports may count activity differently across providers. Second, users’ recall of past use may be inaccurate. Third, survey responses can be affected by low participation or by differences in how respondents interpret questions. Fourth, evidence of usefulness does not by itself establish that a reference source caused improved academic performance (ISO, 2014). Resource and staffing constraints may also limit the depth of evaluation. The framework can be implemented in stages, beginning with a manageable set of high-priority indicators. For example, the library might initially assess awareness, access, and perceived relevance for selected resources, then extend the evaluation to additional formats and user groups (ISO, 2022). Finally, the proposed framework has not been validated through field data at the Federal Polytechnic Bali Library in this article. Its suitability should therefore be tested through piloting, review by library and academic stakeholders, and revision in light of local conditions. Conclusion Evaluating reference-source utilization requires attention to more than collection size or recorded transactions. A useful assessment examines whether resources are relevant, discoverable, accessible, used for meaningful purposes, and supported by appropriate instruction and reference services. This article proposes a seven-dimension framework for the Federal Polytechnic Bali Library that combines collection analysis, usage evidence, user feedback, practical skill assessment, and follow-up action.Implementation should begin with clear definitions and a baseline assessment, use multiple data sources, protect participants’ privacy, and connect findings to a prioritized action plan. Repeated evaluation can help the library make better-informed decisions about acquisitions, technology, promotion, and user education. The framework is intended as a practical starting point; local piloting and adaptation are necessary before it can be treated as a validated institutional model. www.ijee.io References Association of College and Research Libraries. (2016). Framework for information literacy for higher education. American Library Association. https://www.ala.org/acrl/standards/ilframework Association of College and Research Libraries. (2018). Standards for libraries in higher education. American Library Association. https://www.ala.org/acrl/standards/standardslibraries International Organization for Standardization. (2022). ISO 11620:2022 Information and documentation—Library performance indicators. ISO. International Organization for Standardization. (2014). ISO 16439:2014 Information and documentation—Methods and procedures for assessing the impact of libraries. ISO. Oakleaf, M. (2010). The value of academic libraries: A comprehensive research review and report. Association of College and Research Libraries. https://www.ala.org/acrl/sites/ala.org.acrl/files/content/issues/value/val_report.pdf Poll, R., & Boekhorst, P. te. (2007). Measuring quality: Performance measurement in libraries (2nd rev. ed.). K. G. Saur.