Evaluating AI in Library Discovery

Stephen Zweibel · CUNY Graduate Center

AI is already in your discovery tools

Primo Research Assistant

Conversational search, multi-document synthesis

EBSCO AI

Research assistant in EBSCOhost

JSTOR AI · Scopus AI · Semantic Scholar

AI across research databases

  • Conversational search
  • Semantic search and navigation help
  • AI-generated summaries and "key points"
  • Multi-document synthesis across sources

Not requested by librarians. Arrived as vendor defaults.

Two Kinds of AI

Some of these systems are moving from discovery tools toward answer machines.

Discovery assistance vs. answer generation

 Discovery Assistance

  • Helps find and navigate sources
  • Suggests relevant materials
  • Leaves synthesis to the learner

 Answer Generation

  • Synthesizes across sources
  • Generates narrative summaries
  • Blends sources into a single answer

Cognitive offloading

A student who receives a summary of five articles has not done the work of reading them.

Gerlich (2025)

Cognitive offloading to AI → reduced critical thinking
r = −0.75

Strongest among younger, less experienced researchers

Deng et al. (2025)

Meta-analysis, 69 studies: ChatGPT improves performance while reducing mental effort

Better metrics may reflect the tool, not the learning

Source-forward design

AI assists with finding. Synthesis stays with the learner.

Remove unhelpful friction

  • Confusing database interfaces
  • Boolean syntax requirements
  • Navigating unfamiliar library systems

Preserve valuable friction

  • Reading actual sources
  • Comparing arguments
  • Forming your own conclusions

The CUNY AI Discovery Guide

Seven evaluation criteria · Adopted October 2025

1  Transparency & Citation Traceability

Students should see what was searched, which sources were used, and how each claim ties back to evidence.

Look for

  • Inline citations for each claim
  • Query breakdown showing interpretation
  • Clear indication of which corpus was searched

Red flags

  • Citations only in aggregate lists
  • Synthesis without showing query logic
  • No way to see what was actually searched

Examples from current tools

  • Scopus AI: inline references with clickable links, Copilot query breakdown
  • JSTOR: grounded in the specific item being read, clear source attribution

2  User Agency

The interface should pull students back into the sources.

Look for

  • Prompts to explore, read more, follow up
  • Socratic scaffolding (questions, not answers)
  • AI output as starting point, not destination

Red flags

  • Responses formatted as complete answers
  • No friction between retrieving and moving on
  • Design that signals "you're done"

Examples from current tools

  • ChatGPT Study Mode: steps through problems incrementally
  • Semantic Scholar: TLDRs keep users anchored to the paper

3  Discovery Assistance vs. Answer Generation

The tool can support discovery. Reading and synthesis remain student work.

Look for

  • Search refinement, query expansion
  • Citation networks, concept maps
  • Per-document previews for triage
  • Extractive highlights pointing to passages

Red flags

  • Multi-source synthesis into unified prose
  • Responses that read like literature reviews
  • "Research shows..." from limited results

Examples from current tools

  • Primo RA & Scopus AI: Weak. Blends sources into narratives
  • EBSCO AI: Strong. Presents sources for direct engagement

4  Sensitive-Topic Search

Safety filters designed for consumer chat do not map cleanly onto scholarly research.

Examples from current tools

Primo Research Assistant

Blocked or errored on: "Gaza war" · "Tulsa race massacre" · "abortion access" · "transgender rights"

Likely from Azure OpenAI upstream content moderation

JSTOR AI, EBSCO AI, Scopus AI: no widespread blocking reported as of late 2025

5  Privacy & Data Ethics

Patron queries should remain confidential and not be used to train commercial models. Data flows should be documented.

Look for

  • Written commitment: queries not used for training
  • Named AI service provider
  • Data processing agreement available

Red flags

  • Privacy policy absent or generic
  • "May use data to improve services"
  • Can't name which AI company processes queries

Examples from current tools

  • Scopus AI: explicitly states prompts sent to Azure OpenAI are private, not used for training

6  Library Alignment & Access

AI links should take students to copies the library can actually provide.

Look for

  • Priority linking to subscribed content
  • Clear labels: subscription vs. open access
  • Integration with link resolvers

Red flags

  • Links bypass proxy or authentication
  • Suggests sources library doesn't have
  • "Access denied" after following AI links

7  Librarian Control

Libraries need to be able to configure the system, review its behavior, and turn features off.

Look for

  • Toggle switches to enable/disable features
  • Role-based settings (faculty vs. students)
  • Usage statistics and analytics access
  • Configurable scope and disclosure language

Red flags

  • Features auto-enabled without notice
  • Configuration requires a support ticket
  • A/B testing on patrons without consent

How the tools compare

Tool Transparency Agency Discovery vs. Answers Sensitive Topics Privacy Library Alignment Control
JSTOR AI Strong Adequate Adequate Adequate Strong Strong Adequate
Primo RA Adequate Weak Weak Weak Adequate Adequate Adequate
EBSCO AI Strong Strong Strong Adequate Strong Strong Unknown
Scopus AI Strong Adequate Weak Adequate Strong Adequate Unknown
Semantic Scholar Adequate Adequate Strong Adequate Adequate Weak N/A
CUNY AI Discovery Guide, August–September 2025. Tools evolve; verify current capabilities.

Support learning over efficiency

  • Use AI to help students find sources
  • Keep reading and synthesis with the learner
  • Test tools with the Guide
  • Prefer designs that preserve source engagement

Guide and Article

  • CUNY Libraries AI Discovery Guide
    Evaluation criteria and vendor comparison
  • "Evaluating AI Discovery Tools for Academic Libraries"
    Zweibel, under review at C&RL

With the CUNY Library Research AI Task Force

  • John DeLooper, Lehman
  • Mark Eaton, Kingsborough CC
  • Iris Finkel, Hunter
  • Kelly Karst, Brooklyn
  • Robin Naughton, Queens
  • Anne O'Reilly, LaGuardia CC
  • Jana Porter, Bronx CC
  • Alevtina Verbovetskaya, OLS

Stephen Zweibel

szweibel@gc.cuny.edu