Workshop Series: “Understanding and Evaluating Complex Automation Systems for Journals”

Titled “Understanding and Evaluating Complex Automation Systems for Journals,” this new workshop-discussion series responds to the concerns and needs raised during the workshop series held in winter 2025. Our goal is to give teams from partner journals involved in the project the opportunity to better understand AI systems and to gain practical tools for the qualitative (and quantitative) evaluation of the tools and algorithms that are transforming your professional practices. Each session is based on a 30-minute presentation focused on a theme illustrated with cases directly related to the role of journals (e.g., understanding complex AI architectures such as RAG and agent-based AI, evaluating automatically generated abstracts, understanding common myths about AI).

Schedule

February 12: Introduction to Complex System Architectures Integrating “AI” – Alexia Schneider

This first session is a theoretical introduction to current systems that integrate LLMs into their operations. Most tools — from general-purpose chatbots to specialized information retrieval or text-editing tools — rely on complex processing of user requests. To clarify the challenges of this automation and understand where evaluation matters for users, I will present two so-called “complex” architectures that integrate large language models: agent-based AI systems and RAG (Retrieval-Augmented Generation).

March 19: Evaluation of LLMs for the Production of an Article’s Abstract (topic subject to change) – Gauransh Kumar

April 16: Myths of Automation, the Example of HTR (topic subject to change) – Alix Chagué