Evaluation of Information Access Agents under Simulated AI Marketplace Dynamics

Modern information access ecosystems consist of mixtures of systems, such as information retrieval systems and large language models. These ecosystems increasingly rely on marketplaces to mediate access to models, tools, and data, making competition between systems an inherent part of deployment. In such settings, outcomes are shaped not only by benchmark quality but also by competitive pressure, including user switching, routing decisions, and operational constraints. Yet evaluation is still largely conducted on static benchmarks with accuracy-focused measures that assume systems operate in isolation. This mismatch makes it difficult to predict which systems will succeed after deployment and obscures competitive effects such as early-adoption advantages and market dominance. We introduce Marketplace Evaluation, a simulation-based paradigm that evaluates information access systems as participants in a competitive marketplace. By explicitly modeling and simulating repeated interactions and the evolving preferences of both users and agents, the framework enables longitudinal evaluation and yields marketplace-level metrics, such as retention and market share, that complement and can extend beyond traditional accuracy-based metrics. We formalize the framework and outline a research agenda, motivated by business and economic perspectives, around marketplace simulation, evaluation metrics, optimization, and adoption within evaluation campaigns such as TREC.