REMSA: Foundation Model Selection for Remote Sensing via a Constraint-Aware Agent
Binger Chen, Tacettin Emre B\"ok, Behnood Rasti, Volker Markl, Beg\"um Demir
Why It Matters
What makes this one worth your time
This work addresses the challenge of selecting appropriate foundation models for remote sensing tasks, potentially improving efficiency and accuracy in deploying these models for practical applications.
REMSA automates the selection of remote sensing models using a constraint-aware agent and a comprehensive database.
Summary
The paper introduces REMSA, a constraint-aware agent designed to automate the selection of remote sensing foundation models (RSFMs) based on natural language queries. It leverages a newly created RSFM Database (RS-FMD) containing metadata on over 160 RSFMs. REMSA interprets user input, clarifies constraints, ranks models, and provides justifications, aiming to streamline the model selection process for various remote sensing tasks.
Key contributions
- Introduction of the RSFM Database (RS-FMD) covering over 160 RSFMs.
- Development of REMSA, a constraint-aware agent for automated RSFM selection.
- Evaluation of REMSA using a benchmark of 100 expert-verified RS query scenarios.
Notable insights
- The use of a structured database (RS-FMD) to facilitate model selection is a clever approach to handle the complexity of remote sensing tasks.
- Incorporating a task-driven decision workflow that interprets and clarifies user constraints enhances the usability of the system.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2511.17442v3 Announce Type: replace-cross Abstract: Foundation Models (FMs) are increasingly integrated into remote sensing (RS) pipelines. These models include unimodal vision encoders and multimodal architectures. FMs are adapted to diverse perception tasks, such as image classification, change detection, and visual question answering. However, selecting the most suitable remote sensing foundation model (RSFM) for a specific task remains challenging due to scattered documentation, heterogeneous formats, and complex deployment constraints. To address this, we first introduce the RSFM Database (RS-FMD), the first structured and schema-guided resource covering over 160 RSFMs trained on various data modalities, spanning different spatial, spectral, and temporal resolutions, considering different learning paradigms. Built upon RS-FMD, we further present REMSA, a constraint-aware agent that enables automated RSFM selection from natural language queries. REMSA combines structured FM metadata retrieval with a task-driven decision workflow. In detail, it interprets user input, clarifies missing constraints, ranks models via in-context learning, and provides transparent justifications. Our system supports various RS tasks and data modalities, enabling personalized, reproducible, and efficient FM selection. To evaluate REMSA, we construct a benchmark of 100 expert-verified RS query scenarios. Each query is evaluated across 4 systems and 3 LLM backbones, with the top-3 selected models manually assessed by domain experts. This results in 3,000 expert-scored task--system--model configurations under our novel expert-centered evaluation protocol. REMSA outperforms multiple baselines, showing its practical utility in real decision-making applications. REMSA operates entirely on publicly available metadata of open source RSFMs, without accessing private or sensitive data.