RECOR: Reasoning-focused Multi-turn Conversational Retrieval Benchmark

A benchmark for evaluating reasoning-intensive conversational information retrieval systems.

Overview

RECOR addresses the gap between traditional conversational search evaluation and the complex reasoning requirements of real-world information-seeking scenarios.

Pipeline

Statistics

Metric	Value
Total Conversations	707
Total Turns	2,971
Domains	11
Avg. Turns per Conversation	4.2

Domains

Source	Domains
BRIGHT	biology, earth_science, economics, psychology, robotics, sustainable_living
StackExchange	Drones, hardware, law, medicalsciences, politics

Installation

git clone https://github.com/RECOR-Benchmark/RECOR.git
cd RECOR
pip install -r requirements.txt

Dataset

Download

Option 1: Python (Recommended)

from datasets import load_dataset

# Load specific subset and domain
benchmark = load_dataset("RECOR-Benchmark/RECOR", "benchmark", split="biology")
corpus = load_dataset("RECOR-Benchmark/RECOR", "corpus", split="biology")

# Load all domains for a subset
all_benchmarks = load_dataset("RECOR-Benchmark/RECOR", "benchmark")  # Returns all splits
all_corpus = load_dataset("RECOR-Benchmark/RECOR", "corpus")  # Returns all splits

# Available domains: biology, earth_science, economics, psychology, robotics,
#                    sustainable_living, Drones, hardware, law, medicalsciences, politics

Option 2: Command Line

# Download entire dataset to local folder
huggingface-cli download RECOR-Benchmark/RECOR --repo-type dataset --local-dir ./RECOR-data

Option 3: Browse & Download Files

Visit HuggingFace Files to browse and download individual files.

Data Format

Benchmark files ({domain}_benchmark.jsonl):

{
  "id": "biology_0",
  "task": "biology",
  "original_query": "How do mitochondria generate ATP?",
  "original_answer": "Mitochondria generate ATP through...",
  "turns": [
    {
      "turn_id": 1,
      "query": "What happens during the electron transport chain?",
      "answer": "The electron transport chain...",
      "gold_doc_ids": ["doc_123", "doc_456"],
      "conversation_history": "No previous conversation.",
      "subquestion_reasoning": "Understanding ETC is foundational...",
      "subquestion_reasoning_metadata": {
        "target_information": "...",
        "relevance_signals": ["..."],
        "irrelevance_signals": ["..."]
      }
    }
  ],
  "metadata": {
    "num_turns": 3,
    "gold_doc_count": 5,
    "version": "unified_v1",
    "created_at": "...",
    "source": "bright",
    "method": "unified_bright_workflow"
  }
}

Document files ({domain}_documents.jsonl):

{"doc_id": "document_id", "content": "Document text content..."}

File Structure

data/
├── benchmark/                              # Conversational benchmark (11 files)
│   └── {domain}_benchmark.jsonl
└── corpus/                                 # Document corpus (11 files)
    └── {domain}_documents.jsonl

Usage

Retrieval

python -m experiments.retrieval.run_retrieval \
    --model bge \
    --dataset_dir . \
    --output_dir ./results

Ablation Study (§5.1)

# Query + Conversation History
python -m experiments.retrieval.ablation_eval \
    --model bge \
    --dataset_dir . \
    --append-history

# Query + Reasoning + Metadata
python -m experiments.retrieval.ablation_eval \
    --model bge \
    --dataset_dir . \
    --append-reasoning --append-reasoning-metadata

RAG Generation + Evaluation

python -m experiments.generation.generate_and_evaluate \
    --retrieval-cache ./results/retrieval_bge \
    --generators "vllm:Qwen/Qwen2.5-14B-Instruct" \
    --output-dir ./rag_results

LLM-as-Judge Evaluation

# Evaluate generated answers (requires Azure OpenAI credentials)
python -m experiments.generation.llm_judge \
    --input ./rag_results/generation_TIMESTAMP

Evaluation Metrics

Retrieval: Recall@K, MRR, nDCG@10

Generation (automatic): ROUGE-L, METEOR, BERTScore

Generation (LLM-judge): Correctness, Completeness, Relevance, Coherence, Faithfulness

Repository Structure

RECOR/
├── experiments/                       # Run experiments on RECOR benchmark
│   │
│   ├── retrieval/                     # RETRIEVAL evaluation → Recall, MRR, nDCG
│   │   ├── retrievers.py              # Model implementations (BGE, BM25, E5, Contriever, DIVER)
│   │   ├── run_retrieval.py           # Run retrieval, compute metrics
│   │   ├── ablation_eval.py           # Ablation: +history, +reasoning, +metadata
│   │   └── evaluate_last_turn.py      # Evaluate with previous turn context
│   │
│   └── generation/                    # GENERATION evaluation → ROUGE, BERTScore, LLM-judge
│       ├── generate_and_evaluate.py   # Generate RAG answers + automatic metrics
│       └── llm_judge.py               # GPT-4 judge (5 dimensions, 1-10 scale)
│
├── pipeline/                          # Benchmark creation (for maintainers)
│   ├── generate_bright.py             # Generate BRIGHT domain conversations
│   ├── generate_stackexchange.py      # Generate StackExchange conversations
│   └── quality_analysis.py            # Validate benchmark quality (4 dimensions)
│
├── assets/
│   └── pipeline.png                   # Dataset generation pipeline diagram
├── requirements.txt
└── README.md

License

MIT License - see LICENSE for details.