bfcl

Specification for a reproducible, provenance-bound multi-lane LLM benchmark suite.

What 200 steps of fully simulated multi-turn tool-use RL do to a 4B policy: every 10th checkpoint scored on BFCL v4, with the pipeline that produced the measurement.

Public benchmark harnesses and reproducible evaluations from PixelSpaceAI

Canonical IR, schema validation, and deterministic delivery for function calls.

A bf16 LoRA fine-tune of Qwen 3.5 4B for function calling on xLAM. v1.0 ships below the BFCL gate with full per-category failure analysis.

Independent audit of a fine-tuned LLM tool-calling PoC — BFCL regression decomposition, inference stack risk assessment, and production recommendation for a FinTech client. Qwen-2.5, LoRA, SGLang, H100.

Task-conditioned tail reliability for tool-using agents under equivalent interfaces

The first Turkish-native tool-calling benchmark: BFCL-style AST scoring, Turkish difficulty layer, HF leaderboard

Paired metamorphic evaluation of tool-calling robustness under realistic user phrasing, built on BFCL.

Post-train a small instruction-tuned LLM for reliable multi-step tool use

Single-GPU trajectory SFT for Qwen3-1.7B: +30 pp in-domain BFCL task success, with open data, LoRA weights and reproducible evaluation.

OpenEuroLLM snapshot of the Berkeley Function Calling Leaderboard evaluation harness and OLMo evaluation orchestration.

A compact, model-first notation for LLM tool definitions. Re-encodes JSON Schema with a median ~30% input-token reduction and behavior-preserving fallback. Includes the specification, a reference converter, a deployment protocol, and the full evaluation.

QLoRA fine-tune of Qwen2.5-1.5B for tool calling on one 8 GiB GPU: +8.5% on BFCL call categories, -55 points on the ability to decline. Refusal data recovers ~49 of them, reproduced across 3 seeds.

Tool-call data and evaluation lab: context isolation, strict JSON scoring, grouped splits, CPU contract evidence, and optional QLoRA training.