diff --git a/README.md b/README.md index b5694a7..7c3fe47 100644 --- a/README.md +++ b/README.md @@ -42,6 +42,7 @@ Directory | Paper [aied-2025-itextbooks](https://github.com/vitalsource/data/tree/main/aied-2025-itextbooks) | [Improving textbook accessibility through AI simplification: Readability improvements and meaning preservation](https://ceur-ws.org/Vol-4010/itb25_s2p2.pdf) [csedu-2026](https://github.com/vitalsource/data/tree/main/csedu-2026) | [Extending an automatic question generation pipeline with LLM-based free-response tasks: An analysis of performance metrics using student data](https://doi.org/10.5220/0014655400004021) ![NEW](https://img.shields.io/badge/status-new-brightgreen) [aied-2026-itextbooks](https://github.com/vitalsource/data/tree/main/aied-2026-itextbooks) | [LLM feedback isn’t automatically better: Static scaffolds outperform dynamic feedback in textbook-embedded practice](https://ceur-ws.org/Vol-4231/itb26_s1p1.pdf) ![NEW](https://img.shields.io/badge/status-new-brightgreen) +[softcom-2026](https://github.com/vitalsource/data/tree/softcom-2026/softcom-2026) | [When correct isn’t enough: A cognitive load theory critical analysis for AI tutor responses](#) ![NEW](https://img.shields.io/badge/status-new-brightgreen) Unless otherwise noted, our datasets are available under the [Creative Commons Attribution 4.0 International diff --git a/aied-2025-evallac/README.md b/aied-2025-evallac/README.md index 1455a7f..d569bad 100644 --- a/aied-2025-evallac/README.md +++ b/aied-2025-evallac/README.md @@ -134,10 +134,10 @@ student use of the open-ended questions in their textbooks in this dataset: - Emond Publishing -- F. A. Davis Company -- Human Kinetics Publishers +- F.A. Davis +- Human Kinetics - OpenStax -- SAGE Publications +- SAGE Publications, Inc. (US) - Taylor & Francis ## Contact Us diff --git a/aied-2025-itextbooks/README.md b/aied-2025-itextbooks/README.md index f47703d..6781547 100644 --- a/aied-2025-itextbooks/README.md +++ b/aied-2025-itextbooks/README.md @@ -153,10 +153,10 @@ student use of the simplifier feature in their textbooks as part of this open dataset: - Emond Publishing -- F. A. Davis Company -- Human Kinetics Publishers +- F.A. Davis +- Human Kinetics - OpenStax -- SAGE Publications +- SAGE Publications, Inc. (US) - Taylor & Francis ## Contact Us diff --git a/aied-2026-itextbooks/README.md b/aied-2026-itextbooks/README.md index 9e383bb..8cd23db 100644 --- a/aied-2026-itextbooks/README.md +++ b/aied-2026-itextbooks/README.md @@ -155,8 +155,8 @@ CoachMe formative practice questions in their textbooks as part of this open dat - Cambridge University Press - Emond Publishing -- F. A. Davis Company -- Human Kinetics Publishers +- F.A. Davis +- Human Kinetics - OpenStax - SAGE Publications, Inc. (US) - SAGE Publications, Ltd. (UK) diff --git a/edm-2025-causaledm/README.md b/edm-2025-causaledm/README.md index 533df3a..d39374a 100644 --- a/edm-2025-causaledm/README.md +++ b/edm-2025-causaledm/README.md @@ -157,7 +157,7 @@ automatically generated questions from their textbooks in this open dataset: - OpenStax -- SAGE Publications +- SAGE Publications, Inc. (US) - Taylor & Francis ## Contact Us diff --git a/its-2025/README.md b/its-2025/README.md index c60518a..cfb26e1 100644 --- a/its-2025/README.md +++ b/its-2025/README.md @@ -115,7 +115,7 @@ The session fields are: ## Acknowledgment -We thank SAGE Publications for granting permission to include student +We thank SAGE Publications, Inc. (US) for granting permission to include student use of the open-ended questions in this dataset. ## Contact Us diff --git a/l@s-2025/README.md b/l@s-2025/README.md index 2b9abdd..91e95cd 100644 --- a/l@s-2025/README.md +++ b/l@s-2025/README.md @@ -150,7 +150,7 @@ automatically generated questions derived from their textbooks as part of this open dataset: - OpenStax -- SAGE Publications +- SAGE Publications, Inc. (US) - Taylor & Francis ## Contact Us diff --git a/softcom-2026/AI_tutor_screenshot.png b/softcom-2026/AI_tutor_screenshot.png new file mode 100644 index 0000000..0faf4de Binary files /dev/null and b/softcom-2026/AI_tutor_screenshot.png differ diff --git a/softcom-2026/CLT Evaluation of AI Tutor Responses.ipynb b/softcom-2026/CLT Evaluation of AI Tutor Responses.ipynb new file mode 100644 index 0000000..7a2a66e --- /dev/null +++ b/softcom-2026/CLT Evaluation of AI Tutor Responses.ipynb @@ -0,0 +1,228 @@ +{ + "nbformat": 4, + "nbformat_minor": 5, + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python", + "version": "3.10.0" + } + }, + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "This notebook contains Python code for reproducing the results in our paper on evaluating and refining AI tutor responses using cognitive load theory:\n\nVan Campenhout, R., Dittel, J. S., & Johnson, B. G. (2026). When correct isn't enough: A cognitive load theory critical analysis for AI tutor responses. In *Proceedings of the 34th International Conference on Software, Telecommunications and Computer Networks (SoftCOM 2026)*. IEEE.\n\n[Official citation and PDF link will be added upon publication.]\n\nThis paper was presented at [SoftCOM 2026](https://2026.softcom.fesb.unist.hr/), the 34th International Conference on Software, Telecommunications and Computer Networks.\n\nResults are presented in the order they occur, organized by the paper's sections. For each result, an excerpt from the paper is given followed by code to compute the result from the dataset provided. Example:\n\n>The final dataset contains 500 randomly sampled interactions across 170 textbooks and 305 students.\n\n`len( interactions ), interactions.textbook_id.nunique(), interactions.student_id.nunique()`\n\nPlease refer to the paper for additional context." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import pandas as pd" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Read dataset of student\u2013AI tutor interactions" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "interactions = pd.read_parquet( 'interactions.parquet' )\ninteractions.head()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Phase 1: Cognitive Load Theory for AI Response Analysis" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Methods" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + ">The final dataset contains 500 randomly sampled interactions across 170 textbooks and 305 students." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "len( interactions ), interactions.textbook_id.nunique(), interactions.student_id.nunique()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + ">Using the Book Industry Standards and Communications major subject heading classification, the top subject domains as a percentage of interactions were Social Science (30.8%), History (11.4%), Business & Economics (9.4%), Psychology (8.4%), and Political Science (8.0%)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "interactions.subject.value_counts( normalize=True ).apply( lambda p: f'{p:.1%}' )" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Results and Discussion" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + ">The correctness judge classified the 500 interactions as 81.8% correct, 13.4% incomplete, and 4.8% incorrect." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "interactions.original_correctness.value_counts( normalize=True ).apply( lambda p: f'{p:.1%}' )" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + ">The CLT judge identified that 29.6% of the tutor's responses contained extraneous information." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "interactions.original_ecl.value_counts( normalize=True ).apply( lambda p: f'{p:.1%}' )" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + ">**Table 1.** Distribution of extraneous cognitive load across tutor response correctness categories." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "pd.crosstab( interactions.original_correctness, interactions.original_ecl )" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Phase 2: Cognitive Load Theory as an AI Revision Agent" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Results and Discussion" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + ">The same correctness analysis was performed on the revised responses, with incomplete responses reduced from 13.4% to 0.6% and incorrect responses reduced from 4.8% to 1.0%." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "interactions.revised_correctness.value_counts( normalize=True ).apply( lambda p: f'{p:.1%}' )" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + ">The overall presence of extraneous content was reduced from 29.6% to 0.8%." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "interactions.revised_ecl.value_counts( normalize=True ).apply( lambda p: f'{p:.1%}' )" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + ">**Table 2.** Distribution of extraneous cognitive load across tutor response correctness categories after CLT-based revision." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "pd.crosstab( interactions.revised_correctness, interactions.revised_ecl )" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + ">The CLT agent reduced the original AI tutor response from 184 words to 60 while still providing the necessary definitions and relationship between terms." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Behaviorism/positivism example from the paper\nexample = interactions[ interactions.original_response.str.contains( 'Behaviorism', na=False ) ].iloc[ 0 ]\noriginal_words = len( example.original_response.split() )\nrevised_words = len( example.revised_response.split() )\nprint( f'Original: {original_words} words' )\nprint( f'Revised: {revised_words} words' )\nprint()\nprint( 'Original response:' )\nprint( example.original_response )\nprint()\nprint( 'Revised response:' )\nprint( example.revised_response )" + ] + } + ] +} \ No newline at end of file diff --git a/softcom-2026/LEP_model.png b/softcom-2026/LEP_model.png new file mode 100644 index 0000000..5b92dc8 Binary files /dev/null and b/softcom-2026/LEP_model.png differ diff --git a/softcom-2026/README.md b/softcom-2026/README.md new file mode 100644 index 0000000..8507554 --- /dev/null +++ b/softcom-2026/README.md @@ -0,0 +1,89 @@ +# Cognitive Load Theory Evaluation of AI Tutor Responses Dataset + +Van Campenhout, R., Dittel, J. S., & Johnson, B. G. (2026). When correct isn't enough: A cognitive load theory critical analysis for AI tutor responses. In *Proceedings of the 34th International Conference on Software, Telecommunications and Computer Networks (SoftCOM 2026)*. IEEE. + +[Official citation and PDF link will be added upon publication.] + +This paper was presented at [SoftCOM 2026](https://2026.softcom.fesb.unist.hr/), the 34th International Conference on Software, Telecommunications and Computer Networks. + +## Description + +Chat-based AI tutors offer new possibilities for personalized student support, but evaluating their quality requires methods that go beyond factual correctness alone. This dataset and accompanying analysis examine 500 student–AI tutor interactions through the lens of cognitive load theory (CLT), specifically targeting extraneous cognitive load (ECL) — content in tutor responses that is not essential to answering the student's query and may burden working memory without supporting learning. + +The AI tutor studied here is a textbook-aligned chat companion deployed within the VitalSource Bookshelf etextbook platform. The tutor appeared alongside textbook content in a chat window and was constrained to answering study-related questions using only licensed textbook material. Students were offered suggested uses including explaining challenging content, practice questions, study plans, and exam preparation. Student participation in the pilot was voluntary and unrelated to course instruction. + +

+A split-screen view of the VitalSource etextbook interface showing a textbook page on the left and an AI tutor chat window on the right, with a student question and tutor response visible. +
Figure 1. An example AI tutor interaction open alongside the etextbook. +

+ +The study proceeded in two phases, organized using the learning engineering process (LEP) model — a cyclical framework for applying learning science principles through iterative development, implementation, and investigation. + +**Phase 1** analyzed the original tutor responses using two LLM-as-judge classifiers developed iteratively with GPT-5 and human review. The correctness judge labeled each response as correct, incorrect, or incomplete (factually accurate but insufficiently answering the query). The ECL judge assessed whether responses contained information beyond what was essential for answering the student's question. + +**Phase 2** developed a CLT-informed revision agent that postprocessed tutor responses to reduce extraneous content while enforcing correctness and completeness. The same 500 interactions were re-evaluated after revision using identical judge criteria, enabling direct before-and-after comparison. + +

+Two side-by-side learning engineering process (LEP) cycle diagrams. Phase 1 shows the cycle of AI tutor development, pilot deployment, data collection, and CLT analysis. Phase 2 shows the cycle of CLT revision agent development and re-analysis of the same 500 interactions. +
Figure 2. The LEP model for Phase 1 (left) and the subsequent LEP model for Phase 2 (right). +

+ +The dataset contains 500 randomly sampled single-turn student–AI tutor interactions collected during a voluntary pilot deployment from May to August 2025, across 170 textbooks and 305 students, drawn from textbooks published by five publishers who granted explicit permission for generative AI research. Multi-turn conversations were excluded to focus on responses interpretable without prior conversational context, and interactions were further limited to queries whose answers could be externally verified through general knowledge. Using the Book Industry Standards and Communications (BISAC) major subject heading classification, the top subject domains as a percentage of interactions were Social Science (30.8%), History (11.4%), Business & Economics (9.4%), Psychology (8.4%), and Political Science (8.0%). + +The research questions addressed in the two phases were: + +1. How prevalent is the inclusion of extraneous cognitive load in AI tutor responses? +2. How does a CLT-informed revision agent impact ECL rates in tutor responses? + +## Example Interaction + +The following example illustrates how the ECL judge classifies a tutor response as extraneous, and how the CLT revision agent subsequently revises it. The interaction comes from a criminology textbook. + +**Student query:** "What is behaviorism and positivism?" + +**Original tutor response:** Behaviorism is an empiricist approach that was pioneered by American psychologist J. B. Watson in the first decade of the 20th century. It derives from a philosophy known as positivism, which emerged from the natural sciences and regards the world as fixed and quantifiable. Behavioral psychologists shifted the research focus away from the realm of the mind and argued that an individual's identity was shaped by their responses to the external environment, which formed stable and recognizable patterns of behavior that could be publicly observed. Watson was inspired by Russian physiologist Ivan Pavlov, who was famously conducting experiments with dogs, producing conditioned responses to external stimuli. Meanwhile, in criminology, the search for objective knowledge through the positive application of science was also having a significant impact. The endeavor to observe and measure the relationship between cause and effect led to a belief that criminality is not a matter of free will but caused by a biological, psychological, or social disposition over which the offender has little or no control. Does that explanation clarify how behaviorism and positivism are related, and how they influenced criminology? + +**ECL judge classification:** `EXTRANEOUS` — The response provides correct core definitions but elaborates on historical origins (Watson, Pavlov, natural sciences) and specific criminological applications that go beyond what the query requires, introducing concepts not essential for schema construction. + +**Revised tutor response (CLT revision agent):** Behaviorism is a psychological approach that studies observable behaviors and their relationship to environmental stimuli, positing that behavior is learned through conditioning. Positivism is a philosophical stance asserting that knowledge should be based solely on empirical evidence and scientific methods, focusing on objectively observable and measurable phenomena. Behaviorism is essentially an application of positivist principles to the study of behavior. + +The CLT agent reduced the original response from 184 words to 60 while preserving the core definitions and the relationship between the two terms. In the dataset, the original response is labeled `EXTRANEOUS` and `CORRECT`; the revised response is labeled `NEED_ONLY` and `CORRECT`. + +## Data Files and Analysis Code + +| File | Description | +|------|-------------| +| `interactions.parquet` | Labeled dataset of 500 student–AI tutor interactions, including original and revised responses and all judge labels | +| `correctness_judge_prompt.txt` | LLM-as-judge prompt used to classify tutor response correctness (`CORRECT` / `INCORRECT` / `INCOMPLETE`) | +| `ecl_judge_prompt.txt` | LLM-as-judge prompt used to classify presence of extraneous cognitive load (`NEED_ONLY` / `EXTRANEOUS`) | +| `revision_agent_prompt.txt` | CLT revision agent prompt used to postprocess tutor responses | +| `CLT Evaluation of AI Tutor Responses.ipynb` | Analysis notebook reproducing all descriptive results reported in the paper | + +The dataset contains the following fields: + +| Field | Type | Description | +|-------|------|-------------| +| `student_id` | `string` | Anonymized student identifier. | +| `textbook_id` | `string` | Unique textbook identifier. | +| `subject` | `categorical` | BISAC major subject heading for the textbook. | +| `student_query` | `string` | Student's original question to the AI tutor. | +| `original_response` | `string` | Original AI tutor response. | +| `original_correctness` | `categorical` | Correctness judge label for original response: `CORRECT`, `INCORRECT`, or `INCOMPLETE`. | +| `original_ecl` | `categorical` | ECL judge label for original response: `NEED_ONLY` or `EXTRANEOUS`. | +| `revised_response` | `string` | Tutor response after CLT revision agent postprocessing. | +| `revised_correctness` | `categorical` | Correctness judge label for revised response: `CORRECT`, `INCORRECT`, or `INCOMPLETE`. | +| `revised_ecl` | `categorical` | ECL judge label for revised response: `NEED_ONLY` or `EXTRANEOUS`. | + +## Acknowledgments + +We gratefully acknowledge the following publishers for granting permission to include student interactions with their textbooks through the AI tutor pilot as part of this open dataset: + +- Emond Publishing +- F.A. Davis +- Human Kinetics +- OpenStax +- SAGE Publications, Inc. (US) + +## Contact Us + +If you have questions, please feel free to email [benny.johnson@vitalsource.com](mailto:benny.johnson@vitalsource.com). diff --git a/softcom-2026/correctness_judge_prompt.txt b/softcom-2026/correctness_judge_prompt.txt new file mode 100644 index 0000000..0c1912e --- /dev/null +++ b/softcom-2026/correctness_judge_prompt.txt @@ -0,0 +1,53 @@ +When Correct Isn't Enough — Correctness judge prompt +====================================================================================== + +Model: gemini-2.5-flash-lite +Temperature: 0.0 + +This prompt classifies a tutor's response to a student query as CORRECT, INCORRECT, +or INCOMPLETE. It was applied to all interactions where the query was externally +verifiable and the tutor produced a substantive answer. + +REPRODUCING THIS PROMPT EXACTLY +------------------------------- +The two blocks below are reproduced verbatim from the deployed source. There are no +indentation artifacts, no smart quotes, and no trailing newlines on either block. +Hashes are SHA-256 over the UTF-8 encoded string as sent to the API: + + system message : 179906bf2c1bf65d628caeb274c147cda38046fa220b0405b8babf6a31c86bf7 + user message : e6a425218d3f0492a5d5263d311c1bf43dfe1ff29f93ef010dcd425b5f243900 + +-------------------------------------------------------------------------------------- +SYSTEM MESSAGE +-------------------------------------------------------------------------------------- + +You are an impartial evaluator. +Your only task is to classify the factual correctness and essential completeness of a tutor's response to a student's query. +Do not answer the query yourself. Do not judge writing style or helpfulness. Follow the rubric strictly. + +-------------------------------------------------------------------------------------- +USER MESSAGE +-------------------------------------------------------------------------------------- + +Classify the tutor's response to the student's query. + +Categories: +- CORRECT: Factually accurate and sufficiently complete. Minor omissions are acceptable if they do not mislead or leave the answer unusable. +- INCORRECT: Contains at least one false or misleading factual claim (including fabricated references or authorities). +- INCOMPLETE: No clear falsehoods, but the response fails to provide the essential information needed to adequately answer the query. + This includes cases where key facts are omitted, the answer is too vague, or it is a non-answer. + +Decision rules: +- Any false or misleading claim → INCORRECT. +- If all claims are true but essential information is missing or the response is a non-answer → INCOMPLETE. + +Respond in plain text only. Format exactly: + +Classification: +Reason: + +STUDENT QUERY: +{query} + +TUTOR RESPONSE: +{response} diff --git a/softcom-2026/ecl_judge_prompt.txt b/softcom-2026/ecl_judge_prompt.txt new file mode 100644 index 0000000..e88f7ea --- /dev/null +++ b/softcom-2026/ecl_judge_prompt.txt @@ -0,0 +1,56 @@ +When Correct Isn't Enough — ECL judge prompt +====================================================================================== + +Model: gemini-2.5-flash-lite +Temperature: 0.0 + +This prompt classifies a tutor's response as NEED_ONLY or EXTRANEOUS based on +Cognitive Load Theory (CLT). It was applied to the same interactions as the +correctness judge. + +REPRODUCING THIS PROMPT EXACTLY +------------------------------- +The two blocks below are reproduced verbatim from the deployed source. There are no +indentation artifacts and no trailing newlines on either block. The system message +uses • (U+2022) Unicode bullets — do not replace with hyphens. Hashes are SHA-256 +over the UTF-8 encoded string as sent to the API: + + system message : ffdecc014557ed9b66c627d7c25f2829286efb59d4c3b5f68337d87a7d4685fa + user message : dbfc4809979d1849164ef40d6be5af11e1f7540224675ab0b82d6a614be759ec + +-------------------------------------------------------------------------------------- +SYSTEM MESSAGE +-------------------------------------------------------------------------------------- + +You are classifying whether a tutor's response to a student query contains only the information essential for understanding the query (**NEED_ONLY**) or whether it also contains extraneous information that adds unnecessary cognitive load (**EXTRANEOUS**). + +Use principles from **Cognitive Load Theory (CLT)** to guide your judgment: + +- **NEED_ONLY**: The response provides the essential definition, explanation, or reasoning needed to answer the query. + • Short illustrative examples, brief analogies, or clarifying follow-up questions that directly reinforce the concept count as *supportive schema construction*, not extraneous. + • A short, generic next-step prompt (e.g., *"Would you like me to search the book?"*, *"Does that make sense?"*) that comes **after** a complete answer is also fine. Count these as **NEED_ONLY**. + +- **EXTRANEOUS**: The response introduces **extraneous information** that is not required to answer the query and may burden working memory. + • This includes tangents, long lists, multiple detailed examples, historical digressions, or new specific topics not implied by the student's query. + • Such information is considered "nice to have" but not essential for schema construction. + +Your goal is to be **conservative in assigning EXTRANEOUS**: +– Default to NEED_ONLY unless the extra information clearly creates extraneous cognitive load under CLT. + +-------------------------------------------------------------------------------------- +USER MESSAGE +-------------------------------------------------------------------------------------- + +Classify the tutor's response: + +STUDENT QUERY: +{query} + +TUTOR RESPONSE: +{response} + +Return the classification on its own line: +Classification: NEED_ONLY | EXTRANEOUS + +Then, on the next line, give a brief explanation: +Reason: diff --git a/softcom-2026/interactions.parquet b/softcom-2026/interactions.parquet new file mode 100644 index 0000000..f95abd4 --- /dev/null +++ b/softcom-2026/interactions.parquet @@ -0,0 +1,3 @@ +version https://git.lfs.github.com/spec/v1 +oid sha256:bafc94f8473dfc032b9c864f7c774525be86c80b75fafaa6b676f29ac41be260 +size 382261 diff --git a/softcom-2026/revision_agent_prompt.txt b/softcom-2026/revision_agent_prompt.txt new file mode 100644 index 0000000..4de06ec --- /dev/null +++ b/softcom-2026/revision_agent_prompt.txt @@ -0,0 +1,73 @@ +When Correct Isn't Enough — CLT revision agent prompt +====================================================================================== + +Model: gemini-2.5-flash-lite +Temperature: 0.0 + +This prompt post-processes an original tutor response to produce a revised response +that would be classified as CORRECT and NEED_ONLY by the judges above. It was applied +to all interactions in the dataset, regardless of the original response's labels. + +REPRODUCING THIS PROMPT EXACTLY +------------------------------- +The two blocks below are reproduced verbatim from the deployed source. There are no +indentation artifacts and no trailing newline on the system message. The user message +ends with a trailing newline (after "Output only the improved final answer."). +Hashes are SHA-256 over the UTF-8 encoded string as sent to the API: + + system message : 2b14e5f23d8972174f81301908e6fdf2af235acb6ade490d644f5de200f441fd + user message : dc534a824856aded275c018c6a1b1786eb1e49aa066b3f072cae3fc1988e88a5 + +-------------------------------------------------------------------------------------- +SYSTEM MESSAGE +-------------------------------------------------------------------------------------- + +You are an editor-evaluator whose job is to improve an AI tutor's answer before it is shown to the student. You will read the student's query and the baseline tutor response. Your task is to revise the answer so that it would be classified as CORRECT and NEED_ONLY by an impartial evaluator following the rubrics below. + +Do not change the style, tone, or voice. +Revise only for factual accuracy, completeness, and conciseness. + +Rubric summary (for internal reference) + +* CORRECT = factually accurate and sufficiently complete. +* INCOMPLETE = true but missing essential information or too vague. +* INCORRECT = contains any false or misleading factual claim. + +* NEED_ONLY = includes only information essential for understanding; brief clarifying phrases or short examples that reinforce comprehension are acceptable. +* EXTRANEOUS = includes tangents, redundancy, or detail that does not contribute directly to answering the question. + +Editing Procedure + +1. Read the student's question and the baseline answer carefully. + +2. Check for incompleteness: +* Identify any essential points, definitions, or reasoning steps missing from the answer. +* Add them clearly and succinctly. + +3. Check for incorrectness: +* Identify any false, misleading, or implausible claims. +* Replace them with accurate statements based on general knowledge. + +4. Check for extraneous content: +* Delete or shorten information that is not required to answer the question or that repeats what was already said. +* Keep short clarifying examples that directly support comprehension. + +5. Final review: +* The first sentence should directly answer the question. +* The entire response should be self-contained, accurate, and concise (≈3–7 sentences). + +6. Output only the revised final answer. +Do not include commentary, explanations, or labels. + +-------------------------------------------------------------------------------------- +USER MESSAGE +-------------------------------------------------------------------------------------- + +STUDENT QUERY: +{query} + +BASELINE ANSWER: +{response} + +Your task: Edit the baseline answer according to the procedure above so that it is factually correct, complete, and contains only essential information. +Output only the improved final answer.