文本差异评分器

🌐 Textual Difference Scorer

createTextualDifferenceScorer() 函数使用序列匹配来测量两个字符串之间的文本差异。它提供有关更改的详细信息，包括将一个文本转换为另一个文本所需的操作次数。

🌐 The createTextualDifferenceScorer() function uses sequence matching to measure the textual differences between two strings. It provides detailed information about changes, including the number of operations needed to transform one text into another.

参数

🌐 Parameters

createTextualDifferenceScorer() 函数不接受任何选项。

🌐 The createTextualDifferenceScorer() function does not take any options.

此函数返回 MastraScorer 类的一个实例。有关 .run() 方法及其输入/输出的详细信息，请参见 MastraScorer 参考。

🌐 This function returns an instance of the MastraScorer class. See the MastraScorer reference for details on the .run() method and its input/output.

.run() 返回

🌐 .run() Returns

runId:

string

The id of the run (optional).

analyzeStepResult:

object

Object with difference metrics: { confidence: number, changes: number, lengthDiff: number }

score:

number

Similarity ratio (0-1) where 1 indicates identical texts.

.run() 返回的结果形状如下：

{
  runId: string,
  analyzeStepResult: {
    confidence: number,
    ratio: number,
    changes: number,
    lengthDiff: number
  },
  score: number
}

评分详情

🌐 Scoring Details

评分者计算几项指标：

🌐 The scorer calculates several measures:

• 相似度比率：基于文本之间的序列匹配（0-1）
• 更改：需要的不匹配操作次数
• 长度差异：文本长度的标准化差异
• 置信度：与长度差成反比

评分流程

🌐 Scoring Process

1. 分析文本差异：
   • 执行输入和输出之间的序列匹配
   • 计算所需更改操作的数量
   • 测量长度差异
2. 计算指标：
   • 计算相似度比率
   • 确定置信度评分
   • 合并为加权分数

最终得分：(similarity_ratio * confidence) * scale

🌐 Final score: (similarity_ratio * confidence) * scale

分数解释

🌐 Score interpretation

文本差异得分在0到1之间：

🌐 A textual difference score between 0 and 1:

• 1.0：文本相同——未检测到差异。
• 0.7–0.9：轻微差异——只需少量修改。
• 0.4–0.6：中等差异——需要明显的修改。
• 0.1–0.3：主要差异——需要进行大幅修改。
• 0.0：完全不同的文本。

示例

🌐 Example

衡量预期输出和实际代理输出之间的文本差异：

🌐 Measure textual differences between expected and actual agent outputs:

src/example-textual-difference.ts

import { runEvals } from "@mastra/core/evals";
import { createTextualDifferenceScorer } from "@mastra/evals/scorers/prebuilt";
import { myAgent } from "./agent";

const scorer = createTextualDifferenceScorer();

const result = await runEvals({
  data: [
    {
      input: "Summarize the concept of recursion",
      groundTruth:
        "Recursion is when a function calls itself to solve a problem by breaking it into smaller subproblems.",
    },
    {
      input: "What is the capital of France?",
      groundTruth: "The capital of France is Paris.",
    },
  ],
  scorers: [scorer],
  target: myAgent,
  onItemComplete: ({ scorerResults }) => {
    console.log({
      score: scorerResults[scorer.id].score,
      groundTruth: scorerResults[scorer.id].groundTruth,
    });
  },
});

console.log(result.scores);

有关 runEvals 的更多详细信息，请参阅 runEvals 参考。

🌐 For more details on runEvals, see the runEvals reference.

要将此评分器添加到代理中，请参阅 评分器概览 指南。

🌐 To add this scorer to an agent, see the Scorers overview guide.

相关

🌐 Related

• 内容相似度评分器
• 完整性评分器
• 关键词覆盖评分器