TQE analysis includes blocked segments and breaks the analysis result

XMLWordPrintable

    • Type: Task
    • Resolution: Unresolved
    • None
    • Affects Version/s: None
    • Component/s: translate5 AI

      Problem

       

      Blocked segments (untranslatable context segments, autoStateId = BLOCKED) are currently included in the TQE evaluation and in the TQE analysis. They are iterated together with all other segments of the task, so two things happen: blocked segments that have an empty target are stored with score 0 (with the reasoning "Empty target segments are not evaluated by LLM and receive score 0 by default." and blocked segments that do have content are really sent to the LLM, which costs tokens for content nobody can edit or is expected to review.

      The result of both is written to the segment (qualityScore), to the segment history aggregation and to the TQE analysis results table. Consequently the TQE analysis grid lists hundreds of 0-score rows with their word counts for segments that are not part of the deliverable work at all.

      What makes this inconsistent: the average quality score itself looks correct, because the task word count used as the denominator (QualityEstimationCalculator::calculateForAnalysis()) already excludes blocked and locked segments (editor_Models_Segment_Meta::getWordCountSum():156). So the denominator ignores blocked segments while the analysis rows count them. The regular match analysis already handles this correctly and skips blocked segments (Plugins/MatchAnalysis/Analysis.php:143), TQE does not.

      Beside the analysis, the 0 score is also visible in the segment grid, in the TQE filter and in the KPI data of these blocked segments, which is misleading for the same reason.

      Solution

      • Exclude blocked segments from the TQE run: when building the segment iterator in the TQE worker, call setIgnoreBlockedSegments(true) (editor_Models_Segment_Iterator::setIgnoreBlockedSegments()), analog to what the match analysis already does.
      • Apply the same rule as the task word count so that numerator and denominator match: skip segments with autoStateId = BLOCKED always, and skip segments with autoStateId = LOCKED when the task has "edit 100% matches" disabled - this is exactly the condition used in editor_Models_Segment_Meta::getWordCountSum(). The iterator only supports the BLOCKED case today, so the LOCKED case has to be added there or filtered in the TQE worker.
      • Effect: no quality score, no reasoning, no aggregation entry and no analysis result row is written for these segments any more, and they are not sent to the LLM, which also reduces the token cost of a TQE run.
      • Existing tasks keep their old data - a re-run of the TQE analysis produces the corrected result. Whether old analysis rows should be cleaned up by a migration is to be decided; the simple path is to leave them and let a re-run fix it.

            Assignee:
            Aleksandar Mitrev
            Reporter:
            Aleksandar Mitrev
            Aleksandar Mitrev Aleksandar Mitrev
            Sylvia Schumacher
            Axel Becher
            None
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              None
              None