Handle HTML-escaped characters (e.g. `>` to `>`) in TermTagger responses without skipping terminology markup

XMLWordPrintable

    • High
    • None
    • Fix processing of segments with characters that may be escaped by TermTagger
    • None

      Problem

      When a segment text contains characters such as `>` (commonly seen in navigation instructions like `"Connexions > Interfaces"`), the external TermTagger service converts these characters into HTML entities (such as `>`) in its response. 

      Before applying terminology highlights, Translate5 checks that the text returned by TermTagger matches the requested text. Because `>` does not match `>`, Translate5 assumes the text was modified or corrupted. As a result, Translate5 logs a warning (`E1696: TermTagger changed the plain field text; terminology markup was skipped for this field`) and discards all recognized terms for that segment. The user therefore does not see their terms highlighted in the editor.

      Solution

      1. TermExtractor: Added support for decoding HTML entities ($decodeHtmlEntities = true) when extracting terms from TermTagger responses, which decodes entities back to plain characters and computes term $start and $end character boundaries directly against the decoded text.
      2. TermFinder: When the raw response string from TermTagger does not match the requested segment text, attempt extraction with entity decoding enabled. If the decoded text matches the requested text, accept the terms and their calculated offsets without logging warning `E1696`.

      Test Case for Manual Testing

      Preconditions:

      1. A project/task with TermTagger enabled and a termbase assigned containing at least one term present in the test segment (e.g., term: `KNX` or `adresse IP`).

      Test Steps:

      1. In the editor open a segment containing a `>` character followed by a known term, for example:

      > `dans l’onglet Bus, sous Connexions > Interfaces, relever l’adresse IP du module KNX`

      2. Trigger terminology tagging for this segment (e.g., update and save the segment).
      3. Inspect the segment in the editor UI.
      4. Check the application log (data/logs/php.log or in task/system log).

       

      Expected Result:

      • The terms (e.g. `adresse IP`, `KNX`) are properly recognized and highlighted in the editor.
      • The term markup wraps the exact words (offsets are not shifted or displaced).
      • The warning `E1696` (`"TermTagger changed the plain field text; terminology markup was skipped for this field"`) does *not* appear in the log.

            Assignee:
            Sanya Mikhliaiev
            Reporter:
            Sanya Mikhliaiev
            None
            None
            Thomas Lauria
            Stephan Bergmann
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved:
              None
              None