AI pre-translation fails when a term ends with a quote character

XMLWordPrintable

    • High
    • None
    • FIX: pre-translation with AI may fails when a text/segment was ending with a quote
    • None

      Problem

      AI pre-translation fails when the terminology of a task contains a term that ends with a straight quote character, for example the French term Vitesse "Override".

      When translate5 builds the request for the AI translation service, it includes the matching terminology. During this step it converts straight quotes (") into typographic quotes (“ ” or „ ”). This conversion has a bug. It damages every text that ends with a quote. The damaged text cannot be sent to the AI service anymore, and the request fails before it is sent.

      The consequences for the user:

      • In batch pre-translation, all segments of the affected batch stay untranslated, even though only one term is damaged. The log shows "OpenAI batch: failed to encode the chat payload to JSON" (E9999). For a file translation, InstantTranslate shows "The file could not be fully translated, most likely because an external automatic translation source was not working properly" (E1794).
      • A single query against the resource, for example a test query in the language resources overview, fails with a JsonException.

      One such term is enough to make every translation of matching content fail. The term itself is stored correctly; only the request building damages it. This makes the error hard to trace, also because the current log message does not say which part of the request is broken.

      Solution

      The bug is in editor_Utils::typographizeQuotes(). The function replaces the straight quote (1 byte) with a typographic quote (3 bytes in UTF-8). When the text ends with a quote, it then swaps this last quote for a closing quote with substr($text, 0, -1). This removes only 1 byte of the 3-byte quote character and leaves a broken 2-byte fragment behind. The result is invalid UTF-8, and PHP refuses to JSON-encode it:

      $result = editor_Utils::typographizeQuotes('Vitesse "Override"', 'fr-FR');
      // expected: Vitesse “Override”
      // actual bytes: Vitesse “Override\xE2\x80”  (broken fragment before the closing quote)
      json_encode($result); // returns false: "Malformed UTF-8 characters"
      

            Assignee:
            Aleksandar Mitrev
            Reporter:
            Aleksandar Mitrev
            Aleksandar Mitrev Aleksandar Mitrev
            Axel Becher
            Axel Becher
            NO-FRONTEND-TESTING NEEDED
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved:
              None
              None