MQM-based TQE: fixes from the first test round

XMLWordPrintable

    • High
    • None
    • Additional improvements to MQM - TQE.
    • None

      Problem

      The first test round of the MQM-based TQE (TRANSLATE-5572) found the following problems:

      1. Wrong overall score: the AI often leaves out segments in its answer (mostly clean ones). These segments got no result and counted as 0, so the overall score was far too low (42 instead of 97 in the test).
      2. MQM criteria and TQE at import: (a) a custom criteria file in the import zip is only used with the exact name QM_Subsegment_Issues.xml in the root folder; otherwise the default criteria are used without any message. (b) After a wizard import no TQE ran, although the client has a TQE default resource. 
      3. Black bar in the segment grid: with track changes on, deleting an MQM tag paints a bold black bar over the whole segment text.
      4. Terminology ignored in MQM mode: the MQM prompt did not send the task terminology, so the AI marked correct terms as errors.
      5. TQE error on tasks without MQM criteria: for example InstantTranslate file translations, where MQM tags are disabled.
      6. Slow runs with many log errors: segments missing from a batch answer were sent one by one while the other batches were still open. These batches then timed out, so a run could take several minutes and fill the task log with errors.
      7. MQM issues lost at protected whitespace: when the AI writes a real space where the text has a protected whitespace, its quote is not found and the issue is dropped.

      Solution

      1. Segments without a usable answer are sent again in new batches (up to three rounds), and only the remaining ones one by one after all batches are done. Every segment gets a result: without an AI answer, MQM mode calculates the score from the existing MQM tags, classic mode gives 0. The task log shows one summary per round (E1798) and one list of segments without an AI answer (E1799). In the test the run took 79 instead of 264 seconds.
      2. (a) The import logs which MQM criteria were used (E1802) and warns about a criteria file with a wrong name or place (E1803). (b) No code change needed: TQE at import works like the pre-translation. In the wizard, "Import (use defaults)" starts TQE with the default resources, resources removed on the TQE card are not used, and "Import (skip next steps)" starts no TQE. API imports start TQE from the client's defaults. InstantTranslate uses its own TQE defaults.
        Improvement: values from default criteria from disk will be inserted/shown in new overwrite config runtimeOptions.editor.qmFlagXmlContent which can be adjusted by user
      3. The "deleted MQM flag" style is only used for a deletion that contains just the flag.
      4. The MQM prompt now lists the task terms, with the rule that the given translation of a term is correct.
      5. Tasks without MQM criteria are scored with classic TQE and a warning (E1800).
      6. See 1: no single requests while batches are open; failed batch requests are retried automatically before they count as failed (E1797).
      7. Quotes are also searched with all whitespace removed.

      Additional improvements:

      • The TQE start button shows a label that matches the TQE mode.
      • TQE analysis results are marked as AI results or as calculated by translate5 (MQM mode without AI resource), and the analysis list shows the matching label.

            Assignee:
            Aleksandar Mitrev
            Reporter:
            Aleksandar Mitrev
            Aleksandar Mitrev Aleksandar Mitrev
            Sylvia Schumacher
            Leon Kiz
            Sylvia Schumacher
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              None
              None