Uploaded image for project: 'translate5'
  1. translate5
  2. TRANSLATE-4602

Improve Batch Exchange & Import by cloning Visuals

XMLWordPrintable

      Problem:

      When exchanging the visual in a batch or importing a task with several target-languages, all workloads of all visual workers before the segmentation-worker are processing identical workloads. That spoils resources, time and costs.

      Solution:

      Only for one task (master-task) the import/exchange of the visual is done and before segmentation the transformed resources are copied over to the other tasks/workers.

      Challenges:

      • The imports waiting for the cloned resources may wait forever, when the master-task fails. "Communication" between workers only possible with flag-files.
      • The process cannot be achieved for VIDEO visuals and XML/XSLT visuals - what is no loss since there no resource-hungry/expensive services are involved
      • The detection, a multi target-language import is a batch does not exist yet and this must be implemented in the core first to enable batching for imports

       

      Implementation Idea:

      • One task will import the visual (master-task), a special "CloneToOthers" worker will be added before segmentation with the other batchTaskGuids and the operation-type as param
      • This worker copies the unsegmented visual and the linked resources (contents of /visual) to all batch-clones. 
      • It also adds a special JSON-file "t5-visual-cloned.json" containing all added DataModels during import in JSON Format (entries from  LEK_visualreview_files & LEK_visualreview_font_taskassoc)
      • The other tasks are queued with a special "ImportClonedResources" Worker (with the master-task Guid and the operation-type as param) that will add the file-based resources from "t5-visual-cloned.json" to the data-base. This worker will delay itself with an interval of  60 sec until he finds the "t5-visual-clone.json". After 5 min waiting it always checks, if the master-task is not erroneus & then also fails. As usual, after max-delay-time the import/exchange fails as with other delays

       

       

            axelbecher Axel Becher
            axelbecher Axel Becher
            Thomas Lauria
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: