-
Type:
Improvement
-
Resolution: Unresolved
-
None
-
Affects Version/s: None
-
Component/s: VisualReview / VisualTranslation
-
Medium
-
None
-
Improvement: Speed up Visual Export of PDF Visuals by paralleling the workload
-
None
-
Emptyshow more show less
PROBLEM
The export/printing of PDFs for visual export can take excessively long since each page needs to be printed individually with the headless browser. PDF with several hundret pages may take up to 30 min to be printed
SOLUTION
The Workload can be parallelized between several headless converter containers (or even with a single one as browserless works with multiple chrome instances internally). This is steered with the usual maxParallelWorkers config. The workload should be split into the maxParallel config with a min batch size of 10 pages
IMPEMENTATION DETAILS
The "Export Visual" process is implemented as a three-step process with a sequence of three workers (see: MittagQI\Translate5\Plugins\VisualReview\Export\ExportOperation and MittagQI\Translate5\Plugins\VisualReview\Worker\QueueingService::queueExportWorkers)
-> The queing can not queue the VisualExportPrintWorker (which needs to be done dynamically in the VisualExportStartWorker). Instead, a dependency of the VisualExportMergePdfWorker to the VisualExportStartWorker must be added to prevent the VisualExportMergePdfWorker starts too early
a) VisualExportStartWorker / ExportSourceCreation: Performing a ExportSourceCreation - what creates from the visual HTML source file (the WYSIWYG version / right iframe) the HTML-files to be printed in the next step
- in case of a visual PDF generated by pdf2htmlEX each "internal" page will be converted/extracted to a single HTML file
- in case of website downloads or image based visuals each "physical" HTML file
- A config-file is generated containing an inventory of the files
-> here the splitting of the workload has to take place and indexed config-files with the need to be created. The worker needs to queue the needed VisualExportPrintWorker and add the index (or the name of the config/inventory file) as worker-param
-> the annotations also need to be distributed to the confid-files with the related page
-> we may add the overall number of pages as new prop in the inventory-files to e.g. evaluate it to detect errors
b) VisualExportPrintWorker / MittagQI\Translate5\Plugins\VisualReview\PrintPdf\PrintPdfs implements the process of printing the individual pages
- Currently, all single HTML pages created in a) are printed as PDF and the individual PDFs are returned to t5. This will be done by the Headless converter
-> the workload must be tailored to the index the worker is responsible for and the returned zip must be extracted to a general folder which will hold the sum of processed single-page PDFs
c) VisualExportMergePdfWorker / MittagQI\Translate5\Plugins\VisualReview\Pdf\MergeAnnotatePdfs
- Will send the individual PDFs to the pdfconverter to merge them and add the annotations
-> here a new check must be added if all expected files have been prointed in b) and an exception generated otherwise
- relates to
-
TRANSLATE-5613 InstantTranslate visual PDF Download: Print PDF as final part of translation process already
- Selected for dev