Details

    • Sub-task
    • Resolution: Invalid
    • None
    • None
    • Installation & Update

    Description

      File updater

      After long discussion we decided to use an own system to fetch and deploy files, instead using pyrus as distribution and file roleout system.
      Since the file updater is called from frontend, it runs as web server user. That means that file write permissions has to be set accordingly. The client is responsible to ensure the correct file permissions. There are multiple ways to do this:

      • files and directories belong to the same user as the server is running
      • if files should be kept as a different user, the client has to provide a different “updater” vhost with suPHP or apache MPM.

      Own file fetcher

      All fetched external libraries are only extracted and not moved. We don't move / copy subdirectories from the extracted library to another place (currently we use only lib/Horde directory from whole Horde Package). That means that we have to configure the include paths in the application.ini per downloaded package.
      We will provide a simple “fetch” configuration to simplify the download / fetch process.

      JSON config structure:

      [{
      	name: “zend”,						→ internal name of the package
      	label: “Zend Foo Framework 1.2.3”		→ human readable name of the package, used in frontend
      	url: “http://foo.bar.de/lala-1.2.3.zip”, 	→ URL where to get the package
      	basename: “lala-1.2.3.zip”			→ basename as which the download is saved. Optional, if omitted the URL basename is used.
      	version: “1.2.3”, 					→ desired version as simple string for internal usage
      	target: “library/”,					→ target base directory where the content of the downloaded file can be extracted. Depends on the internal data structure of the zip (if no subdir in the zip is given, we have to provide it here). Target can be omitted, default is “library/”
      	symlink: [“library/lala-1.2.3”, “public/lala-1.2.3”], → for ExtJS or Horde
      	md5: “hashOfZip” 					→ md5 hash of the desired file to download, if download does not match: exit.
      	license: “path/to/licensefile/after/extraction” → path to the license file which content should be displayed to the user. Is mandatory.
      	licenseDisabled: true				→ Optionally, we should have the possibility to disable the license check, since for each update we dont want to reaccept our license.
      }]
      

      The Config Array can contain multiple objects, each Object represents one package.

      Dependency Libray

      The JSON dependency informations reside on two places:
      application/config/dependencies.json → these are the original dependencies as required by the release
      library/dependencies.json → these are the installed dependencies, content is same JSON as above, plus the fields:
      fetched:timestamp when fetched the external
      licenseAccepted: timestamp when (and if) accepted the license of the external.
      In Updating process we have only to compare these two JSON to determine if we have to: fetch, unzip, symlink or accept the license of one of the externals.
      The fetch algorithm should be able to recognize partly fetch steps: that means if the filepart of the url exists in the target dir (ext-4.0.1.zip already exists) and md5 hash is correct, we dont download the file again, but use the existing zip. Same for symlinking, if only the symlink targets are missing but the extracted data exists, we don't download the package again, we restore only the symlinks.
      With this algorithm we have only to run the “dependency check” after updating to restore the extjs dirs in the updated public directory.
      Through comparing the two dependency files we can also delete unused externals.

      Thoughts to Directory Layout

      We have to refactor our directory layout to simplify the exchange of files on update. Our thoughts are sketched afterwards:

      Based on beo, with comments and thoughts for planing:

      application
      application/config → new with config refactoring (TRANSLATE-123)
      application/config/dependencies.json → contains the externals definition for our fetcher
      application/docs → new with structure refactoring
      application/docs-beospezifisch → new with structure refactoring
      application/version → new with structure refactoring
      application/application.ini → goes to application/config
      application/Bootstrap.php
      application/extVersionMapping.ini → merged into app.ini
      application/factoryOverwrites
      application/helpDeactivated.ini → merged into module.ini
      application/iniOverwrites
      application/languages.ini → removed, goes to DB
      application/layouts
      application/modules
      data → created empty
      docs → goes to application/
      docs-beospezifisch → goes to application/
      index.php → BEO special, should be done by rewrite rules
      library → created empty (remains as container for our libs)
      public → remains?
      publish.sh → obsolete
      scripts → obsolete
      seleniumTests → obsolete
      tests → obsolete
      update.sh → obsolete
      version → goes to application

      Resulting Structure install

      After extracting the installation package one gets:

      	app-foo-1.2/application
      	app-foo-1.2/data			← only this should be writeable by default apache
      	app-foo-1.2/updates	
      	app-foo-1.2/public			← this dir has to be choosen as htdocs
      	app-foo-1.2/library			← contains our libs, must be copied to master library dir
      

      All these directories and the parent app-foo-1.2/ must be writeable by the updater!
      Admin should rename “app-foo-1.2” to his needs: should be ”app-foo” only, since the installation is updateable and therefore version independent.

      Operations of the updater:

      Get update and extract it (can be done by default fetcher):

      	app-foo/updates/app-foo-1.2.3.zip
      	app-foo/updates/app-foo-1.2.3/
      

      loop over app-foo-1.2.3/ contents and do for each (expect data and library):

      	rename app-foo/FOUND app-foo/FOUND-before-1.2.3
      	rename app-foo/updates/app-foo-1.2.3/FOUND  app-foo/FOUND
      

      library needs a special treatment: copy the files from app-foo/updates/library to app-foo/library. If it already exists delete the existing directory before.

      Exceptions for public directory:

      After replacing the public directory we have to restore some data. First we have to restore imageTags and imageJsons, copy them from app-foo/public-before-1.2.3 to app-foo/public. Affected files: ./modules/editor/images/imageTags/ of the ????????????????????????????????-[right|single|left].png
      For easier copying, we move the generated tags into a own directory which has then to be copyied. This need a fix in the JS where the URL of the tags are set.
      If a path based separation is not possible in JS, we can do this separation be a regex based rewrite rule.

      ./modules/editor/images/imageTagsJson/ → Feature is unused, so do not generate the JSON files anymore and remove this directory!

      Second we have to restore external packages in public, currently this is extjs (in different versions).
      The GET to fetch the update package zip can be done by a generated fetcher config (same syntax as in dependencies.json):

      [{
      	name: “app-foo”,
      	url: “https://mittagqi.com/supportreleases/stable/app-foo-1.2.3.zip”,
      	version: “1.2.3”,
      	target: “updates/”,
      	post: {
      		randomValue: ZZZZ
      		supportKeyHash: YYYY
      		supportId: XX
      	},		→ instead of using post we can pack alle the values into the URL (for logging and error tracing)??? 
      	md5: “hashofzip” 
      }]
      

      All values expect post and target are delivered from the fetched meta files from our release channel.
      Where YYYY must be a type of one time key based on the support id XX, and the only internal stored support key (UUID). The support key therefore is not transferred as plain key, only hashed.

      Impacts on Meta Data

      Our releases Meta Data is delivered by mittagqi.com/supportreleases/
      This URL provides an json array with available packages.
      The JSON is provided only if a valid supportId and a support key hash and a radom value is given in the POST request. Also the count of current users is posted.
      POST to our server:

      randomValue: ZZZZ
      supportKeyHash: YYYY
      supportId: XX
      userCount: 123
      

      The resulting JSON contains:

      [{
      	name: “app-foo”,
      	url: “https://mittagqi.com/supportreleases/stable/app-foo-1.2.3.zip”,
      	version: “1.2.3”,
      	updateableversion: “*”, [or something like “1.2.3” or “>1.2.3” or “=1.2.3” or “>=1.2.3”]
      	autoupdateable: true |  false,
      	dbbackupneeded: true | false,
      	md5: “hashofzip”  → hash this again with the support key for security reasons!
      }]
      

      The Admin can then choose a update out of the above array / list in the wizard. If the admin decides to install a update, the updater fetches from the above URL (plus the support key hash as described above in the fetcher config definition).

      Our Support / Download Server is defined in TRANSLATE-134!

      Attachments

        Activity

          People

            Unassigned Unassigned
            tlauria Thomas Lauria
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: