File updater

After long discussion we decided to use an own system to fetch and deploy files, instead using pyrus as distribution and file roleout system.
Since the file updater is called from frontend, it runs as web server user. That means that file write permissions has to be set accordingly. The client is responsible to ensure the correct file permissions. There are multiple ways to do this:

• files and directories belong to the same user as the server is running
• if files should be kept as a different user, the client has to provide a different “updater” vhost with suPHP or apache MPM.

Own file fetcher

All fetched external libraries are only extracted and not moved. We don't move / copy subdirectories from the extracted library to another place (currently we use only lib/Horde directory from whole Horde Package). That means that we have to configure the include paths in the application.ini per downloaded package.
We will provide a simple “fetch” configuration to simplify the download / fetch process.

JSON config structure:

[{
	name: “zend”,						→ internal name of the package
	label: “Zend Foo Framework 1.2.3”		→ human readable name of the package, used in frontend
	url: “http://foo.bar.de/lala-1.2.3.zip”, 	→ URL where to get the package
	basename: “lala-1.2.3.zip”			→ basename as which the download is saved. Optional, if omitted the URL basename is used.
	version: “1.2.3”, 					→ desired version as simple string for internal usage
	target: “library/”,					→ target base directory where the content of the downloaded file can be extracted. Depends on the internal data structure of the zip (if no subdir in the zip is given, we have to provide it here). Target can be omitted, default is “library/”
	symlink: [“library/lala-1.2.3”, “public/lala-1.2.3”], → for ExtJS or Horde
	md5: “hashOfZip” 					→ md5 hash of the desired file to download, if download does not match: exit.
	license: “path/to/licensefile/after/extraction” → path to the license file which content should be displayed to the user. Is mandatory.
	licenseDisabled: true				→ Optionally, we should have the possibility to disable the license check, since for each update we dont want to reaccept our license.
}]

The Config Array can contain multiple objects, each Object represents one package.

Dependency Libray

The JSON dependency informations reside on two places:
application/config/dependencies.json → these are the original dependencies as required by the release
library/dependencies.json → these are the installed dependencies, content is same JSON as above, plus the fields:
fetched:timestamp when fetched the external
licenseAccepted: timestamp when (and if) accepted the license of the external.
In Updating process we have only to compare these two JSON to determine if we have to: fetch, unzip, symlink or accept the license of one of the externals.
The fetch algorithm should be able to recognize partly fetch steps: that means if the filepart of the url exists in the target dir (ext-4.0.1.zip already exists) and md5 hash is correct, we dont download the file again, but use the existing zip. Same for symlinking, if only the symlink targets are missing but the extracted data exists, we don't download the package again, we restore only the symlinks.
With this algorithm we have only to run the “dependency check” after updating to restore the extjs dirs in the updated public directory.
Through comparing the two dependency files we can also delete unused externals.

Thoughts to Directory Layout

We have to refactor our directory layout to simplify the exchange of files on update. Our thoughts are sketched afterwards:

Based on beo, with comments and thoughts for planing:

application
application/config → new with config refactoring (TRANSLATE-123)
application/config/dependencies.json → contains the externals definition for our fetcher
application/docs → new with structure refactoring
application/docs-beospezifisch → new with structure refactoring
application/version → new with structure refactoring
application/application.ini → goes to application/config
application/Bootstrap.php
application/extVersionMapping.ini → merged into app.ini
application/factoryOverwrites
application/helpDeactivated.ini → merged into module.ini
application/iniOverwrites
application/languages.ini → removed, goes to DB
application/layouts
application/modules
data → created empty
docs → goes to application/
docs-beospezifisch → goes to application/
index.php → BEO special, should be done by rewrite rules
library → created empty (remains as container for our libs)
public → remains?
publish.sh → obsolete
scripts → obsolete
seleniumTests → obsolete
tests → obsolete
update.sh → obsolete
version → goes to application

Resulting Structure install

After extracting the installation package one gets:

	app-foo-1.2/application
	app-foo-1.2/data			← only this should be writeable by default apache
	app-foo-1.2/updates	
	app-foo-1.2/public			← this dir has to be choosen as htdocs
	app-foo-1.2/library			← contains our libs, must be copied to master library dir

All these directories and the parent app-foo-1.2/ must be writeable by the updater!
Admin should rename “app-foo-1.2” to his needs: should be ”app-foo” only, since the installation is updateable and therefore version independent.

Operations of the updater:

Get update and extract it (can be done by default fetcher):

	app-foo/updates/app-foo-1.2.3.zip
	app-foo/updates/app-foo-1.2.3/

loop over app-foo-1.2.3/ contents and do for each (expect data and library):

	rename app-foo/FOUND app-foo/FOUND-before-1.2.3
	rename app-foo/updates/app-foo-1.2.3/FOUND  app-foo/FOUND

library needs a special treatment: copy the files from app-foo/updates/library to app-foo/library. If it already exists delete the existing directory before.

Exceptions for public directory:

After replacing the public directory we have to restore some data. First we have to restore imageTags and imageJsons, copy them from app-foo/public-before-1.2.3 to app-foo/public. Affected files: ./modules/editor/images/imageTags/ of the ????????????????????????????????-[right|single|left].png
For easier copying, we move the generated tags into a own directory which has then to be copyied. This need a fix in the JS where the URL of the tags are set.
If a path based separation is not possible in JS, we can do this separation be a regex based rewrite rule.

./modules/editor/images/imageTagsJson/ → Feature is unused, so do not generate the JSON files anymore and remove this directory!

Second we have to restore external packages in public, currently this is extjs (in different versions).
The GET to fetch the update package zip can be done by a generated fetcher config (same syntax as in dependencies.json):

[{
	name: “app-foo”,
	url: “https://mittagqi.com/supportreleases/stable/app-foo-1.2.3.zip”,
	version: “1.2.3”,
	target: “updates/”,
	post: {
		randomValue: ZZZZ
		supportKeyHash: YYYY
		supportId: XX
	},		→ instead of using post we can pack alle the values into the URL (for logging and error tracing)??? 
	md5: “hashofzip” 
}]

All values expect post and target are delivered from the fetched meta files from our release channel.
Where YYYY must be a type of one time key based on the support id XX, and the only internal stored support key (UUID). The support key therefore is not transferred as plain key, only hashed.

Impacts on Meta Data

Our releases Meta Data is delivered by mittagqi.com/supportreleases/
This URL provides an json array with available packages.
The JSON is provided only if a valid supportId and a support key hash and a radom value is given in the POST request. Also the count of current users is posted.
POST to our server:

randomValue: ZZZZ
supportKeyHash: YYYY
supportId: XX
userCount: 123

The resulting JSON contains:

[{
	name: “app-foo”,
	url: “https://mittagqi.com/supportreleases/stable/app-foo-1.2.3.zip”,
	version: “1.2.3”,
	updateableversion: “*”, [or something like “1.2.3” or “>1.2.3” or “=1.2.3” or “>=1.2.3”]
	autoupdateable: true |  false,
	dbbackupneeded: true | false,
	md5: “hashofzip”  → hash this again with the support key for security reasons!
}]

The Admin can then choose a update out of the above array / list in the wizard. If the admin decides to install a update, the updater fetches from the above URL (plus the support key hash as described above in the fetcher config definition).

Our Support / Download Server is defined in TRANSLATE-134!