diff options
Diffstat (limited to 'README')
| -rw-r--r-- | README | 108 |
1 files changed, 108 insertions, 0 deletions
@@ -0,0 +1,108 @@ +Guix Python Package Utilities +============================= + +This library provides utilities for working with Python package indexes, or +PyPIs for short. + +It provides a generic framework, with which the contents of a PyPI can be +used from the Guix package manager. + +For example, the [Guix Python Integration Project][1] uses this library to +provide Guix packages for the contents of [pypi.org][2]. + +[1]: http://git.cbaines.net/guix-python-integration-project +[2]: https://pypi.org/ + +Implementation Details +---------------------- + +### sdist records + +The sdist record acts as a higher level representation of a set of Guix package +records. It is from this set of sdist records that a Guix package can be +generated for a specific version with a specific set of extras of the desired +software project. + +### Guix package generation + +A Guix package is a function of the following: + + - A sdist store + - The name and version of the sdist in the store, from which to create the + package + - A set of extras which can apply to this sdist + - A function, which takes a sdist and package, and can return a modified + version of the package (fix-function) + - A python package (e.g. python-3.4) (python) + +The fix-function should be used to adjust packages in ways which cannot be +represented by the sdist. For example, it can be used to unconditionally +disable the tests for a given sdist, or add a input from outside of the sdist +store. + +None of the inputs to the package returned by sdist->package should be used +directly. This is because they may have disabled test suites, or missing +propagated inputs as a result of breaking dependency loops when constructing +the package. sdist->package should just be called for each desired package. + +### Importer service + +*TODO:* Currently there is no importer service, but this describes how it could +work. + +The importer service keeps the repository up to date with the pypi.org Python +package index. It attempts to create sdist records for a given version of a +software project, and any packages in its dependency tree which do not have a +corresponding sdist record. + +This is quite complex, as the information for setup_requires, and tests_require +is inside the setup.py. They may differ depending on the environment in which +the setup.py is run, making it difficult to extract the metadata in a reliable +way. + +Due to current limitations described above with extracting information from +sdist archives, the importer service uses a number of strategies to create +sdist records, this is optimised for compatibility, while compromising speed. + +The steps are listed below: + + - pypi.org is queried to get the url for the sdist + - The sdist is downloaded and unpacked in a temporary directory + - The egg info metadata is read, which gives the version and install_requires + (including extras) + - An initial sdist record can now be constructed + - From this initial sdist record, a Guix package record is created if there + are inputs that are not present in the sdist store, these will be fetched + first (with this process) + - The package record is first built, first without running the tests + if the build fails, the log is read (through the use of regular expressions) + in an attempt to determine if the build failed due to a missing dependency, + if no match is found, the process stops, but if a missing dependency is + identified, this is added as a build requirement to the sdist, and the step + is attempted again. + - The package record is now built, but with the tests enabled. Like the + previous step, the logs are checked if the build fails, and any missing + requirements identified are added as test requirements to the sdist record. + - To confirm that it is possible to build the packages for all of the + combinations of extras, every possible combination is built. Again, the logs + for any build failure are analysed to determine missing sdist records + (*TODO:* This is disabled, as some projects, e.g. kombu have a large number + of extras). + +### Updater service + +*TODO:* There is currently no updater service. Its unclear how this could +relate to the import service should it exist. + +To reduce the possibility of packages failing to build, the versions of each +package in the dependency tree for an sdist are limited at the point at which +it is imported (as described int he previous section). The updater service is +used to bump when possible the version limits associated with an individual +sdist record. + +This involves raising all the limits to the latest versions for which there are +sdists available, while not violating the limits or requirements for any +propagated package in the dependency tree of the set of propagated packages. + +Inputs that are not propagated (inputs and native-inputs) are still subject to +the version limits, but only those of the corresponding sdist. |