Guix Python Package Utilities ============================= This library provides utilities for working with Python package indexes, or PyPIs for short. It provides a generic framework, with which the contents of a PyPI can be used from the Guix package manager. For example, the [Guix Python Integration Project][1] uses this library to provide Guix packages for the contents of [pypi.org][2]. [1]: http://git.cbaines.net/guix-python-integration-project [2]: https://pypi.org/ Implementation Details ---------------------- ### sdist records The sdist record acts as a higher level representation of a set of Guix package records. It is from this set of sdist records that a Guix package can be generated for a specific version with a specific set of extras of the desired software project. ### Guix package generation The [packaging][3] library is used to handle requirements (what Guix would consider an input). The dependency of every package in the build graph is decided by the [libsolv][4] dependency solver. [3]: https://github.com/pypa/packaging [4]: https://github.com/openSUSE/libsolv A Guix package is a function of the following: - A sdist store - The name and version of the sdist in the store, from which to create the package - A set of extras which can apply to this sdist - A function, which takes a sdist and package, and can return a modified version of the package (fix-function) - A python package (e.g. python-3.4) (python) The fix-function should be used to adjust packages in ways which cannot be represented by the sdist. For example, it can be used to unconditionally disable the tests for a given sdist, or add a input from outside of the sdist store. None of the inputs to the package returned by sdist->package should be used directly. This is because they may have disabled test suites, or missing propagated inputs as a result of breaking dependency loops when constructing the package. sdist->package should just be called for each desired package. ### Importer service *TODO:* Currently there is no importer service, but this describes how it could work. The importer service keeps the repository up to date with the pypi.org Python package index. It attempts to create sdist records for a given version of a software project, and any packages in its dependency tree which do not have a corresponding sdist record. This is quite complex, as the information for setup_requires, and tests_require is inside the setup.py. They may differ depending on the environment in which the setup.py is run, making it difficult to extract the metadata in a reliable way. Due to current limitations described above with extracting information from sdist archives, the importer service uses a number of strategies to create sdist records, this is optimised for compatibility, while compromising speed. The steps are listed below: - pypi.org is queried to get the url for the sdist - The sdist is downloaded and unpacked in a temporary directory - The egg info metadata is read, which gives the version and install_requires (including extras) - An initial sdist record can now be constructed - From this initial sdist record, a Guix package record is created if there are inputs that are not present in the sdist store, these will be fetched first (with this process) - The package record is first built, first without running the tests if the build fails, the log is read (through the use of regular expressions) in an attempt to determine if the build failed due to a missing dependency, if no match is found, the process stops, but if a missing dependency is identified, this is added as a build requirement to the sdist, and the step is attempted again. - The package record is now built, but with the tests enabled. Like the previous step, the logs are checked if the build fails, and any missing requirements identified are added as test requirements to the sdist record. - To confirm that it is possible to build the packages for all of the combinations of extras, every possible combination is built. Again, the logs for any build failure are analysed to determine missing sdist records (*TODO:* This is disabled, as some projects, e.g. kombu have a large number of extras). ### Updater service *TODO:* There is currently no updater service. Its unclear how this could relate to the import service should it exist. To reduce the possibility of packages failing to build, the versions of each package in the dependency tree for an sdist are limited at the point at which it is imported (as described int he previous section). The updater service is used to bump when possible the version limits associated with an individual sdist record. This involves raising all the limits to the latest versions for which there are sdists available, while not violating the limits or requirements for any propagated package in the dependency tree of the set of propagated packages. Inputs that are not propagated (inputs and native-inputs) are still subject to the version limits, but only those of the corresponding sdist.