Guix Python Package Utilities
=============================

This library provides utilities for working with Python package indexes, or
PyPIs for short.

It provides a generic framework, with which the contents of a PyPI can be
used from the Guix package manager.

For example, the [Guix Python Integration Project][1] uses this library to
provide Guix packages for the contents of [pypi.org][2].

[1]: http://git.cbaines.net/guix-python-integration-project
[2]: https://pypi.org/

Implementation Details
----------------------

### sdist records

The sdist record acts as a higher level representation of a set of Guix package
records. It is from this set of sdist records that a Guix package can be
generated for a specific version with a specific set of extras of the desired
software project.

### Guix package generation

The [packaging][3] library is used to handle requirements (what Guix would
consider an input). The dependency of every package in the build graph is
decided by the [libsolv][4] dependency solver.

[3]: https://github.com/pypa/packaging
[4]: https://github.com/openSUSE/libsolv

A Guix package is a function of the following:

 - A sdist store
 - The name and version of the sdist in the store, from which to create the
   package
 - A set of extras which can apply to this sdist
 - A function, which takes a sdist and package, and can return a modified
   version of the package (fix-function)
 - A python package (e.g. python-3.4) (python)

The fix-function should be used to adjust packages in ways which cannot be
represented by the sdist. For example, it can be used to unconditionally
disable the tests for a given sdist, or add a input from outside of the sdist
store.

None of the inputs to the package returned by sdist->package should be used
directly. This is because they may have disabled test suites, or missing
propagated inputs as a result of breaking dependency loops when constructing
the package. sdist->package should just be called for each desired package.

### Importer service

*TODO:* Currently there is no importer service, but this describes how it could
work.

The importer service keeps the repository up to date with the pypi.org Python
package index. It attempts to create sdist records for a given version of a
software project, and any packages in its dependency tree which do not have a
corresponding sdist record.

This is quite complex, as the information for setup_requires, and tests_require
is inside the setup.py. They may differ depending on the environment in which
the setup.py is run, making it difficult to extract the metadata in a reliable
way.

Due to current limitations described above with extracting information from
sdist archives, the importer service uses a number of strategies to create
sdist records, this is optimised for compatibility, while compromising speed.

The steps are listed below:

 - pypi.org is queried to get the url for the sdist
 - The sdist is downloaded and unpacked in a temporary directory
 - The egg info metadata is read, which gives the version and install_requires
   (including extras)
 - An initial sdist record can now be constructed
 - From this initial sdist record, a Guix package record is created if there
   are inputs that are not present in the sdist store, these will be fetched
   first (with this process)
 - The package record is first built, first without running the tests
   if the build fails, the log is read (through the use of regular expressions)
   in an attempt to determine if the build failed due to a missing dependency,
   if no match is found, the process stops, but if a missing dependency is
   identified, this is added as a build requirement to the sdist, and the step
   is attempted again.
 - The package record is now built, but with the tests enabled. Like the
   previous step, the logs are checked if the build fails, and any missing
   requirements identified are added as test requirements to the sdist record.
 - To confirm that it is possible to build the packages for all of the
   combinations of extras, every possible combination is built. Again, the logs
   for any build failure are analysed to determine missing sdist records
   (*TODO:* This is disabled, as some projects, e.g. kombu have a large number
   of extras).

### Updater service

*TODO:* There is currently no updater service. Its unclear how this could
relate to the import service should it exist.

To reduce the possibility of packages failing to build, the versions of each
package in the dependency tree for an sdist are limited at the point at which
it is imported (as described int he previous section). The updater service is
used to bump when possible the version limits associated with an individual
sdist record.

This involves raising all the limits to the latest versions for which there are
sdists available, while not violating the limits or requirements for any
propagated package in the dependency tree of the set of propagated packages.

Inputs that are not propagated (inputs and native-inputs) are still subject to
the version limits, but only those of the corresponding sdist.