Initial commit

1 files changed, 108 insertions, 0 deletions

diff --git a/README b/README
new file mode 100644
index 0000000..73c46e9
--- /dev/null
+++ b/README
@@ -0,0 +1,108 @@
+Guix Python Package Utilities
+=============================
+
+This library provides utilities for working with Python package indexes, or
+PyPIs for short.
+
+It provides a generic framework, with which the contents of a PyPI can be
+used from the Guix package manager.
+
+For example, the [Guix Python Integration Project][1] uses this library to
+provide Guix packages for the contents of [pypi.org][2].
+
+[1]: http://git.cbaines.net/guix-python-integration-project
+[2]: https://pypi.org/
+
+Implementation Details
+----------------------
+
+### sdist records
+
+The sdist record acts as a higher level representation of a set of Guix package
+records. It is from this set of sdist records that a Guix package can be
+generated for a specific version with a specific set of extras of the desired
+software project.
+
+### Guix package generation
+
+A Guix package is a function of the following:
+
+ - A sdist store
+ - The name and version of the sdist in the store, from which to create the
+   package
+ - A set of extras which can apply to this sdist
+ - A function, which takes a sdist and package, and can return a modified
+   version of the package (fix-function)
+ - A python package (e.g. python-3.4) (python)
+
+The fix-function should be used to adjust packages in ways which cannot be
+represented by the sdist. For example, it can be used to unconditionally
+disable the tests for a given sdist, or add a input from outside of the sdist
+store.
+
+None of the inputs to the package returned by sdist->package should be used
+directly. This is because they may have disabled test suites, or missing
+propagated inputs as a result of breaking dependency loops when constructing
+the package. sdist->package should just be called for each desired package.
+
+### Importer service
+
+*TODO:* Currently there is no importer service, but this describes how it could
+work.
+
+The importer service keeps the repository up to date with the pypi.org Python
+package index. It attempts to create sdist records for a given version of a
+software project, and any packages in its dependency tree which do not have a
+corresponding sdist record.
+
+This is quite complex, as the information for setup_requires, and tests_require
+is inside the setup.py. They may differ depending on the environment in which
+the setup.py is run, making it difficult to extract the metadata in a reliable
+way.
+
+Due to current limitations described above with extracting information from
+sdist archives, the importer service uses a number of strategies to create
+sdist records, this is optimised for compatibility, while compromising speed.
+
+The steps are listed below:
+
+ - pypi.org is queried to get the url for the sdist
+ - The sdist is downloaded and unpacked in a temporary directory
+ - The egg info metadata is read, which gives the version and install_requires
+   (including extras)
+ - An initial sdist record can now be constructed
+ - From this initial sdist record, a Guix package record is created if there
+   are inputs that are not present in the sdist store, these will be fetched
+   first (with this process)
+ - The package record is first built, first without running the tests
+   if the build fails, the log is read (through the use of regular expressions)
+   in an attempt to determine if the build failed due to a missing dependency,
+   if no match is found, the process stops, but if a missing dependency is
+   identified, this is added as a build requirement to the sdist, and the step
+   is attempted again.
+ - The package record is now built, but with the tests enabled. Like the
+   previous step, the logs are checked if the build fails, and any missing
+   requirements identified are added as test requirements to the sdist record.
+ - To confirm that it is possible to build the packages for all of the
+   combinations of extras, every possible combination is built. Again, the logs
+   for any build failure are analysed to determine missing sdist records
+   (*TODO:* This is disabled, as some projects, e.g. kombu have a large number
+   of extras).
+
+### Updater service
+
+*TODO:* There is currently no updater service. Its unclear how this could
+relate to the import service should it exist.
+
+To reduce the possibility of packages failing to build, the versions of each
+package in the dependency tree for an sdist are limited at the point at which
+it is imported (as described int he previous section). The updater service is
+used to bump when possible the version limits associated with an individual
+sdist record.
+
+This involves raising all the limits to the latest versions for which there are
+sdists available, while not violating the limits or requirements for any
+propagated package in the dependency tree of the set of propagated packages.
+
+Inputs that are not propagated (inputs and native-inputs) are still subject to
+the version limits, but only those of the corresponding sdist.