diff options
author | Stephen Finucane <stephen@that.guru> | 2017-03-22 23:17:25 +0000 |
---|---|---|
committer | Stephen Finucane <stephen@that.guru> | 2017-04-18 22:08:32 +0100 |
commit | 1cb38956875fecd42e1eaaf4c0c4064c577c7aab (patch) | |
tree | 564b0bb6903ba41dce8db8b74b19b12b7ff15025 /docs/deployment | |
parent | 5d608aeaa86f2819770098214e92f1dbc1c72d00 (diff) | |
download | patchwork-1cb38956875fecd42e1eaaf4c0c4064c577c7aab.tar patchwork-1cb38956875fecd42e1eaaf4c0c4064c577c7aab.tar.gz |
docs: Remove old Markdown files
Everything is now ported to rST/Sphinx meaning we can remove the
Markdown versions.
Signed-off-by: Stephen Finucane <stephen@that.guru>
Diffstat (limited to 'docs/deployment')
-rw-r--r-- | docs/deployment/installation.md | 489 | ||||
-rw-r--r-- | docs/deployment/upgrading.md | 138 |
2 files changed, 0 insertions, 627 deletions
diff --git a/docs/deployment/installation.md b/docs/deployment/installation.md deleted file mode 100644 index 48bf979..0000000 --- a/docs/deployment/installation.md +++ /dev/null @@ -1,489 +0,0 @@ -# Deployment - -This document describes the necessary steps to configure Patchwork in a -production environment. This requires a significantly "harder" deployment than -the one used for development. If you are interested in developing Patchwork, -please refer to [the development guide][doc-development] instead. - -This document describes a two-node installation of Patchwork, consisting of -a database sever and an application server. It should be possible to combine -these machines for smaller Patchwork instances. It should also be possible to -configure high availability deployment through use of additional database and -application machines, though this is out of the scope of this document. - -## Deployment Guides, Provisioning Tools and Platform-as-a-Service - -Before continuing, it's worth noting that Patchwork is a Django application. -With the exception of the handling of incoming mail (described below), it -can be deployed like any other Django application. This means there are tens, -if not hundreds, of existing articles and blogs detailing how to deploy an -application like this. As such, if any of the below information is unclear -then we'd suggest you go search for "Django deployment guide" or similar, -deploy your application, and submit [a patch for this guide][doc-contributing] -to clear up that confusion for others. - -You'll also find that the same search reveals a significant number of existing -deployment tools aimed at Django. These tools, be they written in Ansible, -Puppet, Chef or something else entirely, can be used to avoid much of the -manual configuration described below. If possible, embrace these tools to make -your life easier. - -Finally, many Platform-as-a-Service (PaaS) providers and tools support -deployment of Django applications with minimal effort. Should you wish to -avoid much of the manual configuration, we suggest you investigate the many -options available to find one that best suits your requirements. The only issue -here will likely be the handling of incoming mail - something which many of -these providers don't support. We address this in the appropriate section -below. - -## Requirements - -For the purpose of this guide, we will assume the following machines: - -| server role | IP address | -|-------------|------------| -| database | 10.1.1.1 | -| application | 10.1.1.2 | - -We will use the database server to, ostensibly enough, host the database for -the Patchwork instance. The application server, on the other hand, will host -the Patchwork instance along with the required reverse proxy and WSGI HTTP -servers. - -We expect a Ubuntu 15.04 installation on each of these hosts: commands, -package names and/or package versions will likely change if using a -different distro or release. In addition, usage of different package versions -to the ones suggested may require slightly different configuration. - -Before beginning, you should update these systems: - - $ sudo apt-get update - $ sudo apt-get upgrade - -We also need to configure some environment variables to ease deployment. These -should be exported on all systems: - -<dl> - <dt>PW_HOST_DB=10.1.1.1</dt> - <dd>IP of the database host</dd> - <dt>PW_HOST_APP=10.1.1.2</dt> - <dd>IP of the application host</dd> - <dt>PW_DB_NAME=patchwork</dt> - <dd>Name of the database</dd> - <dt>PW_DB_USER=www-data</dt> - <dd>Username that the Patchwork app will access the database with</dd> -</dl> - -## Database - -These steps should be run on the database server. - -**NOTE:** If you already have a database server on site, you can skip much of -this section. - -### Install Requirements - -We're going to rely on PostgreSQL. You can adjust the below steps if using a -different RDBMS. Install the required packages. - - $ sudo apt-get install -y postgresql postgresql-contrib - -### Configure Database - -PostgreSQL created a user account called `postgres`; you will need to run -commands as this user. Use this account to create the database that Patchwork -will use, using the credentials we configured earlier. - - $ sudo -u postgres createdb $PW_DB_NAME - $ sudo -u postgres createuser $PW_DB_USER - -We will also need to apply permissions to the tables in this database but -seeing as the tables haven't actually been created yet this will have to be -done later. - -**TODO** `pg_hba.conf` configuration - -## Patchwork - -These steps should be run on the application server. - -### Install Packages - -The first requirement is Patchwork itself. It can be downloaded like so: - - $ wget https://github.com/getpatchwork/patchwork/archive/v1.1.0.tar.gz - -We will install this under `/opt`, though this is only a suggestion: - - $ tar -xvzf v1.1.0.tar.gz - $ sudo mv v1.1.0 /opt/patchwork - -**NOTE:** Per the [Django documentation][ref-django-files], source code should -not be placed in your web server's document root as this risks the possibility -that people may be able to view your code over the Web. This is a security -risk. - -Next we require Python. If not already installed, then you should do so now. -Patchwork supports both Python 2.7 and Python 3.3+, though we would suggest -using the latter to ease future upgrades: - - $ sudo apt-get install python3 # or 'python' if using Python 2.7 - -We require a number of Python packages. These can be installed using `pip`: - - $ sudo pip install -r /opt/patchwork/requirements-prod.txt - -If you're not using `pip`, you will need to identify and install the -corresponding distro package for each of these requirements. For example: - - $ sudo apt-get install python3-django - -**NOTE:** The [pkgs.org][ref-pkgs] website provides a great reference for -identifying the name of these dependencies. - -### Configure Patchwork - -You will also need to configure a [settings][ref-django-settings] file for -Django. A sample settings file is provided that defines default settings for -Patchwork. You'll need to configure settings for your own setup and save this -as `production.py`. - - $ cp patchwork/settings/production.example.py \ - patchwork/settings/production.py - -Alternatively, you can override the `DJANGO_SETTINGS_MODULE` environment -variable and provide a completely custom settings file. - -**NOTE:** You should not include shell variables in settings but rather -hardcoded values. These settings files are evaluated in Python - not a shell. - -### Databases - -You can configure the `DATABASES` setting using the variables we set earlier. - - DATABASES = { - 'default': { - 'ENGINE': 'django.db.backends.postgresql_psycopg2', - 'HOST': '$PW_HOST_DB', # don't use sh variables but actual values - 'PORT': '', - 'NAME': '$PW_DB_NAME', - 'USER': '$PW_DB_USER', - 'PASSWORD': '$PW_DB_PASS', - 'TEST': { - 'CHARSET': 'utf8', - }, - }, - } - -**NOTE:** `TEST/CHARSET` is used when creating tables for the test suite. -Without it, tests checking for the correct handling of non-ASCII characters -fail. It is not necessary if you don't plan to run tests, however. - -#### Static Files - -While we have not yet configured our proxy server, we do need to configure -the location that these files will be stored in. We will install these under -`/var/www/patchwork`, though this is only a suggestion and can be changed. - - $ mkdir /var/www/patchwork - -You can configure this by setting the `STATIC_ROOT` variable. - - STATIC_ROOT = '/var/www/patchwork' - -#### Other Options - -Finally, the following settings need to be configured. The purpose of these -variables is described in the [Django documentation][ref-django-settings]: - -* `SECRET_KEY` -* `ADMINS` -* `TIME_ZONE` -* `LANGUAGE_CODE` -* `DEFAULT_FROM_EMAIL` -* `NOTIFICATION_FROM_EMAIL` - -You can generate the `SECRET_KEY` with the following python code: - - import string, random - chars = string.letters + string.digits + string.punctuation - print repr("".join([random.choice(chars) for i in range(0,50)])) - -If you wish to enable the XML-RPC interface, you should add the following to -the file: - - ENABLE_XMLRPC = True - -### Final Steps - -Once done, we should be able to check that all requirements are met using -the `check` command of the `manage.py` executable: - - $ /opt/patchwork/manage.py check - -We should also take this opportunity to both configure the database and static -files: - - $ /opt/patchwork/manage.py migrate - $ /opt/patchwork/manage.py loaddata \ - /opt/patchwork/patchwork/fixtures/default_tags.xml - $ /opt/patchwork/manage.py loaddata \ - /opt/patchwork/patchwork/fixtures/default_states.xml - $ /opt/patchwork/manage.py collectstatic - -**NOTE:** The above `default_tags` and `default_states` are just that: -defaults. You can modify these to fit your own requirements. - -Finally, it may be helpful to start the development server quickly to ensure -you can see *something*: - - $ /opt/patchwork/manage.py runserver 0.0.0.0:8080 - -Browse this instance at `http://[your_server_ip]:8000`. If everything is -working, kill the development server using `Ctrl`+`C`. - -## Reverse Proxy and WSGI HTTP Servers - -These steps should be run on the application server. - -### Install Packages - -We will use nginx and uWSGI to deploy Patchwork, acting as reverse proxy server -and WSGI HTTP server respectively. Other options are available, such as -Apache+mod_wsgi or nginx+Gunicorn. While we don't document these, sample -configuration files for the former case are provided in `lib/apache2/`. - - $ sudo apt-get install nginx-full uwsgi uwsgi-plugin-python - -### Configure nginx and uWSGI - -Configuration files for nginx and uWSGI are provided in the `lib` subdirectory -of the Patchwork source code. These can be modified as necessary, but for now -we will simply copy them. - -First, let's load the provided configuration for nginx: - - $ sudo cp /opt/patchwork/lib/nginx/patchwork.conf \ - /etc/nginx/sites-available/ - -If you wish to modify this configuration, now is the time to do so. Once done, -validate and enable your configuration: - - $ sudo nginx -t - $ sudo ln -s /etc/nginx/sites-available/patchwork.conf \ - /etc/nginx/sites-enabled/patchwork.conf - -Now use the provided configuration for uWSGI: - - $ sudo mkdir -p /etc/uwsgi/sites - $ sudo cp /opt/patchwork/lib/uwsgi/patchwork.ini \ - /etc/uwsgi/sites/patchwork.ini - -**NOTE** We created the `/etc/uwsgi` directory above because we're going to run -uWSGI in ["emperor mode][ref-uwsgi-emperor]". This has benefits for multi-app -deployments. - -### Create systemd Unit File - -As things stand, uWSGI will need to be started manually every time the system -boots, in addition to any time it may fail. We can automate this process using -systemd. To this end a [systemd unit file][ref-uwsgi-systemd] should be created -to start uWSGI at boot: - - $ sudo cat << EOF > /etc/systemd/system/uwsgi.service - [Unit] - Description=uWSGI Emperor service - - [Service] - ExecStartPre=/usr/bin/bash -c 'mkdir -p /run/uwsgi; chown user:nginx /run/uwsgi' - ExecStart=/usr/bin/uwsgi --emperor /etc/uwsgi/sites - Restart=always - KillSignal=SIGQUIT - Type=notify - NotifyAccess=all - - [Install] - WantedBy=multi-user.target - EOF - -**NOTE:** On older version of Ubuntu you may need to tweak these steps to use -[upstart][ref-uwsgi-upstart] instead. - -### Final Steps - -Start the uWSGI service we created above: - - $ sudo systemctl start uwsgi - $ sudo systemctl status uwsgi - -Next up, restart the nginx service: - - $ sudo systemctl restart nginx - $ sudo systemctl status nginx - -Patchwork uses a cron script to clean up expired registrations and send -notifications of patch changes (for projects with this enabled). Something like -this in your crontab should work. - - # m h dom mon dow command - */10 * * * * cd patchwork; ./manage.py cron - -**NOTE**: The frequency should be the same as the `NOTIFICATION_DELAY_MINUTES` -setting, which defaults to 10 minutes. - -Finally, browse to the instance using your browser of choice. - -You may wish to take this opportunity to setup your projects and configure your -website address (in the Sites section of the admin console, found at `/admin`). - -## Django administrative console - -In order to access the administrative console at `/admin`, you need at least -one user account to be registered and configured as a super user or staff -account to access the Django administrative console. This can be achieved by -doing the following: - - $ /opt/patchwork/manage.py createsuperuser - -Once the administrative console is accessible, you would want to configure your -different sites and their corresponding domain names, which is required for the -different emails sent by patchwork (registration, password recovery) as well as -the sample `pwclientrc` files provided by your project's page. - -## Incoming Email - -Patchwork is designed to parse incoming mails which means you need an address -to receive email at. This is a problem that has been solved for many webapps, -thus there are many ways to go about this. Some of these ways are discussed -below. - -### IMAP/POP3 - -The easiest option for getting mail into Patchwork is to use an existing email -address in combination with a mail retriever like [getmail][ref-getmail], which -will download mails from your inbox and pass them to Patchwork for processing. -getmail is easy to set up and configure: to begin, you need to install it: - - $ sudo apt-get install getmail4 - -Once installed, you should configure it, sustituting your own configuration -details where required below: - - $ sudo cat << EOF > /etc/getmail/user@example.com/getmailrc - [retriever] - type = SimpleIMAPSSLRetriever - server = imap.example.com - port = 993 - username = XXX - password = XXX - mailboxes = ALL - - [destination] - # we configure Patchwork as a "mail delivery agent", in that it will - # handle our mails - type = MDA_external - path = /opt/patchwork/patchwork/bin/parsemail.sh - - [options] - # retrieve only new emails - read_all = false - # do not add a Delivered-To: header field - delivered_to = false - # do not add a Received: header field - received = false - EOF - -Validate that this works as expected by starting `getmail`: - - $ getmail --getmaildir=/etc/getmail/user@example.com --idle INBOX - -If everything works as expected, you can create a systemd script to ensure this -starts on boot: - - $ sudo cat << EOF > /etc/systemd/system/getmail.service - [Unit] - Description=Getmail for user@example.com - - [Service] - User=pathwork - ExecStart=/usr/bin/getmail --getmaildir=/etc/getmail/user@example.com --idle INBOX - Restart=always - - [Install] - WantedBy=multi-user.target - EOF - -And start the service: - - $ sudo systemctl start getmail - $ sudo systemctl status getmail - -### Mail Transfer Agent (MTA) - -The most flexible option is to configure our own mail transfer agent (MTA) or -"email server". There are many options, of which [Postfix][ref-postfix] is one. -While we don't cover setting up Postfix here (it's complicated and there are -many guides already available), Patchwork does include a script to take -received mails and create the relevant entries in Patchwork for you. To use -this, you should configure your system to forward all emails to a given -localpart (the bit before the `@`) to this script. Using the `patchwork` -localpart (e.g. `patchwork@example.com`) you can do this like so: - - $ sudo cat << EOF > /etc/aliases - patchwork: "|/opt/patchwork/patchwork/bin/parsemail.sh" - EOF - -You should ensure the appropriate user is created in PostgreSQL and that -it has (minimal) access to the database. Patchwork provides scripts for the -latter and they can be loaded as seen below: - - $ sudo -u postgres createuser nobody - $ sudo -u postgre psql -f \ - /opt/patchwork/lib/sql/grant-all.postgres.sql patchwork - -**NOTE:** This assumes your Postfix process is running as the `nobody` user. -If this is not correct (use of `postfix` user is also common), you should -change both the username in the `createuser` command above and substitute the -username in the `grant-all-postgres.sql` script with the appropriate -alternative. - -### Use a Email-as-a-Service Provider - -Setting up an email server can be a difficult task and, in the case of -deployment on PaaS provider, may not even be an option. In this case, there -are a variety of web services available that offer "Email-as-as-Service". -These services typically convert received emails into HTTP POST requests to -your endpoint of choice, allowing you to sidestep configuration issues. We -don't cover this here, but a simple wrapper script coupled with one of these -services can be more than to get email into Patchwork. - -You can also create such as service yourself using a PaaS provider that -supports incoming mail and writing a little web app. - -## (Optional) Configure your VCS to Automatically Update Patches - -The `tools` directory of the Patchwork distribution contains a file named -`post-receive.hook` which is a sample git hook that can be used to -automatically update patches to the `Accepted` state when corresponding -commits are pushed via git. - -To install this hook, simply copy it to the `.git/hooks` directory on your -server, name it `post-receive`, and make it executable. - -This sample hook has support to update patches to different states depending -on which branch is being pushed to. See the `STATE_MAP` setting in that file. - -If you are using a system other than git, you can likely write a similar hook -using `pwclient` to update patch state. If you do write one, please contribute -it. - -[doc-contributing]: ../development/contributing.md -[doc-development]: ../development/installation.md -[ref-django-files]: https://docs.djangoproject.com/en/dev/intro/tutorial01/#creating-a-project -[ref-django-settings]: https://docs.djangoproject.com/en/1.8/ref/settings/ -[ref-getmail]: http://pyropus.ca/software/getmail/ -[ref-pkgs]: http://pkgs.org/ -[ref-postfix]: http://www.postfix.org/ -[ref-uwsgi-emperor]: https://uwsgi-docs.readthedocs.io/en/latest/Emperor.html -[ref-uwsgi-systemd]: https://uwsgi-docs.readthedocs.io/en/latest/Systemd.html -[ref-uwsgi-upstart]: https://uwsgi-docs.readthedocs.io/en/latest/Upstart.html diff --git a/docs/deployment/upgrading.md b/docs/deployment/upgrading.md deleted file mode 100644 index 014ec51..0000000 --- a/docs/deployment/upgrading.md +++ /dev/null @@ -1,138 +0,0 @@ -# Upgrading - -**NOTE:** This document provides some general tips and tricks that one can use -when upgrading an existing, production installation of Patchwork. If you are -interested in the specific changes between each release, please refer to the -[`UPGRADING` document][gh-upgrading] instead. If this is your first time -installing Patchwork, please refer to the -[installation guide][doc-installation] instead. - -## Before You Start - -Before doing anything, always **backup your data**. This generally means -backing up your database, but it might also be a good idea to backup your -environment in case you encounter issues during the upgrade process. - -While Patchwork won't explicitly prevent it, it's generally wise to avoid -upgrades spanning multiple releases in one go. An iterative upgrade approach -will provide an easier, if slower, upgrade process. - -## Identify Changed Scripts, Requirements, etc. - -The `UPGRADING` document provides a comprehensive listing of all -backwards-incompatible changes that occur between releases of Patchwork. -Examples of such changes include: - -* Moved/removed scripts and files -* Changes to the requirements, e.g. supported Django versions -* Changes to API that may affect, for example, third-party tools - -It is important that you understand these changes and ensure any scripts you -may have, such as systemd/upstart scripts, are modified accordingly. - -## Understand What Requirements Have Changed - -New versions of Patchwork can often require additional or updated version of -dependencies, e.g. newer versions of Django. It is important that you -understand these requirements and can fulfil them. This is particularly true -for users relying on distro-provided packages, who may have to deal with older -versions of a package or may be missing a package altogether (though we try to -avoid this). Such changes are usually listed in the `UPGRADING` document, but -you can also diff the `requirements.txt` files in each release for comparison. - -## Collect Static Files - -New versions of Patchwork generally contain changes to the additional files -like images, CSS and JavaScript. To do this, run the `collectstatic` -management commands: - - $ ./manage.py collectstatic - -## Upgrade Your Database - -Migrations of the database can be tricky. Prior to [`v1.0.0`][gh-v1], database -migrations were provided by way of manual, SQL migration scripts. After this -release, Patchwork moved to support [Django migrations][ref-django-migrate]. -If you are upgrading from `v1.0.0` or later, it is likely that you can rely -entirely on the later to bring your database up-to-date. This can be done like -so: - - $ ./manage.py migrate - -However, there are a number of scenarios in which you may need to fall back to -the provided SQL migrations or provide your own: - -* You are using Django < 1.6 - - Patchwork supports Django 1.6. However, Django Migrations was added in 1.7 - and is [not available for previous versions][ref-south2]. As such, you must - continue to use manual migrations or upgrade your version of Django. For - many of the migrations, this can be done automatically: - - $ ./manage.py sqlmigrate patchwork 0003_add_check_model - - However, this only works for schema migrations. For data migrations, - however, this will fail. In this cases, these migrations will need to be - handwritten. - -* You are using Django > 1.6, but upgrading from Patchwork < 1.0.0 - - Patchwork only started providing migrations in `v1.0.0`. SQL migrations are - provided for versions prior to this and must be applied to get the database - to the "initial" state that Django migrations expects. - -* You have diverged from upstream Patchwork - - If you have applied custom patches that change the database models, the - database in an "inconsistent state" and the provided migrations will likely - fail to apply. - -Steps to handle the latter two of these are described below. - -### Upgrading a pre-v1.0.0 Patchwork instance - -The process for this type of upgrade is quite simple: upgrade using manual -SQL upgrades until better options become available. As such, you should apply -all unapplied SQL migrations that are not duplicated by Django migrations. -Once such duplication occurs, rely on the Django migrations only and continue -to do so going forward. - -### Upgrading a "diverged" Patchwork instance - -This type of upgrade is a little trickier. There are two options you can take: - -1. Bring your Patchwork instance back in sync with upstream -2. Provide your own migrations - -The former option is particularly suitable if you decide to upstream your -change or decide it's not valuable enough to retain. This will require either -reworking any migrations that exist prior to your feature being upstreamed, or -deleting any added database fields and tables, respectively. In both cases, -manually, hand-written SQL migrations will be required to get the databse into -a consistent state (remember: **backup**!). Once this is done, you can resume -using the upstream-provided migrations, ensuring any Django migrations that you -may have skipped are not applied again: - - $ ./manage.py migrate 000x-abc --fake # when 000x-abc is last "skippable" - -It's worth adding that with the databases now back in sync it should be -possible to return to using upstream code rather than maintaining a fork. - -The latter option is best chosen if you wish to retain the aforementioned fork. -How you do this depends on the extensiveness of your changes, but getting the -latest version of Patchwork, deleting the provided migrations, applying any -patches you may have and regenerating the migrations seems like the best -option. - -**NOTE**: To prevent the latter case above from occurring, we'd ask that you -submit any patches you may have to the upstream Patchwork so that the wider -community can benefit from this new functionality. Please see -[the contributing document][doc-contributing] for more information on this -process. - -[doc-installation]: installation.md -[doc-contributing]: ../development/contributing.md -[gh-upgrading]: https://github.com/getpatchwork/patchwork/blob/master/UPGRADING.md -[gh-v1]: https://github.com/getpatchwork/patchwork/releases/tag/v1.0.0 -[ref-django-migrate]: https://docs.djangoproject.com/en/1.8/topics/migrations/ -[ref-south2]: http://blog.allenap.me/2015/05/south-south-2-and-django-migrations.html |