Running LTD Mason on Travis

If your project already uses Travis CI for continuous integration, you may want to run LTD Mason there for documentation deployment as well. Projects well suited for Travis-based Mason deployments are:

Documentation for EUPS-based packages need to be run from a Jenkins-based CI environment. See Running LTD Mason on Jenkins.

Composing .travis.yml for LSST the Docs

Projects need to include a .travis.yml file in their root to activate Travis CI. You can read more about .travis.yml from the Travis documenation.

The exact construction of this file depends on how you need to use Travis to run tests for your project, or if you are only building documentation. This guide will outline how to deploy documentation in both cases.

When crafting .travis.yml, we recommend using Travis’ validator, available online or through the travis command line app.

Environment variables and secrets

Regardless of context, all projects using Travis to deploy to LSST the Docs need to configure the Travis CI build environment through environment variables. This is a sample env section:

env:
  global:
    - LTD_MASON_BUILD=false  # disable builds in regular text matrix
    - LTD_MASON_PRODUCT="sims-operations"
    - LTD_KEEPER_URL="https://keeper.lsst.codes"
    # travis encrypt "LTD_MASON_AWS_ID=... LTD_MASON_AWS_SECRET=..."
    - secure: "K332NczWP7D7qczaqXmcFq1nNT6/PeZ14rtRK0sFDo7/7u6KwLvhe5biXOgmKJmoOFlXoBg93uRn3tewRKvVpffPzrWwOlOHMNPR90X6uOubFxnd/IXFSWXOtEoKguKv71kTJANJvjhJkEtKm0cjrfAIBo8hMiiOqR9782oB/FQ="
    # travis encrypt "LTD_KEEPER_USER=... LTD_KEEPER_PASSWORD=..."
    - secure: "B3bdOBOnrhb+YFaFKG0LDZ5ETGb8MnH3VdmkmrM9BNmcO9X8q39gqI9cXVaOFrqLHCBkAtLZus5pGeEetxG/87gAE04JDPWjSMCkMyszUMn8fpv3WgRGxmhKxNAlHHVFOiRJLaE2j58YnQtcXYLyIeGWe3QH6Zk+LHHaOJqtqeU="

By including these environment variables under env.global we ensure that the entire set is applied to all builds; including environment variables directly in env creates a build matrix.

LTD_MASON_BUILD=false should always be present to ensure that LTD Mason is only run on the build you specify, rather than all entries of a Travis build matrix.

LTD_MASON_PRODUCT should be set to the project’s slug (specified when the project was created on LTD Keeper).

LTD_KEEPER_URL is the URL of the Keeper API server. For LSST the Docs this is https://keeper.lsst.codes.

The next two lines contain encrypted environment variable settings for LTD_KEEPER_USER/LTD_KEEPER_PASSWORD (credentials for Travis to work with LTD Keeper) and LTD_MASON_AWS_ID/LTD_MASON_AWS_SECRET (credentials for Travis to upload to AWS S3).

Encrypting variables requires the Travis gem, which can be installed with:

gem install travis

From the root of your Git repository, encrypt the environment variables using

travis encrypt "LTD_MASON_AWS_ID=... LTD_MASON_AWS_SECRET=..."
travis encrypt "LTD_KEEPER_USER=... LTD_KEEPER_PASSWORD=..."

Paste the results as list items into the env.global section of .travis.yml.

Alternatively you could pass the --add env.global flag to travis encrypt, but note that your entire .travis.yml file will be reformatted.

These settings can be securely checked into GitHub since only Travis possesses the private key needed to decrypt those secure lines.

You may need to ask an LSST the Docs administrator to add these encrypted lines since the Keeper and AWS credentials are not widely distributed.

Travis setup for a pure Sphinx repository

The sample travis.yml file below shows how Travis can be configured to build a pure-documentation repository (such as a Technote).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
sudo: false
language: python
matrix:
  include:
    # This is the ltd-mason documentation deployment build
    - python: "3.5"
      env: LTD_MASON_BUILD=true
install:
  - pip install -r requirements.txt
  - pip install ltd-mason
script:
  - sphinx-build -b html -a -n -W -d _build/doctree . _build/html
after_success:
  - ltd-mason-travis --html-dir _build/html
env:
  global:
    - LTD_MASON_BUILD=false  # disable builds in regular text matrix
    - LTD_MASON_PRODUCT="dmtn-013"
    # travis encrypt "LTD_MASON_AWS_ID=... LTD_MASON_AWS_SECRET=... LTD_KEEPER_URL=... LTD_KEEPER_USER=... LTD_KEEPER_PASSWORD=..." --add env.global
    - secure: "edPD6RMPjPCvuuqtXc3nmQ2T5tvVWWFnhC957tmqyzvFS++cEZLXRCRRenlxd+9ygm7qONWXtcvJeyWiaVi9pooeFpIKlcVWeJQAZD+oLsONrFBzdPQrP8ObaeD845T9meFR+k48uFpvB2yHc3e0ZUczhxJbKUxSQzlX1xRIVh9YwwWCLbBitNHcTNsnNDfdvxFw5w3CTkWdd8j6962eOzOQHgO/Ta5B71Ab/0XwVWEh7C5mE3jNq4GRLlEXp9vyDr0DzHihVN/XkMMeSTrUj20pHePvxJh0lJi6zmseX5RCKu1hVe7dvv2FUySVB/BzW6O4kFgS7L3ynaF5jGhiIjkGMwIgd+o1ucDbuPtFqk6e1SBGwc2y2ZVijN/D2ZeQOWGDx2lUrVbdmP3MW5YQDuHfep/8H0npmvd/pUlnIjT/cxSMVW0rQlH0O+ZyTbR30lX1JRCzHgjDqX78m1JHaaOdAjcJ+3GfLRr7vWaCr4mb9NbKMCgtkt6efN6E/cfTB8xbFC0x/TF2QcKIWGQKWxCnGuXgqXlOsoR367kgqWps5l5jGmndth24/sK0UWBlqE9Vkhi3ea58Uprh6RjlMvG/6syAhhsDR1u+T48T3ABpKJHhkrbRncxpiWyki3s9t8z85MLD2dBqPF8RanaeI/86ecoNln1imcWRViCwyFQ="

In this Travis configuration, the Sphinx documentation repository’s dependencies specified in a requirements.txt are installed, along with ltd-mason.

In the script stage we build the Sphinx project with sphinx-build. The flags and options are as follows:

  • -b html to build HTML.
  • -a to force a complete build. This should be an issue on Travis, but including the flag does protect against cases where the HTML build output was accidentally checked into the Git repository.
  • -n -W activates ‘nitpicky mode’ where broken references become warnings, and warnings are elevated to errors that cause sphinx-build to exit 1 and fail the build.
  • . is the root directory of the documentation relative to the root of the Git repository; here the Sphinx project resides at the root of the Git repository.
  • _build/html specifies an output directory for the built HTML.

Note: Unlike the ltd-mason build tool, ltd-mason-travis defers the Sphinx build step to the Travis environment (Travis excels at executing and logging scripts already).

If the Sphinx build was successful, the after_success stage is executed. Here we run ltd-mason-travis to deploy the built documentation. The HTML output directory from sphinx-build must match the --html-dir argument for ltd-mason-travis.

Note that we use matrix.include to include a single Travis build. This technique is for forwards compatibility if continuous integration testing against multiple environments is later added to a project—such as the next example.

Travis setup for LSST the Docs deployment alongside Python CI

In many cases you will want to use Travis for continuous integration of a software package, in addition to documentation deployment. The following travis.yml sample shows how this can be achieved.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
sudo: false
language: python
python:
  - '2.7'
  - '3.4'
  - '3.5'
  - '3.5-dev'
matrix:
  allow_failures:
    - python: "3.5-dev"
  include:
    # This is the ltd-mason documentation deployment build
    - python: "3.5"
      env: LTD_MASON_BUILD=true
install:
  - pip install -r requirements.txt
  - pip install ltd-mason
  - pip install -e .
script:
  - py.test --flake8 --cov=ltdmason
  - sphinx-build -b html -a -n -W -d docs/_build/doctree docs docs/_build/html
after_success:
  - ltd-mason-travis --html-dir docs/_build/html
env:
  global:
    - LTD_MASON_BUILD=false  # disable builds in regular text matrix
    - LTD_MASON_PRODUCT="ltd-mason"
    # travis encrypt "LTD_MASON_AWS_ID=... LTD_MASON_AWS_SECRET=... LTD_KEEPER_URL=... LTD_KEEPER_USER=... LTD_KEEPER_PASSWORD=..." --add env.global
    - secure: "CIpaoNzWwEQngjmj0/OQBRUOnkT9Rq8273N5ZgXmZTtVSliukfJMROQnp9m42x3a2XFamaYV60mmuAvMRNU8VHi4nePxF2vp7utVnp8cF4zFQQzL6KnN2rqWv0H3Snqc1sfMT2n4H9qgBlYG7w5Cv52VIXdwh8MqGSxl8HAiYgqcVNJ+q1Rxeb1Yk+Bv3VW6O0/K4AlrhGY2Gl/zbwgM4ph0K0UvT1IZg8ZjCdddOpgwxPq66kvzHNcpCR6JUnvy5vRVH+IgC83Ar+oJqOA/4pizcFccriLF7nANkVJMrRSL8B1h2IHuuGYpC2VzDPMlAuEPmU6t8QAhVCOq9BSy98902TgKkvt4enPcxS2iNqMoOJSNUW7q9yqvVacz4JApJfHWlq5K7uTy00p4XHV4TUs+9NEgBUCwEFE5CXcRQvg+Y2y1wqUUkH+12nb1Nv4CdGxG6k7yG+eM+qmANJ87jZK9vX0RmDLKXuA3gpJyVomrAKX1+MqqwD0Qu885AUsHCQevO+oDmXv6nKLK/x2ZeyHQrgWISj3LXU6B7LarLrqsrE7JWTwgo/iX6xiVHS422tj94/+rab3JarBWe+ntdG9rZBdILU92kLqzgMA570ryVxtsnu8GnzOB0/3yvdtW+duAgrrBUusBcg9E/Kz/68Cm5RbMLyjaeA6HxP6mfM4="

Here, the python section defines a matrix of builds against several versions of Python. During each build, the lines specified in the install section are executed, followed by the script section.

The install stage first installs the Python project’s development dependencies specified in requirements.txt, followed by the ltd-mason package and the Python package itself.

In the script stage we run tests (py.test, in this example) and then sphinx-build. Again, we have configured Sphinx to fail the build on any warnings so you effectively get CI of the documentation in addition to code.

Finally, in the after_success stage we trigger ltd-mason-travis to deploy the previously-built HTML resources. For Travis projects that CI code we want ltd-mason-travis to run in the after_success stage to ensure that documentation is only deployed for commits that don’t have issues.

Notice that each environment in the build matrix (Python 2.7, 3.4, and so on) executes ltd-mason-travis. However, we only want a single documentation deployment from each code push. We solve this by setting the default value of LTD_MASON_BUILD to false, which causes ltd-mason-build to skip deployment. It’s only in the extra build environment specified in matrix.include that we change LTD_MASON_BUILD to true. This technique allows you to create a build matrix for CI without worrying about multiple accidental documentation deployments.