Disecting the Bolt File

The boltfile.py is the main script used by Bolt to execute the tasks which with it is invoked. The file contains the task definitions, as well as, the configuration parameters for the tasks. If you haven’t done so yet, review the Getting Started guide to familiarize your-self with a very basic example of a boltfile.py. In this section, we will look at more advanced examples to learn the different features of Bolt.

In essence, a Bolt File is just a Python script. Within it, you can use the Bolt API to define and configure the tasks you want to execute. You can name the script whatever you want and place it in any location, but Bolt, by default, will look in the current working directory for a file named boltfile.py and use it if no other file is specified. This is the recommended way to work with Bolt.

# Assumes boltfile.py in the current working directory.
bolt task-to-execute

# Uses specified file.
bolt task-to-execute --bolt-file myboltfile.py

Tip

You can run bolt --help to see Bolt’s usage and supported arguments.

The Structure of a Bolt File

There are three distinct sections in a boltfile.py. The first section, like in any other Python module is the imports, and your can bring in any module you want.

The second part is the registration of tasks. This involves registering custom task modules you want to use, as well as, defining custom tasks to create more complex execution workflows.

The third section of the boltfile.py is the configuration. Every Bolt File must declare a config variable set to a dictionary where the configuration parameters are defined. The dictionary can be empty, but it is required to define the variable. Of course, an empty dictionary will not help us to do much.

The following shows the contents of the boltfile.py that we created in the Getting Started guide and illustrates the three sections.

# Imports section
import bolt

# Task registration section
bolt.register_task('run-tests', ['pip', 'delete-pyc', 'nose'])

# Configuration section
config = {
    'pip': {
        'command': 'install',
        'options': {
            'r': 'requirements.txt'
        }
    },
    'delete-pyc': {
        'sourcedir': './source',
        'recursive': True
    },
    'nose': {
        'directory': 'tests'
    }
}

Tip

It doesn’t make any difference if you include your configuration before the registration of tasks. As a matter of fact, I started doing it that way myself because of the experience I had with Grunt. But my experience has been that it makes the boltfile.py more readable if you register your tasks right after the imports, and users can see the tasks available immediately after opening the file. Overtime, your configuration will grow with the usage of Bolt and users will have to scroll all the way down to find the available tasks, which is, usually, the most important part of the file.

The Import Section

The import section of the boltfile.py is no different than the imports in any other python script or module. You will always need to import bolt to gain access to its API, which is used, among other things, to register tasks. You will import and register other modules containing custom tasks. Finally, you can import any other libraries you might need.

The Task Registration Section

There are different ways to define and register tasks with Bolt. In this section, we will take a look at the different options and when we should use each one of them.

Bolt Provided Tasks

Bolt provides a set of tasks that are always available when executing Bolt. You don’t need to register them because Bolt does that for you, and they can be configured without prior registration. This is the case for the tasks shown in the following example, which we will use as starting point.

import bolt

config = {
    'pip': {
        'command': 'install',
        'options': {
            'r': 'requirements.txt'
        }
    },
    'delete-pyc': {
        'sourcedir': './source',
        'recursive': True
    },
    'nose': {
        'directory': 'tests'
    }
}

The tasks in the example (pip, delete-pyc, and nose) are provided by Bolt; therefore, we don’t need to register them to use them. With this simple example you can still run each task independently to execute them.

# Install requirements
bolt pip

# Delete existing .pyc files
bolt delete-pyc

# Execute unit tests
bolt nose

As you can see, it is very easy to leverage the existing functionality in Bolt, but the true power comes from the ability to define and create your own tasks or use other tasks provided by tool and library implementers. Let’s take a look at other ways to define tasks.

Composing Tasks From Existing Ones

In the example above, we can use any of the three tasks provided by Bolt, but most of the time I will want to run all those tasks together. I want to make sure that when anyone working on my project gets source changes they can have the correct environment setup; therefore, I want them to install any required packages, and execute the tests with a clean run. For that I can define a composite task that will execute all three. The following shows the full contents of the boltfile.py after adding the composite tasks.

import bolt

bolt.register_task('run-tests', ['pip', 'delete-pyc', 'nose'])
bolt.register_task('default', ['run-tests'])

config = {
    'pip': {
        'command': 'install',
        'options': {
            'r': 'requirements.txt'
        }
    },
    'delete-pyc': {
        'sourcedir': './source',
        'recursive': True
    },
    'nose': {
        'directory': 'tests'
    }
}

We added two additional lines to our bolt file. The first one defines a composite task run-tests that execute the previous three. The second line registers a default task that executes the previously defined run-tests. Both of this tasks will execute the same set of steps.

Now, I can execute bolt run-tests from the command line to execute all tasks, or I can simply call bolt.

Tip

The default task is a special task that gets executed when calling Bolt without specifying a task to execute. You should always provide a default task in your boltfile.py.

Tip

You want your default task to be compose of the steps you will execute more often. I like to define default as the task that I will always execute when I pull new changes from my central repo and before publishing those changes, so I usually include steps to install new required packages, clean the project tree, and execute the unit tests.

Registering Additional Modules

As you start using Bolt more, you will find your-self implementing your own custom tasks or using modules provided by third-party libraries you use (see Creating Custom Tasks ). In order to use those tasks, you need to import the module containing them and register the module. The following example shows how can your register the tasks in a custom or third-party provided module.

# Removed contents for simplicity.
import my_custom_tasks

bolt.register_module_tasks(my_custom_tasks)

Now, all the tasks registered by my_custom_tasks become available for use and configure (see Creating Custom Tasks for more information about how to create your own).

The Power of Configuration

Bolt provides a very powerful configuration mechanism that abstracts what the user wants to do from task implementers that expose configuration settings. This means Bolt gives users the power to describe the configuration parameters of a task, and it takes care of resolving the configuration before it is sent to the task implementation, so that developers implementing tasks get a consistent set of configuration options.

To illustrate how Bolt processes configuration options, I will describe a scenario that I recently run into in one of my projects.

In a recent project, I found myself using the awscli and boto3 libraries available for Python. Without going too much into the details of what I was doing, let’s just say that I usually work on a Windows machine, but many of my applications and scripts are executed in Linux; therefore, cross-platform it is very important for my projects (and one of the reasons why I choose Python).

Turns out that when you use awscli and/or boto3 in Windows, you need to install an additional dependency called pypiwin32. This dependency is not installed nor can be installed on Linux, so that simple fact threw me out for a few seconds on how I was going to manage the requirements for my project. Thankfully, I had Bolt at my disposal and I was able to fix the problem in a very simple, elegant way.

The first step was to add awscli and boto3 to my requirements.txt file.

# In requirements.txt
awscli>=1.11
boto3>=1.4

Then, I created a second requirements file called requirements_win.txt and added the Windows specific library.

# In requirements_win.txt
pypiwin32>=219

I still want all the people collaborating in my code to have the correct set of requirements, but I don’t want them to have to worry about what they need to install because we use bolt for that. So, this is what I did in my bolt file:

# Many lines removed for simplicity.

import bolt
import sys

# Define a task to install the requirements.
if sys.platform.startswith('win'):
    bolt.register_task('requirements', ['pip', 'pip.win']) # More on this below.
else:
    bolt.register_task('requirements', ['pip'])


bolt.register_task('run-tests', ['requirements', 'delete-pyc', 'nose'])
bolt.register_task('default', ['run-tests'])


config = {
    'pip': {
        'command': 'install',
        'options': {
            'r': 'requirements.txt'
        },
        'win': {
            'options': {
                'r': 'requirements_win.txt'
            }
        }
    },
}

This may seem more complicated than it really is once you understand how Bolt processes configurations, so let’s take a look at it step by step.

The first change I made was to check for the OS in which we are running and register a requirements task to install the requirements accordingly. Since, the boltfile.py is just a Python script, I can import sys and create conditional code if I want to.

Now, let’s take a look at what I do on Windows because it is something we haven’t seen yet bolt.register_task('requirements', ['pip', 'pip.win']). What is this ``pip.win thing?

There might be times when I want to configure a task differently depending on the environment I’m running (I will show another example later, but this is so cool that we will expain it first). In those circumstances, instead of providing a completely different boltfile.py with a different configuration, Bolt allows me to nest configuration options that I name my self.

The pip task knows nothing about the win option specified, and it doesn’t have to worry about it, but when the pip task is invoked as pip.win, Bolt takes the configuration options for pip and then adds or overwrites any options defined in the nested win configuration. Therefore, the configuration passed to the pip task when called as pip.win will look like the following:

config = {
    'commmand': 'install',  # Taken from parent
    'options': {
        'r': 'requirements_win.txt'
    }
}

When the task is invoked as pip, the configuration passed is:

config = {
    'commmand': 'install',  # Taken from parent
    'options': {
        'r': 'requirements.txt'
    }
}

In the registration of the requirements task for Windows, we execute both, where if we run on Linux we just execute pip.

Tip

You can nest configurations as deep as you want, so it will be possible to define tasks as pip.win.32 and pip.win.64 if needed. In my experience, one level of nesting is what you will need for most practical cases, and it keeps the configuration readable.

A More Common Configuration Example

The previous example is pretty cool, and it solve a very real problem, but most of the time you will not need or want to have a lot of conditional code in your boltfile.py. The following scenario illustrates a more common approach to define and configure tasks differently for different environments.

Many times I find my self wanting to execute a task differently when I run it in my local development environment than when that task is running in the CI/CD pipeline for my project. A very common scenario for all my projects is that when I run the unit tests locally, which I do all the time, I run them with bare options, so I configure the task in the same way as the examples above.

During the build process, however, I want to get more information about the execution of the tests, and I want to produce some reports and post them to my CI/CD system. Usually, I want a tests results report, and a code coverage report. The following shows the tasks I normally register and configure to execute the unit tests in the different environments.

# Lines omitted for simplicity.

# Developer's tasks. I like to keep the names short, to type less when
# I run them.
#
bolt.register_task('ut', ['pip', 'delete-pyc', 'nose'])

# Ci/CD Tasks
#
bolt.register_task('run-tests', ['pip', 'nose.ci'])

config = {
    # Again, lines omitted for simplicity.

    'nose': {
        'directory': './tests',
        'ci': {
            'options': {
                'with-xunit': True,
                'xunit-file': os.path.join('output', 'unit_tests_log.xml'),
                'with-coverage': True,
                'cover-erase': True,
                'cover-package': './source',
                'cover-branches': True,
                'cover-html': True,
                'cover-html-dir': os.path.join('output', 'coverage')
            }
        }
    }
}

When I’m working on the project, I execute bolt ut, which does all the operations I want in my local development environment. In CI/CD, I execute bolt run-tests, which runs different tasks, but I want you to focus on the different options that I use with nose.

Without using any conditional code in the boltfile.py` itself, I can run ``nose in different ways by specifying a nested configuration ci.

Tip

If you look at the options set for nose.ci, you can see that I use os.path.join() to resolve the location where reports will be generated. This illustrates the power of configuration as code.