salt.pillar.git_pillar

Use a git repository as a Pillar source

Note

This external pillar has been rewritten for the 2015.8.0 release. The old method of configuring this external pillar will be maintained for a couple releases, allowing time for configurations to be updated to reflect the new usage.

This external pillar allows for a Pillar top file and Pillar SLS files to be sourced from a git repository.

However, since git_pillar does not have an equivalent to the pillar_roots parameter, configuration is slightly different. A Pillar top file is required to be in the git repository and must still contain the relevant environment, like so:

base:
  '*':
    - foo

The branch/tag which maps to that environment must then be specified along with the repo's URL. Configuration details can be found below.

Important

Each branch/tag used for git_pillar must have its own top file. This is different from how the top file works when configuring States. The reason for this is that each git_pillar branch/tag is processed separately from the rest. Therefore, if the qa branch is to be used for git_pillar, it would need to have its own top file, with the qa environment defined within it, like this:

qa:
  'dev-*':
    - bar

Additionally, while git_pillar allows for the branch/tag to be overridden (see here, or here for Salt releases before 2015.8.0), keep in mind that the top file must reference the actual environment name. It is common practice to make the environment in a git_pillar top file match the branch/tag name, but when remapping, the environment of course no longer matches the branch/tag, and the top file needs to be adjusted accordingly. When expected Pillar values configured in git_pillar are missing, this is a common misconfiguration that may be to blame, and is a good first step in troubleshooting.

Configuring git_pillar for Salt releases before 2015.8.0

Note

This legacy configuration for git_pillar will no longer be supported as of the Oxygen release of Salt.

For Salt releases earlier than 2015.8.0, GitPython is the only supported provider for git_pillar. Individual repositories can be configured under the ext_pillar configuration parameter like so:

ext_pillar:
  - git: master https://gitserver/git-pillar.git root=subdirectory

The repository is specified in the format <branch> <repo_url>, with an optional root parameter (added in the 2014.7.0 release) which allows the pillar SLS files to be served up from a subdirectory (similar to gitfs_root in gitfs).

To use more than one branch from the same repo, multiple lines must be specified under ext_pillar:

ext_pillar:
  - git: master https://gitserver/git-pillar.git
  - git: dev https://gitserver/git-pillar.git

To remap a specific branch to a specific Pillar environment, use the format <branch>:<env>:

ext_pillar:
  - git: develop:dev https://gitserver/git-pillar.git
  - git: master:prod https://gitserver/git-pillar.git

In this case, the develop branch would need its own top.sls with a dev section in it, like this:

dev:
  '*':
    - bar

The master branch would need its own top.sls with a prod section in it:

prod:
  '*':
    - bar

If __env__ is specified as the branch name, then git_pillar will first look at the minion's environment option. If unset, it will fall back to using branch specified by the master's gitfs_base:

ext_pillar:
  - git: __env__ https://gitserver/git-pillar.git root=pillar

The corresponding Pillar top file would look like this:

{{saltenv}}:
  '*':
    - bar

Note

This feature was unintentionally omitted when git_pillar was rewritten for the 2015.8.0 release. It was added again in the 2016.3.4 release, but it has changed slightly in that release. On Salt masters running 2015.8.0 through 2016.3.3, this feature can only be accessed using the legacy config described above. For 2016.3.4 and later, refer to explanation of the __env__ parameter in the below section.

Versions 2016.3.0 through 2016.3.4 incorrectly check the master's environment config option (instead of the minion's) before falling back to gitfs_base. This has been fixed in the 2016.3.5 and 2016.11.1 releases (2016.11.0 contains the incorrect behavior).

Additionally, in releases before 2016.11.0, both {{env}} and {{saltenv}} could be used as a placeholder for the environment. Starting in 2016.11.0, {{env}} is no longer supported.

Configuring git_pillar for Salt releases 2015.8.0 and later

Note

In version 2015.8.0, the method of configuring git external pillars has changed, and now more closely resembles that of the Git Fileserver Backend. If Salt detects the old configuration schema, it will use the pre-2015.8.0 code to compile the external pillar. A warning will also be logged.

Beginning with Salt version 2015.8.0, pygit2 is now supported in addition to GitPython. The requirements for GitPython and pygit2 are the same as for GitFS, as described here.

Important

git_pillar has its own set of global configuration parameters. While it may seem intuitive to use the global gitfs configuration parameters (gitfs_base, etc.) to manage git_pillar, this will not work. The main difference for this is the fact that the different components which use Salt's git backend code do not all function identically. For instance, in git_pillar it is necessary to specify which branch/tag to be used for git_pillar remotes. This is the reverse behavior from gitfs, where branches/tags make up your environments.

See here for documentation on the git_pillar configuration options and their usage.

Here is an example git_pillar configuration:

ext_pillar:
  - git:
    # Use 'prod' instead of the branch name 'production' as the environment
    - production https://gitserver/git-pillar.git:
      - env: prod
    # Use 'dev' instead of the branch name 'develop' as the environment
    - develop https://gitserver/git-pillar.git:
      - env: dev
    # No per-remote config parameters (and no trailing colon), 'qa' will
    # be used as the environment
    - qa https://gitserver/git-pillar.git
    # SSH key authentication
    - master git@other-git-server:pillardata-ssh.git:
      # Pillar SLS files will be read from the 'pillar' subdirectory in
      # this repository
      - root: pillar
      - privkey: /path/to/key
      - pubkey: /path/to/key.pub
      - passphrase: CorrectHorseBatteryStaple
    # HTTPS authentication
    - master https://other-git-server/pillardata-https.git:
      - user: git
      - password: CorrectHorseBatteryStaple

The main difference between this and the old way of configuring git_pillar is that multiple remotes can be configured under one git section under ext_pillar. More than one git section can be used, but it is not necessary. Remotes will be evaluated sequentially.

Per-remote configuration parameters are supported (similar to gitfs), and global versions of the git_pillar configuration parameters can also be set.

To remap a specific branch to a specific Pillar environment, use the env per-remote parameter:

ext_pillar:
  - git:
    - production https://gitserver/git-pillar.git:
      - env: prod

If __env__ is specified as the branch name, then git_pillar will decide which branch to use based on the following criteria:

  • If the minion has a pillarenv configured, it will use that pillar environment. (2016.11.2 and later)
  • Otherwise, if the minion has an environment configured, it will use that environment.
  • Otherwise, the master's git_pillar_base will be used.

Note

The use of environment to choose the pillar environment dates from a time before the pillarenv parameter was added. In a future release, it will be ignored and either the minion's pillarenv or the master's git_pillar_base will be used.

Here's an example of using __env__ as the git_pillar environment:

ext_pillar:
  - git:
    - __env__ https://gitserver/git-pillar.git:
      - root: pillar

The corresponding Pillar top file would look like this:

{{saltenv}}:
  '*':
    - bar

Note

This feature was unintentionally omitted when git_pillar was rewritten for the 2015.8.0 release. It was added again in the 2016.3.4 release, but it has changed slightly in that release. The fallback value replaced by {{env}} is :conf_master: is git_pillar_base, while the legacy config's version of this feature replaces {{env}} with gitfs_base.

On Salt masters running 2015.8.0 through 2016.3.3, this feature can only be accessed using the legacy config in the previous section of this page.

The same issue which affected the behavior of the minion's environment config value using the legacy configuration syntax (see the documentation in the pre-2015.8.0 section above for the legacy support of this feature) also affects the new-style git_pillar syntax in version 2016.3.4. This has been corrected in version 2016.3.5 and 2016.11.1 (2016.11.0 contains the incorrect behavior).

2016.3.4 incorrectly checks the master's environment config option (instead of the minion's) before falling back to the master's git_pillar_base.

Additionally, in releases before 2016.11.0, both {{env}} and {{saltenv}} could be used as a placeholder for the environment. Starting in 2016.11.0, {{env}} is no longer supported.

With the addition of pygit2 support, git_pillar can now interact with authenticated remotes. Authentication works just like in gitfs (as outlined in the Git Fileserver Backend Walkthrough), only with the global authenication parameter names prefixed with git_pillar instead of gitfs (e.g. git_pillar_pubkey, git_pillar_privkey, git_pillar_passphrase, etc.).

Note

The name parameter can be used to further differentiate between two remotes with the same URL and branch. When using two remotes with the same URL, the name option is required.

How Multiple Remotes Are Handled

As noted above, multiple remotes can be included in the same git ext_pillar configuration. Consider the following:

my_etcd_config:
  etcd.host: 127.0.0.1
  etcd.port: 4001

ext_pillar:
  - etcd: my_etcd_config
  - git:
    - master https://mydomain.tld/foo.git:
      - root: pillar
    - master https://mydomain.tld/bar.git
    - master https://mydomain.tld/baz.git
    - dev https://mydomain.tld/qux.git
  - git:
    - master https://mydomain.tld/abc.git
    - dev https://mydomain.tld/123.git

To understand how pillar data from these repos will be compiled, it's important to know how Salt will process them. The following points should be kept in mind:

  1. Each ext_pillar is called separately from the others. So, in the above example, the etcd ext_pillar will be evaluated first, with the first group of git_pillar remotes evaluated next (and merged into the etcd pillar data). Lastly, the second group of git_pillar remotes will be evaluated, and then merged into the ext_pillar data evaluated before it.

  2. Within a single group of git_pillar remotes, each remote will be evaluated in order, with results merged together as each remote is evaluated.

    Note

    Prior to the 2017.7.0 release, remotes would be evaluated in a non-deterministic order.

  3. By default, when a repo is evaluated, other remotes' which share its pillar environment will have their files made available to the remote being processed.

The first point should be straightforward enough, but the second and third could use some additional clarification.

First, point #2. In the first group of git_pillar remotes, the top file and pillar SLS files in the foo remote will be evaluated first. The bar remote will be evaluated next, and its results will be merged into the pillar data compiled when the foo remote was evaluated. As the subsequent remotes are evaluated, their data will be merged in the same fashion.

But wait, don't these repositories belong to more than one pillar environments? Well, yes. The default method of generating pillar data compiles pillar data from all environments. This behavior can be overridden using a pillarenv. Setting a pillarenv in the minion config file will make that minion tell the master to ignore any pillar data from environments which don't match that pillarenv. A pillarenv can also be specified for a given minion or set of minions when running states, by using he pillarenv argument. The CLI pillarenv will override one set in the minion config file. So, assuming that a pillarenv of base was set for a minion, it would not get any of the pillar variables configured in the qux remote, since that remote is assigned to the dev environment. The only way to get its pillar data would be to specify a pillarenv of dev, which would mean that it would then ignore any items from the base pillarenv. A more detailed explanation of pillar environments can be found here.

Moving on to point #3, and looking at the example ext_pillar configuration, as the foo remote is evaluated, it will also have access to the files from the bar and baz remotes, since all three are assigned to the base pillar environment. So, if an SLS file referenced by the foo remotes's top file does not exist in the foo remote, it will be searched for in the bar remote, followed by the baz remote. When it comes time to evaluate the bar remote, SLS files referenced by the bar remote's top file will first be looked for in the bar remote, followed by foo, and baz, and when the baz remote is processed, SLS files will be looked for in baz, followed by foo and bar. This "failover" logic is called a directory overlay, and it is also used by file_roots and :conf_minion`pillar_roots`. The ordering of which remote is checked for SLS files is determined by the order they are listed. First the remote being processed is checked, then the others that share the same environment are checked. However, before the 2017.7.0 release, since evaluation was unordered, the remote being processed would be checked, followed in no specific order by the other repos which share the same environment.

Beginning with the 2017.7.0 release, this behavior of git_pillar remotes having access to files in other repos which share the same environment can be disabled by setting git_pillar_includes to False. If this is done, then all git_pillar remotes will only have access to their own SLS files. Another way of ensuring that a git_pillar remote will not have access to SLS files from other git_pillar remotes which share the same pillar environment is to put them in a separate git section under ext_pillar. Look again at the example configuration above. In the second group of git_pillar remotes, the abc remote would not have access to the SLS files from the foo, bar, and baz remotes, and vice-versa.

Mountpoints

New in version 2017.7.0.

Assume the following pillar top file:

base:
  'web*':
    - common
    - web.server.nginx
    - web.server.appdata

Now, assume that you would like to configure the web.server.nginx and web.server.appdata SLS files in separate repos. This could be done using the following ext_pillar configuration (assuming that git_pillar_includes has not been set to False):

ext_pillar:
  - git:
    - master https://mydomain.tld/pillar-common.git
    - master https://mydomain.tld/pillar-nginx.git
    - master https://mydomain.tld/pillar-appdata.git

However, in order to get the files in the second and third git_pillar remotes to work, you would need to first create the directory structure underneath it (i.e. place them underneath web/server/ in the repository). This also makes it tedious to reorganize the configuration, as changing web.server.nginx to web.nginx in the top file would require you to also move the SLS files in the pillar-nginx up a directory level.

For these reasons, much like gitfs, git_pillar now supports a "mountpoint" feature. Using the following ext_pillar configuration, the SLS files in the second and third git_pillar remotes can be placed in the root of the git repository:

ext_pillar:
  - git:
    - master https://mydomain.tld/pillar-common.git
    - master https://mydomain.tld/pillar-nginx.git:
      - mountpoint: web/server/
    - master https://mydomain.tld/pillar-appdata.git:
      - mountpoint: web/server/

Now, if the top file changed the SLS target from web.server.nginx, instead of reorganizing the git repository, you would just need to adjust the mountpoint to web/ (and restart the salt-master daemon).

Note

  • Leading and trailing slashes on the mountpoints are optional.
  • Use of the mountpoint feature requires that git_pillar_includes is not disabled.
  • Content from mounted git_pillar repos can only be referenced by a top file in the same pillar environment.
salt.pillar.git_pillar.ext_pillar(minion_id, repo, pillar_dirs)

Checkout the ext_pillar sources and compile the resulting pillar SLS