rest_tornado

A non-blocking REST API for Salt

depends:
  • tornado Python module

configuration:

All authentication is done through Salt's external auth system which requires additional configuration not described here.

In order to run rest_tornado with the salt-master add the following to the Salt master config file.

rest_tornado:
    # can be any port
    port: 8000
    # address to bind to (defaults to 0.0.0.0)
    address: 0.0.0.0
    # socket backlog
    backlog: 128
    ssl_crt: /etc/pki/api/certs/server.crt
    # no need to specify ssl_key if cert and key
    # are in one single file
    ssl_key: /etc/pki/api/certs/server.key
    debug: False
    disable_ssl: False
    webhook_disable_auth: False
    cors_origin: null

Authentication

Authentication is performed by passing a session token with each request. Tokens are generated via the SaltAuthHandler URL.

The token may be sent in one of two ways:

  • Include a custom header named X-Auth-Token.

  • Sent via a cookie. This option is a convenience for HTTP clients that automatically handle cookie support (such as browsers).

See also

You can bypass the session handling via the RunSaltAPIHandler URL.

CORS

rest_tornado supports Cross-site HTTP requests out of the box. It is by default deactivated and controlled by the cors_origin config key.

You can allow all origins by settings cors_origin to *.

You can allow only one origin with this configuration:

rest_tornado:
    cors_origin: http://salt.yourcompany.com

You can also be more specific and select only a few allowed origins by using a list. For example:

rest_tornado:
    cors_origin:
        - http://salt.yourcompany.com
        - http://salt-preprod.yourcampany.com

The format for origin are full URL, with both scheme and port if not standard.

In this case, rest_tornado will check if the Origin header is in the allowed list if it's the case allow the origin. Else it will returns nothing, effectively preventing the origin to make request.

For reference, CORS is a mechanism used by browser to allow (or disallow) requests made from browser from a different origin than salt-api. It's complementary to Authentication and mandatory only if you plan to use a salt client developed as a Javascript browser application.

Usage

Commands are sent to a running Salt master via this module by sending HTTP requests to the URLs detailed below.

Content negotiation

This REST interface is flexible in what data formats it will accept as well as what formats it will return (e.g., JSON, YAML, x-www-form-urlencoded).

  • Specify the format of data in the request body by including the Content-Type header.

  • Specify the desired data format for the response body with the Accept header.

Data sent in POST and PUT requests must be in the format of a list of lowstate dictionaries. This allows multiple commands to be executed in a single HTTP request.

lowstate

A dictionary containing various keys that instruct Salt which command to run, where that command lives, any parameters for that command, any authentication credentials, what returner to use, etc.

Salt uses the lowstate data format internally in many places to pass command data between functions. Salt also uses lowstate for the LocalClient() Python API interface.

The following example (in JSON format) causes Salt to execute two commands:

[{
    "client": "local",
    "tgt": "*",
    "fun": "test.fib",
    "arg": ["10"]
},
{
    "client": "runner",
    "fun": "jobs.lookup_jid",
    "jid": "20130603122505459265"
}]

Multiple commands in a Salt API request will be executed in serial and makes no guarantees that all commands will run. Meaning that if test.fib (from the example above) had an exception, the API would still execute "jobs.lookup_jid".

Responses to these lowstates are an in-order list of dicts containing the return data, a yaml response could look like:

- ms-1: true
  ms-2: true
- ms-1: foo
  ms-2: bar

In the event of an exception while executing a command the return for that lowstate will be a string, for example if no minions matched the first lowstate we would get a return like:

- No minions matched the target. No command was sent, no jid was assigned.
- ms-1: true
  ms-2: true

x-www-form-urlencoded

Sending JSON or YAML in the request body is simple and most flexible, however sending data in urlencoded format is also supported with the caveats below. It is the default format for HTML forms, many JavaScript libraries, and the curl command.

For example, the equivalent to running salt '*' test.ping is sending fun=test.ping&arg&client=local&tgt=* in the HTTP request body.

Caveats:

  • Only a single command may be sent per HTTP request.

  • Repeating the arg parameter multiple times will cause those parameters to be combined into a single list.

    Note, some popular frameworks and languages (notably jQuery, PHP, and Ruby on Rails) will automatically append empty brackets onto repeated parameters. E.g., arg=one, arg=two will be sent as arg[]=one, arg[]=two. This is not supported; send JSON or YAML instead.

A Websockets add-on to saltnado

depends:
  • tornado Python module

In order to enable saltnado_websockets you must add websockets: True to your saltnado config block.

rest_tornado:
    # can be any port
    port: 8000
    ssl_crt: /etc/pki/api/certs/server.crt
    # no need to specify ssl_key if cert and key
    # are in one single file
    ssl_key: /etc/pki/api/certs/server.key
    debug: False
    disable_ssl: False
    websockets: True

All Events

Exposes all "real-time" events from Salt's event bus on a websocket connection. It should be noted that "Real-time" here means these events are made available to the server as soon as any salt related action (changes to minions, new jobs etc) happens. Clients are however assumed to be able to tolerate any network transport related latencies. Functionality provided by this endpoint is similar to the /events end point.

The event bus on the Salt master exposes a large variety of things, notably when executions are started on the master and also when minions ultimately return their results. This URL provides a real-time window into a running Salt infrastructure. Uses websocket as the transport mechanism.

Exposes GET method to return websocket connections. All requests should include an auth token. A way to obtain obtain authentication tokens is shown below.

% curl -si localhost:8000/login \
    -H "Accept: application/json" \
    -d username='salt' \
    -d password='salt' \
    -d eauth='pam'

Which results in the response

{
    "return": [{
        "perms": [".*", "@runner", "@wheel"],
        "start": 1400556492.277421,
        "token": "d0ce6c1a37e99dcc0374392f272fe19c0090cca7",
        "expire": 1400599692.277422,
        "user": "salt",
        "eauth": "pam"
    }]
}

In this example the token returned is d0ce6c1a37e99dcc0374392f272fe19c0090cca7 and can be included in subsequent websocket requests (as part of the URL).

The event stream can be easily consumed via JavaScript:

// Note, you must be authenticated!

// Get the Websocket connection to Salt
var source = new Websocket('wss://localhost:8000/all_events/d0ce6c1a37e99dcc0374392f272fe19c0090cca7');

// Get Salt's "real time" event stream.
source.onopen = function() { source.send('websocket client ready'); };

// Other handlers
source.onerror = function(e) { console.debug('error!', e); };

// e.data represents Salt's "real time" event data as serialized JSON.
source.onmessage = function(e) { console.debug(e.data); };

// Terminates websocket connection and Salt's "real time" event stream on the server.
source.close();

Or via Python, using the Python module websocket-client for example. Or the tornado client.

# Note, you must be authenticated!

from websocket import create_connection

# Get the Websocket connection to Salt
ws = create_connection('wss://localhost:8000/all_events/d0ce6c1a37e99dcc0374392f272fe19c0090cca7')

# Get Salt's "real time" event stream.
ws.send('websocket client ready')


# Simple listener to print results of Salt's "real time" event stream.
# Look at https://pypi.python.org/pypi/websocket-client/ for more examples.
while listening_to_events:
    print ws.recv()       #  Salt's "real time" event data as serialized JSON.

# Terminates websocket connection and Salt's "real time" event stream on the server.
ws.close()

# Please refer to https://github.com/liris/websocket-client/issues/81 when using a self signed cert

Above examples show how to establish a websocket connection to Salt and activating real time updates from Salt's event stream by signaling websocket client ready.

Formatted Events

Exposes formatted "real-time" events from Salt's event bus on a websocket connection. It should be noted that "Real-time" here means these events are made available to the server as soon as any salt related action (changes to minions, new jobs etc) happens. Clients are however assumed to be able to tolerate any network transport related latencies. Functionality provided by this endpoint is similar to the /events end point.

The event bus on the Salt master exposes a large variety of things, notably when executions are started on the master and also when minions ultimately return their results. This URL provides a real-time window into a running Salt infrastructure. Uses websocket as the transport mechanism.

Formatted events parses the raw "real time" event stream and maintains a current view of the following:

  • minions

  • jobs

A change to the minions (such as addition, removal of keys or connection drops) or jobs is processed and clients are updated. Since we use salt's presence events to track minions, please enable presence_events and set a small value for the loop_interval in the salt master config file.

Exposes GET method to return websocket connections. All requests should include an auth token. A way to obtain obtain authentication tokens is shown below.

% curl -si localhost:8000/login \
    -H "Accept: application/json" \
    -d username='salt' \
    -d password='salt' \
    -d eauth='pam'

Which results in the response

{
    "return": [{
        "perms": [".*", "@runner", "@wheel"],
        "start": 1400556492.277421,
        "token": "d0ce6c1a37e99dcc0374392f272fe19c0090cca7",
        "expire": 1400599692.277422,
        "user": "salt",
        "eauth": "pam"
    }]
}

In this example the token returned is d0ce6c1a37e99dcc0374392f272fe19c0090cca7 and can be included in subsequent websocket requests (as part of the URL).

The event stream can be easily consumed via JavaScript:

// Note, you must be authenticated!

// Get the Websocket connection to Salt
var source = new Websocket('wss://localhost:8000/formatted_events/d0ce6c1a37e99dcc0374392f272fe19c0090cca7');

// Get Salt's "real time" event stream.
source.onopen = function() { source.send('websocket client ready'); };

// Other handlers
source.onerror = function(e) { console.debug('error!', e); };

// e.data represents Salt's "real time" event data as serialized JSON.
source.onmessage = function(e) { console.debug(e.data); };

// Terminates websocket connection and Salt's "real time" event stream on the server.
source.close();

Or via Python, using the Python module websocket-client for example. Or the tornado client.

# Note, you must be authenticated!

from websocket import create_connection

# Get the Websocket connection to Salt
ws = create_connection('wss://localhost:8000/formatted_events/d0ce6c1a37e99dcc0374392f272fe19c0090cca7')

# Get Salt's "real time" event stream.
ws.send('websocket client ready')


# Simple listener to print results of Salt's "real time" event stream.
# Look at https://pypi.python.org/pypi/websocket-client/ for more examples.
while listening_to_events:
    print ws.recv()       #  Salt's "real time" event data as serialized JSON.

# Terminates websocket connection and Salt's "real time" event stream on the server.
ws.close()

# Please refer to https://github.com/liris/websocket-client/issues/81 when using a self signed cert

Above examples show how to establish a websocket connection to Salt and activating real time updates from Salt's event stream by signaling websocket client ready.

Example responses

Minion information is a dictionary keyed by each connected minion's id (mid), grains information for each minion is also included.

Minion information is sent in response to the following minion events:

  • connection drops
    • requires running manage.present periodically every loop_interval seconds

  • minion addition

  • minion removal

# Not all grains are shown
data: {
    "minions": {
        "minion1": {
            "id": "minion1",
            "grains": {
                "kernel": "Darwin",
                "domain": "local",
                "zmqversion": "4.0.3",
                "kernelrelease": "13.2.0"
            }
        }
    }
}

Job information is also tracked and delivered.

Job information is also a dictionary in which each job's information is keyed by salt's jid.

data: {
    "jobs": {
        "20140609153646699137": {
            "tgt_type": "glob",
            "jid": "20140609153646699137",
            "tgt": "*",
            "start_time": "2014-06-09T15:36:46.700315",
            "state": "complete",
            "fun": "test.ping",
            "minions": {
                "minion1": {
                    "return": true,
                    "retcode": 0,
                    "success": true
                }
            }
        }
    }
}

Setup

REST URI Reference

/

class salt.netapi.rest_tornado.saltnado.SaltAPIHandler(*args, **kwargs)

Main API handler for base "/"

disbatch()

Disbatch all lowstates to the appropriate clients

get()

An endpoint to determine salt-api capabilities

GET /
Request Headers:
  • Accept -- the desired response format.

Status Codes:

Example request:

curl -i localhost:8000
GET / HTTP/1.1
Host: localhost:8000
Accept: application/json

Example response:

HTTP/1.1 200 OK
Content-Type: application/json
Content-Legnth: 83

{"clients": ["local", "local_async", "runner", "runner_async"], "return": "Welcome"}
post()

Send one or more Salt commands (lowstates) in the request body

POST /
Request Headers:
Response Headers:
  • Content-Type -- the format of the response body; depends on the Accept request header.

Status Codes:

lowstate data describing Salt commands must be sent in the request body.

Example request:

curl -si https://localhost:8000 \
        -H "Accept: application/x-yaml" \
        -H "X-Auth-Token: d40d1e1e" \
        -d client=local \
        -d tgt='*' \
        -d fun='test.ping' \
        -d arg
POST / HTTP/1.1
Host: localhost:8000
Accept: application/x-yaml
X-Auth-Token: d40d1e1e
Content-Length: 36
Content-Type: application/x-www-form-urlencoded

fun=test.ping&arg&client=local&tgt=*

Example response:

Responses are an in-order list of the lowstate's return data. In the event of an exception running a command the return will be a string instead of a mapping.

HTTP/1.1 200 OK
Content-Length: 200
Allow: GET, HEAD, POST
Content-Type: application/x-yaml

return:
- ms-0: true
    ms-1: true
    ms-2: true
    ms-3: true
    ms-4: true

multiple commands

Note that if multiple lowstate structures are sent, the Salt API will execute them in serial, and will not stop execution upon failure of a previous job. If you need to have commands executed in order and stop on failure please use compound-command-execution.

/login

class salt.netapi.rest_tornado.saltnado.SaltAuthHandler(*args, **kwargs)

Handler for login requests

get()

All logins are done over post, this is a parked endpoint

GET /login
Status Codes:

Example request:

curl -i localhost:8000/login
GET /login HTTP/1.1
Host: localhost:8000
Accept: application/json

Example response:

HTTP/1.1 401 Unauthorized
Content-Type: application/json
Content-Length: 58

{"status": "401 Unauthorized", "return": "Please log in"}
post()

Authenticate against Salt's eauth system

POST /login
Request Headers:
Form Parameters:
  • eauth -- the eauth backend configured for the user

  • username -- username

  • password -- password

Status Codes:

Example request:

curl -si localhost:8000/login \
        -H "Accept: application/json" \
        -d username='saltuser' \
        -d password='saltpass' \
        -d eauth='pam'
POST / HTTP/1.1
Host: localhost:8000
Content-Length: 42
Content-Type: application/x-www-form-urlencoded
Accept: application/json

username=saltuser&password=saltpass&eauth=pam

Example response:

HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 206
X-Auth-Token: 6d1b722e
Set-Cookie: session_id=6d1b722e; expires=Sat, 17 Nov 2012 03:23:52 GMT; Path=/

{"return": {
    "token": "6d1b722e",
    "start": 1363805943.776223,
    "expire": 1363849143.776224,
    "user": "saltuser",
    "eauth": "pam",
    "perms": [
        "grains.*",
        "status.*",
        "sys.*",
        "test.*"
    ]
}}

/minions

class salt.netapi.rest_tornado.saltnado.MinionSaltAPIHandler(*args, **kwargs)

A convenience endpoint for minion related functions

get(mid=None)

A convenience URL for getting lists of minions or getting minion details

GET /minions/(mid)
Request Headers:
Status Codes:

Example request:

curl -i localhost:8000/minions/ms-3
GET /minions/ms-3 HTTP/1.1
Host: localhost:8000
Accept: application/x-yaml

Example response:

HTTP/1.1 200 OK
Content-Length: 129005
Content-Type: application/x-yaml

return:
- ms-3:
    grains.items:
        ...
post()

Start an execution command and immediately return the job id

POST /minions
Request Headers:
Response Headers:
  • Content-Type -- the format of the response body; depends on the Accept request header.

Status Codes:

lowstate data describing Salt commands must be sent in the request body. The client option will be set to local_async().

Example request:

curl -sSi localhost:8000/minions \
    -H "Accept: application/x-yaml" \
    -d tgt='*' \
    -d fun='status.diskusage'
POST /minions HTTP/1.1
Host: localhost:8000
Accept: application/x-yaml
Content-Length: 26
Content-Type: application/x-www-form-urlencoded

tgt=*&fun=status.diskusage

Example response:

HTTP/1.1 202 Accepted
Content-Length: 86
Content-Type: application/x-yaml

return:
- jid: '20130603122505459265'
  minions: [ms-4, ms-3, ms-2, ms-1, ms-0]

/jobs

class salt.netapi.rest_tornado.saltnado.JobsSaltAPIHandler(*args, **kwargs)

A convenience endpoint for job cache data

get(jid=None)

A convenience URL for getting lists of previously run jobs or getting the return from a single job

GET /jobs/(jid)

List jobs or show a single job from the job cache.

Status Codes:

Example request:

curl -i localhost:8000/jobs
GET /jobs HTTP/1.1
Host: localhost:8000
Accept: application/x-yaml

Example response:

HTTP/1.1 200 OK
Content-Length: 165
Content-Type: application/x-yaml

return:
- '20121130104633606931':
    Arguments:
    - '3'
    Function: test.fib
    Start Time: 2012, Nov 30 10:46:33.606931
    Target: jerry
    Target-type: glob

Example request:

curl -i localhost:8000/jobs/20121130104633606931
GET /jobs/20121130104633606931 HTTP/1.1
Host: localhost:8000
Accept: application/x-yaml

Example response:

HTTP/1.1 200 OK
Content-Length: 73
Content-Type: application/x-yaml

info:
- Arguments:
    - '3'
    Function: test.fib
    Minions:
    - jerry
    Start Time: 2012, Nov 30 10:46:33.606931
    Target: '*'
    Target-type: glob
    User: saltdev
    jid: '20121130104633606931'
return:
- jerry:
    - - 0
    - 1
    - 1
    - 2
    - 6.9141387939453125e-06

/run

class salt.netapi.rest_tornado.saltnado.RunSaltAPIHandler(*args, **kwargs)

Endpoint to run commands without normal session handling

post()

Run commands bypassing the normal session handling

POST /run

This entry point is primarily for "one-off" commands. Each request must pass full Salt authentication credentials. Otherwise this URL is identical to the root URL (/).

lowstate data describing Salt commands must be sent in the request body.

Status Codes:

Example request:

curl -sS localhost:8000/run \
    -H 'Accept: application/x-yaml' \
    -d client='local' \
    -d tgt='*' \
    -d fun='test.ping' \
    -d username='saltdev' \
    -d password='saltdev' \
    -d eauth='pam'
POST /run HTTP/1.1
Host: localhost:8000
Accept: application/x-yaml
Content-Length: 75
Content-Type: application/x-www-form-urlencoded

client=local&tgt=*&fun=test.ping&username=saltdev&password=saltdev&eauth=pam

Example response:

HTTP/1.1 200 OK
Content-Length: 73
Content-Type: application/x-yaml

return:
- ms-0: true
    ms-1: true
    ms-2: true
    ms-3: true
    ms-4: true

/events

class salt.netapi.rest_tornado.saltnado.EventsSaltAPIHandler(*args, **kwargs)

Expose the Salt event bus

The event bus on the Salt master exposes a large variety of things, notably when executions are started on the master and also when minions ultimately return their results. This URL provides a real-time window into a running Salt infrastructure.

See also

Events & Reactor

get()

An HTTP stream of the Salt master event bus

This stream is formatted per the Server Sent Events (SSE) spec. Each event is formatted as JSON.

GET /events
Status Codes:

Example request:

curl -NsS localhost:8000/events
GET /events HTTP/1.1
Host: localhost:8000

Example response:

HTTP/1.1 200 OK
Connection: keep-alive
Cache-Control: no-cache
Content-Type: text/event-stream;charset=utf-8

retry: 400
data: {'tag': '', 'data': {'minions': ['ms-4', 'ms-3', 'ms-2', 'ms-1', 'ms-0']}}

data: {'tag': '20130802115730568475', 'data': {'jid': '20130802115730568475', 'return': True, 'retcode': 0, 'success': True, 'cmd': '_return', 'fun': 'test.ping', 'id': 'ms-1'}}

The event stream can be easily consumed via JavaScript:

<!-- Note, you must be authenticated! -->
var source = new EventSource('/events');
source.onopen = function() { console.debug('opening') };
source.onerror = function(e) { console.debug('error!', e) };
source.onmessage = function(e) { console.debug(e.data) };

Or using CORS:

var source = new EventSource('/events', {withCredentials: true});

Some browser clients lack CORS support for the EventSource() API. Such clients may instead pass the X-Auth-Token value as an URL parameter:

curl -NsS localhost:8000/events/6d1b722e

It is also possible to consume the stream via the shell.

Records are separated by blank lines; the data: and tag: prefixes will need to be removed manually before attempting to unserialize the JSON.

curl's -N flag turns off input buffering which is required to process the stream incrementally.

Here is a basic example of printing each event as it comes in:

curl -NsS localhost:8000/events |\
        while IFS= read -r line ; do
            echo $line
        done

Here is an example of using awk to filter events based on tag:

curl -NsS localhost:8000/events |\
        awk '
            BEGIN { RS=""; FS="\\n" }
            $1 ~ /^tag: salt\/job\/[0-9]+\/new$/ { print $0 }
        '
tag: salt/job/20140112010149808995/new
data: {"tag": "salt/job/20140112010149808995/new", "data": {"tgt_type": "glob", "jid": "20140112010149808995", "tgt": "jerry", "_stamp": "2014-01-12_01:01:49.809617", "user": "shouse", "arg": [], "fun": "test.ping", "minions": ["jerry"]}}
tag: 20140112010149808995
data: {"tag": "20140112010149808995", "data": {"fun_args": [], "jid": "20140112010149808995", "return": true, "retcode": 0, "success": true, "cmd": "_return", "_stamp": "2014-01-12_01:01:49.819316", "fun": "test.ping", "id": "jerry"}}

/hook

class salt.netapi.rest_tornado.saltnado.WebhookSaltAPIHandler(*args, **kwargs)

A generic web hook entry point that fires an event on Salt's event bus

External services can POST data to this URL to trigger an event in Salt. For example, Amazon SNS, Jenkins-CI or Travis-CI, or GitHub web hooks.

Note

Be mindful of security

Salt's Reactor can run any code. A Reactor SLS that responds to a hook event is responsible for validating that the event came from a trusted source and contains valid data.

This is a generic interface and securing it is up to you!

This URL requires authentication however not all external services can be configured to authenticate. For this reason authentication can be selectively disabled for this URL. Follow best practices -- always use SSL, pass a secret key, configure the firewall to only allow traffic from a known source, etc.

The event data is taken from the request body. The Content-Type header is respected for the payload.

The event tag is prefixed with salt/netapi/hook and the URL path is appended to the end. For example, a POST request sent to /hook/mycompany/myapp/mydata will produce a Salt event with the tag salt/netapi/hook/mycompany/myapp/mydata.

The following is an example .travis.yml file to send notifications to Salt of successful test runs:

language: python
script: python -m unittest tests
after_success:
    - 'curl -sS http://saltapi-url.example.com:8000/hook/travis/build/success -d branch="${TRAVIS_BRANCH}" -d commit="${TRAVIS_COMMIT}"'

See also

Events, Reactor

post(tag_suffix=None)

Fire an event in Salt with a custom event tag and data

POST /hook
Status Codes:

Example request:

curl -sS localhost:8000/hook -d foo='Foo!' -d bar='Bar!'
POST /hook HTTP/1.1
Host: localhost:8000
Content-Length: 16
Content-Type: application/x-www-form-urlencoded

foo=Foo&bar=Bar!

Example response:

HTTP/1.1 200 OK
Content-Length: 14
Content-Type: application/json

{"success": true}

As a practical example, an internal continuous-integration build server could send an HTTP POST request to the URL http://localhost:8000/hook/mycompany/build/success which contains the result of a build and the SHA of the version that was built as JSON. That would then produce the following event in Salt that could be used to kick off a deployment via Salt's Reactor:

Event fired at Fri Feb 14 17:40:11 2014
*************************
Tag: salt/netapi/hook/mycompany/build/success
Data:
{'_stamp': '2014-02-14_17:40:11.440996',
    'headers': {
        'X-My-Secret-Key': 'F0fAgoQjIT@W',
        'Content-Length': '37',
        'Content-Type': 'application/json',
        'Host': 'localhost:8000',
        'Remote-Addr': '127.0.0.1'},
    'post': {'revision': 'aa22a3c4b2e7', 'result': True}}

Salt's Reactor could listen for the event:

reactor:
  - 'salt/netapi/hook/mycompany/build/*':
    - /srv/reactor/react_ci_builds.sls

And finally deploy the new build:

{% set secret_key = data.get('headers', {}).get('X-My-Secret-Key') %}
{% set build = data.get('post', {}) %}

{% if secret_key == 'F0fAgoQjIT@W' and build.result == True %}
deploy_my_app:
  cmd.state.sls:
    - tgt: 'application*'
    - arg:
      - myapp.deploy
    - kwarg:
        pillar:
          revision: {{ revision }}
{% endif %}