cdillc.splunk.ksconf_package module – Create a Splunk app from a local directory

Note

This module is part of the cdillc.splunk collection (version 0.26.1).

It is not included in ansible-core. To check whether it is installed, run ansible-galaxy collection list.

To install it, use: ansible-galaxy collection install cdillc.splunk. You need further requirements to be able to use this module, see Requirements for details.

To use it in a playbook, specify: cdillc.splunk.ksconf_package.

New in cdillc.splunk 0.10.0

Synopsis

• Build a Splunk app using the ksconf package command. This can be as simple drop-in replacement for the community.general.archive module. Advanced use cases can be supported by a combination of ksconf layers and/or file handlers. Idempotent behavior is fully supported.

• The file handling mechanism allows for things like template rendering based on file matching.

• Jinja2 template expansion is supported for (*.j2) files by either using pure Jinja or Ansible Jinja handlers.

• Ksconf layers are fully supported and can be dynamically included or excluded with filters.

• There are two Jinja template modes: Standard jinja mode uses plain Jinja syntax and is more portable (e.g., as it’s also available via the ksconf package command.) The ansible-jinja mode supports all the features of Jinja within Ansible, which includes access to inventory variables, Ansible’s full range of filters and tests, as well as lookup functionality. By default, all file handling is disabled to avoid any unwanted content modification. Use the enable_handler option to enable a template handler.

Note

This module has a corresponding action plugin.

Requirements

The below requirements are needed on the host that executes this module.

• ksconf>=0.11.5

Parameters

Parameter	Comments
app_name string	Specify the top-level folder (app) name. If this is not given, the app folder name is automatically extracted from the basename of source. Placeholder variables, such as {{app_id}} can be used here.
block list / elements=path	Pattern for files/directories to exclude. Default: []
cache string	Control caching behavior. To fully enable or disable cache use on or off. Use rebuild to ignore an existing cache entries and force but allow cache to be stored. read-only allows existing cache to be used but writes or updates to cache is prohibited. Generally rebuild and read-only shouldn’t be necessary unless debugging the caching mechanism or a bug is found or suspected. Note that using rebuild may not result in an actual updated action. The package will fully repackaged internally but this does not bypass the idempotent mechanism. Please see the general notes for additional explanation of this behavior. Choices: "on" ← (default) "off" "rebuild" "read-only"
cache_storage path	Path to local cache storage. Avoid sharing this too broadly, for example in a multitenant scenario. Default: "~/.cache/cdillc-splunk-ksconf-package"
context dictionary	Free-form metadata that is passed through to the output. Use this to pass around important app context variables that can be conveniently retained when looping and using register.
enable_handler list / elements=string	Enable one or more file handlers for template expansion and encrypted file support. Use jinja for basic Jinja2 syntax support. All necessary variables must be passed in via the template_vars argument. Use ansible-jinja to use the Ansible engine to handle all jinja rendering. By default, all Ansible variables, filters, tests, and lookups are available. This is effectively like using the ansible.builtin.template module to render all *.j2 files before packaging an app. Use ansible-vault to enable ansible vault decryption of files. This will only decrypt files matching *.vault. Choices: "ansible-jinja" "ansible-vault" "jinja"
encrypt string	Encrypt the resulting archive file. Set to vault to encrypt with ansible-vault. Choices: "False" ← (default) "vault"
file path / required	Tarball file created of the app. This can be .spl or .tar.gz This parameter supports dynamic placeholders. Variables are listed here. This value may require extra planning in scenarios where ksconf layers are in use and/or when templates are being used in combinations with variables. Any time a single source input directory is used to build multiple variations or versions of an app a unique output file is needed. Without taking this into consideration, various inefficiencies (cache misses) or the wrong variation being deployed or other confusing behavior. This can be avoided with planning. When using various layers, add [[layers_hash]] to the filename to easily solve this problem. When using templates with variables a custom approach is needed. For example, if building apps for different regions based on the region variable. simply ensure that {{region}} appears in in file value. These approaches are not mutually exclusive. It’s possible to combine both layers and Jinja variable approaches in combination.
follow_symlink boolean	Follow symbolic links pointing to directories. Symlinks to files are always followed. Choices: false ← (default) true
layer_method string	Type of layers used within the source directory. Choices: "auto" "dir.d" ← (default) "disable"
layers list / elements=dictionary	Include and exclude rules regarding which layers to include in the generated app. Layer filters rules are evaluated sequentially, and the last match wins. List of dictionaries with a single key, either include or exclude Default: []
exclude string	Specify a layer or layer glob pattern to exclude.
include string	Specify a layer or layer glob pattern to include.
local string	Define handling of of local directory and local.meta file. Use preserve to keep the local artifacts as-is. block will exclude local artifacts from the generated app archive. promote will merge any local artifacts into the default layer. Choices: "preserve" ← (default) "block" "promote"
source aliases: src path / required	Path of app directory
template_vars dictionary	Add-hoc variables useable during template expansion. This dictionary can be structured any way that’s helpful. There are no restrictions imposed, but be aware that sending more variables than needed could result in extra processing. When using the ansible-jinja handler, these values will be added to existing Ansible variables. Variables set here will have the highest precedence. Default: {}

Attributes

Attribute	Support	Description
check_mode	Support: none	Can run in check_mode and return changed status prediction without modifying target
diff_mode	Support: none	Will return details on what has changed (or possibly needs changing in check_mode), when in diff mode
platform	Platform: posix	Target OS/families that can be operated against

Notes

Note

• As of v0.19.0, the ksconf_package modules is implemented as an action. This means that it must run on the controller not the target machine. In practice, this should not impact most use cases as specifying delegate_to: localhostwas the most common way to use this module anyways. Switching from a module to an action allows us access to the full variable inventory that isn’t accessible to remote modules without explicitly passing in every variable needed.

• Several parameters accept ksconf variables. Traditionally these are written in a Jinja-2 like syntax, which is familiar, but leads to some confusion when embedded in an Ansible playbook. To avoid Jinja escaping these variables manually, this modules supports [[var]] syntax too. If the path includes [[version]] that will be translated to {{version}} before be handed to the ksconf tool.

• Jinja template files are detected based on the *.j2 pattern. The .j2 extension will be removed from the final name. Remember this off by default, and must be enabled with enable_handler.

• Idempotent operations are supported by rendering the app to a temporary file and then comparing the content signature of a newly generated tarball against the previous one. To speed this up a cached .manifest file is stored along side file so that the previous hash doesn’t need to be recalculated in many cases. As this requires a non-trivial amount of work for larger apps, this can feel slow.

• As of v0.26 caching behavior is supported to reduce the amount of work that needs to be done in many no-change operations. (Please avoid the first attempt of caching support added v0.25.) Caching is implemented by saving the output of previous runs and determining if any parameters or if the source directory itself has undergone any changes since the last execution. A cache hit occurs when no changes have been detected, and therefore reuse occurs; the output file and return values are re-used from a previous execution. Specifically any change to file, block, layer_method, local, follow_symlink, app_name, or encrypt can modify the resulting tarball, and therefore will trigger a cache miss. Any layers given are also taken into consideration, but only changes to which layers match or the content within those layers will result in a cache miss. For non-layered apps, the entire source directory is scanned for changes Any change will trigger a full rebuild, even if the change is to a file that is blocked. Change detection for source is based on file name, size, timestamps and not a hash. This allows quick execution when no inputs have changed which is a very common scenario. Templates are handled differently from normal files. All templates are expanded and a hash of the rendered content is compared. This is more expensive, but it ensures that templates or variable changes are handled predictably.

• How to force a changed outcome? Or, how to force an app re-deployment. This is a bit difficult to do by design and should not be necessary most of the time. Both caching and idempotent mechanisms will attempt to keep avoid unnecessary changes. So simply clearing the cache, or deleting the manifest file is not enough. Simply touching a files modification time will trigger a cache miss, but ultimately will report action=unchanged as there is no change to the actual content. To force a change result (or created action), either change the actual content of a file, or remove the archive file from the filesystem. Also keep in mind that if the ultimate goal is to trigger an app re-install and if installation is handle by cdillc.splunk.ksconf_app_sideload, then simply removing the archive file will still be irrelevant as that module also builds a local manifest in attempt to avoid unnecessary installations, unless app content has changed.

• When using both templates and layering, be aware that Jinja2 templates are expanded before layer filtering. This allows one layer to include indexes.conf and another layer to include indexes.conf.j2. All templates will be expanded first, then the resulting layers will be merged.

• Normal use case: Often apps are contained within a version control system are packaged on the controller node and shipped to various Splunk nodes. App installation can be done using the cdillc.splunk.ksconf_app_sideload module. Alternative installation methods include using Splunk’s app install CLI, or ship apps to Splunk Cloud via API.

Examples

- name: Build addon using a specific set of layers
  cdillc.splunk.ksconf_package:
    source: "{{ app_repo }}/Splunk_TA_nix"
    file: "{{ install_root }}/build/Splunk_TA_nix.spl"
    block: ["*.sample"]
    local: preserve
    follow_symlink: false
    layers:
      - exclude: "30-*"
      - include: "30-{{role}}"
      - exclude: "40-*"
      - include: "40-{{env}}"

# More complex example that loops over an 'apps_inventory' list that contains both
# local directories and pre-packaged tarballs (which don't need to be re-packaged)
- name: Render apps from version control
  cdillc.splunk.ksconf_package:
    source: "{{ rendered_apps_folder }}/{{ item.name }}"
    file: "{{ tarred_apps_folder }}/{{ item.name }}-[[ layers_hash ]].tgz"
    local: preserve
    layers:
      - include: "10-upstream"
      - include: "20-common"
      - include: "30-{{ app_role }}"
      - include: "40-{{ layer_env }}"
      - include: "50-{{ app_role }}-{{ layer_env }}"
      - include: "60-{{ org }}"
    enable_handler: ansible-jinja
    template_vars:
      org_name: acme
      default_retention: 7d
      splunk:
        key_password: "{{ splunk.key_password }}"
  delegate_to: localhost
  run_once: true
  loop: >
    {{ apps_inventory
    | selectattr("state", "eq", "present")
    | rejectattr("tarball")
    }}
  register: app_render_output
  tags: render

Return Values

Common return values are documented here, the following are the fields unique to this module:

Key	Description
action string	Resulting action code. Values are created, updated, unchanged, or cached. Both cached and unchanged indicate that the output file was reused from a previous run. cached means that no-changes were found early in the process (based on the source folder and parameters), whereas unchanged means that no change was detected but only after packaging the entire app. created means no existing output file was present. updated means that the output checksum changed. If encrypt has changed, without any changes in source, then actions will be either encrypt and decrypt. Returned: always Sample: "unchanged"
app_name string	Final name of the splunk app, which is the top-level directory included in the generated archive. Returned: always Sample: "org_custom_tech"
archive path	Location of the generated archive. This will either be the literal value passed to file, or the expanded version of file when ksconf variables are used. Returned: always Sample: "/tmp/splunk_apps/org_custom_tech-1.4.3.tgz"
archive_size integer	Size of the generated archive file in bytes. Returned: always
cache dictionary	Cache result status or message in case of a cache error. Values include: hit, miss, created, updated, or disabled. Other values start with failed and a reason message. Note that miss will rarely occur unless new cache cannot be created or if cache is set to read-only. Both updated and created imply a cache miss. Returned: always
context dictionary	Optional pass-through field. See the context parameter. Returned: when provided
encryption string	Final encryption state of the archive. Returned: always Sample: "vault"
encryption_size integer	Size of file on disk, after encryption. Returned: if encrypt is enabled
new_hash string	Checksum of the previous (existing) tarball, if present. This is a SHA256 of the uncompressed content. Returned: always
old_hash string	Checksum of the new tarball. See notes regarding new_hash for more details. Returned: always Sample: "e1617a87ea51c0ca930285c0ce60af4308513ea426ae04be42b1d7b47aba16a5"
stdout string	Output stream of details from the ksconf packaging operations. Returned: when package created

Authors

• Lowell C. Alleman (@lowell80)

Collection links

• Issue Tracker
• Homepage
• Repository (Sources)