From ddae3788c49e6acf4bf555ec229aa35cf70692c9 Mon Sep 17 00:00:00 2001 From: Matt Davis Date: Mon, 3 Mar 2025 12:05:42 -0800 Subject: [PATCH 01/15] WIP 2.19 Porting Guide stub with data tagging --- .../porting_guide_core_2.19.rst | 446 +++++++++++++++++- 1 file changed, 439 insertions(+), 7 deletions(-) diff --git a/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst b/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst index a4f59df5f14..2d820ab00ee 100644 --- a/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst +++ b/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst @@ -5,24 +5,455 @@ Ansible-core 2.19 Porting Guide ******************************* -This section discusses the behavioral changes between ``ansible-core`` 2.18 and ``ansible-core`` 2.19. +This section discusses the behavioral changes between ansible-core 2.18 and ansible-core 2.19. -It is intended to assist in updating your playbooks, plugins and other parts of your Ansible infrastructure so they will work with this version of Ansible. +It is intended to assist in updating your playbooks, plugins and other parts of your Ansible infrastructure so they will +work with this version of Ansible. -We suggest you read this page along with `ansible-core Changelog for 2.19 `_ to understand what updates you may need to make. +We suggest you read this page along with +`ansible-core Changelog for 2.19 `_ +to understand what updates you may need to make. -This document is part of a collection on porting. The complete list of porting guides can be found at :ref:`porting guides `. +This document is part of a collection on porting. +The complete list of porting guides can be found at :ref:`porting guides `. .. contents:: Topics +Engine +====== + +This release includes an overhaul of the templating system along with a new feature dubbed Data Tagging. +These changes have wide-ranging positive effects on security, performance, and user experience. +Backward compatibility has been preserved where practical, but some breaking changes were necessary. + + +Templating +---------- + +Template Trust Model Inversion +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Previously, ansible-core implicitly trusted all string values to be rendered as Jinja templates, but applied an "unsafe" +wrapper object around strings obtained from untrusted sources (eg, module results). +Unsafe-wrapped strings were silently ignored by the template engine, as many templating operations can execute arbitrary +code on the control host as the user running ansible-core. +This required any code that operated on strings to correctly propagate the wrapper object, which resulted in numerous +CVE-worthy RCE (remote code execution) vulnerabilities. + +This release inverts the previous trust model. +Only strings marked as loaded from a trusted source are eligible to be rendered as templates. +Untrusted values can (as before) be referenced by templates, but the template expression itself must always be trusted. +While this change still requires consideration for propagation of trust markers when manipulating strings, failure to do +so now results in a loss of templating ability instead of a potentially high-severity security issue. + +Attempts to render a template appearing in an untrusted string will (as before) return the original string unmodified. +By default, attempting to render an untrusted template fails silently, though such failures can be elevated to a +warning or error via configuration. + +Newly-created string results from template operations will never have trust automatically applied, though templates that +return existing trusted string values unmodified will not strip their trust. +It is also possible for plugins to explicitly apply trust. + +Backward-compatible template trust behavior is applied automatically in most cases; e.g., templates appearing in +playbooks, roles, variable files, and most built-in inventory plugins will yield trusted template strings. +Custom plugins that source template strings will be required to use new public APIs to apply trust where appropriate, +or declaratively request automatic trust application for templates to execute properly (see `Plugin API`_ for additional +information). + +See `Troubleshooting Untrusted Templates`_ for additional information. + + +Native Jinja Mode Required +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Previous versions supported templating in two different modes: + +The first, Jinja's original string templating mode, converted the result of each templating operation to a string. + +The second, Jinja's native mode, attempted to preserve variable types while templating, but could not always do so. + +In both modes, ansible-core evaluated template string results as Python literals, falling back to the original string. +Selection of the templating mode was controlled by configuration, defaulting to Jinja's original string templating. + +Starting with this release, use of Jinja's native templating is required. +Preservation of native types in templating has been improved to correct gaps in the previous implementation, +entirely eliminating the final literal evaluation pass (a frequent source of confusion, errors, and performance issues). +In rare cases where playbooks relied on implicit object conversion from strings, an explicit conversion will be +required. + +The configuration option for setting the templating mode is deprecated and no longer has any effect. + + +Lazy Templating +^^^^^^^^^^^^^^^ + +Ansible's interface with the Jinja templating engine has been heavily refined, yielding significant performance +improvements for many complex templating operations. +Previously, deeply-nested, recursive, or self-referential templating operations were always resolved to their full depth +and breadth on every access, including repeated access to the same data within a single templating operation. +This resulted in expensive and repetitive evaluation of the same templates within a single logical template operation, +even for templates deep inside nested data structures that were never directly accessed. +The new template engine lazily defers nearly all recursion and templating until values are accessed or known to be +exiting the template engine, and intermediate nested or indirected templated results are cached for the duration of the +template operation, reducing repetitive templating. +These changes have shown exponential performance improvements for many real-world complex templating scenarios. + + +Error Handling +-------------- + +Contextual Warnings and Errors +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Changes to internal error handling in ansible-core will be visible in many situations that result in a warning or error. +In most cases, the operational context (what was happening when the error or warning was generated) and data element(s) +involved are captured and included in user-facing messages. +Errors and warnings that occur during task execution are more consistently included in the task result, with the full +details accessible to callbacks and (in the case of errors), a minimal error message in the ``msg`` field of the result. +Due to the standardized nature of this error handling, seemingly redundant elements may appear in some error messages. +These will improve over time as other error handling improvements are made, but are currently necessary to ensure proper +context is available in all error situations. +Error message contents are not considered stable, so automation that relies on them should be avoided when possible. + + +Variable Provenance Tracking +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The new Data Tagging feature expands provenance tracking on variables to nearly every source. +This allows for much more descriptive error messaging, as the entire chain of execution can be consulted to include +contextual information about what was happening when an error occurred- in many cases including display of the offending +source lines with column markers. +Limited support for non-file provenance-tracking is included (e.g., variables from CLI arguments, inventory plugins, +environment variables) and will likely be expanded over time. + + +Deprecation Warnings on Value Access +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +New features allow most ansible-core variables and values to be tagged as deprecated. +Plugins and modules can apply these tags to augment deprecated elements of their return values with a description and +help text to suggest alternatives, which will be displayed in a runtime warning when the tagged value is accessed by +e.g., a playbook or template. +This allows for easier evolution and removal of module/fact results and obsolete core behaviors. +A specific example included in this release is the deprecation of the legacy ``INJECT_FACTS_AS_VARS`` behavior, which +injects fact values directly into the top-level variable namespace. +The injection behavior is potentially dangerous, and the configuration to disable it has existed for many Ansible +releases, but deprecation and an eventual default behavior change were not possible until the current release, since +warnings on access could not be issued reliably. + + +Improved Ansible Module Error Handling +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Ansible modules implemented in Python now have exception handling provided by the AnsiballZ wrapper. +In previous versions of ansible-core, unhandled exceptions in an Ansible module simply printed a traceback and exited +without providing a standard module response, which caused the task result to contain a generic ``MODULE FAILURE`` +message and any raw output text produced by the module. + +To address this, modules often implemented unnecessary ``try/except`` blocks around most code where specific error +handling was not possible, only to call ``AnsibleModule.fail_json`` with a generic failure message. +This pattern is no longer necessary, as all unhandled exceptions in Ansible Python modules are now captured by the +AnsiballZ wrapper and returned as a structured module result, including the automatic inclusion of traceback information +(when enabled by the controller). + + +Improved Handling of Undefined +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Undefined handling has been improved to avoid situations where a Jinja plugin silently ignores undefined values. + +This commonly occurs when a Jinja plugin, such as a filter or test, checks the type of variables without accounting for +the possibility of undefined values being present. + +Example - Missing Attribute +""""""""""""""""""""""""""" + +This task incorrectly references a non-existent ``exists`` attribute from a ``stat`` result in a conditional. +The undefined value goes undetected in previous versions because it is passed to the ``false`` Jinja test plugin, +which silently ignores undefined values. +As a result, this conditional could never be ``True`` in earlier versions of ansible-core, +and there was no indication that the ``failed_when`` expression was invalid. + +.. code-block:: yaml+jinja + + - stat: + path: /does-not-exist + register: result + failed_when: result.exists is false + +In the current release the faulty expression is detected and results in an error. + +This can be corrected by adding the missing ``stat`` attribute to the conditional: + +.. code-block:: yaml+jinja + + - stat: + path: /does-not-exist + register: result + failed_when: result.stat.exists is false + + +Displaying Tracebacks +^^^^^^^^^^^^^^^^^^^^^ + +In previous ansible-core versions, tracebacks from some controller-side errors were available by increasing verbosity +with the ``-vvv`` option, but the availability and behavior was inconsistent. +This feature was also limited to errors. + +Handling of errors, warnings and deprecations throughout much of the ansible-core codebase has now been standardized. +Tracebacks can be optionally collected and displayed for all exceptions, as well as at the call site of errors, +warnings, or deprecations (even in module code) using the ``ANSIBLE_DISPLAY_TRACEBACK`` environment variable. + +Valid options are: + +* ``always`` - Tracebacks will always be displayed. This option takes precedence over others below. +* ``never`` - Tracebacks will never be displayed. This option takes precedence over others below. +* ``error`` - Tracebacks will be displayed for errors. +* ``warning`` - Tracebacks will be displayed for warnings other than deprecation warnings. +* ``deprecated`` - Tracebacks will be displayed for deprecation warnings. + +Multiple options can be combined by separating them with commas. +A CLI option for this setting is under consideration. + + Playbook ======== -* Timeout waiting on privilege escalation (``become``) is now an unreachable error instead of a task error. Existing playbooks should be changed to replace ``ignore_errors`` with ``ignore_unreachable`` on tasks where timeout on ``become`` should be ignored. - .. error:: - Timeout (12s) waiting for privilege escalation prompt: +Task Error Handling +------------------- + +Timeout waiting on privilege escalation (``become``) is now an unreachable error instead of a task error. +Existing playbooks should be changed to replace ``ignore_errors`` with ``ignore_unreachable`` on tasks where timeout on ``become`` should be ignored. + + +Broken Conditionals +------------------- + +Broken conditionals occur when the input expression or template is not a string, or the result is not a boolean. +Python and Jinja provide implicit "truthy" evaluation of most non-empty non-boolean values in conditional expressions. +While sometimes desirable for brevity, truthy conditional evaluation often masks serious logic errors in playbooks that +could not be reliably detected by previous versions of ansible-core. + +Changes to templating in this release detects non-boolean conditionals during expression evaluation and reports an error +by default. The error can be temporarily reduced to a warning via the ``ALLOW_BROKEN_CONDITIONALS`` config setting. + +The following examples are derived from broken conditionals that masked logic errors in actual playbooks. + + +Example - Implicit Boolean Conversion +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +This expression relies on an implicit truthy evaluation of ``inventory_hostname``. +An explicit predicate with a boolean result, such as ``| length > 0`` or ``is truthy``, should be used instead. + +.. code-block:: yaml+jinja + + - assert: + that: inventory_hostname + +The error reported is:: + + Conditional result was 'localhost' of type 'str', which evaluates to True. Conditionals must have a boolean result. + + +This can be resolved by using an explicit boolean conversion: + +.. code-block:: yaml+jinja + + - assert: + that: inventory_hostname | length > 0 + + +Example - Unintentional Truthy Conditional +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The second part of this conditional is erroneously quoted. +The quoted part becomes the expression result considered ``True``, so the expression will never be ``False``. + +.. code-block:: yaml+jinja + + - assert: + that: inventory_hostname is defined and 'inventory_hostname | length > 0' + + +The error reported is:: + + Conditional result was 'inventory_hostname | length > 0' of type 'str', which evaluates to True. Conditionals must have a boolean result. + + +This can be resolved by removing the unnecessary quotes: + +.. code-block:: yaml+jinja + + - assert: + that: inventory_hostname is defined and inventory_hostname | length > 0 + + +Example - Jinja Order of Operations +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +This expression uses the ``~`` concatenation operator, which is evaluated after the contains test. +The result is always a non-empty string, which evaluates to ``True``. + +.. code-block:: yaml+jinja + + - assert: + that: inventory_hostname is contains "local" ~ "host" + + +The error reported is:: + + Conditional result was 'Truehost' of type 'str', which evaluates to True. Conditionals must have a boolean result. + + +This can be resolved by inserting parenthesis to resolve the concatenation operation before the ``contains`` test: + +.. code-block:: yaml+jinja + + - assert: + that: inventory_hostname is contains("local" ~ "host") + + +Example - Dictionary as Conditional +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +This conditional should have been quoted. +In a list, a space after the colon means the YAML parser sees the value as a mapping, not a string. +Any non-empty mapping will always be ``True``. + +.. code-block:: yaml+jinja + + - assert: + that: + - result.msg == "some_key: some_value" + # ^^ colon+space == problem + +The error reported is:: + + Conditional expressions must be strings. + + +This can be resolved by quoting the entire assertion expression: + +.. code-block:: yaml+jinja + + - assert: + - 'result.msg == "some_key: some_value"' + + +Multi-pass Templating +--------------------- + +Embedding templates within other templates or expressions can result in untrusted templates being executed. +The overhauled templating engine in this release no longer supports this insecure behavior. + +Example - Dynamic Expression Construction +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +This conditional is dynamically created using a template, which is expected to be evaluated as an expression. +Previously, the template was rendered by task argument templating, resulting in a plain string, which was later +evaluated by the ``assert`` action. + +.. code-block:: yaml+jinja + + - assert: + that: inventory_hostname {{ comparison }} 'localhost' + vars: + comparison: == + + +The error reported is:: + + Syntax error in expression. Template delimiters are not supported in expressions: chunk after expression + + +Troubleshooting Untrusted Templates +----------------------------------- + +By default, untrusted templates are silently ignored. +To troubleshoot trust issues with templates, it can be helpful to trigger a warning or error for untrusted templates. +The environment variable ``_ANSIBLE_TEMPLAR_UNTRUSTED_TEMPLATE_BEHAVIOR`` can be used to control this behavior. + +Valid options are: + +* ``warn`` - A warning will be issued when an untrusted template is encountered. +* ``fail`` - An error will be raised when an untrusted template is encountered. +* ``ignore`` - Untrusted templates are silently ignored and used as-is. This is the default behavior. + +.. note:: + This optional warning and failure behavior is experimental and subject to change in future versions. + + +Plugin API +========== + +Deprecating Values +------------------ + +Plugins and Python modules can tag returned values as deprecated with the new ``deprecate_value`` function from +``ansible.module_utils.datatag``. +A description of the deprecated feature, optional help text, and removal timeframes can be attached to the value, which +will appear in a runtime warning if the deprecated value is referenced in an expression. +The warning message will include information about the module/plugin that applied the deprecation tag and the +location of the expression that accessed it. + +.. code-block:: python + + from ansible.module_utils.datatag import deprecate_value + + ... + + module.exit_json( + color_name=deprecate_value( + value="blue", + msg="The `color_name` return value is deprecated.", + help_text="Use `color_code` instead.", + ), + color_code="#0000ff", + ) + + +When accessing the `color_name` from the module result, the following warning will be shown:: + + [DEPRECATION WARNING]: The `color_name` return value is deprecated. This feature will be removed from the 'ns.collection.paint' module in a future release. + Origin: /examples/use_deprecated.yml:8:14 + + 6 + 7 - debug: + 8 var: result.color_name + ^ column 14 + + Use `color_code` instead. + + +Applying Template Trust to Individual Values +-------------------------------------------- + +String values are no longer trusted to be rendered as templates by default. Strings loaded from playbooks, vars files, +and other built-in trusted sources are usually marked trusted by default. +Plugins that create new string instances with embedded templates must use the new ``trust_value`` function from +``ansible.module_utils.datatag`` to tag those values as originating from a trusted source to allow the templates to be +rendered. + + +.. warning:: + This section and the associated public API are currently incomplete. + + +Applying Template Trust in Inventory and Vars Plugins +----------------------------------------------------- + +Inventory plugins can set group and host variables. +In most cases, these variables are static values from external sources and do not require trust. +Values that can contain templates will require explicit trust via ``trust_value`` to be allowed to render, but +trust should not be applied to variable values from external sources that could be maliciously altered to include +templates. + +.. warning:: + This section and the associated public API are currently incomplete. + Command Line ============ @@ -90,6 +521,7 @@ Plugins ansible_ssh_password_mechanism: sshpass + Porting custom scripts ====================== From 3695d5cbdfe94b63a0e890224bdd015c7e53da64 Mon Sep 17 00:00:00 2001 From: Matt Clay Date: Tue, 11 Mar 2025 11:37:52 -0700 Subject: [PATCH 02/15] Add another multi-pass templating example --- .../porting_guide_core_2.19.rst | 28 +++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst b/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst index 2d820ab00ee..4fda05daac9 100644 --- a/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst +++ b/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst @@ -349,6 +349,34 @@ Multi-pass Templating Embedding templates within other templates or expressions can result in untrusted templates being executed. The overhauled templating engine in this release no longer supports this insecure behavior. +Example - Unnecessary Template in Expression +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +This conditional references a variable using a template, instead of using the variable directly in the expression. + +.. code-block:: yaml+jinja + + - assert: + that: 1 + {{ value }} == 2 + vars: + value: 1 + + +The error reported is:: + + Syntax error in expression. Template delimiters are not supported in expressions: expected token ':', got '}' + + +This can be resolved by referencing the variable without a template: + +.. code-block:: yaml+jinja + + - assert: + that: 1 + value == 2 + vars: + value: 1 + + Example - Dynamic Expression Construction ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ From cb885270208c8ae40d9f8719087f5c0e7b2c50e0 Mon Sep 17 00:00:00 2001 From: Matt Clay Date: Tue, 18 Mar 2025 11:21:49 -0700 Subject: [PATCH 03/15] Add example for unintentional string conversion --- .../porting_guide_core_2.19.rst | 40 ++++++++++++++++++- 1 file changed, 39 insertions(+), 1 deletion(-) diff --git a/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst b/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst index 4fda05daac9..2caf8aa813a 100644 --- a/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst +++ b/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst @@ -77,12 +77,50 @@ In both modes, ansible-core evaluated template string results as Python literals Selection of the templating mode was controlled by configuration, defaulting to Jinja's original string templating. Starting with this release, use of Jinja's native templating is required. +The configuration option for setting the templating mode is deprecated and no longer has any effect. + Preservation of native types in templating has been improved to correct gaps in the previous implementation, entirely eliminating the final literal evaluation pass (a frequent source of confusion, errors, and performance issues). In rare cases where playbooks relied on implicit object conversion from strings, an explicit conversion will be required. -The configuration option for setting the templating mode is deprecated and no longer has any effect. +Some existing templates may unintentionally convert non-strings to strings. +In previous versions this was sometimes masked by the evaluation of strings as Python literals. + +Example - Unintentional String Conversion +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +This expression erroneously passes a list to the ``replace`` filter, which operates only on strings. +The filter silently converts the list input to a string. +Due to some string results previously parsing as lists, this mistake often went undetected in earlier versions. + +.. code-block:: yaml+jinja + + - debug: + msg: "{{ ['test1', 'test2'] | replace('test', 'prod') }}" + +The result of this template becomes a string:: + + ok: [localhost] => { + "msg": "['prod1', 'prod2']" + } + + +This can be resolved by using the ``map`` filter to apply the ``replace`` filter to each list element: + +.. code-block:: yaml+jinja + + - debug: + msg: "{{ ['test1', 'test2'] | map('replace', 'test', 'prod') }}" + +The result of the corrected template remains a list:: + + ok: [localhost] => { + "msg": [ + "prod1", + "prod2" + ] + } Lazy Templating From 2566ca58746adaaabc3a658662490571a8b57a91 Mon Sep 17 00:00:00 2001 From: Matt Clay Date: Tue, 18 Mar 2025 11:44:25 -0700 Subject: [PATCH 04/15] Add blank lines after code blocks --- docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst b/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst index 2caf8aa813a..0b47135c8f9 100644 --- a/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst +++ b/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst @@ -99,6 +99,7 @@ Due to some string results previously parsing as lists, this mistake often went - debug: msg: "{{ ['test1', 'test2'] | replace('test', 'prod') }}" + The result of this template becomes a string:: ok: [localhost] => { @@ -113,6 +114,7 @@ This can be resolved by using the ``map`` filter to apply the ``replace`` filter - debug: msg: "{{ ['test1', 'test2'] | map('replace', 'test', 'prod') }}" + The result of the corrected template remains a list:: ok: [localhost] => { From 03eebca366b65b386c2ce617b7a5da1aa9bcc756 Mon Sep 17 00:00:00 2001 From: Matt Clay Date: Tue, 18 Mar 2025 11:46:21 -0700 Subject: [PATCH 05/15] Fix heading type --- docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst b/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst index 0b47135c8f9..5570be60cf6 100644 --- a/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst +++ b/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst @@ -88,7 +88,7 @@ Some existing templates may unintentionally convert non-strings to strings. In previous versions this was sometimes masked by the evaluation of strings as Python literals. Example - Unintentional String Conversion -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +""""""""""""""""""""""""""""""""""""""""" This expression erroneously passes a list to the ``replace`` filter, which operates only on strings. The filter silently converts the list input to a string. From 201d57f4182e32682b2711ac77132d39853dacd6 Mon Sep 17 00:00:00 2001 From: Matt Clay Date: Tue, 18 Mar 2025 11:56:05 -0700 Subject: [PATCH 06/15] Attempt to fix warnings --- .../docsite/rst/porting_guides/porting_guide_core_2.19.rst | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst b/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst index 5570be60cf6..f33321edc59 100644 --- a/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst +++ b/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst @@ -100,7 +100,9 @@ Due to some string results previously parsing as lists, this mistake often went msg: "{{ ['test1', 'test2'] | replace('test', 'prod') }}" -The result of this template becomes a string:: +The result of this template becomes a string: + +.. code-block:: console ok: [localhost] => { "msg": "['prod1', 'prod2']" @@ -114,8 +116,9 @@ This can be resolved by using the ``map`` filter to apply the ``replace`` filter - debug: msg: "{{ ['test1', 'test2'] | map('replace', 'test', 'prod') }}" +The result of the corrected template remains a list: -The result of the corrected template remains a list:: +.. code-block:: console ok: [localhost] => { "msg": [ From a5ab4f93ec10f22c2a109aae408904ae40df7a5c Mon Sep 17 00:00:00 2001 From: Matt Davis Date: Tue, 1 Apr 2025 15:47:09 -0700 Subject: [PATCH 07/15] add conditional syntax error example --- .../porting_guide_core_2.19.rst | 20 +++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst b/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst index f33321edc59..36923040418 100644 --- a/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst +++ b/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst @@ -334,6 +334,26 @@ This can be resolved by removing the unnecessary quotes: that: inventory_hostname is defined and inventory_hostname | length > 0 +Example - Expression Syntax Error +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Previous Ansible releases could mask some expression syntax errors as a truthy result. + +.. code-block:: yaml+jinja + + - assert: + that: 1 == 2, + # ^ invalid comma + + +The error reported is:: + + Syntax error in expression: chunk after expression + + +This can be resolved by removing the invalid comma after the expression. + + Example - Jinja Order of Operations ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ From 52fce52b153ee48bc8545b677bdd52caaa911d38 Mon Sep 17 00:00:00 2001 From: Matt Davis <6775756+nitzmahone@users.noreply.github.com> Date: Tue, 15 Apr 2025 15:17:06 -0700 Subject: [PATCH 08/15] Update docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst Co-authored-by: Maxwell G --- docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst b/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst index 36923040418..ad45828202e 100644 --- a/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst +++ b/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst @@ -35,7 +35,7 @@ Template Trust Model Inversion ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Previously, ansible-core implicitly trusted all string values to be rendered as Jinja templates, but applied an "unsafe" -wrapper object around strings obtained from untrusted sources (eg, module results). +wrapper object around strings obtained from untrusted sources (for example, module results). Unsafe-wrapped strings were silently ignored by the template engine, as many templating operations can execute arbitrary code on the control host as the user running ansible-core. This required any code that operated on strings to correctly propagate the wrapper object, which resulted in numerous From db244d65bea6602426f731e53735a58935c661ea Mon Sep 17 00:00:00 2001 From: Matt Davis <6775756+nitzmahone@users.noreply.github.com> Date: Tue, 15 Apr 2025 15:20:03 -0700 Subject: [PATCH 09/15] Update docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst Co-authored-by: Maxwell G --- docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst b/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst index ad45828202e..58994912000 100644 --- a/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst +++ b/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst @@ -155,7 +155,7 @@ involved are captured and included in user-facing messages. Errors and warnings that occur during task execution are more consistently included in the task result, with the full details accessible to callbacks and (in the case of errors), a minimal error message in the ``msg`` field of the result. Due to the standardized nature of this error handling, seemingly redundant elements may appear in some error messages. -These will improve over time as other error handling improvements are made, but are currently necessary to ensure proper +These will improve over time as other error handling improvements are made but are currently necessary to ensure proper context is available in all error situations. Error message contents are not considered stable, so automation that relies on them should be avoided when possible. From 042ec2fe16a3c9d90563fed6409ebb88ab26f89b Mon Sep 17 00:00:00 2001 From: Matt Davis <6775756+nitzmahone@users.noreply.github.com> Date: Tue, 15 Apr 2025 15:20:12 -0700 Subject: [PATCH 10/15] Update docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst Co-authored-by: Maxwell G --- docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst b/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst index 58994912000..f1ba58a8c88 100644 --- a/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst +++ b/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst @@ -167,7 +167,7 @@ The new Data Tagging feature expands provenance tracking on variables to nearly This allows for much more descriptive error messaging, as the entire chain of execution can be consulted to include contextual information about what was happening when an error occurred- in many cases including display of the offending source lines with column markers. -Limited support for non-file provenance-tracking is included (e.g., variables from CLI arguments, inventory plugins, +Limited support for non-file provenance-tracking is included (for example, variables from CLI arguments, inventory plugins, environment variables) and will likely be expanded over time. From b4c338334497cb2e08da96f652f82f07e596acde Mon Sep 17 00:00:00 2001 From: Matt Davis <6775756+nitzmahone@users.noreply.github.com> Date: Tue, 15 Apr 2025 15:25:42 -0700 Subject: [PATCH 11/15] Update docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst Co-authored-by: Maxwell G --- docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst b/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst index f1ba58a8c88..f4b6ef1a1da 100644 --- a/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst +++ b/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst @@ -176,8 +176,8 @@ Deprecation Warnings on Value Access New features allow most ansible-core variables and values to be tagged as deprecated. Plugins and modules can apply these tags to augment deprecated elements of their return values with a description and -help text to suggest alternatives, which will be displayed in a runtime warning when the tagged value is accessed by -e.g., a playbook or template. +help text to suggest alternatives, which will be displayed in a runtime warning when the tagged value is accessed by, +for example, a playbook or template. This allows for easier evolution and removal of module/fact results and obsolete core behaviors. A specific example included in this release is the deprecation of the legacy ``INJECT_FACTS_AS_VARS`` behavior, which injects fact values directly into the top-level variable namespace. From fa4973a125bf3f28077fa881cfc805d24aa1909f Mon Sep 17 00:00:00 2001 From: Matt Davis Date: Tue, 15 Apr 2025 17:18:24 -0700 Subject: [PATCH 12/15] Refresh DT porting guide for 2.19.0b1 * misc reorder, wordsmith * address review feedback --- .../porting_guide_core_2.19.rst | 546 +++++++++--------- 1 file changed, 278 insertions(+), 268 deletions(-) diff --git a/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst b/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst index f4b6ef1a1da..3b7da873b42 100644 --- a/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst +++ b/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst @@ -7,270 +7,32 @@ Ansible-core 2.19 Porting Guide This section discusses the behavioral changes between ansible-core 2.18 and ansible-core 2.19. -It is intended to assist in updating your playbooks, plugins and other parts of your Ansible infrastructure so they will -work with this version of Ansible. +It is intended to assist in updating your playbooks, plugins, +and other parts of your Ansible infrastructure so they will work with this version of Ansible. -We suggest you read this page along with +Review this page and the `ansible-core Changelog for 2.19 `_ -to understand what updates you may need to make. +to understand necessary changes. This document is part of a collection on porting. The complete list of porting guides can be found at :ref:`porting guides `. .. contents:: Topics +Introduction +============ -Engine -====== +This release includes an overhaul of the templating system and a new feature dubbed Data Tagging. +These changes enable reporting of numerous problematic behaviors that went undetected in previous releases, +with wide-ranging positive effects on security, performance, and user experience. -This release includes an overhaul of the templating system along with a new feature dubbed Data Tagging. -These changes have wide-ranging positive effects on security, performance, and user experience. Backward compatibility has been preserved where practical, but some breaking changes were necessary. - - -Templating ----------- - -Template Trust Model Inversion -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Previously, ansible-core implicitly trusted all string values to be rendered as Jinja templates, but applied an "unsafe" -wrapper object around strings obtained from untrusted sources (for example, module results). -Unsafe-wrapped strings were silently ignored by the template engine, as many templating operations can execute arbitrary -code on the control host as the user running ansible-core. -This required any code that operated on strings to correctly propagate the wrapper object, which resulted in numerous -CVE-worthy RCE (remote code execution) vulnerabilities. - -This release inverts the previous trust model. -Only strings marked as loaded from a trusted source are eligible to be rendered as templates. -Untrusted values can (as before) be referenced by templates, but the template expression itself must always be trusted. -While this change still requires consideration for propagation of trust markers when manipulating strings, failure to do -so now results in a loss of templating ability instead of a potentially high-severity security issue. - -Attempts to render a template appearing in an untrusted string will (as before) return the original string unmodified. -By default, attempting to render an untrusted template fails silently, though such failures can be elevated to a -warning or error via configuration. - -Newly-created string results from template operations will never have trust automatically applied, though templates that -return existing trusted string values unmodified will not strip their trust. -It is also possible for plugins to explicitly apply trust. - -Backward-compatible template trust behavior is applied automatically in most cases; e.g., templates appearing in -playbooks, roles, variable files, and most built-in inventory plugins will yield trusted template strings. -Custom plugins that source template strings will be required to use new public APIs to apply trust where appropriate, -or declaratively request automatic trust application for templates to execute properly (see `Plugin API`_ for additional -information). - -See `Troubleshooting Untrusted Templates`_ for additional information. - - -Native Jinja Mode Required -^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Previous versions supported templating in two different modes: - -The first, Jinja's original string templating mode, converted the result of each templating operation to a string. - -The second, Jinja's native mode, attempted to preserve variable types while templating, but could not always do so. - -In both modes, ansible-core evaluated template string results as Python literals, falling back to the original string. -Selection of the templating mode was controlled by configuration, defaulting to Jinja's original string templating. - -Starting with this release, use of Jinja's native templating is required. -The configuration option for setting the templating mode is deprecated and no longer has any effect. - -Preservation of native types in templating has been improved to correct gaps in the previous implementation, -entirely eliminating the final literal evaluation pass (a frequent source of confusion, errors, and performance issues). -In rare cases where playbooks relied on implicit object conversion from strings, an explicit conversion will be -required. - -Some existing templates may unintentionally convert non-strings to strings. -In previous versions this was sometimes masked by the evaluation of strings as Python literals. - -Example - Unintentional String Conversion -""""""""""""""""""""""""""""""""""""""""" - -This expression erroneously passes a list to the ``replace`` filter, which operates only on strings. -The filter silently converts the list input to a string. -Due to some string results previously parsing as lists, this mistake often went undetected in earlier versions. - -.. code-block:: yaml+jinja - - - debug: - msg: "{{ ['test1', 'test2'] | replace('test', 'prod') }}" - - -The result of this template becomes a string: - -.. code-block:: console - - ok: [localhost] => { - "msg": "['prod1', 'prod2']" - } - - -This can be resolved by using the ``map`` filter to apply the ``replace`` filter to each list element: - -.. code-block:: yaml+jinja - - - debug: - msg: "{{ ['test1', 'test2'] | map('replace', 'test', 'prod') }}" - -The result of the corrected template remains a list: - -.. code-block:: console - - ok: [localhost] => { - "msg": [ - "prod1", - "prod2" - ] - } - - -Lazy Templating -^^^^^^^^^^^^^^^ - -Ansible's interface with the Jinja templating engine has been heavily refined, yielding significant performance -improvements for many complex templating operations. -Previously, deeply-nested, recursive, or self-referential templating operations were always resolved to their full depth -and breadth on every access, including repeated access to the same data within a single templating operation. -This resulted in expensive and repetitive evaluation of the same templates within a single logical template operation, -even for templates deep inside nested data structures that were never directly accessed. -The new template engine lazily defers nearly all recursion and templating until values are accessed or known to be -exiting the template engine, and intermediate nested or indirected templated results are cached for the duration of the -template operation, reducing repetitive templating. -These changes have shown exponential performance improvements for many real-world complex templating scenarios. - - -Error Handling --------------- - -Contextual Warnings and Errors -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Changes to internal error handling in ansible-core will be visible in many situations that result in a warning or error. -In most cases, the operational context (what was happening when the error or warning was generated) and data element(s) -involved are captured and included in user-facing messages. -Errors and warnings that occur during task execution are more consistently included in the task result, with the full -details accessible to callbacks and (in the case of errors), a minimal error message in the ``msg`` field of the result. -Due to the standardized nature of this error handling, seemingly redundant elements may appear in some error messages. -These will improve over time as other error handling improvements are made but are currently necessary to ensure proper -context is available in all error situations. -Error message contents are not considered stable, so automation that relies on them should be avoided when possible. - - -Variable Provenance Tracking -^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -The new Data Tagging feature expands provenance tracking on variables to nearly every source. -This allows for much more descriptive error messaging, as the entire chain of execution can be consulted to include -contextual information about what was happening when an error occurred- in many cases including display of the offending -source lines with column markers. -Limited support for non-file provenance-tracking is included (for example, variables from CLI arguments, inventory plugins, -environment variables) and will likely be expanded over time. - - -Deprecation Warnings on Value Access -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -New features allow most ansible-core variables and values to be tagged as deprecated. -Plugins and modules can apply these tags to augment deprecated elements of their return values with a description and -help text to suggest alternatives, which will be displayed in a runtime warning when the tagged value is accessed by, -for example, a playbook or template. -This allows for easier evolution and removal of module/fact results and obsolete core behaviors. -A specific example included in this release is the deprecation of the legacy ``INJECT_FACTS_AS_VARS`` behavior, which -injects fact values directly into the top-level variable namespace. -The injection behavior is potentially dangerous, and the configuration to disable it has existed for many Ansible -releases, but deprecation and an eventual default behavior change were not possible until the current release, since -warnings on access could not be issued reliably. - - -Improved Ansible Module Error Handling -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Ansible modules implemented in Python now have exception handling provided by the AnsiballZ wrapper. -In previous versions of ansible-core, unhandled exceptions in an Ansible module simply printed a traceback and exited -without providing a standard module response, which caused the task result to contain a generic ``MODULE FAILURE`` -message and any raw output text produced by the module. - -To address this, modules often implemented unnecessary ``try/except`` blocks around most code where specific error -handling was not possible, only to call ``AnsibleModule.fail_json`` with a generic failure message. -This pattern is no longer necessary, as all unhandled exceptions in Ansible Python modules are now captured by the -AnsiballZ wrapper and returned as a structured module result, including the automatic inclusion of traceback information -(when enabled by the controller). - - -Improved Handling of Undefined -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Undefined handling has been improved to avoid situations where a Jinja plugin silently ignores undefined values. - -This commonly occurs when a Jinja plugin, such as a filter or test, checks the type of variables without accounting for -the possibility of undefined values being present. - -Example - Missing Attribute -""""""""""""""""""""""""""" - -This task incorrectly references a non-existent ``exists`` attribute from a ``stat`` result in a conditional. -The undefined value goes undetected in previous versions because it is passed to the ``false`` Jinja test plugin, -which silently ignores undefined values. -As a result, this conditional could never be ``True`` in earlier versions of ansible-core, -and there was no indication that the ``failed_when`` expression was invalid. - -.. code-block:: yaml+jinja - - - stat: - path: /does-not-exist - register: result - failed_when: result.exists is false - -In the current release the faulty expression is detected and results in an error. - -This can be corrected by adding the missing ``stat`` attribute to the conditional: - -.. code-block:: yaml+jinja - - - stat: - path: /does-not-exist - register: result - failed_when: result.stat.exists is false - - -Displaying Tracebacks -^^^^^^^^^^^^^^^^^^^^^ - -In previous ansible-core versions, tracebacks from some controller-side errors were available by increasing verbosity -with the ``-vvv`` option, but the availability and behavior was inconsistent. -This feature was also limited to errors. - -Handling of errors, warnings and deprecations throughout much of the ansible-core codebase has now been standardized. -Tracebacks can be optionally collected and displayed for all exceptions, as well as at the call site of errors, -warnings, or deprecations (even in module code) using the ``ANSIBLE_DISPLAY_TRACEBACK`` environment variable. - -Valid options are: - -* ``always`` - Tracebacks will always be displayed. This option takes precedence over others below. -* ``never`` - Tracebacks will never be displayed. This option takes precedence over others below. -* ``error`` - Tracebacks will be displayed for errors. -* ``warning`` - Tracebacks will be displayed for warnings other than deprecation warnings. -* ``deprecated`` - Tracebacks will be displayed for deprecation warnings. - -Multiple options can be combined by separating them with commas. -A CLI option for this setting is under consideration. +This guide describes some common problem scenarios with example content, error messsages, and suggested solutions. Playbook ======== - -Task Error Handling -------------------- - -Timeout waiting on privilege escalation (``become``) is now an unreachable error instead of a task error. -Existing playbooks should be changed to replace ``ignore_errors`` with ``ignore_unreachable`` on tasks where timeout on ``become`` should be ignored. - - Broken Conditionals ------------------- @@ -313,7 +75,7 @@ Example - Unintentional Truthy Conditional ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The second part of this conditional is erroneously quoted. -The quoted part becomes the expression result considered ``True``, so the expression will never be ``False``. +The quoted part becomes the expression result (evaluated as truthy), so the expression can never be ``False``. .. code-block:: yaml+jinja @@ -326,7 +88,7 @@ The error reported is:: Conditional result was 'inventory_hostname | length > 0' of type 'str', which evaluates to True. Conditionals must have a boolean result. -This can be resolved by removing the unnecessary quotes: +This can be resolved by removing the erroneous quotes: .. code-block:: yaml+jinja @@ -357,8 +119,8 @@ This can be resolved by removing the invalid comma after the expression. Example - Jinja Order of Operations ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -This expression uses the ``~`` concatenation operator, which is evaluated after the contains test. -The result is always a non-empty string, which evaluates to ``True``. +This expression uses the ``~`` concatenation operator, which is evaluated after the ``contains`` test. +The result is always a non-empty string, which is truthy. .. code-block:: yaml+jinja @@ -371,7 +133,7 @@ The error reported is:: Conditional result was 'Truehost' of type 'str', which evaluates to True. Conditionals must have a boolean result. -This can be resolved by inserting parenthesis to resolve the concatenation operation before the ``contains`` test: +This can be resolved by inserting parentheses to resolve the concatenation operation before the ``contains`` test: .. code-block:: yaml+jinja @@ -383,8 +145,8 @@ Example - Dictionary as Conditional ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This conditional should have been quoted. -In a list, a space after the colon means the YAML parser sees the value as a mapping, not a string. -Any non-empty mapping will always be ``True``. +In a YAML list element, an unquoted string with a space after a colon is interpreted by the YAML parser as a mapping. +Non-empty mappings are always truthy. .. code-block:: yaml+jinja @@ -403,19 +165,21 @@ This can be resolved by quoting the entire assertion expression: .. code-block:: yaml+jinja - assert: + that: - 'result.msg == "some_key: some_value"' Multi-pass Templating --------------------- -Embedding templates within other templates or expressions can result in untrusted templates being executed. +Embedding templates within other templates or expressions could previously result in untrusted templates being executed. The overhauled templating engine in this release no longer supports this insecure behavior. + Example - Unnecessary Template in Expression ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -This conditional references a variable using a template, instead of using the variable directly in the expression. +This conditional references a variable using a template instead of using the variable directly in the expression. .. code-block:: yaml+jinja @@ -444,8 +208,8 @@ Example - Dynamic Expression Construction ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This conditional is dynamically created using a template, which is expected to be evaluated as an expression. -Previously, the template was rendered by task argument templating, resulting in a plain string, which was later -evaluated by the ``assert`` action. +Previously, the template was rendered by task argument templating, resulting in a plain string, +which was later evaluated by the ``assert`` action. .. code-block:: yaml+jinja @@ -460,11 +224,14 @@ The error reported is:: Syntax error in expression. Template delimiters are not supported in expressions: chunk after expression +Dynamic expression construction from playbooks is insecure and unsupported. + + Troubleshooting Untrusted Templates ----------------------------------- By default, untrusted templates are silently ignored. -To troubleshoot trust issues with templates, it can be helpful to trigger a warning or error for untrusted templates. +Troubleshooting trust issues with templates can be aided by enabling warnings or errors for untrusted templates. The environment variable ``_ANSIBLE_TEMPLAR_UNTRUSTED_TEMPLATE_BEHAVIOR`` can be used to control this behavior. Valid options are: @@ -477,6 +244,250 @@ Valid options are: This optional warning and failure behavior is experimental and subject to change in future versions. +Privilege Escalation Timeouts +----------------------------- + +Timeout waiting on privilege escalation (``become``) is now an unreachable error instead of a task error. +Existing playbooks should be changed to replace ``ignore_errors`` with ``ignore_unreachable`` on tasks where +timeout on ``become`` should be ignored. + + +Engine +====== + +Templating +---------- + +Template Trust Model Inversion +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Previously, ansible-core implicitly trusted all string values to be rendered as Jinja templates, +but applied an "unsafe" wrapper object around strings obtained from untrusted sources (for example, module results). +Unsafe-wrapped strings were silently ignored by the template engine, +as many templating operations can execute arbitrary code on the control host as the user running ansible-core. +This required any code that operated on strings to correctly propagate the wrapper object, +which resulted in numerous CVE-worthy RCE (remote code execution) vulnerabilities. + +This release inverts the previous trust model. +Only strings marked as loaded from a trusted source are eligible to be rendered as templates. +Untrusted values can (as before) be referenced by templates, but the template expression itself must always be trusted. +While this change still requires consideration for propagation of trust markers when manipulating strings, +failure to do so now results in a loss of templating ability instead of a potentially high-severity security issue. + +Attempts to render a template appearing in an untrusted string will (as before) return the original string unmodified. +By default, attempting to render an untrusted template fails silently, +though such failures can be elevated to a warning or error via configuration. + +Newly-created string results from template operations will never have trust automatically applied, +though templates that return existing trusted string values unmodified will not strip their trust. +It is also possible for plugins to explicitly apply trust. + +Backward-compatible template trust behavior is applied automatically in most cases; +for example, templates appearing in playbooks, roles, variable files, +and most built-in inventory plugins will yield trusted template strings. +Custom plugins that source template strings will be required to use new public APIs to apply trust where appropriate. + +See `Plugin API`_ and `Troubleshooting Untrusted Templates`_ for additional information. + + +Native Jinja Mode Required +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Previous versions supported templating in two different modes: + +* Jinja's original string templating mode converted the result of each templating operation to a string. +* Jinja's native mode *usually* preserved variable types in template results. + +In both modes, ansible-core evaluated the final template string results as Python literals, falling back to the +original string if the evaluation resulted in an error. +Selection of the templating mode was controlled by configuration, defaulting to Jinja's original string templating. + +Jinja's native templating mode is now used exclusively. +The configuration option for setting the templating mode is deprecated and no longer has any effect. + +Preservation of native types in templating has been improved to correct gaps in the previous implementation, +entirely eliminating the final literal evaluation pass (a frequent source of confusion, errors, and performance issues). +In rare cases where playbooks relied on implicit object conversion from strings, +an explicit conversion will be required. + +Some existing templates may unintentionally convert non-strings to strings. +In previous versions this conversion could be masked by the evaluation of strings as Python literals. + + +Example - Unintentional String Conversion +""""""""""""""""""""""""""""""""""""""""" + +This expression erroneously passes a list to the ``replace`` filter, which operates only on strings. +The filter silently converts the list input to a string. +Due to some string results previously parsing as lists, this mistake often went undetected in earlier versions. + +.. code-block:: yaml+jinja + + - debug: + msg: "{{ ['test1', 'test2'] | replace('test', 'prod') }}" + + +The result of this template becomes a string: + +.. code-block:: console + + ok: [localhost] => { + "msg": "['prod1', 'prod2']" + } + + +This can be resolved by using the ``map`` filter to apply the ``replace`` filter to each list element: + +.. code-block:: yaml+jinja + + - debug: + msg: "{{ ['test1', 'test2'] | map('replace', 'test', 'prod') }}" + + +The result of the corrected template remains a list: + +.. code-block:: console + + ok: [localhost] => { + "msg": [ + "prod1", + "prod2" + ] + } + + +Lazy Templating +^^^^^^^^^^^^^^^ + +Ansible's interface with the Jinja templating engine has been heavily refined, +yielding significant performance improvements for many complex templating operations. +Previously, deeply-nested, recursive, +or self-referential templating operations were always resolved to their full depth and breadth on every access, +including repeated access to the same data within a single templating operation. +This resulted in expensive and repetitive evaluation of the same templates within a single logical template operation, +even for templates deep inside nested data structures that were never directly accessed. +The new template engine lazily defers nearly all recursion and templating until values are accessed, +or known to be exiting the template engine, +and intermediate nested or indirected templated results are cached for the duration of the template operation, +reducing repetitive templating. +These changes have shown exponential performance improvements for many real-world complex templating scenarios. + + +Error Handling +-------------- + +Contextual Warnings and Errors +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Changes to internal error handling in ansible-core will be visible in many situations that result in a warning or error. +In most cases, the operational context (what was happening when the error or warning was generated) +and data element(s) involved are captured and included in user-facing messages. +Errors and warnings that occur during task execution are more consistently included in the task result, with the full +details accessible to callbacks and (in the case of errors), a minimal error message in the ``msg`` field of the result. +Due to the standardized nature of this error handling, seemingly redundant elements may appear in some error messages. +These will improve over time as other error handling improvements are made but are currently necessary to ensure proper +context is available in all error situations. +Error message contents are not considered stable, so automation that relies on them should be avoided when possible. + + +Variable Provenance Tracking +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The new Data Tagging feature expands provenance tracking on variables to nearly every source. +This allows for much more descriptive error messaging, as the entire chain of execution can be consulted to include +contextual information about what was happening when an error occurred. +In most cases, this includes file path, source lines, and column markers. +Non-file variable sources such as CLI arguments, inventory plugins and environment are also supported. + + +Deprecation Warnings on Value Access +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +New features allow most ansible-core variables and values to be tagged as deprecated. +Plugins and modules can apply these tags to augment deprecated elements of their return values with a description and +help text to suggest alternatives, which will be displayed in a runtime warning when the tagged value is accessed by, +for example, a playbook or template. +This allows for easier evolution and removal of module and fact results, and obsolete core behaviors. + +For example, accessing the deprecated ``play_hosts`` magic variable will trigger a deprecation warning that suggests +the use of the ``ansible_play_batch`` variable instead. + + +Improved Ansible Module Error Handling +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Ansible modules implemented in Python now have exception handling provided by the AnsiballZ wrapper. +In previous versions of ansible-core, unhandled exceptions in an Ansible module simply printed a traceback and exited +without providing a standard module response, which caused the task result to contain a generic ``MODULE FAILURE`` +message and any raw output text produced by the module. + +To address this, modules often implemented unnecessary ``try/except`` blocks around most code where specific error +handling was not possible, only to call ``AnsibleModule.fail_json`` with a generic failure message. +This pattern is no longer necessary, as all unhandled exceptions in Ansible Python modules are now captured by the +AnsiballZ wrapper and returned as a structured module result, +with automatic inclusion of traceback information when enabled by the controller. + + +Improved Handling of Undefined +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Undefined handling has been improved to avoid situations where a Jinja plugin silently ignores undefined values. + +This commonly occurs when a Jinja plugin, such as a filter or test, +checks the type of a variable without accounting for the possibility of an undefined value being present. + + +Example - Missing Attribute +""""""""""""""""""""""""""" + +This task incorrectly references an undefined ``exists`` attribute from a ``stat`` result in a conditional. +The undefined value was not detected in previous versions because it is passed to the ``false`` Jinja test plugin, +which silently ignores undefined values. +As a result, this conditional could never be ``True`` in earlier versions of ansible-core, +and there was no indication that the ``failed_when`` expression was invalid. + +.. code-block:: yaml+jinja + + - stat: + path: /does-not-exist + register: result + failed_when: result.exists is false + # ^ missing reference to stat + +In the current release the faulty expression is detected and results in an error. + +This can be corrected by adding the missing ``stat`` attribute to the conditional: + +.. code-block:: yaml+jinja + + - stat: + path: /does-not-exist + register: result + failed_when: result.stat.exists is false + + +Displaying Tracebacks +^^^^^^^^^^^^^^^^^^^^^ + +In previous ansible-core versions, tracebacks from some controller-side errors were available by increasing verbosity +with the ``-vvv`` option, but the availability and behavior was inconsistent. +This feature was also limited to errors. + +Handling of errors, warnings and deprecations throughout much of the ansible-core codebase has now been standardized. +Tracebacks can be optionally collected and displayed for all exceptions, as well as at the call site of errors, +warnings, or deprecations (even in module code) using the ``ANSIBLE_DISPLAY_TRACEBACK`` environment variable. + +Valid options are: + +* ``always`` - Tracebacks will always be displayed. This option takes precedence over others below. +* ``never`` - Tracebacks will never be displayed. This option takes precedence over others below. +* ``error`` - Tracebacks will be displayed for errors. +* ``warning`` - Tracebacks will be displayed for warnings other than deprecation warnings. +* ``deprecated`` - Tracebacks will be displayed for deprecation warnings. + +Multiple options can be combined by separating them with commas. + + Plugin API ========== @@ -485,8 +496,8 @@ Deprecating Values Plugins and Python modules can tag returned values as deprecated with the new ``deprecate_value`` function from ``ansible.module_utils.datatag``. -A description of the deprecated feature, optional help text, and removal timeframes can be attached to the value, which -will appear in a runtime warning if the deprecated value is referenced in an expression. +A description of the deprecated feature, optional help text, and removal timeframes can be attached to the value, +which will appear in a runtime warning if the deprecated value is referenced in an expression. The warning message will include information about the module/plugin that applied the deprecation tag and the location of the expression that accessed it. @@ -524,10 +535,9 @@ Applying Template Trust to Individual Values String values are no longer trusted to be rendered as templates by default. Strings loaded from playbooks, vars files, and other built-in trusted sources are usually marked trusted by default. -Plugins that create new string instances with embedded templates must use the new ``trust_value`` function from -``ansible.module_utils.datatag`` to tag those values as originating from a trusted source to allow the templates to be -rendered. - +Plugins that create new string instances with embedded templates must use the new ``trust_as_template`` function +from ``ansible.template`` to tag those values as originating from a trusted source to allow the templates +to be rendered. .. warning:: This section and the associated public API are currently incomplete. @@ -538,8 +548,8 @@ Applying Template Trust in Inventory and Vars Plugins Inventory plugins can set group and host variables. In most cases, these variables are static values from external sources and do not require trust. -Values that can contain templates will require explicit trust via ``trust_value`` to be allowed to render, but -trust should not be applied to variable values from external sources that could be maliciously altered to include +Values that can contain templates will require explicit trust via ``trust_as_template`` to be allowed to render, +but trust should not be applied to variable values from external sources that could be maliciously altered to include templates. .. warning:: From f7177883cc2a1c18ab4c09b40aab4d7fe663d250 Mon Sep 17 00:00:00 2001 From: Matt Clay Date: Wed, 16 Apr 2025 09:18:48 -0700 Subject: [PATCH 13/15] Apply suggestions from code review Update capitalization and quoting. Co-authored-by: Sandra McCann --- .../porting_guide_core_2.19.rst | 62 +++++++++---------- 1 file changed, 31 insertions(+), 31 deletions(-) diff --git a/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst b/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst index 3b7da873b42..13e55366cf7 100644 --- a/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst +++ b/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst @@ -39,15 +39,15 @@ Broken Conditionals Broken conditionals occur when the input expression or template is not a string, or the result is not a boolean. Python and Jinja provide implicit "truthy" evaluation of most non-empty non-boolean values in conditional expressions. While sometimes desirable for brevity, truthy conditional evaluation often masks serious logic errors in playbooks that -could not be reliably detected by previous versions of ansible-core. +could not be reliably detected by previous versions of ``ansible-core``. Changes to templating in this release detects non-boolean conditionals during expression evaluation and reports an error -by default. The error can be temporarily reduced to a warning via the ``ALLOW_BROKEN_CONDITIONALS`` config setting. +by default. The error can be temporarily reduced to a warning with the ``ALLOW_BROKEN_CONDITIONALS`` config setting. The following examples are derived from broken conditionals that masked logic errors in actual playbooks. -Example - Implicit Boolean Conversion +Example - implicit boolean conversion ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This expression relies on an implicit truthy evaluation of ``inventory_hostname``. @@ -71,7 +71,7 @@ This can be resolved by using an explicit boolean conversion: that: inventory_hostname | length > 0 -Example - Unintentional Truthy Conditional +Example - unintentional truthy conditional ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The second part of this conditional is erroneously quoted. @@ -96,7 +96,7 @@ This can be resolved by removing the erroneous quotes: that: inventory_hostname is defined and inventory_hostname | length > 0 -Example - Expression Syntax Error +Example - expression syntax error ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Previous Ansible releases could mask some expression syntax errors as a truthy result. @@ -141,7 +141,7 @@ This can be resolved by inserting parentheses to resolve the concatenation opera that: inventory_hostname is contains("local" ~ "host") -Example - Dictionary as Conditional +Example - dictionary as conditional ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This conditional should have been quoted. @@ -169,14 +169,14 @@ This can be resolved by quoting the entire assertion expression: - 'result.msg == "some_key: some_value"' -Multi-pass Templating +Multi-pass templating --------------------- Embedding templates within other templates or expressions could previously result in untrusted templates being executed. The overhauled templating engine in this release no longer supports this insecure behavior. -Example - Unnecessary Template in Expression +Example - unnecessary template in expression ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This conditional references a variable using a template instead of using the variable directly in the expression. @@ -204,7 +204,7 @@ This can be resolved by referencing the variable without a template: value: 1 -Example - Dynamic Expression Construction +Example - dynamic expression construction ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This conditional is dynamically created using a template, which is expected to be evaluated as an expression. @@ -244,7 +244,7 @@ Valid options are: This optional warning and failure behavior is experimental and subject to change in future versions. -Privilege Escalation Timeouts +Privilege escalation timeouts ----------------------------- Timeout waiting on privilege escalation (``become``) is now an unreachable error instead of a task error. @@ -258,10 +258,10 @@ Engine Templating ---------- -Template Trust Model Inversion +Template trust model inversion ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Previously, ansible-core implicitly trusted all string values to be rendered as Jinja templates, +Previously, ``ansible-core`` implicitly trusted all string values to be rendered as Jinja templates, but applied an "unsafe" wrapper object around strings obtained from untrusted sources (for example, module results). Unsafe-wrapped strings were silently ignored by the template engine, as many templating operations can execute arbitrary code on the control host as the user running ansible-core. @@ -298,7 +298,7 @@ Previous versions supported templating in two different modes: * Jinja's original string templating mode converted the result of each templating operation to a string. * Jinja's native mode *usually* preserved variable types in template results. -In both modes, ansible-core evaluated the final template string results as Python literals, falling back to the +In both modes, ``ansible-core`` evaluated the final template string results as Python literals, falling back to the original string if the evaluation resulted in an error. Selection of the templating mode was controlled by configuration, defaulting to Jinja's original string templating. @@ -314,7 +314,7 @@ Some existing templates may unintentionally convert non-strings to strings. In previous versions this conversion could be masked by the evaluation of strings as Python literals. -Example - Unintentional String Conversion +Example - unintentional string conversion """"""""""""""""""""""""""""""""""""""""" This expression erroneously passes a list to the ``replace`` filter, which operates only on strings. @@ -356,7 +356,7 @@ The result of the corrected template remains a list: } -Lazy Templating +Lazy templating ^^^^^^^^^^^^^^^ Ansible's interface with the Jinja templating engine has been heavily refined, @@ -373,13 +373,13 @@ reducing repetitive templating. These changes have shown exponential performance improvements for many real-world complex templating scenarios. -Error Handling +Error handling -------------- -Contextual Warnings and Errors +Contextual warnings and errors ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Changes to internal error handling in ansible-core will be visible in many situations that result in a warning or error. +Changes to internal error handling in ``ansible-core`` will be visible in many situations that result in a warning or error. In most cases, the operational context (what was happening when the error or warning was generated) and data element(s) involved are captured and included in user-facing messages. Errors and warnings that occur during task execution are more consistently included in the task result, with the full @@ -390,7 +390,7 @@ context is available in all error situations. Error message contents are not considered stable, so automation that relies on them should be avoided when possible. -Variable Provenance Tracking +Variable provenance tracking ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The new Data Tagging feature expands provenance tracking on variables to nearly every source. @@ -400,10 +400,10 @@ In most cases, this includes file path, source lines, and column markers. Non-file variable sources such as CLI arguments, inventory plugins and environment are also supported. -Deprecation Warnings on Value Access +Deprecation warnings on value access ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -New features allow most ansible-core variables and values to be tagged as deprecated. +New features allow most ``ansible-core`` variables and values to be tagged as deprecated. Plugins and modules can apply these tags to augment deprecated elements of their return values with a description and help text to suggest alternatives, which will be displayed in a runtime warning when the tagged value is accessed by, for example, a playbook or template. @@ -413,11 +413,11 @@ For example, accessing the deprecated ``play_hosts`` magic variable will trigger the use of the ``ansible_play_batch`` variable instead. -Improved Ansible Module Error Handling +Improved Ansible module error handling ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Ansible modules implemented in Python now have exception handling provided by the AnsiballZ wrapper. -In previous versions of ansible-core, unhandled exceptions in an Ansible module simply printed a traceback and exited +In previous versions of ``ansible-core``, unhandled exceptions in an Ansible module simply printed a traceback and exited without providing a standard module response, which caused the task result to contain a generic ``MODULE FAILURE`` message and any raw output text produced by the module. @@ -428,7 +428,7 @@ AnsiballZ wrapper and returned as a structured module result, with automatic inclusion of traceback information when enabled by the controller. -Improved Handling of Undefined +Improved handling of undefined ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Undefined handling has been improved to avoid situations where a Jinja plugin silently ignores undefined values. @@ -437,7 +437,7 @@ This commonly occurs when a Jinja plugin, such as a filter or test, checks the type of a variable without accounting for the possibility of an undefined value being present. -Example - Missing Attribute +Example - missing attribute """"""""""""""""""""""""""" This task incorrectly references an undefined ``exists`` attribute from a ``stat`` result in a conditional. @@ -466,14 +466,14 @@ This can be corrected by adding the missing ``stat`` attribute to the conditiona failed_when: result.stat.exists is false -Displaying Tracebacks +Displaying tracebacks ^^^^^^^^^^^^^^^^^^^^^ -In previous ansible-core versions, tracebacks from some controller-side errors were available by increasing verbosity +In previous ``ansible-core`` versions, tracebacks from some controller-side errors were available by increasing verbosity with the ``-vvv`` option, but the availability and behavior was inconsistent. This feature was also limited to errors. -Handling of errors, warnings and deprecations throughout much of the ansible-core codebase has now been standardized. +Handling of errors, warnings and deprecations throughout much of the ``ansible-core`` codebase has now been standardized. Tracebacks can be optionally collected and displayed for all exceptions, as well as at the call site of errors, warnings, or deprecations (even in module code) using the ``ANSIBLE_DISPLAY_TRACEBACK`` environment variable. @@ -491,7 +491,7 @@ Multiple options can be combined by separating them with commas. Plugin API ========== -Deprecating Values +Deprecating values ------------------ Plugins and Python modules can tag returned values as deprecated with the new ``deprecate_value`` function from @@ -530,7 +530,7 @@ When accessing the `color_name` from the module result, the following warning wi Use `color_code` instead. -Applying Template Trust to Individual Values +Applying template trust to individual values -------------------------------------------- String values are no longer trusted to be rendered as templates by default. Strings loaded from playbooks, vars files, @@ -543,7 +543,7 @@ to be rendered. This section and the associated public API are currently incomplete. -Applying Template Trust in Inventory and Vars Plugins +Applying template trust in inventory and vars plugins ----------------------------------------------------- Inventory plugins can set group and host variables. From d9080b509e9b05a19827042e607d77f550d93982 Mon Sep 17 00:00:00 2001 From: Matt Clay Date: Wed, 16 Apr 2025 09:30:47 -0700 Subject: [PATCH 14/15] Apply suggestions from code review Co-authored-by: Sandra McCann --- .../rst/porting_guides/porting_guide_core_2.19.rst | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst b/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst index 13e55366cf7..b55347f83a2 100644 --- a/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst +++ b/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst @@ -5,7 +5,7 @@ Ansible-core 2.19 Porting Guide ******************************* -This section discusses the behavioral changes between ansible-core 2.18 and ansible-core 2.19. +This section discusses the behavioral changes between ``ansible-core`` 2.18 and ``ansible-core`` 2.19. It is intended to assist in updating your playbooks, plugins, and other parts of your Ansible infrastructure so they will work with this version of Ansible. @@ -227,7 +227,9 @@ The error reported is:: Dynamic expression construction from playbooks is insecure and unsupported. -Troubleshooting Untrusted Templates +.. _untrusted_templates: + +Troubleshooting untrusted templates ----------------------------------- By default, untrusted templates are silently ignored. @@ -287,7 +289,7 @@ for example, templates appearing in playbooks, roles, variable files, and most built-in inventory plugins will yield trusted template strings. Custom plugins that source template strings will be required to use new public APIs to apply trust where appropriate. -See `Plugin API`_ and `Troubleshooting Untrusted Templates`_ for additional information. +See :ref:`plugin_api` and :ref:`untrusted_templates` for additional information. Native Jinja Mode Required @@ -488,6 +490,8 @@ Valid options are: Multiple options can be combined by separating them with commas. +.. _plugin_api: + Plugin API ========== From fbdd2aaaa835a0d20855645cb433d1f14e1f323a Mon Sep 17 00:00:00 2001 From: Matt Clay Date: Wed, 16 Apr 2025 09:32:32 -0700 Subject: [PATCH 15/15] Apply suggestions from code review --- docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst b/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst index b55347f83a2..2b812be572c 100644 --- a/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst +++ b/docs/docsite/rst/porting_guides/porting_guide_core_2.19.rst @@ -116,7 +116,7 @@ The error reported is:: This can be resolved by removing the invalid comma after the expression. -Example - Jinja Order of Operations +Example - Jinja order of operations ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This expression uses the ``~`` concatenation operator, which is evaluated after the ``contains`` test. @@ -292,7 +292,7 @@ Custom plugins that source template strings will be required to use new public A See :ref:`plugin_api` and :ref:`untrusted_templates` for additional information. -Native Jinja Mode Required +Native Jinja mode required ^^^^^^^^^^^^^^^^^^^^^^^^^^ Previous versions supported templating in two different modes: