[Bug]: ToHaveAccesssibleErrorMessage does not work with group of inputs

[MattyBalaam]

Version

1.50.1

Steps to reproduce

We use component tests, which probably allows me to illustrate the code that is the edge case.

Standard inputs work very well with accessible error message, but grouped fields do not

    test('standard input passes', async ({ mount }) => {
      const component = await mount(
        <div>
          <div>
            <label htmlFor="the-id">This is the label</label>
            <input
              id="the-id"
              aria-invalid="true"
              aria-errormessage="some-error-id"
              name="name"
            />
          </div>
          <div id="some-error-id" role="alert">
            I am the error message
          </div>
        </div>,
      );

      await expect(component.getByLabel('This is the label')).toHaveAttribute(
        'aria-errormessage',
        'some-error-id',
      );

      await expect(
        component.getByLabel('This is the label'),
      ).toHaveAccessibleErrorMessage('I am the error message');
    });

    test('fieldset fails', async ({ mount }) => {
      const component = await mount(
        <fieldset aria-invalid="true" aria-errormessage="some-error-id">
          <legend>This is the label</legend>

          <input type="radio" name="some-name" value="some-value" />
          <input type="radio" name="some-name" value="some-other-value" />

          <div id="some-error-id" role="alert">
            I am the error message
          </div>
        </fieldset>,
      );

      // manual check works:
      await expect(
        component.page().locator(
          // Regex is workaround for Playwright not accepting IDs with colons
          `[id='${await component.getByLabel('This is the label').getAttribute('aria-errormessage')}']`,
        ),
      ).toHaveText('I am the error message');

      // fails
      await expect(
        component.getByLabel('This is the label'),
      ).toHaveAccessibleErrorMessage('I am the error message');
    });

    test('role group fails', async ({ mount }) => {
      const component = await mount(
        <div
          role="group"
          aria-labelledby="some-label"
          aria-invalid="true"
          aria-errormessage="some-error-id"
        >
          <label htmlFor="some-name" id="some-label">
            This is the label
          </label>

          <input type="radio" name="some-name" value="some-value" />
          <input type="radio" name="some-name" value="some-other-value" />

          <div id="some-error-id" role="alert">
            I am the error message
          </div>
        </div>,
      );

      // manual check works:
      await expect(
        component.page().locator(
          // Regex is workaround for Playwright not accepting IDs with colons
          `[id='${await component.getByLabel('This is the label').getAttribute('aria-errormessage')}']`,
        ),
      ).toHaveText('I am the error message');

      // fails
      await expect(
        component.getByLabel('This is the label'),
      ).toHaveAccessibleErrorMessage('I am the error message');
    });

Expected behavior

Grouped inputs should work: https://www.w3.org/WAI/tutorials/forms/grouping/

Actual behavior

Locator: locator('#root').locator('internal:control=component').getByLabel('This is the label')
Expected string: "I am the error message"
Received string: ""

Additional context

New issue based on work released here: #31249

Environment

npx envinfo --preset playwright     
sh: /Users/*******/node_modules/.bin/envinfo: Permission denied

[dgozman]

@MattyBalaam Could you please explain why should this aria-errormessage be respected?

• According to https://w3c.github.io/aria/#aria-errormessage, it only applies to specific roles, and group is not in the list.
• According to https://w3c.github.io/aria/#group, aria-errormessage has been deprecated on this role in ARIA 1.2, so it should not be respected.

Is there any spec/document/explainer that shows how and why error message should be used with groups/fieldsets?

[MattyBalaam]

This is interesting, I have historically used this because I know that testing-library respects this usage of this pattern with role="group", and it is something that a lot of guides online recommend to use for a group of elements. e.g. https://www.w3.org/WAI/ARIA/apg/patterns/checkbox/

I have tested with "radiogroup" role instead and that works with the error id it appears, but if I try to make a set of checkboxes it does not. When I look at a large library, I see that react-aria for example instead used aria-describedby for their checkbox group: https://react-spectrum.adobe.com/react-aria/useCheckboxGroup.html#group-validation. I have also been able to foree the matter and use the role "checkbox" for this group of checkboxes, but I don’t think that would be correct?

Basically I am confused now about the best way to show these error messages. Perhaps this case of a group of checkboxes is something that has fallen through the cracks of standards.

The fieldset example still does not work though. it is possible I might have implemented the example html incorrectly too though, do you have any ideas theres?

[dgozman]

Basically I am confused now about the best way to show these error messages. Perhaps this case of a group of checkboxes is something that has fallen through the cracks of standards.

You are not alone! It seems like many vendors have only shipped support for aria-errormessage in 2024, and there are still discrepancies between browsers and assistive software.

The fieldset example still does not work though. it is possible I might have implemented the example html incorrectly too though, do you have any ideas theres?

Instead of getByLabel('This is the label') it should be getByRole('group', { name: 'This is the label' }). Label is not an accessible name, and technically fieldset does not have a label.

[dgozman]

Closing as there is no action item for Playwright here. If you think there is something to improve, please file a new issue and link to this one.