Add configurable full-text stopword policies (#2606)

test/README.md

@@ -167,6 +167,110 @@ The UI allows you to select a specific test to run and to see the results of the
 
 See the [Playwright docs](https://playwright.dev/docs/running-tests) for more information on running tests.
 
+### Full-text stopword CI coverage
+
+The `Full-text stopwords` suite in `sql\container.spec.ts` runs in the normal
+Playwright CI shards and browser projects without opt-in variables or mocked ARM
+responses. It uses the existing SQL test-account configuration and requires
+`EnableNoSQLFullTextSearchPreviewFeatures` on every SQL shard account. A missing
+capability fails with an explicit prerequisite message rather than skipping.
+The test never changes account settings.
+Trace and video recording are disabled for this test file because the harness URL
+contains authentication tokens.
+
+The five cases cover multiword defaults (including case, order, and duplicates),
+invalid entries in each word list, Discard, and path overrides returning to
+inheritance. Saved policies are checked after a fresh app load. Each case creates
+an isolated 400-RU/s container and deletes its generated database in `afterEach`,
+including after failed setup or assertions. Setup and cleanup use ARM, like the
+policy operations; Settings still requires data-plane access. These tests do not
+assert search semantics.
+
+The Settings-created CI policy uses legacy analysis, which does not expose the
+standard-only stop-filter toggle. Its inheritance case checks the outgoing policy
+contains only the inherited path, without hidden analysis overrides. Standard
+filter-toggle/reset behavior is covered by the unit regressions. Restoring
+inheritance resets all supported per-field analysis settings (including tokenizer
+and filters) rather than copying
+the current defaults; later default changes therefore still apply. Unknown fields
+are preserved, and unsupported policies remain read-only.
+
+`resources\account.bicep` includes the capability for newly provisioned SQL test
+accounts. CI does not deploy that template: `.github\workflows\ci.yml` uses
+pre-existing `${DE_ACCOUNT_PREFIX}-de-test-sql-1` through `-20` accounts in the
+subscription/resource group configured by the `E2ETESTS_*` secrets. Their owner
+must enable and verify the capability before running this suite, preserving all
+existing capabilities. Do not redeploy the generic template over those accounts
+to enable a single capability. The suite checks the actual account metadata, not
+a feature flag or mocked response.
+
+Run it with:
+
+```powershell
+npx playwright test container.spec.ts --grep "Full-text stopwords" --project "Microsoft Edge"
+```
+
+### Full-text stopword manual acceptance
+
+The CI cases above use the existing Playwright harness unchanged and cover policy
+management, not New Container or query semantics. Unit regressions cover creation,
+full-text-only accounts without vector capability, and unrelated Settings saves
+with preserved read-only or indexed analysis.
+
+For manual browser acceptance, use an approved provisioned-throughput NoSQL test
+account with `EnableNoSQLFullTextSearch` and
+`EnableNoSQLFullTextSearchPreviewFeatures`. Create only uniquely named test containers
+and remove them afterward. Do not change account capabilities, network access, or
+permissions as part of the test.
+
+To validate Hosted Entra sign-in, run the dev server on **port 1234** and open
+`https://localhost:1234/hostedExplorer.html`.
+The development sign-in bridge redirects to `https://localhost:1234/redirectBridge.html`;
+a server on another port cannot complete that flow. Use real sign-in, directory,
+subscription, and account selection, then verify create, indexed locks, invalid
+input, Save/Discard, a full browser reload, and policy preservation during an unrelated
+Settings update. Do not substitute injected tokens or fabricated account capabilities.
+
+Configuration round trips do not prove query semantics. Report search-result
+failures separately from configuration acceptance.
+
+For visual acceptance, inspect both creation and Settings in light/dark themes and
+at 1440px, 768px, and 375px widths. Confirm that labels align with their fields,
+inheritance text wraps without clipping, default/override descriptions are clear,
+locked values remain readable without becoming editable, service-default selection
+explains disabled word fields, and errors are associated with the offending word
+list. These checks can use unsaved edits followed by Discard; no new container is
+needed for a visual-only audit.
+
+Check that collapsed path headers retain the path name, inheritance/index state,
+and any validation warning. Space/Enter must toggle expansion without losing
+values, and Delete must be a separate keyboard target. Adding a path should focus
+its input; deleting should return focus to Add, which must still respond to a mouse
+click. Check the empty state and a long path name at narrow width. In wide Settings,
+the word lists should sit side by side; in the create panel they should stack.
+Controls should not stretch to fill the available workspace: language and stopword
+dropdowns and individual word-list boxes are at most 240px wide, path inputs at
+most 320px, and the Settings form at most 496px. They must still shrink to fit a
+narrow panel without clipping values, errors, or accessible labels.
+
+Repeat creation and policy management with each supported language: `en-US`,
+`fr-FR`, `de-DE`, `es-ES`, `it-IT`, `pt-PT`, and `pt-BR`. Check 20-word lists with
+case differences, duplicates, and accented text through creation, indexed locks,
+an unindexed-path save, Discard, and full page reload. Verify that persisted custom
+words affect search as expected; do not treat current backend behavior as the
+expected result merely because the policy was accepted.
+
+For UI acceptance, exercise 0, 1, 5, 20, and 100 entries in **both** word lists and
+both creation/Settings. Test real newline paste, CRLF, middle edits, whole-line
+deletion, long and multilingual words, a late invalid entry, differing list
+lengths, filter off/on, and keyboard inspection of locked values. Include all
+default/path language combinations and service-default reset. A passing
+serialization test or tidy one-word screenshot is not multiword UX acceptance.
+
+Verify modern policy fields with raw ARM JSON. Older typed management clients,
+including `az cosmosdb sql container show`, may omit fields unknown to their models
+even when the service preserved them.
+
 ### Testing with Data Plane RBAC Authentication
 
 By default, the tests will use key based authentication to access the database accounts. For APIs that support data plane RBAC, the

test/resources/account.bicep

@@ -20,7 +20,7 @@ param totalThroughputLimit int = 10000
 param testAccountType string
 
 var kind = (testAccountType == 'mongo' || testAccountType == 'mongo32') ? 'MongoDB' : 'GlobalDocumentDB'
-var capabilities = (testAccountType == 'tables') ? [{name: 'EnableTable'}] : (testAccountType == 'cassandra') ? [{name: 'EnableCassandra'}] : (testAccountType == 'gremlin') ? [{name: 'EnableGremlin'}] : []
+var capabilities = (testAccountType == 'tables') ? [{name: 'EnableTable'}] : (testAccountType == 'cassandra') ? [{name: 'EnableCassandra'}] : (testAccountType == 'gremlin') ? [{name: 'EnableGremlin'}] : (testAccountType == 'sql') ? [{name: 'EnableNoSQLFullTextSearchPreviewFeatures'}] : []
 var serverVersion = (testAccountType == 'mongo32') ? '3.2' : (testAccountType == 'mongo') ? '6.0' : null
 
 resource testCosmosAccount 'Microsoft.DocumentDB/databaseAccounts@2024-02-15-preview' = {

test/sql/container.spec.ts

@@ -1,6 +1,21 @@
-import { expect, test } from "@playwright/test";
+import { CosmosDBManagementClient } from "@azure/arm-cosmosdb";
+import { expect, Page, Response, test } from "@playwright/test";
 
-import { DataExplorer, TEST_AUTOSCALE_THROUGHPUT_RU, TestAccount, generateUniqueName } from "../fx";
+import {
+  CommandBarButton,
+  DataExplorer,
+  generateUniqueName,
+  getAccountName,
+  getAzureCLICredentials,
+  ONE_MINUTE_MS,
+  resourceGroupName,
+  subscriptionId,
+  TEST_AUTOSCALE_THROUGHPUT_RU,
+  TestAccount,
+} from "../fx";
+
+// The test harness URL contains authentication tokens.
+test.use({ trace: "off", video: "off" });
 
 test("SQL database and container CRUD", async ({ page }) => {
   const databaseId = generateUniqueName("db");
@@ -50,3 +65,204 @@ test("SQL database and container CRUD", async ({ page }) => {
 
   await expect(databaseNode.element).not.toBeAttached();
 });
+
+test.describe("Full-text stopwords", () => {
+  test.use({ actionTimeout: 30000 });
+
+  let armClient: CosmosDBManagementClient;
+  let accountName: string;
+  let databaseId: string | undefined;
+  let explorer: DataExplorer;
+  const containerId = "stopwords";
+  const additionalWords = "cosmos\nCatalog\ncosmos";
+  const wordsToKeep = "the\nand";
+  const inheritanceLabel = "Inherit the container's language and stopwords";
+
+  const openPolicy = async (page: Page): Promise<DataExplorer> => {
+    if (!databaseId) {
+      throw new Error("The stopword test database has not been created.");
+    }
+    const current = await DataExplorer.open(page, TestAccount.SQL);
+    const containerNode = await current.waitForContainerNode(databaseId, containerId);
+    await containerNode.expand();
+    await current.frame.getByTestId(`TreeNodeContainer:${databaseId}/${containerId}/Scale & Settings`).press("Enter");
+    await current.frame.getByRole("tab", { name: "Container Policies", exact: true }).press("Enter");
+    await current.frame.getByRole("tab", { name: "Full Text Policy", exact: true }).press("Enter");
+    return current;
+  };
+
+  const savePolicy = async (): Promise<Response> => {
+    const save = explorer.commandBarButton(CommandBarButton.Save);
+    await expect(save).toBeEnabled();
+    const [response] = await Promise.all([
+      explorer.frame
+        .page()
+        .waitForResponse(
+          (response) =>
+            response.request().method() === "PUT" &&
+            new URL(response.url()).pathname.endsWith(`/sqlDatabases/${databaseId}/containers/${containerId}`),
+          { timeout: 2 * ONE_MINUTE_MS },
+        ),
+      save.click(),
+    ]);
+    expect(response.ok(), `Saving the policy returned HTTP ${response.status()}`).toBe(true);
+    await expect(explorer.getConsoleHeaderStatus()).toContainText(`Successfully updated container ${containerId}`, {
+      timeout: 2 * ONE_MINUTE_MS,
+    });
+    await expect(save).toBeDisabled({ timeout: 2 * ONE_MINUTE_MS });
+    return response;
+  };
+
+  test.beforeEach(async ({ page }) => {
+    databaseId = undefined;
+    armClient = new CosmosDBManagementClient(getAzureCLICredentials(), subscriptionId);
+    accountName = getAccountName(TestAccount.SQL);
+    const account = await armClient.databaseAccounts.get(resourceGroupName, accountName);
+    expect(
+      account.capabilities?.map(({ name }) => name),
+      "The SQL CI account must enable EnableNoSQLFullTextSearchPreviewFeatures.",
+    ).toContain("EnableNoSQLFullTextSearchPreviewFeatures");
+
+    const newDatabaseId = generateUniqueName("stopwords", { length: 8 });
+    for await (const database of armClient.sqlResources.listSqlDatabases(resourceGroupName, accountName)) {
+      expect(database.name).not.toBe(newDatabaseId);
+    }
+    databaseId = newDatabaseId;
+    await armClient.sqlResources.beginCreateUpdateSqlDatabaseAndWait(resourceGroupName, accountName, databaseId, {
+      resource: { id: databaseId },
+      options: {},
+    });
+    await armClient.sqlResources.beginCreateUpdateSqlContainerAndWait(
+      resourceGroupName,
+      accountName,
+      databaseId,
+      containerId,
+      {
+        resource: { id: containerId, partitionKey: { paths: ["/pk"], kind: "Hash" } },
+        options: { throughput: 400 },
+      },
+    );
+
+    explorer = await openPolicy(page);
+    await explorer.frame.getByRole("button", { name: "Create new full text search policy" }).click();
+    await explorer.frame.getByRole("button", { name: "Add full text path", exact: true }).click();
+    await explorer.frame.getByRole("textbox", { name: /^Path/ }).fill("/text");
+    await explorer.frame.getByRole("checkbox", { name: inheritanceLabel }).check();
+    const defaults = explorer.frame.getByRole("group", { name: "Default stopwords" });
+    await defaults.getByRole("combobox", { name: "Stopword list" }).click();
+    await explorer.frame.getByRole("option", { name: "None", exact: true }).click();
+    await defaults.getByRole("textbox", { name: "Additional stopwords" }).fill(additionalWords);
+    await defaults.getByRole("textbox", { name: "Words to keep" }).fill(wordsToKeep);
+  });
+
+  test.afterEach(async () => {
+    test.setTimeout(2 * ONE_MINUTE_MS);
+    if (!databaseId) {
+      return;
+    }
+    await armClient.sqlResources
+      .beginDeleteSqlDatabaseAndWait(resourceGroupName, accountName, databaseId)
+      .catch((error: unknown) => {
+        if (!error || typeof error !== "object" || !("statusCode" in error) || error.statusCode !== 404) {
+          throw error;
+        }
+      });
+    databaseId = undefined;
+  });
+
+  test("saves multiword defaults and an inherited path across reload", async ({ page }) => {
+    await savePolicy();
+    explorer = await openPolicy(page);
+    const defaults = explorer.frame.getByRole("group", { name: "Default stopwords" });
+    await expect(explorer.frame.getByRole("combobox", { name: /^Default language/ })).toContainText("English (US)");
+    await expect(defaults.getByRole("combobox", { name: "Stopword list" })).toContainText("None");
+    await expect(defaults.getByRole("textbox", { name: "Additional stopwords" })).toHaveValue(additionalWords);
+    await expect(defaults.getByRole("textbox", { name: "Words to keep" })).toHaveValue(wordsToKeep);
+    await expect(explorer.frame.getByRole("textbox", { name: /^Path/ })).toHaveValue("/text");
+    await expect(explorer.frame.getByRole("checkbox", { name: inheritanceLabel })).toBeChecked();
+    await expect(explorer.commandBarButton(CommandBarButton.Save)).toBeDisabled();
+  });
+
+  for (const label of ["Additional stopwords", "Words to keep"]) {
+    test(`blocks invalid entries in ${label} until corrected`, async () => {
+      const defaults = explorer.frame.getByRole("group", { name: "Default stopwords" });
+      const field = defaults.getByRole("textbox", { name: label });
+      const original = await field.inputValue();
+      await field.fill(`${original}\ninvalid phrase`);
+      await expect(field).toHaveAttribute("aria-invalid", "true");
+      await expect(defaults.getByRole("alert")).toBeVisible();
+      await expect(explorer.commandBarButton(CommandBarButton.Save)).toBeDisabled();
+      await field.fill(original);
+      await expect(field).not.toHaveAttribute("aria-invalid", "true");
+      await expect(defaults.getByRole("alert")).toHaveCount(0);
+      await expect(explorer.commandBarButton(CommandBarButton.Save)).toBeEnabled();
+    });
+  }
+
+  test("discards unsaved language, preset, word-list, and path changes", async ({ page }) => {
+    await savePolicy();
+    const defaults = explorer.frame.getByRole("group", { name: "Default stopwords" });
+    await explorer.frame.getByRole("combobox", { name: /^Default language/ }).click();
+    await explorer.frame.getByRole("option", { name: "French", exact: true }).click();
+    await defaults.getByRole("textbox", { name: "Additional stopwords" }).fill("galaxy\nnebula");
+    await defaults.getByRole("textbox", { name: "Words to keep" }).fill("le\nla");
+    await defaults.getByRole("combobox", { name: "Stopword list" }).click();
+    await explorer.frame.getByRole("option", { name: "Use service default (clear custom words)", exact: true }).click();
+    await explorer.frame.getByRole("textbox", { name: /^Path/ }).fill("/changed");
+    await expect(explorer.commandBarButton(CommandBarButton.Save)).toBeEnabled();
+    await explorer.frame.getByRole("menuitem", { name: "Discard", exact: true }).click();
+    await expect(defaults.getByRole("textbox", { name: "Additional stopwords" })).toHaveValue(additionalWords);
+    await expect(defaults.getByRole("textbox", { name: "Words to keep" })).toHaveValue(wordsToKeep);
+    await expect(defaults.getByRole("combobox", { name: "Stopword list" })).toContainText("None");
+    await expect(explorer.frame.getByRole("combobox", { name: /^Default language/ })).toContainText("English (US)");
+    await expect(explorer.frame.getByRole("textbox", { name: /^Path/ })).toHaveValue("/text");
+    await expect(explorer.commandBarButton(CommandBarButton.Save)).toBeDisabled();
+    explorer = await openPolicy(page);
+    await expect(
+      explorer.frame
+        .getByRole("group", { name: "Default stopwords" })
+        .getByRole("textbox", { name: "Additional stopwords" }),
+    ).toHaveValue(additionalWords);
+    await expect(explorer.frame.getByRole("textbox", { name: /^Path/ })).toHaveValue("/text");
+  });
+
+  test("saves a path override and resets it to inherited defaults", async ({ page }) => {
+    await explorer.frame.getByRole("checkbox", { name: inheritanceLabel }).uncheck();
+    await explorer.frame.getByRole("combobox", { name: /^Language\b/ }).click();
+    await explorer.frame.getByRole("option", { name: "French", exact: true }).click();
+    const override = explorer.frame.getByRole("group", { name: "Stopwords for /text" });
+    await override.getByRole("combobox", { name: "Stopword list" }).click();
+    await explorer.frame.getByRole("option", { name: "None", exact: true }).click();
+    await override.getByRole("textbox", { name: "Additional stopwords" }).fill("galaxy\nnebula");
+    await override.getByRole("textbox", { name: "Words to keep" }).fill("le\nla");
+    await savePolicy();
+    explorer = await openPolicy(page);
+    const persisted = explorer.frame.getByRole("group", { name: "Stopwords for /text" });
+    await expect(explorer.frame.getByRole("combobox", { name: /^Language\b/ })).toContainText("French");
+    await expect(persisted.getByRole("textbox", { name: "Additional stopwords" })).toHaveValue("galaxy\nnebula");
+    await expect(persisted.getByRole("textbox", { name: "Words to keep" })).toHaveValue("le\nla");
+    const defaults = explorer.frame.getByRole("group", { name: "Default stopwords" });
+    await expect(defaults.getByRole("textbox", { name: "Additional stopwords" })).toHaveValue(additionalWords);
+    await expect(defaults.getByRole("textbox", { name: "Words to keep" })).toHaveValue(wordsToKeep);
+    await expect(persisted.getByRole("checkbox", { name: "Enable stopword filtering" })).toHaveCount(0);
+    await explorer.frame.getByRole("checkbox", { name: inheritanceLabel }).check();
+    await expect(persisted).toHaveCount(0);
+    const inheritedSave = await savePolicy();
+    expect(inheritedSave.request().postDataJSON()).toEqual(
+      expect.objectContaining({
+        properties: expect.objectContaining({
+          resource: expect.objectContaining({
+            fullTextPolicy: expect.objectContaining({ fullTextPaths: [{ path: "/text" }] }),
+          }),
+        }),
+      }),
+    );
+    explorer = await openPolicy(page);
+    await expect(explorer.frame.getByRole("checkbox", { name: inheritanceLabel })).toBeChecked();
+    await expect(explorer.frame.getByRole("group", { name: "Stopwords for /text" })).toHaveCount(0);
+    await explorer.frame.getByRole("checkbox", { name: inheritanceLabel }).uncheck();
+    const reset = explorer.frame.getByRole("group", { name: "Stopwords for /text" });
+    await expect(reset.getByRole("textbox", { name: "Additional stopwords" })).toHaveValue("");
+    await expect(reset.getByRole("textbox", { name: "Words to keep" })).toHaveValue("");
+  });
+});