Skip to content

Fixes #25712: handle OpenAPI Path Item fields in REST ingestion - #32958

Merged
harshsoni2024 merged 1 commit into
mainfrom
macau
Sep 10, 2026
Merged

harshsoni2024 merged 1 commit into
mainfrom
macau

Conversation

@harshsoni2024

Copy link
Copy Markdown
Contributor

Describe your changes:

Fixes #25712

Valid OpenAPI Path Item objects can contain metadata fields such as path-level parameters. The REST source was iterating every Path Item field as though it were an HTTP operation, so a parameters array could trigger a list/.get failure and discard every endpoint collected for that collection. Endpoint filtering also stopped after the first field and retained the entire Path Item, which assigned different operations on the same path to the wrong collection.

This change adds a shared iterator for the OpenAPI HTTP operation keys (get, put, post, delete, options, head, patch, and trace) and uses it for both collection discovery and endpoint filtering. Path Item metadata is ignored, each operation is filtered by its own tags, and untagged operations continue to use the existing default collection.

The behavior was reproduced against the pinned Open Food Facts OpenAPI document. Path-level parameters is valid according to the OpenAPI 3.1.1 Path Item Object specification.

Type of change:

  • Bug fix

High-level design:

N/A — small, two-file bug fix with no public API, schema, generated model, database, UI, or documentation changes.

Tests:

Use cases covered

  • A path-level parameters array before an operation does not abort endpoint ingestion.
  • A path-level parameters array after an operation is not emitted as an endpoint.
  • GET and POST operations on one path are assigned independently to their own tagged collections.
  • An untagged operation on the same path is assigned only to the existing default collection.
  • Missing or malformed paths continue to be handled without raising.

Unit tests

  • I added unit tests for the new/changed logic.
  • File updated: ingestion/tests/unit/topology/api/test_rest.py
  • Coverage: 95% for metadata.ingestion.source.api.rest.metadata (375 statements, 12 missed; branch coverage enabled by repository configuration).
  • Verification:
    • python -m pytest tests/unit/topology/api/test_rest.py::TestRest::test_path_parameters_are_not_ingested_as_operations tests/unit/topology/api/test_rest.py::TestRest::test_operations_on_one_path_use_their_own_collections -v — 2 passed.
    • python -m pytest tests/unit/topology/api/test_rest.py -v — 98 passed.
    • python -m pytest tests/unit/topology/api/ -v — 120 passed.
    • python -m pytest tests/unit/topology/api/test_rest.py --cov=metadata.ingestion.source.api.rest.metadata --cov-report=term-missing — 98 passed, 95% module coverage.
    • make unit_ingestion_dev_env — 9,552 passed, 12 skipped, 1 expected xfail.
    • make py_format, make py_format_check, and make static-checks — passed (0 type-check errors; 9 existing warnings outside the changed files).
    • git diff --check — passed.

Backend integration tests

  • Not applicable — no backend API changes.

Ingestion integration tests

  • Not applicable — this is local OpenAPI document traversal logic with no external service, credential, or network-client behavior change; source behavior is covered by unit tests and the pinned-document replay below.

Playwright (UI) tests

  • Not applicable — no UI changes.

Manual testing performed

  1. Downloaded and parsed the pinned Open Food Facts OpenAPI 3.1.0 document through the REST source parser.
  2. Replayed the parsed document through RestSource collection and endpoint producers.
  3. Verified 9 collections and 13 endpoints were emitted.
  4. Verified both Path Items containing path-level parameters were accepted, no parameters endpoint was emitted, and the source recorded zero endpoint errors and zero source failures.

UI screen recording / screenshots:

Not applicable — no UI changes.

Checklist:

  • I have read the CONTRIBUTING document.
  • My PR title is Fixes <issue-number>: <short explanation>.
  • My PR is linked to a GitHub issue via Fixes #<issue-number> above.
  • I have commented on my code, particularly in hard-to-understand areas.
  • Not applicable for JSON Schema changes — this PR does not change a JSON Schema.
  • Not applicable for UI changes — this PR does not change the UI.
  • I have added tests and listed them above.
  • I have added tests that cover the exact bug-fix scenarios.

🤖 Generated with Claude Code

Copilot AI lite review requested due to automatic review settings September 8, 2026 12:56
@harshsoni2024
harshsoni2024 requested review from a team and akashverma0786 as code owners September 8, 2026 12:56
@github-actions

github-actions Bot commented Sep 8, 2026 •

Copy link
Copy Markdown
Contributor

✅ PR checks passed

The linked issue has a description and all required Shipping project fields set. Thanks!

@github-actions github-actions Bot added Ingestion safe to test Add this label to run secure Github workflows on PRs labels Sep 8, 2026
@gitar-bot

gitar-bot Bot commented Sep 8, 2026 •

Copy link
Copy Markdown
Code Review ✅ Approved

Fixes REST ingestion to correctly handle OpenAPI Path Item metadata fields like path-level parameters. The fix adds a shared iterator for HTTP operation keys and uses it for both collection discovery and endpoint filtering, so metadata fields no longer trigger parsing failures and operations are independently assigned to their correct tagged collections. Unit tests cover path parameters, multi-operation paths, and untagged operations, with 95% module coverage and all 98 REST tests passing.

Options

Display: compact → Counting what did not apply, without listing it.

Comment with these commands to change the behavior for this request:

Compact
gitar display:verbose         

Was this helpful? React with 👍 / 👎 | Powered by Gitar — free for open source

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The fix is a targeted, spec-aligned traversal change with clear regression tests that would fail under the prior behavior and pass with the new iterator-based logic.

Pull request overview

This PR fixes REST/OpenAPI ingestion when a Path Item contains non-operation metadata fields (e.g., path-level parameters). It introduces a shared iterator over valid OpenAPI operation keys so only HTTP operations are treated as endpoints, preventing list/.get failures and ensuring each operation on the same path is assigned to the correct collection by its own tags.

Changes:

  • Add a shared _iter_path_operations() iterator (backed by OPENAPI_OPERATION_METHODS) to yield only valid OpenAPI Operation Objects from paths.
  • Use the iterator for both collection discovery (_path_collection_names) and endpoint filtering (_filter_collection_endpoints) so Path Item metadata is ignored and per-operation tagging is respected.
  • Add unit tests covering path-level parameters ordering and multi-operation-per-path collection assignment.
File summaries
File Description
ingestion/src/metadata/ingestion/source/api/rest/metadata.py Filters paths traversal to OpenAPI HTTP operations only; fixes per-operation collection filtering and avoids treating Path Item metadata as endpoints.
ingestion/tests/unit/topology/api/test_rest.py Adds regression tests for path-level parameters and for assigning multiple operations on the same path to their correct collections (including default).
Review details
  • Files reviewed: 2/2 changed files
  • Comments generated: 0
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@sonarqubecloud

sonarqubecloud Bot commented Sep 8, 2026

Copy link
Copy Markdown

@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

✅ Playwright Results — workflow succeeded

Validated commit 2677bb3d13c13c1986feb6d183ee7ccb31b3a365 in Playwright run 34228992826, attempt 1.

✅ 109 passed · ❌ 0 failed · 🟡 0 flaky · ⏭️ 0 skipped · 🧰 0 lifecycle flaky

Performance

Blocking targets: ✅ met · Optimization targets: 🟡 in progress

Shard-job maxima below are not the full workflow wall time; the linked run includes build, fixture, planning, and reporting.

🕒 Full workflow signal wall (to summary) 49m 46s

⏱️ Max setup 4m 30s · max shard execution 11m 53s · max shard-job elapsed before upload 18m 44s · reporting 3s

🌐 236.12 requests/attempt · 1.77 app boots/UI scenario · 0.00% common-shard skew

Optimization targets still in progress:

  • Browser traffic was 236.12 requests per attempt (convergence target: fewer than 200).
  • Application boot ratio was 1.77 per UI scenario (213 boots / 120 scenarios; convergence target: at most 1).
Shard Passed Failed Flaky Skipped Lifecycle failed Lifecycle flaky
✅ Shard chromium-01 46 0 0 0 0 0
✅ Shard ingestion-01 34 0 0 0 0 0
✅ Shard ingestion-02 29 0 0 0 0 0

📦 Download artifacts

How to debug locally
# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip    # view trace

@harshsoni2024
harshsoni2024 added this pull request to the merge queue Sep 10, 2026
Merged via the queue into main with commit ff3d31b Sep 10, 2026
117 of 118 checks passed
@harshsoni2024
harshsoni2024 deleted the macau branch September 10, 2026 05:19

This branch was previously deployed

1 inactive deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Ingestion safe to test Add this label to run secure Github workflows on PRs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Parse causes an error with OpenAPI JSON

3 participants