Repository navigation
fix(mssql): read descriptions from the right database, honour encryption per driver, bound query timeouts - #32574
Conversation
…ion per driver, bound query timeouts Descriptions are read per database through DB_NAME()-scoped queries, but the maps were loaded before the inspector switched, so in an ingest-all-databases run every database was documented from the previously connected one, which no lookup can match. The maps now load after the switch. setup_ssl only ever applied encryption to pyodbc. pytds, the default scheme, has no encrypt switch: it enables TLS only when a CA certificate is supplied, and then always verifies the chain, so trustServerCertificate can drop the host name check alone - which it now does. Encrypt with no CA certificate cannot be honoured by that driver, and warns instead of connecting in the clear while the configuration claims otherwise. pymssql, which takes no TLS parameters at all, warns as well. pytds and pymssql default to no socket timeout and pyodbc to no query timeout, and the shared engine builder sets neither, so a hung read stalled a workflow indefinitely. Each driver now gets a ten minute bound, applied to a copy of the connection so that a stored service connection never gains a timeout nobody configured, and engines are built with pre-ping. Unique constraints were dropped for every table: the dialect method raised NotImplementedError, which the catalogue swallows. They are now read from INFORMATION_SCHEMA alongside the primary keys the dialect already reads. Indexed views are persisted and kept up to date by the engine, so they are reported as materialized views rather than plain views. The container suite gains a second database, descriptions at every level, a unique constraint and an indexed view, so what is read per database is exercised against more than one of them. The shadowed duplicate of test_mssql_url asserted a connection with no database, which the schema has since made required, so it is removed rather than revived.
❌ PR checklist incompleteThis PR cannot be merged until the following are addressed on its linked issue:
The fields live on the linked issue in the Shipping project (open the issue → right sidebar → Projects). After you set them, re-run this check (or push a commit) — issue/project changes do not re-trigger it automatically. Maintainers can bypass this check by adding the |
|
Hi there 👋 Thanks for your contribution! The OpenMetadata team will review the PR shortly! Once it has been labeled as Let us know if you need any help! |
✅ Playwright Results — workflow succeededValidated commit ✅ 4556 passed · ❌ 0 failed · 🟡 9 flaky · ⏭️ 1 skipped · 🧰 0 lifecycle flaky PerformanceBlocking targets: ✅ met · Optimization targets: 🟡 in progress Shard-job maxima below are not the full workflow wall time; the linked run includes build, fixture, planning, and reporting. 🕒 Full workflow signal wall (to summary) 53m 56s ⏱️ Max setup 5m 32s · max shard execution 21m 35s · max shard-job elapsed before upload 24m 58s · reporting 25s 🌐 224.66 requests/attempt · 2.23 app boots/UI scenario · 47.10% common-shard skew Optimization targets still in progress:
🟡 9 flaky test(s) (passed on retry)
How to debug locally# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip # view trace |
|
The Python checkstyle failed. Please run You can install the pre-commit hooks with |
Ten minutes was picked for metadata and query-log reads, but every workflow builds its engine through the same connection: the profiler and data quality reach _get_client through get_ssl_connection too, and the profiler's own default budget is 12 hours per table. A ten minute socket bound on pytds and pymssql - where the wait for the first byte is the server-side execution time - would have failed profiling runs that are working as designed. The bound now matches that budget, so it rules out only the wait that never ends, which is the failure it exists for; a dead pooled connection is still caught far sooner by pre-ping. Tightening it per service through `timeout` in connectionArguments is unchanged and documented.
|
The Python checkstyle failed. Please run You can install the pre-commit hooks with |
✅ Generated Sources Auto-UpdatedThe generated TypeScript types ( |
|
The Python checkstyle failed. Please run You can install the pre-commit hooks with |
pyodbc spends `timeout` on the login, so a service that set one still ran its queries against the 12 hour default - the opposite of what setting it means. The listener now takes the configured value when there is one, and the docs say that on pyodbc the single value bounds both. The description-ordering tests asserted the order of three mocked calls, which holds whether or not the load reads the database it was meant to. They now record which database the connection pointed at when each load ran and assert that against the database being ingested, so reverting the fix fails them with the symptom it causes: the entry-point database read first, then each database read from its predecessor.
…nto fix/mssql-connector-hardening
…a server The catalogue assertion read table.tableConstraints straight off get_by_name, but tableConstraints is not a default field - TableRepository returns null for it unless it is named in fields - so the composite UNIQUE looked missing when it had in fact been ingested. The gap that let that read as a connector failure is that nothing covered the step between reflection and the entity, so add it to the server-free suite: get_columns_and_constraints against the container, asserting the single-column UNIQUE lands on the column and the composite one on the table.
…en-metadata/OpenMetadata into fix/mssql-connector-hardening
The per-driver encryption behaviour was argued from the drivers' source and asserted at the level of connection arguments, which cannot show what actually travels on the wire. sys.dm_exec_connections.encrypt_option can, so ask it: with Encrypt Connection on and no CA certificate, both pytds and pymssql report FALSE, and the connector's warning is what the user gets instead. pyodbc is left out deliberately. It needs Microsoft's ODBC driver, which the CI runners no more install than a dev machine does - unixodbc-dev is the manager, not a driver - so a connection test there could only ever skip. The part this connector owns, Encrypt and TrustServerCertificate reaching the connection arguments, is already covered in the unit suite.
Installing Microsoft's ODBC driver makes the pyodbc half of the matrix runnable, and the server confirms it: Encrypt plus TrustServerCertificate gives encrypt_option TRUE, which is the claim that pyodbc is the one scheme able to enforce encryption against a certificate OpenMetadata was given nothing to verify. It skips where the driver is absent, which includes CI.
sys.extended_properties.value is sql_variant, and FreeTDS hands that back as bytes where the other drivers return text, so every table description ingested through mssql+pymssql arrived as b'...' - the repr of a bytes object reaching the catalogue, or a validation failure. The database, schema and stored procedure queries already cast it; the table one did not. Found by running the source end to end against a real server once per driver, which is also why the regression test builds a pymssql source: the container suite only ever exercised pytds, so no driver-specific difference could show.
The guard skipped only when no ODBC driver at all was registered, but the connection asks the driver manager for the schema's default by name. A machine carrying some other driver - FreeTDS, which this connector supports - would therefore have run the test and failed during connection setup instead of skipping. It now reads that name from the schema rather than repeating the literal, and says which driver is missing when it skips.
…en-metadata/OpenMetadata into fix/mssql-connector-hardening
The pre-ping and query-timeout tests mocked the engine factory and read back the arguments it was called with, so they would have stayed green had the engine stopped applying either. SQLAlchemy resolves a dialect and builds a pool without opening a socket, so the connector can be driven for real: the assertions now read pre-ping off the pool, and fire the engine's own connect listener to see the bound a fresh connection comes away with. All three fail when the behaviour is removed, which is what they could not do before.
Code Review ✅ Approved 2 closed / 2 findings🔴 High risk · MSSQL driver TLS handling changes encryption and certificate-validation behavior Hardens the MSSQL connector by reading descriptions after switching to the target database, applying driver-specific encryption configuration, bounding query timeouts to 12 hours, and reflecting unique constraints and indexed views. Resolved socket timeout handling on pytds/pymssql and pyodbc driver matching in tests. ✅ 2 closed✅ Edge Case: Default 600s socket timeout may break long healthy reads on pytds/pymssql
✅ Edge Case: pyodbc skip guard accepts any ODBC driver, not the one the test uses
OptionsDisplay: compact → Counting what did not apply, without listing it. Comment with these commands to change the behavior for this request:
Was this helpful? React with 👍 / 👎 | Powered by Gitar — free for open source |
|
|



Fixes #33342
Connector-local fixes from the MSSQL connector audit. Each one is a silent failure today: the run reports success and the metadata is quietly wrong.
What changes
DB_NAME()-scoped queries ran before the inspector switched, so an ingest-all-databases run documented every database from the previously connected one. Nothing matched, leaving table, column, schema, database and stored-procedure descriptions silently missing.setup_sslonly ever applied it to pyodbc, so onmssql+pytds— the default scheme — a service configured to encrypt connected in the clear. pytds has no encrypt switch: it enables TLS only when a CA certificate is supplied, and then always verifies the chain, sotrustServerCertificatenow drops the host-name check alone. Encrypt with no CA certificate warns instead of lying about it;mssql+pymssql, which takes no TLS parameters at all, warns too.pool_pre_ping. The default is 12h — the profiler's owntimeoutSecondsbudget — so it rules out only the wait that never ends;timeoutinconnectionArgumentstightens it per service.NotImplementedError, which the catalogue swallows, so they were dropped for every table.Mssql.mdand the schema descriptions now state the per-driver encryption behaviour and the timeout default.Files outside
mssql/utils/ssl_manager.pyMssqlConnectiondispatch branch. The only shared file here — worth a deliberate look.ssisConnection.json,wherescapeConnection.json,serviceConnection.json, generated.tstests/unit/test_source_connection.pytest_mssql_url(hence the# noqa: F811), so the first never ran. Merged into the survivor; itsdatabase=Nonecase could not be revived, as the schema has since made the field required.Testing
The container suite gains a second database with descriptions at all five levels, a unique constraint and an indexed view, so anything read per database is exercised against more than one.
test_reflection.pyasserts what the source reads straight from SQL Server (no OpenMetadata server needed) andtest_metadata.pyasserts the catalogue side. Every new assertion was checked to fail with its fix reverted.Two calls I would like a second opinion on
timeoutis the login timeout, so a service that already sets it had unbounded queries; it now bounds queries at the same value. If you would rather keep the two separate, I will move the query bound to its own key.Not included, deliberately
The shared
builders.pytimeout/retry and pool bound, four-part name resolution insql_lineage.py, ownership, and theresultLimitdocs — each wants its own PR.The PR appears safe to merge; the latest changes resolve the remaining test-quality concern without introducing a new actionable issue.
Summary
This PR corrects several silent MSSQL ingestion failures and strengthens their coverage.
Diagram
%%{init: {'theme': 'neutral'}}%% flowchart TD A[MSSQL service connection] --> B{Driver} B -->|pyodbc| C[ODBC TLS switches] B -->|pytds| D[CA-backed TLS parameters] B -->|pymssql| E[Warn when connector TLS settings cannot be applied] C --> F[Engine with pre-ping] D --> F E --> F F --> G{Query timeout mechanism} G -->|pyodbc| H[DBAPI connect listener sets connection.timeout] G -->|pytds or pymssql| I[Connect argument supplies socket timeout] F --> J[Switch inspector to target database] J --> K[Load database-scoped descriptions] K --> L[Reflect tables, unique constraints, procedures, and views] L --> M[Classify indexed views as materialized views]Reviews (10) · Last reviewed commit: "test(mssql): assert the engine's behavio..."