Skip to content

FIX: Restore bulk copy after upgrading from 1.15.0 - #860

Open
Jahnvi Thakkar (jahnvi480) wants to merge 3 commits into
mainfrom
jahnvi/bulkcopy-packaging-triage
Open

Jahnvi Thakkar (jahnvi480) wants to merge 3 commits into
mainfrom
jahnvi/bulkcopy-packaging-triage

Conversation

@jahnvi480

Copy link
Copy Markdown
Contributor

Work Item / Issue Reference

N/A

GitHub Issue: #855


Summary

Load mssql_py_core.mssql_py_core directly for both bulk-copy methods and the internal async loader. During a 1.15.0 upgrade, pip installs the new Rust dependency before uninstalling the old driver; the old uninstall removes the initializer shared by both distributions. The native extension remains available, but top-level re-exports disappear.

Add regression cases for a namespace-only parent package and missing native dependencies, update existing bulk-copy mocks, and document the repair command in the changelog. Package ownership and public APIs remain unchanged.

Validation on Windows x64 / Python 3.13:

  • Rebuilt the native extension successfully with CMake in a shorter build directory; the standard build directory exceeded Windows MAX_PATH.
  • python -m pytest -q tests\test_008_auth.py tests\test_019_bulkcopy.py tests\test_020_bulkcopy_auth_cleanup.py tests\test_024_bulkcopy_arrow.py tests\AsyncTest --tb=short: 511 passed, including live SQL Server coverage.
  • python -m pre_commit run black-check --all-files --hook-stage pre-push: passed.
  • Built and installed the patched wheel in disposable environments: fresh installation and upgrade from 1.15.0 both passed live row bulk copy, Arrow bulk copy, and async SQL. The upgrade scenario explicitly confirmed that the initializer was missing.
  • Full repository suite and other OS/architecture matrices were not run locally.

Fixes: #855

Load the nested Rust extension directly when an upgrade removes its package initializer. Cover namespace-only parents and missing native dependencies for bulk copy and async queries.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 22, 2026 07:02
@github-actions github-actions Bot added the pr-size: medium Moderate update size label Oct 22, 2026
@github-actions

github-actions Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

PR Performance Report

✅ No regression detected

No consistent slowdowns detected across all 2 environments.

0 IMPROVEMENTS 0 SLOWDOWNS 2/2 ENVIRONMENTS

Coverage: 2 of 2 environments completed. Advisory result; does not block merging.

Performance diagnostics

Phase times are inclusive diagnostics and must not be added together. They identify where measured time changed, not why it changed.

No affected phases or call-count changes were recorded.

All database tasks and timings

Unix / SQL Server 2022

Database task Before After Paired change Result
Connection opening 10.354 ms 10.354 ms -0.8% no signal
SELECT queries 1.063 ms 1.071 ms +1.5% no signal
Row insertion 34.421 ms 34.093 ms -1.0% no signal
Executemany inserts 155.031 ms 155.999 ms -0.2% no signal
Fetch-all queries 119.621 ms 119.106 ms -0.8% no signal
Row-by-row fetching 14.663 ms 14.435 ms -1.1% no signal
Batched row fetching 116.475 ms 116.981 ms +0.4% no signal
Transaction commit and rollback 114.828 ms 113.656 ms -0.6% no signal
Arrow row fetching 95.100 ms 93.545 ms -1.0% no signal
100,000-row insertion 437.150 ms 441.815 ms +0.5% no signal
Row fetching in batches of 100 120.726 ms 120.519 ms -0.2% no signal
Row fetching in batches of 10,000 122.393 ms 134.356 ms +7.8% no signal
Repeated positional queries 33.577 ms 33.595 ms +1.3% no signal
Repeated named-parameter queries 35.992 ms 35.838 ms -0.1% no signal
Legacy 100,000-row insertion 347.039 ms 346.269 ms +0.2% no signal
Insertion with explicit input sizes 481.287 ms 480.873 ms +0.4% no signal
Joined aggregation queries 176.674 ms 177.488 ms +0.6% no signal
Large joined-result fetching 179.085 ms 178.939 ms +0.2% no signal
1.2-million-row fetching 3402.819 ms 3387.580 ms -0.4% no signal
Common table expression queries 5.394 ms 5.290 ms -1.4% no signal
256 KiB VARCHAR(MAX) / fetchall() 1.250 ms 1.222 ms -1.3% no signal
10,000 scalar values / fetchval() (debug disabled) 107.550 ms 107.043 ms -0.1% no signal

Unix / SQL Server 2025

Database task Before After Paired change Result
Connection opening 98.283 ms 98.672 ms -0.0% no signal
SELECT queries 1.078 ms 1.074 ms -1.2% no signal
Row insertion 34.092 ms 35.671 ms +3.0% no signal
Executemany inserts 158.826 ms 160.557 ms +1.1% no signal
Fetch-all queries 122.894 ms 121.073 ms -1.5% no signal
Row-by-row fetching 14.549 ms 14.795 ms +1.9% no signal
Batched row fetching 120.797 ms 118.860 ms -1.6% no signal
Transaction commit and rollback 114.308 ms 114.179 ms -0.1% no signal
Arrow row fetching 93.962 ms 94.968 ms -2.4% no signal
100,000-row insertion 463.574 ms 437.635 ms -1.2% no signal
Row fetching in batches of 100 122.482 ms 123.607 ms -1.0% no signal
Row fetching in batches of 10,000 133.854 ms 140.809 ms +3.5% no signal
Repeated positional queries 34.036 ms 33.933 ms +0.0% no signal
Repeated named-parameter queries 36.408 ms 36.887 ms +2.4% no signal
Legacy 100,000-row insertion 358.202 ms 389.266 ms +14.1% no signal
Insertion with explicit input sizes 494.352 ms 497.016 ms +3.4% no signal
Joined aggregation queries 162.905 ms 163.109 ms +0.1% no signal
Large joined-result fetching 190.562 ms 187.744 ms +0.1% no signal
1.2-million-row fetching 3561.930 ms 3543.087 ms -0.7% no signal
Common table expression queries 5.256 ms 5.135 ms -3.2% no signal
256 KiB VARCHAR(MAX) / fetchall() 1.509 ms 1.532 ms +3.8% no signal
10,000 scalar values / fetchval() (debug disabled) 108.199 ms 108.945 ms -1.7% no signal
Build and measurement details

ADO build 184973

PR head: ed7e3a51872dcc820325a2a6930a8256ff61c6c7
Base: bd0aaf89c724a92c934b68bca41469655bfc733c
Measured merge: 8d5523ee70e111afe3928b4d24094d3450ef1d1b

  • Unix / SQL Server 2022: Python 3.12.3, x86_64, SQL 16.0.4305.5; 5 paired comparisons and 1 warmup.
  • Unix / SQL Server 2025: Python 3.12.3, x86_64, SQL 17.0.5015.4; 5 paired comparisons and 1 warmup.

A consistent change requires more than 20% median paired movement, at least 1 ms between the median runtimes, and at least 80% of pairs exceeding the relative threshold in the same direction. A slowdown without enough pair agreement is reported as inconsistent.

The displayed change is the median of paired before-and-after ratios. It is not recalculated from the two displayed median runtimes.

Both revisions use profiling-enabled builds on the same agent and database, with alternating order and discarded warmups. Results are diagnostic and do not represent production-wheel latency.

Raw samples and logs are attached to the ADO run as profiler-* artifacts.

Comment thread tests/test_020_bulkcopy_auth_cleanup.py Fixed
Comment thread tests/test_020_bulkcopy_auth_cleanup.py Fixed
Replace localhost with example.test in the two import regression tests to address DevSkim comments without suppressing the rule.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Reported Windows validation supports the fix, but cross-platform native-extension upgrade behavior remains unverified and warrants final human review.

No open issues 🎉

What changed in this PR

Addresses #855 by loading the Rust extension directly, allowing bulk copy and internal async queries to work when an upgrade from 1.15.0 removes the parent package initializer. Public APIs and package ownership remain unchanged.

Changes:

  • Switch bulk-copy and async loaders to mssql_py_core.mssql_py_core.
  • Add namespace-package and missing-dependency regression tests, and update existing mocks.
  • Document the initializer repair command.
File Description
tests/​test_024_bulkcopy_arrow.py Update Arrow dependency imports and mocks.
tests/​test_020_bulkcopy_auth_cleanup.py Add loader regression tests and update authentication mocks.
tests/​test_008_auth.py Target the native submodule in the bulk-copy mock.
tests/​AsyncTest/​test_001_async_query_native.py Verify direct loading and namespace-package support.
mssql_python/​cursor.py Load the native submodule for both bulk-copy methods.
mssql_python/​async_query/​_native.py Load the native submodule for internal async operations.
CHANGELOG.md Describe the upgrade fix and repair command.

Review effort: Balanced

Copilot AI balanced review requested due to automatic review settings October 22, 2026 07:04

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Reported Windows validation supports the fix, but native-loading and upgrade behavior on other supported platforms remain unverified.

No open issues 🎉

Review effort: Balanced

@github-actions

github-actions Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

📊 Code Coverage Report

🔥 Diff Coverage

100%


🎯 Overall Coverage

85%


📈 Total Lines Covered: 9444 out of 11093
📁 Project: mssql-python


Diff Coverage

Diff: main...HEAD, staged and unstaged changes

  • mssql_python/async_query/_native.py (100%)
  • mssql_python/cursor.py (100%)

Summary

  • Total: 3 lines
  • Missing: 0 lines
  • Coverage: 100%

📋 Files Needing Attention

📉 Files with overall lowest coverage (click to expand)
mssql_python.pybind.performance_counter.hpp: 0.7%
mssql_python.pybind.logger_bridge.cpp: 57.9%
mssql_python.pybind.ddbc_bindings.h: 62.6%
mssql_python.pybind.logger_bridge.hpp: 70.8%
mssql_python.pybind.connection.connection_pool.cpp: 82.3%
mssql_python.pybind.connection.connection.cpp: 83.1%
mssql_python.logging.py: 86.2%
mssql_python.pooling.py: 90.1%
mssql_python.pybind.fetch_temporal.hpp: 92.1%
mssql_python.cursor.py: 92.5%

🔗 Quick Links

⚙️ Build Summary 📋 Coverage Details

View Azure DevOps Build

Browse Full Coverage Report

Copilot AI balanced review requested due to automatic review settings October 22, 2026 08:53

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Reported Windows validation supports the fix, but cross-platform confirmation of the packaging-sensitive import change remains outstanding.

No open issues 🎉

Review effort: Balanced

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the PR for correctness, security, reliability, performance, test coverage, repository conventions, and applicable architecture and design specifications. No actionable issues were identified. The implementation is consistent with repository standards and the applicable approved design requirements.

Recommendation: Approve

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-size: medium Moderate update size

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bulkcopy_arrow() / bulkcopy()` broken in 1.16.0

4 participants