apache-airflow-providers-common-sql

變更日誌¶

1.26.0¶

特性¶

feat: integrate 'polars' in 'get_df', 'get_df_by_chunks' (#48875)

雜項¶

remove superfluous else block (#49199)

1.25.0¶

特性¶

Add requires_result_fetch Configuration to SQLExecuteQueryOperator (#46997)

1.24.1¶

Bug 修復¶

Fix get_hook() got an unexpected keyword argument 'hook_params' in SQLExecuteQueryTrigger (#47728)
Do not allow semicolon in partition_clause (#48098)

雜項¶

AIP-72: Handle Custom XCom Backend on Task SDK (#47339)
AIP-72: Implement short circuit and branch operators (#46584)

1.24.0¶

特性¶

Add deferred pagination mode to GenericTransfer (#44809)

Bug 修復¶

fIx deprecation warnings in common.sql (#47169)
Add missing 'methodtools' dependency of the 'common-sql' provider (#47148)

雜項¶

Upgrade flit to 3.11.0 (#46938)

1.23.0¶

注意

此版本沒有程式碼更改。釋出此版本是因為之前的版本因打包問題被撤回。

1.22.0¶

特性¶

Introduce notion of dialects in DbApiHook (#41327)

Bug 修復¶

Fix DbApiHook.insert_rows logging incorrect number of rows commited (#45789)
Fix escaping of special characters or reserved words as column names in dialects of common sql provider (#45640)

雜項¶

Add run_after column to DagRun model (#45732)
Adding type ignore to sql hook (#46163)
Added ADR document describing why the notion of dialects is introduced (#45456)
Bring back mypy checks for new-structure providers (#45815)
Remove obsolete pandas specfication for pre-python 3.9 (#45399)

1.21.0¶

注意

此版本的 provider 僅適用於 Airflow 2.9+，如 Apache Airflow provider 支援策略中所述。

警告

所有已棄用的類、引數和特性已從 Common SQL Provider 包中移除。可能存在非常舊的 Provider 版本依賴於 common.sql 的私有函式，導致使用者在不知情的情況下圍繞這些私有函式建立了自定義程式碼。這個問題早已修復，並且可能會影響使用非常舊的 Provider 版本的使用者。我們不認為這是一項破壞性更改，但出於可見性考慮，我們在此明確列出這些更改。引入了以下破壞性更改：

Hooks
- 從 DbApiHook 中移除 _make_serializable 方法。請改用 _make_common_data_structure。

特性¶

Added output_processor parameter to SQLQueryOperator and fixed bug with return_single_query_results handler when None is passed as split_statements (#44781)
Moved common SQL handler methods of common-sql-provider into dedicated module (#43747)

雜項¶

Bump minimum Airflow version in providers to Airflow 2.9.0 (#44956)
Fix static checks in common SQL hooks (#44930)
Added documentation for the SqlThresholdCheckOperator (#44787)
Remove Provider Deprecations in Common SQL (#44645)
Update DAG example links in multiple providers documents (#44034)

1.20.0¶

特性¶

Add support for semicolon stripping to DbApiHook, PrestoHook, and TrinoHook (#41916)

1.19.0¶

特性¶

Feature: Added fast_executemany parameter to insert_rows of DbApiHook (#43357)
Make conn id parameters templated in GenericTransfer and also allow passing hook parameters like in BaseSQLOperator (#42891)
Allow capture of replication lag in SqlSensor (#43107)

雜項¶

fix mypy failure (#43206)

1.18.0¶

特性¶

feat(providers/common/sql): add warning to connection setter (#42736)

Bug 修復¶

FIX: Only pass connection to sqlalchemy engine in JdbcHook (#42705)

1.17.1¶

Bug 修復¶

fix(providers/common/sql): add dummy connection setter for backward compatibility (#42490)
Changed type hinting for handler function (#42275)

1.17.0¶

特性¶

注意

DB Hook 中的連線現在被快取，以避免解析 extras 中的屬性時多次查詢。

Generalize caching of connection in DbApiHook to improve performance (#40751)

雜項¶

feat: log client db messages for provider postgres (#40171)
remove deprecated soft_fail from providers (#41710)

1.16.0¶

注意

此版本的 provider 僅適用於 Airflow 2.8+，如 Apache Airflow provider 支援策略中所述。

Bug 修復¶

fix: rm deprecated import (#41461)

雜項¶

Bump minimum Airflow version in providers to Airflow 2.8.0 (#41396)

1.15.0¶

特性¶

Create SQLAlchemy engine from connection in DB Hook and added autocommit param to insert_rows method (#40669)

1.14.2¶

Bug 修復¶

FIX: DbApiHook.insert_rows unnecessarily restarting connections (#40615)

雜項¶

Enable enforcing pydocstyle rule D213 in ruff. (#40448)

1.14.1¶

雜項¶

implement per-provider tests with lowest-direct dependency resolution (#39946)
Update pandas minimum requirement for Python 3.12 (#40272)
standardizes template fields for 'BaseSQLOperator' and adds 'database' as a templated field (#39826)

1.14.0¶

特性¶

Add 'parameters' as template field for SqlSensor (#39588)

Bug 修復¶

DbAPiHook: Don't log a warning message if placeholder is None and make sure warning message is formatted correctly (#39690)

雜項¶

refactor: The executemany parameter of insert_rows should not be deprecated as for some hooks we don't want to enable a system-wide supports_executemany parameter, that way we can also keep using it in dedicated situations (#39630)
Faster 'airflow_version' imports (#39552)
Simplify 'airflow_version' imports (#39497)
Add typing for SqlSensor (#39773)

1.13.0¶

注意

此版本的 provider 僅適用於 Airflow 2.7+，如 Apache Airflow provider 支援策略中所述。

特性¶

Add 'sqlalchemy_url' property to 'DbApiHook' class (#38871)
Always use the executemany method when inserting rows in DbApiHook as it's way much faster (#38715)

Bug 修復¶

Fix 'DbApiHook.insert_rows' when 'rows' is a generator (#38972)
Fix 'update-common-sql-api-stubs' pre-commit check (#38915)

雜項¶

Bump minimum Airflow version in providers to Airflow 2.7.0 (#39240)
openlineage, snowflake: do not run external queries for Snowflake (#39113)

1.12.0¶

特性¶

Add hook_params to template_fields for BaseSQLOperator-related Operators (#38724)
Make 'placeholder' of DbApiHook configurable in UI (#38528)

雜項¶

Undeprecating 'DBApiHookForTests._make_common_data_structure' (#38573)

1.11.1¶

Bug 修復¶

Make 'executemany' keyword arguments only in 'DbApiHook.insert_rows' (#37840)
Limit 'pandas' to '<2.2' (#37748)

1.11.0¶

特性¶

Enhancement: Performance enhancement for insert_rows method DbApiHook with fast executemany + SAP Hana support (#37246)

Bug 修復¶

Fix SQLThresholdCheckOperator error on falsey vals (#37150)

雜項¶

feat: Switch all class, functions, methods deprecations to decorators (#36876)
Add more-itertools as dependency of common-sql (#37359)

1.10.1¶

雜項¶

Set min pandas dependency to 1.2.5 for all providers and airflow (#36698)

1.10.0¶

Make "placeholder" of ODBC configurable in UI (#36000)

Bug 修復¶

Return common data structure in DBApi derived classes
SQLCheckOperator fails if returns dict with any False values (#36273)

1.9.0¶

注意

此版本的 provider 僅適用於 Airflow 2.6+，如 Apache Airflow provider 支援策略中所述。

雜項¶

Bump minimum Airflow version in providers to Airflow 2.6.0 (#36017)
Add Architecture Decision Record for common.sql introduction (#36015)

1.8.1¶

雜項¶

Add '_make_serializable' method which other SQL operators can overrides when result from cursor is not JSON-serializable (#32319)
Remove backcompat inheritance for DbApiHook (#35754)

1.8.0¶

注意

此版本的 provider 僅適用於 Airflow 2.5+，如 Apache Airflow provider 支援策略中所述。

Bug 修復¶

fix(providers/sql): respect soft_fail argument when exception is raised (#34199)

雜項¶

Bump min airflow version of providers (#34728)
Use 'airflow.exceptions.AirflowException' in providers (#34511)

1.7.2¶

Bug 修復¶

Fix BigQueryValueCheckOperator deferrable mode optimisation (#34018)

雜項¶

Refactor regex in providers (#33898)

1.7.1¶

雜項¶

Refactor: Better percentage formatting (#33595)
Refactor: Simplify code in smaller providers (#33234)

1.7.0¶

特性¶

Add a new parameter to SQL operators to specify conn id field (#30784)

1.6.2¶

雜項¶

Make SQLExecute Query signature consistent with other SQL operators (#32974)
Get rid of Python2 numeric relics (#33050)

1.6.1¶

Bug 修復¶

Fix local OpenLineage import in 'SQLExecuteQueryOperator'. (#32400)

雜項¶

Add default port to Openlineage authority method. (#32828)
Add more accurate typing for DbApiHook.run method (#31846)

1.6.0¶

特性¶

openlineage, common.sql: provide OL SQL parser as internal OpenLineage provider API (#31398)

雜項¶

Pass SQLAlchemy engine to construct information schema query. (#32371)

1.5.2¶

雜項¶

Remove Python 3.7 support (#30963)

1.5.1¶

雜項¶

Bring back min-airflow-version for preinstalled providers (#31469)

1.5.0 (YANKED)¶

警告

此版本已被 撤回 (yanked)，原因：此版本可能導致舊版 airflow 的無限制安裝，從而引發執行時錯誤。

注意

此版本的 provider 僅適用於 Airflow 2.4+，如 Apache Airflow provider 支援策略中所述。

特性¶

Add conditional output processing in SQL operators (#31136)

雜項¶

Remove noisy log from SQL table check (#31037)

1.4.0¶

特性¶

新增選項以在日誌中顯示 'SQLExecuteQueryOperator' 的輸出 (#29954)

雜項¶

修復 Common SQL provider 的 Python API 文件格式 (#29863)

1.3.4¶

錯誤修復¶

當 do_xcom_push=False 時不處理輸出 (#29599)

1.3.3¶

錯誤修復¶

處理 Exasol cursor 的非相容行為 (#28744)

1.3.2¶

錯誤修復¶

修復最近 isort 版本引入的 isort 問題 (#28434)
修復 Common SQL operators 的模板渲染 (#28202)
在 SQLExecuteQueryOperator 中遵從 hook setting 處理 split_statements (#28635)

雜項¶

澄清更新的 DbApiHook 的 docstrings (#27966)

1.3.1¶

此版本修復了重構通用部分時在 common.sql operator 中引入的一些錯誤

SQLExecuteQueryOperator 中的 _process_output 方法現在具有一致的語義和型別，它還可以修改從 SQLExecuteQueryOperator 派生的 operators 中的返回值（並存儲在 XCom 中）。
所有返回結果的描述都作為 DBApiHook 中的 descriptions 屬性儲存
遊標的最後描述（無論是否返回單查詢結果值）現在透過 DBApiHook 中的 last_description 屬性公開。

操作員缺乏一致性導致 1.3.0 被撤下（yanked）—— 不應使用 1.3.0 —— 如果您安裝了 1.3.0，請升級到 1.3.1。

錯誤修復¶

恢復在 common.sql 中已刪除（但仍在使用）的方法 (#27843)
修復重構時在 Databricks SQL operator 中引入的錯誤 (#27854)
將 common.sql provider 版本提升至 1.3.1 (#27888)
最終修復 SQL Hooks 和 Operators 的行為 (#27912)

1.3.0 (已撤下)¶

警告

此版本已被 撤下 (yanked)，原因是：導致 Google 8.4.0 provider 中的 SQLExecute 無法工作

注意

此 provider 版本僅適用於 Airflow 2.3+，如 Apache Airflow provider 支援策略中所述。

雜項¶

將所有 provider 的最低 Airflow 版本移至 2.3.0 (#27196)

特性¶

新增 SQLExecuteQueryOperator (#25717)
DbApiHook.get_records 和 DbApiHook.get_first 使用 DbApiHook.run (#26944)
DbApiHook 一致的 insert_rows 日誌記錄 (#26758)

錯誤修復¶

Common sql bugfixes 和改進 (#26761)
在 SQLCheckOperator.execute 中使用未使用的 SQLCheckOperator.parameters (#27599)

1.2.0¶

特性¶

使 placeholder 樣式可配置 (#25939)
為 pre-common-sql providers 提供更好的錯誤訊息 (#26051)

錯誤修復¶

修復（並測試）postgresql 上的 SQLTableCheckOperator (#25821)
SQLTableCheckOperator 不使用 Pandas (#25822)
在 SQL hook 中放棄分號去除 (#25855)

1.1.0¶

特性¶

使用 ParamSpec 改進 taskflow 型別提示 (#25173)
將所有“舊”SQL operators 移至 common.sql providers (#25350)
廢棄 hql parameters 並同步 DBApiHook 方法 APIs (#25299)
將 DbApiHook.run() 方法與覆蓋它的方法統一 (#23971)
Common SQLCheckOperators 多項功能更新 (#25164)

錯誤修復¶

允許 Legacy SqlSensor 使用 common.sql providers (#25293)
修復 fetch_all_handler 和 db-api 測試 (#25430)
對齊 Common SQL provider 標誌位置 (#25538)
修復 SQL split string 以包含 ';-less' 語句 (#25713)

1.0.0¶

provider 的初始版本。新增 SQLColumnCheckOperator 和 SQLTableCheckOperator。將 DBApiHook、SQLSensor 和 ConnectorProtocol 移至 provider。