tests.system.google.cloud.dataprep.example_dataprep¶

演示如何使用 Google Dataprep 的 Airflow DAG 示例。

此 DAG 依賴於以下作業系統環境變數

SYSTEM_TESTS_DATAPREP_TOKEN - Dataprep API 訪問令牌。有關生成方法，請參閱說明： https://docs.trifacta.com/display/DP/Manage+API+Access+Tokens#:~:text=Enable%20individual%20access-,Generate%20New%20Token,-Via%20UI。

屬性¶

`ENV_ID`
`DAG_ID`
`CONNECTION_ID`
`DATAPREP_TOKEN`
`GCP_PROJECT_ID`
`GCS_BUCKET_NAME`
`GCS_BUCKET_PATH`
`DATASET_URI`
`DATASET_NAME`
`DATASET_WRANGLED_NAME`
`DATASET_WRANGLED_ID`
`FLOW_ID`
`FLOW_COPY_ID`
`RECIPE_NAME`
`WRITE_SETTINGS`
`log`
`create_bucket_task`
`test_run`

模組內容¶

tests.system.google.cloud.dataprep.example_dataprep.ENV_ID[source]¶

tests.system.google.cloud.dataprep.example_dataprep.DAG_ID = 'dataprep'[source]¶

tests.system.google.cloud.dataprep.example_dataprep.CONNECTION_ID = ''[source]¶

tests.system.google.cloud.dataprep.example_dataprep.DATAPREP_TOKEN[source]¶

tests.system.google.cloud.dataprep.example_dataprep.GCP_PROJECT_ID[source]¶

tests.system.google.cloud.dataprep.example_dataprep.GCS_BUCKET_NAME = 'dataprep-bucket-dataprep-Uninferable'[source]¶

tests.system.google.cloud.dataprep.example_dataprep.GCS_BUCKET_PATH = 'gs://dataprep-bucket-dataprep-Uninferable/task_results/'[source]¶

tests.system.google.cloud.dataprep.example_dataprep.DATASET_URI = 'gs://airflow-system-tests-resources/dataprep/dataset-00000.parquet'[source]¶

tests.system.google.cloud.dataprep.example_dataprep.DATASET_NAME = ''[source]¶

tests.system.google.cloud.dataprep.example_dataprep.DATASET_WRANGLED_NAME = 'wrangled_'[source]¶

tests.system.google.cloud.dataprep.example_dataprep.DATASET_WRANGLED_ID = "{{ task_instance.xcom_pull('create_wrangled_dataset')['id'] }}"[source]¶

tests.system.google.cloud.dataprep.example_dataprep.FLOW_ID = "{{ task_instance.xcom_pull('create_flow')['id'] }}"[source]¶

tests.system.google.cloud.dataprep.example_dataprep.FLOW_COPY_ID = "{{ task_instance.xcom_pull('copy_flow')['id'] }}"[source]¶

tests.system.google.cloud.dataprep.example_dataprep.RECIPE_NAME = 'wrangled_'[source]¶

tests.system.google.cloud.dataprep.example_dataprep.WRITE_SETTINGS[source]¶

tests.system.google.cloud.dataprep.example_dataprep.log[source]¶

tests.system.google.cloud.dataprep.example_dataprep.create_bucket_task[source]¶

tests.system.google.cloud.dataprep.example_dataprep.test_run[source]¶

此條目有幫助嗎？