Metadata-Version: 2.1
Name: acryl-datahub
Version: 0.1.3
Summary: A CLI to work with DataHub metadata
Home-page: https://datahubproject.io/
Author: DataHub Committers
License: Apache License 2.0
Project-URL: Documentation, https://datahubproject.io/docs/
Project-URL: Source, https://github.com/linkedin/datahub
Project-URL: Changelog, https://github.com/linkedin/datahub/releases
Platform: UNKNOWN
Classifier: Development Status :: 5 - Production/Stable
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Information Technology
Classifier: Intended Audience :: System Administrators
Classifier: License :: OSI Approved
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: Unix
Classifier: Operating System :: POSIX :: Linux
Classifier: Environment :: Console
Classifier: Environment :: MacOS X
Classifier: Topic :: Software Development
Requires-Python: >=3.6
Description-Content-Type: text/markdown
Requires-Dist: avro-gen3 (==0.4.1)
Requires-Dist: avro-python3 (>=1.8.2)
Requires-Dist: python-dateutil
Requires-Dist: click (>=6.0.0)
Requires-Dist: expandvars (>=0.6.5)
Requires-Dist: docker
Requires-Dist: typing-inspect
Requires-Dist: mypy-extensions (>=0.4.3)
Requires-Dist: toml (>=0.10.0)
Requires-Dist: PyYAML
Requires-Dist: pydantic (>=1.5.1)
Requires-Dist: dataclasses (>=0.6) ; python_version < "3.7"
Requires-Dist: typing-extensions (>=3.7.4) ; python_version < "3.8"
Provides-Extra: airflow
Requires-Dist: apache-airflow (>=1.10.2) ; extra == 'airflow'
Requires-Dist: avro-python3 (>=1.8.2) ; extra == 'airflow'
Requires-Dist: python-dateutil ; extra == 'airflow'
Requires-Dist: click (>=6.0.0) ; extra == 'airflow'
Requires-Dist: expandvars (>=0.6.5) ; extra == 'airflow'
Requires-Dist: docker ; extra == 'airflow'
Requires-Dist: avro-gen3 (==0.4.1) ; extra == 'airflow'
Requires-Dist: toml (>=0.10.0) ; extra == 'airflow'
Requires-Dist: PyYAML ; extra == 'airflow'
Provides-Extra: all
Requires-Dist: cx-Oracle ; extra == 'all'
Requires-Dist: boto3 ; extra == 'all'
Requires-Dist: apache-airflow (>=1.10.2) ; extra == 'all'
Requires-Dist: sqlalchemy (>=1.3.24) ; extra == 'all'
Requires-Dist: pydruid (>=0.6.2) ; extra == 'all'
Requires-Dist: psycopg2-binary ; extra == 'all'
Requires-Dist: click (>=6.0.0) ; extra == 'all'
Requires-Dist: expandvars (>=0.6.5) ; extra == 'all'
Requires-Dist: GeoAlchemy2 ; extra == 'all'
Requires-Dist: requests ; extra == 'all'
Requires-Dist: PyAthena[sqlalchemy] ; extra == 'all'
Requires-Dist: docker ; extra == 'all'
Requires-Dist: pyhive[hive] ; extra == 'all'
Requires-Dist: sqlalchemy-pytds (>=0.3) ; extra == 'all'
Requires-Dist: pymysql (>=1.0.2) ; extra == 'all'
Requires-Dist: toml (>=0.10.0) ; extra == 'all'
Requires-Dist: python-ldap (>=2.4) ; extra == 'all'
Requires-Dist: pybigquery (>=0.6.0) ; extra == 'all'
Requires-Dist: avro-python3 (>=1.8.2) ; extra == 'all'
Requires-Dist: python-dateutil ; extra == 'all'
Requires-Dist: fastavro (>=1.2.0) ; extra == 'all'
Requires-Dist: snowflake-sqlalchemy ; extra == 'all'
Requires-Dist: pymongo (>=3.11) ; extra == 'all'
Requires-Dist: avro-gen3 (==0.4.1) ; extra == 'all'
Requires-Dist: confluent-kafka (>=1.5.0) ; extra == 'all'
Requires-Dist: PyYAML ; extra == 'all'
Provides-Extra: athena
Requires-Dist: sqlalchemy (>=1.3.24) ; extra == 'athena'
Requires-Dist: avro-python3 (>=1.8.2) ; extra == 'athena'
Requires-Dist: python-dateutil ; extra == 'athena'
Requires-Dist: click (>=6.0.0) ; extra == 'athena'
Requires-Dist: expandvars (>=0.6.5) ; extra == 'athena'
Requires-Dist: PyAthena[sqlalchemy] ; extra == 'athena'
Requires-Dist: docker ; extra == 'athena'
Requires-Dist: avro-gen3 (==0.4.1) ; extra == 'athena'
Requires-Dist: toml (>=0.10.0) ; extra == 'athena'
Requires-Dist: PyYAML ; extra == 'athena'
Provides-Extra: base
Requires-Dist: avro-python3 (>=1.8.2) ; extra == 'base'
Requires-Dist: python-dateutil ; extra == 'base'
Requires-Dist: click (>=6.0.0) ; extra == 'base'
Requires-Dist: expandvars (>=0.6.5) ; extra == 'base'
Requires-Dist: docker ; extra == 'base'
Requires-Dist: avro-gen3 (==0.4.1) ; extra == 'base'
Requires-Dist: toml (>=0.10.0) ; extra == 'base'
Requires-Dist: PyYAML ; extra == 'base'
Provides-Extra: bigquery
Requires-Dist: pybigquery (>=0.6.0) ; extra == 'bigquery'
Requires-Dist: sqlalchemy (>=1.3.24) ; extra == 'bigquery'
Requires-Dist: avro-python3 (>=1.8.2) ; extra == 'bigquery'
Requires-Dist: python-dateutil ; extra == 'bigquery'
Requires-Dist: click (>=6.0.0) ; extra == 'bigquery'
Requires-Dist: expandvars (>=0.6.5) ; extra == 'bigquery'
Requires-Dist: docker ; extra == 'bigquery'
Requires-Dist: avro-gen3 (==0.4.1) ; extra == 'bigquery'
Requires-Dist: toml (>=0.10.0) ; extra == 'bigquery'
Requires-Dist: PyYAML ; extra == 'bigquery'
Provides-Extra: datahub-kafka
Requires-Dist: confluent-kafka (>=1.5.0) ; extra == 'datahub-kafka'
Requires-Dist: avro-python3 (>=1.8.2) ; extra == 'datahub-kafka'
Requires-Dist: python-dateutil ; extra == 'datahub-kafka'
Requires-Dist: click (>=6.0.0) ; extra == 'datahub-kafka'
Requires-Dist: fastavro (>=1.2.0) ; extra == 'datahub-kafka'
Requires-Dist: expandvars (>=0.6.5) ; extra == 'datahub-kafka'
Requires-Dist: docker ; extra == 'datahub-kafka'
Requires-Dist: avro-gen3 (==0.4.1) ; extra == 'datahub-kafka'
Requires-Dist: toml (>=0.10.0) ; extra == 'datahub-kafka'
Requires-Dist: PyYAML ; extra == 'datahub-kafka'
Provides-Extra: datahub-rest
Requires-Dist: avro-python3 (>=1.8.2) ; extra == 'datahub-rest'
Requires-Dist: python-dateutil ; extra == 'datahub-rest'
Requires-Dist: click (>=6.0.0) ; extra == 'datahub-rest'
Requires-Dist: expandvars (>=0.6.5) ; extra == 'datahub-rest'
Requires-Dist: requests ; extra == 'datahub-rest'
Requires-Dist: docker ; extra == 'datahub-rest'
Requires-Dist: avro-gen3 (==0.4.1) ; extra == 'datahub-rest'
Requires-Dist: toml (>=0.10.0) ; extra == 'datahub-rest'
Requires-Dist: PyYAML ; extra == 'datahub-rest'
Provides-Extra: dev
Requires-Dist: boto3 ; extra == 'dev'
Requires-Dist: sqlalchemy-stubs ; extra == 'dev'
Requires-Dist: black (>=19.10b0) ; extra == 'dev'
Requires-Dist: sqlalchemy (>=1.3.24) ; extra == 'dev'
Requires-Dist: apache-airflow (>=1.10.2) ; extra == 'dev'
Requires-Dist: build ; extra == 'dev'
Requires-Dist: click (>=6.0.0) ; extra == 'dev'
Requires-Dist: expandvars (>=0.6.5) ; extra == 'dev'
Requires-Dist: flake8 (>=3.8.3) ; extra == 'dev'
Requires-Dist: sqlalchemy-pytds (>=0.3) ; extra == 'dev'
Requires-Dist: docker ; extra == 'dev'
Requires-Dist: requests ; extra == 'dev'
Requires-Dist: apache-airflow (==1.10.15) ; extra == 'dev'
Requires-Dist: pymysql (>=1.0.2) ; extra == 'dev'
Requires-Dist: apache-airflow-backport-providers-snowflake ; extra == 'dev'
Requires-Dist: twine ; extra == 'dev'
Requires-Dist: typing-inspect ; extra == 'dev'
Requires-Dist: mypy-extensions (>=0.4.3) ; extra == 'dev'
Requires-Dist: toml (>=0.10.0) ; extra == 'dev'
Requires-Dist: pydantic (>=1.5.1) ; extra == 'dev'
Requires-Dist: isort (>=5.7.0) ; extra == 'dev'
Requires-Dist: deepdiff ; extra == 'dev'
Requires-Dist: python-ldap (>=2.4) ; extra == 'dev'
Requires-Dist: pybigquery (>=0.6.0) ; extra == 'dev'
Requires-Dist: avro-python3 (>=1.8.2) ; extra == 'dev'
Requires-Dist: pytest-cov (>=2.8.1) ; extra == 'dev'
Requires-Dist: freezegun ; extra == 'dev'
Requires-Dist: python-dateutil ; extra == 'dev'
Requires-Dist: fastavro (>=1.2.0) ; extra == 'dev'
Requires-Dist: mypy (>=0.782) ; extra == 'dev'
Requires-Dist: pymongo (>=3.11) ; extra == 'dev'
Requires-Dist: pytest (>=6.2.2) ; extra == 'dev'
Requires-Dist: coverage (>=5.1) ; extra == 'dev'
Requires-Dist: pytest-docker ; extra == 'dev'
Requires-Dist: avro-gen3 (==0.4.1) ; extra == 'dev'
Requires-Dist: confluent-kafka (>=1.5.0) ; extra == 'dev'
Requires-Dist: PyYAML ; extra == 'dev'
Requires-Dist: dataclasses (>=0.6) ; (python_version < "3.7") and extra == 'dev'
Requires-Dist: typing-extensions (>=3.7.4) ; (python_version < "3.8") and extra == 'dev'
Provides-Extra: druid
Requires-Dist: sqlalchemy (>=1.3.24) ; extra == 'druid'
Requires-Dist: avro-python3 (>=1.8.2) ; extra == 'druid'
Requires-Dist: pydruid (>=0.6.2) ; extra == 'druid'
Requires-Dist: python-dateutil ; extra == 'druid'
Requires-Dist: click (>=6.0.0) ; extra == 'druid'
Requires-Dist: expandvars (>=0.6.5) ; extra == 'druid'
Requires-Dist: docker ; extra == 'druid'
Requires-Dist: avro-gen3 (==0.4.1) ; extra == 'druid'
Requires-Dist: toml (>=0.10.0) ; extra == 'druid'
Requires-Dist: PyYAML ; extra == 'druid'
Provides-Extra: glue
Requires-Dist: boto3 ; extra == 'glue'
Requires-Dist: avro-python3 (>=1.8.2) ; extra == 'glue'
Requires-Dist: python-dateutil ; extra == 'glue'
Requires-Dist: click (>=6.0.0) ; extra == 'glue'
Requires-Dist: expandvars (>=0.6.5) ; extra == 'glue'
Requires-Dist: docker ; extra == 'glue'
Requires-Dist: avro-gen3 (==0.4.1) ; extra == 'glue'
Requires-Dist: toml (>=0.10.0) ; extra == 'glue'
Requires-Dist: PyYAML ; extra == 'glue'
Provides-Extra: hive
Requires-Dist: sqlalchemy (>=1.3.24) ; extra == 'hive'
Requires-Dist: avro-python3 (>=1.8.2) ; extra == 'hive'
Requires-Dist: python-dateutil ; extra == 'hive'
Requires-Dist: click (>=6.0.0) ; extra == 'hive'
Requires-Dist: expandvars (>=0.6.5) ; extra == 'hive'
Requires-Dist: docker ; extra == 'hive'
Requires-Dist: pyhive[hive] ; extra == 'hive'
Requires-Dist: avro-gen3 (==0.4.1) ; extra == 'hive'
Requires-Dist: toml (>=0.10.0) ; extra == 'hive'
Requires-Dist: PyYAML ; extra == 'hive'
Provides-Extra: kafka
Requires-Dist: confluent-kafka (>=1.5.0) ; extra == 'kafka'
Requires-Dist: avro-python3 (>=1.8.2) ; extra == 'kafka'
Requires-Dist: python-dateutil ; extra == 'kafka'
Requires-Dist: click (>=6.0.0) ; extra == 'kafka'
Requires-Dist: fastavro (>=1.2.0) ; extra == 'kafka'
Requires-Dist: expandvars (>=0.6.5) ; extra == 'kafka'
Requires-Dist: docker ; extra == 'kafka'
Requires-Dist: avro-gen3 (==0.4.1) ; extra == 'kafka'
Requires-Dist: toml (>=0.10.0) ; extra == 'kafka'
Requires-Dist: PyYAML ; extra == 'kafka'
Provides-Extra: ldap
Requires-Dist: avro-python3 (>=1.8.2) ; extra == 'ldap'
Requires-Dist: python-dateutil ; extra == 'ldap'
Requires-Dist: click (>=6.0.0) ; extra == 'ldap'
Requires-Dist: expandvars (>=0.6.5) ; extra == 'ldap'
Requires-Dist: docker ; extra == 'ldap'
Requires-Dist: avro-gen3 (==0.4.1) ; extra == 'ldap'
Requires-Dist: toml (>=0.10.0) ; extra == 'ldap'
Requires-Dist: PyYAML ; extra == 'ldap'
Requires-Dist: python-ldap (>=2.4) ; extra == 'ldap'
Provides-Extra: mongodb
Requires-Dist: avro-python3 (>=1.8.2) ; extra == 'mongodb'
Requires-Dist: python-dateutil ; extra == 'mongodb'
Requires-Dist: click (>=6.0.0) ; extra == 'mongodb'
Requires-Dist: expandvars (>=0.6.5) ; extra == 'mongodb'
Requires-Dist: pymongo (>=3.11) ; extra == 'mongodb'
Requires-Dist: docker ; extra == 'mongodb'
Requires-Dist: avro-gen3 (==0.4.1) ; extra == 'mongodb'
Requires-Dist: toml (>=0.10.0) ; extra == 'mongodb'
Requires-Dist: PyYAML ; extra == 'mongodb'
Provides-Extra: mssql
Requires-Dist: sqlalchemy (>=1.3.24) ; extra == 'mssql'
Requires-Dist: avro-python3 (>=1.8.2) ; extra == 'mssql'
Requires-Dist: python-dateutil ; extra == 'mssql'
Requires-Dist: click (>=6.0.0) ; extra == 'mssql'
Requires-Dist: expandvars (>=0.6.5) ; extra == 'mssql'
Requires-Dist: sqlalchemy-pytds (>=0.3) ; extra == 'mssql'
Requires-Dist: docker ; extra == 'mssql'
Requires-Dist: avro-gen3 (==0.4.1) ; extra == 'mssql'
Requires-Dist: toml (>=0.10.0) ; extra == 'mssql'
Requires-Dist: PyYAML ; extra == 'mssql'
Provides-Extra: mysql
Requires-Dist: sqlalchemy (>=1.3.24) ; extra == 'mysql'
Requires-Dist: avro-python3 (>=1.8.2) ; extra == 'mysql'
Requires-Dist: python-dateutil ; extra == 'mysql'
Requires-Dist: click (>=6.0.0) ; extra == 'mysql'
Requires-Dist: expandvars (>=0.6.5) ; extra == 'mysql'
Requires-Dist: docker ; extra == 'mysql'
Requires-Dist: pymysql (>=1.0.2) ; extra == 'mysql'
Requires-Dist: avro-gen3 (==0.4.1) ; extra == 'mysql'
Requires-Dist: toml (>=0.10.0) ; extra == 'mysql'
Requires-Dist: PyYAML ; extra == 'mysql'
Provides-Extra: oracle
Requires-Dist: cx-Oracle ; extra == 'oracle'
Requires-Dist: sqlalchemy (>=1.3.24) ; extra == 'oracle'
Requires-Dist: avro-python3 (>=1.8.2) ; extra == 'oracle'
Requires-Dist: python-dateutil ; extra == 'oracle'
Requires-Dist: click (>=6.0.0) ; extra == 'oracle'
Requires-Dist: expandvars (>=0.6.5) ; extra == 'oracle'
Requires-Dist: docker ; extra == 'oracle'
Requires-Dist: avro-gen3 (==0.4.1) ; extra == 'oracle'
Requires-Dist: toml (>=0.10.0) ; extra == 'oracle'
Requires-Dist: PyYAML ; extra == 'oracle'
Provides-Extra: postgres
Requires-Dist: sqlalchemy (>=1.3.24) ; extra == 'postgres'
Requires-Dist: avro-python3 (>=1.8.2) ; extra == 'postgres'
Requires-Dist: python-dateutil ; extra == 'postgres'
Requires-Dist: click (>=6.0.0) ; extra == 'postgres'
Requires-Dist: psycopg2-binary ; extra == 'postgres'
Requires-Dist: expandvars (>=0.6.5) ; extra == 'postgres'
Requires-Dist: GeoAlchemy2 ; extra == 'postgres'
Requires-Dist: docker ; extra == 'postgres'
Requires-Dist: avro-gen3 (==0.4.1) ; extra == 'postgres'
Requires-Dist: toml (>=0.10.0) ; extra == 'postgres'
Requires-Dist: PyYAML ; extra == 'postgres'
Provides-Extra: snowflake
Requires-Dist: sqlalchemy (>=1.3.24) ; extra == 'snowflake'
Requires-Dist: avro-python3 (>=1.8.2) ; extra == 'snowflake'
Requires-Dist: python-dateutil ; extra == 'snowflake'
Requires-Dist: click (>=6.0.0) ; extra == 'snowflake'
Requires-Dist: expandvars (>=0.6.5) ; extra == 'snowflake'
Requires-Dist: snowflake-sqlalchemy ; extra == 'snowflake'
Requires-Dist: docker ; extra == 'snowflake'
Requires-Dist: avro-gen3 (==0.4.1) ; extra == 'snowflake'
Requires-Dist: toml (>=0.10.0) ; extra == 'snowflake'
Requires-Dist: PyYAML ; extra == 'snowflake'
Provides-Extra: sqlalchemy
Requires-Dist: sqlalchemy (>=1.3.24) ; extra == 'sqlalchemy'
Requires-Dist: avro-python3 (>=1.8.2) ; extra == 'sqlalchemy'
Requires-Dist: python-dateutil ; extra == 'sqlalchemy'
Requires-Dist: click (>=6.0.0) ; extra == 'sqlalchemy'
Requires-Dist: expandvars (>=0.6.5) ; extra == 'sqlalchemy'
Requires-Dist: docker ; extra == 'sqlalchemy'
Requires-Dist: avro-gen3 (==0.4.1) ; extra == 'sqlalchemy'
Requires-Dist: toml (>=0.10.0) ; extra == 'sqlalchemy'
Requires-Dist: PyYAML ; extra == 'sqlalchemy'

# DataHub Metadata Ingestion

![Python version 3.6+](https://img.shields.io/badge/python-3.6%2B-blue)

This module hosts an extensible Python-based metadata ingestion system for DataHub.
This supports sending data to DataHub using Kafka or through the REST API.
It can be used through our CLI tool, with an orchestrator like Airflow, or as a library.

## Getting Started

### Prerequisites

Before running any metadata ingestion job, you should make sure that DataHub backend services are all running. If you are trying this out locally, the easiest way to do that is through [quickstart Docker images](../docker).

### Install from PyPI

The folks over at [Acryl](https://www.acryl.io/) maintain a PyPI package for DataHub metadata ingestion.

```sh
# Requires Python 3.6+
python3 -m pip install --upgrade pip wheel setuptools
python3 -m pip uninstall datahub acryl-datahub || true  # sanity check - ok if it fails
python3 -m pip install --upgrade acryl-datahub
datahub version
# If you see "command not found", try running this instead: python3 -m datahub version
```

If you run into an error, try checking the [_common setup issues_](./developing.md#Common-setup-issues).

#### Installing Plugins

We use a plugin architecture so that you can install only the dependencies you actually need.

| Plugin Name   | Install Command                                            | Provides                   |
| ------------- | ---------------------------------------------------------- | -------------------------- |
| file          | _included by default_                                      | File source and sink       |
| console       | _included by default_                                      | Console sink               |
| athena        | `pip install 'acryl-datahub[athena]'`                      | AWS Athena source          |
| bigquery      | `pip install 'acryl-datahub[bigquery]'`                    | BigQuery source            |
| glue          | `pip install 'acryl-datahub[glue]'`                        | AWS Glue source            |
| hive          | `pip install 'acryl-datahub[hive]'`                        | Hive source                |
| mssql         | `pip install 'acryl-datahub[mssql]'`                       | SQL Server source          |
| mysql         | `pip install 'acryl-datahub[mysql]'`                       | MySQL source               |
| oracle        | `pip install 'acryl-datahub[oracle]'`                      | Oracle source              |
| postgres      | `pip install 'acryl-datahub[postgres]'`                    | Postgres source            |
| sqlalchemy    | `pip install 'acryl-datahub[sqlalchemy]'`                  | Generic SQLAlchemy source  |
| snowflake     | `pip install 'acryl-datahub[snowflake]'`                   | Snowflake source           |
| mongodb       | `pip install 'acryl-datahub[mongodb]'`                     | MongoDB source             |
| ldap          | `pip install 'acryl-datahub[ldap]'` ([extra requirements]) | LDAP source                |
| kafka         | `pip install 'acryl-datahub[kafka]'`                       | Kafka source               |
| druid         | `pip install 'acryl-datahub[druid]'`                       | Druid Source               |
| dbt           | _no additional dependencies_                               | DBT source                 |
| datahub-rest  | `pip install 'acryl-datahub[datahub-rest]'`                | DataHub sink over REST API |
| datahub-kafka | `pip install 'acryl-datahub[datahub-kafka]'`               | DataHub sink over Kafka    |

These plugins can be mixed and matched as desired. For example:

```sh
pip install 'acryl-datahub[bigquery,datahub-rest]'
```

You can check the active plugins:

```sh
datahub check plugins
```

[extra requirements]: https://www.python-ldap.org/en/python-ldap-3.3.0/installing.html#build-prerequisites

#### Basic Usage

```sh
pip install 'acryl-datahub[datahub-rest]'  # install the required plugin
datahub ingest -c ./examples/recipes/example_to_datahub_rest.yml
```

### Install using Docker

[![Docker Hub](https://img.shields.io/docker/pulls/linkedin/datahub-ingestion?style=plastic)](https://hub.docker.com/r/linkedin/datahub-ingestion)
[![datahub-ingestion docker](https://github.com/linkedin/datahub/actions/workflows/docker-ingestion.yml/badge.svg)](https://github.com/linkedin/datahub/actions/workflows/docker-ingestion.yml)

If you don't want to install locally, you can alternatively run metadata ingestion within a Docker container.
We have prebuilt images available on [Docker hub](https://hub.docker.com/r/linkedin/datahub-ingestion). All plugins will be installed and enabled automatically.

_Limitation: the datahub_docker.sh convenience script assumes that the recipe and any input/output files are accessible in the current working directory or its subdirectories. Files outside the current working directory will not be found, and you'll need to invoke the Docker image directly._

```sh
./scripts/datahub_docker.sh ingest -c ./examples/recipes/example_to_datahub_rest.yml
```

### Install from source

If you'd like to install from source, see the [developer guide](./developing.md).

## Recipes

A recipe is a configuration file that tells our ingestion scripts where to pull data from (source) and where to put it (sink).
Here's a simple example that pulls metadata from MSSQL and puts it into datahub.

```yaml
# A sample recipe that pulls metadata from MSSQL and puts it into DataHub
# using the Rest API.
source:
  type: mssql
  config:
    username: sa
    password: ${MSSQL_PASSWORD}
    database: DemoData

transformers:
  - type: "fully-qualified-class-name-of-transformer"
    config:
      some_property: "some.value"

sink:
  type: "datahub-rest"
  config:
    server: "http://localhost:8080"
```

We automatically expand environment variables in the config,
similar to variable substitution in GNU bash or in docker-compose files. For details, see
https://docs.docker.com/compose/compose-file/compose-file-v2/#variable-substitution.

Running a recipe is quite easy.

```sh
datahub ingest -c ./examples/recipes/mssql_to_datahub.yml
```

A number of recipes are included in the examples/recipes directory.

## Sources

### Kafka Metadata `kafka`

Extracts:

- List of topics - from the Kafka broker
- Schemas associated with each topic - from the schema registry

```yml
source:
  type: "kafka"
  config:
    connection:
      bootstrap: "broker:9092"
      schema_registry_url: http://localhost:8081
      consumer_config: {} # passed to https://docs.confluent.io/platform/current/clients/confluent-kafka-python/index.html#deserializingconsumer
```

### MySQL Metadata `mysql`

Extracts:

- List of databases and tables
- Column types and schema associated with each table

```yml
source:
  type: mysql
  config:
    username: root
    password: example
    database: dbname
    host_port: localhost:3306
    table_pattern:
      deny:
        # Note that the deny patterns take precedence over the allow patterns.
        - "performance_schema"
      allow:
        - "schema1.table2"
      # Although the 'table_pattern' enables you to skip everything from certain schemas,
      # having another option to allow/deny on schema level is an optimization for the case when there is a large number
      # of schemas that one wants to skip and you want to avoid the time to needlessly fetch those tables only to filter
      # them out afterwards via the table_pattern.
    schema_pattern:
      deny:
        - "garbage_schema"
      allow:
        - "schema1"
```

### Microsoft SQL Server Metadata `mssql`

Extracts:

- List of databases, schema, and tables
- Column types associated with each table

```yml
source:
  type: mssql
  config:
    username: user
    password: pass
    host_port: localhost:1433
    database: DemoDatabase
    table_pattern:
      deny:
        - "^.*\\.sys_.*" # deny all tables that start with sys_
      allow:
        - "schema1.table1"
        - "schema1.table2"
    options:
      # Any options specified here will be passed to SQLAlchemy's create_engine as kwargs.
      # See https://docs.sqlalchemy.org/en/14/core/engines.html for details.
      charset: "utf8"
```

### Hive `hive`

Extracts:

- List of databases, schema, and tables
- Column types associated with each table

```yml
source:
  type: hive
  config:
    username: user
    password: pass
    host_port: localhost:10000
    database: DemoDatabase
    # table_pattern/schema_pattern is same as above
    # options is same as above
```

### PostgreSQL `postgres`

Extracts:

- List of databases, schema, and tables
- Column types associated with each table
- Also supports PostGIS extensions

```yml
source:
  type: postgres
  config:
    username: user
    password: pass
    host_port: localhost:5432
    database: DemoDatabase
    # table_pattern/schema_pattern is same as above
    # options is same as above
```

### Snowflake `snowflake`

Extracts:

- List of databases, schema, and tables
- Column types associated with each table

```yml
source:
  type: snowflake
  config:
    username: user
    password: pass
    host_port: account_name
    database: db_name
    # table_pattern/schema_pattern is same as above
    # options is same as above
```

### Oracle `oracle`

Extracts:

- List of databases, schema, and tables
- Column types associated with each table

```yml
source:
  type: oracle
  config:
    # For more details on authentication, see the documentation:
    # https://docs.sqlalchemy.org/en/14/dialects/oracle.html#dialect-oracle-cx_oracle-connect and
    # https://cx-oracle.readthedocs.io/en/latest/user_guide/connection_handling.html#connection-strings.
    username: user
    password: pass
    host_port: localhost:5432
    database: dbname
    # table_pattern/schema_pattern is same as above
    # options is same as above
```

### Google BigQuery `bigquery`

Extracts:

- List of databases, schema, and tables
- Column types associated with each table

```yml
source:
  type: bigquery
  config:
    project_id: project # optional - can autodetect from environment
    dataset: dataset_name
    options: # options is same as above
      # See https://github.com/mxmzdlv/pybigquery#authentication for details.
      credentials_path: "/path/to/keyfile.json" # optional
    # table_pattern/schema_pattern is same as above
```

### AWS Athena `athena`

Extracts:

- List of databases and tables
- Column types associated with each table

```yml
source:
  type: athena
  config:
    username: aws_access_key_id # Optional. If not specified, credentials are picked up according to boto3 rules.
    # See https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html
    password: aws_secret_access_key # Optional.
    database: database # Optional, defaults to "default"
    aws_region: aws_region_name # i.e. "eu-west-1"
    s3_staging_dir: s3_location # "s3://<bucket-name>/prefix/"
    # The s3_staging_dir parameter is needed because Athena always writes query results to S3.
    # See https://docs.aws.amazon.com/athena/latest/ug/querying.html
    # However, the athena driver will transparently fetch these results as you would expect from any other sql client.
    work_group: athena_workgroup # "primary"
    # table_pattern/schema_pattern is same as above
```

### AWS Glue `glue`

Extracts:

- List of tables
- Column types associated with each table
- Table metadata, such as owner, description and parameters

```yml
source:
  type: glue
  config:
    aws_region: aws_region_name # i.e. "eu-west-1"
    env: environment used for the DatasetSnapshot URN, one of "DEV", "EI", "PROD" or "CORP". # Optional, defaults to "PROD".
    database_pattern: # Optional, to filter databases scanned, same as schema_pattern above.
    table_pattern: # Optional, to filter tables scanned, same as table_pattern above.
    aws_access_key_id # Optional. If not specified, credentials are picked up according to boto3 rules.
    # See https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html
    aws_secret_access_key # Optional.
    aws_session_token # Optional.
```

### Druid `druid`

Extracts:

- List of databases, schema, and tables
- Column types associated with each table

**Note** It is important to define a explicitly define deny schema pattern for internal druid databases (lookup & sys)
if adding a schema pattern otherwise the crawler may crash before processing relevant databases.
This deny pattern is defined by default but is overriden by user-submitted configurations

```yml
source:
  type: druid
  config:
    # Point to broker address
    host_port: localhost:8082
    schema_pattern:
      deny:
        - "^(lookup|sys).*"
    # options is same as above
```

### Other databases using SQLAlchemy `sqlalchemy`

The `sqlalchemy` source is useful if we don't have a pre-built source for your chosen
database system, but there is an [SQLAlchemy dialect](https://docs.sqlalchemy.org/en/14/dialects/)
defined elsewhere. In order to use this, you must `pip install` the required dialect packages yourself.

Extracts:

- List of schemas and tables
- Column types associated with each table

```yml
source:
  type: sqlalchemy
  config:
    # See https://docs.sqlalchemy.org/en/14/core/engines.html#database-urls
    connect_uri: "dialect+driver://username:password@host:port/database"
    options: {} # same as above
    schema_pattern: {} # same as above
    table_pattern: {} # same as above
```

### MongoDB `mongodb`

Extracts:

- List of databases
- List of collections in each database

```yml
source:
  type: "mongodb"
  config:
    # For advanced configurations, see the MongoDB docs.
    # https://pymongo.readthedocs.io/en/stable/examples/authentication.html
    connect_uri: "mongodb://localhost"
    username: admin
    password: password
    authMechanism: "DEFAULT"
    options: {}
    database_pattern: {}
    collection_pattern: {}
    # database_pattern/collection_pattern are similar to schema_pattern/table_pattern from above
```

### LDAP `ldap`

Extracts:

- List of people
- Names, emails, titles, and manager information for each person

```yml
source:
  type: "ldap"
  config:
    ldap_server: ldap://localhost
    ldap_user: "cn=admin,dc=example,dc=org"
    ldap_password: "admin"
    base_dn: "dc=example,dc=org"
    filter: "(objectClass=*)" # optional field
```

### File `file`

Pulls metadata from a previously generated file. Note that the file sink
can produce such files, and a number of samples are included in the
[examples/mce_files](examples/mce_files) directory.

```yml
source:
  type: file
  filename: ./path/to/mce/file.json
```

### DBT `dbt`

Pull metadata from DBT output files:

- [dbt manifest file](https://docs.getdbt.com/reference/artifacts/manifest-json)
  - This file contains model, source and lineage data.
- [dbt catalog file](https://docs.getdbt.com/reference/artifacts/catalog-json)
  - This file contains schema data.
  - DBT does not record schema data for Ephemeral models, as such datahub will show Ephemeral models in the lineage, however there will be no associated schema for Ephemeral models

```yml
source:
  type: "dbt"
  config:
    manifest_path: "./path/dbt/manifest_file.json"
    catalog_path: "./path/dbt/catalog_file.json"
```

## Sinks

### DataHub Rest `datahub-rest`

Pushes metadata to DataHub using the GMA rest API. The advantage of the rest-based interface
is that any errors can immediately be reported.

```yml
sink:
  type: "datahub-rest"
  config:
    server: "http://localhost:8080"
```

### DataHub Kafka `datahub-kafka`

Pushes metadata to DataHub by publishing messages to Kafka. The advantage of the Kafka-based
interface is that it's asynchronous and can handle higher throughput. This requires the
Datahub mce-consumer container to be running.

```yml
sink:
  type: "datahub-kafka"
  config:
    connection:
      bootstrap: "localhost:9092"
      producer_config: {} # passed to https://docs.confluent.io/platform/current/clients/confluent-kafka-python/index.html#serializingproducer
```

### Console `console`

Simply prints each metadata event to stdout. Useful for experimentation and debugging purposes.

```yml
sink:
  type: "console"
```

### File `file`

Outputs metadata to a file. This can be used to decouple metadata sourcing from the
process of pushing it into DataHub, and is particularly useful for debugging purposes.
Note that the file source can read files generated by this sink.

```yml
sink:
  type: file
  config:
    filename: ./path/to/mce/file.json
```

## Transformations

Beyond basic ingestion, sometimes there might exist a need to modify the source data before passing it on to the sink.
Example use cases could be to add ownership information, add extra tags etc.

In such a scenario, it is possible to configure a recipe with a list of transformers.

```yml
transformers:
  - type: "fully-qualified-class-name-of-transformer"
    config:
      some_property: "some.value"
```

A transformer class needs to inherit from [`Transformer`](./src/datahub/ingestion/api/transform.py)
At the moment there are no built-in transformers.

## Using as a library

In some cases, you might want to construct the MetadataChangeEvents yourself but still use this framework to emit that metadata to DataHub. In this case, take a look at the emitter interfaces, which can easily be imported and called from your own code.

- [DataHub emitter via REST](./src/datahub/emitter/rest_emitter.py) (same requirements as `datahub-rest`). Basic usage [example](./examples/library/lineage_emitter_rest.py).
- [DataHub emitter via Kafka](./src/datahub/emitter/kafka_emitter.py) (same requirements as `datahub-kafka`). Basic usage [example](./examples/library/lineage_emitter_kafka.py).

## Usage with Airflow

There's a couple ways to integrate DataHub with Airflow.

### Running ingestion on a schedule

Take a look at these sample DAGs:

- [`generic_recipe_sample_dag.py`](./examples/airflow/generic_recipe_sample_dag.py) - a simple Airflow DAG that picks up a DataHub ingestion recipe configuration and runs it.
- [`mysql_sample_dag.py`](./examples/airflow/mysql_sample_dag.py) - an Airflow DAG that runs a MySQL metadata ingestion pipeline using an inlined configuration.

### Emitting lineage via a separate operator

Take a look at this sample DAG:

- [`lineage_emission_dag.py`](./examples/airflow/lineage_emission_dag.py) - emits lineage using the DatahubEmitterOperator.

In order to use this example, you must first configure the Datahub hook. Like in ingestion, we support a Datahub REST hook and a Kafka-based hook.

```sh
# For REST-based:
airflow connections add  --conn-type 'datahub_rest' 'datahub_rest_default' --conn-host 'http://localhost:8080'
# For Kafka-based (standard Kafka sink config can be passed via extras):
airflow connections add  --conn-type 'datahub_kafka' 'datahub_kafka_default' --conn-host 'broker:9092' --conn-extra '{}'
```

### Using Datahub's Airflow lineage backend

_Note: The Airflow lineage backend is only supported in Airflow 1.10.15+ and 2.0.2+._

1. First, you must configure the Airflow hooks. See above for details.
2. Add the following lines to your `airflow.cfg` file. You might need to
   ```ini
   [lineage]
   backend = datahub.integrations.airflow.DatahubAirflowLineageBackend
   datahub_conn_id = datahub_rest_default  # or datahub_kafka_default - whatever you named the connection in step 1
   ```
3. Configure `inlets` and `outlets` for your Airflow operators. For reference, look at the sample DAG in [`lineage_backend_demo.py`](./examples/airflow/lineage_backend_demo.py).
4. [optional] Learn more about [Airflow lineage](https://airflow.apache.org/docs/apache-airflow/stable/lineage.html), including shorthand notation and some automation.

## Developing

See the [developing guide](./developing.md).


