Skip to main content

Installation

Introduction

To use the Tokern Lineage engine, you need to:

  • Deploy the engine with access to a Postgres database.
  • Set up and test integration libraries with your favorite ETL tools like Apache Airflow or dbt.
  • Browse lineage graphs or programmatically analyze graphs in Python notebooks.

Architecture

Tokern Lineage Engine consists for the following components:

  • API Server
  • Kedro Viz
  • API SDK
  • DB SDK

Install using packages

Setup the postgres database

# Create database and role in psql prompt
CREATE DATABASE tokern;
CREATE ROLE tokern_admin WITH LOGIN PASSWORD '<secret password>';
GRANT ALL PRIVILEGES ON DATABASE tokern TO tokern_admin;

Setup the API Server

python3 -m venv .env
source .env/bin/activate
pip install data-lineage

data_lineage --help
Usage: data_lineage [OPTIONS]

Options:
--version Show the version and exit.
-l, --log-level TEXT Logging Level
--catalog-user TEXT Database user name [required]
--catalog-password TEXT Database Password [required]
--catalog-host TEXT Database Host
--catalog-port INTEGER Database Password
--catalog-db TEXT Postgres Database
--server-address TEXT The socket to bind to
--is-production / --not-production
Run server in development mode
--help Show this message and exit.

# Start the server
data_lineage --catalog-user tokern_admin --catalog-password <password>

Setup Kedro Viz

git clone https://github.com/tokern/kedro-viz.git
cd kedro-viz
npm start

Install using Docker

This guide will help you get the Tokern Lineage engine and Postgres running as Docker containers using Docker Compose. This is the easiest way to set up Tokern Lineage engine on your local environment.

Prerequisites

Download and start services

# in a new directory run
wget https://raw.githubusercontent.com/tokern/data-lineage/master/install-manifests/docker-compose/tokern-lineage-engine.yml
# or run
curl https://raw.githubusercontent.com/tokern/data-lineage/master/install-manifests/docker-compose/catalog-demo.yml -o tokern-lineage-engine.yml

Run docker-compose

docker-compose up -d

If you want to use an external Postgres database, change the following parameters in tokern-lineage-engine.yml:

  • CATALOG_HOST
  • CATALOG_USER
  • CATALOG_PASSWORD
  • CATALOG_DB

You can also override default values using environement variables.

CATALOG_HOST=... CATALOG_USER=... CATALOG_PASSWORD=... CATALOG_DB=... docker-compose -f ... up -d

For more advanced usage of environment variables with docker-compose, refer to docker-compose docs

Head to http://localhost:8000/ to open the Tokern Lineage app