This contains the code for the new Hamilton UI. For an overview of getting started & features see this documentation. For a lengthier post and intro see our blog post.
One operational UI for all your dataflows
The Hamilton UI is a system that provides the following capabilities:
- Execution tracking with associated metadata
- Provides a persistent database to store/manage these
- Provides a server that allows reading/writing/authentication
- Data/artifact observability: provides telemetry/observability of Hamilton executions + specific function results/code through a web interface
- Lineage & provenance: allows you to quickly inspect how code and data is connected.
- Catalog: everything is observed and cataloged, so you can quickly search and find what exists and when it was run.
The UI is meant to monitor/debug Hamilton dataflows both in development and production. The aim is to enable dataflow authors to move faster during all phases of the software development lifecycle.
For an overview of some of these features you can watch this quick video.
Execution Tracking
See what's slow (left), pinpoint errors (middle) compare execution performance (right)
Data/Artifact Observability
Visualize data for a run (left), track code the run used (middle) compare data across executions (right)
Lineage & Provenance
See how things connect: what's upstream/downstream (left), walk through code visually (right)
Catalog
Understand artifacts produced (left), find features and when they were used (right)
Getting started
You can watch this video walkthrough on getting set up.
Make sure you have docker running:
# clone the repository if you haven't git clone https://github.com/dagworks-inc/hamilton # change into the UI directory cd hamilton/ui # run docker ./run.sh
Once docker is running navigate to http://localhost:8242 and create an email and a project; then follow instructions on integrating with Hamilton.
A fuller guide can be found here.
Architecture
The architecture is simple.
The tracking server stores data on postgres, as well as any blobs on s3. This is stored in a docker volume on local mode. The frontend is a simple React application. There are a few authentication/ACL capabilities, but the default is to use local/unauthenticated (open). Please talk to us if you have a need for more custom authentication.
Development
The structure involves a bit of cleverness to ensure the UI can easily be deployed and served from the CLI.
We have a symlink from backend/hamilton_ui to backend/server, allowing us to work with django's structure
while simultaneously allowing for import as hamilton_ui. (this should probably be changed at some point but not worth it now).
To deploy, use the admin.py script in the UI directory.
This:
- Builds the frontend
- Copies it into the build/ directory
- Publishes to the sf-hamilton-ui package on pypi
Then you'll run it with hamilton ui after installing sf-hamilton[ui]. Note to
talk to it you'll need the hamilton_sdk pacakge which can be installed with pip install sf-hamilton[sdk].
Building docker
Dev mode
For development you'll want to run
cd hamilton/ui ./dev.sh --build # to build it all ./dev.sh # to pull docker images but use local code
You need 9GB assigned to Docker or more to build the frontend
The frontend build requires around 8GB of memory to be assigned to docker to build. If you run into this, bump your docker memory allocation up to 9GB or more.
Prod mode
For production build you'll want to run
cd hamilton/ui ./run.sh # to pull from docker and run ./run.sh --build # to rebuild images for prod
Caveats:
You'll want to clean the backend/dist/ directory to not add unnecessary files to the docker image.
Pushing
How to push to docker hub:
# retag if needed docker tag local-image:tagname dagworks/ui-backend:VERSION # push built image docker push dagworks/ui-backend:VERSION # retag as latest docker tag dagworks/ui-backend:VERSION dagworks/ui-backend:latest # push latest docker push dagworks/ui-backend:latest
# retag if needed docker tag local-image:tagname dagworks/ui-frontend:VERSION # push built image docker push dagworks/ui-frontend:VERSION # retag as latest docker tag dagworks/ui-backend:VERSION dagworks/ui-backend:latest # push latest docker push dagworks/ui-backend:latest