stackablectl demo

A demo is an end-to-end demonstration of the usage of the Stackable data platform. It is tied to a specific stack of the Stackable data platform, which will provide the required products for the demo.

General Usage

Interact with demos, which are end-to-end usage demonstrations of the Stackable data platform Usage: stackablectl demo [OPTIONS] <COMMAND> Commands: list List available demos describe Print out detailed demo information install Install a specific demo help Print this message or the help of the given subcommand(s) Options: -l, --log-level <LOG_LEVEL> Log level this application uses --no-cache Do not cache the remote (default) demo, stack and release files Cached files are saved at '$XDG_CACHE_HOME/stackablectl', which is usually '$HOME/.cache/stackablectl' when not explicitly set. --offline Do not request any remote files via the network -h, --help Print help (see a summary with '-h') -V, --version Print version File options: -d, --demo-file <DEMO_FILE> Provide one or more additional (custom) demo file(s) Demos are loaded in the following order: Remote (default) demo file, custom demo files provided via the 'STACKABLE_DEMO_FILES' environment variable, and lastly demo files provided via the '-d/--demo-file' argument(s). If there are demos with the same name, the last demo definition will be used. Use "stackablectl [OPTIONS] <COMMAND> -d path/to/demos1.yaml -d path/to/demos2.yaml" to provide multiple additional demo files. -s, --stack-file <STACK_FILE> Provide one or more additional (custom) stack file(s) Stacks are loaded in the following order: Remote (default) stack file, custom stack files provided via the 'STACKABLE_STACK_FILES' environment variable, and lastly demo files provided via the '-s/--stack-file' argument(s). If there are stacks with the same name, the last stack definition will be used. Use "stackablectl [OPTIONS] <COMMAND> -s path/to/stacks1.yaml -s path/to/stacks2.yaml" to provide multiple additional stack files. -r, --release-file <RELEASE_FILE> Provide one or more additional (custom) release file(s) Releases are loaded in the following order: Remote (default) release file, custom release files provided via the 'STACKABLE_RELEASE_FILES' environment variable, and lastly release files provided via the '-r/--release-file' argument(s). If there are releases with the same name, the last release definition will be used. Use "stackablectl [OPTIONS] <COMMAND> -r path/to/releases1.yaml -r path/to/releases2.yaml" to provide multiple additional release files. Helm repository options: --helm-repo-stable <URL> Provide a custom Helm stable repository URL [default: https://repo.stackable.tech/repository/helm-stable/] --helm-repo-test <URL> Provide a custom Helm test repository URL [default: https://repo.stackable.tech/repository/helm-test/] --helm-repo-dev <URL> Provide a custom Helm dev repository URL [default: https://repo.stackable.tech/repository/helm-dev/]

Browse Available Demos

To list the available demos, run the following command:

$ stackablectl demo list ┌────┬───────────────────────┬─────────┬─────────────────────────────────────────────────────────┐ │ # ┆ NAME ┆ STACK ┆ DESCRIPTION │ ╞════╪═══════════════════════╪═════════╪═════════════════════════════════════════════════════════╡ │ 1 ┆ airflow-scheduled-job ┆ airflow ┆ Activate a simple Airflow DAG to run continuously at a │ │ ┆ ┆ ┆ set interval │ ├╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ 2 ┆ ... ┆ ... ┆ ... │ ├╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ 11 ┆ local-demo-test-env ┆ test ┆ Test demo. Provided with the STACKABLE_DEMO_FILES env │ │ ┆ ┆ ┆ var in the .env file │ └────┴───────────────────────┴─────────┴─────────────────────────────────────────────────────────┘

Detailed information of a demo can be queried using the describe command:

$ stackablectl demo describe trino-taxi-data DEMO trino-taxi-data DESCRIPTION Demo loading 2.5 years of New York taxi data into S3 bucket, creating a Trino table and a Superset dashboard DOCUMENTATION https://docs.stackable.tech/stackablectl/stable/demos/trino-taxi-data.html STACK trino-superset-s3 LABELS trino, superset, minio, s3, ny-taxi-data

Installing a Demo

Using an Existing Kubernetes Cluster

If you want to access a Kubernetes cluster, make sure your kubectl Kubernetes client is configured to interact with the Kubernetes cluster. After that, run the following command:

$ stackablectl demo install trino-taxi-data

Using a Local Kubernetes Cluster

If you don’t have a Kubernetes cluster available, stackablectl can spin up a kind or minikube Kubernetes cluster for you. Based on the type of local cluster you want to use, ensure you have either kind or minikube installed on your system. stackablectl will perform a check to verify that these tools are available in your PATH and check if Docker is running.

Local Kind Kubernetes Cluster

$ stackablectl demo install trino-taxi-data -c kind Creating cluster "stackable-data-platform" ... ✓ Ensuring node image (kindest/node:v1.26.3) 🖼 ✓ Preparing nodes 📦 📦 ✓ Writing configuration 📜 ✓ Starting control-plane 🕹️ ✓ Installing CNI 🔌 ✓ Installing StorageClass 💾 ✓ Joining worker nodes 🚜 Set kubectl context to "kind-stackable-data-platform" You can now use your cluster with: kubectl cluster-info --context kind-stackable-data-platform Have a nice day! 👋 Installed demo trino-taxi-data Use "stackablectl operator installed" to display the installed operators Use "stackablectl stacklet list" to display the installed stacklets

Local Minikube Kubernetes Cluster

$ stackablectl demo install trino-taxi-data -c minikube 😄 [stackable-data-platform] minikube v1.30.1 on Ubuntu 22.04.2 ✨ Using the docker driver based on user configuration 🎉 minikube 1.31.2 is available! Download it: https://github.com/kubernetes/minikube/releases/tag/v1.31.2 💡 To disable this notice, run: 'minikube config set WantUpdateNotification false' 📌 Using Docker driver with root privileges 👍 Starting control plane node stackable-data-platform in cluster stackable-data-platform 🚜 Pulling base image ... 🔥 Creating docker container (CPUs=2, Memory=8000MB) ... 🐳 Preparing Kubernetes v1.26.3 on Docker 23.0.2 ... ▪ Generating certificates and keys ... ▪ Booting up control plane ... ▪ Configuring RBAC rules ... 🔗 Configuring CNI (Container Networking Interface) ... ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5 🌟 Enabled addons: storage-provisioner, default-storageclass 🔎 Verifying Kubernetes components... 👍 Starting worker node stackable-data-platform-m02 in cluster stackable-data-platform 🚜 Pulling base image ... 🔥 Creating docker container (CPUs=2, Memory=8000MB) ... 🌐 Found network options: ▪ NO_PROXY=192.168.58.2 🐳 Preparing Kubernetes v1.26.3 on Docker 23.0.2 ... ▪ env NO_PROXY=192.168.58.2 🔎 Verifying Kubernetes components... 🏄 Done! kubectl is now configured to use "stackable-data-platform" cluster and "default" namespace by default Installed demo trino-taxi-data Use "stackablectl operator installed" to display the installed operators Use "stackablectl stacklet list" to display the installed stacklets

The demos create Kubernetes jobs that will populate test data and interact with the installed products to process the data. Until the products are ready, it is expected that the pods of these Jobs will fail with an error. They will get retried with an exponentially growing back-off time. After the products are ready, they should turn green, and everything should settle down.

Listing Deployed Stacklets

After installing your demo you can use the stackablectl stacklets command to list the installed stacklets as follows:

$ stackablectl stacklets list ┌──────────┬───────────────┬───────────┬─────────────────────────────────────────────┬────────────────────────────────────────────┐ │ PRODUCT ┆ NAME ┆ NAMESPACE ┆ ENDPOINTS ┆ CONDITIONS │ ╞══════════╪═══════════════╪═══════════╪═════════════════════════════════════════════╪════════════════════════════════════════════╡ │ hive ┆ hive ┆ default ┆ ┆ Available, Reconciling, Running │ ├╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ opa ┆ opa ┆ default ┆ ┆ Available, Reconciling, Running │ ├╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ superset ┆ superset ┆ default ┆ external-superset http://192.168.58.2:30788 ┆ Available, Reconciling, Running │ ├╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ trino ┆ trino ┆ default ┆ ┆ Unavailable: See [1], Reconciling, Running │ ├╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ minio ┆ minio-console ┆ default ┆ ┆ │ └──────────┴───────────────┴───────────┴─────────────────────────────────────────────┴────────────────────────────────────────────┘ [1]: StatefulSet ["trino-coordinator-default", "trino-worker-default"] missing ready replicas.

Uninstalling a Demo

Currently, there is no support for uninstalling a demo again. However, this functionality will come soon.