Installation

Overview #

Flink Agents provides both Python and Java APIs to define a Flink Agents job.

The sections below show how to install the required dependencies.

NOTE: To run on flink cluster, Flink-Agents requires flink version be a stable release of Flink 1.20.3.

Before installing Flink Agents, you need to have Apache Flink 1.20.3 installed.

Download and extract Flink 1.20.3:

# Download Flink 1.20.3: https://www.apache.org/dyn/closer.lua/flink/flink-1.20.3/flink-1.20.3-bin-scala_2.12.tgz curl -LO https://archive.apache.org/dist/flink/flink-1.20.3/flink-1.20.3-bin-scala_2.12.tgz  # Extract the archive tar -xzf flink-1.20.3-bin-scala_2.12.tgz  # Set FLINK_HOME environment variable export FLINK_HOME=$(pwd)/flink-1.20.3  # Copy the flink-python JAR from opt to lib (required for PyFlink) cp $FLINK_HOME/opt/flink-python-1.20.3.jar $FLINK_HOME/lib/
Note: For more detailed Flink installation instructions, refer to the Flink local installation guide.

Prerequisites #

  • Python 3.10 or 3.11
  • For building from source, you also need:
    • Unix-like environment (Linux, Mac OS X, Cygwin, or WSL)
    • Git
    • Maven 3
    • Java 11

We recommend using a Python virtual environment to isolate Flink Agents dependencies from your system Python packages.

Create a virtual environment:

Using venv (built-in with Python 3):

# Create a virtual environment in a directory named 'venv' python3 -m venv venv  # Activate the virtual environment # On Linux/macOS: source venv/bin/activate # On Windows: # venv\Scripts\activate
Note: If python3 command is not found, try python instead. Some systems alias python to Python 3.

To deactivate when you’re done:

deactivate

Choose one of the following installation methods:

From Official Release #

Note: This will be available after Flink Agents is released.

Install Flink Agents using pip:

pip install flink-agents

From Source #

Obtain the source code:

You can obtain the Flink Agents source code in two ways:

Clone the repository and checkout the release tag:

git clone https://github.com/apache/flink-agents.git cd flink-agents # Checkout the release tag git checkout release-0.1.0

Download the source tarball from the Apache mirror:

# Download Flink Agents 0.1.0 source: https://archive.apache.org/dist/flink/flink-agents-0.1.0 curl -LO https://archive.apache.org/dist/flink/flink-agents-0.1.0/flink-agents-0.1.0-src.tgz  # Extract the archive tar -xzf flink-agents-0.1.0-src.tgz cd flink-agents-0.1.0-src

Build and install:

Run the build script to build both Java and Python components:

./tools/build.sh

This script will:

  • Build all Java modules using Maven
  • Build the Python package
  • Install the Python package into your current Python environment
  • Package the distribution JAR with all dependencies
Note: If you activated a virtual environment earlier, the Python package will be installed into that virtual environment. Otherwise, it will be installed into your system Python environment.

After building:

  • The Python package is installed and ready to use
  • The distribution JAR is located at: dist/target/flink-agents-dist-0.1.0.jar

After installing Flink Agents package, you need to deploy it to your Flink cluster so that Flink can run your agent jobs.

Configure PYTHONPATH #

Flink runs in its own JVM process and needs the PYTHONPATH environment variable to locate the flink-agents Python package. You need to set PYTHONPATH to the directory where flink-agents is installed.

Determine your Python package installation path:

The path depends on your Python environment setup:

  • If using a virtual environment, it’s the site-packages directory within your venv
  • If using system Python, it’s the system site-packages directory

Tip: You can use this command to help find the path:

python3 -c 'import sysconfig; print(sysconfig.get_paths()["purelib"])'

Set PYTHONPATH before starting Flink:

# Set PYTHONPATH to your Python site-packages directory  export PYTHONPATH=$(python -c 'import sysconfig; print(sysconfig.get_paths()["purelib"])')
Note: You can add the export PYTHONPATH=... line to your shell profile (~/.bash_profile, ~/.bashrc, ~/.zprofile, or ~/.zshrc) to set it permanently. This way, it will be automatically configured in all future terminal sessions.

Copy the Flink Agents distribution JAR to your Flink installation’s lib directory:

The Flink Agents JAR is bundled inside the Python package. Use the PYTHONPATH you configured above to locate and copy it:

# Copy the JAR from the Python package to Flink's lib directory cp $PYTHONPATH/flink_agents/lib/flink-agents-dist-0.1.0.jar $FLINK_HOME/lib/

After building from source, the distribution JAR is located in the dist/target/ directory:

# Copy the JAR to Flink's lib directory cp dist/target/flink-agents-dist-0.1.0.jar $FLINK_HOME/lib/

Once the Java library is installed and PYTHONPATH is configured, you can start your Flink cluster or submit jobs:

# Start your Flink cluster $FLINK_HOME/bin/start-cluster.sh  # Or submit your job directly $FLINK_HOME/bin/flink run -py /path/to/your/job.py

See deployment for more details on running Flink Agents jobs.