Creating a new dependency package¶

Creating a new dependency package is recommended in a few different scenarios.

The most common case is for depending on a third-party Python module available from PyPI.
More complex Python dependencies could require an install script, Conda environments or virtual environments with version constraint files.
External third-party software could be installed automatically or a virtual marker package can be used to indicate workers that have this software pre-installed.

Install new dependecy package¶

A .xfp dependency package file can be imported via the XamFlow menu Dependencies > Import Dependency Package. The .xfp file is a standard ZIP file (renamed to the .xfp file extension) that can be extracted or created as any ZIP file. It contains a metadata.json text file. A folder containing a metadata.json text file can also be checked in directly via the XamFlow menu Dependencies > Checkin New Dependency Package.

After import / checkin, the package can be installed on the worker computer via XamFlow menu Workers > View Workers. Then right-click the worker computer and select Install Dependency Package, and double click the dependency. At this point the install command is executed on the worker computer.

Python module¶

When a Python module is imported like import numpy often it has to be installed from PyPI with the pip install numpy command. In XamFlow an isolated Python environment is used, to avoid version conflicts, and such Python modules are referenced by declaring a dependency on a dependency package in the metadata.json file of the task type.

Create a new folder YourName.Python.numpy containing a new file metadata.json with this minimal text:

YourName.Python.numpy/metadata.json¶

{
    "package_format": "XFP-DEP1.0",
    "name": "YourName.Python.numpy",
    "version": "1.10.6.0",
    "install": "%xfpip% install numpy",
    "dependencies": [
        {
            "name": "Core.Host.Python3",
            "version": "1.10.7.0"
        }
    ]
}

For comparison here is the Core.Python.numpy/metadata.json file:

Core.Python.numpy/metadata.json¶

{
    "package_format": "XFP-DEP1.0",
    "name": "Core.Python.numpy",
    "version": "1.10.9.0",
    "summary": "numpy Python Library",
    "description_filename": "readme.txt",
    "keywords": [
        "Recommended"
    ],
    "install": "%xfpip% install numpy==1.26.4 --only-binary :all: --no-warn-script-location",
    "dependencies": [
        {
            "name": "Core.Host.Python3",
            "version": "1.10.9.0"
        }
    ]
}

The requirements are:

The package_format must be specified as XFP-DEP1.0.
The name must consist of three parts separated by two dots.
The version must be a four part number separated by three dots.
The dependencies section containing Core.Host.Python3 is required for the %xfpip% command.

Additional recommendations are:

Use a short abbreviation indicating your lab / group / institution as the first part of the name.
Use Python as the second part of the name.
Use the PyPI name / module import name (numpy) as the third part of the name.
Use the same version number as your XamFlow version.
Use %xfpip% in the install command instead of pip, to indicate it should use the pip tool from XamFlow’s Python environment.

More complex cases¶

While in most cases Python modules can be handled as described above, sometimes more complex steps are required. An install script can be used to execute multiple commands and setup more complex environments. Conda environments are sometimes needed for special libraries only available in that ecosystem. Virtual environments with version constraint files can also sometimes be useful to avoid version conflicts.

Virtual environments¶

For example Core.Python.langchain/metadata.json defines an install command and environment variable as follows:

Core.Python.langchain/metadata.json¶

{
    "package_format": "XFP-DEP1.0",
    "name": "Core.Python.langchain",
    "version": "1.10.9.0",
    "summary": "langchain Python Library",
    "description_filename": "readme.txt",
    "install": "powershell.exe -ExecutionPolicy bypass -File install.ps1",
    "environment": {
        "langchain_venv": "%xamflow_package_content_path%\\.venv"
    },
    "dependencies": [
        {
            "name": "Core.Host.Python3",
            "version": "1.10.9.0"
        }
    ]
}

The Core.Python.langchain/install/install.ps1 file itself then executes the commands to set up the virtual environment with specific versions:

Core.Python.langchain/install/install.ps1¶

Set-PSDebug -Trace 1
$ErrorActionPreference = "Stop"

$contentPath="${PSScriptRoot}\..\content"
$venvPath="${contentPath}\.venv"

Write-Output "Using xfpy: ${Env:xfpy}"
Write-Output "Using venv: ${venvPath}"

# Ignore pip_constraint.txt file here to avoid conflict with imageio constraint.
Remove-Item Env:PIP_CONSTRAINT

Write-Output "=== Create venv"
. cmd.exe /c ${Env:xfpy} -m venv --system-site-packages "`"${venvPath}`""
if ($lastexitcode -ne 0) { exit $lastexitcode }

$venv_py="${venvPath}\Scripts\python.exe"

Write-Output "=== Install langchain"
. ${venv_py} -m pip install langchain==0.3.25 langchain_community==0.3.25 langchain-anthropic==0.3.15 langchain-google-genai==2.1.5 langchain-mistralai==0.2.10 langchain-openai==0.3.24 langchain-ollama==0.3.3
if ($lastexitcode -ne 0) { exit $lastexitcode }

Write-Output "Done."

Writing a requirements.txt file is also possible.

Task types would then use "command": "%xfpy_with_venv% %langchain_venv% script.py" to use this virtual environment.

Conda environments¶

Some Python libraries are only available via the Conda ecosystem in channels like conda-forge. XamFlow includes Core.Host.micromamba to make such libraries available.

One approach is to write an install script. For example here is the Core.Python.tomopy/install/install.ps1 script that uses MICROMAMBA_EXE to install various libraries in a custom tomopy-env environment.

Core.Python.tomopy/install/install.ps1¶

Set-PSDebug -Trace 2
$ErrorActionPreference = "Stop"

$venvPath="tomopy-env"

Write-Output "Using venv: ${venvPath}"

. cmd.exe /c ${Env:MICROMAMBA_EXE} --yes create -n $venvPath python=3.11
if ($lastexitcode -ne 0) { exit $lastexitcode }

. cmd.exe /c  ${Env:MICROMAMBA_EXE} --yes install --name $venvPath `
    --channel conda-forge `
    --channel astra-toolbox `
    --channel nvidia `
    tomopy==1.15.0 dxchange==0.2.0 numpy==1.26.4 astra-toolbox==2.3.1
if ($lastexitcode -ne 0) { exit $lastexitcode }

Writing an environment.yml Conda file is also possible.

The task type should then use a command like %xfpy% -n tomopy-env python script.py to use this environment, for example as in Example.tomopy.Reconstruction/metadata.json:

Example.tomopy.Reconstruction/metadata.json¶

{
    "package_format": "XFP-TT1.0",
    "name": "Example.tomopy.Reconstruction",
    "version": "1.10.9.0",
    "display_name": "Tomographic Reconstruction",
    "summary": "Tomographic Reconstruction using tomopy (optionally via ASTRA).",
    "behavior": "ProcessingFollower",
    "description_filename": "readme.txt",
    "citation_cff_filename": "CITATION.cff",
    "command": "%xfpy% -n tomopy-env python script.py",
    "dependencies": [
        {
            "name": "Core.Host.micromamba",
            "version": "1.10.9.0"
        },
        {
            "name": "Core.Python.Library",
            "version": "1.10.9.0"
        },
        {
            "name": "Core.Type.Base",
            "version": "1.10.9.0"
        },
        {
            "name": "Core.Python.tomopy",
            "version": "1.10.9.0"
        }
    ]
}

External software¶

External third-party software can be installed automatically using install scripts. Alternatively a virtual marker dependency can be created to indicate any workers that have this software pre-installed.

Virtual marker dependencies¶

Virtual marker dependencies are “empty” dependency packages that contain no install command. They just exist to indicate that a dependency exists in principle, and that the relevant software must be pre-installed manually. For example the Core.Host.CUDA/metadata.json file contains no install command:

Core.Host.CUDA/metadata.json¶

{
    "package_format": "XFP-DEP1.0",
    "name": "Core.Host.CUDA",
    "version": "1.10.9.0",
    "summary": "NVIDIA CUDA 11.4",
    "description_filename": "readme.txt"
}

The readme.txt contains instructions for how to install the dependency manually:

Core.Host.CUDA/readme.txt¶

# Core.Host.CUDA

Install NVIDIA CUDA 11.4.
Download from https://developer.nvidia.com/cuda-downloads

Dependency packages like this can be useful when multiple worker machines are used, and only some of them have specialized additional software installed. This allows controlling on which worker certain types of XamFlow jobs should (not) be scheduled for processing.

Install scripts¶

The install command can execute any kind of shell command. If a longer sequence of commands is required, it is recommended to create a script in the install folder. Usually a PowerShell script install/install.ps1 is used, and the command to invoke it is "install": "powershell.exe -ExecutionPolicy bypass -File install.ps1".

For example Core.Host.ImageJ/install/install.ps1 executes several commands to install the software:

Core.Host.ImageJ/install/install.ps1¶

Set-PSDebug -Trace 2
$ErrorActionPreference = "Stop"

$zipUri="https://downloads.imagej.net/fiji/stable/fiji-stable-win64-jdk.zip"
$zipPath="${PSScriptRoot}\fiji.zip"
$unpackPath="${PSScriptRoot}\..\content"

Write-Output "Download"
$ProgressPreference = 'SilentlyContinue'
[Net.ServicePointManager]::SecurityProtocol = "Tls12, Tls11, Tls"
Invoke-WebRequest -OutFile $zipPath -Uri $zipUri

Write-Output "Unpack"
Add-Type -AssemblyName System.IO.Compression.FileSystem
[System.IO.Compression.ZipFile]::ExtractToDirectory($zipPath, $unpackPath)
Remove-Item -LiteralPath $zipPath

Write-Output "Enable BoneJ"
. cmd.exe /c $unpackPath\Fiji.app\fiji-windows-x64.exe --update add-update-site BoneJ https://sites.imagej.net/BoneJ
if ($lastexitcode -ne 0) { exit $lastexitcode }

Write-Output "Update and install BoneJ"
. cmd.exe /c $unpackPath\Fiji.app\fiji-windows-x64.exe --update update
if ($lastexitcode -ne 0) { exit $lastexitcode }

Write-Output "Done."

And Core.Host.ImageJ/metadata.json contains this install command:

Core.Host.ImageJ/metadata.json¶

{
    "package_format": "XFP-DEP1.0",
    "name": "Core.Host.ImageJ",
    "version": "1.10.9.0",
    "summary": "ImageJ",
    "description_filename": "readme.txt",
    "install": "powershell.exe -ExecutionPolicy bypass -File install.ps1",
    "environment": {
        "IMAGEJ_EXE": "%xamflow_package_content_path%\\Fiji.app\\ImageJ-win64.exe"
    }
}

Additionally an environment variable IMAGEJ_EXE is declared so the software can be located and used by task types later.