Reporting Progress from a task type

Logging

Any information that a task type logs gets stores and can be viewed in XF Desktop. The Python module logutil provides some helpers, and xfworkerutil.XFWorkerJob automatically configures the standard Python logging infrastructure.

Progress estimation

XamFlow estimates the progress for each processing task type automatically. The interface.json of each processing task type provides some information about how this should be done.

For example Lucid.Core.Threshold uses a linear estimation function that scales the estimated duration with the size of the input image:

{
    "inputs": [
        {
            "description": "The image to segment.",
            "key": "image",
            "type": "Image",
            "supported_modes": [
                "output"
            ]
        },
        {
            "description": "The lower threshold",
            "key": "threshold_lower0",
            "type": "Float",
            "supported_modes": [
                "fixed",
                "output",
                "none"
            ],
            "default": 1
        },
        {
            "description": "The upper threshold",
            "key": "threshold_upper0",
            "type": "Float",
            "supported_modes": [
                "none",
                "output",
                "fixed"
            ]
        },
        {
            "description": "The calibration to use for the threshold values. Requires image calibration metadata of this type.",
            "key": "calibration",
            "type": "String",
            "supported_modes": [
                "none",
                "fixed"
            ],
            "enum": [
                "dens",
                "hu",
                "linatt"
            ],
            "default": "dens"
        }
    ],
    "outputs": [
        {
            "key": "segments",
            "type": "Image"
        }
    ],
    "processing_effort_factor": {
        "function": "LinearWithSizeOfInput",
        "input_key": "image"
    }
}

Another option is a constant duration, i.e. to assume that all jobs of this task type take about the same amount of time:

{
    "inputs": [
        {
            "key": "contours",
            "type": "Contours",
            "supported_modes": [
                "output"
            ]
        },
        {
            "key": "image",
            "type": "Image",
            "supported_modes": [
                "output"
            ]
        }
    ],
    "outputs": [
        {
            "key": "contours",
            "type": "Contours"
        }
    ],
    "processing_effort_factor": {
        "function": "Constant"
    }
}

Live progress reporting

It is also possible to provide all kinds of data live via parameter outputs. Example.Basic.LiveProgress implements an example:

# Avoid requiring tkinter
import matplotlib
matplotlib.use('agg', force=True)

import seaborn
import time

import xfworkerutil


worker = xfworkerutil.XFWorkerJob()

inputs = worker.job['inputs']

# Download input files
image_path = worker.download_input_file(inputs['image'])

plot_path = worker.create_related_path(image_path, '_plot.png')


# Live Progress Demo
demo_data = seaborn.load_dataset("tips")
demo_data_x = "total_bill"
demo_data_y = "tip"
demo_phases = [
    (7.3, "Preprocessing"),
    (5.2, "Analyzing"),
    (3.9, "Reticulating"),
    (0.1, "Cross checking"),
    (2.6, "Collating"),
    (0.5, "Verifying"),
    (1.2, "Optimizing"),
    (0.8, "Triple checking"),
    (1.1, "Postprocessing"),
    (0.9, "Concluding"),
    (1.0, "Polished"),
]
for phase in range(len(demo_phases)):
    # Plot
    progressive_data = demo_data[:(len(demo_data) * (phase + 1)) // len(demo_phases)]
    plot_grid = seaborn.jointplot(x=demo_data_x, y=demo_data_y, data=progressive_data, kind="reg",
                                  xlim=(0, 60), ylim=(0, 12))
    plot_grid.set_axis_labels("X", "Y")

    plot_grid.savefig(str(plot_path))

    # Upload output files
    worker.upload_output_file(plot_path)

    # Save progress
    worker.progress(100 * phase / len(demo_phases), save=False)
    outputs = {
        "live_metric": demo_phases[phase][0],
        "live_status": demo_phases[phase][1],
        "live_plot": worker.create_output_bitmap(plot_path.name),
    }
    worker.save_job(outputs=outputs)

    # Ignore input for this simple demo
    time.sleep(5)


# Save outputs
outputs = {
    "live_metric": demo_phases[-1][0],
    "live_status": demo_phases[-1][1],
    "live_plot": worker.create_output_bitmap(plot_path.name),
}
worker.finish(outputs)

This can be used for simple outputs like numbers or for any other kind of data, like e.g. advanced plots.