If your input is complete, you can now turn to writing your workflow steps. You can connect workflow steps by using expressions. For example, if step A has an input that needs data from step B, Arcaflow will automatically run step B first.
To define a step type, you can do the following:
workflow.yaml
version:v0.2.0steps:step_a:# Specify any ID here you want to reference the step byplugin:deployment_type:imagesrc:quay.io/some/container/image# This must be an Arcaflow-compatible imageinput:# specify input values as a data structure, mixing in expressions as neededsome:key:!expr$.steps.step_b.outputs.success.some_valuestep_b:plugin:deployment_type:imagesrc:quay.io/some/container/imageinput:some:key:!expr$.input.some_value# Reference an input value
Plugin steps
Plugin steps run Arcaflow plugins in containers. They can use Docker, Podman, or Kubernetes as deployers. If no deployer is specified in the workflow, the plugin will use the local deployer.
Plugin steps have the following properties:
Property
Description
plugin
Full name of the container image to run. This must be an Arcaflow-compatible container image.
step
If a plugin provides more than one possible step, you can specify the step ID here.
deploy
Configuration for the deployer. (See below.) This can contain expressions, so you can dynamically specify deployment parameters.
input
Input data for the plugin. This can contain expressions, so you can dynamically define inputs.
You can reference plugin outputs in the format of $.steps.your_step_id.outputs.your_plugin_output_id.some_variable.
Deployers
The deploy key for plugins lets you control how the plugin container is deployed. You can use expressions to use other plugins (e.g. the kubeconfig plugin) to generate the deployment configuration and feed it into other steps.
You can configure the Docker deployer like this:
version:v0.2.0step:your_step_id:plugin:...input:...deploy:# You can use expressions heredeployer_name:dockerconnection:# Change this to point to a TCP-based Docker sockethost:host-to-docker# Add a certificates here. This is usually needed in TCP mode.cacert:|Add your CA cert PEM herecert:|Add your client cert PEM here.key:|Add your client key PEM here.deployment:# For more options here see: https://docs.docker.com/engine/api/v1.42/#tag/Container/operation/ContainerCreatecontainer:# Add your container config here.host:# Add your host config here.network:# Add your network config hereplatform:# Add your platform config hereimagePullPolicy:Always|IfNotPresent|Nevertimeouts:# HTTP timeouthttp:5s
All options for the Docker deployer
Type:
scope
Root object:
Config
Propertiesconnection (reference[Connection])
Name:
Connection
Description:
Docker connection information.
Required:
No
Referenced object:
Connection (see in the Objects section below)
deployment (reference[Deployment])
Name:
Deployment
Description:
Deployment configuration for the plugin.
Required:
No
Referenced object:
Deployment (see in the Objects section below)
timeouts (reference[Timeouts])
Name:
Timeouts
Description:
Timeouts for the Docker connection.
Required:
No
Referenced object:
Timeouts (see in the Objects section below)
ObjectsConfig (object)
Type:
object
Propertiesconnection (reference[Connection])
Name:
Connection
Description:
Docker connection information.
Required:
No
Referenced object:
Connection (see in the Objects section below)
deployment (reference[Deployment])
Name:
Deployment
Description:
Deployment configuration for the plugin.
Required:
No
Referenced object:
Deployment (see in the Objects section below)
timeouts (reference[Timeouts])
Name:
Timeouts
Description:
Timeouts for the Docker connection.
Required:
No
Referenced object:
Timeouts (see in the Objects section below)
Connection (object)
Type:
object
Propertiescacert (string)
Name:
CA certificate
Description:
CA certificate in PEM format to verify the Dockerd server certificate against.
Ports to expose on the host machine. Ports are specified in the format of portnumber/protocol.
Required:
No
Key type
Type:
string
Must match pattern:
^[0-9]+(/[a-zA-Z0-9]+)$
Value type
Type:
list[reference[PortBinding]]
List Items
Type:
reference[PortBinding]
Referenced object:
PortBinding (see in the Objects section below)
NetworkConfig (object)
Type:
object
Properties
None
PlatformConfig (object)
Type:
object
Properties
None
PortBinding (object)
Type:
object
PropertiesHostIP (string)
Name:
Host IP
Required:
No
HostPort (string)
Name:
Host port
Required:
No
Must match pattern:
^0-9+$
Timeouts (object)
Type:
object
Propertieshttp (int)
Name:
HTTP
Description:
HTTP timeout for the Docker API.
Required:
No
Minimum:
100000000
Units:
nanoseconds
Default
"15s"
If you want to use Podman as your local deployer, you can do so like this:
version:v0.2.0step:your_step_id:plugin:...input:...deploy:# You can use expressions heredeployer_name:podmanpodman:# Change where Podman is. (You can use this to point to a shell scriptpath:/path/to/your/podman# Change the network modenetworkMode:hostdeployment:# For more options here see: https://docs.docker.com/engine/api/v1.42/#tag/Container/operation/ContainerCreatecontainer:# Add your container config here.host:# Add your host config here.imagePullPolicy:Always|IfNotPresent|Nevertimeouts:# HTTP timeouthttp:5s
All options for the Podman deployer
Type:
scope
Root object:
Config
Propertiesdeployment (reference[Deployment])
Name:
Deployment
Description:
Deployment configuration for the plugin.
Required:
No
Referenced object:
Deployment (see in the Objects section below)
podman (reference[Podman])
Name:
Podman
Description:
Podman CLI configuration
Required:
No
Referenced object:
Podman (see in the Objects section below)
ObjectsConfig (object)
Type:
object
Propertiesdeployment (reference[Deployment])
Name:
Deployment
Description:
Deployment configuration for the plugin.
Required:
No
Referenced object:
Deployment (see in the Objects section below)
podman (reference[Podman])
Name:
Podman
Description:
Podman CLI configuration
Required:
No
Referenced object:
Podman (see in the Objects section below)
ContainerConfig (object)
Type:
object
PropertiesDomainname (string)
Name:
Domain name
Description:
Domain name for the plugin container.
Required:
No
Minimum:
1
Maximum:
255
Must match pattern:
^[a-zA-Z0-9-_.]+$
Env (list[string])
Name:
Environment variables
Description:
Environment variables to set on the plugin container.
Required:
No
List Items
Type:
string
Minimum:
1
Maximum:
32760
Must match pattern:
^.+=.+$
Hostname (string)
Name:
Hostname
Description:
Hostname for the plugin container.
Required:
No
Minimum:
1
Maximum:
255
Must match pattern:
^[a-zA-Z0-9-_.]+$
MacAddress (string)
Name:
MAC address
Description:
Media Access Control address for the container.
Required:
No
Must match pattern:
^[a-fA-F0-9]{2}(:[a-fA-F0-9]{2}){5}$
NetworkDisabled (bool)
Name:
Disable network
Description:
Disable container networking completely.
Required:
No
User (string)
Name:
Username
Description:
User that will run the command inside the container. Optionally, a group can be specified in the user:group format.
Required:
No
Minimum:
1
Maximum:
255
Must match pattern:
^[a-z_][a-z0-9_-]*[$]?(:[a-z_][a-z0-9_-]*[$]?)$
Deployment (object)
Type:
object
Propertiescontainer (reference[ContainerConfig])
Name:
Container configuration
Description:
Provides information about the container for the plugin.
Required:
No
Referenced object:
ContainerConfig (see in the Objects section below)
host (reference[HostConfig])
Name:
Host configuration
Description:
Provides information about the container host for the plugin.
Required:
No
Referenced object:
HostConfig (see in the Objects section below)
imagePullPolicy (enum[string])
Name:
Image pull policy
Description:
When to pull the plugin image.
Required:
No
Values
Always Always
IfNotPresent If not present
Never Never
Default
"IfNotPresent"
HostConfig (object)
Type:
object
PropertiesBinds (list[string])
Name:
Volume Bindings
Description:
Volumes
Required:
No
List Items
Type:
string
Minimum:
1
Maximum:
32760
Must match pattern:
^.+:.+$
CapAdd (list[string])
Name:
Add capabilities
Description:
Add capabilities to the container.
Required:
No
List Items
Type:
string
CapDrop (list[string])
Name:
Drop capabilities
Description:
Drop capabilities from the container.
Required:
No
List Items
Type:
string
CgroupnsMode (enum[string])
Name:
CGroup namespace mode
Description:
CGroup namespace mode to use for the container.
Required:
No
Values
`` Empty
host Host
private Private
Dns (list[string])
Name:
DNS servers
Description:
DNS servers to use for lookup.
Required:
No
List Items
Type:
string
DnsOptions (list[string])
Name:
DNS options
Description:
DNS options to look for.
Required:
No
List Items
Type:
string
DnsSearch (list[string])
Name:
DNS search
Description:
DNS search domain.
Required:
No
List Items
Type:
string
ExtraHosts (list[string])
Name:
Extra hosts
Description:
Extra hosts entries to add
Required:
No
List Items
Type:
string
NetworkMode (string)
Name:
Network mode
Description:
Specifies either the network mode, the container network to attach to, or a name of a Docker network to use.
The Kubernetes deployer deploys on top of Kubernetes. You can set up the deployer like this:
version:v0.2.0step:your_step_id:plugin:...input:...deploy:# You can use expressions heredeployer_name:kubernetesconnection:host:localhost:6443cert:|Add your client cert in PEM format here.key:|Add your client key in PEM format here.cacert:|Add the server CA cert in PEM format here.
All options for the Kubernetes deployer
Type:
scope
Root object:
Config
Propertiesconnection (reference[Connection])
Name:
Connection
Description:
Docker connection information.
Required:
No
Referenced object:
Connection (see in the Objects section below)
pod (reference[Pod])
Name:
Pod
Description:
Pod configuration for the plugin.
Required:
No
Referenced object:
Pod (see in the Objects section below)
timeouts (reference[Timeouts])
Name:
Timeouts
Description:
Timeouts for the Docker connection.
Required:
No
Referenced object:
Timeouts (see in the Objects section below)
ObjectsAWSElasticBlockStoreVolumeSource (object)
Type:
object
Properties
None
AzureDiskVolumeSource (object)
Type:
object
Properties
None
AzureFileVolumeSource (object)
Type:
object
Properties
None
CSIVolumeSource (object)
Type:
object
Properties
None
CephFSVolumeSource (object)
Type:
object
Properties
None
CinderVolumeSource (object)
Type:
object
Properties
None
Config (object)
Type:
object
Propertiesconnection (reference[Connection])
Name:
Connection
Description:
Docker connection information.
Required:
No
Referenced object:
Connection (see in the Objects section below)
pod (reference[Pod])
Name:
Pod
Description:
Pod configuration for the plugin.
Required:
No
Referenced object:
Pod (see in the Objects section below)
timeouts (reference[Timeouts])
Name:
Timeouts
Description:
Timeouts for the Docker connection.
Required:
No
Referenced object:
Timeouts (see in the Objects section below)
ConfigMapVolumeSource (object)
Type:
object
Properties
None
Connection (object)
Type:
object
PropertiesbearerToken (string)
Name:
Bearer token
Description:
Bearer token to authenticate against the Kubernetes API with.
Required:
No
burst (int)
Name:
Burst
Description:
Burst value for query throttling.
Required:
No
Minimum:
0
Default
10
cacert (string)
Name:
CA certificate
Description:
CA certificate in PEM format to verify Kubernetes server certificate against.