In order to use Arcaflow, you will need to download the Arcaflow Engine. You can simply unpack and run it, no need for installing it.
On Linux and macOS, you may need to run chmod +x on the engine binary.
Configuration
If you are using Docker as the local deployer (see below), you generally do not need to perform any extra configuration.
If you wish to customize Arcaflow, you can pass a YAML configuration file to Arcaflow with the -config your-arcaflow-config.yaml parameter.
Local deployer
The Arcaflow Engine needs a local container deployer to temporarily run plugins and read their schema. We recommend either Docker (default) or Podman for this purpose. You can use a Kubernetes cluster for this purpose too, but a local container engine is the better choice for performance reasons.
You can then change the deployer type like this:
config.yaml
deployer:type:podman# Deployer-specific options
Docker is the default local deployer. You can configure it like this:
config.yaml
deployer:type:dockerconnection:# Change this to point to a TCP-based Docker sockethost:host-to-docker# Add a certificates here. This is usually needed in TCP mode.cacert:|Add your CA cert PEM herecert:|Add your client cert PEM here.key:|Add your client key PEM here.deployment:# For more options here see: https://docs.docker.com/engine/api/v1.42/#tag/Container/operation/ContainerCreatecontainer:# Add your container config here.host:# Add your host config here.network:# Add your network config hereplatform:# Add your platform config hereimagePullPolicy:Always|IfNotPresent|Nevertimeouts:# HTTP timeouthttp:5s
All options for the Docker deployer
Type:
scope
Root object:
Config
Propertiesconnection (reference[Connection])
Name:
Connection
Description:
Docker connection information.
Required:
No
Referenced object:
Connection (see in the Objects section below)
deployment (reference[Deployment])
Name:
Deployment
Description:
Deployment configuration for the plugin.
Required:
No
Referenced object:
Deployment (see in the Objects section below)
timeouts (reference[Timeouts])
Name:
Timeouts
Description:
Timeouts for the Docker connection.
Required:
No
Referenced object:
Timeouts (see in the Objects section below)
ObjectsConfig (object)
Type:
object
Propertiesconnection (reference[Connection])
Name:
Connection
Description:
Docker connection information.
Required:
No
Referenced object:
Connection (see in the Objects section below)
deployment (reference[Deployment])
Name:
Deployment
Description:
Deployment configuration for the plugin.
Required:
No
Referenced object:
Deployment (see in the Objects section below)
timeouts (reference[Timeouts])
Name:
Timeouts
Description:
Timeouts for the Docker connection.
Required:
No
Referenced object:
Timeouts (see in the Objects section below)
Connection (object)
Type:
object
Propertiescacert (string)
Name:
CA certificate
Description:
CA certificate in PEM format to verify the Dockerd server certificate against.
Ports to expose on the host machine. Ports are specified in the format of portnumber/protocol.
Required:
No
Key type
Type:
string
Must match pattern:
^[0-9]+(/[a-zA-Z0-9]+)$
Value type
Type:
list[reference[PortBinding]]
List Items
Type:
reference[PortBinding]
Referenced object:
PortBinding (see in the Objects section below)
NetworkConfig (object)
Type:
object
Properties
None
PlatformConfig (object)
Type:
object
Properties
None
PortBinding (object)
Type:
object
PropertiesHostIP (string)
Name:
Host IP
Required:
No
HostPort (string)
Name:
Host port
Required:
No
Must match pattern:
^0-9+$
Timeouts (object)
Type:
object
Propertieshttp (int)
Name:
HTTP
Description:
HTTP timeout for the Docker API.
Required:
No
Minimum:
100000000
Units:
nanoseconds
Default
"15s"
If you want to use Podman as your local deployer instead of Docker, you can do so like this:
config.yaml
deployer:type:podmanpodman:# Change where Podman is. (You can use this to point to a shell scriptpath:/path/to/your/podman# Change the network modenetworkMode:hostdeployment:# For more options here see: https://docs.docker.com/engine/api/v1.42/#tag/Container/operation/ContainerCreatecontainer:# Add your container config here.host:# Add your host config here.imagePullPolicy:Always|IfNotPresent|Nevertimeouts:# HTTP timeouthttp:5s
All options for the Podman deployer
Type:
scope
Root object:
Config
Propertiesdeployment (reference[Deployment])
Name:
Deployment
Description:
Deployment configuration for the plugin.
Required:
No
Referenced object:
Deployment (see in the Objects section below)
podman (reference[Podman])
Name:
Podman
Description:
Podman CLI configuration
Required:
No
Referenced object:
Podman (see in the Objects section below)
ObjectsConfig (object)
Type:
object
Propertiesdeployment (reference[Deployment])
Name:
Deployment
Description:
Deployment configuration for the plugin.
Required:
No
Referenced object:
Deployment (see in the Objects section below)
podman (reference[Podman])
Name:
Podman
Description:
Podman CLI configuration
Required:
No
Referenced object:
Podman (see in the Objects section below)
ContainerConfig (object)
Type:
object
PropertiesDomainname (string)
Name:
Domain name
Description:
Domain name for the plugin container.
Required:
No
Minimum:
1
Maximum:
255
Must match pattern:
^[a-zA-Z0-9-_.]+$
Env (list[string])
Name:
Environment variables
Description:
Environment variables to set on the plugin container.
Required:
No
List Items
Type:
string
Minimum:
1
Maximum:
32760
Must match pattern:
^.+=.+$
Hostname (string)
Name:
Hostname
Description:
Hostname for the plugin container.
Required:
No
Minimum:
1
Maximum:
255
Must match pattern:
^[a-zA-Z0-9-_.]+$
MacAddress (string)
Name:
MAC address
Description:
Media Access Control address for the container.
Required:
No
Must match pattern:
^[a-fA-F0-9]{2}(:[a-fA-F0-9]{2}){5}$
NetworkDisabled (bool)
Name:
Disable network
Description:
Disable container networking completely.
Required:
No
User (string)
Name:
Username
Description:
User that will run the command inside the container. Optionally, a group can be specified in the user:group format.
Required:
No
Minimum:
1
Maximum:
255
Must match pattern:
^[a-z_][a-z0-9_-]*[$]?(:[a-z_][a-z0-9_-]*[$]?)$
Deployment (object)
Type:
object
Propertiescontainer (reference[ContainerConfig])
Name:
Container configuration
Description:
Provides information about the container for the plugin.
Required:
No
Referenced object:
ContainerConfig (see in the Objects section below)
host (reference[HostConfig])
Name:
Host configuration
Description:
Provides information about the container host for the plugin.
Required:
No
Referenced object:
HostConfig (see in the Objects section below)
imagePullPolicy (enum[string])
Name:
Image pull policy
Description:
When to pull the plugin image.
Required:
No
Values
Always Always
IfNotPresent If not present
Never Never
Default
"IfNotPresent"
HostConfig (object)
Type:
object
PropertiesBinds (list[string])
Name:
Volume Bindings
Description:
Volumes
Required:
No
List Items
Type:
string
Minimum:
1
Maximum:
32760
Must match pattern:
^.+:.+$
CapAdd (list[string])
Name:
Add capabilities
Description:
Add capabilities to the container.
Required:
No
List Items
Type:
string
CapDrop (list[string])
Name:
Drop capabilities
Description:
Drop capabilities from the container.
Required:
No
List Items
Type:
string
CgroupnsMode (enum[string])
Name:
CGroup namespace mode
Description:
CGroup namespace mode to use for the container.
Required:
No
Values
`` Empty
host Host
private Private
Dns (list[string])
Name:
DNS servers
Description:
DNS servers to use for lookup.
Required:
No
List Items
Type:
string
DnsOptions (list[string])
Name:
DNS options
Description:
DNS options to look for.
Required:
No
List Items
Type:
string
DnsSearch (list[string])
Name:
DNS search
Description:
DNS search domain.
Required:
No
List Items
Type:
string
ExtraHosts (list[string])
Name:
Extra hosts
Description:
Extra hosts entries to add
Required:
No
List Items
Type:
string
NetworkMode (string)
Name:
Network mode
Description:
Specifies either the network mode, the container network to attach to, or a name of a Docker network to use.
Kubernetes can be used as the “local” deployer, but this is typically not recommended for performance reasons. You can set up the Kubernetes deployer like this:
config.yaml
deployer:type:kubernetesconnection:host:localhost:6443cert:|Add your client cert in PEM format here.key:|Add your client key in PEM format here.cacert:|Add the server CA cert in PEM format here.
All options for the Kubernetes deployer
Type:
scope
Root object:
Config
Propertiesconnection (reference[Connection])
Name:
Connection
Description:
Docker connection information.
Required:
No
Referenced object:
Connection (see in the Objects section below)
pod (reference[Pod])
Name:
Pod
Description:
Pod configuration for the plugin.
Required:
No
Referenced object:
Pod (see in the Objects section below)
timeouts (reference[Timeouts])
Name:
Timeouts
Description:
Timeouts for the Docker connection.
Required:
No
Referenced object:
Timeouts (see in the Objects section below)
ObjectsAWSElasticBlockStoreVolumeSource (object)
Type:
object
Properties
None
AzureDiskVolumeSource (object)
Type:
object
Properties
None
AzureFileVolumeSource (object)
Type:
object
Properties
None
CSIVolumeSource (object)
Type:
object
Properties
None
CephFSVolumeSource (object)
Type:
object
Properties
None
CinderVolumeSource (object)
Type:
object
Properties
None
Config (object)
Type:
object
Propertiesconnection (reference[Connection])
Name:
Connection
Description:
Docker connection information.
Required:
No
Referenced object:
Connection (see in the Objects section below)
pod (reference[Pod])
Name:
Pod
Description:
Pod configuration for the plugin.
Required:
No
Referenced object:
Pod (see in the Objects section below)
timeouts (reference[Timeouts])
Name:
Timeouts
Description:
Timeouts for the Docker connection.
Required:
No
Referenced object:
Timeouts (see in the Objects section below)
ConfigMapVolumeSource (object)
Type:
object
Properties
None
Connection (object)
Type:
object
PropertiesbearerToken (string)
Name:
Bearer token
Description:
Bearer token to authenticate against the Kubernetes API with.
Required:
No
burst (int)
Name:
Burst
Description:
Burst value for query throttling.
Required:
No
Minimum:
0
Default
10
cacert (string)
Name:
CA certificate
Description:
CA certificate in PEM format to verify Kubernetes server certificate against.
VsphereVirtualDiskVolumeSource (see in the Objects section below)
VsphereVirtualDiskVolumeSource (object)
Type:
object
Properties
None
Logging
Logging is useful when you need more information about what is happening while you run a workload.
Basic logging
Here is the syntax for setting the log level:
config.yaml
log:level:info
Options for the level are:
debug: Extra verbosity useful to developers
info: General info
warning: Something went wrong, and you should know about it
error: Something failed. This inf o should help you figure out why
This sets which types of log output are shown or hidden. debug shows everything, while error shows the least, only showing error output. Each output shows more, rather than just its type, so debug, info, and warning still show error output.
Step logging
Step logging is useful for getting output from failed steps, or general debugging.
It is not recommended that you rely on this long term, as there may be better methods of debugging failed workflows.
To make the workflow output just error level logs when a step fails, set it as shown:
config.yaml
logged_outputs:error:level:error
Tip
The standard name for the output path when a step fails is called error, which happens to also be the name of the log level here, but these are independent values.
You can specify multiple types of outputs and their log levels. For example, if you also want to output success steps as debug, set it as shown: