Backends¶
There are currently four different backends. They can also be initialized in few different ways,
consistent with general settings. You can use the ARCA_BACKEND
setting
or you can pass a backend
keyword directly to Arca
.
The backend setting can be either a string, class or an instance. All the initializations shown bellow are equivalent,
but again, as mentioned in Configuring Arca, the backend
keyword cannot be overridden by settings
or environ variables.
from arca import Arca, DockerBackend
Arca(settings={"ARCA_BACKEND": "arca.backend.DockerBackend"})
Arca(settings={"ARCA_BACKEND": DockerBackend})
Arca(backend="arca.backend.DockerBackend")
Arca(backend=DockerBackend)
Arca(backend=DockerBackend())
Setting up backends is based on the same principle as setting up Arca
.
You can either pass keyword arguments when initializing the backend class
or you can use settings (described in more details in Configuring Arca). For example these two calls are equivalent:
from arca import Arca, DockerBackend
Arca(settings={
"ARCA_BACKEND": "arca.backend.DockerBackend",
"ARCA_BACKEND_PYTHON_VERSION": "3.6.4"
})
Arca(backend=DockerBackend(python_version="3.6.4"))
As mentioned in Basic options, there are two options common for all backends. (See that section for more details.)
- requirements_location
- requirements_timeout
- pipfile_location
- cwd
Current Environment¶
arca.backend.CurrentEnvironmentBackend
This backend is the default option, it runs the tasks with the same Python that’s used to run Arca, in a subprocess. There are no extra settings for this backend. All the requirements in repositories are ignored completely.
(possible settings prefixes: ARCA_CURRENT_ENVIRONMENT_BACKEND_
and ARCA_BACKEND_
)
Virtual Environment¶
arca.backend.VenvBackend
This backend uses the Python virtual environments to run the tasks. The environments are created from the Python
used to run Arca and they are shared between repositories that have the same exact requirement file.
The virtual environments are stored in folder venv
in folder
determined by the Arca
base_dir
setting, usually .arca
.
For installing requirements using Pipenv it must be available to be launched by the current user.
Disabling Pipenv can be done by setting the pipfile_location to None
.
(possible settings prefixes: ARCA_VENV_BACKEND_
and ARCA_BACKEND_
)
Docker¶
arca.backend.DockerBackend
This backend runs tasks in docker containers. To use this backend the user running Arca needs to be able to interact
with docker
(see documentation).
This backend first creates an image with requirements and dependencies installed so the installation only runs once.
By default the images are based on custom images, which have Python
and several build tools pre-installed.
These images are based on debian
(slim stretch
version) and use pyenv
to install Python.
You can specify you want to base your images on a different image with the inherit_image
setting.
Once Arca has an image with the requirements installed, it launches a container for each task and
kills it when the task finishes. This can be modify by setting keep_container_running
to True
,
then the container is not killed and can be used by different tasks running from the same repository, branch and commit.
This can save time on starting up containers before each task.
You can then kill the containers by calling DockerBackend
method stop_containers
.
If you’re using Arca on a CI/CD tool or somewhere docker images are not kept long-term, you can setup pushing
images with the installed requirements and dependencies to a docker registry and they will be pulled next time instead
of building them each time. It’s set using use_registry_name
and you’ll have to be logged in to docker
using docker login
. If you can’t use docker login
(for example in PRs on Travis CI), you can set
registry_pull_only
and Arca will only attempt to pull from the registry and not push new images.
Settings:
- python_version: What Python version should be used.
In theory any of
these versions could be used,
but only CPython 3.6 has been tested. The default is the Python version of the current environment.
This setting is ignored if
inherit_image
is set. - keep_container_running: When
True
, containers aren’t killed once the task finishes. Default isFalse
. - apt_dependencies: For some python libraries, system dependencies are required,
for example
libxml2-dev
andlibxslt-dev
are needed forlxml
. With this settings you can specify a list of system dependencies that will be installed via debianapt-get
. This setting is ignored ifinherit_image
is set since Arca can’t determined how to install requirements on an unknown system. - disable_pull: Disable pulling prebuilt Arca images from Docker Hub and build even the base images locally.
- inherit_image: If you don’t wish to use the Arca images you can specify what image should be used instead.
Pipenv must be available in the image if the repositories contain Pipenv files. Alternativelly Pipenv can be
disabled by setting the option pipfile_location to
None
. - use_registry_name: Uses this registry to store images with installed requirements and dependencies to, tries to pull image from the registry before building it locally to save time.
- registry_pull_only: Disables pushing to registry.
(possible settings prefixes: ARCA_DOCKER_BACKEND_
and ARCA_BACKEND_
)
Vagrant¶
arca.backend.VagrantBackend
If you’re extra paranoid you can use Vagrant to completely isolate the runtime in a Virtual Machine (VM).
This backend is actually a subclass of DockerBackend
and uses docker in the VM to run the tasks.
Docker and Vagrant must be runnable by the current user.
The backend works by building the image with requirements and dependencies locally and pushing it to registry using use_to_registry_name
.
Then a VM is launched and the image is pulled there from the registry.
This takes some time when first launching the VM, but if the VM is reused often, the upload/download time is shorted.
The built images are also not lost when the VM is destroyed.
The backend inherits all the settings of DockerBackend
(keep_containers_running is True
by default) and has these extra settings:
- box: Vagrant box used in the VM. Either has to have docker version >= 1.8 or not have docker at all, in which case it will be installed when spinning up the VM. The default is ailispaw/barge.
- provider: Vagrant provider, default is
virtualbox
. Visit vagrant docs for more. - quiet: Tells Vagrant and Fabric (which is used to run the task in the VM) to be quiet. Default is
True
. Vagrant and Docker output is logged in separate files for each run in a folderlogs
in theArca
base_dir
. The filename is logged in the arca logger (see bellow) - keep_vm_running: Should the VM be kept up once a task finishes? By default
False
. If set toTrue
,stop_vm
can be used to stop the VM. - destroy: When stopping the VM (either after a task or after
stop_vm()
is called), should the VM be destroyed (= deleted) or just halted?False
by default.
(possible settings prefixes: ARCA_VAGRANT_BACKEND_
and ARCA_BACKEND_
)
Your own¶
You can also create your own backend and pass it to Arca
. It has be a subclass of arca.BaseBackend
and
it has to implement its run
method.