Cornflow-server architecture

Components

../_images/architecture.png

Components with the connections between them

Cornflow-server

The main component. It stores the user information, their instances, executions, etc. It is built in python+flask and is served via an REST API.

Airflow service

Apache Foundation project that is used to handle the new executions. It manages the set of workers (machines or processors) that will execute the solution methods on the instances.

Airflow has its own nomenclature, which includes some concepts we refer to in Cornflow, such as: DAGs, DAG-runs (or dagrun), workers, etc.

A DAG usually represents an Abstract Problem. A Dagrun represents an Execution.

Airflow is also built on top of flask+python. The project’s site is: airflow.apache.org/

Worker

A worker is a machine that is capable of executing an airflow task. Some of these workers will have a commercial solver license (e.g., Gurobi or CPLEX) in order to use these solvers.

Client

Any library that uses the REST API to use Cornflow. There are two official clients that are currently maintained: a SPA web app and a python package.

The REST API is created with Open API standards (see REST API), so it should be fairly easy to build new apis.

Main data flows

../_images/main_flows.png

The main data flows between components

Create instance (data):

  1. Client calls Cornflow and sends instance.

  2. Cornflow validates the instance matches the app’s jsonschema.

  3. Cornflow saves the instance and returns the instance code to client.

Solve instance (instance, config, dag):

  1. Client calls Cornflow and gives instance code, execution configuration and eventually an existing solution.

  2. Cornflow validates the instance matches the jsonschema of the dag to execute.

  3. Cornflow calls airflow dag and assigns dagrun code to execution.

  4. Cornflow creates execution and returns the execution code to client.

  5. Airflow creates a rundag for the selected dag and sends it to a worker.

  6. The worker asks Cornflow for the instance, the config, and eventually the solution data.

  7. The worker solves the problem.

  8. The worker sends Cornflow the results (solution and log) of the execution.

Retrieve results (execution):

  1. Client asks Cornflow for the results to an execution.

  2. Cornflow returns the execution solution and / or log.

Secondary flows

../_images/secondary_flows.png

The secondary data flows between components

Interrupt execution (execution):

  1. Client asks Cornflow to stop an execution.

  2. Cornflow checks the current status and asks Airflow to stop the execution.

  3. Airflow kills the process and worker by deleting the execution.

Retrieve status (execution):

  1. Client asks Cornflow for the results to an execution.

  2. If Cornflow does not know it: it asks Airflow for the status.

  3. Cornflow returns the status to the user.

Retrieve schema:

  1. Client asks Cornflow for the schema to a problem.

  2. If Cornflow does not know it: it asks Airflow for the schema of a problem.

  3. Airflow returns the schema if it exists.

Get instance data:

  1. Client asks Cornflow for the data of an instance.

  2. Cornflow returns the data of the instance if it exists.

Save manual execution:

  1. Client sends Cornflow a solved problem.

  2. Cornflow check the problem matches the schema.

  3. Cornflow saves the execution.