AiiDA

AiiDA aims to help researchers with managing the complex workflows of high-throughput research and making their results fully reproducible. It does so by providing the following key features:

• Workflows: Write complex, auto-documenting workflows in python, managing both simple user-defined functions and external executables on local or remote computers. The AiiDA workflow engine enables users to encode their workflow logic in functional building blocks and combine them in sophisticated simulations. The event-based engine can deal with demanding high-throughput workloads, supporting tens of thousands of processes per hour with full checkpointing.    
     

• Data provenance: Automatically track inputs, outputs & metadata of all calculations and data in a provenance graph for full reproducibility, which means being able to reconstruct the complete history of each calculation or scientific result. Perform fast queries on graphs containing millions of nodes.
     

• HPC interface: Move your calculations to a different computer by changing one line of code. AiiDA is compatible with many schedulers, including SLURM, PBS Pro, torque, SGE or LSF. The daemon prepares  the necessary input files, sends them to the cluster, submits a new job to the scheduler, and continues monitoring the status of calculations, retrieving and parsing the relevant files as soon as the job finishes.
     

• Plugin interface: Extend AiiDA with plugins for new simulation codes (input generation & parsing), data types, schedulers, transport modes and more. The public AiiDA registry is the central place to find AiiDA plugins, listing plugins, authors, installation instructions and which extensions they provide.
     

• Open Science: Export subsets of your provenance graph and share them with peers or make them available online for everyone on the Materials Cloud. When uploaded to this platform, peers can browse the graph interactively, download individual files or the whole database, and start their research right from where the original author left off.
     

Open source: AiiDA is developed publicly on the aiida-core GitHub repository and is released under the MIT open source license, making it suitable for use both in academic and commercial settings. We are strong supporters of open source software; nevertheless, AiiDA plugins can connect both to open source and commercial codes, and plugin developers can make the choice to develop their plugin privately or under different licenses if they want to.

AiiDA is an open-source Python infrastructure to help researchers with automating, managing, persisting, sharing and reproducing the complex workflows associated with modern computational science and all associated data

DATA STORAGE AND QUERYING

By default, data are stored both in a file repository and in a PostgreSQL database for efficient querying. The data does not only contain the raw and parsed inputs and outputs, but also the whole provenance graph. Sharing is enabled by export functionalities to export the full provenance graph or parts of it in the form of archive files, and then reimport it in a different AiiDA instance.

The directed graph is stored in the SQL database; AiiDA provides a QueryBuilder engine to express general graph queries with filtering and projections in a simple and uniform python-like syntax, without the need to know SQL or the underlying design of the database. The AiiDA ORM leverages the SQLAlchemy python ORM for the connection to the PostgreSQL database.

Moreover, the data management backend has been completely abstracted and modularized in the last major 2.0 release. This means not only that exported archive files can now be accessed and explored directly without needing to import them into AiiDA first, but also that the backend classes can be extended and alternatives to the default file repository + PostgreSQL could be developed in the future (for example, to use remote resources such as cloud object stores).

ENGINE

AiiDA defines a rich and powerful workflow language that allows declarative definition of inputs, outputs, error codes and the outline of the code, making it self-documenting, and enabling automatic input and output validation and automatic documentation generation features. In addition, AiiDA provides python decorators that make it straightforward, by just adding one line of code, to convert any python function (used e.g. for pre- or post-processing) into an AiiDA process, that gets automatically tracked in the AiiDA provenance graph along with links to its inputs and outputs.

The engine that automates the simulations and the workflows is based on an event queue (using RabbitMQ as the queue manager) to enable "real-time" instantaneous reaction to events, like submission or completion of a calculation. It is also empowered by advanced algorithms to provide sustained throughput on HPC resources with automatic rate-limiting and batching of connections to avoid overloading. Additional algorithms ensure robustness to downtimes and temporary failures, with an exponential backoff mechanism and the possibility to pause and restart any process.

The engine is run by a multi-slot, multi-process daemon managed via the Circus python library. Control of the workflow is maintained centrally, allowing for stopping and restarting/rebooting the AiiDA server without any data loss; the engine will pick up the workflow and continue execution from where it was left off (while e.g. simulations continue to run on HPC supercomputers).