PyFarm - A Python Based Distributed Job System¶
Attention
These documents are still undergoing revision
PyFarm is a Python based distributed job system which is intended to be easy to deploy and maintain. Initially developed for individual use new revisions have been engineered with larger deployments in mind. It’s a system that’s focused on providing a framework for customized command execution while taking into account resource management.
Since this project is under active development, if you have specific questions or comments you’re always welcome to post to the google groups discussion list or via email to pyfarm@googlegroups.com.
Components¶
PyFarm has several distinct roles that it must fill in order to accomplish what it was designed to do. Because of this the project has been broken down into three components to help manage scope and limit interdependencies as much as possible.
Core¶
| | Documentation | Source Code
This is the base library which provides a few modules and objects which the other two components of PyFarm use. The primary purpose of this component is to centralize code basic functionality such as logging and configuration.
Master¶
| | Documentation | Source Code
This is the component responsible for storing jobs and tasks to run as well as allocation of work to remote hosts. This component is the largest of the three components and contains the code necessary to run the web interface, interact with the relational database, REST APIs and scheduler.
Agent¶
| | Documentation | Source Code
This component controls the execution of commands on remote systems as instructed by the master. This component also contains the job types which are used as a framework for executing commands.
Contents