Commands

Note

The default values provided are based on the configuration at the time this page was generated. They may not be the same defaults you will see.

Standard Commands

pyfarm-agent

usage: pyfarm-agent [status|start|stop]

positional arguments:
  {start,stop,status}   individual operations pyfarm-agent can run
    start               starts the agent
    stop                stops the agent
    status              query the 'running' state of the agent

optional arguments:
  -h, --help            show this help message and exit

Agent Network Service:
  Main flags which control the network services running on the agent.

  --port PORT           The port number which the agent is either running on
                        or will run on when started. This port is also
                        reported the master when an agent starts. [default:
                        None]
  --host HOST           The host to communicate with or hostname to present to
                        the master when starting. Defaults to the fully
                        qualified hostname.
  --agent-api-username AGENT_API_USERNAME
                        The username required to access or manipulate the
                        agent using REST. [default: agent]
  --agent-api-password AGENT_API_PASSWORD
                        The password required to access manipulate the agent
                        using REST. [default: agent]
  --agent-id AGENT_ID   The UUID used to identify this agent to the master. By
                        default the agent will attempt to load a cached value
                        however a specific UUID could be provided with this
                        flag.
  --agent-id-file AGENT_ID_FILE
                        The location to store the agent's id. By default the
                        path is platform specific and defined by the
                        `agent_id_file_platform_defaults` key in the
                        configuration. [default: /etc/pyfarm/agent/uuid.dat]

Network Resources:
  Resources which the agent will be communicating with.

  --master MASTER       This is a convenience flag which will allow you to set
                        the hostname for the master. By default this value
                        will be substituted in --master-api
  --master-api MASTER_API
                        The location where the master's REST api is located.
                        [default: None]
  --master-api-version MASTER_API_VERSION
                        Sets the version of the master's REST api the agent
                        shoulduse [default: None]

Process Control:
  These settings apply to the parent process of the agent and contribute to
  allowing the process to run as other users or remain isolated in an
  environment. They also assist in maintaining the 'running state' via a
  process id file.

  --pidfile PIDFILE     The file to store the process id in. [default: None]
  -n, --no-daemon       If provided then do not run the process in the
                        background.
  --chdir CHDIR         The working directory to change the agent into upon
                        launch
  --uid UID             The user id to run the agent as. *This setting is
                        ignored on Windows.*
  --gid GID             The group id to run the agent as. *This setting is
                        ignored on Windows.*
  --pdb-on-unhandled    When set pdb.set_trace() will be called if an
                        unhandled error is caught in the logger

pyfarm-agent is a command line client for working with a local agent. You can
use it to stop, start, and report the general status of a running agent
process.
usage: pyfarm-agent [status|start|stop] status [-h]

optional arguments:
  -h, --help  show this help message and exit
usage: pyfarm-agent [status|start|stop] start [-h] [--state STATE]
                                              [--time-offset TIME_OFFSET]
                                              [--ntp-server NTP_SERVER]
                                              [--ntp-server-version NTP_SERVER_VERSION]
                                              [--no-pretty-json]
                                              [--shutdown-timeout SHUTDOWN_TIMEOUT]
                                              [--updates-drop-dir UPDATES_DROP_DIR]
                                              [--run-control-file RUN_CONTROL_FILE]
                                              [--farm-name FARM_NAME]
                                              [--cpus CPUS] [--ram RAM]
                                              [--ram-check-interval RAM_CHECK_INTERVAL]
                                              [--ram-max-report-frequency RAM_MAX_REPORT_FREQUENCY]
                                              [--ram-report-delta RAM_REPORT_DELTA]
                                              [--master-reannounce MASTER_REANNOUNCE]
                                              [--log LOG]
                                              [--capture-process-output]
                                              [--task-log-dir TASK_LOG_DIR]
                                              [--ip-remote IP_REMOTE]
                                              [--enable-manhole]
                                              [--manhole-port MANHOLE_PORT]
                                              [--manhole-username MANHOLE_USERNAME]
                                              [--manhole-password MANHOLE_PASSWORD]
                                              [--html-templates-reload]
                                              [--static-files STATIC_FILES]
                                              [--http-retry-delay-offset HTTP_RETRY_DELAY_OFFSET]
                                              [--http-retry-delay-factor HTTP_RETRY_DELAY_FACTOR]
                                              [--jobtype-no-cache]

optional arguments:
  -h, --help            show this help message and exit

General Configuration:
  These flags configure parts of the agent related to hardware, state, and
  certain timing and scheduling attributes.

  --state STATE         The current agent state, valid values are ['disabled',
                        'offline', 'running', 'online']. [default: online]
  --time-offset TIME_OFFSET
                        If provided then don't talk to the NTP server at all
                        to calculate the time offset. If you know for a fact
                        that this host's time is always up to date then
                        setting this to 0 is probably a safe bet.
  --ntp-server NTP_SERVER
                        The default network time server this agent should
                        query to retrieve the real time. This will be used to
                        help determine the agent's clock skew if any. Setting
                        this value to '' will effectively disable this query.
                        [default: None]
  --ntp-server-version NTP_SERVER_VERSION
                        The version of the NTP server in case it's running an
                        olderor newer version. [default: None]
  --no-pretty-json      If provided do not dump human readable json via the
                        agent's REST api
  --shutdown-timeout SHUTDOWN_TIMEOUT
                        How many seconds the agent should spend attempting to
                        inform the master that it's shutting down.
  --updates-drop-dir UPDATES_DROP_DIR
                        The directory to drop downloaded updates in. This
                        should be the same directory pyfarm-supervisor will
                        look for updates in. [default: None]
  --run-control-file RUN_CONTROL_FILE
                        The path to a file that will signal to the supervisor
                        that agent is supposed to be restarted if it stops for
                        whatever reason.[default:
                        /tmp/pyfarm/agent/should_be_running]
  --farm-name FARM_NAME
                        The name of the farm the agent should join. If unset,
                        the agent will join any farm.

Physical Hardware:
  Command line flags which describe the hardware of the agent.

  --cpus CPUS           The total amount of cpus installed on the system.
                        Defaults to the number of cpus installed on the
                        system.
  --ram RAM             The total amount of ram installed on the system in
                        megabytes. Defaults to the amount of ram the system
                        has installed.

Interval Controls:
  Controls which dictate when certain internal intervals should occur.

  --ram-check-interval RAM_CHECK_INTERVAL
                        How often ram resources should be checked for changes.
                        The amount of memory currently being consumed on the
                        system is checked after certain events occur such as a
                        process but this flag specifically controls how often
                        we should check when no such events are occurring.
                        [default: None]
  --ram-max-report-frequency RAM_MAX_REPORT_FREQUENCY
                        This is a limiter that prevents the agent from
                        reporting memory changes to the master more often than
                        a specific time interval. This is done in order to
                        ensure that when 100s of events fire in a short period
                        of time cause changes in ram usage only one or two
                        will be reported to the master. [default: None]
  --ram-report-delta RAM_REPORT_DELTA
                        Only report a change in ram if the value has changed
                        at least this many megabytes. [default: None]
  --master-reannounce MASTER_REANNOUNCE
                        Controls how often the agent should reannounce itself
                        to the master. The agent may be in contact with the
                        master more often than this however during long period
                        of inactivity this is how often the agent will
                        'inform' the master the agent is still online.

Logging Options:
  Settings which control logging of the agent's parent process and/or any
  subprocess it runs.

  --log LOG             If provided log all output from the agent to this
                        path. This will append to any existing log data.
                        [default: None]
  --capture-process-output
                        If provided then all log output from each process
                        launched by the agent will be sent through agent's
                        loggers.
  --task-log-dir TASK_LOG_DIR
                        The directory tasks should log to.

Network Service:
  Controls how the agent is seen or interacted with by external services
  such as the master.

  --ip-remote IP_REMOTE
                        The remote IPv4 address to report. In situation where
                        the agent is behind a firewall this value will
                        typically be different.

Manhole Service:
  Controls the manhole service which allows a telnet connection to be made
  directly into the agent as it's running.

  --enable-manhole      When provided the manhole service will be started once
                        the reactor is running.
  --manhole-port MANHOLE_PORT
                        The port the manhole service should run on if enabled.
  --manhole-username MANHOLE_USERNAME
                        The telnet username that's allowed to connect to the
                        manhole service running on the agent.
  --manhole-password MANHOLE_PASSWORD
                        The telnet password to use when connecting to the
                        manhole service running on the agent.

HTTP Configuration:
  Options for how the agent will interact with the master's REST api and how
  it should run it's own REST api.

  --html-templates-reload
                        If provided then force Jinja2, the html template
                        system, to check the file system for changes with
                        every request. This flag should not be used in
                        production but is useful for development and debugging
                        purposes.
  --static-files STATIC_FILES
                        The default location where the agent's http server
                        should find static files to serve.
  --http-retry-delay-offset HTTP_RETRY_DELAY_OFFSET
                        If a http request to the master has failed, wait at
                        least this amount of time before resending the
                        request.
  --http-retry-delay-factor HTTP_RETRY_DELAY_FACTOR
                        The value provided here is used in combination with
                        --http-retry-delay-offset to calculate the retry
                        delay. This is used as a multiplier against random()
                        before being added to the offset.

Job Types:
  --jobtype-no-cache    If provided then do not cache job types, always
                        directly retrieve them. This is beneficial if you're
                        testing the agent or a new job type class.
usage: pyfarm-agent [status|start|stop] stop [-h] [--no-wait]

optional arguments:
  -h, --help  show this help message and exit

optional flags:
  Flags that control how the agent is stopped

  --no-wait   If provided then don't wait on the agent to shut itself down. By
              default we would want to wait on each task to stop so we can
              catch any errors and then finally wait on the agent to shutdown
              too. If you're in a hurry or stopping a bunch of agents at once
              then setting this flag will let the agent continue to stop
              itself without waiting for each agent
usage: pyfarm-supervisor [-h] [--updates-drop-dir UPDATES_DROP_DIR]
                         [--agent-package-dir AGENT_PACKAGE_DIR]
                         [--pidfile PIDFILE] [-n] [--chdir CHDIR] [--uid UID]
                         [--gid GID]

Start and monitor the agent process

optional arguments:
  -h, --help            show this help message and exit
  --updates-drop-dir UPDATES_DROP_DIR
                        Where to look for agent updates
  --agent-package-dir AGENT_PACKAGE_DIR
                        Path to the actual agent code
  --pidfile PIDFILE     The file to store the process id in. [default: None]
  -n, --no-daemon       If provided then do not run the process in the
                        background.
  --chdir CHDIR         The directory to chdir to upon launch.
  --uid UID             The user id to run the supervisor as. *This setting is
                        ignored on Windows.*
  --gid GID             The group id to run the supervisor as. *This setting
                        is ignored on Windows.*

Development Commands

pyfarm-dev-fakerender

usage: pyfarm-dev-fakerender [-h] [--ram RAM] [--duration DURATION]
                             [--return-code RETURN_CODE]
                             [--duration-jitter DURATION_JITTER]
                             [--ram-jitter RAM_JITTER] -s START [-e END]
                             [-b BY] [--spew] [--segfault]

Very basic command line tool which vaguely simulates a render.

optional arguments:
  -h, --help            show this help message and exit
  --ram RAM             How much ram in megabytes the fake command should
                        consume
  --duration DURATION   How many seconds it should take to run this command
  --return-code RETURN_CODE
                        The return code to return, declaring this flag
                        multiple times will result in a random return code.
                        [default: [0]]
  --duration-jitter DURATION_JITTER
                        Randomly add or subtract this amount to the total
                        duration
  --ram-jitter RAM_JITTER
                        Randomly add or subtract this amount to the ram
  -s START, --start START
                        The start frame. If no other flags are provided this
                        will also be the end frame.
  -e END, --end END     The end frame
  -b BY, --by BY        The by frame
  --spew                Spews lots of random output to stdout which is
                        generally a decent stress test for log processing
                        issues. Do note however that this will disable the
                        code which is consuming extra CPU cycles. Also, use
                        this option with care as it can generate several
                        gigabytes of data per frame.
  --segfault            If provided then there's a 25% chance of causing a
                        segmentation fault.

pyfarm-dev-fakework

usage: pyfarm-dev-fakework [-h] [--master-api MASTER_API]
                           [--agent-api AGENT_API] [--jobtype JOBTYPE]
                           [--job JOB]

Quick and dirty script to create a job type, a job, and some tasks which are
then posted directly to the agent. The primary purpose of this script is to
test the internal of the job types

optional arguments:
  -h, --help            show this help message and exit
  --master-api MASTER_API
                        The url to the master's api [default:
                        http://127.0.0.1/api/v1]
  --agent-api AGENT_API
                        The url to the agent's api [default:
                        http://127.0.0.1:50000/api/v1]
  --jobtype JOBTYPE     The job type to use [default: FakeRender]
  --job JOB             If provided then this will be the job we pull tasks
                        from and assign to the agent. Please note we'll only
                        be pulling tasks that aren't running or assigned.