Configuration Object¶

Basic module used for reading configuration data into PyFarm in various forms.

class pyfarm.core.config.Configuration(name, version=None, cwd=None)¶

Bases: builtins.dict

Main object responsible for finding, loading, and merging configuration data. By default this class does nothing until load() is called. Once this method is called Configuration class will populate itself with data loaded from the configuration files. The configuration files themselves can be loaded from multiple location depending on the system’s setup. For example on Linux you might end up attempting to load these files for pyfarm.agent v1.2.3:

Override paths set by DEFAULT_ENVIRONMENT_PATH_VARIABLE. By default this path will not be set, this is only an example. * /tmp/pyfarm/agent/1.2.3/agent.yml * /tmp/pyfarm/agent/1.2/agent.yml * /tmp/pyfarm/agent/1/agent.yml * /tmp/pyfarm/agent/agent.yml

Paths relative to the current working directory or the directory provided to cwd when Configuration was instanced. * etc/pyfarm/agent/1.2.3/agent.yml * etc/pyfarm/agent/1.2/agent.yml * etc/pyfarm/agent/1/agent.yml * etc/pyfarm/agent/agent.yml

User’s home directory * ~/.pyfarm/agent/1.2.3/agent.yml * ~/.pyfarm/agent/1.2/agent.yml * ~/.pyfarm/agent/1/agent.yml * ~/.pyfarm/agent/agent.yml

System level paths * /etc/pyfarm/agent/1.2.3/agent.yml * /etc/pyfarm/agent/1.2/agent.yml * /etc/pyfarm/agent/1/agent.yml * /etc/pyfarm/agent/agent.yml

Finally, if we don’t locate a configuration file in any of the above paths we’ll use the file which was installed along side the source code.

Configuration will only attempt to load data from files which exist on the file system when load() is called. If multiple files exist the data will be loaded from each file with the successive data overwriting the value from the previously loaded configuration file. So if you have two files containing the same data:

• /etc/pyfarm/agent/agent.yml

  env:
      a: 0
  foo: 1
  bar: true
  

• etc/pyfarm/agent/1.2.3/agent.yml

  env:
      a: 1
      b: 1
  foo: 0
  

You’ll end up with a single merged configuration. Please note that the only keys which will be merged in the configuration are the env key. Configuration files are meant to store simple data and while it can be used to store more complicate data it won’t merge any other data structures.

env:
    a: 1
    b: 1
foo: 0
bar: true

Variables:

DEFAULT_SYSTEM_ROOT (string) – The system level directory that we should look for configuration files in. This path is platform dependent: Linux - /etc/ Mac - /Library/ Windows - %ProgramData%. An environment variable that varies depending on the Windows version. See Microsoft’s docs: https://www.microsoft.com/security/portal/mmpc/shared/variables.aspx The value built here will be copied onto the instance as system_root DEFAULT_USER_ROOT (string) – The user level directory that we should look for configuration files in. This path is platform dependent: Linux/Mac - ~ (home directory) Windows - %APPDATA%. An environment variable that varies depending on the Windows version. See Microsoft’s docs: https://www.microsoft.com/security/portal/mmpc/shared/variables.aspx The value built here will be copied onto the instance as user_root DEFAULT_FILE_EXTENSION (string) – The default file extension of the configuration files. This will default to .yml and will be copied to file_extension when the class is instanced. DEFAULT_LOCAL_DIRECTORY_NAME (string) – A directory local to the current process which we should search for configuration files in. This will default to etc and will be copied to local_dir when the class is instanced. DEFAULT_PARENT_APPLICATION_NAME (string) – The base name of the parent application. This used used to build child directories and will default to pyfarm. DEFAULT_ENVIRONMENT_PATH_VARIABLE (string) – A environment variable to search for a configuration path in. The value defined here, which defaults to PYFARM_CONFIG_ROOT, will be read from the environment when Configuration is instanced. This allows for an non-standard configuration location to be loaded first for testing forced-override of the configuration. DEFAULT_TEMP_DIRECTORY_ROOT – The directory which will store any temporary files.

Parameters:

name (string) – The name of the configuration itself, typically ‘master’ or ‘agent’. This may also be the name of a package such as ‘pyfarm.agent’. When the package name is provided we can usually automatically determine the version number. version (string) – The version the version of the program running. cwd (string) – The current working directory to construct the local path from. If not provided then we’ll use os.getcwd() to determine the current working directory.

_expandvars(value)¶

Performs variable expansion for value. This method is run when a string value is returned from get() or __getitem__(). The default behavior of this method is to recursively expand variables using sources in the following order:

• The environment, os.environ
• The environment (from the configuration), env
• Other values in the configuration
• ~ to the user’s home directory

For example, the following configuration:

foo: foo
bar: bar
foobar: $foo/$bar
path: ~/$foobar/$TEST

Would result in the following assuming $TEST is an environment variable set to somevalue and the current user’s name is user:

{
    "foo": "foo",
    "bar": "bar",
    "foobar": "foo/bar",
    "path": "/home/user/foo/bar/somevalue"
}

DEFAULT_ENVIRONMENT_PATH_VARIABLE = 'PYFARM_CONFIG_ROOT'¶

DEFAULT_FILE_EXTENSION = '.yml'¶

DEFAULT_LOCAL_DIRECTORY_NAME = 'etc'¶

DEFAULT_PARENT_APPLICATION_NAME = 'pyfarm'¶

DEFAULT_SYSTEM_ROOT = '/etc'¶

DEFAULT_TEMP_DIRECTORY_ROOT = '/tmp/pyfarm'¶

DEFAULT_USER_ROOT = '/home/docs'¶

MAX_EXPANSION_RECURSION = 10¶

directories(validate=True, unversioned_only=False)¶

Returns a list of platform dependent directories which may contain configuration files.

Parameters:	validate (bool) – When True this method will only return directories which exist on disk. unversioned_only (bool) – When True this method will only return versionless directories instead of both versionless and versioned directories.

files(validate=True, unversioned_only=False)¶

Returns a list of configuration files.

Parameters:	validate (bool) – When True this method will only return files which exist on disk. Note This method calls directories() and will be passed the value that is provided to validate here. unversioned_only (bool) – See the keyword documentation for unversioned_only in directories`()

get(key, default=None)¶

Overrides dict.get() to provide internal variable expansion through _expandvars().

load(environment=None)¶

Loads data from the configuration files. Any data present in the env key in the configuration files will update environment

Parameters:	environment (dict) – A dictionary to load data in the env key from the configuration files into. This would typically be set to os.environ so the environment itself could be updated.

split_version(sep='.')¶

Splits self.version into a tuple of individual versions. For example 1.2.3 would be split into ['1', '1.2', '1.2.3']

pyfarm.core.config.read_env(envvar, default=<object object at 0x7f4daa778660>, warn_if_unset=False, eval_literal=False, raise_eval_exception=True, log_result=True, desc=None)¶

Lookup and evaluate an environment variable.

Parameters:

envvar (string) – The environment variable to lookup in os.environ default (object) – Alternate value to return if envvar is not present. If this is instead set to NOTSET then an exception will be raised if envvar is not found. warn_if_unset (bool) – If True, log a warning if the value being returned is the same as default eval_literal – if True, run literal_eval() on the value retrieved from the environment raise_eval_exception (bool) – If True and we failed to parse envvar with literal_eval() then raise a EnvironmentKeyError log_result (bool) – If True, log the query and result to INFO. If False, only log the query itself to DEBUG. This keyword mainly exists so environment variables such as PYFARM_SECRET or PYFARM_DATABASE_URI stay out of log files. desc (string) – Describes the purpose of the value being returned. This may also be read in at the time the documentation is built.

pyfarm.core.config.read_env_bool(*args, **kwargs)¶

Wrapper around read_env() which converts environment variables to boolean values. Please see the documentation for read_env() for additional information on exceptions and input arguments.

Raises:	AssertionError – raised if a default value is not provided TypeError – raised if the environment variable found was a string and could not be converted to a boolean.

pyfarm.core.config.read_env_number(*args, **kwargs)¶

Wrapper around read_env() which will read a numerical value from an environment variable. Please see the documentation for read_env() for additional information on exceptions and input arguments.

Raises TypeError:
	raised if we either failed to convert the value from the environment variable or the value was not a float, integer, or long

pyfarm.core.config.read_env_strict_number(*args, **kwargs)¶

Strict version of read_env_number() which will only return an integer

Parameters:	number_type – the type of number(s) this function must return
Raises:	AsssertionError – raised if the number_type keyword is not provided (required to check the type on output) TypeError – raised if the type of the result is not an instance of number_type

Enums¶

Provides enum values for certain aspect of PyFarm. See below for more detailed information.

Utilities¶

General utility functions that are not specific to individual components of PyFarm.

class pyfarm.core.utility.ImmutableDict(iterable=None, **kwargs)¶

Bases: builtins.dict

A basic immutable dictionary that’s built on top of Python’s standard dict class. Once __init__() has been run the contents of the instance can no longer be modified

clear(*args, **kwargs)¶

pop(*args, **kwargs)¶

popitem(*args, **kwargs)¶

setdefault(*args, **kwargs)¶

update(*args, **kwargs)¶

class pyfarm.core.utility.PyFarmJSONEncoder(skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False, indent=None, separators=None, default=None)¶

Bases: json.encoder.JSONEncoder

encode(o)¶

class pyfarm.core.utility.convert¶

Bases: builtins.object

Namespace containing various static methods for converting data.

Some staticmethods are named the same as builtin types. The name indicates the expected result but the staticmethod may not behave the same as the equivalently named Python object. Read the documentation for each staticmethod to learn the differences, expected input and output.

static bool(value)¶

Converts value into a boolean object. This function mainly exits so human-readable booleans such as ‘yes’ or ‘y’ can be handled in a single location. Internally it does not use bool() and instead checks value against pyfarm.core.enums.BOOLEAN_TRUE and pyfarm.core.enums.BOOLEAN_FALSE.

Parameters:	value – The value to attempt to convert to a boolean. If this value is a string it will be run through .lower().strip() first.
Raises ValueError:
	Raised if we can’t convert value to a true boolean object

static bytetomb(value)¶

Convert bytes to megabytes

>>> convert.bytetomb(10485760)
10.0

static list(value, sep=', ', strip=True, filter_empty=True)¶

Converts value into a list object by splitting on sep.

Parameters:	value (str) – The string we should convert into a list sep (str) – The string that we should split value by. strip (bool) – If True, strip extra whitespace from the results so 'foo, bar' becomes ['foo', 'bar'] filter_empty (bool) – If True, any result that evaluates to False will be removed so 'foo,,' would become ['foo']

static mbtogb(value)¶

Convert megabytes to gigabytes

>>> convert.mbtogb(2048)
2.0

static none(value)¶

Converts value into None. This function mainly exits so human-readable values such as ‘None’ or ‘null’ can be handled in a single location. Internally this checks value against pyfarm.core.enums.NONE

Parameters:	value – The value to attempt to convert to None. If this value is a string it will be run through .lower().strip() first.
Raises ValueError:
	Raised if we can’t convert value to None

static ston(value, types=(<class 'int'>, <class 'float'>, <class 'complex'>))¶

Converts a string to an integer or fails with a useful error message

Parameters:	value (string) – The value to convert to an integer
Raises:	ValueError – Raised if value could not be converted using literval_eval() TypeError – Raised if value was not converted to a float, integer, or long