pyfarm.agent.http.core.client module

HTTP Client

The client library the manager uses to communicate with the master server.

class pyfarm.agent.http.core.client.HTTPLog

Bases: object

Provides a wrapper around the http logger so requests and responses can be logged in a standardized fashion.

static queue(method, url, uid=None)

Logs the request we’re asking treq to queue

static response(response, uid=None)

Logs the return code of a request that treq completed

static error(failure, uid=None, method=None, url=None)

Called when the treq request experiences an error and calls the errback method.

pyfarm.agent.http.core.client.build_url(url, params=None)

Builds the full url when provided the base url and some url parameters:

>>> build_url("/foobar", {"first": "foo", "second": "bar"})
'/foobar?first=foo&second=bar'
>>> build_url("/foobar bar/")
''/foobar%20bar/'

Parameters:

• url (str) – The url to build off of.
• params (dict) – A dictionary of parameters that should be added on to url. If this value is not provided url will be returned by itself. Arguments to a url are unordered by default however they will be sorted alphabetically so the results are repeatable from call to call.

pyfarm.agent.http.core.client.http_retry_delay(initial=None, uniform=False, get_delay=<built-in method random of Random object at 0x19f12b0>, minimum=1)

Returns a floating point value that can be used to delay an http request. The main purpose of this is to ensure that not all requests are run with the same interval between then. This helps to ensure that if the same request, such as agents coming online, is being run on multiple systems they should be staggered a little more than they would be without the non-uniform delay.

Parameters:

• initial (int) – The initial delay value to start off with before any extra calculations are done. If this value is not provided the value provided to --http-retry-delay at startup will be used.
• uniform (bool) – If True then use the value produced by get_delay as a multiplier.
• get_delay (callable) – A function which should produce a number to multiply delay by. By default this uses random.random()
• minimum – Ensures that the value returned from this function is greater than or equal to a minimum value.

class pyfarm.agent.http.core.client.Request

Bases: pyfarm.agent.http.core.client.Request

Contains all the information used to perform a request such as the method, url, and original keyword arguments (kwargs). These values contain the basic information necessary in order to retry() a request.

retry(**kwargs)

When called this will rerun the original request with all of the original arguments to request()

Parameters:kwargs – Additional keyword arguments which should override the original keyword argument(s).

class pyfarm.agent.http.core.client.Response(deferred, response, request)

Bases: twisted.internet.protocol.Protocol

This class receives the incoming response body from a request constructs some convenience methods and attributes around the data.

Parameters:

• deferred (Deferred) – The deferred object which contains the target callback and errback.
• response – The initial response object which will be passed along to the target deferred.
• request (Request) – Named tuple object containing the method name, url, headers, and data.

data()

Returns the data currently contained in the buffer.

Raises RuntimeError:  Raised if this method id called before all data has been received.

json(loader=<function loads at 0x7f9c4921d320>)

Returns the json data from the incoming request

Raises:

• RuntimeError – Raised if this method id called before all data has been received.
• ValueError – Raised if the content type for this request is not application/json.

dataReceived(data)

Overrides Protocol.dataReceived() and appends data to _body.

connectionLost(reason=<twisted.python.failure.Failure <class 'twisted.internet.error.ConnectionDone'>>)

Overrides Protocol.connectionLost() and sets the _done when complete. When called with ResponseDone for reason this method will call the callback on _deferred

pyfarm.agent.http.core.client.request(method, url, **kwargs)

Wrapper around treq.request() with some added arguments and validation.

Parameters:

• method (str) – The HTTP method to use when making the request.
• url (str) – The url this request will be made to.
• data (str, list, tuple, set, dict) – The data to send along with some types of requests such as POST or PUT
• headers (dict) – The headers to send along with the request to url. Currently only single values per header are supported.
• callback (function) – The function to deliver an instance of Response once we receive and unpack a response.
• errback (function) – The function to deliver an error message to. By default this will use log.err().
• response_class (class) – The class to use to unpack the internal response. This is mainly used by the unittests but could be used elsewhere to add some custom behavior to the unpack process for the incoming response.

Raises NotImplementedError:  

Raised whenever a request is made of this function that we can’t implement such as an invalid http scheme, request method or a problem constructing data to an api.

pyfarm.agent.http.core.client.random() → x in the interval [0, 1).