Computational graph¶

class blocks.graph.ComputationGraph(outputs)[source]¶

Bases: object

Encapsulates a managed Theano computation graph.

This implies that it not only contains the variables required to compute the given outputs, but also all the auxiliary variables and updates that were attached to these variables through the annotation system.

All variables are presented in topologically sorted order according to the apply nodes that they are an input to.

Parameters:	outputs ((list of) `TensorVariable`) – The output(s) of the computation graph.

inputs¶: list of TensorVariable – The inputs of the computation graph. This does not include shared variables and constants.

shared_variables¶: list of TensorSharedVariable – All the shared variables in the graph.

parameters¶: list of TensorSharedVariable – All the shared variables which have the PARAMETER role.

outputs¶: list of TensorVariable – The outputs of the computations graph (as passed to the constructor).

auxiliary_variables¶: list of TensorVariable – All variables which have the AUXILIARY role.

intermediary_variables¶: list of TensorVariable – Any variable that is not part of inputs or outputs.

variables¶: list of TensorVariable – All variables (including auxiliary) in the managed graph.

scans¶: list of Scan – All Scan ops used in this computation graph.

scan_variables¶: list of TensorVariable – All variables of the inner graphs of Scan ops.

updates¶: TensorSharedVariable updates – All the updates found attached to the annotations.

auxiliary_variables

dict_of_inputs()[source]¶: Return a mapping from an input name to the input.

get_snapshot(data)[source]¶

Evaluate all role-carrying Theano variables on given data.

Parameters:	data (dict of (data source, data) pairs) – Data for input variables. The sources should match with the names of the input variables.
Returns:
Return type:	Dictionary of (variable, variable value on given data) pairs.

get_theano_function(additional_updates=None, **kwargs)[source]¶

Create Theano function from the graph contained.

Parameters:	*kwargs (dict*) – Keyword arguments to theano.function. Useful for specifying compilation modes or profiling.

has_inputs(variable)[source]¶

Check if a variable depends on input variables.

Returns:	`True` if the given variable depends on input variables, `False` otherwise.
Return type:	bool

inputs: Inputs to the graph, excluding constants and shared variables.

intermediary_variables

parameters

replace(replacements)[source]¶

Replace certain variables in the computation graph.

Parameters:	replacements (dict) – The mapping from variables to be replaced to the corresponding substitutes.

Examples

>>> import theano
>>> from theano import tensor, function
>>> x = tensor.scalar('x')
>>> y = x + 2
>>> z = y + 3
>>> a = z + 5

Let’s suppose we have dependent replacements like

>>> replacements = {y: x * 2, z: y * 3}
>>> cg = ComputationGraph([a])
>>> theano.pprint(a)  
'(((x + TensorConstant{2}) + TensorConstant{3}) +
TensorConstant{5})'
>>> cg_new = cg.replace(replacements)
>>> theano.pprint(
...     cg_new.outputs[0])  
'(((x * TensorConstant{2}) * TensorConstant{3}) +
TensorConstant{5})'

First two sums turned into multiplications

>>> float(function(cg_new.inputs, cg_new.outputs)(3.)[0])
23.0

scan_variables: Variables of Scan ops.

shared_variables

blocks.graph.apply_dropout(computation_graph, variables, drop_prob, rng=None, seed=None, custom_divisor=None)[source]¶

Apply dropout to specified variables in a graph.

Parameters:	computation_graph (instance of `ComputationGraph`) – The computation graph. variables (list of `TensorVariable`) – Variables to be dropped out. drop_prob (float) – Probability of dropping out. If you want to apply the dropout with different probabilities for different layers, call it several times. rng (`MRG_RandomStreams`) – Random number generator. seed (int) – Random seed to be used if rng was not specified. custom_divisor (float or None, optional) – Divide dropped variables by a given scalar value. If None, (default) dropped variables will be divided by (1 - drop_prob) which is equivalent to scaling by (1 - drop_prob) at test time as recommended in [DROPOUT].
Returns:	dropped_computation_graph – A new computation graph with dropout applied to the specified variables. In order to train with, or monitor, the outputs of the original computation graph with dropout applies, use the variables contained in dropped_computation_graph.outputs.
Return type:	instance of `ComputationGraph`

Notes

For more information, see [DROPOUT].

[DROPOUT]

(1, 2) Hinton et al. Improving neural networks by preventing co-adaptation of feature detectors, arXiv:1207.0580.

Examples

>>> import numpy
>>> from theano import tensor, function
>>> from blocks.bricks import MLP, Identity
>>> from blocks.filter import VariableFilter
>>> from blocks.initialization import Constant
>>> from blocks.roles import INPUT
>>> linear = MLP([Identity(), Identity()], [2, 10, 2],
...              weights_init=Constant(1), biases_init=Constant(2))
>>> x = tensor.matrix('x')
>>> y = linear.apply(x)
>>> cg = ComputationGraph(y)

We are going to drop out all the input variables

>>> inputs = VariableFilter(roles=[INPUT])(cg.variables)

Here we apply dropout with default setting to our computation graph

>>> cg_dropout = apply_dropout(cg, inputs, 0.5)

Dropped out variables have role DROPOUT and are tagged with replacement_of tag. Let’s filter these variables and check if they have the links to original ones.

>>> dropped_out = VariableFilter(roles=[DROPOUT])(cg_dropout.variables)
>>> inputs_referenced = [var.tag.replacement_of for var in dropped_out]
>>> set(inputs) == set(inputs_referenced)
True

Compiling theano functions to forward propagate in original and dropped out graphs

>>> fprop = function(cg.inputs, cg.outputs[0])
>>> fprop_dropout = function(cg_dropout.inputs, cg_dropout.outputs[0])

Initialize an MLP and apply these functions

>>> linear.initialize()
>>> fprop(numpy.ones((3, 2),
...       dtype=theano.config.floatX))  
array([[ 42.,  42.],
       [ 42.,  42.],
       [ 42.,  42.]]...
>>> fprop_dropout(numpy.ones((3, 2),
...               dtype=theano.config.floatX))  
array([[ 0.,  0.],
       [ 0.,  0.],
       [ 0.,  0.]]...

And after the second run answer is different

>>> fprop_dropout(numpy.ones((3, 2),
...               dtype=theano.config.floatX))  
array([[   0.,   52.],
       [ 100.,    0.],
       [   0.,    0.]]...

blocks.graph.apply_noise(computation_graph, variables, level, seed=None)[source]¶

Add Gaussian noise to certain variable of a computation graph.

Parameters:	computation_graph (instance of `ComputationGraph`) – The computation graph. variables (`TensorVariable`) – Variables to add noise to. level (float) – Noise level. seed (int, optional) – The seed with which `MRG_RandomStreams` is initialized, is set to 1 by default.

blocks.graph.collect_parameters(computation_graph, parameters)[source]¶

Replace parameters with a single shared variable.

This can be useful if you need to calculate the full Hessian of a computational graph. It replaces parameters with slices of a single large vectors like

>>> from blocks.utils import shared_floatx
>>> W1 = shared_floatx(numpy.random.rand(10, 10))
>>> W2 = shared_floatx(numpy.random.rand(10, 10))
>>> all_parameters = shared_floatx(numpy.concatenate(
...     [W1.get_value().flatten(), W2.get_value().flatten()]))
>>> W1 = all_parameters[:W1.size]
>>> W2 = all_parameters[W1.size:]

Parameters:	computation_graph (`ComputationGraph` instance) – The managed Theano graph in which to collect parameters. parameters (list of Theano shared variables) – The parameters whose values should be collected.
Returns:	A new Theano graph which has all the given parameters collected into a single large shared variable.
Return type:	ComputationGraph instance

Notes

Note that this replacement makes the training of the model significantly slower because of the large amount of Theano’s set_subtensor calls needed to train the model.

Examples

>>> from blocks.bricks import MLP, Logistic
>>> from blocks.bricks.cost import SquaredError
>>> from theano import tensor
>>> x = tensor.matrix()
>>> mlp = MLP(activations=[Logistic(), Logistic()],
...           dims=[784, 100, 784])
>>> cost = SquaredError().apply(x, mlp.apply(x))
>>> cg = ComputationGraph(cost)
>>> new_cg = collect_parameters(cg, cg.shared_variables)

The new graph only has a single shared variable. This variable receives the COLLECTOR role.

>>> new_cg.shared_variables
[collected_parameters]

The bricks’ variables have been replaced with reshaped segments of this single shared variable. These replacements are given the COLLECTED role.

>>> from blocks.filter import VariableFilter
>>> from blocks.roles import PARAMETER
>>> var_filter = VariableFilter(roles=[COLLECTED])
>>> var_filter(new_cg.variables)  
[Reshape{1}.0, Reshape{1}.0, Reshape{2}.0, Reshape{2}.0]