Parameter initialization¶
-
class
blocks.initialization.
Constant
(constant)[source]¶ Bases:
blocks.initialization.NdarrayInitialization
Initialize parameters to a constant.
The constant may be a scalar or a
ndarray
of any shape that is broadcastable with the requested parameter arrays.Parameters: constant ( ndarray
) – The initialization value to use. Must be a scalar or an ndarray (or compatible object, such as a nested list) that has a shape that is broadcastable with any shape requested by initialize.-
generate
(rng, shape)[source]¶ Generate an initial set of parameters from a given distribution.
Parameters: - rng (
numpy.random.RandomState
) – - shape (tuple) – A shape tuple for the requested parameter array shape.
Returns: output – An ndarray with values drawn from the distribution specified by this object, of shape shape, with dtype
config.floatX
.Return type: - rng (
-
-
class
blocks.initialization.
Identity
(mult=1)[source]¶ Bases:
blocks.initialization.NdarrayInitialization
Initialize to the identity matrix.
Only works for 2D arrays. If the number of columns is not equal to the number of rows, the array will be truncated or padded with zeros.
Parameters: mult (float, optional) – Multiply the identity matrix with a scalar. Defaults to 1. -
generate
(rng, shape)[source]¶ Generate an initial set of parameters from a given distribution.
Parameters: - rng (
numpy.random.RandomState
) – - shape (tuple) – A shape tuple for the requested parameter array shape.
Returns: output – An ndarray with values drawn from the distribution specified by this object, of shape shape, with dtype
config.floatX
.Return type: - rng (
-
-
class
blocks.initialization.
IsotropicGaussian
(std=1, mean=0)[source]¶ Bases:
blocks.initialization.NdarrayInitialization
Initialize parameters from an isotropic Gaussian distribution.
Parameters: Notes
Be careful: the standard deviation goes first and the mean goes second!
-
generate
(rng, shape)[source]¶ Generate an initial set of parameters from a given distribution.
Parameters: - rng (
numpy.random.RandomState
) – - shape (tuple) – A shape tuple for the requested parameter array shape.
Returns: output – An ndarray with values drawn from the distribution specified by this object, of shape shape, with dtype
config.floatX
.Return type: - rng (
-
-
class
blocks.initialization.
NdarrayInitialization
[source]¶ Bases:
object
Base class specifying the interface for ndarray initialization.
-
generate
(rng, shape)[source]¶ Generate an initial set of parameters from a given distribution.
Parameters: - rng (
numpy.random.RandomState
) – - shape (tuple) – A shape tuple for the requested parameter array shape.
Returns: output – An ndarray with values drawn from the distribution specified by this object, of shape shape, with dtype
config.floatX
.Return type: - rng (
-
initialize
(var, rng, shape=None)[source]¶ Initialize a shared variable with generated parameters.
Parameters: - var (object) – A Theano shared variable whose value will be set with values
drawn from this
NdarrayInitialization
instance. - rng (
numpy.random.RandomState
) – - shape (tuple) – A shape tuple for the requested parameter array shape.
- var (object) – A Theano shared variable whose value will be set with values
drawn from this
-
-
class
blocks.initialization.
Orthogonal
(scale=1)[source]¶ Bases:
blocks.initialization.NdarrayInitialization
Initialize a random orthogonal matrix.
Only works for 2D arrays.
Parameters: scale (float, optional) – Multiply the resulting matrix with a scalar. Defaults to 1. For a discussion of the importance of scale for training time and generalization refer to [Saxe2013].
[Saxe2013] Saxe, A.M., McClelland, J.L., Ganguli, S., 2013., Exact solutions to the nonlinear dynamics of learning in deep linear neural networks, arXiv:1312.6120 [cond-mat, q-bio, stat]. -
generate
(rng, shape)[source]¶ Generate an initial set of parameters from a given distribution.
Parameters: - rng (
numpy.random.RandomState
) – - shape (tuple) – A shape tuple for the requested parameter array shape.
Returns: output – An ndarray with values drawn from the distribution specified by this object, of shape shape, with dtype
config.floatX
.Return type: - rng (
-
-
class
blocks.initialization.
Sparse
(num_init, weights_init, sparse_init=None)[source]¶ Bases:
blocks.initialization.NdarrayInitialization
Initialize only a fraction of the weights, row-wise.
Parameters: - num_init (int or float) – If int, this is the number of weights to initialize per row. If float, it’s the fraction of the weights per row to initialize.
- weights_init (
NdarrayInitialization
instance) – The initialization scheme to initialize the weights with. - sparse_init (
NdarrayInitialization
instance, optional) – What to set the non-initialized weights to (0. by default)
-
generate
(rng, shape)[source]¶ Generate an initial set of parameters from a given distribution.
Parameters: - rng (
numpy.random.RandomState
) – - shape (tuple) – A shape tuple for the requested parameter array shape.
Returns: output – An ndarray with values drawn from the distribution specified by this object, of shape shape, with dtype
config.floatX
.Return type: - rng (
-
class
blocks.initialization.
SparseND
(axis, **kwargs)[source]¶ Bases:
blocks.initialization.Sparse
Initialize only a fraction of the weights with configurable axes.
Parameters: axis (int or sequence) – Which axis or axes are to be treated as a “unit” for the purpose of the number of elements initialized. For example, an axis of (0, 1) when initializing a 4D tensor W will treat the first two axes of the weight tensor as a grid and initialize num_init elements of W[0, 0, :, :], another num_init elements of W[0, 1, :, :], and so on. Notes
See
Sparse
for documentation of other arguments.-
generate
(rng, shape)[source]¶ Generate an initial set of parameters from a given distribution.
Parameters: - rng (
numpy.random.RandomState
) – - shape (tuple) – A shape tuple for the requested parameter array shape.
Returns: output – An ndarray with values drawn from the distribution specified by this object, of shape shape, with dtype
config.floatX
.Return type: - rng (
-
-
class
blocks.initialization.
Uniform
(mean=0.0, width=None, std=None)[source]¶ Bases:
blocks.initialization.NdarrayInitialization
Initialize parameters from a uniform distribution.
Parameters: - mean (float, optional) – The mean of the uniform distribution (i.e. the center of mass for the density function); Defaults to 0.
- width (float, optional) – One way of specifying the range of the uniform distribution. The support will be [mean - width/2, mean + width/2]. Exactly one of width or std must be specified.
- std (float, optional) – An alternative method of specifying the range of the uniform distribution. Chooses the width of the uniform such that random variates will have a desired standard deviation. Exactly one of width or std must be specified.
-
generate
(rng, shape)[source]¶ Generate an initial set of parameters from a given distribution.
Parameters: - rng (
numpy.random.RandomState
) – - shape (tuple) – A shape tuple for the requested parameter array shape.
Returns: output – An ndarray with values drawn from the distribution specified by this object, of shape shape, with dtype
config.floatX
.Return type: - rng (