Logging

Log has two different backends configurable in .blocksrc, see Configuration.

Dictionary backend

class blocks.log.log.TrainingLog

Bases: collections.defaultdict, blocks.log.log.TrainingLogBase

Training log using a defaultdict as backend.

Notes

For analysis of the logs, it can be useful to convert the log to a Pandas data frame:

df = DataFrame.from_dict(log, orient='index')

class blocks.log.log.TrainingLogBase(uuid=None)

Bases: object

Base class for training log.

A training log stores the training timeline, statistics and other auxiliary information. Training logs can use different backends e.g. in-memory Python objects or an SQLite database.

Information is stored similar to a nested dictionary, so use log[time][key] to read data. An entry without stored data will return an empty dictionary-like object that can be written to, log[time][key] = value.

Depending on the backend, log[time] = {'key': 'value'} could fail. Use log[time].update({'key': 'value'}) for compatibility across backends.

In addition to the set of records displaying training dynamics, a training log has a status attribute, which is a dictionary with data that is not bound to a particular time.

Warning

Changes to mutable objects might not be reflected in the log, depending on the backend. So don’t use log.status['key'].append(...), use log.status['key'] = ... instead.

Parameters:uuid (uuid.UUID, optional) – The UUID of this log. For persistent log backends, passing the UUID will result in an old log being loaded. Otherwise a new, random UUID will be created.

status

dict – A dictionary with data representing the current state of training. By default it contains iterations_done, epochs_done and _epoch_ends (a list of time stamps when epochs ended).

current_row

h_uuid

Return a hexadecimal version of the UUID bytes.

This is necessary to store ids in an SQLite database.

last_epoch_row

previous_row

resume()

Resume a log by setting a new random UUID.

Keeps a record of the old log that this is a continuation of. It copies the status of the old log into the new log.

Sqlite backend

class blocks.log.sqlite.SQLiteEntry(log, time)

Bases: _abcoll.MutableMapping

Store log entries in an SQLite database.

Each entry is a row with the columns uuid, time (iterations done), key and value. Note that SQLite only supports numeric values, strings, and bytes (e.g. the uuid column), all other objects will be pickled before being stored.

Entries are automatically retrieved from ancestral logs (i.e. logs that were resumed from).

class blocks.log.sqlite.SQLiteLog(database=None, **kwargs)

Bases: blocks.log.log.TrainingLogBase, _abcoll.Mapping

Training log using SQLite as a backend.

Parameters:

• database (str, optional) – The database (file) to connect to. Can also be :memory:. See sqlite3.connect() for details. Uses config.sqlite_database by default.
• **kwargs – Arguments to pass to TrainingLogBase

conn

class blocks.log.sqlite.SQLiteStatus(log)

Bases: _abcoll.MutableMapping

blocks.log.sqlite.adapt_ndarray(obj)

Convert NumPy scalars to floats before storing in SQLite.

This makes it easier to inspect the database, and speeds things up.

Parameters:obj (ndarray) – A NumPy array. Returns:If the array was a scalar, it returns a floating point number. Otherwise it binarizes the NumPy array using adapt_obj() Return type:float or memoryview

blocks.log.sqlite.adapt_obj(obj)

Binarize objects to be stored in an SQLite database.

Parameters:obj (object) – Any picklable object. Returns:blob – A buffer (Python 2) or memoryview (Python 3) of the pickled object that can be stored as a BLOB in an SQLite database. Return type:memoryview