Logging¶
Log has two different backends configurable in .blocksrc
,
see Configuration.
Dictionary backend¶
-
class
blocks.log.log.
TrainingLog
[source]¶ Bases:
collections.defaultdict
,blocks.log.log.TrainingLogBase
Training log using a defaultdict as backend.
Notes
For analysis of the logs, it can be useful to convert the log to a Pandas data frame:
df = DataFrame.from_dict(log, orient='index')
-
class
blocks.log.log.
TrainingLogBase
(uuid=None)[source]¶ Bases:
object
Base class for training log.
A training log stores the training timeline, statistics and other auxiliary information. Training logs can use different backends e.g. in-memory Python objects or an SQLite database.
Information is stored similar to a nested dictionary, so use
log[time][key]
to read data. An entry without stored data will return an empty dictionary-like object that can be written to,log[time][key] = value
.Depending on the backend,
log[time] = {'key': 'value'}
could fail. Uselog[time].update({'key': 'value'})
for compatibility across backends.In addition to the set of records displaying training dynamics, a training log has a
status
attribute, which is a dictionary with data that is not bound to a particular time.Warning
Changes to mutable objects might not be reflected in the log, depending on the backend. So don’t use
log.status['key'].append(...)
, uselog.status['key'] = ...
instead.Parameters: uuid ( uuid.UUID
, optional) – The UUID of this log. For persistent log backends, passing the UUID will result in an old log being loaded. Otherwise a new, random UUID will be created.-
status
¶ dict – A dictionary with data representing the current state of training. By default it contains
iterations_done
,epochs_done
and_epoch_ends
(a list of time stamps when epochs ended).
-
current_row
¶
-
h_uuid
¶ Return a hexadecimal version of the UUID bytes.
This is necessary to store ids in an SQLite database.
-
last_epoch_row
¶
-
previous_row
¶
-
Sqlite backend¶
-
class
blocks.log.sqlite.
SQLiteEntry
(log, time)[source]¶ Bases:
_abcoll.MutableMapping
Store log entries in an SQLite database.
Each entry is a row with the columns uuid, time (iterations done), key and value. Note that SQLite only supports numeric values, strings, and bytes (e.g. the uuid column), all other objects will be pickled before being stored.
Entries are automatically retrieved from ancestral logs (i.e. logs that were resumed from).
-
class
blocks.log.sqlite.
SQLiteLog
(database=None, **kwargs)[source]¶ Bases:
blocks.log.log.TrainingLogBase
,_abcoll.Mapping
Training log using SQLite as a backend.
Parameters: - database (str, optional) – The database (file) to connect to. Can also be :memory:. See
sqlite3.connect()
for details. Uses config.sqlite_database by default. - **kwargs – Arguments to pass to
TrainingLogBase
-
conn
¶
- database (str, optional) – The database (file) to connect to. Can also be :memory:. See
-
blocks.log.sqlite.
adapt_ndarray
(obj)[source]¶ Convert NumPy scalars to floats before storing in SQLite.
This makes it easier to inspect the database, and speeds things up.
Parameters: obj (ndarray) – A NumPy array. Returns: If the array was a scalar, it returns a floating point number. Otherwise it binarizes the NumPy array using adapt_obj()
Return type: float or memoryview
-
blocks.log.sqlite.
adapt_obj
(obj)[source]¶ Binarize objects to be stored in an SQLite database.
Parameters: obj (object) – Any picklable object. Returns: blob – A buffer (Python 2) or memoryview (Python 3) of the pickled object that can be stored as a BLOB in an SQLite database. Return type: memoryview