{
"cells": [
{
"cell_type": "raw",
"metadata": {
"collapsed": true,
"pycharm": {
"name": "#%% raw\n"
},
"raw_mimetype": "text/restructuredtext"
},
"source": [
".. _example-evaluation2:\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Logging the training progress\n",
"=============================\n",
"\n",
"This example shows how to use the `pymia.evaluation` package to log the performance of a neural network during training.\n",
"The [TensorBoard](https://www.tensorflow.org/tensorboard) is commonly used to visualize the training in deep learning. We will log the Dice coefficient of predicted segmentations calculated against a reference ground truth to the TensorBoard to visualize the performance of a neural network during the training.\n",
"\n",
"This example uses PyTorch. At the end of it, you can find the required modifications for TensorFlow.\n",
"\n",
"\n",
"\n",
"Tip\n",
"\n",
"This example is available as Jupyter notebook at [./examples/evaluation/logging.ipynb](https://github.com/rundherum/pymia/blob/master/examples/evaluation/logging.ipynb) and Python scripts for PyTorch and TensorFlow at [./examples/evaluation/logging_torch.py](https://github.com/rundherum/pymia/blob/master/examples/evaluation/logging_torch.py) and [./examples/evaluation/logging_tensorflow.py](https://github.com/rundherum/pymia/blob/master/examples/evaluation/logging_tensorflow.py), respectively.\n",
"\n",
"
\n",
"\n",
"\n",
"\n",
"Note\n",
"\n",
"To be able to run this example:\n",
"\n",
"- Get the example data by executing [./examples/example-data/pull_example_data.py](https://github.com/rundherum/pymia/blob/master/examples/example-data/pull_example_data.py).\n",
"- Install torch (`pip install torch`).\n",
"- Install tensorboard (`pip install tensorboard`).\n",
"\n",
"Further, it might be good to be familiar with [Data extraction and assembling](examples.data.extraction_assembly.rst) and [Evaluation of results](examples.evaluation.basic.rst).\n",
"\n",
"
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Import the required modules."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"import os\n",
"\n",
"import numpy as np\n",
"import pymia.data.assembler as assm\n",
"import pymia.data.backends.pytorch as pymia_torch\n",
"import pymia.data.definition as defs\n",
"import pymia.data.extraction as extr\n",
"import pymia.data.transformation as tfm\n",
"import pymia.evaluation.metric as metric\n",
"import pymia.evaluation.evaluator as eval_\n",
"import pymia.evaluation.writer as writer\n",
"import torch\n",
"import torch.nn as nn\n",
"import torch.utils.data as torch_data\n",
"import torch.utils.tensorboard as tensorboard"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let us create a list with the metric to log, the Dice coefficient."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"metrics = [metric.DiceCoefficient()]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now, we need to define the labels we want to log during the training. In the provided example data, we have five labels for different brain structures. Here, we are only interested in three of them: white matter, grey matter, and the thalamus."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"labels = {1: 'WHITEMATTER',\n",
" 2: 'GREYMATTER',\n",
" 5: 'THALAMUS'\n",
" }"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Using the metrics and labels, we can initialize an evaluator."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"evaluator = eval_.SegmentationEvaluator(metrics, labels)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The evaluator will return results for all subjects in the dataset. However, we would like to log only statistics like the mean and the standard deviation of the metrics among all subjects. Therefore, we initialize a statistics aggregator."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"functions = {'MEAN': np.mean, 'STD': np.std}\n",
"statistics_aggregator = writer.StatisticsAggregator(functions=functions)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"PyTorch provides a module to log to the TensorBoard, which we will use."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"log_dir = '../example-data/log'\n",
"tb = tensorboard.SummaryWriter(os.path.join(log_dir, 'logging-example'))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We now initialize the data handling, please refer to the above mentioned example to understand what is going on."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"hdf_file = '../example-data/example-dataset.h5'\n",
"transform = tfm.Permute(permutation=(2, 0, 1), entries=(defs.KEY_IMAGES,))\n",
"dataset = extr.PymiaDatasource(hdf_file, extr.SliceIndexing(), extr.DataExtractor(categories=(defs.KEY_IMAGES,)), transform)\n",
"pytorch_dataset = pymia_torch.PytorchDatasetAdapter(dataset)\n",
"loader = torch_data.dataloader.DataLoader(pytorch_dataset, batch_size=100, shuffle=False)\n",
"\n",
"assembler = assm.SubjectAssembler(dataset)\n",
"direct_extractor = extr.ComposeExtractor([\n",
" extr.SubjectExtractor(), # extraction of the subject name for evaluation\n",
" extr.ImagePropertiesExtractor(), # extraction of image properties (origin, spacing, etc.) for evaluation in physical space\n",
" extr.DataExtractor(categories=(defs.KEY_LABELS,)) # extraction of \"labels\" entries for evaluation\n",
"])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's now define a dummy network, which will actually just return a random prediction."
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [
{
"data": {
"text/plain": ""
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"class DummyNetwork(nn.Module):\n",
"\n",
" def forward(self, x):\n",
" return torch.randint(0, 5, (x.size(0), 1, *x.size()[2:]))\n",
"\n",
"dummy_network = DummyNetwork()\n",
"torch.manual_seed(0) # set seed for reproducibility"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can now start the training loop. We will loop over the samples in our dataset, feed them to the \"neural network\", and assemble them to back to entire volumetric predictions. As soon as a prediction is fully assembled, it will be evaluated against its reference. We do this evaluation in the physical space, as the spacing might be important for metrics like the Hausdorff distance (distances in mm rather than voxels). At the end of each epoch, we can calculate the mean and standard deviation of the metrics among all subjects in the dataset, and log them to the TensorBoard.\n",
"Note that this example is just for illustration because usually you would want to log the performance on the validation set."
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 1/10\n",
"Epoch 2/10\n",
"Epoch 3/10\n",
"Epoch 4/10\n",
"Epoch 5/10\n",
"Epoch 6/10\n",
"Epoch 7/10\n",
"Epoch 8/10\n",
"Epoch 9/10\n",
"Epoch 10/10\n"
]
}
],
"source": [
"nb_batches = len(loader)\n",
"\n",
"epochs = 10\n",
"for epoch in range(epochs):\n",
" print(f'Epoch {epoch + 1}/{epochs}')\n",
" for i, batch in enumerate(loader):\n",
" # get the data from batch and predict\n",
" x, sample_indices = batch[defs.KEY_IMAGES], batch[defs.KEY_SAMPLE_INDEX]\n",
" prediction = dummy_network(x)\n",
"\n",
" # translate the prediction to numpy and back to (B)HWC (channel last)\n",
" numpy_prediction = prediction.numpy().transpose((0, 2, 3, 1))\n",
"\n",
" # add the batch prediction to the assembler\n",
" is_last = i == nb_batches - 1\n",
" assembler.add_batch(numpy_prediction, sample_indices.numpy(), is_last)\n",
"\n",
" # process the subjects/images that are fully assembled\n",
" for subject_index in assembler.subjects_ready:\n",
" subject_prediction = assembler.get_assembled_subject(subject_index)\n",
"\n",
" # extract the target and image properties via direct extract\n",
" direct_sample = dataset.direct_extract(direct_extractor, subject_index)\n",
" reference, image_properties = direct_sample[defs.KEY_LABELS], direct_sample[defs.KEY_PROPERTIES]\n",
"\n",
" # evaluate the prediction against the reference\n",
" evaluator.evaluate(subject_prediction[..., 0], reference[..., 0], direct_sample[defs.KEY_SUBJECT])\n",
"\n",
" # calculate mean and standard deviation of each metric\n",
" results = statistics_aggregator.calculate(evaluator.results)\n",
" # log to TensorBoard into category train\n",
" for result in results:\n",
" tb.add_scalar(f'train/{result.metric}-{result.id_}', result.value, epoch)\n",
"\n",
" # clear results such that the evaluator is ready for the next evaluation\n",
" evaluator.clear()"
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {
"name": "#%% md\n"
}
},
"source": [
"You can now start the TensorBoard and point the location to the log directory:\n",
"\n",
"```\n",
"tensorboard --logdir=/examples/example-data/log\n",
"```\n",
"\n",
"Open a browser and type [localhost:6006](https://localhost:6006) to see the logged training progress. It should look similar to the figure below (the data does not make a lot of sense as we create random predictions).\n",
"\n",
""
]
},
{
"cell_type": "markdown",
"source": [
"TensorFlow adaptions\n",
"--------------------\n",
"\n",
"For the presented logging to work with the TensorFlow framework, only minor modifications are required:\n",
"(1) Modifications of the imports, (2) framework-specific TensorBoard logging, and (3) framework-specific data handling."
],
"metadata": {
"collapsed": false
}
},
{
"cell_type": "markdown",
"source": [
"```python\n",
"# 1)\n",
"import tensorflow as tf\n",
"import pymia.data.backends.tensorflow as pymia_tf\n",
"\n",
"# 2)\n",
"tb = tf.summary.create_file_writer(os.path.join(log_dir, 'logging-example'))\n",
"\n",
"for result in results:\n",
" with tb.as_default():\n",
" tf.summary.scalar(f'train/{result.metric}-{result.id_}', result.value, epoch)\n",
"\n",
"# 3)\n",
"gen_fn = pymia_tf.get_tf_generator(dataset)\n",
"tf_dataset = tf.data.Dataset.from_generator(generator=gen_fn,\n",
" output_types={defs.KEY_IMAGES: tf.float32,\n",
" defs.KEY_SAMPLE_INDEX: tf.int64})\n",
"loader = tf_dataset.batch(100)\n",
"\n",
"class DummyNetwork(tf.keras.Model):\n",
"\n",
" def call(self, inputs):\n",
" return tf.random.uniform((*inputs.shape[:-1], 1), 0, 6, dtype=tf.int32)\n",
"\n",
"dummy_network = DummyNetwork()\n",
"tf.random.set_seed(0) # set seed for reproducibility\n",
"\n",
"# no permutation transform needed. Thus the lines\n",
"transform = tfm.Permute(permutation=(2, 0, 1), entries=(defs.KEY_IMAGES,))\n",
"numpy_prediction = prediction.numpy().transpose((0, 2, 3, 1))\n",
"# become\n",
"transform = None\n",
"numpy_prediction = prediction.numpy()\n",
"```"
],
"metadata": {
"collapsed": false
}
}
],
"metadata": {
"celltoolbar": "Raw Cell Format",
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.9"
}
},
"nbformat": 4,
"nbformat_minor": 1
}