{ "cells": [ { "cell_type": "markdown", "id": "8c2da21a", "metadata": {}, "source": [ "# Guide 4: Research Projects with JAX\n", "\n", "**Filled notebook:** \n", "[![View filled on Github](https://img.shields.io/static/v1.svg?logo=github&label=Repo&message=View%20On%20Github&color=lightgrey)](https://github.com/phlippe/uvadlc_notebooks/blob/master/docs/tutorial_notebooks/guide4/Research_Projects_with_JAX.ipynb)\n", "[![Open filled In Collab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/phlippe/uvadlc_notebooks/blob/master/docs/tutorial_notebooks/guide4/Research_Projects_with_JAX.ipynb) \n", "**Author:** Phillip Lippe" ] }, { "cell_type": "markdown", "id": "095dacfc", "metadata": {}, "source": [ "This guide summarizes some tips, tricks and practices that are useful when working with JAX for a research project. In my opinion, one key aspect that JAX is missing compared to PyTorch is a framework like [PyTorch Lightning](https://pytorch-lightning.readthedocs.io/en/stable/) that can massively reduce code overhead while still being flexible enough for supporting almost any model/task. Although there exist such libraries for certain common tasks, like [trax](https://github.com/google/trax) or [scenic](https://github.com/google-research/scenic) (attention-based CV), I have not come across one so far which was sufficiently flexible for my research. Hence, in this guide, we build a simpler version of a PyTorch Lightning trainer, that summarizes all training, logging, etc. behavior that we need for almost any model, and allows training various models with much fewer lines than from scratch. Moreover, we implement some simple examples to showcase possible training structures, and underline its flexibility by performing automatic hyperparameter tuning with [Optuna](https://optuna.readthedocs.io/en/stable/index.html). Since this guide will be about code structures, it is more code-heavy than the other guides and can also be run in Google Colab if preferred.\n", "\n", "First, let's import some standard libraries. For this guide, we will use the data loading functionalities of PyTorch, but one could also use the [TensorFlow](https://www.tensorflow.org/api_docs/python/tf/data) dataset API. Additionally, we integrate loggers from PyTorch Lightning since they support a flexible API and have most popular logging application implemented (e.g. [TensorBoard](https://www.tensorflow.org/tensorboard), [Weights and Biases](https://wandb.ai/site))." ] }, { "cell_type": "code", "execution_count": 1, "id": "b66ed17f", "metadata": {}, "outputs": [], "source": [ "# Standard libraries\n", "import os\n", "import sys\n", "from typing import Any, Sequence, Optional, Tuple, Iterator, Dict, Callable, Union\n", "import json\n", "import time\n", "from tqdm.auto import tqdm\n", "import numpy as np\n", "from copy import copy\n", "from glob import glob\n", "from collections import defaultdict\n", "\n", "# JAX/Flax\n", "# If you run this code on Colab, remember to install flax and optax\n", "# !pip install --quiet --upgrade flax optax\n", "import jax\n", "import jax.numpy as jnp\n", "from jax import random\n", "from flax import linen as nn\n", "from flax.training import train_state, checkpoints\n", "import optax\n", "\n", "# PyTorch for data loading\n", "import torch\n", "import torch.utils.data as data\n", "\n", "# Logging with Tensorboard or Weights and Biases\n", "# If you run this code on Colab, remember to install pytorch_lightning\n", "# !pip install --quiet --upgrade pytorch_lightning\n", "from pytorch_lightning.loggers import TensorBoardLogger, WandbLogger" ] }, { "cell_type": "markdown", "id": "2199fd59", "metadata": {}, "source": [ "## Trainer module for JAX with Flax\n", "\n", "As seen in previous tutorials, [Flax](https://flax.readthedocs.io/en/latest/) gives us already some basic functionalities for training models. One part of it is the `TrainState`, which holds the model parameters and optimizers, and allows updating it. However, there might be more model aspects that we would like to add to the `TrainState`. For instance, if a model uses Batch Normalization as in [Tutorial 5](https://uvadlc-notebooks.readthedocs.io/en/latest/tutorial_notebooks/JAX/tutorial5/Inception_ResNet_DenseNet.html), we need to keep the batch statistics in order to evaluate the models on a test dataset. Furthermore, many models contain stochastic elements such as dropout or sampling in generative models (e.g. [Normalizing Flows](https://uvadlc-notebooks.readthedocs.io/en/latest/tutorial_notebooks/JAX/tutorial11/NF_image_modeling.html)). Thus, we extend the `TrainState` class from Flax to also include the batch statistics as `batch_stats` and a pseudo-random number generation `rng`. Note that if models do not require these elements, they can simply be `None` without breaking our code." ] }, { "cell_type": "code", "execution_count": 2, "id": "7963119b", "metadata": {}, "outputs": [], "source": [ "class TrainState(train_state.TrainState):\n", " # A simple extension of TrainState to also include batch statistics\n", " # If a model has no batch statistics, it is None\n", " batch_stats : Any = None\n", " # You can further extend the TrainState by any additional part here\n", " # For example, rng to keep for init, dropout, etc.\n", " rng : Any = None" ] }, { "cell_type": "markdown", "id": "cdb3e264", "metadata": {}, "source": [ "Now we already come to the main part of this guide: the Trainer module for JAX/Flax. The shown module here is not meant to be the 'one and only' way of doing it, and is more meant as showcasing one possible option of obtaining a Lightning-like API in JAX. The module can easily be extended by more functionalities, depending on what is needed/preferred by the individual users.\n", "\n", "First let's make a list of functionalities that we would want the Trainer module to include:\n", "\n", "* **Logging**: For basically all usecases and models, we want to log our hyperparameters, training/validation performance, and model checkpoints. For the second point, we can make use of PyTorch Lightning's logger classes like `TensorBoardLogger` and `WandbLogger`. For the model checkpoints, we use `flax.checkpoints`. In terms of flexibility, the trainer should support arbitrary sets of hyperparameters, since different models may require different hyperparameters. Similarly, it should be easy to add new metrics for logging, like accuracy for classification or intersection over union for segmentation.\n", " * Implemented in: `init_logger`, `save_model`, `load_model`, `save_metrics`\n", "* **Model state initialization**: In contrast to PyTorch, JAX separates the model itself from the learnable parameters. Creating a set of parameters for a model requires some boiler-template code, like creating a PRNG for the parameter generation and creating an initial `TrainState`. At the same time, we need to allow overwriting the `model.init` code, since different architectures will have different input arguments for the forward pass (e.g. models with dropout require a dropout-PRNG).\n", " * Implemented in: `init_model`, `run_model_init`, `print_tabulate`\n", "* **Optimizer initialization**: Following with the parameter initialization, we also need to create an optimizer and its eventual parameters (e.g. momentum and adaptive learning rate parameters in Adam). Since most models use a similar set of optimizers (SGD or Adam) and extra functionalities like gradient clipping and learning rate scheduling, we can write a template method that creates an optimizers based on some hyperparameters. However, it should be possible to overwrite this method if very specific optimizer settings/learning rate schedulers are needed. Since some schedulers require information about the overall number of training iterations, we create the optimizer right before starting the training.\n", " * Implemented in: `init_optimizer`\n", "* **Training loop**: Most models follow a similar training procedure where we train a model for several epoch on the training dataset, and evaluate it in between on the validation dataset. If a model is better than all previous models, we want to save its weight for loading them potentially later. Importantly, however, each model will have a very different training and validation step. Thus, similarly to PyTorch Lightning, we expect that an inheriting Trainer module has to define a training step function and evaluation step function, that can be jitted and used in the training loop. This is implemented in the function `train_model`, `train_epoch`, `eval_model`, `create_functions`, `create_jitted_functions`. Additional aspects to consider include:\n", " * Whether a model is better than the previous ones or not depends on the task at hand. For example, classification models are usually compared by their accuracy, trying to achieve the maximum value, while regression models aim for the lowest loss. Hence, we need a flexible API to support different ways of comparing models and finding the best one. Implemented in: `is_new_model_better`\n", " * Within the training loop, we might want to perform additional operations, like logging reconstruction examples of an autoencoder after every few epochs. To do so, PyTorch Lightning provides functions that are called at different stages during training, which we can similarly integrate in our Trainer module. Implemented in: `on_training_start`, `on_training_epoch_end`, `on_validation_epoch_end`\n", " * Depending on whether we run the model on a cluster with no display or on our local machine, we might want to see progress bars that track the training progress. Hence, the Trainer module should have to switch to enable or disable these progress bars. Implemented in: `tracker`\n", "* **Inference**: After we have finished training, we might want to load a model at a later time and perform inference experiments with it. For example, in [Tutorial 9](https://uvadlc-notebooks.readthedocs.io/en/latest/tutorial_notebooks/JAX/tutorial9/AE_CIFAR10.html), we use a trained autoencoder for an image search engine. To support this, two functionalities are needed: (1) loading a model from disk, including its hyperparameters (i.e. the function `load_from_checkpoint` in PyTorch Lightning), and (2) binding parameters to a model to reduce code overhead. Both parts can be implemented in our Trainer module.\n", " * Implemented in: `load_from_checkpoint`, `bind_model` \n", "\n", "With these requirements in mind, let's finally implement the module. Note that it is a considerably long code cell since we want to support many different settings. We recommend to take some time to go through the code and understand how all the elements are implemented, and how one can extend it depending on their own needs." ] }, { "cell_type": "code", "execution_count": 3, "id": "00113c86", "metadata": {}, "outputs": [], "source": [ "class TrainerModule:\n", "\n", " def __init__(self, \n", " model_class : nn.Module,\n", " model_hparams : Dict[str, Any],\n", " optimizer_hparams : Dict[str, Any],\n", " exmp_input : Any,\n", " seed : int = 42,\n", " logger_params : Dict[str, Any] = None,\n", " enable_progress_bar : bool = True,\n", " debug : bool = False,\n", " check_val_every_n_epoch : int = 1,\n", " **kwargs):\n", " \"\"\"\n", " A basic Trainer module summarizing most common training functionalities\n", " like logging, model initialization, training loop, etc.\n", " \n", " Atributes:\n", " model_class: The class of the model that should be trained.\n", " model_hparams: A dictionary of all hyperparameters of the model. Is\n", " used as input to the model when created.\n", " optimizer_hparams: A dictionary of all hyperparameters of the optimizer.\n", " Used during initialization of the optimizer.\n", " exmp_input: Input to the model for initialization and tabulate.\n", " seed: Seed to initialize PRNG.\n", " logger_params: A dictionary containing the specification of the logger.\n", " enable_progress_bar: If False, no progress bar is shown.\n", " debug: If True, no jitting is applied. Can be helpful for debugging.\n", " check_val_every_n_epoch: The frequency with which the model is evaluated\n", " on the validation set.\n", " \"\"\"\n", " super().__init__()\n", " self.model_class = model_class\n", " self.model_hparams = model_hparams\n", " self.optimizer_hparams = optimizer_hparams\n", " self.enable_progress_bar = enable_progress_bar\n", " self.debug = debug\n", " self.seed = seed\n", " self.check_val_every_n_epoch = check_val_every_n_epoch\n", " self.exmp_input = exmp_input\n", " # Set of hyperparameters to save\n", " self.config = {\n", " 'model_class': model_class.__name__,\n", " 'model_hparams': model_hparams,\n", " 'optimizer_hparams': optimizer_hparams,\n", " 'logger_params': logger_params,\n", " 'enable_progress_bar': self.enable_progress_bar,\n", " 'debug': self.debug,\n", " 'check_val_every_n_epoch': check_val_every_n_epoch,\n", " 'seed': self.seed\n", " }\n", " self.config.update(kwargs)\n", " # Create empty model. Note: no parameters yet\n", " self.model = self.model_class(**self.model_hparams)\n", " self.print_tabulate(exmp_input)\n", " # Init trainer parts\n", " self.init_logger(logger_params)\n", " self.create_jitted_functions()\n", " self.init_model(exmp_input)\n", "\n", " def init_logger(self, \n", " logger_params : Optional[Dict] = None):\n", " \"\"\"\n", " Initializes a logger and creates a logging directory.\n", " \n", " Args:\n", " logger_params: A dictionary containing the specification of the logger.\n", " \"\"\"\n", " if logger_params is None:\n", " logger_params = dict()\n", " # Determine logging directory\n", " log_dir = logger_params.get('log_dir', None)\n", " if not log_dir:\n", " base_log_dir = logger_params.get('base_log_dir', 'checkpoints/')\n", " # Prepare logging\n", " log_dir = os.path.join(base_log_dir, self.config[\"model_class\"])\n", " if 'logger_name' in logger_params:\n", " log_dir = os.path.join(log_dir, logger_params['logger_name'])\n", " version = None\n", " else:\n", " version = ''\n", " # Create logger object\n", " logger_type = logger_params.get('logger_type', 'TensorBoard').lower()\n", " if logger_type == 'tensorboard':\n", " self.logger = TensorBoardLogger(save_dir=log_dir, \n", " version=version,\n", " name='')\n", " elif logger_type == 'wandb':\n", " self.logger = WandbLogger(name=logger_params.get('project_name', None),\n", " save_dir=log_dir, \n", " version=version,\n", " config=self.config)\n", " else:\n", " assert False, f'Unknown logger type \\\"{logger_type}\\\"'\n", " # Save hyperparameters\n", " log_dir = self.logger.log_dir\n", " if not os.path.isfile(os.path.join(log_dir, 'hparams.json')):\n", " os.makedirs(os.path.join(log_dir, 'metrics/'), exist_ok=True)\n", " with open(os.path.join(log_dir, 'hparams.json'), 'w') as f:\n", " json.dump(self.config, f, indent=4)\n", " self.log_dir = log_dir\n", "\n", " def init_model(self, \n", " exmp_input : Any):\n", " \"\"\"\n", " Creates an initial training state with newly generated network parameters.\n", " \n", " Args:\n", " exmp_input: An input to the model with which the shapes are inferred.\n", " \"\"\"\n", " # Prepare PRNG and input\n", " model_rng = random.PRNGKey(self.seed)\n", " model_rng, init_rng = random.split(model_rng)\n", " exmp_input = [exmp_input] if not isinstance(exmp_input, (list, tuple)) else exmp_input\n", " # Run model initialization\n", " variables = self.run_model_init(exmp_input, init_rng)\n", " # Create default state. Optimizer is initialized later\n", " self.state = TrainState(step=0, \n", " apply_fn=self.model.apply,\n", " params=variables['params'],\n", " batch_stats=variables.get('batch_stats'),\n", " rng=model_rng,\n", " tx=None,\n", " opt_state=None)\n", "\n", " def run_model_init(self, \n", " exmp_input : Any, \n", " init_rng : Any) -> Dict:\n", " \"\"\"\n", " The model initialization call\n", " \n", " Args:\n", " exmp_input: An input to the model with which the shapes are inferred.\n", " init_rng: A jax.random.PRNGKey.\n", " \n", " Returns:\n", " The initialized variable dictionary.\n", " \"\"\"\n", " return self.model.init(init_rng, *exmp_input, train=True)\n", "\n", " def print_tabulate(self, \n", " exmp_input : Any):\n", " \"\"\"\n", " Prints a summary of the Module represented as table.\n", " \n", " Args:\n", " exmp_input: An input to the model with which the shapes are inferred.\n", " \"\"\"\n", " print(self.model.tabulate(random.PRNGKey(0), *exmp_input, train=True))\n", "\n", " def init_optimizer(self, \n", " num_epochs : int, \n", " num_steps_per_epoch : int):\n", " \"\"\"\n", " Initializes the optimizer and learning rate scheduler.\n", " \n", " Args:\n", " num_epochs: Number of epochs the model will be trained for.\n", " num_steps_per_epoch: Number of training steps per epoch.\n", " \"\"\"\n", " hparams = copy(self.optimizer_hparams)\n", "\n", " # Initialize optimizer\n", " optimizer_name = hparams.pop('optimizer', 'adamw')\n", " if optimizer_name.lower() == 'adam':\n", " opt_class = optax.adam\n", " elif optimizer_name.lower() == 'adamw':\n", " opt_class = optax.adamw\n", " elif optimizer_name.lower() == 'sgd':\n", " opt_class = optax.sgd\n", " else:\n", " assert False, f'Unknown optimizer \"{opt_class}\"'\n", " # Initialize learning rate scheduler\n", " # A cosine decay scheduler is used, but others are also possible\n", " lr = hparams.pop('lr', 1e-3)\n", " warmup = hparams.pop('warmup', 0)\n", " lr_schedule = optax.warmup_cosine_decay_schedule(\n", " init_value=0.0,\n", " peak_value=lr,\n", " warmup_steps=warmup,\n", " decay_steps=int(num_epochs * num_steps_per_epoch),\n", " end_value=0.01 * lr\n", " )\n", " # Clip gradients at max value, and evt. apply weight decay\n", " transf = [optax.clip_by_global_norm(hparams.pop('gradient_clip', 1.0))]\n", " if opt_class == optax.sgd and 'weight_decay' in hparams: # wd is integrated in adamw\n", " transf.append(optax.add_decayed_weights(hparams.pop('weight_decay', 0.0)))\n", " optimizer = optax.chain(\n", " *transf,\n", " opt_class(lr_schedule, **hparams)\n", " )\n", " # Initialize training state\n", " self.state = TrainState.create(apply_fn=self.state.apply_fn,\n", " params=self.state.params,\n", " batch_stats=self.state.batch_stats,\n", " tx=optimizer,\n", " rng=self.state.rng)\n", " \n", " def create_jitted_functions(self):\n", " \"\"\"\n", " Creates jitted versions of the training and evaluation functions.\n", " If self.debug is True, not jitting is applied.\n", " \"\"\"\n", " train_step, eval_step = self.create_functions()\n", " if self.debug: # Skip jitting \n", " print('Skipping jitting due to debug=True')\n", " self.train_step = train_step\n", " self.eval_step = eval_step\n", " else:\n", " self.train_step = jax.jit(train_step)\n", " self.eval_step = jax.jit(eval_step)\n", "\n", " def create_functions(self) -> Tuple[Callable[[TrainState, Any], Tuple[TrainState, Dict]],\n", " Callable[[TrainState, Any], Tuple[TrainState, Dict]]]:\n", " \"\"\"\n", " Creates and returns functions for the training and evaluation step. The\n", " functions take as input the training state and a batch from the train/\n", " val/test loader. Both functions are expected to return a dictionary of\n", " logging metrics, and the training function a new train state. This\n", " function needs to be overwritten by a subclass. The train_step and \n", " eval_step functions here are examples for the signature of the functions.\n", " \"\"\"\n", " def train_step(state : TrainState, \n", " batch : Any):\n", " metrics = {}\n", " return state, metrics\n", " def eval_step(state : TrainState, \n", " batch : Any):\n", " metrics = {}\n", " return metrics\n", " raise NotImplementedError\n", "\n", " def train_model(self, \n", " train_loader : Iterator, \n", " val_loader : Iterator, \n", " test_loader : Optional[Iterator] = None, \n", " num_epochs : int = 500) -> Dict[str, Any]:\n", " \"\"\"\n", " Starts a training loop for the given number of epochs.\n", " \n", " Args:\n", " train_loader: Data loader of the training set.\n", " val_loader: Data loader of the validation set.\n", " test_loader: If given, best model will be evaluated on the test set.\n", " num_epochs: Number of epochs for which to train the model.\n", " \n", " Returns:\n", " A dictionary of the train, validation and evt. test metrics for the\n", " best model on the validation set.\n", " \"\"\"\n", " # Create optimizer and the scheduler for the given number of epochs\n", " self.init_optimizer(num_epochs, len(train_loader))\n", " # Prepare training loop\n", " self.on_training_start()\n", " best_eval_metrics = None\n", " for epoch_idx in self.tracker(range(1, num_epochs+1), desc='Epochs'):\n", " train_metrics = self.train_epoch(train_loader)\n", " self.logger.log_metrics(train_metrics, step=epoch_idx)\n", " self.on_training_epoch_end(epoch_idx)\n", " # Validation every N epochs\n", " if epoch_idx % self.check_val_every_n_epoch == 0:\n", " eval_metrics = self.eval_model(val_loader, log_prefix='val/')\n", " self.on_validation_epoch_end(epoch_idx, eval_metrics, val_loader)\n", " self.logger.log_metrics(eval_metrics, step=epoch_idx)\n", " self.save_metrics(f'eval_epoch_{str(epoch_idx).zfill(3)}', eval_metrics)\n", " # Save best model\n", " if self.is_new_model_better(eval_metrics, best_eval_metrics):\n", " best_eval_metrics = eval_metrics\n", " best_eval_metrics.update(train_metrics)\n", " self.save_model(step=epoch_idx)\n", " self.save_metrics('best_eval', eval_metrics)\n", " # Test best model if possible\n", " if test_loader is not None:\n", " self.load_model()\n", " test_metrics = self.eval_model(test_loader, log_prefix='test/')\n", " self.logger.log_metrics(test_metrics, step=epoch_idx)\n", " self.save_metrics('test', test_metrics)\n", " best_eval_metrics.update(test_metrics)\n", " # Close logger\n", " self.logger.finalize('success')\n", " return best_eval_metrics\n", "\n", " def train_epoch(self, \n", " train_loader : Iterator) -> Dict[str, Any]:\n", " \"\"\"\n", " Trains a model for one epoch.\n", " \n", " Args:\n", " train_loader: Data loader of the training set.\n", " \n", " Returns:\n", " A dictionary of the average training metrics over all batches\n", " for logging.\n", " \"\"\"\n", " # Train model for one epoch, and log avg loss and accuracy\n", " metrics = defaultdict(float)\n", " num_train_steps = len(train_loader)\n", " start_time = time.time()\n", " for batch in self.tracker(train_loader, desc='Training', leave=False):\n", " self.state, step_metrics = self.train_step(self.state, batch)\n", " for key in step_metrics:\n", " metrics['train/' + key] += step_metrics[key] / num_train_steps\n", " metrics = {key: metrics[key].item() for key in metrics}\n", " metrics['epoch_time'] = time.time() - start_time\n", " return metrics\n", "\n", " def eval_model(self, \n", " data_loader : Iterator, \n", " log_prefix : Optional[str] = '') -> Dict[str, Any]:\n", " \"\"\"\n", " Evaluates the model on a dataset.\n", " \n", " Args:\n", " data_loader: Data loader of the dataset to evaluate on.\n", " log_prefix: Prefix to add to all metrics (e.g. 'val/' or 'test/')\n", " \n", " Returns:\n", " A dictionary of the evaluation metrics, averaged over data points\n", " in the dataset.\n", " \"\"\"\n", " # Test model on all images of a data loader and return avg loss\n", " metrics = defaultdict(float)\n", " num_elements = 0\n", " for batch in data_loader:\n", " step_metrics = self.eval_step(self.state, batch)\n", " batch_size = batch[0].shape[0] if isinstance(batch, (list, tuple)) else batch.shape[0]\n", " for key in step_metrics:\n", " metrics[key] += step_metrics[key] * batch_size\n", " num_elements += batch_size\n", " metrics = {(log_prefix + key): (metrics[key] / num_elements).item() for key in metrics}\n", " return metrics\n", "\n", " def is_new_model_better(self, \n", " new_metrics : Dict[str, Any], \n", " old_metrics : Dict[str, Any]) -> bool:\n", " \"\"\"\n", " Compares two sets of evaluation metrics to decide whether the\n", " new model is better than the previous ones or not.\n", " \n", " Args:\n", " new_metrics: A dictionary of the evaluation metrics of the new model.\n", " old_metrics: A dictionary of the evaluation metrics of the previously\n", " best model, i.e. the one to compare to.\n", " \n", " Returns:\n", " True if the new model is better than the old one, and False otherwise.\n", " \"\"\"\n", " if old_metrics is None:\n", " return True\n", " for key, is_larger in [('val/val_metric', False), ('val/acc', True), ('val/loss', False)]:\n", " if key in new_metrics:\n", " if is_larger:\n", " return new_metrics[key] > old_metrics[key]\n", " else:\n", " return new_metrics[key] < old_metrics[key]\n", " assert False, f'No known metrics to log on: {new_metrics}'\n", "\n", " def tracker(self, \n", " iterator : Iterator, \n", " **kwargs) -> Iterator:\n", " \"\"\"\n", " Wraps an iterator in a progress bar tracker (tqdm) if the progress bar \n", " is enabled.\n", " \n", " Args:\n", " iterator: Iterator to wrap in tqdm.\n", " kwargs: Additional arguments to tqdm.\n", " \n", " Returns:\n", " Wrapped iterator if progress bar is enabled, otherwise same iterator\n", " as input.\n", " \"\"\"\n", " if self.enable_progress_bar:\n", " return tqdm(iterator, **kwargs)\n", " else:\n", " return iterator\n", "\n", " def save_metrics(self, \n", " filename : str, \n", " metrics : Dict[str, Any]):\n", " \"\"\"\n", " Saves a dictionary of metrics to file. Can be used as a textual\n", " representation of the validation performance for checking in the terminal.\n", " \n", " Args:\n", " filename: Name of the metrics file without folders and postfix.\n", " metrics: A dictionary of metrics to save in the file.\n", " \"\"\"\n", " with open(os.path.join(self.log_dir, f'metrics/{filename}.json'), 'w') as f:\n", " json.dump(metrics, f, indent=4)\n", "\n", " def on_training_start(self):\n", " \"\"\"\n", " Method called before training is started. Can be used for additional\n", " initialization operations etc.\n", " \"\"\"\n", " pass\n", "\n", " def on_training_epoch_end(self, \n", " epoch_idx : int):\n", " \"\"\"\n", " Method called at the end of each training epoch. Can be used for additional\n", " logging or similar.\n", " \n", " Args:\n", " epoch_idx: Index of the training epoch that has finished.\n", " \"\"\"\n", " pass\n", "\n", " def on_validation_epoch_end(self, \n", " epoch_idx : int, \n", " eval_metrics : Dict[str, Any], \n", " val_loader : Iterator):\n", " \"\"\"\n", " Method called at the end of each validation epoch. Can be used for additional\n", " logging and evaluation.\n", " \n", " Args:\n", " epoch_idx: Index of the training epoch at which validation was performed.\n", " eval_metrics: A dictionary of the validation metrics. New metrics added to\n", " this dictionary will be logged as well.\n", " val_loader: Data loader of the validation set, to support additional \n", " evaluation.\n", " \"\"\"\n", " pass\n", "\n", " def save_model(self, \n", " step : int = 0):\n", " \"\"\"\n", " Saves current training state at certain training iteration. Only the model\n", " parameters and batch statistics are saved to reduce memory footprint. To\n", " support the training to be continued from a checkpoint, this method can be\n", " extended to include the optimizer state as well.\n", " \n", " Args:\n", " step: Index of the step to save the model at, e.g. epoch.\n", " \"\"\"\n", " checkpoints.save_checkpoint(ckpt_dir=self.log_dir,\n", " target={'params': self.state.params,\n", " 'batch_stats': self.state.batch_stats},\n", " step=step,\n", " overwrite=True)\n", "\n", " def load_model(self):\n", " \"\"\"\n", " Loads model parameters and batch statistics from the logging directory.\n", " \"\"\"\n", " state_dict = checkpoints.restore_checkpoint(ckpt_dir=self.log_dir, target=None)\n", " self.state = TrainState.create(apply_fn=self.model.apply,\n", " params=state_dict['params'],\n", " batch_stats=state_dict['batch_stats'],\n", " # Optimizer will be overwritten when training starts\n", " tx=self.state.tx if self.state.tx else optax.sgd(0.1),\n", " rng=self.state.rng\n", " )\n", " \n", " def bind_model(self):\n", " \"\"\"\n", " Returns a model with parameters bound to it. Enables an easier inference\n", " access.\n", " \n", " Returns:\n", " The model with parameters and evt. batch statistics bound to it.\n", " \"\"\"\n", " params = {'params': self.state.params}\n", " if self.state.batch_stats:\n", " params['batch_stats'] = self.state.batch_stats\n", " return self.model.bind(params)\n", "\n", " @classmethod\n", " def load_from_checkpoint(cls, \n", " checkpoint : str, \n", " exmp_input : Any) -> Any:\n", " \"\"\"\n", " Creates a Trainer object with same hyperparameters and loaded model from\n", " a checkpoint directory.\n", " \n", " Args:\n", " checkpoint: Folder in which the checkpoint and hyperparameter file is stored.\n", " exmp_input: An input to the model for shape inference.\n", " \n", " Returns:\n", " A Trainer object with model loaded from the checkpoint folder. \n", " \"\"\"\n", " hparams_file = os.path.join(checkpoint, 'hparams.json')\n", " assert os.path.isfile(hparams_file), 'Could not find hparams file'\n", " with open(hparams_file, 'r') as f:\n", " hparams = json.load(f)\n", " hparams.pop('model_class')\n", " hparams.update(hparams.pop('model_hparams'))\n", " if not hparams['logger_params']:\n", " hparams['logger_params'] = dict()\n", " hparams['logger_params']['log_dir'] = checkpoint\n", " trainer = cls(exmp_input=exmp_input,\n", " **hparams)\n", " trainer.load_model()\n", " return trainer" ] }, { "cell_type": "markdown", "id": "4745efac", "metadata": {}, "source": [ "### Utility functions\n", "\n", "Besides the Trainer module, we have seen other functionalities re-occurring several times in the tutorials. One of them is `numpy_collate`, which is needed for PyTorch's data loader to purely work with NumPy arrays. Similarly, creating Data Loaders for our datasets often follows the same structure, which we can also summarize in a function called `create_data_loaders`." ] }, { "cell_type": "code", "execution_count": 4, "id": "57eaa392", "metadata": {}, "outputs": [], "source": [ "def numpy_collate(batch):\n", " if isinstance(batch[0], np.ndarray):\n", " return np.stack(batch)\n", " elif isinstance(batch[0], (tuple,list)):\n", " transposed = zip(*batch)\n", " return [numpy_collate(samples) for samples in transposed]\n", " else:\n", " return np.array(batch)\n", "\n", "def create_data_loaders(*datasets : Sequence[data.Dataset], \n", " train : Union[bool, Sequence[bool]] = True, \n", " batch_size : int = 128, \n", " num_workers : int = 4, \n", " seed : int = 42):\n", " \"\"\"\n", " Creates data loaders used in JAX for a set of datasets.\n", " \n", " Args:\n", " datasets: Datasets for which data loaders are created.\n", " train: Sequence indicating which datasets are used for \n", " training and which not. If single bool, the same value\n", " is used for all datasets.\n", " batch_size: Batch size to use in the data loaders.\n", " num_workers: Number of workers for each dataset.\n", " seed: Seed to initialize the workers and shuffling with.\n", " \"\"\"\n", " loaders = []\n", " if not isinstance(train, (list, tuple)):\n", " train = [train for _ in datasets]\n", " for dataset, is_train in zip(datasets, train):\n", " loader = data.DataLoader(dataset,\n", " batch_size=batch_size,\n", " shuffle=is_train,\n", " drop_last=is_train,\n", " collate_fn=numpy_collate,\n", " num_workers=num_workers,\n", " persistent_workers=is_train,\n", " generator=torch.Generator().manual_seed(seed))\n", " loaders.append(loader)\n", " return loaders" ] }, { "cell_type": "markdown", "id": "d5f6bd64", "metadata": {}, "source": [ "## Example 1: Function regression\n", "\n", "Using the `TrainerModule` and our few utility functions, we can now write a full training scenario with logging etc. in a few lines. To showcase this, we first consider a very simple scenario: regressing a sine-wave with a neural network." ] }, { "cell_type": "markdown", "id": "b3558d13", "metadata": {}, "source": [ "### Dataset\n", "\n", "The first step is to create a dataset. Since we can use PyTorch's data package, this is straightforward. First, let's import needed plotting libraries for visualization and set the data and checkpoint path, similarly as in any other tutorial." ] }, { "cell_type": "code", "execution_count": 5, "id": "7e6bbdfe", "metadata": {}, "outputs": [], "source": [ "## Imports for plotting\n", "import matplotlib.pyplot as plt\n", "%matplotlib inline\n", "from IPython.display import set_matplotlib_formats\n", "set_matplotlib_formats('svg', 'pdf') # For export\n", "from matplotlib.colors import to_rgb\n", "import matplotlib\n", "matplotlib.rcParams['lines.linewidth'] = 2.0\n", "import seaborn as sns\n", "sns.reset_orig()\n", "sns.set()\n", "\n", "DATASET_PATH = '../data/'\n", "CHECKPOINT_PATH = '../saved_models/guide4/'" ] }, { "cell_type": "markdown", "id": "26cc39bf", "metadata": {}, "source": [ "The dataset is kept very simple and contains pairs of input-output of a sine function:" ] }, { "cell_type": "code", "execution_count": 6, "id": "e6ad10f1", "metadata": {}, "outputs": [], "source": [ "def target_function(x):\n", " return np.sin(x * 3.0)\n", "\n", "class RegressionDataset(data.Dataset):\n", " \n", " def __init__(self, num_points, seed):\n", " super().__init__()\n", " rng = np.random.default_rng(seed)\n", " self.x = rng.uniform(low=-2.0, high=2.0, size=num_points)\n", " self.y = target_function(self.x)\n", " \n", " def __len__(self):\n", " return self.x.shape[0]\n", " \n", " def __getitem__(self, idx):\n", " return self.x[idx:idx+1], self.y[idx:idx+1]" ] }, { "cell_type": "markdown", "id": "256736e3", "metadata": {}, "source": [ "We can create our needed data loaders with the utility function `create_data_loaders` and visualize the dataset for debugging:" ] }, { "cell_type": "code", "execution_count": 7, "id": "2079293e", "metadata": {}, "outputs": [ { "data": { "application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1BhZ2VzIDIgMCBSIC9UeXBlIC9DYXRhbG9nID4+CmVuZG9iago4IDAgb2JqCjw8IC9FeHRHU3RhdGUgNCAwIFIgL0ZvbnQgMyAwIFIgL1BhdHRlcm4gNSAwIFIKL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gL1NoYWRpbmcgNiAwIFIKL1hPYmplY3QgNyAwIFIgPj4KZW5kb2JqCjEwIDAgb2JqCjw8IC9Bbm5vdHMgWyBdIC9Db250ZW50cyA5IDAgUgovR3JvdXAgPDwgL0NTIC9EZXZpY2VSR0IgL1MgL1RyYW5zcGFyZW5jeSAvVHlwZSAvR3JvdXAgPj4KL01lZGlhQm94IFsgMCAwIDM4Ni41NTkzNzUgMjY2LjEzNjg3NSBdIC9QYXJlbnQgMiAwIFIgL1Jlc291cmNlcyA4IDAgUgovVHlwZSAvUGFnZSA+PgplbmRvYmoKOSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDExIDAgUiA+PgpzdHJlYW0KeJzNnc2ObcdxrOf9FHtoD3qz/n+GEmwTMHAHkgl4fEFTsgSKhkjLfv0bX1QfnT5Uli9lkoAsS2Av7t69Vq2qzMjMyMj8+P3LZ7/Ij99+99D/PNLj9/rvf+ufP+fnl6Sf/vBS13j2vuvs+vHr9z+WMZ65jqV//Fof/uTHf395+c1Leu48R5upr/X4/g9tp7xHmuvxLX/687/4wJ9/ePnep19eWvt4D885Suc+537WTy9+/clF/VJJvvru1z9e9B3/8fGXX11re65HyfPZ2uPbrx7/+vjm8dkvylm1f9Z/f6//nlX77B+++q/fffnVrz//5cuX3730/Zxz9j7f3eW7ax//9Mu/vPzq8ccPX5qeuettfPhe//j529WXP75kLdZr0r9qS2uU6tg1r/LQc7LyX/7h5ZdfPD77p/zI+fHFb17Gs5W6ysqF1/fFv738XXmmv3988fuXf/xCX5aeKfOF+s+f/0lfod9//cW3v/u/X7/+4Xff/Om7xz/8x8uv+M/jx63Py8f1eWh99nyuUvmzH9fn3bUfuz7aqLt3fdsadfzw9cnP/jeyPrn25xqrjfFugd5f/LErpId+ppTLXC3t/Ncs0d/KFsqz6hDUXd6v0MdrP3qBRn2mNmYtu7f+wxco/c3soZKzTsH3TOX7i/+rJfrk8Utqz5oqT/+2QGcJvEceWoKf+pG6vju1Vd8/0cdrP8EDtfKsbfXJS//eQ/Wf66HWeqaeymrvn+rdxZ/gsVZ61t1S1WFP+/1j5Z/tXVV9Q9p91v3eU7+7+OMfS97v2fRcCzPfPn2sn+tt6Q89c8t5v9+D7y/+BI/VZVZW203Pl9f7xyo/29v6+FV1PEfra+xPgdXHy/9bizqf5VHLc6Yl4CJw136Yu/lb8Tcfv2pkLUrbe366Qh8v/5gV6rqxPNuerawf5G7m34q/+fhVgl9ptFzGpwv08fKPWaBVnvoK3encc/wwh/y3t4Vy1osTtujt0yV6d/3HrFFO47lqn73lMcsPWqTyt7eNwLsNEF+/t0gfr//vjWyuz1Hy8tfJWzQ5kdLrmB8QzM9vaPPQfe3cdT+fPt7H6z/J48mbaD+lLNc/6idv++d9vNWeO+lPfs+RvLv+kzyeoM0eFdA/+/7kxP+sj1fSfo4kE7S+l4D4eP2neLyS+lNHuIw5AKTvTf7P+3j6CsGPtb9nw99d/0kerwqYLr28kbThP3H5f+3j1XMPa7z939QP/X0iqRYdhTW0TUg7+Taf3Td6+Z3HJ7/z689ffuAnv/1gjzHHCh8VUSiabFlGV3F3fubcS6o6ALivx2f/J2FJP3z+NSdFaKuDa6c+8drzc25Zu9J22cEvKIZ/zjZrm6XqTL0WrVdOvfMHZvQH2n6mWXruaya5CtJSqfXZxqorvKFVn33tmraeQA+09JPMb+J5g8+XsZ5Fd5xG3133r/eKu9HKrFVb9AdGfypAmbJ5q9ZH1/3vMpKAdIsWaOrPD93J2rnoabf21+gAyvBuXkvC2mnb8J1bqwVSb7nOkWsJfmEr7lqCFrqfkc/d571wDlmvL/oDwvt6vDG0jVndClxPG8wefT6P/Sz66qJFX+OxBGaIynqq4csSZsqptarVw9atZ629DEUGTkZ8/+Nau9a2XqyempsfOmhj9jnGwUp/+W6zXGknV9ZTFzx/7i6DqrB+R1uNN6XIWLFxWXq42Z/axtrzevTobrYsYdXarCn7/GgCJq3N1cbM4a1oX9aZ505+uKIbEwzmF6Jl1HnQv5+62a0n5j315yInuvQb4UbQPpcJ1Xbve1a/hlT32ltbJ9qVVaHl1HGfE1P+urWyNY+uJV3xSpbyXDrshYTaPMd4lt24UqMHlv0T7NACziEbwt/Tjqt8fIenpOhd6nXpEGV9fdOuLvpbo+EWgo/LRu5C3lzQ5iH7TRpLNnyvePHnlJtJeplTxvahI7xYSB37FW1LYfClaFfHTsdam1QWaLY6UhvhmdLi6OlKGrrhgs2Rj9yKcnvX243uvuIg5qpToZCOrN5USi0Lv0RLD17St8moKgzXVqgyN7vaPoUf160sELZuWZ/WL6emLxY6Ch916Nn0lMOAWY+atU4yz8KgMzwiGXufel0+VDYPWbZT9rPGt4Nx7TLIOuNYyyrErZfQZJ6i25G/xHh0bfs0HjrfuhN5Bln82HzItLbZdQi53bXlwOywVo43MeaorZx0BvdDe0zrItuphw13cOLb9aTyzqPzlnlt8tU5x7b1VRbjPGiXkXk0nRB9Gk+VoiOb8fO8HK2PzuAEN1ZhoV1iRzXlnzN2t4EWbBJk9HU7+p/w+wvgQ/BjyFpq7YcOmG5Plj7+fq1HTXnqZvU6H7nx56asqyBfbFvlSsA5wjatYODkGvTLhb0X3k+X56xDXyjHXB46Y/JT2saya+G7fWJ8tdKL3KdeRdV7k8kJv/sVmCDULXu2bJ9kjqfedeXeYgOinSbjp20Prm7AHb0FeYZoo8kYPDd3LRhUh94cIaaWSvhohV5TppQgpwi266RoKz13kyUWLoq3gh42yZ4J+pXsm9eiyieXFZ9CreTAvCdOinaG/BqYZPYQsVS5ppl0RNYa7eARGVlZq35ZminQIe+k1Z7b2ygpBBFoWCU84oWvL7P1IkTzKls4ytaOk70Mb6c8ZaZlbbQavGYBbu5Fbi623VtP11gLefk3+AdM7Zjm0LQ2OWxBCO1Dv7c6sa1ybvGL0j7ovbMNhTB5EcKiuhV5udBN6c3olGqfN1k/bTnd2WbXx1hIWHIKHgyBGi38fmoPyPzIK4bmA1vdK4eVdbfLxdoIucbwhpXWe5ftFhzTHtP504HUpRqaP339SkmovuCVXqeMq3Z0kzm5bIPN18uWAej19bobvlw4IXyvvJpB7nt03Bq/rj0k+6AHCoE9B3Strm9cj05SGXdP6jxadq/7HjrdBt0ynbJq+p0647UUqBdeBW2UTg5Ep30JCQnZhycK267bTuCNcRy6zIgO5NyXd8UR1QGXfdI+lFMc2gjyuLGXFThLgCUcVT+wVBhfkFuvI9w5pIASeCaToHrVxpjaNgIhMXAFOQu1CkI4Tth6eO1oIS85h9B+6JQoJCI0ldfW4nKkADg13DhT2HMKuYKJWHxiBv0ipjPElkA9bUqiENZSLl8LK0cd2z59QDGAdv4gaZJ1b3pynSgQVLgvFUDKSy6dDQVchEgEbIreZrzttfay6to5iW1PjLGAAPp8/PXjqchSB0O3rNvJ48mDs03jnamV53619nqVr+Rr9DRafsW4sbVscpuyf/KXQlyspSAuazkugEs3UBUK7DIFmbTfdAzlLJaOYvhu5YZ3gohBKexVNkTWr4BJdvS4Mu6KQrTSWd7YyQiBx9pkpcJ9ufV1sgIyAh1X0oGLBGz5Yi0FzFcmmgJ2s/aEhbJmLfz4VigNkFz6/vbA/jVFJ03vrYeHVhavafUUSGspsBG6eaEtmbg48zCoYunjhdwMYb/8lt5VuG3w4DpB2oWEx8Cbpx5j8JZT6Kcm354WvsRIV/tSJ1L2MI8wxNObla/BzXaw8KsiJHllVitOhByoTTyn9VgPfb3eseI9Tm24ODIaAluCUMI5ehz2cRfeJZMTWkwb5M7BWL47oQbhS2Fdbe+LiW1EnDq3BD0KIbuRZtmhq9VWEBpiPQh6dIQFeBRihdFy6UPoUA+rJ0xgy4Fvkc8fcUClDyheVMSziJAN7oSkh3x+TbH51sufBFVacqJlzqRigiL4GgeQOnZJG6dj92RFNk9Tq7Bmjl1z5huFo/V9GEGhUyFxzpnuKrTIYDgC0l7lDgVH01oY0BWGJq/GKUv+TZt6PhQZgBEW8XZ4sBY5Kr2chiXjJIDwFJonUNfFQciVrYH/EbTQx4XwZERraPD5uKwRGHQSOgA4gdM8ewgBWyct2uSAnNzY0OvkJKqePn698odbZgM3lTr8MFIFGzsan0XdsLbh5G2Vh8yK/MXUcpYwJ0mBRLCryBN5sxGk6phrV8QnF2wxfSyE0uWwhH8FTid2KzY8GDIQ8ZThPkGwtrbANGgktMk6q5lEUZNhJdLDtMioyzLHaTQZgyWDPDmtG/ev6KBsnff4qOf8XEuGJxHoY/QVqmgB9o4DwzUFXMn06PhhCQmGtLG3g9cY7TSZT+2dJrud2Z1CtFXYJUYjWo/c4QLofTYeV9hLyBSvHu224qod6eI8CE+eQ1FupaocP+wspDzXFn7qfL3+mELc7nx1bDd1kBaB+GY3yMFrbXXYhREvq1mFj5Jgsg7UNIAB4ctjy7eE4E7YccnzpLkJmOQ4ZEe09fcKP09Ge3cQ0fbnqQjJ7Otwhbj6FXqF8EHXX5fxIPVVSPEOveHL9nySLZcb2QvyDAdrzTjjItCkk9t1TEdOzonA1FTcSgYm/nLgi4xaGdWr7/w0uD9OvWnVFXXIdEyZz3ZyvMTxdQkEhbZHu+x51nrJPD3gyHF2tUPkYELjkEngCxFV4mXnBnX2Zdtai+5oNn1cq1dI3j6oFhDbyjm1S5JpsibFUSrYHSgvjMGmK6ExXKSBNqlIhTOsmByx45y1L2hfJ3Z3ISUdKNn7RmoVV4dRiRyvjD8gg2cs5A+nFlPGLt6drwXzp9dbZf7qSX2AmmR+UgoPDAG9AO8mIrd5oFokV2xoFgZ+gqPGo/o+vbs+9eZkrsP9JjSeFjlsYdjmoFU4QCcoX/xixtMR8es8bld/yIorsKixZ9R2E6CSdVKYjalNcFAbeXNt2RA5lCcxqwMEfl+uQ7c2FUyEZ53dLKwusDPBy50YmvRnCY+XUIBigSLTUIno9SYUHDfy0NeQe1HaIMeurbMTlkfgdq7YNDhLppOtvTVlWzOO3TnQHR4VyF6bILoq5HBGQsGNAJlQzApjOW1d/Ushi17JYHCWZRKF/y+Wn/qo/r1Cf2qgOlx6GySjKfdFthZar/aaHOFwBQjPXRKWv4dn18ChazHaJAP0qNrLcqFaUZ2e6PtlGzrFQB1z5zSFUxOluEsGVKGl9pmsRzqBMfnfMrZWZ0TPS9wvWKS9uHTCZUz5esx+6mFMoeUW5G1YTywVkbEwjO5vhjgGVzgn+Wws/avsAvnGAcU3Rg0JX6h3ozUnSQLsIDsODogeWDue5z1Bk3FbpSYk6zx6HBwLT8vaCFfLnRKEyFQrfCTgyffE5uJ9TgKDTspauDOvFR7GDJtMhlMOZhFfduGeRdkozINy+vZiv+s+KGHJOOgcL2q5YZSQyYBpaYZgUjF7Vc+ud6jdF64n7SM62nJftXLytU46+uCO2Ipvh7e6GU4rIcImybCEpMO3u52wq1R3dNixi0ugGctye7ltKdjSXpPf0c7TrcthyNRGi0NVWyihyYWs8SCW3fyyosvQyRkU6bYFMtKJXfW+9Gr1DWE4xNbaGS9SCIcGwbRiu7x6XC2Fh6FNLggrN6K7l9XNBAuKnuLVJCMvG66dk+rAye2hkyNcElelqJJ1YJ12r9+FEHEVICZrFdrxoa1AtMiKLLLMFN0HpZswtpddw4vIkG+FEDiNAV1NB7eEbqiRfuxp4hcUSwslgV52pvQVfF4fL1h5/c8AMujzVUulr8hhsYbML9n7usbJJhayoToKcWKC4FV2HsiZmjkdOvZLMVKLIcMQYtFiC7O7mExtuA+71BTWdvQqnwpetZZ6m84vUjeiVhUf3EoRcbDdSQliRbQPmkKYFReHFbCQ+ZDlgHEhOwWzRS66tRFbQjhbWso0KKcQjtZOLC2cEoKSTbZVRrzKODVnuShECwjfApwG67UQUC4CQOyodmrVCw9/objySDE5DSp3HQyNmds13v20PSjCUbg95LOpX2snDw5XfHwJpyn7kok7IQ8epgNiwohL1qND2dX7JwIhpJNRJwW741wbBx7XoxiWOAFDLfMiEyZXF23QSbqhL/2h5Bf2JA8CpyVGkKRThoEjEFM/CcuT/ohRgyD7pN9hbCHfkyx3dV53tcL947vXh7VPMxE+KVX5MPB4eNypRoxKeH8CZCHiSb4F4k4IY5rMmRyLCQwFN6fDpQNG1jyOp+USYQU56QqIkUOFvHEBPSbpsN8GAbp8KnlJrW2+5CfkpfGzZEK3do+rVgs+5QiNlW7Ajk6YCq/bKIYSv+aYMzA6p7HINm3oC0RH2kWK7y5pTvcRJoXgTaZTJ7MR3ZF9CnPAbs3E7uBPeJYnaSGZ0Tgx94olB0BOveBlfoLeEhg1XhwgcnL1VKgFgsSWo+DTM0wUap9nciPa97ob2iYcOcmWhIazQrtx7CDQSJVYbyF18j7xi8pQzAXo9BncaJ3eZx08Ft68dr7gso5VotPidZ8kJrXcGMG4iE79sx24lp5FWHizL6KNSVqw08cBkU/QXCi2aVl3ivMkC0IVJbZFlWyYjFHIToexxCtpNSy+VmM9dOcDMKAoOUTKilQGVdxaFGzxt2RwyDJoY94q9FkWSgshC+sMrVzkoug0r4FiwyXKwsIS1MNWbVH+WshU09KkTfq2VPPW/N1yMSFpDmDfZMlorxjmnZAPS9D4IvegPajPgBT2eEuRDBiGUKviMCvpEzJnXa+1kM1dW5a7U+AP3Zv8q76pwZWZ3mWyHCVTK44pgm5LzBQb5ZNl3RRDT2I+BSqh+dPSN8qTQsc6gjIfwpGDKDasUOnLJ73XRQfpoZuBPbo7KZP4vSYFYfoq6ub14ZJ6IdccRtA6clTwqcTmU93vhQp5TBAkUyPYsQR83xI9UygA/l9YisM2Ccw705ermVlTp1WW70IRdABXFBrJ8w8nEknv10GAFt58Ec5q7OC03lhu2jVzUr68GG59NJPtqMXB8IRwpx19yaz5+7UY9tpaKxlKiKwj9OGCTdpVCqydHMhmKzVtghbn6IupGgrV7MhIs5ltqfAm9LHQgGVYcDP4tCyPq0AolxGWrOVZOf+wB5fzOhPWPPTDcql4CHKUDhFF8JKKQJ3UgoSY4102qZY1UK9+49GG06AkgEMPLgAqf0OOFZctU5LYvt38migHStZBf5z/x30TNZROuT4ywk4xURSkTmd0o5CWVdwXYlWiH0SbntKgeZqKDEbGzod2m9LULtB0m8sjk9vXqxV2j7eZwKjeDazcQWGTHPGQK4KdGfPCtDj4MKropJC0I1x/icv5ejxSWDpROlKTEnGhctPDEH44JSDf3decJxJIEF7I9l7y1TJeMuubrTAfjfK5YhkKf7d0ILxWWSTZEdPIyoAhe+GxUBoQCpa9gCxf30htOl4hzO1ymAUqOkZB5kQYn3yQ/hMaynwINxsSkEN0bYC9QY4xys1YXgp8RALOleof9VpviXZiDGj4beIWMm1FcvZCiiFlGEM8BhVN9/rJm49BPNwvNHxH8I3CWnVMqwho0Lc04wMuiK7waWvxm0MSkjvD9dJL4pYqGq9e698Oac7HvYy44tt99+SqOrkxQTnZesjSPSzgCvqTihQuq4ebvrDlFegVx7QUOkgUktTQA8AVk49TUFPjopq+X15N4UL33ZuhRAQBFSNOQZDZJg9ujhHLS35EgX+Ja4ic2iT80yGZPpxRoMGONEF4/1QdXMhZh24Jn08hWQ1fL36WLgLtNts/RW4Esyl8VTwq1RVZ6rJOEpZ+saVYNUbRUBuGXA/QlT/VBaih7MQcFuDSdKxF1nKBkKmrhqBVXkcbJOGVyWzQrVCx+zOMk0k8aI8ITsC+xlTqSVZpMvsp/gWBG3IxBEPdBoEMkD8fLYx2sL4XcrbicPpN6VSS18/hFgZOTBpN+jbB5HTelA5sjSuZPKzAxD6SGfpU1d+KQ/BXavja3Y6KKSXITQnc6/AqkIpMnyypDj+EVmhMcrcge/LIYQgu07Sy3EGFkmFmpMBxgvEXJkCqcJm+WC6tnRrj5iGFEMJPwyhVqCoPBcyD3a+tozWPH7RRI5kQlYu5e3aHWnTohJemExlK0vocz267WqoMa11x+mzTPFDph9IBZdM0WTaFhbzX6A9o4QfcBoCZvr7QTQ/LvF3snk4qGFGOgb9vFjkbolyoNyQvSOpT4GXXmCdKqnGX2C/ALwbtT0he23wIfUHfdESFZ2q454+CjNORROQKReAursteE9yr4OJlp0+VqDa9cR3cMH2mJelUsGHe78PJkxmXQYGXGO02ggp5firA1QQEuX/awcaFUE0oaFagQEWDuy28ILgSvwCiP50KgCUBjOyN4qha4r4cmRiFlNotI8k4oWIzXP5sMQ5tmb1AvxtMGNLaitQ54jvcm80+ewgN47NN4QejbyrHF44a1BNaGBQjm+jQIVO2mI+nrSNbuVyf03MX/SrbePc4z4zkgnkoC9sKfdjoP6WwQkHaVxDXLupkmWX2ZZd1xi6cKpcru5uySEI24ljYgnpzMSAiF1SQbhhkIfXeqHuyK8M0s6KtJpsw2PZymDTG6KdxWCChXagUF/V9VNbfeLTUqg4CiBzbelKIL0ZCZo5W+7Xaw/oc8dpOEDlST0ZockdsjqZQKLYN/AE4f+7RAumTtNATzbjLAiKSobmA1iAPMSio6VKYHtJuI/D10TqldcFNGWWdqnA/pCfYR2GYjBuBezWYnZc4n4oN3qSB7E0yphxFUiSOMc62LNu9M+R+iCFKC+tDW8i8pEL6nPwvbnyQ7mzx1vRBqn1glZZTzCR45XtlkmOsTgmBFtim9+t8FZV+6tI5pDzJXcokmBLtnUOeQpBlECXGYBpeQj7x9HwQJVUo0PUS4pHCrbC/XS9zeY7kOEIvYdMYqEW3IhPvnl9cvzabguWwUO6WMdk+heq7HUoQqXHQX2wDzdrQ4tM5sN0Cr5CP5pUabmPdfUn0aE2TLEh/Q9XFQsc5YLD5GhQHYdiAo0bHgMblpwpFui9tCLwD3XiKSWAbXfyt7BQtIYWKkOlsslsyCZMyb7gXXI1JDuz09td0z+zkKy5cV232Cb+faiVAzU1U9ENE396227oXXdCNh9GnZYMunUidPIn7EbddIXQXCvxe/ziOcfEPBoddEKtJSyx5oJhBRhg7uwytjofCQjptqPZccgngym7fqiCeQuaEPlPjQpiAjcnynSr6KZyRcKY1o178W3+CC1bTTm8kYoehTOxuCyGyHmy1NywIFY/OJb2+OOetpSO1m2CzVVKvi4CA/GQOM4xW+yqDuv6hilJaLch11RWHeY2GbkFNCCuKsuku995esQXn6GpBEKoZBBwVu1joCwrbDyAkyYYL6Q63JLmdsxHSxHW8ymYUGqnFKYLGw2ifrpgzLySrXZK3sZ0bEGGfCAuXS8zspKK8MdQACxLNArlohckW3GETjjvk05NWl92lLjzjzeCYH2+1DHfoDFMQdAtR3IQHSXsAA+W9MOAgt4tdeOvpWYI8eOcCx0J7x4SOMHzbfliZ4WX/+dRblW3LNVz54vSAMG/Tt8nACjSkRVkuBBblSfegsyen3j9pZCdwDVeS3nYZePkD+Q+65Cle5xqnNiZ9AYp/9FbpiCHbSZMU3PdwHRH8SJ3uFm5Gm6BlKlywjEJXqK8fOqAKMGReaaU2iYNezGhDosgFPCfUIAKmAzVT6L+Rf4kF6QTFEWYLywllCDvEchZwomDVUEdy/yBvteoFX6gA7sGTJxcmGw8etdEVuuOCALmJAfW4wdqnT7yjU5FCHqkDNcwofe18GhIwFeyYIzEx7HQDkiAAUnQ3tCUyOGE6Ly1nB3TkNq3ZFLh0PNqFI/HqFDoMd3dXcxZJdnE8YlYun5iIUJTW6lsxo+s9dVelw+clU0xRCG43i9VBlpzAMPUqNFpBxnTOL3IA0GDp0L+kw+Q3pwIBBZK0M786vwdaqxffQB9YJ5/nnoYByTgBWOK+K4JMoURFjfgGmH4y2Yo5c40rs/Qzy6gq4kUwwrzWTcMyLUnhvmRX6t8pSp6YA2etVgvxWSFVSFK8QdE4Ib9O+64jt/gMIhsymiucGXLZckZCeGTGjIEMka/iKJcihdejioNwR9wx4VBB1mOh7rDcKN8dnI8R037lt7ULFl3LsgmFKHdRWY7r6Nm30yFz5F4Pnxq6Fc1XYX6A1W/QG7BqFfgrww0bf8ZhlVAI6Q9SMjCu6ZmjfaJe+kM4KtpbQiCOwiCcN2uJpJiLtjDADV0SSPfCOPSwVVDJrbFL4GDQMKG9snUQBb50bsJjrkNFhx6VfPRnKIfQ61zpSY4NIO0O7Hmq74d+3zCHu4ctgaSXcPIdwOh+SdY+wdWaMTdLwYi+jdUmuWcSQXZfMV3J4d4UwISZIg9hDi+FhAykioNmSLjyyrLc5YSVNIYo0Gvh15udsuDTU+qW0QEK6TtiMZ9XmFwkzzYc2Worgg2FPRNvTbZ6JbSbmVhGb8spEcRNYyIj5HtsstU+nDYtcj59XqosgKZCDoclfUAKxSLvWLGhwX/VsRLexrd1osgENyhOBDb3ARSST6eNB4UW2bcaG5FGnlBueVY6WV/RkeonZgthAqQ+xcsb7H/oJhN4U4wFY3tP4anwcib7utB9L/MZt/2i9QvFVOCQYwrpEY4SfQBhslyxo0seyAPvt35PhYXy7lqhGHS5taDTmgv+gxwwSH4J3F96/MgPUcnQK5qPQ77We467+ykgCRvI307ox6wPOViS+LFN4ySylPJaBIYkOapMFDyV0KoVSt7aiXC32zq9BE2bmq0WN1lmU5DRtdjkKUiS7OVm55jwQ4mN6lOGX81i4VFazEV3RZ8uAR1TGD9wE+RqISrE/CCST3hyq/g77uvUN8D3cc4Z8hYlBYhnywxGxTI0x67whgZENVnV7U4o/L/uXN5uzDDDZJobOhWIpXQSTklHB95YbBdkCBZaBnq57a2Dk5Z0ubl5wS9zOJmnAEL3jc+SEVLUX2NLRTFYYVQStlyHEV2e7ofV6YqRLOnPwuHepJjMsbUluXFUISboXrU/8fxeAJAtrV5xUgpRvkpkTYsrTrKgx0ArScypdFKEfmoBTFJAwkuZClBYWh9AFgH7ZY4CD69nJZcY5t8Et9C0k63Z25lbzrri8tZuLCQdFqpncNf0dI0+Hbo9LyVMvBriEFTrgAD6sVrP5VKcVrwDv2xB00v0L0+dXQiZK877UxFBPUpm32yYQl0SdML7jaImQtuy6QGgTsDhafiMFfftn6ZAfRvxw2MuBMvlnduM2cfomlCCouG5PyjOUJVUbBmWLOAgAcxydmJ1YJYZmRG781ckzkk3ZBQw9kk3KZwkDxACcfk42VcKqiBvovJB8yDSabeKbYWZb64PzL4FRaPE/Snwpit9gB3BKXBhoa8MZB0HzlqzsvTVpD0R2sbzu0oW+5NE35Mcrs4TUgsda9XCd2SOPTpgHcUiczFHMQlv3NoA89McYnkzEyDY8nj/FFO+7T86rWUKoo+uI+3g2tMzLMO7p38pfsBCmnXv4gayiiFccHGmwdik3IyUxKBPSFZlxkAKbcDkNIKwzUaDkaRkjlvAF/SU3R2TjFNbWrL+BNxxSRLgpV3l7y+H81gBdXr8O+8Y5S5W9LA2aTovCEqEATH+jR5TlCwnGZihwJy3fWk6RSAB1nOlkdJEXu3hRGE4RC+o5maTsk6aEdEmnYNlsYs4woIhxnbZR3OXYS+FNtH4aelQ0frTj4CKCsmFkUhDXwwapG+4TtrAsC46GpC0qIcB63QAltwsA6amOr2M+UPfXyCfQMsYzkoiYUvENG5qA1DuMs00Wm5S1EU4UC8thTttozcKUynBaqZXpioYcptnaCtBydokuZn2U8wrQgL3FmrTJYaoGW2mNj8ZolyrcQejBa0SWUVa6FA+evLJXGvcokeCeio4mocw752Qk+x865dXRUmYVmIKsfgegtdholGIwxdmDLWDQj83nTCdQooTmZHn0cbtKKA0s/lPc/8iHIr9LEISJLuthghHm5yXbi1ktGI0ZJPwensfJiP3PRPiCnc/i9qX0xwD4QPUb1IsQ8LO6m4yKuhgWIaiIQGonRNztJAITKYOuNqSkD8UjKorzvGSwsKBVEjq6GijyUHRM35TabmIWgE5zV0z0Bpp2AxNjkLPQvW4OIHYYOhYJjZEUCCsQkKsWMYNSETbK9Fw+GLZVpR/94dmAaRgNvYgrCNM4HqGhYEKQHu620RIMaTAdusuCh3qYBUnBukh1pV5kdhF21ORBUiV/LA82r71kiF+pd0+qvs4na8QTqz6pRWjm0TY29tC0I/sgw4KnUopjnlkakhzJblWOoLNdSMC7TWmzmenbRfdVzr/VAIRZZsgxTD3A3qqbtwD3cBIr/RxZeQvwl/oPp9W1Cx2awgHJtJHl0wa+no6fF6cdkSGaAkJ622IaWkDkmrpcLgXtF8apWLBS2EU8iQb7EmxKC3tyHUh2yf0mgQhCqUBDsfgIdGIi4+27TTNtrTw4b1h8qdYzFZmoEHyrRAY+TRaLop7W9wrArG2435N1OCQW41jx0qa+UDz7J530lUFqRhswi1PXk2xb85Lk7iXVQUXlXhhGjz7TBs5TfLO6dDrKLQYLsy0zl+mc0aBFtt90x2BctQtq+HUKGW1t6xAJXmGikccSpXiVsICyNVZJeD3douzl4nycuPVmsSIHBoaARAhw3iB0IYGLOrLpizVleB2xXJpuDu9S+x7bsbcUP4Fa8LoQnuwI5XLPijnYelLJUN2cQhPN1VmODAPfJW98k2luFhQUkbPIlN7Ur4stHiH1VO4YR0mO5V8wQTIUeDbMGQkbbph0BoOY9UIXRPCEjGMoOCAbknyJqdTjezWAiLHJH56S0ig4Y6pbC2L7tULGkYJAPZFQbcHUiLdhru2mLNrxUQBNjL6pPsszUv0KGsTm+1sMdqKqgrSDFhZ6xWGK2/V/7qttbnfonfhlUV+PZZgoWRS6U+VC6S/lXTPRgm7xXR1bTM3OWkD5Df2difhr6AkVglkF9NTpUfoTpVS2TWfP06dIXJIrW+S7eNlCCDLT/RYOZQYeTnb2tvbyx5MnZGXuMiwWB+Xxgh6wMApDKSgY/3CDqcGM2Ei2AwAXDqODRmXEIVabI4mDZCbeQbIWc24X5u6qF68fLYTkRa/YubGuGiXk5+gSIvuUzH3GHEahTNxiE9mVseQ4BqIvshTUztbt5z9gFgwEUzrEMpN0LYEbXw3AxX7TfFdFkobWSfYtPuwmN7Iehdk4C0vKfMnkNPGae2NMkkoD4DENkLxFN+RCHTwGDpwJDCsoy4sD5Ge3lOkhy5YCFYzGfHdrU5ixVYo7vGTJhrpC7M9Vjmie7KFGIUw5woPQYGUFhHHaSJxLyDGGcPhzltFnbatw/peENyyorHQsWnpaPqGb+EkHnLUtFXF+URUoTL9nTDt3BPZ0DTCb8UGZ3uQBdX5bUTBCdeDK74P/U7C6zca6i0MJJho+f4Rht/bVP9SYXFX4HAnz6CQ8BIzIkBPmQq6/YL4qs1c6eQID0g6LdBIsLnxBvlhzm64waiT0KwKjd9UR70lymF9tbjOo+1ODw2Ewe7+TXh382aKzYpdMDroLquHslHoaURuL7IFZu81WKvlbc+hbpTIVFzIgonTttxjgQY8nffWbQkNmd7jaTPEdD+o8sD71sPGUnwbvwNzj1kJ3M6yeDaeIs6Ko7nTFAFYNIZeWzrzZGpqHNPBVyObiRLNfhzdXzKnI6aQUH/slIXGhp0HvN+yBFWRUhy9bCOa5jT7chOZjIPBXcxddDeBRxjLljKUgWEZ9SZhCnij3XNlF4RwC1pRJpyEbkpQMXkQg5Ny9MrAUqCCEZeWs5tp3jYCatpaFe37i9qsXskT6h7KkNC9SNBBo0NGMaRGTrh7hSQ3OfrR0bChBDDCnYA8vjMdg74Jf1zQEYJL6GIxCIkM2USpTEeWIwAJM2abbLdoy8RDkNCyTpjxyLXE6TXoyZVCH7RFik3IjO6rwBw0wgqhAnVt08qX1lHhe5gMbaaps+52r5Z/oU5GS2qsdo+wObye5FQxcR7KQ6jv3JosvX9p5j8jsZAInih6hjkkJOJ1oEgkOdBTOIteYuhEEEukm9UH4hVWdqLKNOImZFSSmoxNRlcUMQhiKvqYYliGDCzjFmgz1CZAMZfChU5I6I3R09qEXkDLN3opcZpwaFyWIj2GOsUmsrNMSe2QyS6x2vCwH+oJjtaxxYgayvzEkSOdK6gBYz2mTqNl+NMol3Iv+hXy75XmvkJuoqCZiFRnmBCaMFGFDdA1gcAvwwconWF5AR4K4w4UpjmzhmkgceZWk9AFWpK9ohbYKqwX1KTIDlxaV6rZmDTnm9lL6zuZ+XkRVXzdtM6g2YTCl7c0SUcFZDmsni+GKBBjbCv6mLyWoQLfGm86hl5G2BDBbUGJMlLNcYJCd9Mhw1kbDCji5N2Fx0znaUJWjrYSRES0qA1xrUtL9Kb1nkoLjzqsKN7gB4Whiw9FscD5PLBIgY5uLV90qjtyFCwN+jzAj9N20MpV20kh8ULkm7ZD/K2csUnfl75isp6djZXAKxUvgiCjnuVWD0QulsJ8PyjxjDjZFwYotV3kAtvw4CV+ZD7PpPQeZvxIaDBmJZH/oIeSxvocOgV5DrNcJuK/4zDVtJ+RVd6XqSzNxAmGsJCSq7Ct4PvExU8CSwiih4LtdDqqXGNeOoyWw1atfyWMpYJn3XgaiC5lKTc503wCHx0FvlGMdWKuGqeIyUkNIG/mms97huwWnlr22qICZ+Eu0u+Zqg5kvtgeH+kC/sYErkzGGsuTXAcAoVEOjl9vxecC6N4AyJsJTETtif5Pp09RRaVJLL4hcnPuJC1Hm7jAhkPLZfXw5ArtEqMrpNtnnPZ2E2K4efTdeVFVy4ytwmC1Qbddv8h9IE66GAFGGHU2skcdkb6KAqRqC/ZGoZT9hpmukOPSy4Y9kyGoyGpPz0ZCr0u4Ne4R7cjxbHRQ2fdwfRAGoFktfrHgcvOupxupZe+Hf65xBRHgnPUiId+SVhxMobE+/booxC1ZEbqY6J8/+u/4XfR9whEYbp8fFK5I6xccO6oSPZbMqLIJzb2NcJPpf+dedtwGsSH2oKA1Tk6fqXjQmKDWhPDMQxXS9vBGomSZet3arSOdEgTjHYSw9iFE8qSj4i/C7AfE8YzYHBMBF3PlKhApxZknK9kxLdKJJ+tG7oFoQxxVQxfVaWDlPJglIVKxaEC59I4x0Iz+jeI5DWwEWAWF3HssXsQ0JGGiXZluROF8Envp0xcmDTTB4qZTBgGgjU5zzmwXnr3H4MEutdY63bTMDaBuFzrDhOAuHbMwSqgPNmyr7OhFbBpxN2gcsEQZcCivOTPV8DhJq7/PuLO+XG7MNClu2gQuU/N0qKen0HgIIfaScUCTcxLmb7CPu2QL7z888JBK774osyMrOAUoFzTE5dEmsLKrzkxMcUT2kpGAQoKemNHQT0H3LS49z0OQWRD95yF1QBsYyIqGpxAWFSyEcZKuVBUrUuiXfqFqb0LWJHug45yJ5d+hqpZTxtmUi4mkxttYZgLtuPIIBDUsa+jlHb4MQ8fQ9415CxtOjCJadE30B6Z3KqFQjgVa3IIKYCeZR5GIHI3ZtcGHpxuMdjkNCOTXmaUI5+uim0eujyEPbrjFwCGcRpUlZvfi92iyrB5Rhq/KE5ZYuNOoZDP6gtTfiQb0GAp8FDbFwlfEv7CFIEa8dVZx3kc8+gqlLtqctVuwsEgmTEoB6zJypxTkcBGvoSDgiWkFOcRdYjkoFPrJa6xGsGNB+ob8f40Jc9mKbYy/gl9FlLgdhI6Llmb2pyfttfWNd5k/KIXEJpD1KLC8F8UMc0yscXERcJcvLx59p7NXjBobavEoVIZVs+0mQjKScz8GadjKDJtyqXDuyTiQgp4EnD9EECBSaoViphTzI5BdBklZCgixmYUo5MWO8IFG8UyGpFn2BEINDdtxBwIhwHm5EF7WcItE4nBeOv1o0EURk/SPh9wqTO+QWsJ8yxErpLF+e06c9UkByTcBTpThyCqRpSCV7bQ8wzbidj8+Dj+KEElIECoZZpZBtHFKclmtjPEdMhwFMzQL8l9h6AmpZjlKNU8Qohh6ZQro4gxj56xMHHq1VabAWzwuKyaOeByhIpLp2XawBqxIyeDaMJ60LA4s9QHGpwUY7Q/Fw5dKHio5FYlx6LzOeZDD1AbKcU4HIrJQL45Buw0lYzoRTeGJDuNCKbpb1/mNBE37xEQpMWZUoDpRkVdbiqlweSjYsf/ClGrBZ21E3OsH6eXmidG0W0amZ6GLVxksIwTi6WC4PHx2vDp0VgEfu1OH1XKsizpEeLQqwtvJic7d/0zrM2EthtWmAheL440z7F4bk7poWB2A1YlEwdqlptNjghhr75c6Z/aIRKZDE3If2jzFgaEdErMfNmlYlBncAIoSNCUuz7cNNwNCvmb7LAtVMo+TLtlb+g1FsJHcqm8q0c50piIBEp5b6zUSnbrsI6/iV4vGfwyWJwoa6OpMTz1CSAO9xEshkkEdjT6gxQjGzU5yqmaEAIYYTMFm7qeTyaPgYNxj5OJkv5kbC45oP2ZT4Q3TIOP6HGoh9FpbydToazNS6Trbm7637alCgFO6h4TbxiWQsCAGJVy3Cjt+s7QLmZoLowwdLOFkFMRpd0RrdYLFY5S/mDNCVGIFCro1TPRol+Yhepi1GHL7yRzNhhxQr/FQZMi0m84KojY4LW+jbcdlUDA9/kwAKYzQJvNIRYRW2ZirK1xZ/KdpxXcNB7Xl1GLsQkkomQrMZF6DTrhKKdN4fLHHLTGl2Ok3qoFoHyHeftGqlQXzyI5kMVnoTjRBMa42rPrM0w6OpGl2Sr6Y/D9j/hyDRBELh0d2hGAI3Ki5xYKNpvBDqkGR2qC9uPe9x8N8kAkcJG3X26gs7Cv/fJk431DLIK8B84zfNmOCmkJoEKy8yVQBxtTxJGkaJ8e3nqYzjSSlnYqFpDLQ9EVMLw74sJbdL2c60WvfU2MD4nmvyZr52GIy3LaFfcXM506HkI5gdYRkcVwaspCEjOUsET4V8Bbsdn1/eoIgjPuLdOdEJYdGZkqvpN7mYs74Ch3h8lovBpMwtAupJLgBNc40Tpc2AICmE7nE9UZuv2BqOt8GovkMZKOVCkMWx6kuVnhOB9J/3RoIxsrtoiJAcGUh4UYvhocpYGtiQXUS7Jbg6m6kf+2eNsZU7fhmoJ2hgptKWflt/JOzOzV8Ta0IiloZc3tEENPkEITi3sIS2iA/w5DLtyHB7pRGvSd2sDJH5OlgRhRoVMnc234TYlqV90jXOwcKtFW26b1xYNXoEYWIhhC1g0R5y3URy2Iq44baSBbzceL3VNjCsTK9R5Avs18oPJDUky2IcyeWZp/0rHULrXi+MLmFffPezl4PRiygt04IoYMFSI+7hQpjzICJ5l/LKJczE2xctmQh6M1EvKQkyfIJN5EeudBCyNVnhDThBHF7ZJ91FF2UDm8Ioe5JjY2pm14uemOy91H4ahF5ZOOsdGRFGjKe5E+iBx6UEjCPjUndHtPbmdPtDGtknejLgOFFPyz9QiSiiqP6cOcsy8c2mgP6h1oqtYh1IUrw96lAMTWEmcSoVRWaYWOf7CRswzNQCeZ26MRq6L2GTnnTzsWYB4SCqdvCKIZvGAbw69nMuUA3GXLesBRMjitiTLyjajPoY3+4dfaYA9pbb3mo5F4rkumW79Ku1tGKQxi4Vehhoz+Wj/xiBUvRq33hldHnwaSz7TFjlQ6dStAR5/UUBKA1S+vDae5qnlpM6irWdwThgJyHx3pRi0C/elw62RhVq6CEjFI3oU7xj4LDejm39F7y3WyFckRaifMJyS4R1UjkVohZHh5eUZCu3WE1kkxUW6gXvMkQ03JIS/vul/uBBQM6HER425yczsS5NeJCPJtLcAjKehrupCaIoCxW47dFHR5CwhFMKAzjRIC5xuKg9JIOCrSK0azID18MtS0KuNED4/MXU4Kzu+RIzFNWimVFkANLgF060MkCMS2qxe3rPiTZpBk8FYn2yXiOmi/t+vOU4xazOiudoQz+AwjGDF9tSvAG5cRh6UVoIbPdhs0h/kg/AyWoTVoBRgxzxqNFJ1nbIdcMa2NjqxpUiBmyWjA2jEBjkEH1VO2B/BhzPS9D9dhk8m106p9sbWZkqKzrZbwurQ99vnXQQB5mNM2aMTnc9KDsrgC6tOCQumB/yb2ie6CDrICku0eLtn1Ux7BnYUqgA2uBYqm+OTq9NArm8Uw6+lrlOBR5NYuC4ReLJeAu5MuJX0N1bKOvzwal4AwF6ZZPgkmZaH099bbOHoYWEIc7aEgMwALtE0w2arzqELYizSsHKctNjwI7gXhDcEcxdpzQKEyjhWkzsd8OePZiahv6OlGgjCgrusyZ4cHcDweA7sabkA6zZ5itJNPLjLcNKW3Hk1Ebo3oTjDndSkV/ak+G98W63qjlTxl6FAicLQHaoRdyKS3Kz8DPs3DNPJQfkjceUhcOcWCiX8YYAUDgB8NQr/3C51PYOjw3mFpMY9YMOcy4XIxWv8BVrntadWdWkOZilvpFV8ZSrU3oHPR65tkhf1RnTOmDxgrRkn3l/gNGuiWy5vGAOdckNjLOCOZbnwD+cU9WYbzYhH0m1pjgVN74TqCcOOdNdS1xNBIlnE3b3ESgoMfkTuhQw8qDxO3TEs3dwXBcGF1WwGLmBnmJSe5Md0OgH64P+mR9WZpxHkpisvLFujTfIxtUqYuzSNVREIx8BryEyS0wDZpCrt2ZW8FWRXggZsLDUs4MqXaasxL3AHcZoRY/7nZVhRIk5bkO4UDRcyF5G+bDqFtPLIEWnNwb09ysyRWDb3RBGVBc8unfJM8Ldba22NU2MqcF0U7aIDi7CZhPf0eId5lcu2G7FlhdijZkc4WjGE8aH17awNL02LTilg0Uts2jurBr86hgtGT5A+YRkGRHrjeuTaNOQm/tBJZRaIbEWeLZNPTMbUb2OoB508BC14A2znj30K0/6ShMh5MmaInPiIOr7a5Nufs6LP6CMjXMTdQe4iy2B5fq5KZqTpoxpCDmjNUMWvPIhM5YZnffVzgWuqF4lgOZYGimkCaK715/LFHOi18udta8aYwZiRZBaZ3GElLMtmUjqI4Q+JBfRwJGiPdGEMEtMNp0emOS9GJRu+eRhncPkNJSM0qoHXBUScSuWLAMy5TJZaBGbd5Qon8AIZpLloCEJUN6KwQUSpkKa8aKCyywWiFiTyb/meQJV0EnMR6KAL5NHhlTj4SE9f0g013FcxlEtyFfWhNyMZ8ko50f4mOkfCtlZdelB/wGRD9m3E7vsZZ9DU9qzEcmhq7rxdYMoU6iOYkCBXudbA5N1AOOQsjHYDARfES2TfVgYKBJWKSVs9LaWAqFXntTB+lZbZeZIUdMfTh8p85GGIMeJuqsF12F7V9AQRayCryi5Uk/MU5zpQxx17mNwTdgZ0GwS3FvBPy45LkCDZ3jTYOH3NWttZ/B6VQ0NjRT31z3rh7xHBP5E45sJ8z3XGVm/GGSL+xphN6ZrUWP4nFACiCXuyXi2paA3cRqdEhonpooQK3HjWtVspNPenAIMqip09Ve3b0ddwURxrKPoWe76Qg1XBLVKYbJlmcZW+CsuYrQSQuSLqBnMzqHhT4aBgqtI86yLOYP3rmEzY3xSh1FqgpOhrjpvn8i+9DsCO8QxcCQP1nTyWzAdesFZpAlZrMZTmk7b3pTUM8Kv36bvGmy0H7bHwypl8mPZ2QcWgjp7k0EbNEmmgd6iteTEaF6/QMhg+rEKWkFkkKhv6WU0GAeofHZrNHiblFiifB526G4MDyrnKYyGM6MygrxKfJFw4yEYsUHsm/NFI54u7mxEJUW0iawngWGBR5Qcw3vZoFe2P6AU88rElYj7xK2YSDqTY4a3REfBYpdqcdgAW7TIm1c3TCPPLvl1VOP620eNUeqlvyr++1HIVUHbyxOiCS3KFn8glScM40MHIldhAXeofjB0CZLTFTDGIZYKpa7HyjXde5+eZdOU9JjqX20UqGaCE57cEBhgjljO2IpVIV8lmPRKSmHViybOFbdcdBHZtQFae1MdIaIU+DcwuAI3QSd1M29sc1wYVsYjfE8N6EnkC/ICIbTfDQTVum/C9ECbE+aXhTqedImp9BUrAv1Cn19RijJdisSepy50ExyDrPytElno8TqHDhmjdk/DbZNHM7DfEOO6A1Xo3WK2BkEnTBoVdxO7vB0TjG1kNpOjnsdi9w5HGi4QcdLINFhynUc/qMWQpaUo+imVwoziXMeB7nZDPbEGHEkDph0BPRCJS/UT1twGAlSySfQcQdJTuvZwszLSg756JwUfKK3eTD0h/ccY2Q4wdnE7IdllhDPZIpzuJT2URuVELQ3nMeaCIgwYSbUWrXyi16/edaezDOPhnFIQ8JBmU4OGGF4fPV86/hWzjSTvZ2KyMzJQ1a+x8RNeKwoKsEKcmAGiROZpDhQOipubSi0a55lpR/ym47aJW6W/WLXMATH7GvWPbW44Q7K76ZKIhNYznmlKtXibvkzenJi7xTmNTTfyhGLD/dXwnbIcpBch1i2XVu8aUFl0pJaGpLZSOMQF0D9vsxaoo8564vRdcyeaQJRuw90B8KzQeHUE4PzmR1HwI2+xW20BCNOSYDwuNNZ1s5BRx4x9Atwq6bZz24utN46VKde48QktVSBLtSiFnRresw2ihFukYswQoPdTmIUV2KpxszwWJQvI7fpiiG6S+uMbVH4Q0ct3coXXTaGAlErLs2vt8lpoUAemz7F5LxZxtAD0JgTIGuQWqyPsvAL1fq3w2KGaMNDnbrI0jypMpTuEQeQ5KgttXnJjsnJQpOiYdFG30Eu7KMe9rQzWaJS0ug4TY8vS7SyoIh7ids6DqckD67yhNbs2uJFnsNBKk9LjbnQPTih4CvmjhlFCyoIfakNbeNFaYN8FBKhcU4YukOGA3A60nR3mdFP4clCv+i0Ons42iZmG3GLKRwTIVrPXSPlVXpCMjImkFjoiHp4Suc8UevWXhyxIiyif3r3zMWgDoYAK81HN+E5hL1pj1oewvOG0tCDvvTSMhoyU6/P/H2nR9H9uEnIn+Gstupk8JhyjxbWvuAauhC1V52ypJs2WVYQpl08NDjTwCRYxg7BzNPYl6nQ5XiLmWPSNzDGolPwKDczPFe9jDdYHhsI/QVNWA4AmM8K0zFQeWZGhjC06vQGM6Gz5Ev8C0l30njh+VyIqlfoDBf+HBwypIx4gFMvptKsW6zXcQUKLLoHdze3n7sNRiHBJfh1fm3DInbwyyj6lMdNulrApGTPQiiCEnvihaz9EUu0MMKE4abZ5MPX3fD85J3jHnXOW0PkbTDb+8yvdFMkjOgQQtNCy8R0DpL9EJpMHQnI6ygKSv8bWd1Tz8Nsb/c/x07ZI62Q4FKM7M2ZPN1zxu0XNC42HAPS8G8UPKTSmXQYHt3ikdGMSDpixQjhNrowY8F/5A+oDSDrh61tKD85aoh2vm6Wl0vqvb1Nn0H1qMflRbz2skmyBv4Thhi8k1hS3ehncfPcCQq38CZGiuclJqSPM3nK5nEwFa2enS7zjiE9LPfZeXgMiUgSICtOWxYXca0W3/opo6YF3aTE7EIMGKUdBSLyCPQG7UW2JNxitDyjJs0o6n2CHtga7TYMBkEImknrtthXgfSwE/SiSy5D5mkywg3kvxlsj44XDRJh3rLQPoRQbkOS+2GhfdScaZGM2TjbHFf0dKBsLIpndLbVfNGILmgwMXF0k3alEU7+e0FUuByQN9qoRw171L18BcgunkEhDDxoULImH1uS7QxaibEKKXL2GILPmJtCIwX8mXBx0KxDZ4V6NaZ+kLpZDHrucQphMXec1CJby7LFdGwy0jYcCklPNeH2YnwuyIiloovvMqGN8kLrDGM8B5C8reJBAfzQU9EwyvDoSv+N0yXUrXKHwxj6BtpbFRjB/10Q+FG/ianG1lPdlh2Bc8asObOU8w6D70qG1o0wR2ONt4SO6KXVgg471KkFK+kPFCZD9oNOzsuQW3qA6hmk6z0EJYR932IxnddCqskNe0cOlt58D8CEoxfmehgjxiA5Zjq/5UqKlsrCG/FfoFxjbXYSAjRyIY9XqdNdtBZkV+X50tHjRdUGYXgLXYRJVKfWyNFaQ7sTHE4FYDq5ccyWZOtlUpGXp/GveVKl2y5DT+JOPHgghG2HaAaTfOkJYiId03IERmV2hlNvehTP87gw7bGT2vy7M8ILKWQG89CGEGI0pvUq0qEjBnI4RCqd2XQpKjYmwRSabcebUgQbG4mPuDFJ9hvh5rxsQ7w5mVp2kZXQrSM1wAgmHVs6EPViubtL/II6kTDX9tzaZUUQWbnLfGdUqehrYbrCPrng6uxGfOtyq9oDE3pQJyNXjphQviBLZknNI2zsYZ9M/UaZ/FK3Yc7HZBR3LW8DMOnBgUd/SeF1J3DNYxmHvdidQOvxw2LPQNGNLCFJKHA6rSNx2oFYDglnmQeifG2xiioymqWx12fZK6zbMzQDlh6Jp7B3j2RQlsXXu6mmzJv8bx58tDYInTdGT8k+mT/AbN8Rh4w0R2vzaUOtM4s4y+dr3d2gFMJQFpoa3GjpDYYuczrLReGHltuFZrxlVGDvlxFTuJgf7r4mgQiPUkR1VkdQHuSmouIJaL0zPKiAzLQuFFpjmRZL7AouMfSULBUT0AeJJDK/saHktcM8b2eyUibe0bYx4o3e03p6plOzbjPBLGacOsClWonuW8HNF2No92qM4WlIMR8E1tcAp74ZVoT4l7xhTK8hW4pgKLMUt7lQpDdg48QyNk+GIrEvZdepJUK+INa86fWDJTsx2DrdVJ3jDkH4EmyaYkghq+Xl9ghaJ/qKR9G43ZaZSguTwUxW57hLj2cHIdKC0qxW28DbgzwZX1zypdMlQ8Yh2txuMnKx0h2vsVAmfQXoxWg3ttOzqLdQeH2Xr0+IMjNIfPE26Wan6F4vboccqqwA7FH3AW2YazhRO+koW8Vxwol7WF85YlL8RmyNmYhCsaxQkILWlRYKQZdAAza5WQLZYnG0ntNKRlPkTTKQgkQjq3xa6xlyurFEMeMRogA5/eEiAJBhUtSQx7/hLqL9THM5ipGvyULJdP/Hq0+rnwwDE/3qmYsihIRuTpyqQDiGGgQx+xs7VMeyV5QlQvZrdQiJPrc2P02OSNJ4ovIls0j3LNL4nudL3Z5J1jPulYelB+eikjLzmCEyCdqlIcste8434zXZmC5VMoeiMRcmXkq2ljZZdxyDyLS2HaOYQ0RKjouy7HSy5WGxF8GAYZXFeOcDDSAS09qwiZ+ZJJhkR+MekGwFDWH2BjtcCAr5tEo332W0hTD7sqQsrpyQWPGwoMAK7YjMNcFLbyWR9yWvAwMP9HgrIQwsG3nT7m7wSpaxMBHrk6P1q5dfPf5oVlaHsnACPjkuuYzn0rJpXdvj268e//r45vHZL8rjt9/p2/9Z//39oz7++5Eg8+mMKAKdL0gdTExFqdBKPG9mGM7++vPHZ//w1X/97suvfv35Lx9ffvfy0hluyMQwt5zqGJAL/8OL/nmaImUAjsKdPvH1i14vYHcPs6K0RyiJfv1Cq1xFQtZD7omBU+UyLbGumhZo+H7JXOadAOYpr9CrU/0dPKFs0HlYVqFzVSiV3rVpqU8yUJlvJv5M5oEWs0RpxuBypSOKUIp4H1uW+JJFqG19YTSjaPpf/hIEt2FGy4Lo+wpJaC5P5IfY/Qp8nojOZ3/39sFg1mileRdioS7vRPhQGQNDI3xPGGRd9vSANA4rCEbH4moxo6m2Iz8u49w3lzHWCaY8l2Vmyvm0h1en5W+2tq2/eRyWYvVlZA4mb4aUota0TN91dd+frjKohtK1rjLmePbBTZO/G6SyvSBks/0dlp8p5CW9fjQw7uLrPBkY0FpItAn4tYM48G+T63y9deC5jrZ4ks3wu6T1tGZfZ74uIil+9RmCur8foGgFJ28UZKi6bxMOtZO2Tl7PiXiLrwMumLbg6e+M+ay+f3j9iBZmJ2xkYeB5cZ24HD1DD4GTl67nMoyfSXMeZSZtM5iwut4YFGFWtWubq/VzuRNt04oBFQWE3n15IeSFsjB8n2ptSF3uDEKEF4BNRBamn8soBiSyfVDtsueRcBmjzzRe2tCRjhp+UCsRUruncobkUfan0bnNDBUjoBywfn0n+kaLnG/siIARAS+XIWu6Y54Hk+Oq53J/WuwSLFLhvPoVkfpB1N8EXrkU38YgxNjMq0E5UhHbeT3yg7RtExxO3ubuXtZJ0pLWTaY0QQb2PVuipqMLWh2PYb65DOrWCRtcNuDzWsPYrAVROgdnBWlkLpMWzqhq4TKte+TLw22Fy9oclZN+LiOSYoF2T6rJ3eYnWzrII2w8fd4pc10mxceU1mKmlsKt4mWiMwWKYGNRsaRnjyxS8rBhrdsrmHRWCpvCfO/KC5PZ3OcVIO/SLH+NHBeS4VzdybSg8fDkSPqKfZWwewLCvVdJC/oykRqStnSnUAydfuf04ms36c/Q59E9svfrF7jt+lWKtqg7k+jMvjxQOK+Mk6a07TksXEeKNdMjbywiYJf98NSbmTzU3ppsULnnMUlEKTBpLpsuN8s3fx6vVZkW61IdCXq/CazMm6HgIOv9oDav67TxMSjjHPyaUMzxdchbnlTvEjzdKOfzw85a78hOQid2V18njSf4fLxE94QIXW5nzny1l0jWlfDlQp+4zqO9BJ01/qO0MR/rYTeRPPGJ69CvOsrk9hMJdWlfZ9TTsiPDUQwK276ulz0YjH08BVDiXMeXKd44iJVkiS1Fse1pNLwfXwGu8/VCLg6YdLyF4tHzeSTRaAo43kJA0F+PJiCQ+jgohH+8ywoCnNAFzu0gnJ79uNqJx3LYYQietPNXYdQgQ3tcBpvrXPd8CbrF7DOoMvpurDBALtirqYU+b8TDRztz++wyECz3DsTqFAQ8jstgTJkvU3z3+BZeLCxVW24m7hJkr+KNgKe1qWMwrT5Yh/cNFd/iNzs9MdDd/ZsQyVOvuD5Avcjx2GEoLrV7LEwD08lDYQP2MIVmf56IZZNas8OAIHTunoHEx5TaY8AE5nZAhQybk32x5Kbc9vbl/sGkLLdEVm+ESk5ZUEh+Z7ozf2V/GmF1poDjMFCw8fvQ+jwZzbNsJdIs3quVlBPsxGZ/Md9MW+VQFGYxmWnBiInsyxattkQCdYa0vGMUvclqI5Ntf4GAor+keDAtgg2eQSIT7huxQDhzfuwwqgUluDw+WBocxqYQ5svzA3CrtDmi8+zLCDyQucVjKFSufnLm1x2zhMsQrGjTl/MHOIfLgKjqr64wEzyvoh5m0PI7cJKtQcXFZWRLI3CZwJmyt12G35IvM8AjkwuiUYVI+nx6nhkyxS5jEFj5ssXeyL9hUuiN4yrrMKEnn14geot9OX+Ac/T8mI3iyzBGSAHaY5Cd819s/QNwAzEwb8J/8Th219GJhttBUJV4rMCJZzewj70vKyEh8v3bnDY4Zf5u2uOso4bPQPfxbIe+LDygP4muGVkKTnLlG11UwmdMOJX+Egw/XWe0iTIpAFGz/+H6y784knmLUQoxiu5NIUp+fD/4+Mto5w/vrzUTjvv5Tr04ejg+/fD7i598Ovrm4Bu+/9kP3/E/fHPSAyEM8vitY6/s2Es/fv529eWXXzw++ydSOo8vfmMAtz98jyyGv+eLf3v5u19/9dtvv/ruu9/9xzeP3/zpmy//U//w948vfv/4xy9ePvtFY+WSUcCJ4/7ih/YhpHt8+1vFmb/XteVb+Pixl//pd17squHoAn4zrVrzPDZEBt/un6/qVQPSmOVYPl7l2vr4SYpi5PneXS30d5FPeXl/1dTaP3/rn6/i6jyv7v3fIqfa/bc+3tWHa19+8gQfrgI1ICeegvaHqxZw/fjJt7/0ydW3u/ry5f3VD0/w9SdXPz7t+7/1YV2idf3y5d9fXn758v8J2x9/RdhOz39/+yvNUJm3xw77/tW/3LS3DZvZsLj5D7ffktX/vV8///Y//vTNvz2++PZP//nvj3/6ix1bz46VsTv/x8P097uvImO2LEesHSto7Nv4+PmX++d//fkP/+YXQ8QPa8DrOfvakO/tak3n6teffPbD1U8/++Ebvn73/v76hSyQ47yMX3z7f3/3ze+++e3ju6/+88Pq/erl/wHPovTLCmVuZHN0cmVhbQplbmRvYmoKMTEgMCBvYmoKMTk1MjcKZW5kb2JqCjE3IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggNzIgPj4Kc3RyZWFtCnicMzM0VDBQsDACEqZmpgrmhmYKKYZcYH4uiAIJ5HDBpCAsM2NDIMvQ3ByJZWJoAJI1NEZiGRuZQmURLJAc2PQcrjQAuQIXwwplbmRzdHJlYW0KZW5kb2JqCjE4IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMzQwID4+CnN0cmVhbQp4nEWSy40cMQxE7x0FExhA/EvxjLGn2fyvftQANtANUvxUFSl1b1mSLi9VaU1p2/JHn1AT2yq/1/NK+TyT/u/pWnhVR8r4VDwkW0VL3k9yfpkE5wEOPde+H499PdshLmYt7oR1YFzUCtyS08BMvJqaq28aMJ4x8C5hIWRjn6nOOINCe61D75ai0Myl9jRYN9PRHutOOEONfT8Bz3ipZJCZSWYNdx4YNoiDSlXlQuNXFRlq6phkBfVJ/8xdIARRFrPAHNxDLlHYYPcwJlh5yKClNC9zfbVsPKLBkir6qhzL2uobcddbYWfWPhljCcVkFmg5dWfOzsmgMNmUbrh8icKeK+624eDX1Msx1jmTwTNQlDu1mF62uNZFI8oc5ptrSu6D/ovmZrJZbZU096IxhmHyOsEWeUfJ4s7ozYYPvbWwNlPAcmccb57c59/j+zw/fwF5AX1LCmVuZHN0cmVhbQplbmRvYmoKMTkgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyNzggPj4Kc3RyZWFtCnicRZG7cQQxDEPzrYIlUPxK9ZzH0br/1ADlGQd3xAr8PFFtJip9+KdHepV8rWe+fxh48D5+6k9F6ajUPQUZSwru58kOqe1SihNklSHahlPISeRUItJhhAsHKupIeUhYTa0fY7fa4tmSq8U9JeKIG+cE5vhC/nFxNZCU2FHWoItVzAxzl9wp6yQckiroE3mMAZqFjrzPRg4oIsWMFC0WzTmosVbwuNgucWzDpptXYjIoEu7Cz2uI4BgcbMcV1Gujzid+ntVXvaPI0BeJn6HctnePApTpqEgAYVD0lljOK+LCgSfj+gJjeeXIWX8U8nCS6Hd64l3yKHTKUKwEtYuLJm4cPhHq0vAEdzrjxaX653uf719lwWhXCmVuZHN0cmVhbQplbmRvYmoKMjAgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCA2OCA+PgpzdHJlYW0KeJwzMzRUMFAwMgYSppaGCuaGZgophlxGppZAgVwwbWZsqJDDBVQBZ4AU5XDBlENYEEljUxMklgFIOdikHK40ABYMFBsKZW5kc3RyZWFtCmVuZG9iagoyMSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDQzNyA+PgpzdHJlYW0KeJw1kklyJTEIRPd1Ci7gCDFoOs/v8Mp9/22/pNyLKpAQkJkw57JhuezL3aaXzRz2x58aZavs75PbFc4a5hgfNu3zxBn2NS1qd2J4tv08Pt9S7mFhJ4xyn2dS+6jMTf09N5dyVljx+Ez6WozF9aJsbKNBVNm9FlOv3bfFuuQei307NY4SnFNcng8yb5GGTx4dAJJj05K25Ofli47Io/Nrz2tn/I8cbs4FGnk7reIoMoeV3qJDTaGItqgByb4ZsggF+MrGtvAChoV2dzbznPeVRNL+PJwKjCpGEB61JJmPY4V+nmlzSPzNfIQwBmrGy1PTilZPOeImL9FQLxK5NdPPIwyTkRac6/JN/K1JFnVLGDasqFiHqAt7Hd6IESq3CrLZ1fACPX/a85zEmFh16SWMBVfBGwxpNIbRKAJLFjwcekOi2O+qvdIH5Fm69e6WhhYIGdqO0BqobUjQq61DUGDHuC01NyPNNQCIe6lJ7ySgfR2AEoF42+wcearCUl2YsLynxd8NSfOcQlDWOxgU0fkeRROF9/1dDPYut4phj5r3PC4QICRizj41wXeXfqn+PN//ABlPplMKZW5kc3RyZWFtCmVuZG9iagoyMiAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDI0NiA+PgpzdHJlYW0KeJw1UUlyBDEIu/sV+kCqDMLGfk+nckr+f41gZg5dwsZoodecmODFlxniGhYnvm3EDNid+Bt1aXnxO+KosotYiXSEnRp8BtVdIK1JPFfjM3yyK4sNc5iO6+h+T9VRs5at7SIUzQWNCLFux06Uh2echSiCamXCVvksGjuSlR2X43JdwoVi4isH9X6Z5pu2NCLKvr63/zgutd3qCS4qJsVLvWZGT3IJac0rHjFwalJRPG+jojK6MjmL8A4WVl5MJ6Y6rjl/oe/uqKoV1wurZWx9s5PdhdwdbNdCo0DyaqogtX6BSK7X9WFvh9KuVX9+3TN+/gHOaljNCmVuZHN0cmVhbQplbmRvYmoKMjMgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyNTcgPj4Kc3RyZWFtCnicNVAxkgMxCOv9Cj6QGYMA2+9J5qrN/9sTeFJJi1cgKSJlClxequLbZWnKR4dP4/zbmCHPQB5RF5j2rxar8T1Mo5muRYU6xMCxzi0eslU4TxPVzSlfNXbdsxkNcTgmgU5xE3Bv0tCpMZXu3DwhvlwU/D5Zy5dKcKFRFJjcgka6YYRiUJOgEVATWi9IBjxLsCtonUga7OtkFfsZvwIum4XdwzPUor1+m+lhIGymJWYyXF3Q4xXWjBHYEOdZBWF6EYBXUpCsYO4+y7pwxPmuezValKIYjGXwDzB4afxqEF0JaMtZpOVFBrfLlBpvWy5+bdEoFl9oHPT2i/Ief//jlV6CCmVuZHN0cmVhbQplbmRvYmoKMjQgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyNTUgPj4Kc3RyZWFtCnicNVDJbQQxDPu7CjYQQKdl17NBXpv+v6E0G4wHImyRlJi5IfDClypSE+mCb11hCt2F35VqUHa9V9yCiiFKsBXhBp7X8uvgFzEidp76WiZnkAZ5FBFHHt7nJY421Rpvy2yZooaBr6EyHTHtGgcpGyY101ndqWT0C1FITkcEueS/OKpTxWYjjz3VdnMGZfAmYBxsKq3pYzXovZSaShclU51/JefZs1KgOEpMAr3q7k1dd4OOYF84czvd7ec+gUkHwNk+odKrs5PLeMMexHj1wNOn2w/nJrsxdTrtoL49mdiRTzbm97lhAkF3rcO9xyEZ7eUeTiXu++/4Wj9/SRdcugplbmRzdHJlYW0KZW5kb2JqCjI1IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTYwID4+CnN0cmVhbQp4nE1PORIDMQjr9xV6AmAO857NpEr+30ZeZ7IpPBIWSGA1IWi+oQNlEw89ZrF+L/AQvI7+YWgTKdykfJOUiRbkcHQiQ3EeKo5kg7I7e0BdUVJLSWOaQtuuVBty4XlYxP6Za5/Ye3GeStPFB+NsKlnAkv5eMJ8Xssd0/4gRlz9rejOxOK0Tyn2ia2Pmpfj3Hqv4Y/vopd5M9rELnh/cbTvYCmVuZHN0cmVhbQplbmRvYmoKMjYgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyNDYgPj4Kc3RyZWFtCnicRVE7bsUwDNt9Cl6ggPW1fZ4UnV7vv5ZMAnRIxNgSSTFVjYl0fJmhrLFm49sGT2xv/A6LhJ3CZ1hOWOlpGDttG07iGs6RZfBo9IQTslwjLAQiD1Yj1oHNzfPkW1zpQQ6/q0fpRmgX1BGeiM3xCnGV84uPFeIsisy7UpxO7xM6ikN3J6ilG1NP071m89EMl4NaiNhayZ+FPyNJ/o/aXbekfVFtZEwin4bUltnIVXDKqcpi3Ujmk6az2GkKIplSdN/xxhuzp9YSssV+KhmVspjVnQSzM7okh36MMlV9shYyKnDGOCMirsp8UywL77+7xs8fHkpY9gplbmRzdHJlYW0KZW5kb2JqCjI3IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMzUzID4+CnN0cmVhbQp4nD1Sy40lIRC7dxROoCXqC8TzVnPazf86NszsqSzA7U91VWMgHK+PjVwbFQN/7KmBNx3/HovCW4W/RBvvMlhy2hiw5pWZ4/PYmoS+4NYEMeGVF3we3z8wvO+ryPXLjEml3YjFuxkIPc7UzeYjMlJSdkYvnbfBHWFB634CyEBymm+eYA9MCRfNSs1h+6T0PpIi84OGqIna1Nw8JiV5ZiOQNCLDSWP89jSUKZudelyskGrwVChorEbR40KWOEJlm7WdUv8jpr2ADbJvZm8m7LyNkneaiUQy4ms9bjG2jpy2YjQbY96NOTdzAF3uuNAy9KqYRPtpNdFaT2jDLFtez3ZJ8mApW3sWGowfDVNxzQr8VMvuFtN7Yup1adDMOCBi6TYYw2yftZFIgaRHedX0vp3oF1DdpLHtaDV2OHG7D3Vf1Oo7+e9QVcg2F0bLxqrSji0ajckblwnDb5TP8/UNIeKGVgplbmRzdHJlYW0KZW5kb2JqCjI4IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTczID4+CnN0cmVhbQp4nE1QuxFDMQjrPYVG4GvwPMmletm/DfByuRS2hHQGYfcNwu7LMhG88eQ19buhhWux2x8zP82OwWlDbMOVoHQGH0stbiUZLgJrh6Ic04CdUjxhwXVqrHk7WSrnhNA4N8oZJyvMtYzoh+18WSj0VBfy4tVRupu6TF+tytwhhwcfS/ZXsZ6cEK5EauX0PiYEjkpBAt53knIqrdY/9e4qNig5b4p1pvmva70+/I0+swplbmRzdHJlYW0KZW5kb2JqCjI5IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggNjcgPj4Kc3RyZWFtCnicMzIyUjBQMDMDEoamJgrmhmYKKYZcQL6ZoalCLogBEsrhgklCWCDJHJgqMMMAotjU0BKqBMEygKnI4UoDAJV6FUwKZW5kc3RyZWFtCmVuZG9iagozMCAwIG9iago8PCAvQkJveCBbIC02NjUgLTMyNSAyMDI5IDEwMzggXSAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDM1Ci9TdWJ0eXBlIC9Gb3JtIC9UeXBlIC9YT2JqZWN0ID4+CnN0cmVhbQp4nOMyNbJQMDY0UsjlMjUDM3LADEsTEAMkh2CBJdMA7sYJtQplbmRzdHJlYW0KZW5kb2JqCjMxIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTgxID4+CnN0cmVhbQp4nE1QQRICMQi79xU8oRDownt0POn/rwYcHQ/bBFLSsBFHtpw+PC8JbLnrmvrVEFryXOrxx5wfWUJiqxhyxqB78Lbg+ulc7JgLqn1Axc04Y3Swec6DbqdaOclKxS92rajyxvZWMgSZcx9RH9SZIdtMgqofQuPL6IbiLB2RNZzZ2pdZOptbO0KcG1BBb5bj4OFiZYO3ZTynYzrJtVhrz+ihAyulCq9By960WWeaP/lcjzeeU0O7CmVuZHN0cmVhbQplbmRvYmoKMzIgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyMzIgPj4Kc3RyZWFtCnicLVBBksRACLrnFX5gq1pFO/2e2Zrb/v+6YHKCxEbAqrZlmfbjbuXHKpf9+sU/Ucf+RLLKyBFt7mnYaZ/La/O9W3iMJnYPfq7EHoZF2WpDuaE1weEXN8gncQajNyfD1uL7Y049biI5NX1sc0EyAGHRcUw6lTt8gstc+LliPVUcMCZz7bxlUORQUee2tx1bBN6eYn44zptiInO5y8pP2d4WGdaPVcspmYMkeUBO8673ORyzAMEKB4PRoQlZhk7AIBujwVI6XRislzwDmFcmmNxyFVMIvVCsR6OguenK4BkPPqW+/1TOVsIKZW5kc3RyZWFtCmVuZG9iagozMyAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDExNCA+PgpzdHJlYW0KeJw1TssNQzEMumcKRvDf8Tyv6ind/1rHai8GYUC4BwhM1VdTkVx48bqU8FmyvfEMegwLhRtBtJU2CzGsCs/iSFgWWAMWNqXmdj/NXKvT7Lt7ZFJet2UjRNsjaQh3KBFiJ5RjxjzrP+v8Vp31/gItliJeCmVuZHN0cmVhbQplbmRvYmoKMzQgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCA0MiA+PgpzdHJlYW0KeJwzMrdQMFCwNAQShkDS0MBAIcWQC8zP5YIK5HAZorBANJRKAwB+zAwSCmVuZHN0cmVhbQplbmRvYmoKMzUgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxNDQgPj4Kc3RyZWFtCnicTY8xsgMhDEP7PYWPYMk2sOfJn19t7t9GhhQpQALNk8cRYW6jdEVOq3D7w7Xf75bCbc+FzB+X6e2G3ByGRSt3o06B9roIFTGNMXYh66iSdVxAyu9Ib6Z/kt3LW71B4wzpLZpbRcdxREljT0w2jSUGbhAT4jGmxcxOSi5pKCW+tnJiJ735c3Z9rv8PwzQxjwplbmRzdHJlYW0KZW5kb2JqCjM2IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggNDA4ID4+CnN0cmVhbQp4nC2TOXIDQQhF8zkFF1BVs/R2Hrkc2fdP/T5yoIGhp/kLaI5hw9Lt5W613GYO+/KHis9pv4/7MV/Hfh6PMM/kt8wHv3nsHHs/fobtYeFhNIjZ4f3E7SS5tq5lhZ1JOan5oL6J8R8rdaJspeUCaB+uTPM7dCLYS2WkxThgTIvQiV8QRagW1dEdg/vv51LYZXtb0GMVIsVqgphhtE6aKByVSWqU0aFiinaVyG6ZMu0sqyPaZXVLsLgyeZMXE92+BvG2GXQJsMdtL0VOET/2J0u+nwEfROuuhAuZk7vBgQlVwUKLTmJSdCkwCxfzY+NcWJfMJTE8rxwW+dGGV/Y32FVICkwophWVHeEyojPfqmjW9M8eJs8KKaMbGhTzep+Q7ds7kEzUCytXD6EYjcyft1X5xtbc7QbfZrYbKVfE1eWgnqGRihee5YmeF5rZrWANpD0K5uiK2D0k7ozde+onPnHKwc6km7c7W/7SNNozKFwogNGrJ/C49hJ+9N6L1au3Q9NTJo100sZRZZ9gCQ25/PljvJ/vP4XjmJkKZW5kc3RyZWFtCmVuZG9iagozNyAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDEzOCA+PgpzdHJlYW0KeJw1j0sSwyAMQ/ecQkfwF+PzpNMVvf+2IplsQJaYZ5E5IYjikaooKXx0cJ5m+B1xrD3e8FHTF1XMRK5GaCMt4JWICFzDXeAzYJ2wpbBSaBcTS4d6wcJA0wgS2no32LwX2EizoSTqEpgcogkfLxJdSX6I4Xl2sU9Kw0lOut7rLn+9v9jj+wdnSysWCmVuZHN0cmVhbQplbmRvYmoKMzggMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxNyA+PgpzdHJlYW0KeJwzMrdQMIDDFEMuABrjAvEKZW5kc3RyZWFtCmVuZG9iagozOSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDE3MCA+PgpzdHJlYW0KeJxFUDkOwzAM2/0KfiCARR+y3pOiU/v/tZRTJItJkLJIm75QYQvHBN3gteJlhWPBA9+SmuFT2AeOBrLtydoTzmLOJNYdhwZbxUrVmCtNu5ohGnqqa2B2LCIiTxtMkeijKkDzNxkWIrJuMhUga8YueLHLzKYP+6+Q+zC77xrV0fXcOoQdscu6I6QrRQ1tqZylHBNyWAUDVILgLOQm7ITrH65vOsv7BzKGPYkKZW5kc3RyZWFtCmVuZG9iago0MCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDI2NyA+PgpzdHJlYW0KeJw1UUlywzAMu/sVeIK4i+9Jp6fk/9eCzHTGMmhzA6CIxIE2X3EMJY0feSa8js8GB+/HzgLrVGAGl3lS8HrC0GxUiDr6Qjjx9cyH3IKkQZVHeDKY0eYEvTA3WBFrZk2Psdtjhiv83sVQZWYjzrVuxCWWc/mZHm+kOUwK6QmtL3KPxffPIVFSlkrkucMtKPaSsBXC64tn9zDgqveIimpMC6UL6WWuLJIoDlSR9UqniDhEaiPnoCRNd+Ia5FyVtGBWBCcu6pCfyGmHd8JplNNzt1gizJxaO8YkV4r2uyb1irVwbg+MnbomqdF81uqh9ayV25Q2GaFdo0GSog/1hM71vv7v+f38/gErHWDYCmVuZHN0cmVhbQplbmRvYmoKNDEgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxODkgPj4Kc3RyZWFtCnicTVDBbQAxCPtnCi9QKUAgME+rvq77f2uSq9QHMjJgA+6BiVj4EMHKBZfCl4w1m/85uAPPsHBIwmSeVl1y8HPoy0iSYY87grRoQTZkFkxRAZ9k0xCJvZCFYIM4yVZmD5cQrwO1m77LPENc/2Vq8maSbWeMnqSXZRuHHV2hC3WkFDzr7rknx4+TXifSFGFi3JNVM7vdxr9w2rYeMUuiVReKp4bCeJIwGvsZXYl3zb8/3mw2nnc+4/sX9s1EjAplbmRzdHJlYW0KZW5kb2JqCjQyIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjcyID4+CnN0cmVhbQp4nDVRS24FMQjbzyl8gUr8Sc4zVXe9/7Ym6ZNmBAnYGCezIAjDlypSFlo3vvXhjafjd5LwZolfohYyG++zmMuGElXBGDz3FLQ3mp1mfris88T3cb03Hs2o8C7UTlZCguhNGQtF+mBHMWelCrlZWVRXgdiGNGMlhcKWHM3BWRNH8VQWrIghzkTZZdS3D1tik943kiiqdBlhreC61seEETzxfUxupsnfuatRFe/JoqZjmukM/5+r/vFsMvM8rij30R70OpQCLmrOheWMqqNAT+KxpjrG3PYxZgiCpyGceIdNFtQ9HRkXk2swQ1JWcOWMKA65kcPd7w1NKeOj8cHyPIDS8Dxb0tQWfFZ5n58/9JZlSwplbmRzdHJlYW0KZW5kb2JqCjE1IDAgb2JqCjw8IC9CYXNlRm9udCAvQXJpYWxNVCAvQ2hhclByb2NzIDE2IDAgUgovRW5jb2RpbmcgPDwKL0RpZmZlcmVuY2VzIFsgMzIgL3NwYWNlIDQ2IC9wZXJpb2QgNDggL3plcm8gL29uZSAvdHdvIDUzIC9maXZlIDU1IC9zZXZlbiA3MCAvRiAvRyA4MgovUiA4NCAvVCA5NyAvYSA5OSAvYyAvZCAvZSAvZiAvZyAvaCAvaSAxMTAgL24gL28gMTE0IC9yIC9zIC90IC91IF0KL1R5cGUgL0VuY29kaW5nID4+Ci9GaXJzdENoYXIgMCAvRm9udEJCb3ggWyAtNjY1IC0zMjUgMjAyOSAxMDM4IF0gL0ZvbnREZXNjcmlwdG9yIDE0IDAgUgovRm9udE1hdHJpeCBbIDAuMDAxIDAgMCAwLjAwMSAwIDAgXSAvTGFzdENoYXIgMjU1IC9OYW1lIC9BcmlhbE1UCi9TdWJ0eXBlIC9UeXBlMyAvVHlwZSAvRm9udCAvV2lkdGhzIDEzIDAgUiA+PgplbmRvYmoKMTQgMCBvYmoKPDwgL0FzY2VudCA5MDYgL0NhcEhlaWdodCA3MTYgL0Rlc2NlbnQgLTIxMiAvRmxhZ3MgMzIKL0ZvbnRCQm94IFsgLTY2NSAtMzI1IDIwMjkgMTAzOCBdIC9Gb250TmFtZSAvQXJpYWxNVCAvSXRhbGljQW5nbGUgMAovTWF4V2lkdGggMTAxNSAvU3RlbVYgMCAvVHlwZSAvRm9udERlc2NyaXB0b3IgL1hIZWlnaHQgNTE5ID4+CmVuZG9iagoxMyAwIG9iagpbIDc1MCA3NTAgNzUwIDc1MCA3NTAgNzUwIDc1MCA3NTAgNzUwIDc1MCA3NTAgNzUwIDc1MCA3NTAgNzUwIDc1MCA3NTAgNzUwCjc1MCA3NTAgNzUwIDc1MCA3NTAgNzUwIDc1MCA3NTAgNzUwIDc1MCA3NTAgNzUwIDc1MCA3NTAgMjc4IDI3OCAzNTUgNTU2IDU1Ngo4ODkgNjY3IDE5MSAzMzMgMzMzIDM4OSA1ODQgMjc4IDMzMyAyNzggMjc4IDU1NiA1NTYgNTU2IDU1NiA1NTYgNTU2IDU1NiA1NTYKNTU2IDU1NiAyNzggMjc4IDU4NCA1ODQgNTg0IDU1NiAxMDE1IDY2NyA2NjcgNzIyIDcyMiA2NjcgNjExIDc3OCA3MjIgMjc4CjUwMCA2NjcgNTU2IDgzMyA3MjIgNzc4IDY2NyA3NzggNzIyIDY2NyA2MTEgNzIyIDY2NyA5NDQgNjY3IDY2NyA2MTEgMjc4IDI3OAoyNzggNDY5IDU1NiAzMzMgNTU2IDU1NiA1MDAgNTU2IDU1NiAyNzggNTU2IDU1NiAyMjIgMjIyIDUwMCAyMjIgODMzIDU1NiA1NTYKNTU2IDU1NiAzMzMgNTAwIDI3OCA1NTYgNTAwIDcyMiA1MDAgNTAwIDUwMCAzMzQgMjYwIDMzNCA1ODQgNzUwIDU1NiA3NTAgMjIyCjU1NiAzMzMgMTAwMCA1NTYgNTU2IDMzMyAxMDAwIDY2NyAzMzMgMTAwMCA3NTAgNjExIDc1MCA3NTAgMjIyIDIyMiAzMzMgMzMzCjM1MCA1NTYgMTAwMCAzMzMgMTAwMCA1MDAgMzMzIDk0NCA3NTAgNTAwIDY2NyAyNzggMzMzIDU1NiA1NTYgNTU2IDU1NiAyNjAKNTU2IDMzMyA3MzcgMzcwIDU1NiA1ODQgMzMzIDczNyA1NTIgNDAwIDU0OSAzMzMgMzMzIDMzMyA1NzYgNTM3IDI3OCAzMzMgMzMzCjM2NSA1NTYgODM0IDgzNCA4MzQgNjExIDY2NyA2NjcgNjY3IDY2NyA2NjcgNjY3IDEwMDAgNzIyIDY2NyA2NjcgNjY3IDY2NwoyNzggMjc4IDI3OCAyNzggNzIyIDcyMiA3NzggNzc4IDc3OCA3NzggNzc4IDU4NCA3NzggNzIyIDcyMiA3MjIgNzIyIDY2NyA2NjcKNjExIDU1NiA1NTYgNTU2IDU1NiA1NTYgNTU2IDg4OSA1MDAgNTU2IDU1NiA1NTYgNTU2IDI3OCAyNzggMjc4IDI3OCA1NTYgNTU2CjU1NiA1NTYgNTU2IDU1NiA1NTYgNTQ5IDYxMSA1NTYgNTU2IDU1NiA1NTYgNTAwIDU1NiA1MDAgXQplbmRvYmoKMTYgMCBvYmoKPDwgL0YgMTcgMCBSIC9HIDE4IDAgUiAvUiAxOSAwIFIgL1QgMjAgMCBSIC9hIDIxIDAgUiAvYyAyMiAwIFIgL2QgMjMgMCBSCi9lIDI0IDAgUiAvZiAyNSAwIFIgL2ZpdmUgMjYgMCBSIC9nIDI3IDAgUiAvaCAyOCAwIFIgL2kgMjkgMCBSIC9uIDMxIDAgUgovbyAzMiAwIFIgL29uZSAzMyAwIFIgL3BlcmlvZCAzNCAwIFIgL3IgMzUgMCBSIC9zIDM2IDAgUiAvc2V2ZW4gMzcgMCBSCi9zcGFjZSAzOCAwIFIgL3QgMzkgMCBSIC90d28gNDAgMCBSIC91IDQxIDAgUiAvemVybyA0MiAwIFIgPj4KZW5kb2JqCjMgMCBvYmoKPDwgL0YxIDE1IDAgUiA+PgplbmRvYmoKNCAwIG9iago8PCAvQTEgPDwgL0NBIDAgL1R5cGUgL0V4dEdTdGF0ZSAvY2EgMSA+PgovQTIgPDwgL0NBIDEgL1R5cGUgL0V4dEdTdGF0ZSAvY2EgMSA+PgovQTMgPDwgL0NBIDAuNSAvVHlwZSAvRXh0R1N0YXRlIC9jYSAwLjUgPj4KL0E0IDw8IC9DQSAwLjggL1R5cGUgL0V4dEdTdGF0ZSAvY2EgMC44ID4+ID4+CmVuZG9iago1IDAgb2JqCjw8ID4+CmVuZG9iago2IDAgb2JqCjw8ID4+CmVuZG9iago3IDAgb2JqCjw8IC9GMS1BcmlhbC1taW51cyAzMCAwIFIgL00wIDEyIDAgUiA+PgplbmRvYmoKMTIgMCBvYmoKPDwgL0JCb3ggWyAtMTAuNSAtMTAuNSAxMC41IDEwLjUgXSAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDM2Ci9TdWJ0eXBlIC9Gb3JtIC9UeXBlIC9YT2JqZWN0ID4+CnN0cmVhbQp4nDNUyOIyUPDi0jVWAKJcLmMFY4UcEA/C0QXxuJy4AI05BuYKZW5kc3RyZWFtCmVuZG9iagoyIDAgb2JqCjw8IC9Db3VudCAxIC9LaWRzIFsgMTAgMCBSIF0gL1R5cGUgL1BhZ2VzID4+CmVuZG9iago0MyAwIG9iago8PCAvQ3JlYXRpb25EYXRlIChEOjIwMjIwNzA1MjAwNDU5KzAyJzAwJykKL0NyZWF0b3IgKE1hdHBsb3RsaWIgdjMuMy4yLCBodHRwczovL21hdHBsb3RsaWIub3JnKQovUHJvZHVjZXIgKE1hdHBsb3RsaWIgcGRmIGJhY2tlbmQgdjMuMy4yKSA+PgplbmRvYmoKeHJlZgowIDQ0CjAwMDAwMDAwMDAgNjU1MzUgZiAKMDAwMDAwMDAxNiAwMDAwMCBuIAowMDAwMDI5NTg4IDAwMDAwIG4gCjAwMDAwMjkxMDQgMDAwMDAgbiAKMDAwMDAyOTEzNiAwMDAwMCBuIAowMDAwMDI5MzIxIDAwMDAwIG4gCjAwMDAwMjkzNDIgMDAwMDAgbiAKMDAwMDAyOTM2MyAwMDAwMCBuIAowMDAwMDAwMDY1IDAwMDAwIG4gCjAwMDAwMDAzOTkgMDAwMDAgbiAKMDAwMDAwMDIwOCAwMDAwMCBuIAowMDAwMDIwMDAxIDAwMDAwIG4gCjAwMDAwMjk0MTggMDAwMDAgbiAKMDAwMDAyNzc1OCAwMDAwMCBuIAowMDAwMDI3NTU4IDAwMDAwIG4gCjAwMDAwMjcxMzYgMDAwMDAgbiAKMDAwMDAyODgwOSAwMDAwMCBuIAowMDAwMDIwMDIzIDAwMDAwIG4gCjAwMDAwMjAxNjcgMDAwMDAgbiAKMDAwMDAyMDU4MCAwMDAwMCBuIAowMDAwMDIwOTMxIDAwMDAwIG4gCjAwMDAwMjEwNzEgMDAwMDAgbiAKMDAwMDAyMTU4MSAwMDAwMCBuIAowMDAwMDIxOTAwIDAwMDAwIG4gCjAwMDAwMjIyMzAgMDAwMDAgbiAKMDAwMDAyMjU1OCAwMDAwMCBuIAowMDAwMDIyNzkxIDAwMDAwIG4gCjAwMDAwMjMxMTAgMDAwMDAgbiAKMDAwMDAyMzUzNiAwMDAwMCBuIAowMDAwMDIzNzgyIDAwMDAwIG4gCjAwMDAwMjM5MjEgMDAwMDAgbiAKMDAwMDAyNDA4OCAwMDAwMCBuIAowMDAwMDI0MzQyIDAwMDAwIG4gCjAwMDAwMjQ2NDcgMDAwMDAgbiAKMDAwMDAyNDgzNCAwMDAwMCBuIAowMDAwMDI0OTQ4IDAwMDAwIG4gCjAwMDAwMjUxNjUgMDAwMDAgbiAKMDAwMDAyNTY0NiAwMDAwMCBuIAowMDAwMDI1ODU3IDAwMDAwIG4gCjAwMDAwMjU5NDYgMDAwMDAgbiAKMDAwMDAyNjE4OSAwMDAwMCBuIAowMDAwMDI2NTI5IDAwMDAwIG4gCjAwMDAwMjY3OTEgMDAwMDAgbiAKMDAwMDAyOTY0OCAwMDAwMCBuIAp0cmFpbGVyCjw8IC9JbmZvIDQzIDAgUiAvUm9vdCAxIDAgUiAvU2l6ZSA0NCA+PgpzdGFydHhyZWYKMjk4MDUKJSVFT0YK\n", "image/svg+xml": [ "\n", "\n", "\n", "\n", " \n", " \n", " \n", " \n", " 2022-07-05T20:04:59.201248\n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.3.2, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n" ], "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "train_set = RegressionDataset(num_points=1000, seed=42)\n", "val_set = RegressionDataset(num_points=200, seed=43)\n", "test_set = RegressionDataset(num_points=500, seed=44)\n", "train_loader, val_loader, test_loader = create_data_loaders(train_set, val_set, test_set,\n", " train=[True, False, False],\n", " batch_size=64)\n", "\n", "x = np.linspace(-2, 2, 1000)\n", "plt.scatter(train_set.x, train_set.y, color='C1', marker='x', alpha=0.5, label='Training set')\n", "plt.plot(x, target_function(x), linewidth=3.0, label='Ground Truth Function')\n", "plt.legend()\n", "plt.title('Regression function')\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "07abb34a", "metadata": {}, "source": [ "Note that even with PyTorch Lightning, we would have needed the similar setup here." ] }, { "cell_type": "markdown", "id": "04dc5354", "metadata": {}, "source": [ "### Model\n", "\n", "The Trainer module does not restrict us to any specific model, so let's implement our own MLP class for regression:" ] }, { "cell_type": "code", "execution_count": 8, "id": "f309da60", "metadata": {}, "outputs": [], "source": [ "class MLPRegressor(nn.Module):\n", " hidden_dims : Sequence[int]\n", " output_dim : int\n", " \n", " @nn.compact\n", " def __call__(self, x, **kwargs):\n", " for dims in self.hidden_dims:\n", " x = nn.Dense(dims)(x)\n", " x = nn.silu(x)\n", " x = nn.Dense(self.output_dim)(x)\n", " return x" ] }, { "cell_type": "markdown", "id": "76819164", "metadata": {}, "source": [ "### Trainer\n", "\n", "Now comes the interesting part. Using the `TrainerModule`, we only need to overwrite the aspects that are needed: the training and validation step. For the regression task and the small model, this reduces to writing the mean-squared error loss. Note that we still have some minor repetitive code here, such as creating the gradient function with `jax.value_and_grad` in the training step and applying the update with `state.apply_gradients`. One could reduce this further by restricting possible loss functions, but to keep flexibility high, we implement the whole training step here." ] }, { "cell_type": "code", "execution_count": 9, "id": "233673a6", "metadata": {}, "outputs": [], "source": [ "class MLPRegressTrainer(TrainerModule):\n", " \n", " def __init__(self,\n", " hidden_dims : Sequence[int],\n", " output_dim : int,\n", " **kwargs):\n", " super().__init__(model_class=MLPRegressor,\n", " model_hparams={\n", " 'hidden_dims': hidden_dims,\n", " 'output_dim': output_dim\n", " },\n", " **kwargs)\n", " \n", " def create_functions(self):\n", " def mse_loss(params, batch):\n", " x, y = batch\n", " pred = self.model.apply({'params': params}, x)\n", " loss = ((pred - y) ** 2).mean()\n", " return loss\n", " \n", " def train_step(state, batch):\n", " loss_fn = lambda params: mse_loss(params, batch)\n", " loss, grads = jax.value_and_grad(loss_fn)(state.params)\n", " state = state.apply_gradients(grads=grads)\n", " metrics = {'loss': loss}\n", " return state, metrics\n", " \n", " def eval_step(state, batch):\n", " loss = mse_loss(state.params, batch)\n", " return {'loss': loss}\n", " \n", " return train_step, eval_step" ] }, { "cell_type": "markdown", "id": "eb14ea13", "metadata": {}, "source": [ "And that's already it! This now looks much more like the minimal code of PyTorch Lightning and automatically logs our metrics as we want." ] }, { "cell_type": "markdown", "id": "5707e1bc", "metadata": {}, "source": [ "### Training\n", "\n", "To train the model, we simply specify our hyperparameters and create a Trainer module:" ] }, { "cell_type": "code", "execution_count": 10, "id": "6a5e6914", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
                    MLPRegressor Summary                     \n",
       "┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃ path          outputs          params                   ┃\n",
       "┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│ Inputs       │ - float64[64,1] │                          │\n",
       "│              │ - train: True   │                          │\n",
       "├──────────────┼─────────────────┼──────────────────────────┤\n",
       "│ Dense_0      │ float32[64,128] │ bias: float32[128]       │\n",
       "│              │                 │ kernel: float32[1,128]   │\n",
       "│              │                 │                          │\n",
       "│              │                 │ 256 (1.0 KB)             │\n",
       "├──────────────┼─────────────────┼──────────────────────────┤\n",
       "│ Dense_1      │ float32[64,128] │ bias: float32[128]       │\n",
       "│              │                 │ kernel: float32[128,128] │\n",
       "│              │                 │                          │\n",
       "│              │                 │ 16,512 (66.0 KB)         │\n",
       "├──────────────┼─────────────────┼──────────────────────────┤\n",
       "│ Dense_2      │ float32[64,1]   │ bias: float32[1]         │\n",
       "│              │                 │ kernel: float32[128,1]   │\n",
       "│              │                 │                          │\n",
       "│              │                 │ 129 (516 B)              │\n",
       "├──────────────┼─────────────────┼──────────────────────────┤\n",
       "│ MLPRegressor │ float32[64,1]   │                          │\n",
       "├──────────────┼─────────────────┼──────────────────────────┤\n",
       "│                         Total  16,897 (67.6 KB)         │\n",
       "└──────────────┴─────────────────┴──────────────────────────┘\n",
       "                                                             \n",
       "             Total Parameters: 16,897 (67.6 KB)              \n",
       "
\n" ], "text/plain": [ "\u001b[3m MLPRegressor Summary \u001b[0m\n", "┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n", "┃\u001b[1m \u001b[0m\u001b[1mpath \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1moutputs \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mparams \u001b[0m\u001b[1m \u001b[0m┃\n", "┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n", "│ Inputs │ - \u001b[2mfloat64\u001b[0m[64,1] │ │\n", "│ │ - train: True │ │\n", "├──────────────┼─────────────────┼──────────────────────────┤\n", "│ Dense_0 │ \u001b[2mfloat32\u001b[0m[64,128] │ bias: \u001b[2mfloat32\u001b[0m[128] │\n", "│ │ │ kernel: \u001b[2mfloat32\u001b[0m[1,128] │\n", "│ │ │ │\n", "│ │ │ \u001b[1m256 \u001b[0m\u001b[1;2m(1.0 KB)\u001b[0m │\n", "├──────────────┼─────────────────┼──────────────────────────┤\n", "│ Dense_1 │ \u001b[2mfloat32\u001b[0m[64,128] │ bias: \u001b[2mfloat32\u001b[0m[128] │\n", "│ │ │ kernel: \u001b[2mfloat32\u001b[0m[128,128] │\n", "│ │ │ │\n", "│ │ │ \u001b[1m16,512 \u001b[0m\u001b[1;2m(66.0 KB)\u001b[0m │\n", "├──────────────┼─────────────────┼──────────────────────────┤\n", "│ Dense_2 │ \u001b[2mfloat32\u001b[0m[64,1] │ bias: \u001b[2mfloat32\u001b[0m[1] │\n", "│ │ │ kernel: \u001b[2mfloat32\u001b[0m[128,1] │\n", "│ │ │ │\n", "│ │ │ \u001b[1m129 \u001b[0m\u001b[1;2m(516 B)\u001b[0m │\n", "├──────────────┼─────────────────┼──────────────────────────┤\n", "│ MLPRegressor │ \u001b[2mfloat32\u001b[0m[64,1] │ │\n", "├──────────────┼─────────────────┼──────────────────────────┤\n", "│\u001b[1m \u001b[0m\u001b[1m \u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m Total\u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m16,897 \u001b[0m\u001b[1;2m(67.6 KB)\u001b[0m\u001b[1m \u001b[0m\u001b[1m \u001b[0m│\n", "└──────────────┴─────────────────┴──────────────────────────┘\n", "\u001b[1m \u001b[0m\n", "\u001b[1m Total Parameters: 16,897 \u001b[0m\u001b[1;2m(67.6 KB)\u001b[0m\u001b[1m \u001b[0m\n" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "\n", "\n" ] } ], "source": [ "trainer = MLPRegressTrainer(hidden_dims=[128, 128],\n", " output_dim=1,\n", " optimizer_hparams={'lr': 4e-3},\n", " logger_params={'base_log_dir': CHECKPOINT_PATH},\n", " exmp_input=next(iter(train_loader))[0:1],\n", " check_val_every_n_epoch=5)" ] }, { "cell_type": "markdown", "id": "7e5dbc66", "metadata": {}, "source": [ "As one can see, we also automatically print out all layers with their parameters and outputs with Flax's `nn.tabulate` function. This is quite helpful for debugging and gives an intuition about the size of the model. Since the task is not very difficult, we are fine with using less than 20k parameters.\n", "\n", "Next, let's start the training:" ] }, { "cell_type": "code", "execution_count": 11, "id": "596294f4", "metadata": {}, "outputs": [], "source": [ "metrics = trainer.train_model(train_loader, \n", " val_loader, \n", " test_loader=test_loader, \n", " num_epochs=50)" ] }, { "cell_type": "code", "execution_count": 12, "id": "a5812e65", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Training loss: 0.0008829445578157902\n", "Validation loss: 0.0008724514045752585\n", "Test loss: 0.0007670423365198076\n" ] } ], "source": [ "print(f'Training loss: {metrics[\"train/loss\"]}')\n", "print(f'Validation loss: {metrics[\"val/loss\"]}')\n", "print(f'Test loss: {metrics[\"test/loss\"]}')" ] }, { "cell_type": "markdown", "id": "b6efa103", "metadata": {}, "source": [ "With the speed of JAX, this takes only a few seconds. The logs can be found in `../saved_models/guide4/MLPRegressor/`, and we have a dictionary of the best results here as well. The training, validation and test loss suggest that our model learned the function quite well, but let's check it by predicting the whole function as inference task." ] }, { "cell_type": "markdown", "id": "674ab27d", "metadata": {}, "source": [ "### Inference\n", "\n", "To perform inference, we first bind the model to the parameters. This enables us a simpler API, closer to PyTorch. Applying the model to a values between -2.0 and 2.0 shows that the model learned the sine wave indeed quite well and is only off at the corners." ] }, { "cell_type": "code", "execution_count": 13, "id": "ee055416", "metadata": {}, "outputs": [ { "data": { "application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1BhZ2VzIDIgMCBSIC9UeXBlIC9DYXRhbG9nID4+CmVuZG9iago4IDAgb2JqCjw8IC9FeHRHU3RhdGUgNCAwIFIgL0ZvbnQgMyAwIFIgL1BhdHRlcm4gNSAwIFIKL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gL1NoYWRpbmcgNiAwIFIKL1hPYmplY3QgNyAwIFIgPj4KZW5kb2JqCjEwIDAgb2JqCjw8IC9Bbm5vdHMgWyBdIC9Db250ZW50cyA5IDAgUgovR3JvdXAgPDwgL0NTIC9EZXZpY2VSR0IgL1MgL1RyYW5zcGFyZW5jeSAvVHlwZSAvR3JvdXAgPj4KL01lZGlhQm94IFsgMCAwIDM4Ni41NTkzNzUgMjY1Ljk5NjI1IF0gL1BhcmVudCAyIDAgUiAvUmVzb3VyY2VzIDggMCBSCi9UeXBlIC9QYWdlID4+CmVuZG9iago5IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTEgMCBSID4+CnN0cmVhbQp4nM2aza8U1xHF9/NX9DJZuLnfH0ssx0hZxQ5SFlEWEcYOCLAMcfzv53equmd6BoyeeTwJo4ffFD23b9WtOnVOdcfl5enR47j89G7hryUsL/n5jd+f6PMp8On1KY+21jpzr3x8dfyYWl3nbKliDlef/nM6/XgK64y9lR7qGMvthzJDnC30sbzVjZ+8d8H5w+nm6tOplMsO1q47sss+13xtfHVl5EspmPXw9YvRdvzL8v7SOZd1LCn2tZTl7fPlH8ub5dHj5DH7Kz8v+fGYPfrm+f9ePHv+/ZOvT8/enepce++19sMuD7bLrU9/P323/LIvGtZYOYt9Xfv4ZLOefjlFgvVV4J/KIEYht5njSAt+Djb97PXp66fLo2/jEuPy9MdTW0vKI42YdHhPfzj9Ka3hz8vTl6e/PGWxsIaoBflz/o0l+P5Xj9+++Perr16/ePPru+Wbn0/f6c9yv/icLvFZiM/s60hZt73E52C7b3xI01krq42W293jE9f6hcQn5rqONkprhwAdjfeNEE6vIcTURwkz/pEQfSkpFHumCPJMxwhdbPcOUMtrKK3nNGupdw9Q+GJyKMVIFdxA5dH4SSG6cj+FsuaQ5f0WIA+B5chCCD63S5W1Qxn56NHF9hkcKmnNZdSuQ79xqj6UU2OsoYY0ytGrg/EzuDXCmmcJmWIP8+hWfLCzyqwQZu15Hjv1wXh/t+h+a8GvIZgv12491GlxozWWGOcxB4/Gz+BWBVZGmQX/4ji6lR7stC5L5ba2Ukeb18TqYv5URO1rWnJaexgQF8hdvFu7+VL6zWWpFgkKtdmvI3Qx3ydClY3FXlPrE+pyl37Tv5SGc1kK/hVaieOGnF/M94nQSGtMDR5U07xLDoHcX14OxcjBdRZK1yE62O8ToxjaOnIdJSWq9k5BSl9eGonwlnEr8c7WT0fYmNeW4rDFaBXlzFseHl5jY0MztjpuvLrYP4tf9JBUS6wkU61XR/yw7o2yzhBGvD20i/2zuAehmS3nWEuH0BzL/EHdS2GuLczZ883Y4WL/HO6lUNfWU0F9AJhXOP+w7rFEJgtzvHHvYv8s7mXo6EBhhVnivGr093NvrctvgFeaIyACSX42UnySlDJQt7Z9llSW748zG4XhMKCRjmwNYaNjhtu1mpKSjpoqFeAkCm2sLaaZxxInsrwntetXp468aiUN+G3Ia6ktdxV0R23HMHuyTOmzUJcyd06V3Xb8KkpocBvzEBSEnGGTcdIyZ+gKPI0vbwOSlNhJFOnFDPesMSSZ2WrkLrrlqOvsrCJzkzcxat+jryP0UGQeaB1ioZIcnDVKDtdSmuw7ZkvlGdZRM5loZ5YGq2iDk530Hpof5ci5mTWvdeP2oGrQ/EVWji2Wws3NOno3hjThAtxx+BKa+CVbg+5V4T/DzBV2n7W9CdFn09m3R6CCrUHRjRlDNWvsnITiFENcU4SIN+y4O/ClmB32mWsfw0KS6ixGZyMqlg+cpkUwzR7NHxZe+6AKFauo+WauyeydMCM5saNAFOTmdiKXIe6Y68oFflcUN3SFPUYNM3svtkeqIU/cHlbslbyO3ex8gPcrCXGvkq7T7R23J0mFnb1QN8HumQAGThadEMmPVsa00MSUSS22RvWRk6WPOPx68UtNSKgONMuowy6HmFNuiWvUXvtsvdjmC4cQi7pSJJ+B22wJwIES79DIYjInUGsG7LFwfBRPW/ogpKFlM8595NDIMkREsWsrScR9yNtGNgddYmYqP82y0MAaZ2ib1u8taQRcIzUs6mJmskJNZWggSn01X5hOSgSpNqEIFGhUO2X4+SiRjFoQ+Sru4mYOohGfBVd7DsmNfHFAj1Asc6X23bvWOFWIOWCEe0OVa+ZOXdbSxpL7ajG3kIIQlbzHA8xkw5ZsxIBcVyNEUrE1alnmTnJQJ/giM1K9W5y7QkPRRS3CGp7gwBM0epBfWEsjSX0Nyp8wh6H9sdNYbdudbbegvMQZ8qm6j4APJJpskudgGSExMw6HXik1zUIgD8luOahdPuANQe2zN/ednEN8c9HC0TfqLFragEltcmzoGSV9cT4rAKWeQJNaVwBZxyzrID+4fCxNFGwi6GUmzdH28NeFuANvwzcCngB8qc9FUWD/2aINngziwIrstMwp7JaZ3EtlkqcsB93NQ2Zhc+qNfF8ECuCKgSB+s60ujIuRlAuhWGEDJsSb0FKUnE7grGWlxIIiwSKEmJoiX80Ovc6U/VzkWhjs3W5qGMW+yqLdToA6yCPB4g4nIBG/desZarr8PlI36Ki9Nr8vWyC9q2APtGg66G528JwWrUkYqBBKqAbAiePdhxNsOVfS2tYvcYcUg0Ousf6eStYBa9NqEUSpdotCqeR5IAe8gSNVotuVsHnMZghPTpXg92U/1Dfnk6T70bRzmn2uoBSFgj0LXpLVLVWG8zPVZHYKm1ubXX2v8w/GQ8JoNbidLwNNc7qdDLI2SSlSYTV1Xwe08cxOJN0cpFQ1Ox2pRr++a2LT4vB9trbfF3jh6CkQ+TU5i+bXT6V0U3tWj4ozZfMLgKEWpzgLceujKSSyJ3pHFHAqzmpjweIPyGzYam2HivJlyCXSvnU7xkHdZts+eLJBbiI55yQgFjYAZU5uJjvAhm9GIvCfQiFfRSe5HhgJlp5d9yrsAjuFryjbfbv2RmsgOzWlie4scKDy5askFhAgEJCdwofGkJOLoAvCMz154BfBuUbkfAIFF2z1OXZsUlvlrq7goDMcyhBETw3cYRvRzHWHG25U2ahfDNrBbohTF6sgAt3MU4/DxPCAYtiI86JMNELh+LIaTKYzWnzBfeIIuYw2a+kQjGFmCptGTkmybzattiKzViQWUUjG/4y3wYXhhBM6K9gjmaeddIYlAFPYl6Jc6z26WfGEsxRhJ1VWk5s5CC4mH6k4lF/czHUnhQAw1ePkRZNTJyaUbRcNtIiQaBtVxFy4ybTwicMFVZDMaeTRbNukK8gJYBXNqgQcZlVjBMRtDWolZb9YjRGAKNoHhzwtTYyGT9KgadegdalubjtEihrQqg3bM/1pZvY9FZFSy/SjyWqCMMGp+IHJyRCM7rEzSDCCvExmBI4SjaovAAFV163a4BF0HvCiiilY87fzAnSIHX1BzYRslLeYwRYAEShYWhWUJo91FZ50Jb0T92ztKysxjMiR/dR3M/zLGvA5ZRPngRXUbObz7FdCANZgvOJ3zJt4+oPyZpM2yz8XNs931rL8awmnH1A0o23/SevU41N0glrb4Lzrx7WOHnRRGqNJ6xBr2BUpKXpAlHySTVmFAUFr1tEmgewyU/hbK46Ud1WOmDntrVjAk0l864toIPQ6528wVaoQyCRQKVSY+05QarM1CPyQoDbqTTJFqwL1ds9rwk8/LwZGANPWtp3vz2oAPtQHQjZ8peUkylF3lIrTCNplSkm1mdQ5CCO6AMnZ7MDsqGuPxZtG4grTKSQXuCjQVa+q8DXV0lQLyKortSQ6Y7EyBVDbiCpkV9KlWHVM63JIMDPDfItlFPlC3oRhS6sd2Q2JJNVVs1khz8YdpkKG3vQWS0ueVjOwno2ZmLiCUDjTMrzPJiBheXoi5DrHJqriloofa4iLyS7P6HSuLIGc7Mz2qIu0/KjV7Qdd1G2PTsgleSmtOu3oRVydxqKMBlJPBIFEoSM0J6yx7ERG/E0lbNUqDQT9INdMA5FWwSkhm4CLUGKLaCp9vedbDdT0OA15aOtI+DqhiIC6lMKmXmgYudNZF9IZ4V6z6xf6FaECw4Y6Fy3FzfR4zg4uQmZTNd1CDNAUUo1AUTSw+2CArF6oDowgAkUIZLZpgYQNLE2SEHKCRIM8LS5skICajYgTDRSPmUWPqQNHTdjFCK4/LsKmSAFWp75NMwyIiukdAHm48AT/OGFqFlpIQy1naUNzQQZKOoxurNA1jNKEAgHIiQyAubiGQcbBXUzaqOAs3uBB02GZhIEl+yl01S/In/0Z0qQSFtcwHLQaGuZOIrkco7FnuhrOSPBAWLtLGwFflGfSWPB7a9ri3Z3CB7tEflXru7TZSIXkG1rL88fQPKly8B3rGK5hLtImSba07IoH4qFLrGtrbuRrgypZ1C7rEOg0w6iJzYaAeMLGkaGZ5iabNGPpetjFUbNE3GRJIEvgnMoG2NW2P2BCgxCK0tIVvHAhVHey0TWA0lDfzGNnFdybTMrZNUzSV1VJeqGG9OyueNQkUYzUFDWIs83J6VHawINpuib8NXgqBJ8dSAwT+ersUQqC9prF+4T52cdKxmx9thKBzIETJncFNBtWqJazAmrXA1dJ8fDazwG8sn1mTdtIRW8TnOdGrpUkZDyHZH0ibMkv0QPmSTOoT9TQrTNJ80A2hZZ60YUKSi5tWJK6Lt4oqrqUmcsOINYpYBPbKkSetVP2VkFz8KAVH51ok+oVTY1+uVE8UhjEzBWSuhmMM3q3IPiuwAx9NBDZ2kXyAZYUD3RW7Nb7BaTY9gmKpGJCzvoFvdKFDaySgAfvUeClt1YJHvRI6r4dKn8TkKTimBpjWNNIqGNf/aB3gPUenDUe9Y7aBiXuevaod0gdPhsMkENQsDrlldpG2QZhekcQSJ7iBWobyJqyC54pil/saElgDw4gE+ClGklGjTtL2/RL3GZMSpxeSt3kC3CnpwE+OCNF63AZ1PYxkzUNFIs12Sthw4q0LcPeJJXDHqEsQj9gtXsU4AcbnFrTmH0MVypalP0nNQ16ZXKyeBAwVc3fR6QoP5xKmiaDaHSeYWwno+99MELT4JimTSnogauCOgwmQt8gknTYByNqGvSu4muIsLFDQyAOvDrzjaq6Mn1C09lAcYVwkSpUnEaktm3NNHxQLdIfho/wJFVqNnasrsEOnJrrbQ7HGnUNqtlqkt66kzcgZZBRtm1pYnSSGal8VzU57LCkngGzKK494s7o1DMoZCMqNAb6Dn5N6xm0SBvbSKk0CEaz5hBoS7dKRT1Dh+S3BC2juKpP8bqTESmVbQKuniFl4eaxT8AFKTMkQ4asMPgEXD2jhenh00jFGZ1Gj41O4lefJYx6RsnZiIUkzMbexBrAWk8Gb+4aa6tnwI593xqbkPjDKAR5XPwgwYpteE3uTE1iXdnY+318W02DpjKn66Cxz5T0MF7cyDXMWdrQNDqqqbriOWsYcbuhhwjlI/brJ0NJT2XY23svyN48czq/RfPBd3NZ8wOv977+ndd7ufrOLwhfX7uv8ZGVAw7d6XFXssddwCZYZc+7AAxbRk+6vv31zbP/vvj5DSLvp7fP373j1/3B16PH2R+p3f3d6ZeSgDdvUL/38vX1G9RCMupJ4zQ9+FDDlNNRJMOfZW5WvUOdpQzUxc5W2cZ+pT3WGjlfWxvJrJclT0crZVdD2lc9W+2xRjRUuNwri9pIFB92tdueHR3YjcJnPVCCFuaLNQG9ebvwfJ+jcduSljwb992/ujKeHT3eZw/JByL6TC+Vf3360APK5VMeUJ7sRcEpxpf8kdl+cIioG+P7yfrx181S2HevyXXa3jX729vnP7ywTN3T80MziOUPzCDee+/x7A6Cr8cbbzabO3Nx5J/nu97FIfXV5v48eXp+vnz6P4iICy0KZW5kc3RyZWFtCmVuZG9iagoxMSAwIG9iago0MTg4CmVuZG9iagoxNiAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDcyID4+CnN0cmVhbQp4nDMzNFQwULAwAhKmZqYK5oZmCimGXGB+LogCCeRwwaQgLDNjQyDL0NwciWViaACSNTRGYhkbmUJlESyQHNj0HK40ALkCF8MKZW5kc3RyZWFtCmVuZG9iagoxNyAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDM0MCA+PgpzdHJlYW0KeJxFksuNHDEMRO8dBRMYQPxL8Yyxp9n8r37UADbQDVL8VBUpdW9Zki4vVWlNadvyR59QE9sqv9fzSvk8k/7v6Vp4VUfK+FQ8JFtFS95Pcn6ZBOcBDj3Xvh+PfT3bIS5mLe6EdWBc1ArcktPATLyamqtvGjCeMfAuYSFkY5+pzjiDQnutQ++WotDMpfY0WDfT0R7rTjhDjX0/Ac94qWSQmUlmDXceGDaIg0pV5ULjVxUZauqYZAX1Sf/MXSAEURazwBzcQy5R2GD3MCZYecigpTQvc321bDyiwZIq+qocy9rqG3HXW2Fn1j4ZYwnFZBZoOXVnzs7JoDDZlG64fInCnivutuHg19TLMdY5k8EzUJQ7tZhetrjWRSPKHOaba0rug/6L5mayWW2VNPeiMYZh8jrBFnlHyeLO6M2GD721sDZTwHJnHG+e3Off4/s8P38BeQF9SwplbmRzdHJlYW0KZW5kb2JqCjE4IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjAxID4+CnN0cmVhbQp4nEVQSZIDIQy78wo/AbzI8J5Mzanz/2tkU5UcGgm8SGogZUrWAXXJBflbo+/vgnp4hvmXndvkxqll4gHJeeQ1/Cwyk1CuIg93QQYrgS2II7FTYBBM5QdWwE2RaOnuL+xtxZyqmC522IUQC69tCrGlrazUsYjG11ipzZ5mk9g57tVo5T0sstkzfO1mDq7YDERz3tJRsUylhBzeGG23GQMEZ4NzUXG3Vfh9rTp/J1h1N4ap8K580cMQFf7aLbx2i/38PeP/A/NAS34KZW5kc3RyZWFtCmVuZG9iagoxOSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDY4ID4+CnN0cmVhbQp4nDMzNFQwUDAyBhKmloYK5oZmCimGXEamlkCBXDBtZmyokMMFVAFngBTlcMGUQ1gQSWNTEySWAUg52KQcrjQAFgwUGwplbmRzdHJlYW0KZW5kb2JqCjIwIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjQ2ID4+CnN0cmVhbQp4nDVRSXIEMQi7+xX6QKoMwsZ+T6dySv5/jWBmDl3Cxmih15yY4MWXGeIaFie+bcQM2J34G3VpefE74qiyi1iJdISdGnwG1V0grUk8V+MzfLIriw1zmI7r6H5P1VGzlq3tIhTNBY0IsW7HTpSHZ5yFKIJqZcJW+SwaO5KVHZfjcl3ChWLiKwf1fpnmm7Y0Isq+vrf/OC613eoJLiomxUu9ZkZPcglpzSseMXBqUlE8b6OiMroyOYvwDhZWXkwnpjquOX+h7+6oqhXXC6tlbH2zk92F3B1s10KjQPJqqiC1foFIrtf1YW+H0q5Vf37dM37+Ac5qWM0KZW5kc3RyZWFtCmVuZG9iagoyMSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDI1NyA+PgpzdHJlYW0KeJw1UDGSAzEI6/0KPpAZgwDb70nmqs3/2xN4UkmLVyApImUKXF6q4ttlacpHh0/j/NuYIc9AHlEXmPavFqvxPUyjma5FhTrEwLHOLR6yVThPE9XNKV81dt2zGQ1xOCaBTnETcG/S0Kkxle7cPCG+XBT8PlnLl0pwoVEUmNyCRrphhGJQk6ARUBNaL0gGPEuwK2idSBrs62QV+xm/Ai6bhd3DM9SivX6b6WEgbKYlZjJcXdDjFdaMEdgQ51kFYXoRgFdSkKxg7j7LunDE+a57NVqUohiMZfAPMHhp/GoQXQloy1mk5UUGt8uUGm9bLn5t0SgWX2gc9PaL8h5//+OVXoIKZW5kc3RyZWFtCmVuZG9iagoyMiAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDI1NSA+PgpzdHJlYW0KeJw1UMltBDEM+7sKNhBAp2XXs0Fem/6/oTQbjAcibJGUmLkh8MKXKlIT6YJvXWEK3YXflWpQdr1X3IKKIUqwFeEGntfy6+AXMSJ2nvpaJmeQBnkUEUce3ucljjbVGm/LbJmihoGvoTIdMe0aBykbJjXTWd2pZPQLUUhORwS55L84qlPFZiOPPdV2cwZl8CZgHGwqreljNei9lJpKFyVTnX8l59mzUqA4SkwCveruTV13g45gXzhzO93t5z6BSQfA2T6h0quzk8t4wx7EePXA06fbD+cmuzF1Ou2gvj2Z2JFPNub3uWECQXetw73HIRnt5R5OJe777/haP39JF1y6CmVuZHN0cmVhbQplbmRvYmoKMjMgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyNDYgPj4Kc3RyZWFtCnicRVE7bsUwDNt9Cl6ggPW1fZ4UnV7vv5ZMAnRIxNgSSTFVjYl0fJmhrLFm49sGT2xv/A6LhJ3CZ1hOWOlpGDttG07iGs6RZfBo9IQTslwjLAQiD1Yj1oHNzfPkW1zpQQ6/q0fpRmgX1BGeiM3xCnGV84uPFeIsisy7UpxO7xM6ikN3J6ilG1NP071m89EMl4NaiNhayZ+FPyNJ/o/aXbekfVFtZEwin4bUltnIVXDKqcpi3Ujmk6az2GkKIplSdN/xxhuzp9YSssV+KhmVspjVnQSzM7okh36MMlV9shYyKnDGOCMirsp8UywL77+7xs8fHkpY9gplbmRzdHJlYW0KZW5kb2JqCjI0IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMzUzID4+CnN0cmVhbQp4nD1Sy40lIRC7dxROoCXqC8TzVnPazf86NszsqSzA7U91VWMgHK+PjVwbFQN/7KmBNx3/HovCW4W/RBvvMlhy2hiw5pWZ4/PYmoS+4NYEMeGVF3we3z8wvO+ryPXLjEml3YjFuxkIPc7UzeYjMlJSdkYvnbfBHWFB634CyEBymm+eYA9MCRfNSs1h+6T0PpIi84OGqIna1Nw8JiV5ZiOQNCLDSWP89jSUKZudelyskGrwVChorEbR40KWOEJlm7WdUv8jpr2ADbJvZm8m7LyNkneaiUQy4ms9bjG2jpy2YjQbY96NOTdzAF3uuNAy9KqYRPtpNdFaT2jDLFtez3ZJ8mApW3sWGowfDVNxzQr8VMvuFtN7Yup1adDMOCBi6TYYw2yftZFIgaRHedX0vp3oF1DdpLHtaDV2OHG7D3Vf1Oo7+e9QVcg2F0bLxqrSji0ajckblwnDb5TP8/UNIeKGVgplbmRzdHJlYW0KZW5kb2JqCjI1IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggNjcgPj4Kc3RyZWFtCnicMzIyUjBQMDMDEoamJgrmhmYKKYZcQL6ZoalCLogBEsrhgklCWCDJHJgqMMMAotjU0BKqBMEygKnI4UoDAJV6FUwKZW5kc3RyZWFtCmVuZG9iagoyNiAwIG9iago8PCAvQkJveCBbIC02NjUgLTMyNSAyMDI5IDEwMzggXSAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDM1Ci9TdWJ0eXBlIC9Gb3JtIC9UeXBlIC9YT2JqZWN0ID4+CnN0cmVhbQp4nOMyNbJQMDY0UsjlMjUDM3LADEsTEAMkh2CBJdMA7sYJtQplbmRzdHJlYW0KZW5kb2JqCjI3IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTgxID4+CnN0cmVhbQp4nE1QQRICMQi79xU8oRDownt0POn/rwYcHQ/bBFLSsBFHtpw+PC8JbLnrmvrVEFryXOrxx5wfWUJiqxhyxqB78Lbg+ulc7JgLqn1Axc04Y3Swec6DbqdaOclKxS92rajyxvZWMgSZcx9RH9SZIdtMgqofQuPL6IbiLB2RNZzZ2pdZOptbO0KcG1BBb5bj4OFiZYO3ZTynYzrJtVhrz+ihAyulCq9By960WWeaP/lcjzeeU0O7CmVuZHN0cmVhbQplbmRvYmoKMjggMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyMzIgPj4Kc3RyZWFtCnicLVBBksRACLrnFX5gq1pFO/2e2Zrb/v+6YHKCxEbAqrZlmfbjbuXHKpf9+sU/Ucf+RLLKyBFt7mnYaZ/La/O9W3iMJnYPfq7EHoZF2WpDuaE1weEXN8gncQajNyfD1uL7Y049biI5NX1sc0EyAGHRcUw6lTt8gstc+LliPVUcMCZz7bxlUORQUee2tx1bBN6eYn44zptiInO5y8pP2d4WGdaPVcspmYMkeUBO8673ORyzAMEKB4PRoQlZhk7AIBujwVI6XRislzwDmFcmmNxyFVMIvVCsR6OguenK4BkPPqW+/1TOVsIKZW5kc3RyZWFtCmVuZG9iagoyOSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDExNCA+PgpzdHJlYW0KeJw1TssNQzEMumcKRvDf8Tyv6ind/1rHai8GYUC4BwhM1VdTkVx48bqU8FmyvfEMegwLhRtBtJU2CzGsCs/iSFgWWAMWNqXmdj/NXKvT7Lt7ZFJet2UjRNsjaQh3KBFiJ5RjxjzrP+v8Vp31/gItliJeCmVuZHN0cmVhbQplbmRvYmoKMzAgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCA0MiA+PgpzdHJlYW0KeJwzMrdQMFCwNAQShkDS0MBAIcWQC8zP5YIK5HAZorBANJRKAwB+zAwSCmVuZHN0cmVhbQplbmRvYmoKMzEgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxNDQgPj4Kc3RyZWFtCnicTY8xsgMhDEP7PYWPYMk2sOfJn19t7t9GhhQpQALNk8cRYW6jdEVOq3D7w7Xf75bCbc+FzB+X6e2G3ByGRSt3o06B9roIFTGNMXYh66iSdVxAyu9Ib6Z/kt3LW71B4wzpLZpbRcdxREljT0w2jSUGbhAT4jGmxcxOSi5pKCW+tnJiJ735c3Z9rv8PwzQxjwplbmRzdHJlYW0KZW5kb2JqCjMyIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggNDA4ID4+CnN0cmVhbQp4nC2TOXIDQQhF8zkFF1BVs/R2Hrkc2fdP/T5yoIGhp/kLaI5hw9Lt5W613GYO+/KHis9pv4/7MV/Hfh6PMM/kt8wHv3nsHHs/fobtYeFhNIjZ4f3E7SS5tq5lhZ1JOan5oL6J8R8rdaJspeUCaB+uTPM7dCLYS2WkxThgTIvQiV8QRagW1dEdg/vv51LYZXtb0GMVIsVqgphhtE6aKByVSWqU0aFiinaVyG6ZMu0sqyPaZXVLsLgyeZMXE92+BvG2GXQJsMdtL0VOET/2J0u+nwEfROuuhAuZk7vBgQlVwUKLTmJSdCkwCxfzY+NcWJfMJTE8rxwW+dGGV/Y32FVICkwophWVHeEyojPfqmjW9M8eJs8KKaMbGhTzep+Q7ds7kEzUCytXD6EYjcyft1X5xtbc7QbfZrYbKVfE1eWgnqGRihee5YmeF5rZrWANpD0K5uiK2D0k7ozde+onPnHKwc6km7c7W/7SNNozKFwogNGrJ/C49hJ+9N6L1au3Q9NTJo100sZRZZ9gCQ25/PljvJ/vP4XjmJkKZW5kc3RyZWFtCmVuZG9iagozMyAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDEzOCA+PgpzdHJlYW0KeJw1j0sSwyAMQ/ecQkfwF+PzpNMVvf+2IplsQJaYZ5E5IYjikaooKXx0cJ5m+B1xrD3e8FHTF1XMRK5GaCMt4JWICFzDXeAzYJ2wpbBSaBcTS4d6wcJA0wgS2no32LwX2EizoSTqEpgcogkfLxJdSX6I4Xl2sU9Kw0lOut7rLn+9v9jj+wdnSysWCmVuZHN0cmVhbQplbmRvYmoKMzQgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxNyA+PgpzdHJlYW0KeJwzMrdQMIDDFEMuABrjAvEKZW5kc3RyZWFtCmVuZG9iagozNSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDE3MCA+PgpzdHJlYW0KeJxFUDkOwzAM2/0KfiCARR+y3pOiU/v/tZRTJItJkLJIm75QYQvHBN3gteJlhWPBA9+SmuFT2AeOBrLtydoTzmLOJNYdhwZbxUrVmCtNu5ohGnqqa2B2LCIiTxtMkeijKkDzNxkWIrJuMhUga8YueLHLzKYP+6+Q+zC77xrV0fXcOoQdscu6I6QrRQ1tqZylHBNyWAUDVILgLOQm7ITrH65vOsv7BzKGPYkKZW5kc3RyZWFtCmVuZG9iagozNiAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDI2NyA+PgpzdHJlYW0KeJw1UUlywzAMu/sVeIK4i+9Jp6fk/9eCzHTGMmhzA6CIxIE2X3EMJY0feSa8js8GB+/HzgLrVGAGl3lS8HrC0GxUiDr6Qjjx9cyH3IKkQZVHeDKY0eYEvTA3WBFrZk2Psdtjhiv83sVQZWYjzrVuxCWWc/mZHm+kOUwK6QmtL3KPxffPIVFSlkrkucMtKPaSsBXC64tn9zDgqveIimpMC6UL6WWuLJIoDlSR9UqniDhEaiPnoCRNd+Ia5FyVtGBWBCcu6pCfyGmHd8JplNNzt1gizJxaO8YkV4r2uyb1irVwbg+MnbomqdF81uqh9ayV25Q2GaFdo0GSog/1hM71vv7v+f38/gErHWDYCmVuZHN0cmVhbQplbmRvYmoKMzcgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxODkgPj4Kc3RyZWFtCnicTVDBbQAxCPtnCi9QKUAgME+rvq77f2uSq9QHMjJgA+6BiVj4EMHKBZfCl4w1m/85uAPPsHBIwmSeVl1y8HPoy0iSYY87grRoQTZkFkxRAZ9k0xCJvZCFYIM4yVZmD5cQrwO1m77LPENc/2Vq8maSbWeMnqSXZRuHHV2hC3WkFDzr7rknx4+TXifSFGFi3JNVM7vdxr9w2rYeMUuiVReKp4bCeJIwGvsZXYl3zb8/3mw2nnc+4/sX9s1EjAplbmRzdHJlYW0KZW5kb2JqCjM4IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjcyID4+CnN0cmVhbQp4nDVRS24FMQjbzyl8gUr8Sc4zVXe9/7Ym6ZNmBAnYGCezIAjDlypSFlo3vvXhjafjd5LwZolfohYyG++zmMuGElXBGDz3FLQ3mp1mfris88T3cb03Hs2o8C7UTlZCguhNGQtF+mBHMWelCrlZWVRXgdiGNGMlhcKWHM3BWRNH8VQWrIghzkTZZdS3D1tik943kiiqdBlhreC61seEETzxfUxupsnfuatRFe/JoqZjmukM/5+r/vFsMvM8rij30R70OpQCLmrOheWMqqNAT+KxpjrG3PYxZgiCpyGceIdNFtQ9HRkXk2swQ1JWcOWMKA65kcPd7w1NKeOj8cHyPIDS8Dxb0tQWfFZ5n58/9JZlSwplbmRzdHJlYW0KZW5kb2JqCjE0IDAgb2JqCjw8IC9CYXNlRm9udCAvQXJpYWxNVCAvQ2hhclByb2NzIDE1IDAgUgovRW5jb2RpbmcgPDwKL0RpZmZlcmVuY2VzIFsgMzIgL3NwYWNlIDQ2IC9wZXJpb2QgNDggL3plcm8gL29uZSAvdHdvIDUzIC9maXZlIDU1IC9zZXZlbiA3MCAvRiAvRyA4MAovUCA4NCAvVCA5OSAvYyAvZCAvZSAxMDMgL2cgMTA1IC9pIDExMCAvbiAvbyAxMTQgL3IgL3MgL3QgL3UgXQovVHlwZSAvRW5jb2RpbmcgPj4KL0ZpcnN0Q2hhciAwIC9Gb250QkJveCBbIC02NjUgLTMyNSAyMDI5IDEwMzggXSAvRm9udERlc2NyaXB0b3IgMTMgMCBSCi9Gb250TWF0cml4IFsgMC4wMDEgMCAwIDAuMDAxIDAgMCBdIC9MYXN0Q2hhciAyNTUgL05hbWUgL0FyaWFsTVQKL1N1YnR5cGUgL1R5cGUzIC9UeXBlIC9Gb250IC9XaWR0aHMgMTIgMCBSID4+CmVuZG9iagoxMyAwIG9iago8PCAvQXNjZW50IDkwNiAvQ2FwSGVpZ2h0IDcxNiAvRGVzY2VudCAtMjEyIC9GbGFncyAzMgovRm9udEJCb3ggWyAtNjY1IC0zMjUgMjAyOSAxMDM4IF0gL0ZvbnROYW1lIC9BcmlhbE1UIC9JdGFsaWNBbmdsZSAwCi9NYXhXaWR0aCAxMDE1IC9TdGVtViAwIC9UeXBlIC9Gb250RGVzY3JpcHRvciAvWEhlaWdodCA1MTkgPj4KZW5kb2JqCjEyIDAgb2JqClsgNzUwIDc1MCA3NTAgNzUwIDc1MCA3NTAgNzUwIDc1MCA3NTAgNzUwIDc1MCA3NTAgNzUwIDc1MCA3NTAgNzUwIDc1MCA3NTAKNzUwIDc1MCA3NTAgNzUwIDc1MCA3NTAgNzUwIDc1MCA3NTAgNzUwIDc1MCA3NTAgNzUwIDc1MCAyNzggMjc4IDM1NSA1NTYgNTU2Cjg4OSA2NjcgMTkxIDMzMyAzMzMgMzg5IDU4NCAyNzggMzMzIDI3OCAyNzggNTU2IDU1NiA1NTYgNTU2IDU1NiA1NTYgNTU2IDU1Ngo1NTYgNTU2IDI3OCAyNzggNTg0IDU4NCA1ODQgNTU2IDEwMTUgNjY3IDY2NyA3MjIgNzIyIDY2NyA2MTEgNzc4IDcyMiAyNzgKNTAwIDY2NyA1NTYgODMzIDcyMiA3NzggNjY3IDc3OCA3MjIgNjY3IDYxMSA3MjIgNjY3IDk0NCA2NjcgNjY3IDYxMSAyNzggMjc4CjI3OCA0NjkgNTU2IDMzMyA1NTYgNTU2IDUwMCA1NTYgNTU2IDI3OCA1NTYgNTU2IDIyMiAyMjIgNTAwIDIyMiA4MzMgNTU2IDU1Ngo1NTYgNTU2IDMzMyA1MDAgMjc4IDU1NiA1MDAgNzIyIDUwMCA1MDAgNTAwIDMzNCAyNjAgMzM0IDU4NCA3NTAgNTU2IDc1MCAyMjIKNTU2IDMzMyAxMDAwIDU1NiA1NTYgMzMzIDEwMDAgNjY3IDMzMyAxMDAwIDc1MCA2MTEgNzUwIDc1MCAyMjIgMjIyIDMzMyAzMzMKMzUwIDU1NiAxMDAwIDMzMyAxMDAwIDUwMCAzMzMgOTQ0IDc1MCA1MDAgNjY3IDI3OCAzMzMgNTU2IDU1NiA1NTYgNTU2IDI2MAo1NTYgMzMzIDczNyAzNzAgNTU2IDU4NCAzMzMgNzM3IDU1MiA0MDAgNTQ5IDMzMyAzMzMgMzMzIDU3NiA1MzcgMjc4IDMzMyAzMzMKMzY1IDU1NiA4MzQgODM0IDgzNCA2MTEgNjY3IDY2NyA2NjcgNjY3IDY2NyA2NjcgMTAwMCA3MjIgNjY3IDY2NyA2NjcgNjY3CjI3OCAyNzggMjc4IDI3OCA3MjIgNzIyIDc3OCA3NzggNzc4IDc3OCA3NzggNTg0IDc3OCA3MjIgNzIyIDcyMiA3MjIgNjY3IDY2Nwo2MTEgNTU2IDU1NiA1NTYgNTU2IDU1NiA1NTYgODg5IDUwMCA1NTYgNTU2IDU1NiA1NTYgMjc4IDI3OCAyNzggMjc4IDU1NiA1NTYKNTU2IDU1NiA1NTYgNTU2IDU1NiA1NDkgNjExIDU1NiA1NTYgNTU2IDU1NiA1MDAgNTU2IDUwMCBdCmVuZG9iagoxNSAwIG9iago8PCAvRiAxNiAwIFIgL0cgMTcgMCBSIC9QIDE4IDAgUiAvVCAxOSAwIFIgL2MgMjAgMCBSIC9kIDIxIDAgUiAvZSAyMiAwIFIKL2ZpdmUgMjMgMCBSIC9nIDI0IDAgUiAvaSAyNSAwIFIgL24gMjcgMCBSIC9vIDI4IDAgUiAvb25lIDI5IDAgUgovcGVyaW9kIDMwIDAgUiAvciAzMSAwIFIgL3MgMzIgMCBSIC9zZXZlbiAzMyAwIFIgL3NwYWNlIDM0IDAgUiAvdCAzNSAwIFIKL3R3byAzNiAwIFIgL3UgMzcgMCBSIC96ZXJvIDM4IDAgUiA+PgplbmRvYmoKMyAwIG9iago8PCAvRjEgMTQgMCBSID4+CmVuZG9iago0IDAgb2JqCjw8IC9BMSA8PCAvQ0EgMCAvVHlwZSAvRXh0R1N0YXRlIC9jYSAxID4+Ci9BMiA8PCAvQ0EgMSAvVHlwZSAvRXh0R1N0YXRlIC9jYSAxID4+Ci9BMyA8PCAvQ0EgMC44IC9UeXBlIC9FeHRHU3RhdGUgL2NhIDAuOCA+PiA+PgplbmRvYmoKNSAwIG9iago8PCA+PgplbmRvYmoKNiAwIG9iago8PCA+PgplbmRvYmoKNyAwIG9iago8PCAvRjEtQXJpYWwtbWludXMgMjYgMCBSID4+CmVuZG9iagoyIDAgb2JqCjw8IC9Db3VudCAxIC9LaWRzIFsgMTAgMCBSIF0gL1R5cGUgL1BhZ2VzID4+CmVuZG9iagozOSAwIG9iago8PCAvQ3JlYXRpb25EYXRlIChEOjIwMjIwNzA1MjAwNTA5KzAyJzAwJykKL0NyZWF0b3IgKE1hdHBsb3RsaWIgdjMuMy4yLCBodHRwczovL21hdHBsb3RsaWIub3JnKQovUHJvZHVjZXIgKE1hdHBsb3RsaWIgcGRmIGJhY2tlbmQgdjMuMy4yKSA+PgplbmRvYmoKeHJlZgowIDQwCjAwMDAwMDAwMDAgNjU1MzUgZiAKMDAwMDAwMDAxNiAwMDAwMCBuIAowMDAwMDEyOTIzIDAwMDAwIG4gCjAwMDAwMTI2NjMgMDAwMDAgbiAKMDAwMDAxMjY5NSAwMDAwMCBuIAowMDAwMDEyODM3IDAwMDAwIG4gCjAwMDAwMTI4NTggMDAwMDAgbiAKMDAwMDAxMjg3OSAwMDAwMCBuIAowMDAwMDAwMDY1IDAwMDAwIG4gCjAwMDAwMDAzOTggMDAwMDAgbiAKMDAwMDAwMDIwOCAwMDAwMCBuIAowMDAwMDA0NjYxIDAwMDAwIG4gCjAwMDAwMTEzNDcgMDAwMDAgbiAKMDAwMDAxMTE0NyAwMDAwMCBuIAowMDAwMDEwNzI5IDAwMDAwIG4gCjAwMDAwMTIzOTggMDAwMDAgbiAKMDAwMDAwNDY4MiAwMDAwMCBuIAowMDAwMDA0ODI2IDAwMDAwIG4gCjAwMDAwMDUyMzkgMDAwMDAgbiAKMDAwMDAwNTUxMyAwMDAwMCBuIAowMDAwMDA1NjUzIDAwMDAwIG4gCjAwMDAwMDU5NzIgMDAwMDAgbiAKMDAwMDAwNjMwMiAwMDAwMCBuIAowMDAwMDA2NjMwIDAwMDAwIG4gCjAwMDAwMDY5NDkgMDAwMDAgbiAKMDAwMDAwNzM3NSAwMDAwMCBuIAowMDAwMDA3NTE0IDAwMDAwIG4gCjAwMDAwMDc2ODEgMDAwMDAgbiAKMDAwMDAwNzkzNSAwMDAwMCBuIAowMDAwMDA4MjQwIDAwMDAwIG4gCjAwMDAwMDg0MjcgMDAwMDAgbiAKMDAwMDAwODU0MSAwMDAwMCBuIAowMDAwMDA4NzU4IDAwMDAwIG4gCjAwMDAwMDkyMzkgMDAwMDAgbiAKMDAwMDAwOTQ1MCAwMDAwMCBuIAowMDAwMDA5NTM5IDAwMDAwIG4gCjAwMDAwMDk3ODIgMDAwMDAgbiAKMDAwMDAxMDEyMiAwMDAwMCBuIAowMDAwMDEwMzg0IDAwMDAwIG4gCjAwMDAwMTI5ODMgMDAwMDAgbiAKdHJhaWxlcgo8PCAvSW5mbyAzOSAwIFIgL1Jvb3QgMSAwIFIgL1NpemUgNDAgPj4Kc3RhcnR4cmVmCjEzMTQwCiUlRU9GCg==\n", "image/svg+xml": [ "\n", "\n", "\n", "\n", " \n", " \n", " \n", " \n", " 2022-07-05T20:05:09.705902\n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.3.2, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n" ], "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "model_bd = trainer.bind_model()\n", "x = np.linspace(-2, 2, 1000)[:,None]\n", "y_pred = model_bd(x)\n", "plt.plot(x, y_pred, label='Prediction')\n", "plt.plot(x, target_function(x), '--', label='GT')\n", "plt.title('Function regression')\n", "plt.legend()\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "9db890bb", "metadata": {}, "source": [ "## Example 2: CIFAR10 classification\n", "\n", "As a second example, let's consider image classification on CIFAR10. We have done it before in [Tutorial 5](https://uvadlc-notebooks.readthedocs.io/en/latest/tutorial_notebooks/JAX/tutorial5/Inception_ResNet_DenseNet.html), but here, we want to showcase the flexibility of the Trainer module. For that, we will consider a more complicated setting in Flax: a model with both Batch Normalization and Dropout. However, with the Trainer module, this reduces to a simpler code again." ] }, { "cell_type": "markdown", "id": "f8a54805", "metadata": {}, "source": [ "### Dataset\n", "\n", "First, let's load our dataset again. This is the same data loading as used in [Tutorial 5](https://uvadlc-notebooks.readthedocs.io/en/latest/tutorial_notebooks/JAX/tutorial5/Inception_ResNet_DenseNet.html) and [Tutorial 15](https://uvadlc-notebooks.readthedocs.io/en/latest/tutorial_notebooks/JAX/tutorial15/Vision_Transformer.html), but only considers flipping as regularization technique since we work with simple MLPs here." ] }, { "cell_type": "code", "execution_count": 14, "id": "fdfb1235", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Files already downloaded and verified\n", "Files already downloaded and verified\n", "Files already downloaded and verified\n" ] } ], "source": [ "from torchvision.datasets import CIFAR10\n", "from torchvision import transforms\n", "\n", "# Transformations applied on each image => bring them into a numpy array\n", "DATA_MEANS = np.array([0.49139968, 0.48215841, 0.44653091])\n", "DATA_STD = np.array([0.24703223, 0.24348513, 0.26158784])\n", "def image_to_numpy(img):\n", " img = np.array(img, dtype=np.float32)\n", " img = (img / 255. - DATA_MEANS) / DATA_STD\n", " return img\n", "\n", "\n", "test_transform = image_to_numpy\n", "# For training, we add some augmentation. Networks are too powerful and would overfit.\n", "train_transform = transforms.Compose([transforms.RandomHorizontalFlip(),\n", " image_to_numpy])\n", "# Loading the training dataset. We need to split it into a training and validation part\n", "# We need to do a little trick because the validation set should not use the augmentation.\n", "train_dataset = CIFAR10(root=DATASET_PATH, train=True, transform=train_transform, download=True)\n", "val_dataset = CIFAR10(root=DATASET_PATH, train=True, transform=test_transform, download=True)\n", "train_set, _ = data.random_split(train_dataset, [45000, 5000], generator=torch.Generator().manual_seed(42))\n", "_, val_set = data.random_split(val_dataset, [45000, 5000], generator=torch.Generator().manual_seed(42))\n", "\n", "# Loading the test set\n", "test_set = CIFAR10(root=DATASET_PATH, train=False, transform=test_transform, download=True)\n", "\n", "train_loader, val_loader, test_loader = create_data_loaders(train_set, val_set, test_set,\n", " train=[True, False, False],\n", " batch_size=256)" ] }, { "cell_type": "markdown", "id": "b83f6b66", "metadata": {}, "source": [ "### Model\n", "\n", "The model definition is again relatively simple. We repeat a series of Dropout $\\to$ Linear $\\to$ BatchNorm $\\to$ Swish blocks, with a final Dropout and Linear layer at the end." ] }, { "cell_type": "code", "execution_count": 15, "id": "fc749501", "metadata": {}, "outputs": [], "source": [ "class MLPClassifier(nn.Module):\n", " hidden_dims : Sequence[int]\n", " num_classes : int\n", " dropout_prob : float = 0.0\n", " \n", " @nn.compact\n", " def __call__(self, x, train=True):\n", " x = x.reshape(x.shape[0], -1)\n", " for dims in self.hidden_dims:\n", " x = nn.Dropout(self.dropout_prob)(x, deterministic=not train)\n", " x = nn.Dense(dims)(x)\n", " x = nn.BatchNorm()(x, use_running_average=not train)\n", " x = nn.swish(x)\n", " x = nn.Dropout(self.dropout_prob)(x, deterministic=not train)\n", " x = nn.Dense(self.num_classes)(x)\n", " return x" ] }, { "cell_type": "markdown", "id": "7cf22ac7", "metadata": {}, "source": [ "### Trainer\n", "\n", "For the Trainer module, we again define our model hyperparameters in the init function, and write our own training and evaluation steps in `create_functions`. In these functions, we take care of the mutable batch statistics and the PRNG state for dropout. Note that since both parts are integrated in the `TrainState`, we do not need to alternate the training or validation step signature, and it is sufficient to pass the state and batch to the functions. Additionally, we overwrite the model call during initialization (`run_model_init`) and tabulate function (`print_tabulate`). And that's it! Overall, we didn't need to make many changes, showing that the trainer module is flexible enough to support a variety of layers. For now, we can ignore the `trial` object and come back to it later when discussing automated hyperparameter tuning." ] }, { "cell_type": "code", "execution_count": 16, "id": "6a945b8e", "metadata": {}, "outputs": [], "source": [ "class MLPClassTrainer(TrainerModule):\n", " \n", " def __init__(self,\n", " hidden_dims : Sequence[int],\n", " num_classes : int,\n", " dropout_prob : float,\n", " trial : Any = None,\n", " **kwargs):\n", " super().__init__(model_class=MLPClassifier,\n", " model_hparams={\n", " 'hidden_dims': hidden_dims,\n", " 'num_classes': num_classes,\n", " 'dropout_prob': dropout_prob\n", " },\n", " **kwargs)\n", " self.trial = trial\n", " \n", " def create_functions(self):\n", " def loss_function(params, batch_stats, rng, batch, train):\n", " imgs, labels = batch\n", " rng, dropout_rng = random.split(rng)\n", " output = self.model.apply({'params': params, 'batch_stats': batch_stats},\n", " imgs,\n", " train=train,\n", " rngs={'dropout': dropout_rng},\n", " mutable=['batch_stats'] if train else False)\n", " logits, new_model_state = output if train else (output, None)\n", " loss = optax.softmax_cross_entropy_with_integer_labels(logits, labels).mean()\n", " acc = (logits.argmax(axis=-1) == labels).mean()\n", " return loss, (rng, new_model_state, acc)\n", " \n", " def train_step(state, batch):\n", " loss_fn = lambda params: loss_function(params, state.batch_stats, state.rng, batch, train=True)\n", " ret, grads = jax.value_and_grad(loss_fn, has_aux=True)(state.params)\n", " loss, rng, new_model_state, acc = ret[0], *ret[1]\n", " state = state.apply_gradients(grads=grads, batch_stats=new_model_state['batch_stats'], rng=rng)\n", " metrics = {'loss': loss, 'acc': acc}\n", " return state, metrics\n", " \n", " def eval_step(state, batch):\n", " _, (_, _, acc) = loss_function(state.params, state.batch_stats, state.rng, batch, train=False)\n", " return {'acc': acc}\n", " \n", " return train_step, eval_step\n", " \n", " def run_model_init(self, exmp_input, init_rng):\n", " imgs, _ = exmp_input\n", " init_rng, dropout_rng = random.split(init_rng)\n", " return self.model.init({'params': init_rng, 'dropout': dropout_rng}, x=imgs, train=True)\n", " \n", " def print_tabulate(self, exmp_input):\n", " imgs, _ = exmp_input\n", " print(self.model.tabulate(rngs={'params': random.PRNGKey(0), 'dropout': random.PRNGKey(0)}, x=imgs, train=True))\n", " \n", " def on_validation_epoch_end(self, epoch_idx, eval_metrics, val_loader):\n", " if self.trial:\n", " self.trial.report(eval_metrics['val/acc'], step=epoch_idx)\n", " if self.trial.should_prune():\n", " raise optuna.exceptions.TrialPruned()" ] }, { "cell_type": "markdown", "id": "101d2ff3", "metadata": {}, "source": [ "### Training\n", "\n", "With the Trainer fully defined, we can again start training. Let's pick some reasonable hyperparameters, and look at the layers created by the model:" ] }, { "cell_type": "code", "execution_count": 17, "id": "cb3a3899", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
                                   MLPClassifier Summary                                    \n",
       "┏━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃ path           outputs                  batch_stats         params                    ┃\n",
       "┡━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│ Inputs        │ train: True             │                    │                           │\n",
       "│               │ x: float64[256,32,32,3] │                    │                           │\n",
       "├───────────────┼─────────────────────────┼────────────────────┼───────────────────────────┤\n",
       "│ BatchNorm_0   │ float32[256,512]        │ mean: float32[512] │ bias: float32[512]        │\n",
       "│               │                         │ var: float32[512]  │ scale: float32[512]       │\n",
       "│               │                         │                    │                           │\n",
       "│               │                         │ 1,024 (4.1 KB)1,024 (4.1 KB)            │\n",
       "├───────────────┼─────────────────────────┼────────────────────┼───────────────────────────┤\n",
       "│ BatchNorm_1   │ float32[256,512]        │ mean: float32[512] │ bias: float32[512]        │\n",
       "│               │                         │ var: float32[512]  │ scale: float32[512]       │\n",
       "│               │                         │                    │                           │\n",
       "│               │                         │ 1,024 (4.1 KB)1,024 (4.1 KB)            │\n",
       "├───────────────┼─────────────────────────┼────────────────────┼───────────────────────────┤\n",
       "│ Dense_0       │ float32[256,512]        │                    │ bias: float32[512]        │\n",
       "│               │                         │                    │ kernel: float32[3072,512] │\n",
       "│               │                         │                    │                           │\n",
       "│               │                         │                    │ 1,573,376 (6.3 MB)        │\n",
       "├───────────────┼─────────────────────────┼────────────────────┼───────────────────────────┤\n",
       "│ Dense_1       │ float32[256,512]        │                    │ bias: float32[512]        │\n",
       "│               │                         │                    │ kernel: float32[512,512]  │\n",
       "│               │                         │                    │                           │\n",
       "│               │                         │                    │ 262,656 (1.1 MB)          │\n",
       "├───────────────┼─────────────────────────┼────────────────────┼───────────────────────────┤\n",
       "│ Dense_2       │ float32[256,10]         │                    │ bias: float32[10]         │\n",
       "│               │                         │                    │ kernel: float32[512,10]   │\n",
       "│               │                         │                    │                           │\n",
       "│               │                         │                    │ 5,130 (20.5 KB)           │\n",
       "├───────────────┼─────────────────────────┼────────────────────┼───────────────────────────┤\n",
       "│ Dropout_0     │ float32[256,3072]       │                    │                           │\n",
       "├───────────────┼─────────────────────────┼────────────────────┼───────────────────────────┤\n",
       "│ Dropout_1     │ float32[256,512]        │                    │                           │\n",
       "├───────────────┼─────────────────────────┼────────────────────┼───────────────────────────┤\n",
       "│ Dropout_2     │ float32[256,512]        │                    │                           │\n",
       "├───────────────┼─────────────────────────┼────────────────────┼───────────────────────────┤\n",
       "│ MLPClassifier │ float32[256,10]         │                    │                           │\n",
       "├───────────────┼─────────────────────────┼────────────────────┼───────────────────────────┤\n",
       "│                                  Total  2,048 (8.2 KB)      1,843,210 (7.4 MB)        │\n",
       "└───────────────┴─────────────────────────┴────────────────────┴───────────────────────────┘\n",
       "                                                                                            \n",
       "                            Total Parameters: 1,845,258 (7.4 MB)                            \n",
       "
\n" ], "text/plain": [ "\u001b[3m MLPClassifier Summary \u001b[0m\n", "┏━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n", "┃\u001b[1m \u001b[0m\u001b[1mpath \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1moutputs \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mbatch_stats \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mparams \u001b[0m\u001b[1m \u001b[0m┃\n", "┡━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n", "│ Inputs │ train: True │ │ │\n", "│ │ x: \u001b[2mfloat64\u001b[0m[256,32,32,3] │ │ │\n", "├───────────────┼─────────────────────────┼────────────────────┼───────────────────────────┤\n", "│ BatchNorm_0 │ \u001b[2mfloat32\u001b[0m[256,512] │ mean: \u001b[2mfloat32\u001b[0m[512] │ bias: \u001b[2mfloat32\u001b[0m[512] │\n", "│ │ │ var: \u001b[2mfloat32\u001b[0m[512] │ scale: \u001b[2mfloat32\u001b[0m[512] │\n", "│ │ │ │ │\n", "│ │ │ \u001b[1m1,024 \u001b[0m\u001b[1;2m(4.1 KB)\u001b[0m │ \u001b[1m1,024 \u001b[0m\u001b[1;2m(4.1 KB)\u001b[0m │\n", "├───────────────┼─────────────────────────┼────────────────────┼───────────────────────────┤\n", "│ BatchNorm_1 │ \u001b[2mfloat32\u001b[0m[256,512] │ mean: \u001b[2mfloat32\u001b[0m[512] │ bias: \u001b[2mfloat32\u001b[0m[512] │\n", "│ │ │ var: \u001b[2mfloat32\u001b[0m[512] │ scale: \u001b[2mfloat32\u001b[0m[512] │\n", "│ │ │ │ │\n", "│ │ │ \u001b[1m1,024 \u001b[0m\u001b[1;2m(4.1 KB)\u001b[0m │ \u001b[1m1,024 \u001b[0m\u001b[1;2m(4.1 KB)\u001b[0m │\n", "├───────────────┼─────────────────────────┼────────────────────┼───────────────────────────┤\n", "│ Dense_0 │ \u001b[2mfloat32\u001b[0m[256,512] │ │ bias: \u001b[2mfloat32\u001b[0m[512] │\n", "│ │ │ │ kernel: \u001b[2mfloat32\u001b[0m[3072,512] │\n", "│ │ │ │ │\n", "│ │ │ │ \u001b[1m1,573,376 \u001b[0m\u001b[1;2m(6.3 MB)\u001b[0m │\n", "├───────────────┼─────────────────────────┼────────────────────┼───────────────────────────┤\n", "│ Dense_1 │ \u001b[2mfloat32\u001b[0m[256,512] │ │ bias: \u001b[2mfloat32\u001b[0m[512] │\n", "│ │ │ │ kernel: \u001b[2mfloat32\u001b[0m[512,512] │\n", "│ │ │ │ │\n", "│ │ │ │ \u001b[1m262,656 \u001b[0m\u001b[1;2m(1.1 MB)\u001b[0m │\n", "├───────────────┼─────────────────────────┼────────────────────┼───────────────────────────┤\n", "│ Dense_2 │ \u001b[2mfloat32\u001b[0m[256,10] │ │ bias: \u001b[2mfloat32\u001b[0m[10] │\n", "│ │ │ │ kernel: \u001b[2mfloat32\u001b[0m[512,10] │\n", "│ │ │ │ │\n", "│ │ │ │ \u001b[1m5,130 \u001b[0m\u001b[1;2m(20.5 KB)\u001b[0m │\n", "├───────────────┼─────────────────────────┼────────────────────┼───────────────────────────┤\n", "│ Dropout_0 │ \u001b[2mfloat32\u001b[0m[256,3072] │ │ │\n", "├───────────────┼─────────────────────────┼────────────────────┼───────────────────────────┤\n", "│ Dropout_1 │ \u001b[2mfloat32\u001b[0m[256,512] │ │ │\n", "├───────────────┼─────────────────────────┼────────────────────┼───────────────────────────┤\n", "│ Dropout_2 │ \u001b[2mfloat32\u001b[0m[256,512] │ │ │\n", "├───────────────┼─────────────────────────┼────────────────────┼───────────────────────────┤\n", "│ MLPClassifier │ \u001b[2mfloat32\u001b[0m[256,10] │ │ │\n", "├───────────────┼─────────────────────────┼────────────────────┼───────────────────────────┤\n", "│\u001b[1m \u001b[0m\u001b[1m \u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m Total\u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m2,048 \u001b[0m\u001b[1;2m(8.2 KB)\u001b[0m\u001b[1m \u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m1,843,210 \u001b[0m\u001b[1;2m(7.4 MB)\u001b[0m\u001b[1m \u001b[0m\u001b[1m \u001b[0m│\n", "└───────────────┴─────────────────────────┴────────────────────┴───────────────────────────┘\n", "\u001b[1m \u001b[0m\n", "\u001b[1m Total Parameters: 1,845,258 \u001b[0m\u001b[1;2m(7.4 MB)\u001b[0m\u001b[1m \u001b[0m\n" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "\n", "\n" ] } ], "source": [ "trainer = MLPClassTrainer(hidden_dims=[512, 512],\n", " num_classes=10,\n", " dropout_prob=0.4,\n", " optimizer_hparams={\n", " 'weight_decay': 2e-4,\n", " 'lr': 1e-3\n", " },\n", " logger_params={\n", " 'base_log_dir': CHECKPOINT_PATH\n", " },\n", " exmp_input=next(iter(train_loader)),\n", " check_val_every_n_epoch=5)" ] }, { "cell_type": "markdown", "id": "64f59fac", "metadata": {}, "source": [ "One interesting observation here is that the MLP has way more parameters than any of the CNNs in [Tutorial 5](https://uvadlc-notebooks.readthedocs.io/en/latest/tutorial_notebooks/JAX/tutorial5/Inception_ResNet_DenseNet.html), but yet significantly underperforms the models. Although to really see the performance, let's train the model again with the simple call from before:" ] }, { "cell_type": "code", "execution_count": 18, "id": "ddeb5a9e", "metadata": {}, "outputs": [], "source": [ "metrics = trainer.train_model(train_loader, \n", " val_loader, \n", " test_loader=test_loader, \n", " num_epochs=50)" ] }, { "cell_type": "code", "execution_count": 19, "id": "dfc21f0c", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Validation accuracy: 60.22%\n", "Test accuracy: 60.05%\n" ] } ], "source": [ "print(f'Validation accuracy: {metrics[\"val/acc\"]:4.2%}')\n", "print(f'Test accuracy: {metrics[\"test/acc\"]:4.2%}')" ] }, { "cell_type": "markdown", "id": "b512c20a", "metadata": {}, "source": [ "The MLP achieves decent accuracy, but already for models like this, we have several hyperparameters to tune including learning rate, weight decay, dropout rate. Which should we choose? While we could use intuition to get a reasonable guess, it is unlikely that we hit the best hyperparameter set. In order to find a very strong hyperparameter set, a good practice is to use automatic hyperparameter tuning, which we shortly review next to showcase the flexibility of the Trainer module." ] }, { "cell_type": "markdown", "id": "72a5e760", "metadata": {}, "source": [ "## Automatic hyperparameter tuning with Optuna\n", "\n", "Automatic hyperparameter tuner have the goal to efficiently identify sets of hyperparameters that achieve the best performance. Thereby, the key question is how can we search the hyperparameter space efficiently, since we don't have infinite compute. [Optuna](https://optuna.readthedocs.io/en/stable/) is a library that helps you setup this search with minimal code overhead and perform automatic hyperparameter tuning. Before getting started with Optuna, let's import the library and download a pre-executed hyperparameter search as an example." ] }, { "cell_type": "code", "execution_count": 20, "id": "b89f729b", "metadata": {}, "outputs": [], "source": [ "try:\n", " import optuna\n", "except ModuleNotFoundError:\n", " !pip install --quiet --upgrade optuna pyplot\n", " import optuna" ] }, { "cell_type": "code", "execution_count": 21, "id": "8029b654", "metadata": {}, "outputs": [], "source": [ "import urllib.request\n", "from urllib.error import HTTPError\n", "# Github URL where saved models are stored for this tutorial\n", "base_url = \"https://raw.githubusercontent.com/phlippe/saved_models/main/guide4/\"\n", "# Files to download\n", "pretrained_files = [\"optuna_hparam_search.db\", \"MLPClassifier/version_16/checkpoint_150\", \"MLPClassifier/version_16/hparams.json\"]\n", "# Create checkpoint path if it doesn't exist yet\n", "os.makedirs(CHECKPOINT_PATH, exist_ok=True)\n", "\n", "# For each file, check whether it already exists. If not, try downloading it.\n", "for file_name in pretrained_files:\n", " file_path = os.path.join(CHECKPOINT_PATH, file_name)\n", " if not os.path.isfile(file_path):\n", " file_url = base_url + file_name\n", " print(f\"Downloading {file_url}...\")\n", " try:\n", " urllib.request.urlretrieve(file_url, file_path)\n", " except HTTPError as e:\n", " print(\"Something went wrong. Please contact the author with the full output including the following error:\\n\", e)" ] }, { "cell_type": "markdown", "id": "98cc1ec8", "metadata": {}, "source": [ "### Defining objective and hyperparameters\n", "\n", "The main part a user has to specify in Optuna is intuitively the the objective to optimize, and the hyperparameters over which we want to optimize. In our case, the objective is to optimize the validation accuracy of the MLP. Note that we do not use the test set here, since hyperparameter searches should only be done on the validation set, not the \"unseen\" test set! The function below, `objective(trial)`, creates a MLP with our trainer module and trains it for max. 200 epochs. The input argument, `trial`, is thereby an object which characterizes the current run. This includes, for example, the hyperparameters we want to optimize. In order to add a hyperparameter to our optimization set, we can simply call `trial.suggest_float` for continuous values and `trial.suggest_categorical` for categorical values (e.g. which optimizer to use). For the CIFAR10 classification, we consider the following three hyperparameters: dropout rate, weight decay, and the learning rate, which we define below. Finally, we return the best validation accuracy which will be used by Optuna to guide the next pick of hyperparameters." ] }, { "cell_type": "code", "execution_count": 22, "id": "49e60a72", "metadata": {}, "outputs": [], "source": [ "def objective(trial):\n", " my_train_loader, my_val_loader = create_data_loaders(train_set, val_set,\n", " train=[True, False],\n", " batch_size=256)\n", " trainer = MLPClassTrainer(hidden_dims=[512, 512],\n", " num_classes=10,\n", " dropout_prob=trial.suggest_float('dropout_prob', 0, 0.6),\n", " optimizer_hparams={\n", " 'weight_decay': trial.suggest_float('weight_decay', 1e-6, 1e-2, log=True),\n", " 'lr': trial.suggest_float('lr', 1e-4, 1e-2, log=True)\n", " },\n", " logger_params={\n", " 'base_log_dir': CHECKPOINT_PATH\n", " },\n", " exmp_input=next(iter(my_train_loader)),\n", " check_val_every_n_epoch=5,\n", " trial=trial)\n", " metrics = trainer.train_model(my_train_loader,\n", " my_val_loader,\n", " num_epochs=200)\n", " del trainer\n", " del my_train_loader, my_val_loader\n", " return metrics['val/acc']" ] }, { "cell_type": "markdown", "id": "ab3528c1", "metadata": {}, "source": [ "### Running hyperparameter study\n", "\n", "To run the hyperparameter search, we create a `Study` in Optuna. A study implements the search logic and summarizes the data/logs of all executed experiments. By default, Optuna uses the [Tree-Structured Parzen Estimator](https://optuna.readthedocs.io/en/stable/reference/generated/optuna.samplers.TPESampler.html#optuna.samplers.TPESampler) algorithm, using Gaussian Mixture Models to estimate the performance surface of they hyperparameters. For more information, check out the [documentation](https://optuna.readthedocs.io/en/stable/reference/generated/optuna.samplers.TPESampler.html#optuna.samplers.TPESampler). The studies are usually stored in a database format, but can be easily accessed via the Python interface of Optuna. Let's run the hyperparameter search for up to 25 models:" ] }, { "cell_type": "code", "execution_count": 23, "id": "bb80fe0d", "metadata": { "scrolled": false }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "\u001b[32m[I 2022-07-05 20:06:47,966]\u001b[0m Using an existing study with name 'mlp_cifar10' instead of creating a new one.\u001b[0m\n" ] } ], "source": [ "study = optuna.create_study(\n", " study_name='mlp_cifar10',\n", " storage=f'sqlite:///{CHECKPOINT_PATH}/optuna_hparam_search.db',\n", " direction='maximize',\n", " pruner=optuna.pruners.MedianPruner(n_startup_trials=5, n_warmup_steps=50),\n", " load_if_exists=True\n", ")\n", "study.optimize(objective, n_trials=25-len(study.trials), n_jobs=1)" ] }, { "cell_type": "markdown", "id": "3964e313", "metadata": {}, "source": [ "During the study creation, we used the input argument `pruner`. This specifies a strategy with which we want to stop experiments early if they don't look promising. For instance, a very low learning rate combined with high weight decay and dropout will likely achieve low performance, which we can already judge after 50 epochs and don't have to run the model for much longer. For this, we implemented the `on_validation_epoch_end` callback in our Trainer module before. After each epoch, it reports the current validation performance to Optuna. Depending on the previous performances and Optuna's pruning strategy, it may decide to stop the experiment early, which it does by throwing a `TrialPruned` error. This error is caught by Optuna, and the next trial is directly started." ] }, { "cell_type": "markdown", "id": "e232b4d2", "metadata": {}, "source": [ "### Evaluate hyperparameter search\n", "\n", "After finishing the hyperparameter search, we can analyze the results. First, let's print the best model found and its corresponding hyperparameters:" ] }, { "cell_type": "code", "execution_count": 24, "id": "a8dd8fdf", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Best Validation Accuracy: 63.44%\n", "Best Params:\n", "-> dropout_prob: 0.39573629692783413\n", "-> lr: 0.002097404408052793\n", "-> weight_decay: 0.0012107132860246818\n" ] } ], "source": [ "trial = study.best_trial\n", "print(f'Best Validation Accuracy: {trial.value:4.2%}')\n", "print(f'Best Params:')\n", "for key, value in trial.params.items():\n", " print(f'-> {key}: {value}')" ] }, { "cell_type": "markdown", "id": "a7bd7373", "metadata": {}, "source": [ "The validation performance is quite a bit higher than the model we had manually designed before. Let's load the model and check its test performance. For this, we can make use of the `load_from_checkpoint` function of our Trainer module:" ] }, { "cell_type": "code", "execution_count": 25, "id": "846bfb75", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
                                   MLPClassifier Summary                                    \n",
       "┏━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃ path           outputs                  batch_stats         params                    ┃\n",
       "┡━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│ Inputs        │ train: True             │                    │                           │\n",
       "│               │ x: float64[256,32,32,3] │                    │                           │\n",
       "├───────────────┼─────────────────────────┼────────────────────┼───────────────────────────┤\n",
       "│ BatchNorm_0   │ float32[256,512]        │ mean: float32[512] │ bias: float32[512]        │\n",
       "│               │                         │ var: float32[512]  │ scale: float32[512]       │\n",
       "│               │                         │                    │                           │\n",
       "│               │                         │ 1,024 (4.1 KB)1,024 (4.1 KB)            │\n",
       "├───────────────┼─────────────────────────┼────────────────────┼───────────────────────────┤\n",
       "│ BatchNorm_1   │ float32[256,512]        │ mean: float32[512] │ bias: float32[512]        │\n",
       "│               │                         │ var: float32[512]  │ scale: float32[512]       │\n",
       "│               │                         │                    │                           │\n",
       "│               │                         │ 1,024 (4.1 KB)1,024 (4.1 KB)            │\n",
       "├───────────────┼─────────────────────────┼────────────────────┼───────────────────────────┤\n",
       "│ Dense_0       │ float32[256,512]        │                    │ bias: float32[512]        │\n",
       "│               │                         │                    │ kernel: float32[3072,512] │\n",
       "│               │                         │                    │                           │\n",
       "│               │                         │                    │ 1,573,376 (6.3 MB)        │\n",
       "├───────────────┼─────────────────────────┼────────────────────┼───────────────────────────┤\n",
       "│ Dense_1       │ float32[256,512]        │                    │ bias: float32[512]        │\n",
       "│               │                         │                    │ kernel: float32[512,512]  │\n",
       "│               │                         │                    │                           │\n",
       "│               │                         │                    │ 262,656 (1.1 MB)          │\n",
       "├───────────────┼─────────────────────────┼────────────────────┼───────────────────────────┤\n",
       "│ Dense_2       │ float32[256,10]         │                    │ bias: float32[10]         │\n",
       "│               │                         │                    │ kernel: float32[512,10]   │\n",
       "│               │                         │                    │                           │\n",
       "│               │                         │                    │ 5,130 (20.5 KB)           │\n",
       "├───────────────┼─────────────────────────┼────────────────────┼───────────────────────────┤\n",
       "│ Dropout_0     │ float32[256,3072]       │                    │                           │\n",
       "├───────────────┼─────────────────────────┼────────────────────┼───────────────────────────┤\n",
       "│ Dropout_1     │ float32[256,512]        │                    │                           │\n",
       "├───────────────┼─────────────────────────┼────────────────────┼───────────────────────────┤\n",
       "│ Dropout_2     │ float32[256,512]        │                    │                           │\n",
       "├───────────────┼─────────────────────────┼────────────────────┼───────────────────────────┤\n",
       "│ MLPClassifier │ float32[256,10]         │                    │                           │\n",
       "├───────────────┼─────────────────────────┼────────────────────┼───────────────────────────┤\n",
       "│                                  Total  2,048 (8.2 KB)      1,843,210 (7.4 MB)        │\n",
       "└───────────────┴─────────────────────────┴────────────────────┴───────────────────────────┘\n",
       "                                                                                            \n",
       "                            Total Parameters: 1,845,258 (7.4 MB)                            \n",
       "
\n" ], "text/plain": [ "\u001b[3m MLPClassifier Summary \u001b[0m\n", "┏━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n", "┃\u001b[1m \u001b[0m\u001b[1mpath \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1moutputs \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mbatch_stats \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mparams \u001b[0m\u001b[1m \u001b[0m┃\n", "┡━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n", "│ Inputs │ train: True │ │ │\n", "│ │ x: \u001b[2mfloat64\u001b[0m[256,32,32,3] │ │ │\n", "├───────────────┼─────────────────────────┼────────────────────┼───────────────────────────┤\n", "│ BatchNorm_0 │ \u001b[2mfloat32\u001b[0m[256,512] │ mean: \u001b[2mfloat32\u001b[0m[512] │ bias: \u001b[2mfloat32\u001b[0m[512] │\n", "│ │ │ var: \u001b[2mfloat32\u001b[0m[512] │ scale: \u001b[2mfloat32\u001b[0m[512] │\n", "│ │ │ │ │\n", "│ │ │ \u001b[1m1,024 \u001b[0m\u001b[1;2m(4.1 KB)\u001b[0m │ \u001b[1m1,024 \u001b[0m\u001b[1;2m(4.1 KB)\u001b[0m │\n", "├───────────────┼─────────────────────────┼────────────────────┼───────────────────────────┤\n", "│ BatchNorm_1 │ \u001b[2mfloat32\u001b[0m[256,512] │ mean: \u001b[2mfloat32\u001b[0m[512] │ bias: \u001b[2mfloat32\u001b[0m[512] │\n", "│ │ │ var: \u001b[2mfloat32\u001b[0m[512] │ scale: \u001b[2mfloat32\u001b[0m[512] │\n", "│ │ │ │ │\n", "│ │ │ \u001b[1m1,024 \u001b[0m\u001b[1;2m(4.1 KB)\u001b[0m │ \u001b[1m1,024 \u001b[0m\u001b[1;2m(4.1 KB)\u001b[0m │\n", "├───────────────┼─────────────────────────┼────────────────────┼───────────────────────────┤\n", "│ Dense_0 │ \u001b[2mfloat32\u001b[0m[256,512] │ │ bias: \u001b[2mfloat32\u001b[0m[512] │\n", "│ │ │ │ kernel: \u001b[2mfloat32\u001b[0m[3072,512] │\n", "│ │ │ │ │\n", "│ │ │ │ \u001b[1m1,573,376 \u001b[0m\u001b[1;2m(6.3 MB)\u001b[0m │\n", "├───────────────┼─────────────────────────┼────────────────────┼───────────────────────────┤\n", "│ Dense_1 │ \u001b[2mfloat32\u001b[0m[256,512] │ │ bias: \u001b[2mfloat32\u001b[0m[512] │\n", "│ │ │ │ kernel: \u001b[2mfloat32\u001b[0m[512,512] │\n", "│ │ │ │ │\n", "│ │ │ │ \u001b[1m262,656 \u001b[0m\u001b[1;2m(1.1 MB)\u001b[0m │\n", "├───────────────┼─────────────────────────┼────────────────────┼───────────────────────────┤\n", "│ Dense_2 │ \u001b[2mfloat32\u001b[0m[256,10] │ │ bias: \u001b[2mfloat32\u001b[0m[10] │\n", "│ │ │ │ kernel: \u001b[2mfloat32\u001b[0m[512,10] │\n", "│ │ │ │ │\n", "│ │ │ │ \u001b[1m5,130 \u001b[0m\u001b[1;2m(20.5 KB)\u001b[0m │\n", "├───────────────┼─────────────────────────┼────────────────────┼───────────────────────────┤\n", "│ Dropout_0 │ \u001b[2mfloat32\u001b[0m[256,3072] │ │ │\n", "├───────────────┼─────────────────────────┼────────────────────┼───────────────────────────┤\n", "│ Dropout_1 │ \u001b[2mfloat32\u001b[0m[256,512] │ │ │\n", "├───────────────┼─────────────────────────┼────────────────────┼───────────────────────────┤\n", "│ Dropout_2 │ \u001b[2mfloat32\u001b[0m[256,512] │ │ │\n", "├───────────────┼─────────────────────────┼────────────────────┼───────────────────────────┤\n", "│ MLPClassifier │ \u001b[2mfloat32\u001b[0m[256,10] │ │ │\n", "├───────────────┼─────────────────────────┼────────────────────┼───────────────────────────┤\n", "│\u001b[1m \u001b[0m\u001b[1m \u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m Total\u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m2,048 \u001b[0m\u001b[1;2m(8.2 KB)\u001b[0m\u001b[1m \u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m1,843,210 \u001b[0m\u001b[1;2m(7.4 MB)\u001b[0m\u001b[1m \u001b[0m\u001b[1m \u001b[0m│\n", "└───────────────┴─────────────────────────┴────────────────────┴───────────────────────────┘\n", "\u001b[1m \u001b[0m\n", "\u001b[1m Total Parameters: 1,845,258 \u001b[0m\u001b[1;2m(7.4 MB)\u001b[0m\u001b[1m \u001b[0m\n" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "\n", "\n", "Test accuracy: 62.89%\n" ] } ], "source": [ "trainer = MLPClassTrainer.load_from_checkpoint(os.path.join(CHECKPOINT_PATH, 'MLPClassifier/version_16/'), \n", " exmp_input=next(iter(train_loader)))\n", "test_metrics = trainer.eval_model(test_loader)\n", "print(f'Test accuracy: {test_metrics[\"acc\"]:4.2%}')" ] }, { "cell_type": "markdown", "id": "b291a163", "metadata": {}, "source": [ "The test performance is also quite strong, showing the benefit of the automatic hyperparameter search. However, often, we are not just interested in the best model. Optuna provides several ways of visualizing the results of the hyperparameter study, for instance by plotting all validation accuracy curves:" ] }, { "cell_type": "code", "execution_count": 26, "id": "433aed71", "metadata": {}, "outputs": [ { "data": { "text/html": [ " \n", " " ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/vnd.plotly.v1+json": { "config": { "plotlyServerURL": "https://plot.ly" }, "data": [ { "marker": { "maxdisplayed": 10 }, "mode": "lines+markers", "name": "Trial0", "type": "scatter", "x": [ 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 110, 115, 120, 125, 130, 135, 140, 145, 150, 155, 160, 165, 170, 175, 180, 185, 190, 195, 200 ], "y": [ 0.5258000493049622, 0.5596000552177429, 0.5718000531196594, 0.5824000239372253, 0.5924000144004822, 0.6004000306129456, 0.6038000583648682, 0.6058000326156616, 0.605400025844574, 0.6132000088691711, 0.6172000169754028, 0.612000048160553, 0.6134000420570374, 0.6160000562667847, 0.6212000250816345, 0.6146000027656555, 0.625, 0.6166000366210938, 0.6244000196456909, 0.6258000135421753, 0.6302000284194946, 0.6248000264167786, 0.6282000541687012, 0.6212000250816345, 0.6290000081062317, 0.6230000257492065, 0.626800000667572, 0.6214000582695007, 0.6294000148773193, 0.628000020980835, 0.6300000548362732, 0.6258000135421753, 0.6274000406265259, 0.628600001335144, 0.6292000412940979, 0.6276000142097473, 0.628600001335144, 0.6278000473976135, 0.6304000020027161, 0.6314000487327576 ] }, { "marker": { "maxdisplayed": 10 }, "mode": "lines+markers", "name": "Trial1", "type": "scatter", "x": [ 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 110, 115, 120, 125, 130, 135, 140, 145, 150, 155, 160, 165, 170, 175, 180, 185, 190, 195, 200 ], "y": [ 0.5, 0.5306000113487244, 0.5466000437736511, 0.5600000023841858, 0.5646000504493713, 0.5730000138282776, 0.5706000328063965, 0.5848000049591064, 0.5848000049591064, 0.5888000130653381, 0.5956000089645386, 0.5974000096321106, 0.6034000515937805, 0.5998000502586365, 0.6044000387191772, 0.5972000360488892, 0.6055999994277954, 0.6136000156402588, 0.6068000197410583, 0.6086000204086304, 0.609000027179718, 0.6082000136375427, 0.6088000535964966, 0.6098000407218933, 0.6122000217437744, 0.6154000163078308, 0.6136000156402588, 0.6132000088691711, 0.6144000291824341, 0.6162000298500061, 0.6168000102043152, 0.6134000420570374, 0.6162000298500061, 0.6140000224113464, 0.6154000163078308, 0.6142000555992126, 0.6150000095367432, 0.615600049495697, 0.6152000427246094, 0.6140000224113464 ] }, { "marker": { "maxdisplayed": 10 }, "mode": "lines+markers", "name": "Trial2", "type": "scatter", "x": [ 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 110, 115, 120, 125, 130, 135, 140, 145, 150, 155, 160, 165, 170, 175, 180, 185, 190, 195, 200 ], "y": [ 0.5040000081062317, 0.537600040435791, 0.5468000173568726, 0.5626000165939331, 0.5730000138282776, 0.5782000422477722, 0.58160001039505, 0.5872000455856323, 0.5904000401496887, 0.5908000469207764, 0.6022000312805176, 0.5978000164031982, 0.6038000583648682, 0.6055999994277954, 0.6034000515937805, 0.6084000468254089, 0.6106000542640686, 0.6162000298500061, 0.615600049495697, 0.6150000095367432, 0.6172000169754028, 0.6176000237464905, 0.6198000311851501, 0.6178000569343567, 0.6154000163078308, 0.6222000122070312, 0.6200000047683716, 0.6220000386238098, 0.6260000467300415, 0.6220000386238098, 0.6246000528335571, 0.6240000128746033, 0.6256000399589539, 0.6228000521659851, 0.6252000331878662, 0.6258000135421753, 0.6252000331878662, 0.6254000067710876, 0.6256000399589539, 0.6262000203132629 ] }, { "marker": { "maxdisplayed": 10 }, "mode": "lines+markers", "name": "Trial3", "type": "scatter", "x": [ 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 110, 115, 120, 125, 130, 135, 140, 145, 150, 155, 160, 165, 170, 175, 180, 185, 190, 195, 200 ], "y": [ 0.4846000373363495, 0.5156000256538391, 0.5356000065803528, 0.5466000437736511, 0.5527999997138977, 0.5630000233650208, 0.5618000030517578, 0.5716000199317932, 0.5768000483512878, 0.5774000287055969, 0.5820000171661377, 0.5861999988555908, 0.5908000469207764, 0.5890000462532043, 0.5944000482559204, 0.5960000157356262, 0.5948000550270081, 0.602400004863739, 0.598800003528595, 0.6020000576972961, 0.6028000116348267, 0.6044000387191772, 0.6055999994277954, 0.6062000393867493, 0.6086000204086304, 0.6068000197410583, 0.6092000007629395, 0.6100000143051147, 0.6100000143051147, 0.6104000210762024, 0.6114000082015991, 0.612000048160553, 0.6110000014305115, 0.6132000088691711, 0.6104000210762024, 0.6094000339508057, 0.6094000339508057, 0.6100000143051147, 0.61080002784729, 0.6106000542640686 ] }, { "marker": { "maxdisplayed": 10 }, "mode": "lines+markers", "name": "Trial4", "type": "scatter", "x": [ 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 110, 115, 120, 125, 130, 135, 140, 145, 150, 155, 160, 165, 170, 175, 180, 185, 190, 195, 200 ], "y": [ 0.5178000330924988, 0.5462000370025635, 0.5592000484466553, 0.5716000199317932, 0.5800000429153442, 0.5868000388145447, 0.5926000475883484, 0.5879999995231628, 0.5906000137329102, 0.5978000164031982, 0.6068000197410583, 0.6028000116348267, 0.5940000414848328, 0.6062000393867493, 0.61080002784729, 0.6080000400543213, 0.612000048160553, 0.5952000021934509, 0.6146000027656555, 0.6082000136375427, 0.6152000427246094, 0.6158000230789185, 0.6218000054359436, 0.617400050163269, 0.6198000311851501, 0.6202000379562378, 0.6190000176429749, 0.6228000521659851, 0.6198000311851501, 0.6228000521659851, 0.626800000667572, 0.6256000399589539, 0.6282000541687012, 0.626800000667572, 0.6276000142097473, 0.6262000203132629, 0.6288000345230103, 0.628600001335144, 0.6282000541687012, 0.6282000541687012 ] }, { "marker": { "maxdisplayed": 10 }, "mode": "lines+markers", "name": "Trial5", "type": "scatter", "x": [ 5, 10, 15, 20, 25, 30, 35, 40, 45, 50 ], "y": [ 0.5, 0.5351999998092651, 0.5460000038146973, 0.5570000410079956, 0.5680000185966492, 0.5758000016212463, 0.5790000557899475, 0.5874000191688538, 0.58760005235672, 0.5886000394821167 ] }, { "marker": { "maxdisplayed": 10 }, "mode": "lines+markers", "name": "Trial6", "type": "scatter", "x": [ 5, 10, 15, 20, 25, 30, 35, 40, 45, 50 ], "y": [ 0.42600002884864807, 0.45680001378059387, 0.47200003266334534, 0.4880000352859497, 0.49500003457069397, 0.504800021648407, 0.510200023651123, 0.5170000195503235, 0.5198000073432922, 0.5220000147819519 ] }, { "marker": { "maxdisplayed": 10 }, "mode": "lines+markers", "name": "Trial7", "type": "scatter", "x": [ 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55 ], "y": [ 0.5049999952316284, 0.5486000180244446, 0.5548000335693359, 0.5667999982833862, 0.5770000219345093, 0.5870000123977661, 0.5902000069618225, 0.598800003528595, 0.5974000096321106, 0.598800003528595, 0.6008000373840332 ] }, { "marker": { "maxdisplayed": 10 }, "mode": "lines+markers", "name": "Trial8", "type": "scatter", "x": [ 5, 10, 15, 20, 25, 30, 35, 40, 45, 50 ], "y": [ 0.5502000451087952, 0.58160001039505, 0.5742000341415405, 0.5772000551223755, 0.5834000110626221, 0.5822000503540039, 0.5888000130653381, 0.576200008392334, 0.574400007724762, 0.5820000171661377 ] }, { "marker": { "maxdisplayed": 10 }, "mode": "lines+markers", "name": "Trial9", "type": "scatter", "x": [ 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 110, 115, 120, 125, 130, 135, 140, 145, 150, 155, 160, 165 ], "y": [ 0.5314000248908997, 0.5560000538825989, 0.5658000111579895, 0.578000009059906, 0.5910000205039978, 0.5982000231742859, 0.5954000353813171, 0.6014000177383423, 0.6034000515937805, 0.6098000407218933, 0.6055999994277954, 0.6050000190734863, 0.6142000555992126, 0.6128000020980835, 0.609000027179718, 0.6152000427246094, 0.6136000156402588, 0.6190000176429749, 0.6188000440597534, 0.6202000379562378, 0.6168000102043152, 0.6220000386238098, 0.6226000189781189, 0.6182000041007996, 0.6220000386238098, 0.6248000264167786, 0.6198000311851501, 0.6230000257492065, 0.6220000386238098, 0.6216000318527222, 0.6230000257492065, 0.6230000257492065, 0.6228000521659851 ] }, { "marker": { "maxdisplayed": 10 }, "mode": "lines+markers", "name": "Trial10", "type": "scatter", "x": [ 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65 ], "y": [ 0.5332000255584717, 0.5672000050544739, 0.5760000348091125, 0.5826000571250916, 0.5944000482559204, 0.5878000259399414, 0.6022000312805176, 0.5956000089645386, 0.5946000218391418, 0.5892000198364258, 0.5926000475883484, 0.5928000211715698, 0.5900000333786011 ] }, { "marker": { "maxdisplayed": 10 }, "mode": "lines+markers", "name": "Trial11", "type": "scatter", "x": [ 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 110, 115, 120, 125, 130, 135, 140, 145, 150, 155, 160, 165, 170, 175, 180, 185, 190, 195, 200 ], "y": [ 0.531000018119812, 0.5652000308036804, 0.5750000476837158, 0.5888000130653381, 0.5910000205039978, 0.5998000502586365, 0.6010000109672546, 0.6104000210762024, 0.6044000387191772, 0.6060000061988831, 0.6136000156402588, 0.6080000400543213, 0.607200026512146, 0.6132000088691711, 0.6182000041007996, 0.612000048160553, 0.6124000549316406, 0.6212000250816345, 0.6152000427246094, 0.6194000244140625, 0.6242000460624695, 0.6224000453948975, 0.6202000379562378, 0.6272000074386597, 0.6238000392913818, 0.6254000067710876, 0.6264000535011292, 0.6294000148773193, 0.6308000087738037, 0.6246000528335571, 0.628600001335144, 0.6264000535011292, 0.6294000148773193, 0.6300000548362732, 0.6266000270843506, 0.6292000412940979, 0.628000020980835, 0.6300000548362732, 0.6276000142097473, 0.6290000081062317 ] }, { "marker": { "maxdisplayed": 10 }, "mode": "lines+markers", "name": "Trial12", "type": "scatter", "x": [ 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 110, 115, 120, 125, 130, 135, 140, 145, 150, 155, 160, 165, 170, 175, 180, 185, 190, 195, 200 ], "y": [ 0.5246000289916992, 0.5684000253677368, 0.5742000341415405, 0.5888000130653381, 0.5946000218391418, 0.6002000570297241, 0.6044000387191772, 0.610200047492981, 0.6078000068664551, 0.6146000027656555, 0.6142000555992126, 0.6122000217437744, 0.6150000095367432, 0.6164000034332275, 0.6136000156402588, 0.6206000447273254, 0.6172000169754028, 0.6164000034332275, 0.6190000176429749, 0.615600049495697, 0.6256000399589539, 0.6192000508308411, 0.6226000189781189, 0.6228000521659851, 0.6266000270843506, 0.6272000074386597, 0.628600001335144, 0.6236000061035156, 0.6234000325202942, 0.6260000467300415, 0.628000020980835, 0.6270000338554382, 0.6236000061035156, 0.628600001335144, 0.6270000338554382, 0.628000020980835, 0.626800000667572, 0.626800000667572, 0.6282000541687012, 0.6272000074386597 ] }, { "marker": { "maxdisplayed": 10 }, "mode": "lines+markers", "name": "Trial13", "type": "scatter", "x": [ 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55 ], "y": [ 0.5455999970436096, 0.5774000287055969, 0.5868000388145447, 0.5848000049591064, 0.6026000380516052, 0.5968000292778015, 0.5986000299453735, 0.5960000157356262, 0.597000002861023, 0.5948000550270081, 0.602400004863739 ] }, { "marker": { "maxdisplayed": 10 }, "mode": "lines+markers", "name": "Trial14", "type": "scatter", "x": [ 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 110, 115, 120, 125, 130 ], "y": [ 0.5136000514030457, 0.5502000451087952, 0.5674000382423401, 0.5784000158309937, 0.586400032043457, 0.5910000205039978, 0.5928000211715698, 0.5992000102996826, 0.6080000400543213, 0.6082000136375427, 0.609000027179718, 0.6080000400543213, 0.6092000007629395, 0.6086000204086304, 0.6152000427246094, 0.610200047492981, 0.6176000237464905, 0.6194000244140625, 0.615600049495697, 0.6162000298500061, 0.6200000047683716, 0.6210000514984131, 0.6220000386238098, 0.6210000514984131, 0.6200000047683716, 0.6196000576019287 ] }, { "marker": { "maxdisplayed": 10 }, "mode": "lines+markers", "name": "Trial15", "type": "scatter", "x": [ 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90 ], "y": [ 0.5422000288963318, 0.5748000144958496, 0.5822000503540039, 0.5978000164031982, 0.6004000306129456, 0.6058000326156616, 0.6126000285148621, 0.6060000061988831, 0.6082000136375427, 0.6084000468254089, 0.6106000542640686, 0.6066000461578369, 0.6055999994277954, 0.6070000529289246, 0.6106000542640686, 0.6030000448226929, 0.6034000515937805, 0.6092000007629395 ] }, { "marker": { "maxdisplayed": 10 }, "mode": "lines+markers", "name": "Trial16", "type": "scatter", "x": [ 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 110, 115, 120, 125, 130, 135, 140, 145, 150, 155, 160, 165, 170, 175, 180, 185, 190, 195, 200 ], "y": [ 0.5200000405311584, 0.5488000512123108, 0.5732000470161438, 0.5724000334739685, 0.591200053691864, 0.5962000489234924, 0.6074000000953674, 0.600600004196167, 0.6076000332832336, 0.6064000129699707, 0.6100000143051147, 0.61080002784729, 0.6060000061988831, 0.6140000224113464, 0.6182000041007996, 0.6192000508308411, 0.6172000169754028, 0.6146000027656555, 0.6258000135421753, 0.6224000453948975, 0.6274000406265259, 0.6262000203132629, 0.6292000412940979, 0.6266000270843506, 0.6254000067710876, 0.6284000277519226, 0.6282000541687012, 0.6308000087738037, 0.6292000412940979, 0.6344000101089478, 0.631600022315979, 0.631600022315979, 0.6278000473976135, 0.6306000351905823, 0.6300000548362732, 0.631600022315979, 0.6306000351905823, 0.629800021648407, 0.631600022315979, 0.6308000087738037 ] }, { "marker": { "maxdisplayed": 10 }, "mode": "lines+markers", "name": "Trial18", "type": "scatter", "x": [ 5, 10, 15, 20, 25, 30, 35, 40, 45, 50 ], "y": [ 0.47300001978874207, 0.5014000535011292, 0.5246000289916992, 0.532200038433075, 0.5384000539779663, 0.5464000105857849, 0.5520000457763672, 0.5552000403404236, 0.562000036239624, 0.5606000423431396 ] }, { "marker": { "maxdisplayed": 10 }, "mode": "lines+markers", "name": "Trial19", "type": "scatter", "x": [ 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 110, 115, 120, 125, 130, 135, 140, 145, 150, 155, 160, 165, 170, 175, 180, 185, 190, 195, 200 ], "y": [ 0.5216000080108643, 0.5455999970436096, 0.5670000314712524, 0.5740000009536743, 0.5870000123977661, 0.5972000360488892, 0.6014000177383423, 0.6046000123023987, 0.6076000332832336, 0.6020000576972961, 0.610200047492981, 0.6134000420570374, 0.610200047492981, 0.6114000082015991, 0.6210000514984131, 0.6204000115394592, 0.6214000582695007, 0.6158000230789185, 0.6252000331878662, 0.6232000589370728, 0.6246000528335571, 0.6282000541687012, 0.6258000135421753, 0.6242000460624695, 0.631600022315979, 0.6276000142097473, 0.6296000480651855, 0.6300000548362732, 0.6306000351905823, 0.6302000284194946, 0.629800021648407, 0.6290000081062317, 0.6300000548362732, 0.6314000487327576, 0.6304000020027161, 0.6300000548362732, 0.6296000480651855, 0.6278000473976135, 0.6288000345230103, 0.6296000480651855 ] }, { "marker": { "maxdisplayed": 10 }, "mode": "lines+markers", "name": "Trial20", "type": "scatter", "x": [ 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 110, 115, 120, 125, 130, 135, 140, 145, 150, 155, 160, 165, 170, 175, 180, 185, 190, 195, 200 ], "y": [ 0.5320000052452087, 0.5706000328063965, 0.5756000280380249, 0.5938000082969666, 0.5980000495910645, 0.6110000014305115, 0.6092000007629395, 0.6070000529289246, 0.6088000535964966, 0.6148000359535217, 0.6198000311851501, 0.6164000034332275, 0.6164000034332275, 0.6146000027656555, 0.612000048160553, 0.6218000054359436, 0.6218000054359436, 0.6236000061035156, 0.6254000067710876, 0.6208000183105469, 0.6210000514984131, 0.6290000081062317, 0.6278000473976135, 0.6248000264167786, 0.6198000311851501, 0.6210000514984131, 0.6254000067710876, 0.6270000338554382, 0.6240000128746033, 0.6234000325202942, 0.628600001335144, 0.6262000203132629, 0.6304000020027161, 0.626800000667572, 0.626800000667572, 0.628000020980835, 0.6266000270843506, 0.6274000406265259, 0.6262000203132629, 0.628000020980835 ] }, { "marker": { "maxdisplayed": 10 }, "mode": "lines+markers", "name": "Trial21", "type": "scatter", "x": [ 5, 10, 15, 20, 25, 30, 35, 40, 45, 50 ], "y": [ 0.5126000046730042, 0.5420000553131104, 0.5520000457763672, 0.5672000050544739, 0.5752000212669373, 0.582800030708313, 0.5834000110626221, 0.5902000069618225, 0.593000054359436, 0.5968000292778015 ] }, { "marker": { "maxdisplayed": 10 }, "mode": "lines+markers", "name": "Trial22", "type": "scatter", "x": [ 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55 ], "y": [ 0.5166000127792358, 0.5486000180244446, 0.5630000233650208, 0.5768000483512878, 0.5822000503540039, 0.593000054359436, 0.600600004196167, 0.6014000177383423, 0.602400004863739, 0.6046000123023987, 0.6078000068664551 ] }, { "marker": { "maxdisplayed": 10 }, "mode": "lines+markers", "name": "Trial24", "type": "scatter", "x": [ 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 110, 115, 120, 125, 130, 135, 140, 145, 150, 155, 160, 165, 170, 175, 180, 185, 190, 195, 200 ], "y": [ 0.5404000282287598, 0.5700000524520874, 0.5806000232696533, 0.589400053024292, 0.6012000441551208, 0.6114000082015991, 0.6058000326156616, 0.6058000326156616, 0.6148000359535217, 0.6114000082015991, 0.6204000115394592, 0.6198000311851501, 0.615600049495697, 0.6196000576019287, 0.6274000406265259, 0.6248000264167786, 0.6256000399589539, 0.6270000338554382, 0.6248000264167786, 0.6244000196456909, 0.6264000535011292, 0.6288000345230103, 0.6232000589370728, 0.6220000386238098, 0.6270000338554382, 0.6262000203132629, 0.6292000412940979, 0.6254000067710876, 0.6290000081062317, 0.6282000541687012, 0.631600022315979, 0.6266000270843506, 0.6274000406265259, 0.6292000412940979, 0.6302000284194946, 0.6302000284194946, 0.6312000155448914, 0.6304000020027161, 0.6294000148773193, 0.6290000081062317 ] } ], "layout": { "showlegend": false, "template": { "data": { "bar": [ { "error_x": { "color": "#2a3f5f" }, "error_y": { "color": "#2a3f5f" }, "marker": { "line": { "color": "#E5ECF6", "width": 0.5 }, "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "bar" } ], "barpolar": [ { "marker": { "line": { "color": "#E5ECF6", "width": 0.5 }, "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "barpolar" } ], "carpet": [ { "aaxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "baxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "type": "carpet" } ], "choropleth": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "choropleth" } ], "contour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "contour" } ], "contourcarpet": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "contourcarpet" } ], "heatmap": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "heatmap" } ], "heatmapgl": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "heatmapgl" } ], "histogram": [ { "marker": { "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "histogram" } ], "histogram2d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "histogram2d" } ], "histogram2dcontour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "histogram2dcontour" } ], "mesh3d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "mesh3d" } ], "parcoords": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "parcoords" } ], "pie": [ { "automargin": true, "type": "pie" } ], "scatter": [ { "fillpattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 }, "type": "scatter" } ], "scatter3d": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter3d" } ], "scattercarpet": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattercarpet" } ], "scattergeo": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergeo" } ], "scattergl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergl" } ], "scattermapbox": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattermapbox" } ], "scatterpolar": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolar" } ], "scatterpolargl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolargl" } ], "scatterternary": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterternary" } ], "surface": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "surface" } ], "table": [ { "cells": { "fill": { "color": "#EBF0F8" }, "line": { "color": "white" } }, "header": { "fill": { "color": "#C8D4E3" }, "line": { "color": "white" } }, "type": "table" } ] }, "layout": { "annotationdefaults": { "arrowcolor": "#2a3f5f", "arrowhead": 0, "arrowwidth": 1 }, "autotypenumbers": "strict", "coloraxis": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "colorscale": { "diverging": [ [ 0, "#8e0152" ], [ 0.1, "#c51b7d" ], [ 0.2, "#de77ae" ], [ 0.3, "#f1b6da" ], [ 0.4, "#fde0ef" ], [ 0.5, "#f7f7f7" ], [ 0.6, "#e6f5d0" ], [ 0.7, "#b8e186" ], [ 0.8, "#7fbc41" ], [ 0.9, "#4d9221" ], [ 1, "#276419" ] ], "sequential": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "sequentialminus": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ] }, "colorway": [ "#636efa", "#EF553B", "#00cc96", "#ab63fa", "#FFA15A", "#19d3f3", "#FF6692", "#B6E880", "#FF97FF", "#FECB52" ], "font": { "color": "#2a3f5f" }, "geo": { "bgcolor": "white", "lakecolor": "white", "landcolor": "#E5ECF6", "showlakes": true, "showland": true, "subunitcolor": "white" }, "hoverlabel": { "align": "left" }, "hovermode": "closest", "mapbox": { "style": "light" }, "paper_bgcolor": "white", "plot_bgcolor": "#E5ECF6", "polar": { "angularaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "radialaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "scene": { "xaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "yaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "zaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" } }, "shapedefaults": { "line": { "color": "#2a3f5f" } }, "ternary": { "aaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "baxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "caxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "title": { "x": 0.05 }, "xaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 }, "yaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 } } }, "title": { "text": "Intermediate Values Plot" }, "xaxis": { "title": { "text": "Step" } }, "yaxis": { "title": { "text": "Intermediate Value" } } } }, "text/html": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "fig = optuna.visualization.plot_intermediate_values(study)\n", "fig.show()" ] }, { "cell_type": "markdown", "id": "ea487a3a", "metadata": {}, "source": [ "As we can see, there is quite a big group of models that perform very similarly. At the same time, there were a few models with very poor results, which were lucky stopped early by Optuna to not waste compute. \n", "\n", "Another question we might have is which hyperparameter was the most important for the performance? This can be directly visualized by `plot_param_importances`. By default, Optuna uses a random forest regression to estimate the importance of each hyperparameter ([documentation](https://optuna.readthedocs.io/en/stable/reference/generated/optuna.importance.FanovaImportanceEvaluator.html#optuna.importance.FanovaImportanceEvaluator)), and shows it as values that sum up to 1 for all hyperparameters:" ] }, { "cell_type": "code", "execution_count": 27, "id": "3dd9c34c", "metadata": {}, "outputs": [ { "data": { "application/vnd.plotly.v1+json": { "config": { "plotlyServerURL": "https://plot.ly" }, "data": [ { "cliponaxis": false, "hovertemplate": [ "weight_decay (LogUniformDistribution): 0.10002080050477152", "dropout_prob (UniformDistribution): 0.1840508429559047", "lr (LogUniformDistribution): 0.7159283565393239" ], "marker": { "color": "rgb(66,146,198)" }, "orientation": "h", "text": [ "0.10002080050477152", "0.1840508429559047", "0.7159283565393239" ], "textposition": "outside", "texttemplate": "%{text:.2f}", "type": "bar", "x": [ 0.10002080050477152, 0.1840508429559047, 0.7159283565393239 ], "y": [ "weight_decay", "dropout_prob", "lr" ] } ], "layout": { "showlegend": false, "template": { "data": { "bar": [ { "error_x": { "color": "#2a3f5f" }, "error_y": { "color": "#2a3f5f" }, "marker": { "line": { "color": "#E5ECF6", "width": 0.5 }, "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "bar" } ], "barpolar": [ { "marker": { "line": { "color": "#E5ECF6", "width": 0.5 }, "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "barpolar" } ], "carpet": [ { "aaxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "baxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "type": "carpet" } ], "choropleth": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "choropleth" } ], "contour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "contour" } ], "contourcarpet": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "contourcarpet" } ], "heatmap": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "heatmap" } ], "heatmapgl": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "heatmapgl" } ], "histogram": [ { "marker": { "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "histogram" } ], "histogram2d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "histogram2d" } ], "histogram2dcontour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "histogram2dcontour" } ], "mesh3d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "mesh3d" } ], "parcoords": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "parcoords" } ], "pie": [ { "automargin": true, "type": "pie" } ], "scatter": [ { "fillpattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 }, "type": "scatter" } ], "scatter3d": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter3d" } ], "scattercarpet": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattercarpet" } ], "scattergeo": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergeo" } ], "scattergl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergl" } ], "scattermapbox": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattermapbox" } ], "scatterpolar": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolar" } ], "scatterpolargl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolargl" } ], "scatterternary": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterternary" } ], "surface": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "surface" } ], "table": [ { "cells": { "fill": { "color": "#EBF0F8" }, "line": { "color": "white" } }, "header": { "fill": { "color": "#C8D4E3" }, "line": { "color": "white" } }, "type": "table" } ] }, "layout": { "annotationdefaults": { "arrowcolor": "#2a3f5f", "arrowhead": 0, "arrowwidth": 1 }, "autotypenumbers": "strict", "coloraxis": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "colorscale": { "diverging": [ [ 0, "#8e0152" ], [ 0.1, "#c51b7d" ], [ 0.2, "#de77ae" ], [ 0.3, "#f1b6da" ], [ 0.4, "#fde0ef" ], [ 0.5, "#f7f7f7" ], [ 0.6, "#e6f5d0" ], [ 0.7, "#b8e186" ], [ 0.8, "#7fbc41" ], [ 0.9, "#4d9221" ], [ 1, "#276419" ] ], "sequential": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "sequentialminus": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ] }, "colorway": [ "#636efa", "#EF553B", "#00cc96", "#ab63fa", "#FFA15A", "#19d3f3", "#FF6692", "#B6E880", "#FF97FF", "#FECB52" ], "font": { "color": "#2a3f5f" }, "geo": { "bgcolor": "white", "lakecolor": "white", "landcolor": "#E5ECF6", "showlakes": true, "showland": true, "subunitcolor": "white" }, "hoverlabel": { "align": "left" }, "hovermode": "closest", "mapbox": { "style": "light" }, "paper_bgcolor": "white", "plot_bgcolor": "#E5ECF6", "polar": { "angularaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "radialaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "scene": { "xaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "yaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "zaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" } }, "shapedefaults": { "line": { "color": "#2a3f5f" } }, "ternary": { "aaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "baxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "caxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "title": { "x": 0.05 }, "xaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 }, "yaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 } } }, "title": { "text": "Hyperparameter Importances" }, "xaxis": { "title": { "text": "Importance for Objective Value" } }, "yaxis": { "title": { "text": "Hyperparameter" } } } }, "text/html": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "fig = optuna.visualization.plot_param_importances(study)\n", "fig.show()" ] }, { "cell_type": "markdown", "id": "d1cc14bb", "metadata": {}, "source": [ "Interestingly, the learning rate is the most important hyperparameter. \n", "\n", "Finally, to get a good intuition of the interplay between hyperparameters, we can plot the estimated accuracy surface over hyperparameters:" ] }, { "cell_type": "code", "execution_count": 28, "id": "fcd0530c", "metadata": {}, "outputs": [ { "data": { "application/vnd.plotly.v1+json": { "config": { "plotlyServerURL": "https://plot.ly" }, "data": [ { "colorbar": { "title": { "text": "Objective Value" } }, "colorscale": [ [ 0, "rgb(5,10,172)" ], [ 0.35, "rgb(40,60,190)" ], [ 0.5, "rgb(70,100,245)" ], [ 0.6, "rgb(90,120,245)" ], [ 0.7, "rgb(106,137,247)" ], [ 1, "rgb(220,220,220)" ] ], "connectgaps": true, "contours": { "coloring": "heatmap" }, "hoverinfo": "none", "line": { "smoothing": 1.3 }, "reversescale": false, "type": "contour", "x": [ 0.2656807026267916, 0.27577279675695177, 0.29183134632896607, 0.3251148821458062, 0.3408997139932338, 0.35243531172620524, 0.3601189100414495, 0.39573629692783413, 0.39878166656225567, 0.43296234784961435, 0.47418149810543003, 0.477614679360155, 0.48770677349031516 ], "y": [ 0.00011738966920156188, 0.00014229927952328852, 0.0002673059778626356, 0.001274534085173653, 0.0017800932033490465, 0.0019352582636301293, 0.002097404408052793, 0.0027169253442906613, 0.0028385969798708226, 0.0036962094659705996, 0.005797348676281532, 0.006678195819497592, 0.008095281808812697 ], "z": [ [ null, null, null, null, null, null, null, null, null, null, null, null, null ], [ null, null, null, 0.6168000102043152, null, null, null, null, null, null, null, null, null ], [ null, null, null, null, null, null, null, null, null, null, 0.6132000088691711, null, null ], [ null, null, null, null, null, null, null, null, null, null, null, 0.6262000203132629, null ], [ null, 0.6304000020027161, null, null, null, null, null, null, null, null, null, null, null ], [ null, null, null, null, null, null, null, null, 0.631600022315979, null, null, null, null ], [ null, null, null, null, null, null, null, 0.6344000101089478, null, null, null, null, null ], [ null, null, 0.631600022315979, null, null, null, null, null, null, null, null, null, null ], [ null, null, null, null, null, null, 0.6314000487327576, null, null, null, null, null, null ], [ null, null, null, null, 0.628600001335144, null, null, null, null, null, null, null, null ], [ null, null, null, null, null, null, null, null, null, 0.6288000345230103, null, null, null ], [ null, null, null, null, null, 0.6308000087738037, null, null, null, null, null, null, null ], [ null, null, null, null, null, null, null, null, null, null, null, null, null ] ] }, { "marker": { "color": "black", "line": { "color": "Grey", "width": 0.5 } }, "mode": "markers", "showlegend": false, "type": "scatter", "x": [ 0.3601189100414495, 0.3251148821458062, 0.477614679360155, 0.47418149810543003, 0.43296234784961435, 0.35243531172620524, 0.3408997139932338, 0.39573629692783413, 0.39878166656225567, 0.27577279675695177, 0.29183134632896607 ], "y": [ 0.0028385969798708226, 0.00014229927952328852, 0.001274534085173653, 0.0002673059778626356, 0.005797348676281532, 0.006678195819497592, 0.0036962094659705996, 0.002097404408052793, 0.0019352582636301293, 0.0017800932033490465, 0.0027169253442906613 ] } ], "layout": { "template": { "data": { "bar": [ { "error_x": { "color": "#2a3f5f" }, "error_y": { "color": "#2a3f5f" }, "marker": { "line": { "color": "#E5ECF6", "width": 0.5 }, "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "bar" } ], "barpolar": [ { "marker": { "line": { "color": "#E5ECF6", "width": 0.5 }, "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "barpolar" } ], "carpet": [ { "aaxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "baxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "type": "carpet" } ], "choropleth": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "choropleth" } ], "contour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "contour" } ], "contourcarpet": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "contourcarpet" } ], "heatmap": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "heatmap" } ], "heatmapgl": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "heatmapgl" } ], "histogram": [ { "marker": { "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "histogram" } ], "histogram2d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "histogram2d" } ], "histogram2dcontour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "histogram2dcontour" } ], "mesh3d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "mesh3d" } ], "parcoords": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "parcoords" } ], "pie": [ { "automargin": true, "type": "pie" } ], "scatter": [ { "fillpattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 }, "type": "scatter" } ], "scatter3d": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter3d" } ], "scattercarpet": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattercarpet" } ], "scattergeo": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergeo" } ], "scattergl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergl" } ], "scattermapbox": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattermapbox" } ], "scatterpolar": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolar" } ], "scatterpolargl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolargl" } ], "scatterternary": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterternary" } ], "surface": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "surface" } ], "table": [ { "cells": { "fill": { "color": "#EBF0F8" }, "line": { "color": "white" } }, "header": { "fill": { "color": "#C8D4E3" }, "line": { "color": "white" } }, "type": "table" } ] }, "layout": { "annotationdefaults": { "arrowcolor": "#2a3f5f", "arrowhead": 0, "arrowwidth": 1 }, "autotypenumbers": "strict", "coloraxis": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "colorscale": { "diverging": [ [ 0, "#8e0152" ], [ 0.1, "#c51b7d" ], [ 0.2, "#de77ae" ], [ 0.3, "#f1b6da" ], [ 0.4, "#fde0ef" ], [ 0.5, "#f7f7f7" ], [ 0.6, "#e6f5d0" ], [ 0.7, "#b8e186" ], [ 0.8, "#7fbc41" ], [ 0.9, "#4d9221" ], [ 1, "#276419" ] ], "sequential": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "sequentialminus": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ] }, "colorway": [ "#636efa", "#EF553B", "#00cc96", "#ab63fa", "#FFA15A", "#19d3f3", "#FF6692", "#B6E880", "#FF97FF", "#FECB52" ], "font": { "color": "#2a3f5f" }, "geo": { "bgcolor": "white", "lakecolor": "white", "landcolor": "#E5ECF6", "showlakes": true, "showland": true, "subunitcolor": "white" }, "hoverlabel": { "align": "left" }, "hovermode": "closest", "mapbox": { "style": "light" }, "paper_bgcolor": "white", "plot_bgcolor": "#E5ECF6", "polar": { "angularaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "radialaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "scene": { "xaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "yaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "zaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" } }, "shapedefaults": { "line": { "color": "#2a3f5f" } }, "ternary": { "aaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "baxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "caxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "title": { "x": 0.05 }, "xaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 }, "yaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 } } }, "title": { "text": "Contour Plot" }, "xaxis": { "range": [ 0.2656807026267916, 0.48770677349031516 ], "title": { "text": "dropout_prob" } }, "yaxis": { "range": [ -3.9303701211988815, -2.0917680282099247 ], "title": { "text": "lr" }, "type": "log" } } }, "text/html": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "fig = optuna.visualization.plot_contour(study, params=['lr', 'dropout_prob'])\n", "fig.show()" ] }, { "cell_type": "markdown", "id": "b65998ec", "metadata": {}, "source": [ "As the hyperparameter importance plot already suggested, the learning rate seems to be very important while we are more flexible in the dropout probability. However, there seems to be an optimal spot in the middle, overall maximizing the accuracy." ] }, { "cell_type": "markdown", "id": "59242621", "metadata": {}, "source": [ "## Conclusion\n", "\n", "In this guide, we discussed tips and practices for doing research with JAX. We mainly focused on implementing a Trainer module that summarizes most common functionalities needed for training and testing models. To showcase the flexibility of the module, we implement a regression and classification model in a few lines. Moreover, we show how one can easily perform automatic hyperparameter optimization with Optuna in this setting. While this guide gives a possible template for a research code, it may not cover all parts that is needed for your specific usecase. Moreover, if you are specialized on a certain task, such as classification or generative models, you can further specialize the Trainer module to reduce your code for submodules." ] }, { "cell_type": "markdown", "id": "dad2e975", "metadata": {}, "source": [ "---\n", "\n", "[![Star our repository](https://img.shields.io/static/v1.svg?logo=star&label=⭐&message=Star%20Our%20Repository&color=yellow)](https://github.com/phlippe/uvadlc_notebooks/) If you found this tutorial helpful, consider ⭐-ing our repository. \n", "[![Ask questions](https://img.shields.io/static/v1.svg?logo=star&label=❔&message=Ask%20Questions&color=9cf)](https://github.com/phlippe/uvadlc_notebooks/issues) For any questions, typos, or bugs that you found, please raise an issue on GitHub. \n", "\n", "---" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.2" } }, "nbformat": 4, "nbformat_minor": 5 }