{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Tutorial 2 (JAX): Introduction to JAX+Flax\n", "\n", "![Status](https://img.shields.io/static/v1.svg?label=Status&message=Finished&color=green)\n", "\n", "**Filled notebook:** \n", "[![View filled on Github](https://img.shields.io/static/v1.svg?logo=github&label=Repo&message=View%20On%20Github&color=lightgrey)](https://github.com/phlippe/uvadlc_notebooks/blob/master/docs/tutorial_notebooks/JAX/tutorial2/Introduction_to_JAX.ipynb)\n", "[![Open filled In Collab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/phlippe/uvadlc_notebooks/blob/master/docs/tutorial_notebooks/JAX/tutorial2/Introduction_to_JAX.ipynb) \n", "**Author:** Phillip Lippe" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Welcome to our JAX tutorial for the Deep Learning course at the University of Amsterdam! The following notebook is meant to give a short introduction to JAX, including writing and training your own neural networks with [Flax](https://flax.readthedocs.io/en/latest/). But why should you learn JAX, if there are already so many other deep learning frameworks like [PyTorch](https://pytorch.org/) and [TensorFlow](https://www.tensorflow.org/)? The short answer: because it can be extremely fast. For instance, a small GoogleNet on CIFAR10, which we discuss in detail in [Tutorial 5](https://uvadlc-notebooks.readthedocs.io/en/latest/tutorial_notebooks/JAX/tutorial5/Inception_ResNet_DenseNet.html), can be trained in JAX 3x faster than in PyTorch with a similar setup. Note that for larger models, larger batch sizes, or smaller GPUs, a considerably smaller speedup is expected, and the code has not been designed for benchmarking. Nonetheless, JAX enables this speedup by compiling functions and numerical programs for accelerators (GPU/TPU) *just in time*, finding the optimal utilization of the hardware. Frameworks with dynamic computation graphs like PyTorch cannot achieve the same efficiency, since they cannot anticipate the next operations before the user calls them. For example, in an Inception block of GoogleNet, we apply multiple convolutional layers in parallel on the same input. JAX can optimize the execution of this layer by compiling the whole forward pass for the available accelerator and fusing operations where possible, reducing memory access and speeding up execution. In contrast, when calling the first convolutional layer in PyTorch, the framework does not know that multiple convolutions on the same feature map will follow. It sends each operation one by one to the GPU, and can only adapt the execution after seeing the next Python calls. Hence, JAX can make more efficient use of the GPU than, for instance, PyTorch. \n", "\n", "However, everything comes with a price. In order to efficiently compile programs just-in-time in JAX, the functions need to be written with certain constraints. Firstly, the functions are not allowed to have side-effects, meaning that they are not allowed to affect any variable outside of their namespaces. For instance, in-place operations affect a variable even outside of the function. Moreover, stochastic operations such as `torch.rand(...)` change the global state of pseudo random number generators, which is not allowed in functional JAX (we will see later how JAX handles random number generation). Secondly, JAX compiles the functions based on anticipated shapes of all arrays/tensors in the function. This becomes problematic if the shapes or the program flow within the function depends on the values of the tensor. For instance, in the operation `y = x[x>3]`, the shape of `y` depends on how many values of `x` are greater than 3. We will discuss more of these constraints in this notebook. Still, in most common cases of training neural networks, it is straightforward to write functions within these constraints.\n", "\n", "This tutorial is heavily inspired by many great JAX tutorials before, and a (non-exclusive) list of them are:\n", "\n", "* [JAX 101](https://jax.readthedocs.io/en/latest/jax-101/index.html) with many subtutorials on individual parts of JAX\n", "* [JAX - The Sharp Bits](https://jax.readthedocs.io/en/latest/notebooks/Common_Gotchas_in_JAX.html) discusses the constraints of JAX and how to overcome them\n", "* [Jax for the Impatient](https://flax.readthedocs.io/en/latest/notebooks/jax_for_the_impatient.html) for a quick intro to JAX with focus on deep learning\n", "* [Flax Basics](https://flax.readthedocs.io/en/latest/notebooks/flax_basics.html) as introduction to the Flax framework\n", "\n", "Throughout this tutorial, we will draw comparisons to PyTorch and also use its data loading library (see our [PyTorch tutorial](https://uvadlc-notebooks.readthedocs.io/en/latest/tutorial_notebooks/tutorial2/Introduction_to_PyTorch.html) for a refresher). JAX is not meant to 'redefine the wheel', so we can combine it framework-agnostic parts from PyTorch (e.g., data loading) and TensorFlow (e.g., logging in TensorBoard). Further, we use [Flax](https://flax.readthedocs.io/en/latest/) as a neural network library in JAX, and [Optax](https://optax.readthedocs.io/en/latest/index.html) to implement common deep learning optimizers. More on them later in the notebook. First, let's get started with some basic JAX operations." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "## Standard libraries\n", "import os\n", "import math\n", "import numpy as np \n", "import time\n", "\n", "## Imports for plotting\n", "import matplotlib.pyplot as plt\n", "%matplotlib inline \n", "from IPython.display import set_matplotlib_formats\n", "set_matplotlib_formats('svg', 'pdf') # For export\n", "from matplotlib.colors import to_rgba\n", "import seaborn as sns\n", "sns.set()\n", "\n", "## Progress bar\n", "from tqdm.auto import tqdm" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## JAX as NumPy on accelerators\n", "\n", "Every deep learning framework has its own API for dealing with data arrays. For example, PyTorch uses `torch.Tensor` as data arrays on which it defines several operations like matrix multiplication, taking the mean of the elements, etc. In JAX, this basic API strongly resembles the one of [NumPy](https://numpy.org/), and even has the same name in JAX (`jax.numpy`). So, for now, let's think of JAX as NumPy that runs on accelerators. As a first step, let's import JAX and its NumPy API:" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Using jax 0.3.13\n" ] } ], "source": [ "import jax\n", "import jax.numpy as jnp\n", "print(\"Using jax\", jax.__version__)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "At the current time of writing (June 2022), the newest JAX version is `0.3.13` which supports most of the common NumPy functionalities. The NumPy API of JAX is usually imported as `jnp`, to keep a resemblance to NumPy's import as `np`. In the following subsections, we will discuss the main differences between the classical NumPy API and the one of JAX." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Device Arrays\n", "\n", "As a first test, let's create some arbitrary arrays like we would do in NumPy. For instance, let's create an array of zeros with shape `[2,5]`:" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[[0. 0. 0. 0. 0.]\n", " [0. 0. 0. 0. 0.]]\n" ] } ], "source": [ "a = jnp.zeros((2, 5), dtype=jnp.float32)\n", "print(a)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Similarly, we can create an array with values of 0 to 5 by using `arange`:" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[0 1 2 3 4 5]\n" ] } ], "source": [ "b = jnp.arange(6)\n", "print(b)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You might now wonder whether the arrays `a` and `b` are simply NumPy arrays. To check that, let's print out the class of `b`:" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "jaxlib.xla_extension.DeviceArray" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "b.__class__" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Instead of a simple NumPy array, it shows the type `DeviceArray` which is what JAX uses to represent arrays. In contrast to NumPy, JAX can execute the same code on different backends – CPU, GPU and TPU. A `DeviceArray` therefore represents an array which is on one of the backends. Similar to PyTorch, we can check the device of an array by calling `.device()`:" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "GpuDevice(id=0, process_index=0)" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "b.device()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As you can see, the array `b` is already natively on a GPU although we did not specify this explicitly as you would do in PyTorch (on Colab, remember to select a GPU in your runtime environment). In order to change the device of an array, we can use `jax.device_get`:" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n" ] } ], "source": [ "b_cpu = jax.device_get(b)\n", "print(b_cpu.__class__)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Unsurprisingly, a simple CPU-based array is nothing else than a NumPy array, which allows for a simple conversion between the two frameworks! To explicitly push a NumPy array to the accelerator, you can use `jax.device_put`:" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Device put: on gpu:0\n" ] } ], "source": [ "b_gpu = jax.device_put(b_cpu)\n", "print(f'Device put: {b_gpu.__class__} on {b_gpu.device()}')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Nicely enough, JAX will handle any device clash itself when you try to perform operations on a NumPy array and a DeviceArray by modeling the output as DeviceArray again:" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "DeviceArray([ 0, 2, 4, 6, 8, 10], dtype=int32)" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "b_cpu + b_gpu" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Finally, we can also print all our available devices using `jax.devices()`:" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[GpuDevice(id=0, process_index=0), GpuDevice(id=1, process_index=0)]" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "jax.devices()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "A technical detail of running operations on DeviceArrays is that when a JAX function is called, the corresponding operation takes place asynchronously on the accelerator when possible. For instance, if we call `out = jnp.matmul(b, b)`, JAX first returns a placeholder array for `out` which may not be filled with the values as soon as the function calls finishes. This way, Python will not block the execution of follow-up statements, but instead only does it whenever we strictly need the value of `out`, for instance for printing or putting it on CPU. PyTorch uses a very similar principle to allow asynchronous computation. For more details, see [JAX - Asynchronous Dispatch](https://jax.readthedocs.io/en/latest/async_dispatch.html)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Immutable tensors\n", "\n", "When we would like to change a NumPy array in-place, like replacing the first element of `b` with `1` instead of `0`, we could simply write `b[0]=1`. However, in JAX, this is not possible. A `DeviceArray` object is *immutable*, which means that no in-place operations are possible. The reason for this goes back to our discussion in the introduction: JAX requires programs to be \"pure\" functions, i.e. no effects on variables outside of the function are allowed. Allowing in-place operations of variables would make the program analysis for JAX's just-in-time compilation difficult. Instead, we can use the expression `b.at[0].set(1)` which, analogous to the in-place operation, returns a new array which is identical to `b`, except that its value at the first position is 1. Let's try that out below:" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Original array: [0 1 2 3 4 5]\n", "Changed array: [1 1 2 3 4 5]\n" ] } ], "source": [ "b_new = b.at[0].set(1)\n", "print('Original array:', b)\n", "print('Changed array:', b_new)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "However, we said that JAX is very efficient. Isn't creating a new array in this case the opposite? While it is indeed less efficient, it can made much more efficient with JAX's just-in-time compilation. The compiler can recognize unnecessary array duplications, and replace them with in-place operations again. More on the just-in-time compilation later! " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Pseudo Random Numbers in JAX\n", "\n", "In machine learning, we come across several situations where we need to generate pseudo random numbers: randomly shuffling a dataset, sampling a dropout mask for regularization, training a VAE by sampling from the approximate posterior, etc. In libraries like NumPy and PyTorch, the random number generator are controlled by a seed, which we set initially to obtain the same samples every time we run the code (this is why the numbers are not truly random, hence \"pseudo\"-random). However, if we call `np.random.normal()` 5 times consecutively, we will get 5 different numbers since every execution changes the state/seed of the pseudo random number generation (PRNG). In JAX, if we would try to generate a random number with this approach, a function creating pseudo-random number would have an effect outside of it. To prevent this, JAX takes a different approach by explicitly passing and iterating the PRNG state. First, let's create a PRNG for the seed 42:" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [], "source": [ "rng = jax.random.PRNGKey(42)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now, we can use this PRNG state to generate random numbers. Since with this state, the random number generation becomes deterministic, we sample the same number every time. This is not the case in NumPy if we set the seed once before both operations:" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "JAX - Random number 1: -0.18471177\n", "JAX - Random number 2: -0.18471177\n", "NumPy - Random number 1: 0.4967141530112327\n", "NumPy - Random number 2: -0.13826430117118466\n" ] } ], "source": [ "# A non-desirable way of generating pseudo-random numbers...\n", "jax_random_number_1 = jax.random.normal(rng)\n", "jax_random_number_2 = jax.random.normal(rng)\n", "print('JAX - Random number 1:', jax_random_number_1)\n", "print('JAX - Random number 2:', jax_random_number_2)\n", "\n", "# Typical random numbers in NumPy\n", "np.random.seed(42)\n", "np_random_number_1 = np.random.normal()\n", "np_random_number_2 = np.random.normal()\n", "print('NumPy - Random number 1:', np_random_number_1)\n", "print('NumPy - Random number 2:', np_random_number_2)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Usually, we want to have a behavior like NumPy where we get a different random number every time we sample. To achieve this, we can *split* the PRNG state to get usable subkeys every time we need a new pseudo-random number. We can do this with `jax.random.split(...)`:" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "JAX new - Random number 1: 0.107961535\n", "JAX new - Random number 2: -1.2226542\n" ] } ], "source": [ "rng, subkey1, subkey2 = jax.random.split(rng, num=3) # We create 3 new keys\n", "jax_random_number_1 = jax.random.normal(subkey1)\n", "jax_random_number_2 = jax.random.normal(subkey2)\n", "print('JAX new - Random number 1:', jax_random_number_1)\n", "print('JAX new - Random number 2:', jax_random_number_2)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Every time you run this cell, you will obtain different random numbers for both operations since we create new PRNG states before sampling. In general, you want to split the PRNG key every time before generating a pseudo-number, to prevent accidentally obtaining the exact same numbers (for instance, sampling the exact same dropout mask every time you run the network makes dropout itself quite useless...). For a deeper dive into the ideas behind the random number generation in JAX, see JAX's tutorial on [Pseudo Random Numbers](https://jax.readthedocs.io/en/latest/jax-101/05-random-numbers.html)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Function transformations with Jaxpr\n", "\n", "Rosalia Schneider and Vladimir Mikulik summarize the key points of JAX in the [JAX 101 tutorial](https://jax.readthedocs.io/en/latest/jax-101/01-jax-basics.html) as follows: \n", "\n", "> The most important difference, and in some sense the root of all the rest, is that JAX is designed to be functional, as in functional programming. The reason behind this is that the kinds of program transformations that JAX enables are much more feasible in functional-style programs. [...] The important feature of functional programming to grok when working with JAX is very simple: don’t write code with side-effects.\n", "\n", "Essentially, we want to write our main code of JAX in functions that do not affect anything else besides its outputs. For instance, we do not want to change input arrays in-place, or access global variables. While this might seem limiting at first, you get used to this quite quickly and most JAX functions that need to fulfill these constraints can be written this way without problems. Note that not all possible functions in training a neural network need to fulfill the constraints. For instance, loading or saving of models, the logging, or the data generation can be done in naive functions. Only the network execution, which we want to do very efficiently on our accelerator (GPU or TPU), should strictly follow these constraints.\n", "\n", "What does make JAX functions so special, and how can we think about them? A good way of gaining understanding in how JAX handles function is to understand its intermediate representation: jaxpr. Conceptually, you can think of any operation that JAX does on a function, as first trace-specializing the Python function to be transformed into a small and well-behaved intermediate form. This means that we check which operations are performed on which array, and what shapes the arrays are. Based on this representation, JAX then interprets the function with transformation-specific interpretation rules, which includes automatic differentiation or compiling a function in XLA to efficiently use the accelerator.\n", "\n", "To illustrate this intermediate representation, let's consider the same simple function we used in the [PyTorch tutorial](https://uvadlc-notebooks.readthedocs.io/en/latest/tutorial_notebooks/tutorial2/Introduction_to_PyTorch.html) to discuss the concept of dynamic computation graphs:\n", "\n", "$$ y = \\frac{1}{|x|}\\sum_{i}\\left[\\left(x_i+2\\right)^2+3\\right]$$\n", "\n", "Using common NumPy operations in JAX, we can write it as follows:" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Input [0. 1. 2.]\n", "Output 12.666667\n" ] } ], "source": [ "def simple_graph(x):\n", " x = x + 2\n", " x = x ** 2\n", " x = x + 3\n", " y = x.mean()\n", " return y\n", "\n", "inp = jnp.arange(3, dtype=jnp.float32)\n", "print('Input', inp)\n", "print('Output', simple_graph(inp))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To view the jaxpr representation of this function, we can use `jax.make_jaxpr`. Since the tracing depends on the shape of the input, we need to pass an input to the function (here of shape `[3]`):" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{ lambda ; a:f32[3]. let\n", " b:f32[3] = add a 2.0\n", " c:f32[3] = integer_pow[y=2] b\n", " d:f32[3] = add c 3.0\n", " e:f32[] = reduce_sum[axes=(0,)] d\n", " f:f32[] = div e 3.0\n", " in (f,) }" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "jax.make_jaxpr(simple_graph)(inp)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "A jaxpr representation follows the structure:\n", "\n", "```python\n", "jaxpr ::= { lambda Var* ; Var+.\n", " let Eqn*\n", " in [Expr+] }\n", "```\n", "where `Var*` are constants and `Var+` are input arguments. In the cell above, this is `a:f32[3]`, i.e. an array of shape 3 with type `jnp.float32` (`inp`). The list of equations, `Eqn*`, define the intermediate results of the function. You can see that each operation in `simple_graph` is translated to a corresponding equation, like `x = x + 2` is translated to `b:f32[3] = add a 2.0`. Furthermore, you see the specialization of the operations on the input shape, like `x.mean()` being replacing in `e` and `f` with summing and dividing by 3. Finally, `Expr+` in the jaxpr representation are the outputs of the functions. In the example, this is `f`, i.e. the final result of the function.\n", "Based on these atomic operations, JAX offers all kind of function transformations, of which we will discuss the most important ones later in this section. \n", "Hence, you can consider the jaxpr representation is an intermediate compilation stage of JAX. What happens if we actually try to look at the jaxpr representation of a function with side-effect? Let's consider the following function, which, as an illustrative example, appends the input to a global list:" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{ lambda ; a:f32[3]. let\n", " b:f32[3] = integer_pow[y=2] a\n", " c:f32[] = reduce_sum[axes=(0,)] b\n", " d:f32[] = sqrt c\n", " in (d,) }" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "global_list = []\n", "\n", "# Invalid function with side-effect\n", "def norm(x):\n", " global_list.append(x)\n", " x = x ** 2\n", " n = x.sum()\n", " n = jnp.sqrt(n)\n", " return n\n", "\n", "jax.make_jaxpr(norm)(inp)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As you can see, the jaxpr representation of the function does not contain any operation for `global_list.append(x)`. This is because jaxpr only understand side-effect-free code, and cannot represent such effects. Thus, we need to stick with pure functions without any side effects, to prevent any unwanted errors in our functions. If you are interested in learning more about the jaxpr representation, check out the JAX [documentation](https://jax.readthedocs.io/en/latest/jaxpr.html) on it. But for this tutorial, we just need the basics as discussed above." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Automatic differentiation\n", "\n", "The intermediate jaxpr representation defines a computation graph, on which we can perform an essential operation of deep learning framework: automatic differentiation. In frameworks like PyTorch with a dynamic computation graph, we would compute the gradients based on the loss tensor itself, e.g. by calling `loss.backward()`. However, JAX directly works with functions. Instead of backpropagating gradients through tensors, JAX takes as input a function, and outputs another function which directly calculates the gradients for it. While this might seem quite different to what you are used to from other frameworks, it is quite intuitive: your gradient of parameters is really a function of parameters and data.\n", "\n", "The transformation that allows us to do this is `jax.grad`, which takes as input the function, and returns another function representing the gradient calculation of the (first) input with respect to the output:" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Gradient [1.3333334 2. 2.6666667]\n" ] } ], "source": [ "grad_function = jax.grad(simple_graph)\n", "gradients = grad_function(inp)\n", "print('Gradient', gradients)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The gradient we get here is exactly the one we would obtain when doing the calculation by hand. Moreover, we can also print the jaxpr representation of the gradient function:" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{ lambda ; a:f32[3]. let\n", " b:f32[3] = add a 2.0\n", " c:f32[3] = integer_pow[y=2] b\n", " d:f32[3] = integer_pow[y=1] b\n", " e:f32[3] = mul 2.0 d\n", " f:f32[3] = add c 3.0\n", " g:f32[] = reduce_sum[axes=(0,)] f\n", " _:f32[] = div g 3.0\n", " h:f32[] = div 1.0 3.0\n", " i:f32[3] = broadcast_in_dim[broadcast_dimensions=() shape=(3,)] h\n", " j:f32[3] = mul i e\n", " in (j,) }" ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "jax.make_jaxpr(grad_function)(inp)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This shows an unique property of JAX: we can print out the exact computation graph for determining the gradients. Compared to the original function, you can see new equations like `d:f32[3] = integer_pow[y=1] b` and `e:f32[3] = mul 2.0 d`, which model the intermediate gradient of $\\partial b_i^2/\\partial b_i = 2b_i$. Furthermore, the return value `j` is the multiplication of `e` with $1/3$, which maps to the gradient being:\n", "\n", "$$ \\frac{\\partial y}{\\partial x_i} = \\frac{2}{3}(x_i + 2)$$\n", "\n", "Hence, we can not only use JAX to estimate the gradients at a certain input value, but actually return the analytical gradient function which is quite a nice feature and highlights the properties of JAX!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Often, we do not only want the gradients, but also the actual output of the function, for instance for logging the loss. This can be efficiently done using `jax.value_and_grad`:" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(DeviceArray(12.666667, dtype=float32),\n", " DeviceArray([1.3333334, 2. , 2.6666667], dtype=float32))" ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" } ], "source": [ "val_grad_function = jax.value_and_grad(simple_graph)\n", "val_grad_function(inp)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Further, we can specialize the gradient function to consider multiple input arguments, and add extra outputs that we may not want to differentiate (for instance the accuracy in classification). We will visit the most important ones in the network training later in this section, and refer to other great resources for more details ([JAX Quickstart](https://jax.readthedocs.io/en/latest/notebooks/quickstart.html#taking-derivatives-with-grad), [Autodiff cookbook](https://jax.readthedocs.io/en/latest/notebooks/autodiff_cookbook.html), [Advanced autodiff](https://jax.readthedocs.io/en/latest/jax-101/04-advanced-autodiff.html)).\n", "\n", "To train neural networks, we need to determine the gradient for every parameter in the network with respect to the loss. Listing all parameters as input arguments quickly gets annoying and infeasible. JAX offers an elegant data structure to summarize all parameters: a pytree ([documentation](https://jax.readthedocs.io/en/latest/pytrees.html)). A pytree is a container-like object which structures its elements as a tree. For instance, a linear neural network might have its parameters organized similar to:\n", "\n", "```python\n", "params = {\n", " 'linear1': {\n", " 'weights': ...,\n", " 'bias': ...\n", " },\n", " ...\n", "}\n", "```\n", "JAX offers functions to process pytrees efficiently, such as obtaining all leafs (i.e. all parameters in a network) or applying a function on each element. We will come back to these structures when training a full network." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Speeding up computation with Just-In-Time compilation\n", "\n", "Interestingly, from the previous code cell, you can see in the jaxpr representation of the gradient function that calculating the array `f` and scalar `g` are unnecessary. Intuitively, the gradient of taking the mean is independent of the actual output of the mean, hence we could drop `f` and `g` without any drawback. Finding such cases to improve efficiency and optimizing the code to take full advantage of the available accelerator hardware is one of the big selling points of JAX. It achieves that by *compiling functions just-in-time* with [XLA](https://www.tensorflow.org/xla) (Accelerated Linear Algebra), using their jaxpr representation. Thereby, XLA fuses operations to reduce execution time of short-lived operations and eliminates intermediate storage buffers where not needed. For more details, see the [XLA documentation](https://docs.w3cub.com/tensorflow~guide/performance/xla/index). \n", "\n", "To compile a function, JAX provides the `jax.jit` transformation. We can either apply the transformation directly on a function (as we will do in the next cell), or use the decorator `@jax.jit` before a function." ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [], "source": [ "jitted_function = jax.jit(simple_graph)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The `jitted_function` takes the same input arguments as the original function `simple_graph`. Since the jaxpr representation of a function depends on the input shape, the compilation is started once we put the first input in. However, note that this also means that for every different shape we want to run the function, a new XLA compilation is needed. This is why it is recommended to use padding in cases where your input shape strongly varies (we revisit this topic in the final section of this tutorial). For now, let's create an array with 1000 random values, on which we apply the jitted function:" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [], "source": [ "# Create a new random subkey for generating new random values\n", "rng, normal_rng = jax.random.split(rng)\n", "large_input = jax.random.normal(normal_rng, (1000,))\n", "# Run the jitted function once to start compilation\n", "_ = jitted_function(large_input)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The output is not any different from what you would get from the non-jitted function. However, how is it about the runtime? Let's time both the original and the jitted function. Due to the asynchronous execution on the GPU, we add `.block_until_ready()` on the output, which blocks the Python execution until the accelerator (here GPU) finished computing the result and hence get an accurate time estimate." ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "598 µs ± 104 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)\n" ] } ], "source": [ "%%timeit\n", "simple_graph(large_input).block_until_ready()" ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "19.5 µs ± 52.8 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)\n" ] } ], "source": [ "%%timeit\n", "jitted_function(large_input).block_until_ready()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We see that the compiled function is almost 10-15x faster! This is quite an improvement in performance, and shows the potential of compiling functions with XLA. Furthermore, we can also apply multiple transformations on the same function in JAX, such as applying `jax.jit` on a gradient function:" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [], "source": [ "jitted_grad_function = jax.jit(grad_function)\n", "_ = jitted_grad_function(large_input) # Apply once to compile" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's time the functions once more:" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2.55 ms ± 190 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)\n" ] } ], "source": [ "%%timeit\n", "grad_function(large_input).block_until_ready()" ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "17.4 µs ± 60.6 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)\n" ] } ], "source": [ "%%timeit\n", "jitted_grad_function(large_input).block_until_ready()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Once more, the jitted function is much faster than the original one. Intuitively, this shows the potential speed up we can gain by using `jax.jit` to compile the whole training step of a network. Generally, we want to jit the largest possible chunk of computation to give the compiler maximum freedom. \n", "\n", "There are situations in which applying jit to a function is not straight-forward, for instance, if an input argument cannot be traced, or you need to use loops that depend on input arguments. To keep the tutorial simple, and since most neural network training functions do not run into these issues, we do not discuss such special cases here. Instead, we refer to the section on just-in-time compilation in the great tutorials of [JAX 101 Tutorial](https://jax.readthedocs.io/en/latest/jax-101/02-jitting.html), [JAX Quickstart](https://jax.readthedocs.io/en/latest/notebooks/quickstart.html#using-jit-to-speed-up-functions), and [Thinking in JAX](https://jax.readthedocs.io/en/latest/notebooks/thinking_in_jax.html#to-jit-or-not-to-jit)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Implementing a Neural Network with Flax\n", "\n", "With having reviewed the basics of JAX, we are now ready to implement our own neural network. Technically, we could implement our own neural network from scratch with JAX (see [here](https://jax.readthedocs.io/en/latest/notebooks/Neural_Network_and_Data_Loading.html) for an example), but we do not really want to do that every time. Similarly to PyTorch's `torch.nn` package, there exist neural network libraries based on JAX which provide such basic functionality. A (non-exclusive) collection of them are:\n", "\n", "* [Flax](https://flax.readthedocs.io/en/latest/index.html), started by the Google Brain Team, focuses on flexibility and clarity.\n", "* [Haiku](https://dm-haiku.readthedocs.io/en/latest/), from DeepMind, focuses on simplicity and compositionality.\n", "* [Trax](https://github.com/google/trax), maintained by the Google Brain Team, provides solutions for common training tasks\n", "* [Equinox](https://github.com/patrick-kidger/equinox), created by Patrick Kidger and Cristian Garcia, implements neural networks as callable PyTrees\n", "* [Jraph](https://github.com/deepmind/jraph), from DeepMind, is a graph neural network library (similar to PyTorch Geometric)\n", "\n", "For this tutorial series, we will use Flax due to its flexibility, intuitive API, and larger community. However, this should not mean that the other libraries are necessarily worse, and we recommend giving them a try as well to find the best library for yourself!\n", "\n", "We will introduce the libraries and all additional parts you might need to train a neural network in Flax, using a simple example classifier on a simple yet well known example: XOR. Given two binary inputs $x_1$ and $x_2$, the label to predict is $1$ if either $x_1$ or $x_2$ is $1$ while the other is $0$, or the label is $0$ in all other cases. The example became famous by the fact that a single neuron, i.e. a linear classifier, cannot learn this simple function. Hence, we will learn how to build a small neural network that can learn this function. \n", "To make it a little bit more interesting, we move the XOR into continuous space and introduce some gaussian noise on the binary inputs. Our desired separation of an XOR dataset could look as follows:\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### The model\n", "\n", "The package `flax.linen` defines a series of useful classes like linear networks layers, convolutions, activation functions etc. A full list can be found [here](https://flax.readthedocs.io/en/latest/flax.linen.html). In case you need a certain network layer, check the documentation of the package first before writing the layer yourself as the package likely contains the code for it already. We import it below:" ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [], "source": [ "try:\n", " import flax\n", "except ModuleNotFoundError: # Install flax if missing\n", " !pip install --quiet flax\n", " import flax\n", "\n", "from flax import linen as nn" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### nn.Module\n", "\n", "Similar to PyTorch, a neural network is built up out of modules. Modules can contain other modules, and a neural network is considered to be a module itself as well. The basic template of a module is as follows:" ] }, { "cell_type": "code", "execution_count": 29, "metadata": {}, "outputs": [], "source": [ "class MyModule(nn.Module):\n", " # Some dataclass attributes, like hidden dimension, number of layers, etc. of the form:\n", " # varname : vartype\n", " \n", " def setup(self):\n", " # Flax uses \"lazy\" initialization. This function is called once before you\n", " # call the model, or try to access attributes. In here, define your submodules etc.\n", " pass\n", " \n", " def __call__(self, x):\n", " # Function for performing the calculation of the module.\n", " pass" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The main, obvious difference to PyTorch is that Flax uses lazy initialization. The function `setup` is called once on a module instance before any other methods are called, or when you try to access a attribute of *self* defined in setup. Additional object attributes are defined below the class name. However, in contrast to PyTorch, the parameters are not part of the module. Instead, we can create a set of parameters of the module by calling its `init()` function. This function takes as input a PRNG state for sampling pseudo-random numbers and an example input to the model, and returns a set of parameters for the module as a pytree. Further, since the init function requires an input to the network, we can infer the input shape for all modules and do not need to explicitly define it during the module creation. This becomes more clear in the example we show in a second.\n", "\n", "The `__call__` method represents the `forward` function in PyTorch, and performs the actual computation of the module. It can take additional arguments if needed, like whether we are training or validation." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Simple classifier\n", "\n", "To get an intuition behind how we work with modules in Flax, let's define our own small neural network. We will use a minimal network with a input layer, one hidden layer with tanh as activation function, and a output layer. In other words, our networks should look something like this:\n", "\n", "
\n", "\n", "The input neurons are shown in blue, which represent the coordinates $x_1$ and $x_2$ of a data point. The hidden neurons including a tanh activation are shown in white, and the output neuron in red.\n", "In Flax, we can define this as follows:" ] }, { "cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [], "source": [ "class SimpleClassifier(nn.Module):\n", " num_hidden : int # Number of hidden neurons\n", " num_outputs : int # Number of output neurons\n", "\n", " def setup(self):\n", " # Create the modules we need to build the network\n", " # nn.Dense is a linear layer\n", " self.linear1 = nn.Dense(features=self.num_hidden)\n", " self.linear2 = nn.Dense(features=self.num_outputs)\n", "\n", " def __call__(self, x):\n", " # Perform the calculation of the model to determine the prediction\n", " x = self.linear1(x)\n", " x = nn.tanh(x)\n", " x = self.linear2(x)\n", " return x" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "One thing you may notice is that usually, all layers that we define in `setup` are also used in the `__call__` function. To reduce the code overhead, Flax provides an alternative, more compact network creation with `nn.compact`. With that, we can remove the setup function and instead our model as:" ] }, { "cell_type": "code", "execution_count": 31, "metadata": {}, "outputs": [], "source": [ "class SimpleClassifierCompact(nn.Module):\n", " num_hidden : int # Number of hidden neurons\n", " num_outputs : int # Number of output neurons\n", "\n", " @nn.compact # Tells Flax to look for defined submodules\n", " def __call__(self, x):\n", " # Perform the calculation of the model to determine the prediction\n", " # while defining necessary layers\n", " x = nn.Dense(features=self.num_hidden)(x)\n", " x = nn.tanh(x)\n", " x = nn.Dense(features=self.num_outputs)(x)\n", " return x" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The `nn.compact` annotation of the `__call__` method signals Flax to look for submodules that we define in the forward pass. These are automatically recognized as such, so that we can use them for initialization etc. Which of the two model definition you use is often up to you (see the [Flax documentation](https://flax.readthedocs.io/en/latest/design_notes/setup_or_nncompact.html) for some pros and cons for both methods). In the following tutorials, we will mostly use the compact version, but occasionally come back to the explicit setup function where necessary. For instance, if we define more functions on a module besides `__call__` and want to reuse some modules, it is recommended to use the setup version." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For the examples in this notebook, we will use a tiny neural network with two input neurons and eight hidden neurons. As we perform binary classification, we will use a single output neuron. Note that we do not apply a sigmoid on the output yet. This is because other functions, especially the loss, are more efficient and precise to calculate on the original outputs instead of the sigmoid output. We will discuss the detailed reason later.\n", "\n", "Now, let's create an instance of this network:" ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "SimpleClassifier(\n", " # attributes\n", " num_hidden = 8\n", " num_outputs = 1\n", ")\n" ] } ], "source": [ "model = SimpleClassifier(num_hidden=8, num_outputs=1)\n", "# Printing the model shows its attributes\n", "print(model)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "At this stage, the model has no parameters initialized. To do this, let's create a random input of our dataset, and apply the init function:" ] }, { "cell_type": "code", "execution_count": 33, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "FrozenDict({\n", " params: {\n", " linear1: {\n", " kernel: DeviceArray([[ 0.31476864, -0.4647768 , -0.7862042 , -0.48842615,\n", " -0.65373844, 0.3892545 , 0.3038056 , 0.04179859],\n", " [-0.3298236 , 1.1110363 , 0.54909396, -0.8168818 ,\n", " 0.40057245, -0.8665987 , 1.2087964 , 1.0364622 ]], dtype=float32),\n", " bias: DeviceArray([0., 0., 0., 0., 0., 0., 0., 0.], dtype=float32),\n", " },\n", " linear2: {\n", " kernel: DeviceArray([[-0.27971813],\n", " [-0.7466775 ],\n", " [ 0.29791608],\n", " [-0.26267236],\n", " [-0.5084385 ],\n", " [ 0.04573093],\n", " [-0.47257012],\n", " [ 0.50458497]], dtype=float32),\n", " bias: DeviceArray([0.], dtype=float32),\n", " },\n", " },\n", "})\n" ] } ], "source": [ "rng, inp_rng, init_rng = jax.random.split(rng, 3)\n", "inp = jax.random.normal(inp_rng, (8, 2)) # Batch size 8, input size 2\n", "# Initialize the model\n", "params = model.init(init_rng, inp)\n", "print(params)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now, we have parameters with which we can apply the network. We see that the parameters follow the same structure as defined in our module, and each linear layer contains one `kernel`, i.e. the weights, and a bias parameter. With this, we could apply the model on an input using the `apply` function:" ] }, { "cell_type": "code", "execution_count": 34, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "DeviceArray([[-0.48368204],\n", " [ 0.04365474],\n", " [ 0.06668529],\n", " [-0.34203646],\n", " [ 0.4835147 ],\n", " [ 0.37424874],\n", " [ 0.14232653],\n", " [-0.5916512 ]], dtype=float32)" ] }, "execution_count": 34, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model.apply(params, inp)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The model returns an output array of shape `[8,1]`, which corresponds to the one output neuron in the model for all 8 batch elements. With that, we now know how to initialize a model, and run a model. Next, let's look at the data." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### The data\n", "\n", "As mentioned before, JAX is not meant to 'reinvent the wheel' for every part of the deep learning pipeline. Hence, JAX and Flax do not natively provide a data loading functionality, but instead refer to other available libraries like Tensorflow and PyTorch. Here, let's use again the package `torch.utils.data` library." ] }, { "cell_type": "code", "execution_count": 35, "metadata": {}, "outputs": [], "source": [ "import torch.utils.data as data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The data package defines two classes which are the standard interface for handling data in PyTorch: `data.Dataset`, and `data.DataLoader`. The dataset class provides an uniform interface to access the training/test data, while the data loader makes sure to efficiently load and stack the data points from the dataset into batches during training." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### The dataset class\n", "\n", "The dataset class summarizes the basic functionality of a dataset in a natural way. To define a dataset in PyTorch, we simply specify two functions: `__getitem__`, and `__len__`. The get-item function has to return the $i$-th data point in the dataset, while the len function returns the size of the dataset. For the XOR dataset, we can define the dataset class as follows:" ] }, { "cell_type": "code", "execution_count": 36, "metadata": {}, "outputs": [], "source": [ "class XORDataset(data.Dataset):\n", "\n", " def __init__(self, size, seed, std=0.1):\n", " \"\"\"\n", " Inputs:\n", " size - Number of data points we want to generate\n", " seed - The seed to use to create the PRNG state with which we want to generate the data points\n", " std - Standard deviation of the noise (see generate_continuous_xor function)\n", " \"\"\"\n", " super().__init__()\n", " self.size = size\n", " self.np_rng = np.random.RandomState(seed=seed)\n", " self.std = std\n", " self.generate_continuous_xor()\n", "\n", " def generate_continuous_xor(self):\n", " # Each data point in the XOR dataset has two variables, x and y, that can be either 0 or 1\n", " # The label is their XOR combination, i.e. 1 if only x or only y is 1 while the other is 0.\n", " # If x=y, the label is 0.\n", " data = self.np_rng.randint(low=0, high=2, size=(self.size, 2)).astype(np.float32)\n", " label = (data.sum(axis=1) == 1).astype(np.int32)\n", " # To make it slightly more challenging, we add a bit of gaussian noise to the data points.\n", " data += self.np_rng.normal(loc=0.0, scale=self.std, size=data.shape)\n", "\n", " self.data = data\n", " self.label = label\n", "\n", " def __len__(self):\n", " # Number of data point we have. Alternatively self.data.shape[0], or self.label.shape[0]\n", " return self.size\n", "\n", " def __getitem__(self, idx):\n", " # Return the idx-th data point of the dataset\n", " # If we have multiple things to return (data point and label), we can return them as tuple\n", " data_point = self.data[idx]\n", " data_label = self.label[idx]\n", " return data_point, data_label" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Note that in contrast to our [PyTorch tutorial](https://uvadlc-notebooks.readthedocs.io/en/latest/tutorial_notebooks/tutorial2/Introduction_to_PyTorch.html#The-data), we use NumPy to generate the random data. Similar to JAX, NumPy also allows the pseudo-number generation based on a PRNG state. Hence, for better reproducibility, we are doing the same here. Let's try to create such a dataset and inspect it:" ] }, { "cell_type": "code", "execution_count": 37, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Size of dataset: 200\n", "Data point 0: (array([-0.06800247, 1.0232254 ], dtype=float32), 1)\n" ] } ], "source": [ "dataset = XORDataset(size=200, seed=42)\n", "print(\"Size of dataset:\", len(dataset))\n", "print(\"Data point 0:\", dataset[0])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To better relate to the dataset, we visualize the samples below. " ] }, { "cell_type": "code", "execution_count": 38, "metadata": {}, "outputs": [], "source": [ "def visualize_samples(data, label):\n", " data_0 = data[label == 0]\n", " data_1 = data[label == 1]\n", " \n", " plt.figure(figsize=(4,4))\n", " plt.scatter(data_0[:,0], data_0[:,1], edgecolor=\"#333\", label=\"Class 0\")\n", " plt.scatter(data_1[:,0], data_1[:,1], edgecolor=\"#333\", label=\"Class 1\")\n", " plt.title(\"Dataset samples\")\n", " plt.ylabel(r\"$x_2$\")\n", " plt.xlabel(r\"$x_1$\")\n", " plt.legend()" ] }, { "cell_type": "code", "execution_count": 39, "metadata": {}, "outputs": [ { "data": { "application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1BhZ2VzIDIgMCBSIC9UeXBlIC9DYXRhbG9nID4+CmVuZG9iago4IDAgb2JqCjw8IC9FeHRHU3RhdGUgNCAwIFIgL0ZvbnQgMyAwIFIgL1BhdHRlcm4gNSAwIFIKL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gL1NoYWRpbmcgNiAwIFIKL1hPYmplY3QgNyAwIFIgPj4KZW5kb2JqCjEwIDAgb2JqCjw8IC9Bbm5vdHMgWyBdIC9Db250ZW50cyA5IDAgUgovR3JvdXAgPDwgL0NTIC9EZXZpY2VSR0IgL1MgL1RyYW5zcGFyZW5jeSAvVHlwZSAvR3JvdXAgPj4KL01lZGlhQm94IFsgMCAwIDI4My44MTg3NSAyODAuOTgwNjI1IF0gL1BhcmVudCAyIDAgUiAvUmVzb3VyY2VzIDggMCBSCi9UeXBlIC9QYWdlID4+CmVuZG9iago5IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTEgMCBSID4+CnN0cmVhbQp4nLWbS7MbR3KF9/gVvdQs0Kx3ZS2lkUcRjvBiPIzwmkFzbDHIsSWOHz/f36kG0A3chDwaXUpBSSg1qqsyT548mVWIy8fTm2/j8m9fFv6xhOUjf/6H//5Bn0+BT59PyfJq0Xrl06fDp2RhHRZaqgyH+4//fjr9+cTn2FvpoZotjx/KCHG00G35WS/+4cUDtw+nh6dPp5rXMpdQ4tpL03+xzN7W9jD66TCa6lijXYZvM9yNzlX/tDjTp5TXtKTY11KWnz8s/7L8ZXnzbdoM94/8+cifzXBvvv/w3z++//DPP3x3ev/lNNLac2p3K72O3b379KfTH5efrpOGNVYccp13fvzhMnr67u3y5g9xiXF5++eT1TX0lmsow/KSypr7XPvbfz19E9bwu+Xtx+Uf3s65f9O+Tvu+FvYVG1/sxXK+29ph+FV2F2uV+6qllkJ/sb36tbaXslDeQ3vA2D78KttLqa2xBYull0fnxd15P50iIXIOTBerrXU+M9Y6ZrS9/8ykitW15O27b/6Qlpj0gm/+V3Oc+hpDrHr4HNfR7PpYXmwt87k4n5MlX92Wt7mqCKIGtn6M1n30V9nxYJOU8MTce+lrKL3GYLX1zTAHa7e1pGzJYhKVbQhK133/BM1oEeK8/b+Ygu+fv/35x3efzp9//Mt/fVm+/w/Z6OtZqfd1ZN6f76y0j/4GtNnaJmJ6XsdILWY4tX9tprjNFUNZ+XeK7W5nh+FX2FoMwCn3DnHkZkcnf9295bhaCjna/d724dfYW+qrCb/d+m1v5evvDcrprZZ0v7Xb6GvsrIKB1EcKwGBcdta+/s5Inj2Gmu5j7TD8GnszVtxzSKnWa7DZV99aimltdVL+cWuH4VfYWgpggP2lYqM/Jq2vt7U8eH/oqd9vbR9+ja1lhEfLKfVETr7sLe0JWXnnrBwRwcpQ7okFVm1zj6+SkdNvychwHlo8jxRrJ7GVTUmzKf5Pu2rpIum9GyVNm+zfW47fOz1875J/ZYbRVms9j24jjKXZmkMdvesPllje/FNQzrw+fU6S8yx9BBsYhe22Ua2Ti7zHEVyx5pJqL9jvHMc64Igs1zR39rXDSSUHiCkKGqyqE3nJeRjcriGH1nvrZJ/I5CEMEgf0Wp3n+5pb7dh7AJYEHAZrCTNzOSvJa4fOch2WkpyarZYYWnnyMPVOzmlIWuJQnusDWZ+9dRDDtcUwQhAKzsXWOGqPxZjemzw2agTWMmIOPfNZhukECoshjzgvAGyhhGwlEE0xk3ZAgtUcfZfGUtaUY7aGk/Q5p9VCxwej1Ox8Y6w5hYikryDmTNDlFIkGPDC85RSUPoFdcyo8wINrD/xV5Sx/PZiTxYLKMmTcyNtwwOiuOTuKmoRaAVZZ6lqMXVTDd+7c2Si4gEBusSh6kfCDlMx3SvKMyYQBRUoFS312bitpBe+CeN9X6LzQS4kjUgZHInCwB4DgGR5wo3yNLEx2ycvArIW/WXv2gyOtg4USFDWNBULBP4mQdtetWCNvmdmoyM5M1qKsw0XFxwD1UG0lMVuz6YNIuIwCFpKLAQkIyh4rGMTwKi9ooJKdUFZ5z09iQU0VdsvjawpD5Ny6F9tnxXaFkQBkF3FQaMFlsYYWvckhPoKtE/tY+gycBzRCVLleQtVRtFXwxz+A5wpcKOEyAektpTT5pbc4sLfCT30S/hoKFmd23IiMhT6JOaE3xJDwGmB2Dc9iwCt8Km4RQRJ4wC27YD8PhFbBo/iRtaAoU+l8CNTsLnhjI7lnJCzVN6kuE4wNmnqyFJhpJBxObSGiaaym1CZbRnc1EAUUSTIg9tk5loTpM3kxeS/Ah/hUyixtLs0IR2M3LtqJy4BNwCMko6TZcw/mbjMWqL8CpQICF9FXALZZtOCCK7LSJjpkxy1ujErYsphMKvG+QdJLcFigUG36lOooTUHuz0+wZcONRsbVTgisWsgfPg0AKLaZYyOyxa0IEiE9utxOjWzYePB+vDhzIDbEkqm76MUWsB2xkCAVFkYFTdRafpJryEbosVRjwUCyVNPOjX8PHwIGYIkeYFU3dUAC7vi/+6wBXlPCUWiGBTprFREqKHtrQQxAS7iUTS9FOg3OgF9cBgDqQbQ/hlLSSqYB9h0zPXEQ+g3VAO1DvCsQjABGW3CmJoHWkRUFBkNE8l0X6k1m8RfelOFldZK24IWHVULwTl8/CI4zjiEKoirjJShnDBs+Bipw5e0kglbm+pAS5CXkgZ8KAA27HewiC78RMo5kTCVw16ssOHWWT/iRxiaeeVepfHLDg/2SkxoQBFZVH9ADwZorB9BhrNyAWJc1pfgQT6VkP82ov0S6hq4hFZmytCyRx+LdvSq/oyykaCXaUTKG4REy3mKmuCKBdVH2lvf4zPfh+1bc5SsLk+lano0oFgZKUcvEVHWje0AWaGR836ScEH6lIENdCgbF0JKhxSDRRfUBCU/Jw03xtkp4F1Y6E41E35ic5rLvWcJrvp0QGgIBNMZ3gWV3Q0rBbOhvKsQRy0zJ+FVqBXHo6uuCwoPGpO15VW1ID2uu3SVOqiRwqCjms+BI7JJW3ZQ6BW0u1C5ReUY+6iJfINp9r6oQLCZJilS7RKBhScbcbBMphQtpHWDGjVXFxqJh//Gk0qRLepkAj8ajOAi1PiMysjsYb1hDm4XRDG5A6fmQjMqMpGkKig2RUbMXGdT7Av8/N+wuOlNykiQDCsNciHUUrYR4Jly1VUDWMrmE7OqvvkGWPEHpWMYkD0UIRRzZ1n2ByEZFKJDvbUZsHzJ9L27OycI83h1KGlvRB/hRV9FfTm9rUk2bSpjLKVi3Im0pR4tfH6iB06SCETMELDGYUCFkITfDEicwSJbKbhAY3C+Ngmx9ElUBMBowRrQXyRoMarlKf+A1r3iqlGOkawEaYROohppKlv6M0Jr0QVGeRBfI/mxAprWmctCLFtIJu0M8tyAHEJiUU5mPvrgR+Em0KBaMWbfPMANaCH3uupgpEYchZxHzIhd0I+HNLOqbtCK5q6q6qhdUdGuygkSSMPW20Ci7kmAjqyihQlzUs+OZViAHjpqpMBC30wtoUcyToK/qBY2oNaq9M/Oh9ALlO3LDFxdgVN2bWR2lWfqgzijB2fCTEGbB1NNgYkyLSkIAumkjf8NVqK7QiTKDXojUga3QAX69xoLgKxKPahIp3oBuBIyuYqBqANARVUxQsXhy3VCL3YcDhQJiR4e2ttkSMje1fEp4kihwFnAYisPlrH5WN+SR3zxA7IjdMI2RGqTEyW/MTCryVg5bYkgFN6bWYQMGbNQbfuujSNhRgXS4oC4qZ5BQOok+rvvv68hZu/yljlw9nnVjVagZOq1eR27/3nL83unhe4eOnGUI3dBPlDyD5YhgWKB6bpdtxAcCJe2LkkdsE9h6GFXenIfPWEgNChI19lbBRpgFVbfOw4IxsqyBMZWjk3ooa4kwwmV4sytxSpCAGaY/Uz1UpS2y6PBWM9WcWiBN7pL6kY4FHlfauXu6w5SsHCIjD54lMykJDRFRvaUgBJtSrEpSm+2QIrnTZkPBWUoSD5PUyUKdlE7VPkmzulZUTDT1JbtRXmOoIm9CmSlccXn/BXBOuUECgsQ7dmpEZEozCL3VJ/XIMmxqyuKsXRYMm9OctQNh0NLIKEHdEsNNUP6g0BvuatTtASYS3VGnk4PNUBNKM7g4mBIkSOEkmV7NHr6prkO7thQfXgDMqOsGUj2rJUpZiZxhCgzmvQF6pPiiJgHlWIcMB50I1MGfnvpEOneoI6CKAsPwCRt5uIlqWfeE+NqQEFeeG5Iv1/rt3vY6iAAEcaBbtnIFbQIVDw/ywGQWmxD9XLvODNmjtFpwQ8SmPsMqCXWhrlwkoMCpt/LzoBRU5UPqHMpDaB8pmeZODVeG1irv7UO8XYYOMCiulIpcUOo0CVSihi5pkI2wdNXZ3vM9gfKJmTJ7DwKRqSVp5rpJD+TZyoXi1XIX17A+5J7LNl26Zai+GBfRQCkENaTUvJBV2x2FEtQfGAqSYhhK5Yy7+qy2TR/kJeKCAoWa2XRxI7pzS0EUaZSYwqxTg1QUEdOHuatHV6IPyLNpOxbS2bFq/oKOdGGGyqeuyuqubJAm3UOzsVbXmGqwDp3VESML9Y3IIVOJBg84sjUpGb6jBlNKzhQg5EI/g8wqmAhBEqkymUUBmGG3cbiEg2O7jnpNh4ez5EjTszCsOz25Di/1pJMiuJXkTD4P0YU8D0PqhiVmPSDhhAIg2qNHZSoNpe6j6pEJGj7raIqU4oOy6o6JJbVCMwqhoJRxgfnxN49GIDIUPmte9BHmkIta8rMIjtEZkwrlmTERLFTMgUK1+6YvLBghTe1L7C0DAdMpjTN+cKm+DVyjZl6YRblwmQWcqje4QbuOPC8phbn5WV2pN+Zh8rzdTwpVxpy8Ld2e1bjs1TWQrUgNyI+kXeYJBb7FU725eRPCodYrZOGsfN90zITATe7KL402JHchVW29SJzQGajdwwJQIYlVHY3N21g1m+RE9QO86lriTPKibaI1NLgAjZ79lDl7ylXYykIl3+qTavynQTE+IbtDxlu9pCpbROuKsrzKKeRK1bDbiVYqqh1czGdV/CKOGki/lwNBPpD9XaLneeJBD/fQJpPwLmrQ0lw7nnXnAdBHHWyrayy1YBJ1oQ9PrYg+KLUiM0ppgyISRSawgtoS3hvmXa6uG0zACl1H+kuqwPwQUWsLPoAaVT7qI5hQTWzNXU6ZWo+Xh5G3vpcF8TgG81Hfp/hDaeegxEYiChQ/uh/jhoncBfmqv1vHbBgQ7BGHPMs8VayhzIoyZ7vU+2gv82AJ5nGLpMxsXBEwOgnqpUbfNjrLktYu6kDOU2vIk+KtJ1cJdbXryTnE0OwlBAQFqAtVV+68L0jLSKhurR9dWtRBt2LyGX0oinCtZK5wYLPTa8H11FnNErJ7ADgABbc1HT/e+hoPVK/TJlOzqsfLyygFZlvDd+sU3TpEJDFJpINqVZotd5/tqVaAfFDfYy5NykvbbS496bYiEV6QCLFdWhCgBr+l7K7HVtUHEEeBNER+80hAFxKcpxHMJv0LZk1kZZSyRF9yi5epyNUqgHNk96JjaJHN8IWTDvogVXVwlBWK5DT1CIsvLgpkC7OQxCLsAzoBRaTA4hYvyuBB7Z6uHHJW70OFF4hwQXmWkCT0hzaA59VOSlpN0t07L0ZIeUPdVMWcDpZ1cYGICT6b6dJvSWq/qGF/TthyzJyWXDJW0VsJCF5fmf5MRs46ULD9issLFT1VZdQdiFl3Uv3oUrDP9XKVrkJQPA2UnhJ4nQ3EduuuvCjudFMYKtWXB1wmWUoadalJfW74gg1UCTME16ROXwspv5uOGIw4gfiq8tSYB6Gu+tDVDEOWkThqmKfRFQ6M6mY/EfQ9IolR6badQlHGSggld+3qo1MVZeF86kBd+2hRIfakwE/SnEgPYCLJqsgqaqw/k3LIMFig6kxC6kKS1cg9bpDglwQXkAFJhRKKRTqud3cxoppZqajsvlAPCURd6RiPpv/j8WJb0o05CPjFTw/uLswdb7D7P3w4/cn9+cTnZz+f4Plf8RuM49OHaX5p9sDG/qZbe/NO3dSwebsNj7a7zqRLe9+/++u7Lx/+unx59/k/P334cr3A9+bbvDXl/vZfqHxUrf3wO5UXP3G5/50K8b6qCXORHHU2wdh2AzmX+8G3Yf1URYfrOss8jgO4Nd4eBm0IyLtBpMH25PvTYXjWY+M68T7c1nnuTwF4eJvq5u3hw9L2wffHjezDnxiu69QZ+ThOslTZne5edxy8LU0T34b3jXy6G75t+vi23T6ujd/rpzzfnZzLkMv/exny5UXIX7xAeTqleXlorkBN+M18eDmqXUGOGcdxjSKwxjbIJOTBeSuX+ktkXrd7SnW6U/0USdbtPAMy0GFJ0tWny+v6qgTd2t2gXd+lGfbhIT4ptr3tNu2Y9B3rcQ1tdiri2Ga4rZdhmF03oo5ba3Hf2s0O+yAzBHWRqo4Wjw8HhB51c7ybV+dIVLH5YQ1BZ2jzLt9xwYy+3Nth8GiHw/Bus+O0u3n3Ndz5Yl/vwW+HrR1d7ODhgsgLp/3SDWSye7h9u4TrDeJvfv/p3RewfGWwp+cCL84Tfu05whHQWX2akV8Aeh8/AEQnA0XNsSOYGGwBdfIIaBKEfkh1D2gplaybE3eD6fquO0DPGqqMB0DPdosKybs1lAmaR0AXHcrl9ABo3Q24bu1mh33wHtCHh3eAHObdwXS3hgPyDgveQXrY22HwaIfD8G6z47S7eQ9rOPrisN7db4etHV3s4OHvAbS0Z7n9wG0DdLz9XuD0f+K23WgKZW5kc3RyZWFtCmVuZG9iagoxMSAwIG9iago0NzQwCmVuZG9iagoxOCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDkyID4+CnN0cmVhbQp4nD2MsQ3AMAgEe6b4BSJhjG3YJ0rl7N/mLSdp4PQP19KgOKxxdlU0HziLfHhL9YSNxJSmlUdTnN3aFg4rgxS72BYWXmERpPJqmPF5U9XAklKU5c36f3c9x6sbugplbmRzdHJlYW0KZW5kb2JqCjE2IDAgb2JqCjw8IC9CYXNlRm9udCAvRGVqYVZ1U2Fucy1PYmxpcXVlIC9DaGFyUHJvY3MgMTcgMCBSCi9FbmNvZGluZyA8PCAvRGlmZmVyZW5jZXMgWyAxMjAgL3ggXSAvVHlwZSAvRW5jb2RpbmcgPj4gL0ZpcnN0Q2hhciAwCi9Gb250QkJveCBbIC0xMDE2IC0zNTEgMTY2MCAxMDY4IF0gL0ZvbnREZXNjcmlwdG9yIDE1IDAgUgovRm9udE1hdHJpeCBbIDAuMDAxIDAgMCAwLjAwMSAwIDAgXSAvTGFzdENoYXIgMjU1IC9OYW1lIC9EZWphVnVTYW5zLU9ibGlxdWUKL1N1YnR5cGUgL1R5cGUzIC9UeXBlIC9Gb250IC9XaWR0aHMgMTQgMCBSID4+CmVuZG9iagoxNSAwIG9iago8PCAvQXNjZW50IDkyOSAvQ2FwSGVpZ2h0IDAgL0Rlc2NlbnQgLTIzNiAvRmxhZ3MgOTYKL0ZvbnRCQm94IFsgLTEwMTYgLTM1MSAxNjYwIDEwNjggXSAvRm9udE5hbWUgL0RlamFWdVNhbnMtT2JsaXF1ZQovSXRhbGljQW5nbGUgMCAvTWF4V2lkdGggMTM1MCAvU3RlbVYgMCAvVHlwZSAvRm9udERlc2NyaXB0b3IgL1hIZWlnaHQgMCA+PgplbmRvYmoKMTQgMCBvYmoKWyA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMAo2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDMxOCA0MDEgNDYwIDgzOCA2MzYKOTUwIDc4MCAyNzUgMzkwIDM5MCA1MDAgODM4IDMxOCAzNjEgMzE4IDMzNyA2MzYgNjM2IDYzNiA2MzYgNjM2IDYzNiA2MzYgNjM2CjYzNiA2MzYgMzM3IDMzNyA4MzggODM4IDgzOCA1MzEgMTAwMCA2ODQgNjg2IDY5OCA3NzAgNjMyIDU3NSA3NzUgNzUyIDI5NQoyOTUgNjU2IDU1NyA4NjMgNzQ4IDc4NyA2MDMgNzg3IDY5NSA2MzUgNjExIDczMiA2ODQgOTg5IDY4NSA2MTEgNjg1IDM5MCAzMzcKMzkwIDgzOCA1MDAgNTAwIDYxMyA2MzUgNTUwIDYzNSA2MTUgMzUyIDYzNSA2MzQgMjc4IDI3OCA1NzkgMjc4IDk3NCA2MzQgNjEyCjYzNSA2MzUgNDExIDUyMSAzOTIgNjM0IDU5MiA4MTggNTkyIDU5MiA1MjUgNjM2IDMzNyA2MzYgODM4IDYwMCA2MzYgNjAwIDMxOAozNTIgNTE4IDEwMDAgNTAwIDUwMCA1MDAgMTM1MCA2MzUgNDAwIDEwNzAgNjAwIDY4NSA2MDAgNjAwIDMxOCAzMTggNTE4IDUxOAo1OTAgNTAwIDEwMDAgNTAwIDEwMDAgNTIxIDQwMCAxMDI4IDYwMCA1MjUgNjExIDMxOCA0MDEgNjM2IDYzNiA2MzYgNjM2IDMzNwo1MDAgNTAwIDEwMDAgNDcxIDYxNyA4MzggMzYxIDEwMDAgNTAwIDUwMCA4MzggNDAxIDQwMSA1MDAgNjM2IDYzNiAzMTggNTAwCjQwMSA0NzEgNjE3IDk2OSA5NjkgOTY5IDUzMSA2ODQgNjg0IDY4NCA2ODQgNjg0IDY4NCA5NzQgNjk4IDYzMiA2MzIgNjMyIDYzMgoyOTUgMjk1IDI5NSAyOTUgNzc1IDc0OCA3ODcgNzg3IDc4NyA3ODcgNzg3IDgzOCA3ODcgNzMyIDczMiA3MzIgNzMyIDYxMSA2MDgKNjMwIDYxMyA2MTMgNjEzIDYxMyA2MTMgNjEzIDk5NSA1NTAgNjE1IDYxNSA2MTUgNjE1IDI3OCAyNzggMjc4IDI3OCA2MTIgNjM0CjYxMiA2MTIgNjEyIDYxMiA2MTIgODM4IDYxMiA2MzQgNjM0IDYzNCA2MzQgNTkyIDYzNSA1OTIgXQplbmRvYmoKMTcgMCBvYmoKPDwgL3ggMTggMCBSID4+CmVuZG9iagoyMyAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDgwID4+CnN0cmVhbQp4nEWMuw3AMAhEe6ZgBH4mZp8olbN/GyBK3HBPunu4OhIyU95hhocEngwshlPxBpmjYDW4RlKNneyjsG5fdYHmelOr9fcHKk92dnE9zcsZ9AplbmRzdHJlYW0KZW5kb2JqCjI0IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjQ4ID4+CnN0cmVhbQp4nC1ROZIDQQjL5xV6QnPT77HLkff/6QrKAYOGQyA6LXFQxk8Qlive8shVtOHvmRjBd8Gh38p1GxY5EBVI0hhUTahdvB69B3YcZgLzpDUsgxnrAz9jCjd6cXhMxtntdRk1BHvXa09mUDIrF3HJxAVTddjImcNPpowL7VzPDci5EdZlGKSblcaMhCNNIVJIoeomqTNBkASjq1GjjRzFfunLI51hVSNqDPtcS9vXcxPOGjQ7Fqs8OaVHV5zLycULKwf9vM3ARVQaqzwQEnC/20P9nOzkN97SubPF9Phec7K8MBVY8ea1G5BNtfg3L+L4PePr+fwDqKVbFgplbmRzdHJlYW0KZW5kb2JqCjIxIDAgb2JqCjw8IC9CYXNlRm9udCAvRGVqYVZ1U2FucyAvQ2hhclByb2NzIDIyIDAgUgovRW5jb2RpbmcgPDwgL0RpZmZlcmVuY2VzIFsgNDkgL29uZSAvdHdvIF0gL1R5cGUgL0VuY29kaW5nID4+IC9GaXJzdENoYXIgMAovRm9udEJCb3ggWyAtMTAyMSAtNDYzIDE3OTQgMTIzMyBdIC9Gb250RGVzY3JpcHRvciAyMCAwIFIKL0ZvbnRNYXRyaXggWyAwLjAwMSAwIDAgMC4wMDEgMCAwIF0gL0xhc3RDaGFyIDI1NSAvTmFtZSAvRGVqYVZ1U2FucwovU3VidHlwZSAvVHlwZTMgL1R5cGUgL0ZvbnQgL1dpZHRocyAxOSAwIFIgPj4KZW5kb2JqCjIwIDAgb2JqCjw8IC9Bc2NlbnQgOTI5IC9DYXBIZWlnaHQgMCAvRGVzY2VudCAtMjM2IC9GbGFncyAzMgovRm9udEJCb3ggWyAtMTAyMSAtNDYzIDE3OTQgMTIzMyBdIC9Gb250TmFtZSAvRGVqYVZ1U2FucyAvSXRhbGljQW5nbGUgMAovTWF4V2lkdGggMTM0MiAvU3RlbVYgMCAvVHlwZSAvRm9udERlc2NyaXB0b3IgL1hIZWlnaHQgMCA+PgplbmRvYmoKMTkgMCBvYmoKWyA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMAo2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDMxOCA0MDEgNDYwIDgzOCA2MzYKOTUwIDc4MCAyNzUgMzkwIDM5MCA1MDAgODM4IDMxOCAzNjEgMzE4IDMzNyA2MzYgNjM2IDYzNiA2MzYgNjM2IDYzNiA2MzYgNjM2CjYzNiA2MzYgMzM3IDMzNyA4MzggODM4IDgzOCA1MzEgMTAwMCA2ODQgNjg2IDY5OCA3NzAgNjMyIDU3NSA3NzUgNzUyIDI5NQoyOTUgNjU2IDU1NyA4NjMgNzQ4IDc4NyA2MDMgNzg3IDY5NSA2MzUgNjExIDczMiA2ODQgOTg5IDY4NSA2MTEgNjg1IDM5MCAzMzcKMzkwIDgzOCA1MDAgNTAwIDYxMyA2MzUgNTUwIDYzNSA2MTUgMzUyIDYzNSA2MzQgMjc4IDI3OCA1NzkgMjc4IDk3NCA2MzQgNjEyCjYzNSA2MzUgNDExIDUyMSAzOTIgNjM0IDU5MiA4MTggNTkyIDU5MiA1MjUgNjM2IDMzNyA2MzYgODM4IDYwMCA2MzYgNjAwIDMxOAozNTIgNTE4IDEwMDAgNTAwIDUwMCA1MDAgMTM0MiA2MzUgNDAwIDEwNzAgNjAwIDY4NSA2MDAgNjAwIDMxOCAzMTggNTE4IDUxOAo1OTAgNTAwIDEwMDAgNTAwIDEwMDAgNTIxIDQwMCAxMDIzIDYwMCA1MjUgNjExIDMxOCA0MDEgNjM2IDYzNiA2MzYgNjM2IDMzNwo1MDAgNTAwIDEwMDAgNDcxIDYxMiA4MzggMzYxIDEwMDAgNTAwIDUwMCA4MzggNDAxIDQwMSA1MDAgNjM2IDYzNiAzMTggNTAwCjQwMSA0NzEgNjEyIDk2OSA5NjkgOTY5IDUzMSA2ODQgNjg0IDY4NCA2ODQgNjg0IDY4NCA5NzQgNjk4IDYzMiA2MzIgNjMyIDYzMgoyOTUgMjk1IDI5NSAyOTUgNzc1IDc0OCA3ODcgNzg3IDc4NyA3ODcgNzg3IDgzOCA3ODcgNzMyIDczMiA3MzIgNzMyIDYxMSA2MDUKNjMwIDYxMyA2MTMgNjEzIDYxMyA2MTMgNjEzIDk4MiA1NTAgNjE1IDYxNSA2MTUgNjE1IDI3OCAyNzggMjc4IDI3OCA2MTIgNjM0CjYxMiA2MTIgNjEyIDYxMiA2MTIgODM4IDYxMiA2MzQgNjM0IDYzNCA2MzQgNTkyIDYzNSA1OTIgXQplbmRvYmoKMjIgMCBvYmoKPDwgL29uZSAyMyAwIFIgL3R3byAyNCAwIFIgPj4KZW5kb2JqCjI5IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjkyID4+CnN0cmVhbQp4nCVSS24FMQjb5xS+QKXwJ+eZ6q1e77+tySxGECDGdqZUsREbPyLINpQ2fmVFNzQEf2uKqoXvyjSIHyRPfRAV4OlZoYYDzxfEOm98lu1zM/WEQ07DguULkRBRfoIuSM14KtRkuOiJCcZ9RN9wK6SzpDiKiB4U3UghbJJ3JJR59uArwUsMpn7VGKVMfJbHuVkII8lFNrJmSQo3zBZKDgrIoincwPVVVNmUHxQYZBOWnCnSCTIY5k6MpDY3cvC6FkykBbZvps2O0UjmuaejQqQWCDvhQR3kswdNwuFBVzjrxJ9olD/OMaMJMnrpopRD9+2cqfLJqEyoW+c+J7nnOJpeDrUDq55AynSaiVdiCNAukTEvBpZ0oubFxoz3P3jW5x8e1GeBCmVuZHN0cmVhbQplbmRvYmoKMzAgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyNTAgPj4Kc3RyZWFtCnicPVA5cgRBCMvnFXoCN93vWZej3f+nFrjsYEY0UDpoMwi6+au6aC186bPvz8A03o9b/FX9uxSmREMkUQqvJ66hbiGtUX2QWagMTvIGijrz5VFUKNKNk8qLOLbSYbXotZOphEx8GbcqBaY9E29oJ9kUGrQtRPXRIfdJCnM9qJDknrYqqBbHYULDAh2FmBRwOhU4d4W9zf1+tA0neAC3nGJibyfk4hyEF+54CbpTpg/OVC9SSE4uijKrZiPpUA8xmcN2Qm9WjmzaIt9irX9W4XM5SswlfVLURhDujc7kdUQw/14jqFq83OQon6C2+NoQU/3HeT/fP1IVXH0KZW5kc3RyZWFtCmVuZG9iagozMSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDQzNyA+PgpzdHJlYW0KeJw1kklyJTEIRPd1Ci7gCDFoOs/v8Mp9/22/pNyLKpAQkJkw57JhuezL3aaXzRz2x58aZavs75PbFc4a5hgfNu3zxBn2NS1qd2J4tv08Pt9S7mFhJ4xyn2dS+6jMTf09N5dyVljx+Ez6WozF9aJsbKNBVNm9FlOv3bfFuuQei307NY4SnFNcng8yb5GGTx4dAJJj05K25Ofli47Io/Nrz2tn/I8cbs4FGnk7reIoMoeV3qJDTaGItqgByb4ZsggF+MrGtvAChoV2dzbznPeVRNL+PJwKjCpGEB61JJmPY4V+nmlzSPzNfIQwBmrGy1PTilZPOeImL9FQLxK5NdPPIwyTkRac6/JN/K1JFnVLGDasqFiHqAt7Hd6IESq3CrLZ1fACPX/a85zEmFh16SWMBVfBGwxpNIbRKAJLFjwcekOi2O+qvdIH5Fm69e6WhhYIGdqO0BqobUjQq61DUGDHuC01NyPNNQCIe6lJ7ySgfR2AEoF42+wcearCUl2YsLynxd8NSfOcQlDWOxgU0fkeRROF9/1dDPYut4phj5r3PC4QICRizj41wXeXfqn+PN//ABlPplMKZW5kc3RyZWFtCmVuZG9iagozMiAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDI1NSA+PgpzdHJlYW0KeJw1UMltBDEM+7sKNhBAp2XXs0Fem/6/oTQbjAcibJGUmLkh8MKXKlIT6YJvXWEK3YXflWpQdr1X3IKKIUqwFeEGntfy6+AXMSJ2nvpaJmeQBnkUEUce3ucljjbVGm/LbJmihoGvoTIdMe0aBykbJjXTWd2pZPQLUUhORwS55L84qlPFZiOPPdV2cwZl8CZgHGwqreljNei9lJpKFyVTnX8l59mzUqA4SkwCveruTV13g45gXzhzO93t5z6BSQfA2T6h0quzk8t4wx7EePXA06fbD+cmuzF1Ou2gvj2Z2JFPNub3uWECQXetw73HIRnt5R5OJe777/haP39JF1y6CmVuZHN0cmVhbQplbmRvYmoKMzMgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCA0MTYgPj4Kc3RyZWFtCnicPVJLbgUxCNvPKbhApfBPzjNVd73/tjYzr9LLgwkEbENmyZJQ+VKVVJPWI996abf43vJ7aSC+VFRdwpaclIiU+0JG1BH84oxJD1zT2SW7peyIWkoV07VcGnUMddjEOsfel3uPx3690M0Kb1gr8F+2JbajaDzWjRF4cRDpGBSR/cIKP4MziBf9/GWCiPEL+RniqXiLyCBIdDUgpgAW57GL1ehpsBeYG1owibWWCxBHjXDWj70vvqKnsRFXfE162bzmfdDYahaBk2CEZoiihhtZQ03PFHUH3BL9J6BJkZtDoQcI2iAKIZXVk49N0cBJAzcGyzEZJXPAoue+J8NrvW9821TxAzCU7HxkHg9D8I3tOIigb0HYZ2jleLNwAxkoAS0QoPPcAAkziK2UfYg28TXoq+XDBxF/NPkdT9FNnXEcjENnsbS4hAucN8W0Bck4PJsVg5JLwIh8YUj30HEI3D4EdK2Z3MZWPqJovSKt2TZ6AM4M23jKsyi8J2XDfBQn2STGojmFhKfYRWZo60gCuJi0DRFQw9p8KN7Xzx+IoaQ2CmVuZHN0cmVhbQplbmRvYmoKMzQgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyNDYgPj4Kc3RyZWFtCnicRVE7bsUwDNt9Cl6ggPW1fZ4UnV7vv5ZMAnRIxNgSSTFVjYl0fJmhrLFm49sGT2xv/A6LhJ3CZ1hOWOlpGDttG07iGs6RZfBo9IQTslwjLAQiD1Yj1oHNzfPkW1zpQQ6/q0fpRmgX1BGeiM3xCnGV84uPFeIsisy7UpxO7xM6ikN3J6ilG1NP071m89EMl4NaiNhayZ+FPyNJ/o/aXbekfVFtZEwin4bUltnIVXDKqcpi3Ujmk6az2GkKIplSdN/xxhuzp9YSssV+KhmVspjVnQSzM7okh36MMlV9shYyKnDGOCMirsp8UywL77+7xs8fHkpY9gplbmRzdHJlYW0KZW5kb2JqCjM1IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggOTAgPj4Kc3RyZWFtCnicTY1BEsAgCAPvvoInGChS/tPpyf7/WpFx9EJ2EiCqjSpBxtB6k6HRgyIcxjcVBuoFB7DyABGf671cwEGZxrNNeRrppho/Zk9qbGejmg7PfRXxqnx/MdkhKQplbmRzdHJlYW0KZW5kb2JqCjM2IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggNDggPj4Kc3RyZWFtCnicMzIyUjBQMDMBEoamRgrmhmYKKYZcYH4uiAIJ5HDBpCAsAyANVpHDlQYAgA4MJQplbmRzdHJlYW0KZW5kb2JqCjM3IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjc0ID4+CnN0cmVhbQp4nE1SS3bEMAjb+xQcwfzxeaavq+n9txU409dFImKBkJKUKm2KwC3jkOumL17z/NPgfOi92PxfZRZdBZMlE5eQHSbZGN9JryWKORGSyBHULYOvpbbvCea6Qw86d4Ax2VDBpUWGOTOgnmbqgIG2XZXY9ahFXLVolp1SMFftIB0u/Uwkawao3nu62nAfxX+omHsqZIos0gogcsF57wmoFAUUrPcZkts4EJzYgSfscSOvi6/lLvcEKa37D/Jwe7M05FakRH50DG5uBlV7UnR8UDU/VQb8Yd92zEFVvN9ovy8Dyzb7pORxIJ73RMFYkjB2ajN8ehpfLnMSciBxtjf2Gm32VoxBiTPM9TR/xnt9/wJnsGqfCmVuZHN0cmVhbQplbmRvYmoKMzggMCBvYmoKPDwgL0JCb3ggWyAtNjY1IC0zMjUgMjAyOSAxMDM4IF0gL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAzNQovU3VidHlwZSAvRm9ybSAvVHlwZSAvWE9iamVjdCA+PgpzdHJlYW0KeJzjMjWyUDA2NFLI5TI1AzNywAxLExADJIdggSXTAO7GCbUKZW5kc3RyZWFtCmVuZG9iagozOSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDExNCA+PgpzdHJlYW0KeJw1TssNQzEMumcKRvDf8Tyv6ind/1rHai8GYUC4BwhM1VdTkVx48bqU8FmyvfEMegwLhRtBtJU2CzGsCs/iSFgWWAMWNqXmdj/NXKvT7Lt7ZFJet2UjRNsjaQh3KBFiJ5RjxjzrP+v8Vp31/gItliJeCmVuZHN0cmVhbQplbmRvYmoKNDAgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyNzYgPj4Kc3RyZWFtCnicTZFLcsMwDEP3PgUvkBnxK+k86XSV3H/bR7npdGGThigAhDNLhlTJQ/eS1JL0IV96faB3d6lbXpfG/y5Su6uQmFN0gewppoOZIc/LPCTNxcoOp+2b+3l5jNP53MwuCXXuFicREza+pkmEgjK1Nyc5pnjO49DVTrXyPumuVUeJohULN9Y6UUuwFsgFLkeIWcsDQ4uBhyq27orh+kUw/kg4VSawNt+GegkHmmwVDfM+Ab3+orpzMRJ9n04X15IHA52PjtUybDsZY6AQW9EFV0RF49zGswPriTFYVoNIDIIdp1q1g+56i57oKH3l6eFKQmVlZyKOyDoV8Rw3op2LH4txbGn1DwHBl5vJZ5Xn9f0DZepl8gplbmRzdHJlYW0KZW5kb2JqCjQxIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggNDIgPj4Kc3RyZWFtCnicMzK3UDBQsDQEEoZA0tDAQCHFkAvMz+WCCuRwGaKwQDSUSgMAfswMEgplbmRzdHJlYW0KZW5kb2JqCjQyIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggNDA4ID4+CnN0cmVhbQp4nC2TOXIDQQhF8zkFF1BVs/R2Hrkc2fdP/T5yoIGhp/kLaI5hw9Lt5W613GYO+/KHis9pv4/7MV/Hfh6PMM/kt8wHv3nsHHs/fobtYeFhNIjZ4f3E7SS5tq5lhZ1JOan5oL6J8R8rdaJspeUCaB+uTPM7dCLYS2WkxThgTIvQiV8QRagW1dEdg/vv51LYZXtb0GMVIsVqgphhtE6aKByVSWqU0aFiinaVyG6ZMu0sqyPaZXVLsLgyeZMXE92+BvG2GXQJsMdtL0VOET/2J0u+nwEfROuuhAuZk7vBgQlVwUKLTmJSdCkwCxfzY+NcWJfMJTE8rxwW+dGGV/Y32FVICkwophWVHeEyojPfqmjW9M8eJs8KKaMbGhTzep+Q7ds7kEzUCytXD6EYjcyft1X5xtbc7QbfZrYbKVfE1eWgnqGRihee5YmeF5rZrWANpD0K5uiK2D0k7ozde+onPnHKwc6km7c7W/7SNNozKFwogNGrJ/C49hJ+9N6L1au3Q9NTJo100sZRZZ9gCQ25/PljvJ/vP4XjmJkKZW5kc3RyZWFtCmVuZG9iago0MyAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDM1MyA+PgpzdHJlYW0KeJw9UjtyRDEI698puEBmzB/Os5lU2fu3EThJJQYjQMLuQYe06IOZnA8lN33yY13kxvR+DElXo+/HjpBHkTZKW0kzKU7T61FXCkVGgBYk1YuvR4JvRgMVRcJOgarXwzVsJY4gT6DPHJ8XTLMOYnEy7DCoMXMYnewgk0ImRgK+2Zk5mG7QIgFO4KV7cXbLjewADTwbBdPNsKWCM7L1nEVRwctEs58jy4aOhZnggzN6igyLat9d1oBIOAj9vUZKxSL2YtmIfRRuk1USI0toHeEBXekILMfLawkbwhnLXuChMddeSNoWR969mXZSjh0wIpJ3VRxhlmxIg51/Jx2De4W+b4SzjkjeI9TGqElI54QNRSCPjpI1GgdMEkdz2FU+gDWEJ5iPkLCmQD7Txg7uCIoJMnlRZJ2cKOeeQcqXo3YvZvhbMEfGGcyqixhuv5lTW8H/HHbZLisoi/4kvp6vH1MwiTEKZW5kc3RyZWFtCmVuZG9iago0NCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDE3ID4+CnN0cmVhbQp4nDMyt1AwgMMUQy4AGuMC8QplbmRzdHJlYW0KZW5kb2JqCjQ1IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTcwID4+CnN0cmVhbQp4nEVQOQ7DMAzb/Qp+IIBFH7Lek6JT+/+1lFMki0mQskibvlBhC8cE3eC14mWFY8ED35Ka4VPYB44Gsu3J2hPOYs4k1h2HBlvFStWYK027miEaeqprYHYsIiJPG0yR6KMqQPM3GRYism4yFSBrxi54scvMpg/7r5D7MLvvGtXR9dw6hB2xy7ojpCtFDW2pnKUcE3JYBQNUguAs5CbshOsfrm86y/sHMoY9iQplbmRzdHJlYW0KZW5kb2JqCjQ2IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjY3ID4+CnN0cmVhbQp4nDVRSXLDMAy7+xV4griL70mnp+T/14LMdMYyaHMDoIjEgTZfcQwljR95JryOzwYH78fOAutUYAaXeVLwesLQbFSIOvpCOPH1zIfcgqRBlUd4MpjR5gS9MDdYEWtmTY+x22OGK/zexVBlZiPOtW7EJZZz+Zkeb6Q5TArpCa0vco/F988hUVKWSuS5wy0o9pKwFcLri2f3MOCq94iKakwLpQvpZa4skigOVJH1SqeIOERqI+egJE134hrkXJW0YFYEJy7qkJ/IaYd3wmmU03O3WCLMnFo7xiRXiva7JvWKtXBuD4yduiap0XzW6qH1rJXblDYZoV2jQZKiD/WEzvW+/u/5/fz+ASsdYNgKZW5kc3RyZWFtCmVuZG9iago0NyAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDI3MiA+PgpzdHJlYW0KeJw1UUtuBTEI288pfIFK/EnOM1V3vf+2JumTZgQJ2BgnsyAIw5cqUhZaN7714Y2n43eS8GaJX6IWMhvvs5jLhhJVwRg89xS0N5qdZn64rPPE93G9Nx7NqPAu1E5WQoLoTRkLRfpgRzFnpQq5WVlUV4HYhjRjJYXClhzNwVkTR/FUFqyIIc5E2WXUtw9bYpPeN5IoqnQZYa3gutbHhBE88X1MbqbJ37mrURXvyaKmY5rpDP+fq/7xbDLzPK4o99Ee9DqUAi5qzoXljKqjQE/isaY6xtz2MWYIgqchnHiHTRbUPR0ZF5NrMENSVnDljCgOuZHD3e8NTSnjo/HB8jyA0vA8W9LUFnxWeZ+fP/SWZUsKZW5kc3RyZWFtCmVuZG9iagoyNyAwIG9iago8PCAvQmFzZUZvbnQgL0FyaWFsTVQgL0NoYXJQcm9jcyAyOCAwIFIKL0VuY29kaW5nIDw8Ci9EaWZmZXJlbmNlcyBbIDMyIC9zcGFjZSA0NiAvcGVyaW9kIDQ4IC96ZXJvIC9vbmUgL3R3byA1MiAvZm91ciAvZml2ZSAvc2l4IDU2IC9laWdodCA2NwovQyAvRCA5NyAvYSAxMDEgL2UgMTA4IC9sIC9tIDExMiAvcCAxMTUgL3MgL3QgXQovVHlwZSAvRW5jb2RpbmcgPj4KL0ZpcnN0Q2hhciAwIC9Gb250QkJveCBbIC02NjUgLTMyNSAyMDI5IDEwMzggXSAvRm9udERlc2NyaXB0b3IgMjYgMCBSCi9Gb250TWF0cml4IFsgMC4wMDEgMCAwIDAuMDAxIDAgMCBdIC9MYXN0Q2hhciAyNTUgL05hbWUgL0FyaWFsTVQKL1N1YnR5cGUgL1R5cGUzIC9UeXBlIC9Gb250IC9XaWR0aHMgMjUgMCBSID4+CmVuZG9iagoyNiAwIG9iago8PCAvQXNjZW50IDkwNiAvQ2FwSGVpZ2h0IDcxNiAvRGVzY2VudCAtMjEyIC9GbGFncyAzMgovRm9udEJCb3ggWyAtNjY1IC0zMjUgMjAyOSAxMDM4IF0gL0ZvbnROYW1lIC9BcmlhbE1UIC9JdGFsaWNBbmdsZSAwCi9NYXhXaWR0aCAxMDE1IC9TdGVtViAwIC9UeXBlIC9Gb250RGVzY3JpcHRvciAvWEhlaWdodCA1MTkgPj4KZW5kb2JqCjI1IDAgb2JqClsgNzUwIDc1MCA3NTAgNzUwIDc1MCA3NTAgNzUwIDc1MCA3NTAgNzUwIDc1MCA3NTAgNzUwIDc1MCA3NTAgNzUwIDc1MCA3NTAKNzUwIDc1MCA3NTAgNzUwIDc1MCA3NTAgNzUwIDc1MCA3NTAgNzUwIDc1MCA3NTAgNzUwIDc1MCAyNzggMjc4IDM1NSA1NTYgNTU2Cjg4OSA2NjcgMTkxIDMzMyAzMzMgMzg5IDU4NCAyNzggMzMzIDI3OCAyNzggNTU2IDU1NiA1NTYgNTU2IDU1NiA1NTYgNTU2IDU1Ngo1NTYgNTU2IDI3OCAyNzggNTg0IDU4NCA1ODQgNTU2IDEwMTUgNjY3IDY2NyA3MjIgNzIyIDY2NyA2MTEgNzc4IDcyMiAyNzgKNTAwIDY2NyA1NTYgODMzIDcyMiA3NzggNjY3IDc3OCA3MjIgNjY3IDYxMSA3MjIgNjY3IDk0NCA2NjcgNjY3IDYxMSAyNzggMjc4CjI3OCA0NjkgNTU2IDMzMyA1NTYgNTU2IDUwMCA1NTYgNTU2IDI3OCA1NTYgNTU2IDIyMiAyMjIgNTAwIDIyMiA4MzMgNTU2IDU1Ngo1NTYgNTU2IDMzMyA1MDAgMjc4IDU1NiA1MDAgNzIyIDUwMCA1MDAgNTAwIDMzNCAyNjAgMzM0IDU4NCA3NTAgNTU2IDc1MCAyMjIKNTU2IDMzMyAxMDAwIDU1NiA1NTYgMzMzIDEwMDAgNjY3IDMzMyAxMDAwIDc1MCA2MTEgNzUwIDc1MCAyMjIgMjIyIDMzMyAzMzMKMzUwIDU1NiAxMDAwIDMzMyAxMDAwIDUwMCAzMzMgOTQ0IDc1MCA1MDAgNjY3IDI3OCAzMzMgNTU2IDU1NiA1NTYgNTU2IDI2MAo1NTYgMzMzIDczNyAzNzAgNTU2IDU4NCAzMzMgNzM3IDU1MiA0MDAgNTQ5IDMzMyAzMzMgMzMzIDU3NiA1MzcgMjc4IDMzMyAzMzMKMzY1IDU1NiA4MzQgODM0IDgzNCA2MTEgNjY3IDY2NyA2NjcgNjY3IDY2NyA2NjcgMTAwMCA3MjIgNjY3IDY2NyA2NjcgNjY3CjI3OCAyNzggMjc4IDI3OCA3MjIgNzIyIDc3OCA3NzggNzc4IDc3OCA3NzggNTg0IDc3OCA3MjIgNzIyIDcyMiA3MjIgNjY3IDY2Nwo2MTEgNTU2IDU1NiA1NTYgNTU2IDU1NiA1NTYgODg5IDUwMCA1NTYgNTU2IDU1NiA1NTYgMjc4IDI3OCAyNzggMjc4IDU1NiA1NTYKNTU2IDU1NiA1NTYgNTU2IDU1NiA1NDkgNjExIDU1NiA1NTYgNTU2IDU1NiA1MDAgNTU2IDUwMCBdCmVuZG9iagoyOCAwIG9iago8PCAvQyAyOSAwIFIgL0QgMzAgMCBSIC9hIDMxIDAgUiAvZSAzMiAwIFIgL2VpZ2h0IDMzIDAgUiAvZml2ZSAzNCAwIFIKL2ZvdXIgMzUgMCBSIC9sIDM2IDAgUiAvbSAzNyAwIFIgL29uZSAzOSAwIFIgL3AgNDAgMCBSIC9wZXJpb2QgNDEgMCBSCi9zIDQyIDAgUiAvc2l4IDQzIDAgUiAvc3BhY2UgNDQgMCBSIC90IDQ1IDAgUiAvdHdvIDQ2IDAgUiAvemVybyA0NyAwIFIgPj4KZW5kb2JqCjMgMCBvYmoKPDwgL0YxIDI3IDAgUiAvRjIgMTYgMCBSIC9GMyAyMSAwIFIgPj4KZW5kb2JqCjQgMCBvYmoKPDwgL0ExIDw8IC9DQSAwIC9UeXBlIC9FeHRHU3RhdGUgL2NhIDEgPj4KL0EyIDw8IC9DQSAxIC9UeXBlIC9FeHRHU3RhdGUgL2NhIDEgPj4KL0EzIDw8IC9DQSAwLjggL1R5cGUgL0V4dEdTdGF0ZSAvY2EgMC44ID4+ID4+CmVuZG9iago1IDAgb2JqCjw8ID4+CmVuZG9iago2IDAgb2JqCjw8ID4+CmVuZG9iago3IDAgb2JqCjw8IC9GMS1BcmlhbC1taW51cyAzOCAwIFIgL00wIDEyIDAgUiAvTTEgMTMgMCBSID4+CmVuZG9iagoxMiAwIG9iago8PCAvQkJveCBbIC04IC04IDggOCBdIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTMxIC9TdWJ0eXBlIC9Gb3JtCi9UeXBlIC9YT2JqZWN0ID4+CnN0cmVhbQp4nG2QQQ6EIAxF9z1FL/BJS0Vl69JruJlM4v23A3FATN000L48flH+kvBOpcD4JAlLTrPketOQ0rpMjBjm1bIox6BRLdbOdTioz9BwY3SLsRSm1NboeKOb6Tbekz/6sFkhRj8cDq+EexZDJlwpMQaH3wsv28P/EZ5e1MAfoo1+Y1pD/QplbmRzdHJlYW0KZW5kb2JqCjEzIDAgb2JqCjw8IC9CQm94IFsgLTggLTggOCA4IF0gL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxMzEgL1N1YnR5cGUgL0Zvcm0KL1R5cGUgL1hPYmplY3QgPj4Kc3RyZWFtCnicbZBBDoQgDEX3PUUv8ElLRWXr0mu4mUzi/bcDcUBM3TTQvjx+Uf6S8E6lwPgkCUtOs+R605DSukyMGObVsijHoFEt1s51OKjP0HBjdIuxFKbU1uh4o5vpNt6TP/qwWSFGPxwOr4R7FkMmXCkxBoffCy/bw/8Rnl7UwB+ijX5jWkP9CmVuZHN0cmVhbQplbmRvYmoKMiAwIG9iago8PCAvQ291bnQgMSAvS2lkcyBbIDEwIDAgUiBdIC9UeXBlIC9QYWdlcyA+PgplbmRvYmoKNDggMCBvYmoKPDwgL0NyZWF0aW9uRGF0ZSAoRDoyMDIyMDYwODIxMTMzNCswMicwMCcpCi9DcmVhdG9yIChNYXRwbG90bGliIHYzLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZykKL1Byb2R1Y2VyIChNYXRwbG90bGliIHBkZiBiYWNrZW5kIHYzLjMuMikgPj4KZW5kb2JqCnhyZWYKMCA0OQowMDAwMDAwMDAwIDY1NTM1IGYgCjAwMDAwMDAwMTYgMDAwMDAgbiAKMDAwMDAxNzQ4NyAwMDAwMCBuIAowMDAwMDE2Njc1IDAwMDAwIG4gCjAwMDAwMTY3MjkgMDAwMDAgbiAKMDAwMDAxNjg3MSAwMDAwMCBuIAowMDAwMDE2ODkyIDAwMDAwIG4gCjAwMDAwMTY5MTMgMDAwMDAgbiAKMDAwMDAwMDA2NSAwMDAwMCBuIAowMDAwMDAwMzk4IDAwMDAwIG4gCjAwMDAwMDAyMDggMDAwMDAgbiAKMDAwMDAwNTIxMyAwMDAwMCBuIAowMDAwMDE2OTc5IDAwMDAwIG4gCjAwMDAwMTcyMzMgMDAwMDAgbiAKMDAwMDAwNTkyMiAwMDAwMCBuIAowMDAwMDA1NzE0IDAwMDAwIG4gCjAwMDAwMDUzOTggMDAwMDAgbiAKMDAwMDAwNjk3NSAwMDAwMCBuIAowMDAwMDA1MjM0IDAwMDAwIG4gCjAwMDAwMDc5ODYgMDAwMDAgbiAKMDAwMDAwNzc4NiAwMDAwMCBuIAowMDAwMDA3NDgwIDAwMDAwIG4gCjAwMDAwMDkwMzkgMDAwMDAgbiAKMDAwMDAwNzAwNyAwMDAwMCBuIAowMDAwMDA3MTU5IDAwMDAwIG4gCjAwMDAwMTUzOTQgMDAwMDAgbiAKMDAwMDAxNTE5NCAwMDAwMCBuIAowMDAwMDE0Nzg5IDAwMDAwIG4gCjAwMDAwMTY0NDUgMDAwMDAgbiAKMDAwMDAwOTA4NSAwMDAwMCBuIAowMDAwMDA5NDUwIDAwMDAwIG4gCjAwMDAwMDk3NzMgMDAwMDAgbiAKMDAwMDAxMDI4MyAwMDAwMCBuIAowMDAwMDEwNjExIDAwMDAwIG4gCjAwMDAwMTExMDAgMDAwMDAgbiAKMDAwMDAxMTQxOSAwMDAwMCBuIAowMDAwMDExNTgxIDAwMDAwIG4gCjAwMDAwMTE3MDEgMDAwMDAgbiAKMDAwMDAxMjA0OCAwMDAwMCBuIAowMDAwMDEyMjE1IDAwMDAwIG4gCjAwMDAwMTI0MDIgMDAwMDAgbiAKMDAwMDAxMjc1MSAwMDAwMCBuIAowMDAwMDEyODY1IDAwMDAwIG4gCjAwMDAwMTMzNDYgMDAwMDAgbiAKMDAwMDAxMzc3MiAwMDAwMCBuIAowMDAwMDEzODYxIDAwMDAwIG4gCjAwMDAwMTQxMDQgMDAwMDAgbiAKMDAwMDAxNDQ0NCAwMDAwMCBuIAowMDAwMDE3NTQ3IDAwMDAwIG4gCnRyYWlsZXIKPDwgL0luZm8gNDggMCBSIC9Sb290IDEgMCBSIC9TaXplIDQ5ID4+CnN0YXJ0eHJlZgoxNzcwNAolJUVPRgo=\n", "image/svg+xml": [ "\n", "\n", "\n", "\n", " \n", " \n", " \n", " \n", " 2022-06-08T21:13:34.811143\n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.3.2, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n" ], "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "visualize_samples(dataset.data, dataset.label)\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### The data loader class\n", "\n", "The class `torch.utils.data.DataLoader` represents a Python iterable over a dataset with support for automatic batching, multi-process data loading and many more features. The data loader communicates with the dataset using the function `__getitem__`, and stacks its outputs as tensors over the first dimension to form a batch.\n", "In contrast to the dataset class, we usually don't have to define our own data loader class, but can just create an object of it with the dataset as input. Additionally, we can configure our data loader with the following input arguments (only a selection, see full list [here](https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader)):\n", "\n", "* `batch_size`: Number of samples to stack per batch\n", "* `shuffle`: If True, the data is returned in a random order. This is important during training for introducing stochasticity. \n", "* `num_workers`: Number of subprocesses to use for data loading. The default, 0, means that the data will be loaded in the main process which can slow down training for datasets where loading a data point takes a considerable amount of time (e.g. large images). More workers are recommended for those, but can cause issues on Windows computers. For tiny datasets as ours, 0 workers are usually faster.\n", "* `persistent_workers`: If True, workers will not be shutdown after an iteration over the dataset has finished. This can be useful if the time per epoch is small, or if you face issues with workers being killed during training. \n", "* `drop_last`: If True, the last batch is dropped in case it is smaller than the specified batch size. This occurs when the dataset size is not a multiple of the batch size. Only potentially helpful during training to keep a consistent batch size.\n", "* `collate_fn`: A function that defines how the elements per batch are combined. By default, PyTorch stacks them as PyTorch tensors. For JAX, we will change it to NumPy arrays.\n", "\n", "Let's create a simple data loader below with a function that stacks batch elements as NumPy array instead of PyTorch Tensors:" ] }, { "cell_type": "code", "execution_count": 40, "metadata": {}, "outputs": [], "source": [ "# This collate function is taken from the JAX tutorial with PyTorch Data Loading\n", "# https://jax.readthedocs.io/en/latest/notebooks/Neural_Network_and_Data_Loading.html\n", "def numpy_collate(batch):\n", " if isinstance(batch[0], np.ndarray):\n", " return np.stack(batch)\n", " elif isinstance(batch[0], (tuple,list)):\n", " transposed = zip(*batch)\n", " return [numpy_collate(samples) for samples in transposed]\n", " else:\n", " return np.array(batch)\n", "\n", "data_loader = data.DataLoader(dataset, batch_size=8, shuffle=True, collate_fn=numpy_collate)" ] }, { "cell_type": "code", "execution_count": 41, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Data inputs (8, 2) \n", " [[ 1.0504987 1.0865755 ]\n", " [ 0.02809919 -0.06226995]\n", " [ 0.06141667 1.0757508 ]\n", " [ 0.08356921 -0.11297069]\n", " [ 1.0324166 -0.01301431]\n", " [ 1.0024511 0.04979983]\n", " [ 0.3078881 0.11195749]\n", " [ 1.0371146 0.9396015 ]]\n", "Data labels (8,) \n", " [0 0 1 0 1 1 0 0]\n" ] } ], "source": [ "# next(iter(...)) catches the first batch of the data loader\n", "# If shuffle is True, this will return a different batch every time we run this cell\n", "# For iterating over the whole dataset, we can simple use \"for batch in data_loader: ...\"\n", "data_inputs, data_labels = next(iter(data_loader))\n", "\n", "# The shape of the outputs are [batch_size, d_1,...,d_N] where d_1,...,d_N are the \n", "# dimensions of the data point returned from the dataset class\n", "print(\"Data inputs\", data_inputs.shape, \"\\n\", data_inputs)\n", "print(\"Data labels\", data_labels.shape, \"\\n\", data_labels)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Optimization\n", "\n", "After defining the model and the dataset, it is time to prepare the optimization of the model. During training, we will perform the following steps:\n", "\n", "1. Get a batch from the data loader\n", "2. Obtain the predictions from the model for the batch\n", "3. Calculate the loss based on the difference between predictions and labels\n", "4. Backpropagation: calculate the gradients for every parameter with respect to the loss\n", "5. Update the parameters of the model in the direction of the gradients\n", "\n", "We have seen how we can do step 1, 2 and 4 in JAX and Flax. Now, we will look at step 3 and 5." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Stochastic Gradient Descent\n", "\n", "For updating the parameters, Flax does not directly provide support for optimizers, but instead refers to another package called `optax` ([documentation](https://optax.readthedocs.io/en/latest/index.html)). Optax is an optimization library for JAX, which offers most common deep learning optimizers (SGD, Adam, Adagrad, RMSProp, etc.) and utilities (gradient clipping, weight decay, etc.)." ] }, { "cell_type": "code", "execution_count": 42, "metadata": {}, "outputs": [], "source": [ "try:\n", " import optax\n", "except ModuleNotFoundError: # Install optax if missing\n", " !pip install --quiet optax\n", " import optax" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For now, we will use the simplest optimizer, namely `optax.sgd`. Stochastic Gradient Descent updates parameters by multiplying the gradients with a small constant, called learning rate, and subtracting those from the parameters (hence minimizing the loss). Therefore, we slowly move towards the direction of minimizing the loss. A good default value of the learning rate for a small network as ours is 0.1. Remember that we again aim to write functional code. Hence, the optimizer does not take as input the parameters, but only the optimizer hyperparameters." ] }, { "cell_type": "code", "execution_count": 43, "metadata": {}, "outputs": [], "source": [ "# Input to the optimizer are optimizer settings like learning rate\n", "optimizer = optax.sgd(learning_rate=0.1)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Since JAX calculates gradients via function transformations, we do not have functions like `backward()`, `optimizer.step()` or `optimizer.backward()` as in PyTorch. Instead, a optimizer is a function on the parameters and gradients. To simplify this step and bundle important parts of the training procedure, Flax offers the `flax.training` package. As a first step, we can create a `TrainState` which bundles the parameters, the optimizer, and the forward step of the model:" ] }, { "cell_type": "code", "execution_count": 44, "metadata": {}, "outputs": [], "source": [ "from flax.training import train_state\n", "\n", "model_state = train_state.TrainState.create(apply_fn=model.apply,\n", " params=params,\n", " tx=optimizer)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "With this state object, it is easier to handle the training." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Loss function\n", "\n", "For performing gradient updates, we need a function that can calculate the loss for a batch. Afterwards, we can apply JAX's gradient transformation to obtain a gradient function of it. In our setting, which is binary classification, we can use Binary Cross Entropy (BCE) which is defined as follows:\n", "\n", "$$\\mathcal{L}_{BCE} = -\\sum_i \\left[ y_i \\log x_i + (1 - y_i) \\log (1 - x_i) \\right]$$\n", "\n", "where $y$ are our labels, and $x$ our predictions, both in the range of $[0,1]$. Similar to PyTorch, Optax already provides a function for this: `optax.sigmoid_binary_cross_entropy(logits, labels)`. We calculate the loss on the logits instead of the sigmoid outputs for numerical stability. Let's write a function that takes as input a state (for the forward function), parameters, and a batch, and return the binary cross entropy loss and accuracy: " ] }, { "cell_type": "code", "execution_count": 45, "metadata": {}, "outputs": [], "source": [ "def calculate_loss_acc(state, params, batch):\n", " data_input, labels = batch\n", " # Obtain the logits and predictions of the model for the input data\n", " logits = state.apply_fn(params, data_input).squeeze(axis=-1)\n", " pred_labels = (logits > 0).astype(jnp.float32)\n", " # Calculate the loss and accuracy\n", " loss = optax.sigmoid_binary_cross_entropy(logits, labels).mean()\n", " acc = (pred_labels == labels).mean()\n", " return loss, acc" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Note that we explicitly add the parameters here as an input argument since we want to calculate the gradients with respect to them later. An example execution of the function would look like:" ] }, { "cell_type": "code", "execution_count": 46, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(DeviceArray(0.6830494, dtype=float32), DeviceArray(0.625, dtype=float32))" ] }, "execution_count": 46, "metadata": {}, "output_type": "execute_result" } ], "source": [ "batch = next(iter(data_loader))\n", "calculate_loss_acc(model_state, model_state.params, batch)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Creating an efficient training and validation step\n", "\n", "With this loss function and the optimizer, we are now ready to create an efficient training and validation/test step. First, let's consider the training. As input to each training step, we have a training state and a batch. We then want to calculate the loss for the input and take the gradients of it. Finally, we update the parameters with our optimizer and return the new state. All this can be summarized in the following function:" ] }, { "cell_type": "code", "execution_count": 47, "metadata": {}, "outputs": [], "source": [ "@jax.jit # Jit the function for efficiency\n", "def train_step(state, batch):\n", " # Gradient function\n", " grad_fn = jax.value_and_grad(calculate_loss_acc, # Function to calculate the loss\n", " argnums=1, # Parameters are second argument of the function\n", " has_aux=True # Function has additional outputs, here accuracy\n", " )\n", " # Determine gradients for current model, parameters and batch\n", " (loss, acc), grads = grad_fn(state, state.params, batch)\n", " # Perform parameter update with gradients and optimizer\n", " state = state.apply_gradients(grads=grads)\n", " # Return state and any other value we might want\n", " return state, loss, acc" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "By using the transformation `jax.jit`, the whole gradient calculation and application is optimized in XLA, providing an efficient function for updating the model.\n", "\n", "Next, let's look at the evaluation function. Here, we do not need to calculate gradients, but only want to get the accuracy of the model for the batch. This becomes a simpler version of the training step:" ] }, { "cell_type": "code", "execution_count": 48, "metadata": {}, "outputs": [], "source": [ "@jax.jit # Jit the function for efficiency\n", "def eval_step(state, batch):\n", " # Determine the accuracy\n", " _, acc = calculate_loss_acc(state, state.params, batch)\n", " return acc" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "These two functions provide us now efficient utilities to train our model." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Training\n", "\n", "Finally, we are ready to train our model. As a first step, we create a slightly larger dataset and specify a data loader with a larger batch size. " ] }, { "cell_type": "code", "execution_count": 49, "metadata": {}, "outputs": [], "source": [ "train_dataset = XORDataset(size=2500, seed=42)\n", "train_data_loader = data.DataLoader(train_dataset, batch_size=128, shuffle=True, collate_fn=numpy_collate)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now, we can write a small training function. In contrast to PyTorch, we do not need to explicitly push our model to GPU, since the parameters are already automatically created on GPU. Further, since the model itself is stateless, we do not have a `train()` or `eval()` function to switch between modes of e.g. dropout. When necessary, we can add an argument `train : bool` to the model forward pass. For this simple network here, however, this is not necessary.\n", "\n", "Following the PyTorch tutorial, let's write a function here that trains a model for several epochs:" ] }, { "cell_type": "code", "execution_count": 50, "metadata": {}, "outputs": [], "source": [ "def train_model(state, data_loader, num_epochs=100):\n", " # Training loop\n", " for epoch in tqdm(range(num_epochs)):\n", " for batch in data_loader:\n", " state, loss, acc = train_step(state, batch)\n", " # We could use the loss and accuracy for logging here, e.g. in TensorBoard\n", " # For simplicity, we skip this part here\n", " return state" ] }, { "cell_type": "code", "execution_count": 51, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "c9ddb028438c41c48d19fb9f532175b0", "version_major": 2, "version_minor": 0 }, "text/plain": [ " 0%| | 0/100 [00:00 Don't drop the last batch although it is smaller than 128\n", "test_data_loader = data.DataLoader(test_dataset, \n", " batch_size=128, \n", " shuffle=False, \n", " drop_last=False, \n", " collate_fn=numpy_collate) " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can use our `eval_step` function to efficiently evaluate our model:" ] }, { "cell_type": "code", "execution_count": 56, "metadata": {}, "outputs": [], "source": [ "def eval_model(state, data_loader):\n", " all_accs, batch_sizes = [], []\n", " for batch in data_loader:\n", " batch_acc = eval_step(state, batch)\n", " all_accs.append(batch_acc)\n", " batch_sizes.append(batch[0].shape[0])\n", " # Weighted average since some batches might be smaller\n", " acc = sum([a*b for a,b in zip(all_accs, batch_sizes)]) / sum(batch_sizes)\n", " print(f\"Accuracy of the model: {100.0*acc:4.2f}%\")" ] }, { "cell_type": "code", "execution_count": 57, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Accuracy of the model: 100.00%\n" ] } ], "source": [ "eval_model(trained_model_state, test_data_loader)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If we trained our model correctly, we should see a score close to 100% accuracy. However, this is only possible because of our simple task, and unfortunately, we usually don't get such high scores on test sets of more complex tasks." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Binding model parameters\n", "\n", "Once we have trained the model, we might want to do multiple application of the same model and parameters. It can get a bit annoying to always write `model.apply(params, ...)` and keep track of the model and parameters separately. To prevent this, Flax's module can be bound to specific parameters to simplify our application. Specifically, we can bind the instance `model` of our `SimpleClassifier` class to our trained parameter as follows:" ] }, { "cell_type": "code", "execution_count": 58, "metadata": {}, "outputs": [], "source": [ "trained_model = model.bind(trained_model_state.params)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "With the model being binded to the parameters, we can use it as we would any PyTorch module. For instance, to apply it to an input array, we can simply run:" ] }, { "cell_type": "code", "execution_count": 59, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(8, 1)" ] }, "execution_count": 59, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data_input, labels = next(iter(data_loader))\n", "out = trained_model(data_input) # No explicit parameter passing necessary anymore\n", "out.shape" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This can simplify the analysis of models, and provide a more familiar interface to PyTorch users." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Visualizing classification boundaries\n", "\n", "To visualize what our model has learned, we can perform a prediction for every data point in a range of $[-0.5, 1.5]$, and visualize the predicted class as in the sample figure at the beginning of this section. This shows where the model has created decision boundaries, and which points would be classified as $0$, and which as $1$. We therefore get a background image out of blue (class 0) and orange (class 1). The spots where the model is uncertain we will see a blurry overlap. The specific code is less relevant compared to the output figure which should hopefully show us a clear separation of classes:" ] }, { "cell_type": "code", "execution_count": 60, "metadata": { "scrolled": false }, "outputs": [ { "data": { "application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1BhZ2VzIDIgMCBSIC9UeXBlIC9DYXRhbG9nID4+CmVuZG9iago4IDAgb2JqCjw8IC9FeHRHU3RhdGUgNCAwIFIgL0ZvbnQgMyAwIFIgL1BhdHRlcm4gNSAwIFIKL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gL1NoYWRpbmcgNiAwIFIKL1hPYmplY3QgNyAwIFIgPj4KZW5kb2JqCjEwIDAgb2JqCjw8IC9Bbm5vdHMgWyBdIC9Db250ZW50cyA5IDAgUgovR3JvdXAgPDwgL0NTIC9EZXZpY2VSR0IgL1MgL1RyYW5zcGFyZW5jeSAvVHlwZSAvR3JvdXAgPj4KL01lZGlhQm94IFsgMCAwIDI5MS44NCAyODAuOTgwNjI1IF0gL1BhcmVudCAyIDAgUiAvUmVzb3VyY2VzIDggMCBSCi9UeXBlIC9QYWdlID4+CmVuZG9iago5IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTEgMCBSID4+CnN0cmVhbQp4nK2bS7Mcx3GF9/MrekkvbqMy670kTYthR3hBGxFeI2BIIgKQRUKy/PP9nZo7Mz0XOQRJiQpSmGZ1dVU+zjmZVbTt/enV17b94dPGP7a0vefvv/Hn7/T7lPj18eTT9lH444fLH32kfY7UvPIs3f/84+n0+xO/rbfSUx1je/mjzGSzpT62n/TJ7z4bcP1xejH6dKpzryX3uhXbe2mDP7HA3vj+/dMPh6fOWzaeH19nuHu6Vv3jFkzv1vdSLv/307vtv7Y/ba++9rPN3vP339YmXn377n9/ePvuP777Znv7iameX5AJn/8YTP724/bqX2379n+270/fbz9e5k27VdxxnV4/v3t+evrxZMz5lPhXZeyttyrDe9lzt/Ocp29eb69+Z5vZ9vr3p7YXz8OHMSptr//79FXa6z9tr9+f/uU1k6U9ma1l3v7EFLz/9PVPP7z58PTxhz/99RNLPLHE+5kttd1LP378PH3S9BvT3w9vaceRnw+v4XC3ss/22XB7MLu3uWcPhl9nPxjOGi5o8gU+mStyl9kU8fvyEe+++p1v5pr7q/9b9uq7JVvWfjKWNi7D8jb2ssbZxa7fH7/mTqCtWXPfR5r5l3op/VY3hR9v2HP84o/7PyZGnBBd9p2so818i5EHbryMN8Of5fCCPwiT6wul7dUPL9QvfaHnvafDC/0LX/CU9jGuL9gX9+D8KbXDC1/ag9dKRh1euO0BL+DNJxnf6t7XGCuZOGzXrP97w9dv4Ss0+pVoiItggTzdaidkyhnDPRNTe7ugeBHo34DNF67d3tuO751evPcc1iuBWUCaKeWSLLGBQaykUlrP2Req/ntSOF6GP1nbLQ/LtWefm5CF0X3w2RwM98rGBxNbHXW9nFqqfM/wVzA7oe1evPbeWFreiyU3Hy1FoyeT95HxXWe2ydy5Nx/TUg1GVyF8KkZQFCKWXO69Dis1WjcLGYSDe9LUTVjTRrFuPRqcyeWBrTMoq1X3nkYBIWq0jsk6cETuo2GRPPYxQVw2auFCzFlqqykZdvHtaRbIq1b24TaCF+TPie/qcLPGT3IF+/TM8lr4AUZgheTV+uADlvggsI9hhkcbkOncci+9dwyFpWopeWLNaAOWxg6Z2Jwzaz0ORFYfpTv2jdezFyIYm9QmP2hZTF16FAIKkTmtYdLRtgL6Yppei8dbhWYLsVhIxtH123dyhKV3Phi8UWR6G0TZHNsTiNLyLMVLZJYnOX7OSjg6ljCBdB1wY+wmCCzDEj01gn1D9xkMPVJnK+HkHYTNM0/IZW4O54Ij5FEJ0whcw+kDn2YGwOOTSM7Z4ogkxhwYS7VjSJIDF5Q9KTVaDSNA482xJmGcBy6uYE2VW8GYaDx2ZJ+pEufkHmBU51wwEIf82EmlITsjVmXXVKYSzML0QwkIIMjllYqFPaRU0CQRHiFpKtgysTy5AStUbNJztOqnDDsQhyTO4Bd5i4d4YXg0Ogt8vRRSAyMTt8XXmmcPTT7Q46MxuispwFTemhMJGuZEB+jMmFCi78khHcALOG8livOsGiJ1pUSHaokVGZKlhEkxmgTcMIjQ5HsC00A6XoqtgglHSRUWm6Qk6CRrghSwWphB4KJVUKD05UpCDDCafDGavJIUfHnwr+FcwX8aOTWiJkI6VtJYaq6QwKbkZlXLn2EUku5ZtgAAMrC+oJT0yHxuxKtJ8v9osrx+uCSO4jbMUIgTqBA0+toHudFg9xEtfTIX6AxVMRjnNokhKqzIQYbRyHtk5YBJ2SZQkYAv7zBluE0kkGsl5CeCRESTZk0lxhWMSJLXmkVZ+o1uwISETvHQKNSCQkSAyuZidMZmN2rNCEGpO2v3WZK4xXf2MYicmsKpJwAKaxJNIxPFrAE87bgnNMtOkCTUTUqLJxKB0GfJsSuBz1EQNnM2xyjIHThywjMhZUlKJCfxCb1NYQOOY6OUZrxuXOgDH44qzpqJUJhMn5qFyZ9lM9LN50rnJv8StOw2DgDAJAvJWb4YHfyE3gtOKjF+LuHEspnStTyisRJgwFeKPoBmHoRgRYd0xbH8haKL3V9RcG0adOJdhIvDSMAcrzwvMQn1YznhPgjpQ2wbe7SxToAEaU3CjymI9OTOowd8C4yjLLqdWSVrl7DGYyGV9RfadSyS67ujl4nlEvIQ8NBlZpYvsyc0PCEhsdNCOAKNMsqvA7aVpM4CPVk1yjtbdQOJU9WpgfAQrigNvhclUQNjSceZCO8VL3gAxEWAxYbxVQioChWMPqHp8QBohKdiyUW8Ex9FlMwvDN+lcySWQyEN9ojo0P2bOgsKRVwW+hQhQrEHWhVNLbCYIDRbDRNP2pXcl1NQOfpZEXdgVA0T9WlOipGGkIYB+jnxeuKvXEPMGBloHBJZAqCFpPoUv8KABI6cKM+q8Ngo8nVkpGUoiZ4UrZXonfCd4AbAQyxC/3EVJSGC5B9gjM9zMKIGkVNA9YMaAzWJglKIi4ygrSmNFJI0IS6RPgaghy7GTCoFKUApiePlgBjQCRphavqJ5HJDR7VY654RBp2NZ4FE7O7S4JLKkegiGbq0jZh2SfbUusmU8WJYvKhuldhZthGbgl7sNkwPy65GFTlRlMnCS8iR5CCkY+vDHZTI7E+FMsE/Ee18wUPgGMhEK9TDSxtjS5B7UHTE2rtDFlViCHuODXFlUEFBv9qDXAVlppQ2iI3YnZKCM6+qIPQs2YlURPyrhloYRUFb2UocaEMQ3IVxRoH+NOSKRgjjrmh6eXIijhD+RR9T+hYM5fmBpBJYUEap6bIKfMBpSEliohCGhU3MLQ1AEUYOGiBbcRfjH4g2WIEV+UpV/WaJGSlGHocqwiXqCTVwRomVVMn0SSyNmOtVEZDiqn9YD7EKDAPz5MKDzO1qpXerCA47C6Y5MCdyKNR6MlFrEkFIz754JIFpeTSPM92IR7heISR1QDI4kYkODjFTpQEgAzLVsqm2Bls9U/48ECpoEpMuXZW+MTeokhBDcWGo2h5OM5WRoHfC1YnU7KHwJNTE3lksIyvVggIiDcMmgiiHj8vvlL84YZCspGBIaRRXTIWktaKSBgP1TOpScx4G/8ZW22jPf6nVVo/HJ6haoqapFR602m7vbcf3Ti/eO7baVKFQLDtoMlHDaYmEJVmfA9lewCbWmy4UY/WIPmQRJr2ofrs3vjqqwBgA3jdBbCXoUs/RYIqaHflgajagUAU6UKirLXJpnd1PbmqrQvgwMRjl0AmI6W5WoukHQkglhQ3pMleTDoxqdonJu8FVR0tIlboEAsUIWZ3UHYsXomxVhLW6GhVq6oFnGTRrwfjJSlStNAn3shPnCeBkZdHcUvHYW3UZiSdSUXtD3eQaLVw6DEKlVsLftqpyUJMkh1pCdyZNX4ZKrSWMOohnWeVO5KIkHU7igV5i5E4NAOtg0Utf44Vh3BlPYk4FvU4oVCIINOLpTcePqrSA7QUFIH0V8hFi4W61IDVKOjYh7c6oNkQ93qIvCLZAVcBJXQX0BZiszltqoWMJsNXdQneurnJSMvE/i3Y7qIcaQcJc6oiuRr2i2aJoVIMSSZHgmDqlPlAiaIVb8+ludB+SMpBMU/Ce2QeLkFjjorTuF66UJoGpJJOAoCON2MYcYYh1W6UeZgP5FhXMVsUlbUZL72pwq2/Xez3L+ro6o4ibyOTiCtYODnUVZ+K+glCBPsyj7HiqBCEFkZKnPlMhJYvqFw+DDFPWrO4NpbCra4WDdebHRkKUQaqQqVO9flZflS5UpflKNi8QDM2ZVejoYEz5AeUXqZvQlESUBDkBO6baTJSk1KGyTZjbhKOkMEUWwTvV4U5FABWGjFrgWXYE8tpzEECwuMJzFDRYokicVmha9f9OShRYNYwwrVzIXCQQi8JTXdGktYeZiiBE6jg7I8Aov1VAmdoXMQxIv4CnsAYsjFhVEUBdEU+uvi+FIcGOJFlVRWEGjBK6X8IHI6P2YFdEWM6OSUNUVzdMXA1lqYNSdpgCxkDpRoFo6viC+4T1UqkSxQngdh0rxKGiZktTBc3bqoyyeiMh7p4POMbs0rWrfwLbqX9Jsd6jQJyqxgUShe3iTZC3iOrqA39KATK6OyUHGAFSo5hUHpQULd7UKa469Kmtl0U77N+qOwuKEYaKxlSBKBhhyEyFC6XFObp687AM+ElqrBq6gEajSuXG0w94caq9WNc5wySATGAdDXf1f3gBmGY1oKlKD+Qq3opMCQKoIzOWzJCh4I1SuvooMWdQbCCQdKgAdxP17HmWEC4wxGDdgh9f0CeBiGE8Dkh8hCCtJE9ZZ3SYhBQl00NkHCBpp9I2HW2pyoKI2HAv8eQZO1D9Jp3nnI87ckUJsJeY2zUgq4efZPN1pNfUQSzlgRCUvKR4LnZGeR0Gzj5ymE2IJJ0mDSgUnutL7RG76gXmcLcCjZIk5yTun4+DhGY6Y3xg+0n5g8AdPVN0UP7B1O16pvtSOegQD5yBhcrCVsga+5Ai0XgVnQQwtQFUeq4oWX1CQLaYVDGl7L06mCwHbET1AWhWw/XIV+hAw7dq1qJPG0AmMKlhfqh7oSIIBHbtlYkR2bVGhtHJjPwiJihLIao4E0l6SEvnGzZqpivMntZBF6w0fIS0lLt672iTPnUeSZB2ZQBJM6Lpjckd+i0qGaSVlR8EQe6xHXG7Q0yqyZpuPMCwTTcDapzcqgogRTWVEIcUoi2vsraUcDEmTlXPp7RFqoupFNXERozb6rtDIoNpkUfCzeaJqNZhbyhnKClVHagrttqYUHCRtopJW91prAHpjEU60ARRlHQPIoXeguWLChMWsXrbVVVL1qFfMFqSoUE7SafeJExX49ska+O9Ur7p5FAytZ+v7FTFMAIo3Ov5NJIk6XPBD8GJhXFwtPK5joM9oWSFN20VaEN7jfyq2lBBruJg1ZVSPXPqdCtcyVTxOFLLrqaq2uAmXmaroSrUoarWzYrbuT/sE9HXayw5G6XA1G2AxkZRwEMNYgVOiB1AHXRE6K5TAFVN03vS5nuc3Hx/9dKoQNsqLxvJKw6a4Xj5iSK/gjSSjnB3RtCq5xyypRQ+6ePkrK1ywlSMaDEWphQBX7w1CBtrQFlwflfVHU6+ZtOBmmTwOF+t0BESMd9HrA1M19NWQSC4IWNRUa4vPNLvrraM+g/j3OvT6Qa4GvNUEbsnaKaldY7kahvW/EAhqsijei7qwWYdgRDKGes/ELarzQZOTqRDXxQqt0LmoShTY8DVUUIsdSlE6hwI2mos90GC0XXmpcbPKnVRKOSALkEcXri7Wurbv226h/Z8y/f+1mp4uza+MHv6z/Da7cdH124Z/yvu7h5HH6b5udkTG/vZe7OXG3frSpxoQp2Gdemu1ctMunT37Zu/vPn07i/bpzcf//zh3afL/btXX+dz5+2X32x+r37bi/vNn12Nvr/f3Prua9eSB342wMcT1I1W0O3Mw+MPJykmUpr5js+RCHu+Dl5ybdw9rHs5j3x7OjzGCOV8q/DD3WPsXVqCHI9fa2ryna8g3pZ2e/j2sI/b0w8nvtG1nHx8zMP0PPT6rcOz67Lenm5Pb3v4cHx63e7hQzfDRLZ9q5vf35yCG4zbF28wfn578WdvPZ5OIy0qZgG65jntHNJDV93gqXl8PBwZoGMvPWMGgFvPdOiH+WtTJ2CJudNARpIV6oTwuK+DYh0ZXr6kQgrOasdn4/KZ9frlKTgJnLDW45TUM3VdCLh9vbEN060svX5dJ09nlybcDhtqdt3Qde+3Z29PHRYuoJUdH3eIno/acU6eZXcdvRy/3vve6xzLTNeF9v75jg7PDns/PL1a6Tjl1Zy3rx8tf1vnzUeHDR2cGfj9OfCeEStGqudL9ukStrxd0uV671f//OHNJ0L2gk8PW/ufHQn82qOAQ9xmHS7P/DJub49v8QBaUCaWu7gt+g8ARnoZt6ANRm13cVvUc6TwO8Zt8ctnjnFbso7m5n3clqIWkt3FLc8UIi/itlTFk9/HbanXDV33fnt2F7e3x7d4uM15i5zj1w9BdlvoLRpvOzo8O+z98PRqpeOUV3Mevn6w/GGdVx8dNnRwZuD33xK3vD3K9b/IOMetXeL2+9P/AwIlllkKZW5kc3RyZWFtCmVuZG9iagoxMSAwIG9iago0NTU3CmVuZG9iagoxOSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDkyID4+CnN0cmVhbQp4nD2MsQ3AMAgEe6b4BSJhjG3YJ0rl7N/mLSdp4PQP19KgOKxxdlU0HziLfHhL9YSNxJSmlUdTnN3aFg4rgxS72BYWXmERpPJqmPF5U9XAklKU5c36f3c9x6sbugplbmRzdHJlYW0KZW5kb2JqCjE3IDAgb2JqCjw8IC9CYXNlRm9udCAvRGVqYVZ1U2Fucy1PYmxpcXVlIC9DaGFyUHJvY3MgMTggMCBSCi9FbmNvZGluZyA8PCAvRGlmZmVyZW5jZXMgWyAxMjAgL3ggXSAvVHlwZSAvRW5jb2RpbmcgPj4gL0ZpcnN0Q2hhciAwCi9Gb250QkJveCBbIC0xMDE2IC0zNTEgMTY2MCAxMDY4IF0gL0ZvbnREZXNjcmlwdG9yIDE2IDAgUgovRm9udE1hdHJpeCBbIDAuMDAxIDAgMCAwLjAwMSAwIDAgXSAvTGFzdENoYXIgMjU1IC9OYW1lIC9EZWphVnVTYW5zLU9ibGlxdWUKL1N1YnR5cGUgL1R5cGUzIC9UeXBlIC9Gb250IC9XaWR0aHMgMTUgMCBSID4+CmVuZG9iagoxNiAwIG9iago8PCAvQXNjZW50IDkyOSAvQ2FwSGVpZ2h0IDAgL0Rlc2NlbnQgLTIzNiAvRmxhZ3MgOTYKL0ZvbnRCQm94IFsgLTEwMTYgLTM1MSAxNjYwIDEwNjggXSAvRm9udE5hbWUgL0RlamFWdVNhbnMtT2JsaXF1ZQovSXRhbGljQW5nbGUgMCAvTWF4V2lkdGggMTM1MCAvU3RlbVYgMCAvVHlwZSAvRm9udERlc2NyaXB0b3IgL1hIZWlnaHQgMCA+PgplbmRvYmoKMTUgMCBvYmoKWyA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMAo2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDMxOCA0MDEgNDYwIDgzOCA2MzYKOTUwIDc4MCAyNzUgMzkwIDM5MCA1MDAgODM4IDMxOCAzNjEgMzE4IDMzNyA2MzYgNjM2IDYzNiA2MzYgNjM2IDYzNiA2MzYgNjM2CjYzNiA2MzYgMzM3IDMzNyA4MzggODM4IDgzOCA1MzEgMTAwMCA2ODQgNjg2IDY5OCA3NzAgNjMyIDU3NSA3NzUgNzUyIDI5NQoyOTUgNjU2IDU1NyA4NjMgNzQ4IDc4NyA2MDMgNzg3IDY5NSA2MzUgNjExIDczMiA2ODQgOTg5IDY4NSA2MTEgNjg1IDM5MCAzMzcKMzkwIDgzOCA1MDAgNTAwIDYxMyA2MzUgNTUwIDYzNSA2MTUgMzUyIDYzNSA2MzQgMjc4IDI3OCA1NzkgMjc4IDk3NCA2MzQgNjEyCjYzNSA2MzUgNDExIDUyMSAzOTIgNjM0IDU5MiA4MTggNTkyIDU5MiA1MjUgNjM2IDMzNyA2MzYgODM4IDYwMCA2MzYgNjAwIDMxOAozNTIgNTE4IDEwMDAgNTAwIDUwMCA1MDAgMTM1MCA2MzUgNDAwIDEwNzAgNjAwIDY4NSA2MDAgNjAwIDMxOCAzMTggNTE4IDUxOAo1OTAgNTAwIDEwMDAgNTAwIDEwMDAgNTIxIDQwMCAxMDI4IDYwMCA1MjUgNjExIDMxOCA0MDEgNjM2IDYzNiA2MzYgNjM2IDMzNwo1MDAgNTAwIDEwMDAgNDcxIDYxNyA4MzggMzYxIDEwMDAgNTAwIDUwMCA4MzggNDAxIDQwMSA1MDAgNjM2IDYzNiAzMTggNTAwCjQwMSA0NzEgNjE3IDk2OSA5NjkgOTY5IDUzMSA2ODQgNjg0IDY4NCA2ODQgNjg0IDY4NCA5NzQgNjk4IDYzMiA2MzIgNjMyIDYzMgoyOTUgMjk1IDI5NSAyOTUgNzc1IDc0OCA3ODcgNzg3IDc4NyA3ODcgNzg3IDgzOCA3ODcgNzMyIDczMiA3MzIgNzMyIDYxMSA2MDgKNjMwIDYxMyA2MTMgNjEzIDYxMyA2MTMgNjEzIDk5NSA1NTAgNjE1IDYxNSA2MTUgNjE1IDI3OCAyNzggMjc4IDI3OCA2MTIgNjM0CjYxMiA2MTIgNjEyIDYxMiA2MTIgODM4IDYxMiA2MzQgNjM0IDYzNCA2MzQgNTkyIDYzNSA1OTIgXQplbmRvYmoKMTggMCBvYmoKPDwgL3ggMTkgMCBSID4+CmVuZG9iagoyNCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDgwID4+CnN0cmVhbQp4nEWMuw3AMAhEe6ZgBH4mZp8olbN/GyBK3HBPunu4OhIyU95hhocEngwshlPxBpmjYDW4RlKNneyjsG5fdYHmelOr9fcHKk92dnE9zcsZ9AplbmRzdHJlYW0KZW5kb2JqCjI1IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjQ4ID4+CnN0cmVhbQp4nC1ROZIDQQjL5xV6QnPT77HLkff/6QrKAYOGQyA6LXFQxk8Qlive8shVtOHvmRjBd8Gh38p1GxY5EBVI0hhUTahdvB69B3YcZgLzpDUsgxnrAz9jCjd6cXhMxtntdRk1BHvXa09mUDIrF3HJxAVTddjImcNPpowL7VzPDci5EdZlGKSblcaMhCNNIVJIoeomqTNBkASjq1GjjRzFfunLI51hVSNqDPtcS9vXcxPOGjQ7Fqs8OaVHV5zLycULKwf9vM3ARVQaqzwQEnC/20P9nOzkN97SubPF9Phec7K8MBVY8ea1G5BNtfg3L+L4PePr+fwDqKVbFgplbmRzdHJlYW0KZW5kb2JqCjIyIDAgb2JqCjw8IC9CYXNlRm9udCAvRGVqYVZ1U2FucyAvQ2hhclByb2NzIDIzIDAgUgovRW5jb2RpbmcgPDwgL0RpZmZlcmVuY2VzIFsgNDkgL29uZSAvdHdvIF0gL1R5cGUgL0VuY29kaW5nID4+IC9GaXJzdENoYXIgMAovRm9udEJCb3ggWyAtMTAyMSAtNDYzIDE3OTQgMTIzMyBdIC9Gb250RGVzY3JpcHRvciAyMSAwIFIKL0ZvbnRNYXRyaXggWyAwLjAwMSAwIDAgMC4wMDEgMCAwIF0gL0xhc3RDaGFyIDI1NSAvTmFtZSAvRGVqYVZ1U2FucwovU3VidHlwZSAvVHlwZTMgL1R5cGUgL0ZvbnQgL1dpZHRocyAyMCAwIFIgPj4KZW5kb2JqCjIxIDAgb2JqCjw8IC9Bc2NlbnQgOTI5IC9DYXBIZWlnaHQgMCAvRGVzY2VudCAtMjM2IC9GbGFncyAzMgovRm9udEJCb3ggWyAtMTAyMSAtNDYzIDE3OTQgMTIzMyBdIC9Gb250TmFtZSAvRGVqYVZ1U2FucyAvSXRhbGljQW5nbGUgMAovTWF4V2lkdGggMTM0MiAvU3RlbVYgMCAvVHlwZSAvRm9udERlc2NyaXB0b3IgL1hIZWlnaHQgMCA+PgplbmRvYmoKMjAgMCBvYmoKWyA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMAo2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDMxOCA0MDEgNDYwIDgzOCA2MzYKOTUwIDc4MCAyNzUgMzkwIDM5MCA1MDAgODM4IDMxOCAzNjEgMzE4IDMzNyA2MzYgNjM2IDYzNiA2MzYgNjM2IDYzNiA2MzYgNjM2CjYzNiA2MzYgMzM3IDMzNyA4MzggODM4IDgzOCA1MzEgMTAwMCA2ODQgNjg2IDY5OCA3NzAgNjMyIDU3NSA3NzUgNzUyIDI5NQoyOTUgNjU2IDU1NyA4NjMgNzQ4IDc4NyA2MDMgNzg3IDY5NSA2MzUgNjExIDczMiA2ODQgOTg5IDY4NSA2MTEgNjg1IDM5MCAzMzcKMzkwIDgzOCA1MDAgNTAwIDYxMyA2MzUgNTUwIDYzNSA2MTUgMzUyIDYzNSA2MzQgMjc4IDI3OCA1NzkgMjc4IDk3NCA2MzQgNjEyCjYzNSA2MzUgNDExIDUyMSAzOTIgNjM0IDU5MiA4MTggNTkyIDU5MiA1MjUgNjM2IDMzNyA2MzYgODM4IDYwMCA2MzYgNjAwIDMxOAozNTIgNTE4IDEwMDAgNTAwIDUwMCA1MDAgMTM0MiA2MzUgNDAwIDEwNzAgNjAwIDY4NSA2MDAgNjAwIDMxOCAzMTggNTE4IDUxOAo1OTAgNTAwIDEwMDAgNTAwIDEwMDAgNTIxIDQwMCAxMDIzIDYwMCA1MjUgNjExIDMxOCA0MDEgNjM2IDYzNiA2MzYgNjM2IDMzNwo1MDAgNTAwIDEwMDAgNDcxIDYxMiA4MzggMzYxIDEwMDAgNTAwIDUwMCA4MzggNDAxIDQwMSA1MDAgNjM2IDYzNiAzMTggNTAwCjQwMSA0NzEgNjEyIDk2OSA5NjkgOTY5IDUzMSA2ODQgNjg0IDY4NCA2ODQgNjg0IDY4NCA5NzQgNjk4IDYzMiA2MzIgNjMyIDYzMgoyOTUgMjk1IDI5NSAyOTUgNzc1IDc0OCA3ODcgNzg3IDc4NyA3ODcgNzg3IDgzOCA3ODcgNzMyIDczMiA3MzIgNzMyIDYxMSA2MDUKNjMwIDYxMyA2MTMgNjEzIDYxMyA2MTMgNjEzIDk4MiA1NTAgNjE1IDYxNSA2MTUgNjE1IDI3OCAyNzggMjc4IDI3OCA2MTIgNjM0CjYxMiA2MTIgNjEyIDYxMiA2MTIgODM4IDYxMiA2MzQgNjM0IDYzNCA2MzQgNTkyIDYzNSA1OTIgXQplbmRvYmoKMjMgMCBvYmoKPDwgL29uZSAyNCAwIFIgL3R3byAyNSAwIFIgPj4KZW5kb2JqCjMwIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjkyID4+CnN0cmVhbQp4nCVSS24FMQjb5xS+QKXwJ+eZ6q1e77+tySxGECDGdqZUsREbPyLINpQ2fmVFNzQEf2uKqoXvyjSIHyRPfRAV4OlZoYYDzxfEOm98lu1zM/WEQ07DguULkRBRfoIuSM14KtRkuOiJCcZ9RN9wK6SzpDiKiB4U3UghbJJ3JJR59uArwUsMpn7VGKVMfJbHuVkII8lFNrJmSQo3zBZKDgrIoincwPVVVNmUHxQYZBOWnCnSCTIY5k6MpDY3cvC6FkykBbZvps2O0UjmuaejQqQWCDvhQR3kswdNwuFBVzjrxJ9olD/OMaMJMnrpopRD9+2cqfLJqEyoW+c+J7nnOJpeDrUDq55AynSaiVdiCNAukTEvBpZ0oubFxoz3P3jW5x8e1GeBCmVuZHN0cmVhbQplbmRvYmoKMzEgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyNTAgPj4Kc3RyZWFtCnicPVA5cgRBCMvnFXoCN93vWZej3f+nFrjsYEY0UDpoMwi6+au6aC186bPvz8A03o9b/FX9uxSmREMkUQqvJ66hbiGtUX2QWagMTvIGijrz5VFUKNKNk8qLOLbSYbXotZOphEx8GbcqBaY9E29oJ9kUGrQtRPXRIfdJCnM9qJDknrYqqBbHYULDAh2FmBRwOhU4d4W9zf1+tA0neAC3nGJibyfk4hyEF+54CbpTpg/OVC9SSE4uijKrZiPpUA8xmcN2Qm9WjmzaIt9irX9W4XM5SswlfVLURhDujc7kdUQw/14jqFq83OQon6C2+NoQU/3HeT/fP1IVXH0KZW5kc3RyZWFtCmVuZG9iagozMiAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDQzNyA+PgpzdHJlYW0KeJw1kklyJTEIRPd1Ci7gCDFoOs/v8Mp9/22/pNyLKpAQkJkw57JhuezL3aaXzRz2x58aZavs75PbFc4a5hgfNu3zxBn2NS1qd2J4tv08Pt9S7mFhJ4xyn2dS+6jMTf09N5dyVljx+Ez6WozF9aJsbKNBVNm9FlOv3bfFuuQei307NY4SnFNcng8yb5GGTx4dAJJj05K25Ofli47Io/Nrz2tn/I8cbs4FGnk7reIoMoeV3qJDTaGItqgByb4ZsggF+MrGtvAChoV2dzbznPeVRNL+PJwKjCpGEB61JJmPY4V+nmlzSPzNfIQwBmrGy1PTilZPOeImL9FQLxK5NdPPIwyTkRac6/JN/K1JFnVLGDasqFiHqAt7Hd6IESq3CrLZ1fACPX/a85zEmFh16SWMBVfBGwxpNIbRKAJLFjwcekOi2O+qvdIH5Fm69e6WhhYIGdqO0BqobUjQq61DUGDHuC01NyPNNQCIe6lJ7ySgfR2AEoF42+wcearCUl2YsLynxd8NSfOcQlDWOxgU0fkeRROF9/1dDPYut4phj5r3PC4QICRizj41wXeXfqn+PN//ABlPplMKZW5kc3RyZWFtCmVuZG9iagozMyAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDI1NSA+PgpzdHJlYW0KeJw1UMltBDEM+7sKNhBAp2XXs0Fem/6/oTQbjAcibJGUmLkh8MKXKlIT6YJvXWEK3YXflWpQdr1X3IKKIUqwFeEGntfy6+AXMSJ2nvpaJmeQBnkUEUce3ucljjbVGm/LbJmihoGvoTIdMe0aBykbJjXTWd2pZPQLUUhORwS55L84qlPFZiOPPdV2cwZl8CZgHGwqreljNei9lJpKFyVTnX8l59mzUqA4SkwCveruTV13g45gXzhzO93t5z6BSQfA2T6h0quzk8t4wx7EePXA06fbD+cmuzF1Ou2gvj2Z2JFPNub3uWECQXetw73HIRnt5R5OJe777/haP39JF1y6CmVuZHN0cmVhbQplbmRvYmoKMzQgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyNDYgPj4Kc3RyZWFtCnicRVE7bsUwDNt9Cl6ggPW1fZ4UnV7vv5ZMAnRIxNgSSTFVjYl0fJmhrLFm49sGT2xv/A6LhJ3CZ1hOWOlpGDttG07iGs6RZfBo9IQTslwjLAQiD1Yj1oHNzfPkW1zpQQ6/q0fpRmgX1BGeiM3xCnGV84uPFeIsisy7UpxO7xM6ikN3J6ilG1NP071m89EMl4NaiNhayZ+FPyNJ/o/aXbekfVFtZEwin4bUltnIVXDKqcpi3Ujmk6az2GkKIplSdN/xxhuzp9YSssV+KhmVspjVnQSzM7okh36MMlV9shYyKnDGOCMirsp8UywL77+7xs8fHkpY9gplbmRzdHJlYW0KZW5kb2JqCjM1IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggNDggPj4Kc3RyZWFtCnicMzIyUjBQMDMBEoamRgrmhmYKKYZcYH4uiAIJ5HDBpCAsAyANVpHDlQYAgA4MJQplbmRzdHJlYW0KZW5kb2JqCjM2IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjc0ID4+CnN0cmVhbQp4nE1SS3bEMAjb+xQcwfzxeaavq+n9txU409dFImKBkJKUKm2KwC3jkOumL17z/NPgfOi92PxfZRZdBZMlE5eQHSbZGN9JryWKORGSyBHULYOvpbbvCea6Qw86d4Ax2VDBpUWGOTOgnmbqgIG2XZXY9ahFXLVolp1SMFftIB0u/Uwkawao3nu62nAfxX+omHsqZIos0gogcsF57wmoFAUUrPcZkts4EJzYgSfscSOvi6/lLvcEKa37D/Jwe7M05FakRH50DG5uBlV7UnR8UDU/VQb8Yd92zEFVvN9ovy8Dyzb7pORxIJ73RMFYkjB2ajN8ehpfLnMSciBxtjf2Gm32VoxBiTPM9TR/xnt9/wJnsGqfCmVuZHN0cmVhbQplbmRvYmoKMzcgMCBvYmoKPDwgL0JCb3ggWyAtNjY1IC0zMjUgMjAyOSAxMDM4IF0gL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAzNQovU3VidHlwZSAvRm9ybSAvVHlwZSAvWE9iamVjdCA+PgpzdHJlYW0KeJzjMjWyUDA2NFLI5TI1AzNywAxLExADJIdggSXTAO7GCbUKZW5kc3RyZWFtCmVuZG9iagozOCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDExNCA+PgpzdHJlYW0KeJw1TssNQzEMumcKRvDf8Tyv6ind/1rHai8GYUC4BwhM1VdTkVx48bqU8FmyvfEMegwLhRtBtJU2CzGsCs/iSFgWWAMWNqXmdj/NXKvT7Lt7ZFJet2UjRNsjaQh3KBFiJ5RjxjzrP+v8Vp31/gItliJeCmVuZHN0cmVhbQplbmRvYmoKMzkgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyNzYgPj4Kc3RyZWFtCnicTZFLcsMwDEP3PgUvkBnxK+k86XSV3H/bR7npdGGThigAhDNLhlTJQ/eS1JL0IV96faB3d6lbXpfG/y5Su6uQmFN0gewppoOZIc/LPCTNxcoOp+2b+3l5jNP53MwuCXXuFicREza+pkmEgjK1Nyc5pnjO49DVTrXyPumuVUeJohULN9Y6UUuwFsgFLkeIWcsDQ4uBhyq27orh+kUw/kg4VSawNt+GegkHmmwVDfM+Ab3+orpzMRJ9n04X15IHA52PjtUybDsZY6AQW9EFV0RF49zGswPriTFYVoNIDIIdp1q1g+56i57oKH3l6eFKQmVlZyKOyDoV8Rw3op2LH4txbGn1DwHBl5vJZ5Xn9f0DZepl8gplbmRzdHJlYW0KZW5kb2JqCjQwIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggNDIgPj4Kc3RyZWFtCnicMzK3UDBQsDQEEoZA0tDAQCHFkAvMz+WCCuRwGaKwQDSUSgMAfswMEgplbmRzdHJlYW0KZW5kb2JqCjQxIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggNDA4ID4+CnN0cmVhbQp4nC2TOXIDQQhF8zkFF1BVs/R2Hrkc2fdP/T5yoIGhp/kLaI5hw9Lt5W613GYO+/KHis9pv4/7MV/Hfh6PMM/kt8wHv3nsHHs/fobtYeFhNIjZ4f3E7SS5tq5lhZ1JOan5oL6J8R8rdaJspeUCaB+uTPM7dCLYS2WkxThgTIvQiV8QRagW1dEdg/vv51LYZXtb0GMVIsVqgphhtE6aKByVSWqU0aFiinaVyG6ZMu0sqyPaZXVLsLgyeZMXE92+BvG2GXQJsMdtL0VOET/2J0u+nwEfROuuhAuZk7vBgQlVwUKLTmJSdCkwCxfzY+NcWJfMJTE8rxwW+dGGV/Y32FVICkwophWVHeEyojPfqmjW9M8eJs8KKaMbGhTzep+Q7ds7kEzUCytXD6EYjcyft1X5xtbc7QbfZrYbKVfE1eWgnqGRihee5YmeF5rZrWANpD0K5uiK2D0k7ozde+onPnHKwc6km7c7W/7SNNozKFwogNGrJ/C49hJ+9N6L1au3Q9NTJo100sZRZZ9gCQ25/PljvJ/vP4XjmJkKZW5kc3RyZWFtCmVuZG9iago0MiAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDEzOCA+PgpzdHJlYW0KeJw1j0sSwyAMQ/ecQkfwF+PzpNMVvf+2IplsQJaYZ5E5IYjikaooKXx0cJ5m+B1xrD3e8FHTF1XMRK5GaCMt4JWICFzDXeAzYJ2wpbBSaBcTS4d6wcJA0wgS2no32LwX2EizoSTqEpgcogkfLxJdSX6I4Xl2sU9Kw0lOut7rLn+9v9jj+wdnSysWCmVuZHN0cmVhbQplbmRvYmoKNDMgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxNyA+PgpzdHJlYW0KeJwzMrdQMIDDFEMuABrjAvEKZW5kc3RyZWFtCmVuZG9iago0NCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDE3MCA+PgpzdHJlYW0KeJxFUDkOwzAM2/0KfiCARR+y3pOiU/v/tZRTJItJkLJIm75QYQvHBN3gteJlhWPBA9+SmuFT2AeOBrLtydoTzmLOJNYdhwZbxUrVmCtNu5ohGnqqa2B2LCIiTxtMkeijKkDzNxkWIrJuMhUga8YueLHLzKYP+6+Q+zC77xrV0fXcOoQdscu6I6QrRQ1tqZylHBNyWAUDVILgLOQm7ITrH65vOsv7BzKGPYkKZW5kc3RyZWFtCmVuZG9iago0NSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDI2NyA+PgpzdHJlYW0KeJw1UUlywzAMu/sVeIK4i+9Jp6fk/9eCzHTGMmhzA6CIxIE2X3EMJY0feSa8js8GB+/HzgLrVGAGl3lS8HrC0GxUiDr6Qjjx9cyH3IKkQZVHeDKY0eYEvTA3WBFrZk2Psdtjhiv83sVQZWYjzrVuxCWWc/mZHm+kOUwK6QmtL3KPxffPIVFSlkrkucMtKPaSsBXC64tn9zDgqveIimpMC6UL6WWuLJIoDlSR9UqniDhEaiPnoCRNd+Ia5FyVtGBWBCcu6pCfyGmHd8JplNNzt1gizJxaO8YkV4r2uyb1irVwbg+MnbomqdF81uqh9ayV25Q2GaFdo0GSog/1hM71vv7v+f38/gErHWDYCmVuZHN0cmVhbQplbmRvYmoKNDYgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyNzIgPj4Kc3RyZWFtCnicNVFLbgUxCNvPKXyBSvxJzjNVd73/tibpk2YECdgYJ7MgCMOXKlIWWje+9eGNp+N3kvBmiV+iFjIb77OYy4YSVcEYPPcUtDeanWZ+uKzzxPdxvTcezajwLtROVkKC6E0ZC0X6YEcxZ6UKuVlZVFeB2IY0YyWFwpYczcFZE0fxVBasiCHORNll1LcPW2KT3jeSKKp0GWGt4LrWx4QRPPF9TG6myd+5q1EV78mipmOa6Qz/n6v+8Wwy8zyuKPfRHvQ6lAIuas6F5Yyqo0BP4rGmOsbc9jFmCIKnIZx4h00W1D0dGReTazBDUlZw5YwoDrmRw93vDU0p46PxwfI8gNLwPFvS1BZ8Vnmfnz/0lmVLCmVuZHN0cmVhbQplbmRvYmoKMjggMCBvYmoKPDwgL0Jhc2VGb250IC9BcmlhbE1UIC9DaGFyUHJvY3MgMjkgMCBSCi9FbmNvZGluZyA8PAovRGlmZmVyZW5jZXMgWyAzMiAvc3BhY2UgNDYgL3BlcmlvZCA0OCAvemVybyAvb25lIC90d28gNTMgL2ZpdmUgNTUgL3NldmVuIDY3IC9DIC9EIDk3Ci9hIDEwMSAvZSAxMDggL2wgL20gMTEyIC9wIDExNSAvcyAvdCBdCi9UeXBlIC9FbmNvZGluZyA+PgovRmlyc3RDaGFyIDAgL0ZvbnRCQm94IFsgLTY2NSAtMzI1IDIwMjkgMTAzOCBdIC9Gb250RGVzY3JpcHRvciAyNyAwIFIKL0ZvbnRNYXRyaXggWyAwLjAwMSAwIDAgMC4wMDEgMCAwIF0gL0xhc3RDaGFyIDI1NSAvTmFtZSAvQXJpYWxNVAovU3VidHlwZSAvVHlwZTMgL1R5cGUgL0ZvbnQgL1dpZHRocyAyNiAwIFIgPj4KZW5kb2JqCjI3IDAgb2JqCjw8IC9Bc2NlbnQgOTA2IC9DYXBIZWlnaHQgNzE2IC9EZXNjZW50IC0yMTIgL0ZsYWdzIDMyCi9Gb250QkJveCBbIC02NjUgLTMyNSAyMDI5IDEwMzggXSAvRm9udE5hbWUgL0FyaWFsTVQgL0l0YWxpY0FuZ2xlIDAKL01heFdpZHRoIDEwMTUgL1N0ZW1WIDAgL1R5cGUgL0ZvbnREZXNjcmlwdG9yIC9YSGVpZ2h0IDUxOSA+PgplbmRvYmoKMjYgMCBvYmoKWyA3NTAgNzUwIDc1MCA3NTAgNzUwIDc1MCA3NTAgNzUwIDc1MCA3NTAgNzUwIDc1MCA3NTAgNzUwIDc1MCA3NTAgNzUwIDc1MAo3NTAgNzUwIDc1MCA3NTAgNzUwIDc1MCA3NTAgNzUwIDc1MCA3NTAgNzUwIDc1MCA3NTAgNzUwIDI3OCAyNzggMzU1IDU1NiA1NTYKODg5IDY2NyAxOTEgMzMzIDMzMyAzODkgNTg0IDI3OCAzMzMgMjc4IDI3OCA1NTYgNTU2IDU1NiA1NTYgNTU2IDU1NiA1NTYgNTU2CjU1NiA1NTYgMjc4IDI3OCA1ODQgNTg0IDU4NCA1NTYgMTAxNSA2NjcgNjY3IDcyMiA3MjIgNjY3IDYxMSA3NzggNzIyIDI3OAo1MDAgNjY3IDU1NiA4MzMgNzIyIDc3OCA2NjcgNzc4IDcyMiA2NjcgNjExIDcyMiA2NjcgOTQ0IDY2NyA2NjcgNjExIDI3OCAyNzgKMjc4IDQ2OSA1NTYgMzMzIDU1NiA1NTYgNTAwIDU1NiA1NTYgMjc4IDU1NiA1NTYgMjIyIDIyMiA1MDAgMjIyIDgzMyA1NTYgNTU2CjU1NiA1NTYgMzMzIDUwMCAyNzggNTU2IDUwMCA3MjIgNTAwIDUwMCA1MDAgMzM0IDI2MCAzMzQgNTg0IDc1MCA1NTYgNzUwIDIyMgo1NTYgMzMzIDEwMDAgNTU2IDU1NiAzMzMgMTAwMCA2NjcgMzMzIDEwMDAgNzUwIDYxMSA3NTAgNzUwIDIyMiAyMjIgMzMzIDMzMwozNTAgNTU2IDEwMDAgMzMzIDEwMDAgNTAwIDMzMyA5NDQgNzUwIDUwMCA2NjcgMjc4IDMzMyA1NTYgNTU2IDU1NiA1NTYgMjYwCjU1NiAzMzMgNzM3IDM3MCA1NTYgNTg0IDMzMyA3MzcgNTUyIDQwMCA1NDkgMzMzIDMzMyAzMzMgNTc2IDUzNyAyNzggMzMzIDMzMwozNjUgNTU2IDgzNCA4MzQgODM0IDYxMSA2NjcgNjY3IDY2NyA2NjcgNjY3IDY2NyAxMDAwIDcyMiA2NjcgNjY3IDY2NyA2NjcKMjc4IDI3OCAyNzggMjc4IDcyMiA3MjIgNzc4IDc3OCA3NzggNzc4IDc3OCA1ODQgNzc4IDcyMiA3MjIgNzIyIDcyMiA2NjcgNjY3CjYxMSA1NTYgNTU2IDU1NiA1NTYgNTU2IDU1NiA4ODkgNTAwIDU1NiA1NTYgNTU2IDU1NiAyNzggMjc4IDI3OCAyNzggNTU2IDU1Ngo1NTYgNTU2IDU1NiA1NTYgNTU2IDU0OSA2MTEgNTU2IDU1NiA1NTYgNTU2IDUwMCA1NTYgNTAwIF0KZW5kb2JqCjI5IDAgb2JqCjw8IC9DIDMwIDAgUiAvRCAzMSAwIFIgL2EgMzIgMCBSIC9lIDMzIDAgUiAvZml2ZSAzNCAwIFIgL2wgMzUgMCBSCi9tIDM2IDAgUiAvb25lIDM4IDAgUiAvcCAzOSAwIFIgL3BlcmlvZCA0MCAwIFIgL3MgNDEgMCBSIC9zZXZlbiA0MiAwIFIKL3NwYWNlIDQzIDAgUiAvdCA0NCAwIFIgL3R3byA0NSAwIFIgL3plcm8gNDYgMCBSID4+CmVuZG9iagozIDAgb2JqCjw8IC9GMSAyOCAwIFIgL0YyIDE3IDAgUiAvRjMgMjIgMCBSID4+CmVuZG9iago0IDAgb2JqCjw8IC9BMSA8PCAvQ0EgMCAvVHlwZSAvRXh0R1N0YXRlIC9jYSAxID4+Ci9BMiA8PCAvQ0EgMSAvVHlwZSAvRXh0R1N0YXRlIC9jYSAxID4+Ci9BMyA8PCAvQ0EgMC44IC9UeXBlIC9FeHRHU3RhdGUgL2NhIDAuOCA+PiA+PgplbmRvYmoKNSAwIG9iago8PCA+PgplbmRvYmoKNiAwIG9iago8PCA+PgplbmRvYmoKNyAwIG9iago8PCAvRjEtQXJpYWwtbWludXMgMzcgMCBSIC9JMSAxMiAwIFIgL00wIDEzIDAgUiAvTTEgMTQgMCBSID4+CmVuZG9iagoxMiAwIG9iago8PCAvQml0c1BlckNvbXBvbmVudCA4IC9Db2xvclNwYWNlIC9EZXZpY2VSR0IKL0RlY29kZVBhcm1zIDw8IC9Db2xvcnMgMyAvQ29sdW1ucyAxNTEwIC9QcmVkaWN0b3IgMTAgPj4KL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0hlaWdodCAxNTEwIC9MZW5ndGggNDcgMCBSIC9TdWJ0eXBlIC9JbWFnZQovVHlwZSAvWE9iamVjdCAvV2lkdGggMTUxMCA+PgpzdHJlYW0KeJzs/V12K0uTJdo5CLRQXVI31RKJQOihrnQrK23tpH3hARrJOR/92DH3+AXgm2Os2//r//n/WADwrY7vXsD1bm+eb/gpPS47IfnAixlbZ2lT56o4VJdnKXVOp/RVNwnFRzFedlhrvdbH14ufR1H8rDr8Y/yzavK57qG4GP//tIqrwbXW/7tq8v85HnVxNZ6WEWfsNDl/4J/putQXse78rO6lFS5u7x4Lncu7OnU+/3Clxza97cvy1puw9VbpN2k5+zp9vzHLe/eXAb5izO1Rq9/IAAAAAHwjWzYAAAAA49iyAQAAABjHlg0AAADAOLZsAAAAAMaxZQMAAAAwji0bAAAAgHEe370AAP6Q47sXwJtdd8W/4166nW9xVOs+QufyGHNxY/y64tdRF7/K4tD5edT/pviq/q3xFYqfVXHq/FkVl4NrredxbxT3llF07jYpi9NZqq9L6/yHi5jGe/dYdTt1H4FQ3OlQv24aj+2/mjecPcBdJn+UT1rbho8MWP7KBgAAAGAgWzYAAAAA49iyAQAAABjHlg0AAADAOLZsAAAAAMaxZQMAAAAwjpBvANjrrbmekwJNSzuCsXfM2GpyvrgMJ+4rE44by0jjZdb4SrndnRTnMgc6jbeSp1eItS4zrfcUV2HeK+R5x+KqcwzzbgWWX3bgvSsebvWUaV02Sc9LiOKu9XK7e4HZjSdxizEvdjHV8P38lQ0AAADAOLZsAAAAAMaxZQMAAAAwji0bAAAAgHFs2QAAAACMY8sGAAAAYBxbNgAAAADjPL57AQD8Tsd3L4A3O9btu5ewUXEs+ZZuHHh5llLncjyd5zh+FOOvUFyOp86v052fofgZ/k2xHG81eR6Nzp9pGVWTWHx6Gam+dZZa1yUWn77iK9xOreLWI9B7uHZ8brXehJveKlcZ/jk+Znm/6bOPifyVDQAAAMA4tmwAAAAAxrFlAwAAADCOLRsAAACAcWzZAAAAAIxjywYAAABgHCHfAPCfeXeu55BA0zGBsinotzXjhotYBgNfmdvdDPk+HZNcRjuvPfnQ9b8dvmIGdpUUviO3+/O4f7HDWutzVcVxGUVxKz48rSSdpTpPPRaXieCNzv3U+U5xORq0Hq6gtYzf8xGw5U14nSGfffBd/JUNAAAAwDi2bAAAAADGsWUDAAAAMI4tGwAAAIBxbNkAAAAAjGPLBgAAAGAcWzYAAAAA4zy+ewEA/GzHdy+At7t99wLW6t94rfpQ3DrwVFyMp7WV47E4/IejnrFe3qsaLwdj8VH/c+Cz+mfCcnCt9QwzPqvmn+teFn9WxWWHtJKyQ15GXVwvIx54Y8bWWWpexLq4dS+1xvMj0Jix5aiOccdbolv//tfpiBf4j+Xs/U7pug75iuuvbAAAAADGsWUDAAAAMI4tGwAAAIBxbNkAAAAAjGPLBgAAAGAcWzYAAAAA4wj5BoD/0VtzPYeESq4rV9LsvOX8n23STNFOxaVLM5ULMfK5Tj6u/4XvVY3nMOmqOIRJtzKwe7ndrUTwVFwvoy4+v+YVw9dbxa0rviEDvkzRzs0vSwTf8RZrxYrnCbc0OWvO50tp+PLgW/grGwAAAIBxbNkAAAAAjGPLBgAAAGAcWzYAAAAA49iyAQAAABjHlg0AAADAOLZsAAAAAMZ5fPcCAPgZju9ewK80/qzeJkzXOktbio+jWEmzczqWRudy/FWtba31CjOW46m4XN4zda5W8gz/HPjsFYfxdf96k8+qSSyuxssOqUnrWPoHXjWJd0JRXA6uHbfHEW7fVz1c39XNN2HrvdR6q7z5jXep33Qs7+fssVa4D97/zc1f2QAAAACMY8sGAAAAYBxbNgAAAADj2LIBAAAAGMeWDQAAAMA4tmwAAAAAxhHyDQD/f3801/O6xMrvSDG/LtP3qgzymAjeyRpPTcrA5rJzKs7x4WWYdF0cYqobmdapvsznXiERPOd2t+LDG8XNA0956qdzu8MVD6nzG4rz89Ka8auDq/m8fH26fnHjrfK73r0Nw5cH/7Dlm2LrEfBXNgAAAADj2LIBAAAAGMeWDQAAAMA4tmwAAAAAxrFlAwAAADCOLRsAAACAcWzZAAAAAIzz+O4FADDO8d0L+H3Gn9Lb+RbNYyxmbHU4tqw5TFkOp+WVK0nLu674lZocxb/PpeJX9Y95r6rDWutZNXnG4mL8M/zb4edxr8er+jTjZzVeLiMVl4OpSau4df5XONWxyXH2tsn30tm7d+14uDa9Tze8QFqu+xTY8jL8q5w6vkfrzvNXNgAAAADj2LIBAAAAGMeWDQAAAMA4tmwAAAAAxrFlAwAAADCOLRsAAACAcWzZAAAAAIzz+O4FAMD73b57Ad/jGHPgx9uLm8deFtcdyhnzMqrBUF2u+RWXUY+X9a/wj3bPozHj8yiaPGPnTnGnyee6f73J59Eo3nMs1SlNxa+qeMWL2LgTWsX5hvzq4D/GWw9X6Nx4EnOT88Xvf51OeYG3tE418N/5KxsAAACAcWzZAAAAAIxjywYAAABgHFs2AAAAAOPYsgEAAAAYx5YNAAAAwDhCvgH+rr8RvfnuVNQhZ/W6ZTQ7bzn/5/O563W30oJb8eGxcydFuxyPYd478qHL8O+YSF0WVwHYqUmreK312WlShn83c7sbieD5/FfFzXTzsv4IB966bXp37+ni1X24Tr/I0jJCcbKlyVlDPlyAN/NXNgAAAADj2LIBAAAAGMeWDQAAAMA4tmwAAAAAxrFlAwAAADCOLRsAAACAcWzZAAAAAIzz+O4FAABXuE2Y8ej8/1uKj/AfyuHYpD6WdEobxa3O5fjrqItfoUk5/jzqf7R79oqL8XIwNflc97L4M814FPWfnRl7x9I6S2kZ1Xj3Ih5VfSzuPIll59YNmbQerqz1Vnn/S+86P+9YWi/w7/DzTin8L/7KBgAAAGAcWzYAAAAA49iyAQAAABjHlg0AAADAOLZsAAAAAMaxZQMAAAAwjpBvgD9hfPrmee/O7xxySi9dxtuPMUULN4q7zb8+Yw4trgZj1ngjU7lMcX6Ff2+L41VSdUwEL4OxWynaobiM4m5FbucmISm8Kk6J4GWseBnmvZoHXuZ5t/K5U30r1j3Fim+41Xe8e8us8X+s5LLiDSnmWwz5fAEm8Fc2AAAAAOPYsgEAAAAYx5YNAAAAwDi2bAAAAADGsWUDAAAAMI4tGwAAAIBxbNkAAAAAjPP47gUAAGfczrc4dszYanKcXvYR5kudy/K8jGI8Fb9OF5eDa63X0SkO489q/Bn+0a4cT8Wf1fjnkTrfv9hhrfV5FMVpJanJs1rJlgN/VeP5upTFoXN1xVPz9MSV915+Ls4XN5ax8pP7deffHv0Zr/PuY9niyhNy3o88pfAP/soGAAAAYBxbNgAAAADj2LIBAAAAGMeWDQAAAMA4tmwAAAAAxrFlAwAAADCOkG+AX2V29OYuvynhteEnLmNL8RGSj1ua+cRfHVxheSnyuVxGK9o5J4I3EqnLAOyVsq5TcSdFuwz/fnbCvFcI/95zLKeL8/nfkNTei+Ku7tSc231e9/FsPInn5c4joqCHvNWByfyVDQAAAMA4tmwAAAAAxrFlAwAAADCOLRsAAACAcWzZAAAAAIxjywYAAABgHFs2AAAAAOM8vnsBADDF8d0L+J/cTv7//QM8O2PqEFbSmO7ode42KcaPo1McOr+q8XIwFh/1v7c9Q5NnVf8K/2hXFj9TcTX+edwbxaHzZzzGzvI6x1Ke1XSWXtWd0LyIjXspjTdvyFrr7i2bxM473qdpJaG4dL7DLudfp+82/jMR/gR/ZQMAAAAwji0bAAAAgHFs2QAAAACMY8sGAAAAYBxbNgAAAADj2LIBAAAAGEfIN8BP9QfSN39eJOoWrVzbTbYEZp+VUoF70cKtmOQ9xYUY+VznQ4cw6TrTOoVJ7wjGLotD5HYZ0Z06l+HfeRl1UngZ/p2Swjcc+Jak9uqKtzLg03jr7m09472HKz75rbfK73/b/4GP7Pf7/bcNLH9lAwAAADCQLRsAAACAcWzZAAAAAIxjywYAAABgHFs2AAAAAOPYsgEAAAAYx5YNAAAAwDiP714AALzb8d0L+P+7biXfcYy3/z6Ul1EUtzqnDmnGcjwVv46i+SvMWI6n4qNTHJZR/3vb8wjjVfPPUPxZNS8H11rP494oLjt3lpGa5AMvxl+huL6IsbjqHNZ8/vZYax3VnbrlVm89XGG63pN4ZXFr2VdJJ2SyOZ+JwH/nr2wAAAAAxrFlAwAAADCOLRsAAACAcWzZAAAAAIxjywYAAABgHFs2AAAAAOMI+QaY7m+kb/68VNRNzh548/bYEse74WKF0OLG8nLCcSM++ahStPOMjZDv1DmESdfFdaZ1DMAOTaqk6tikLK7CvFeI4o7F9TLq4lZud+9YOmepF+se76Xq9uiGfJ9usufh2vFp9BMzsINfcyDAaP7KBgAAAGAcWzYAAAAA49iyAQAAABjHlg0AAADAOLZsAAAAAMaxZQMAAAAwji0bAAAAgHEe370AAP5vx3cv4Hq3N8835JT+0GW06svi46iveF0cOxdNysG1VrrHWk3K8VcqPop/AEvFz+pfy17hn9DK8Vc4pWXntdZz3YvBas2pyWfqXDWJxdV4axmpPp295qluXPHyEqR7KXSu5eel9QiUHZLWO7lcxoVC8y0HftaQt3rXz1z2u785wCj+ygYAAABgHFs2AAAAAOPYsgEAAAAYx5YNAAAAwDi2bAAAAADGsWUDAAAAMI4tGwAAAIBxHt+9AAD4C27nWxxvn7FskpfRmrFRXM6YlvFKTar/4QjLeNUH3ih+hX8Vex1F8TMUP3vFYbyqT00+j3ujeFXFcRlF8eeWYwnF5XUpT+kK16vssMKd0Co+wjLSXd16BNKN2ugcltdyfhlrrfMvik22vE4B/hP+ygYAAABgHFs2AAAAAOPYsgEAAAAYx5YNAAAAwDi2bAAAAADGsWUDAAAAMI6Qb4BvcGUW6RxvTUWdc0pb0bY79NKCQ/GOTN9OinYILU7LaERup3ziEMXdSXHudH7GRPBGmHQr0zrndhfjMV27Gi87pJWkzmWTGFjeCf+Op7oqjuHrdRR3p7gV8t0p7jfpdO68KULt7w/AnvP5AvxB/soGAAAAYBxbNgAAAADj2LIBAAAAGMeWDQAAAMA4tmwAAAAAxrFlAwAAADCOLRsAAACAcR7fvQAA6Dm+ewH/y3XLGHKAK6zkWLcdnYsmqXNrGa3xVPw6ivFXKq7GX0f9r2LPqvgZi4vxz/DvbZ/H/etNnqH4s1rJc4XiunPjWFrFK53qUFye6nQRj84VD/dSrezcutX7Wk9o41hK3TWH+saar3xDbni5DTHncwQ4yV/ZAAAAAIxjywYAAABgHFs2AAAAAOPYsgEAAAAYx5YNAAAAwDi2bAAAAADGEfINcK0/ELT5e1JRm7ZkXZ+dsRnH211zVR+mbCUfhw6dzp1lpPGc2138m1YMk66SqlPnsjhlWveKO01SUngdK97K7Q6J4M0DD2evvC6huLwEMdb9fHFYRoiorzUj6lOTanDHR1H/BfLD/IHP6/l++T0G/wF/ZQMAAAAwji0bAAAAgHFs2QAAAACMY8sGAAAAYBxbNgAAAADj2LIBAAAAGMeWDQAAAMA4j+9eAMAvcXz3An6fOad0yEpay9hSfFT/IRbXg7dQXIxvKX51xl9Hpzh2Lv4B7Bn+VawuPurissnzuJfFn2HGz1XUpxnr4tC5bBKLq/F8/tPZK69LXXz0Ziw718p7L90e4UZNxWnGr3dOejOelzs3ln3lu7d19kYb8gkFXMRf2QAAAACMY8sGAAAAYBxbNgAAAADj2LIBAAAAGMeWDQAAAMA4tmwAAAAAxhHyDcAX/Z5I1L6zx94MYd1yqstg7C0zpuLGjCG0uFYGMzczlXtNyvEcxd0q7gRjV01SmHfM7T7dpOyQmvTOUicRfK31qpr3Yt23pM53nufWrb7l2T9CkHmjwx942wvGBn4Kf2UDAAAAMI4tGwAAAIBxbNkAAAAAjGPLBgAAAGAcWzYAAAAA49iyAQAAABjHlg0AAADAOI/vXgDAD3N89wLe4vbm+Yac1ePtB16e6tbZ2LLmI0xZNk/LK8fT8urORypudH51xmPxUfybVqv4Gf5V7Nkp/lz3L3ZIxan586iLP8vi3rG0Tmnd+RXuhOZFLO/ezg35DcW18LyE6k7n82/7LZ2v/Ah4/1sd4D9Tvwv9lQ0AAADAOLZsAAAAAMaxZQMAAAAwji0bAAAAgHFs2QAAAACMY8sGAAAAYBwh3wDRkOTp32TIKb10Gdc1b4Z/V4OdFO3YZEZMcivaeYU86RQyXSZVx6zrTjB2maL9GXK7yyZlh5XDv8832XLgoXhHUnu8qzu3TS91vnH3NrWa9B7ny4p7B/7mT4EhHzpb/KZjgT9jw4Prr2wAAAAAxrFlAwAAADCOLRsAAACAcWzZAAAAAIxjywYAAABgHFs2AAAAAOPYsgEAAAAY5/HdCwDgG92+ewHfZcOBHztmPN8kd2gdY6M4zViOx+KjmPEVllGOp+Kj1aRaxlrrVf2b1uuo/6HrWRU/U3E1/lz3svizLD5Ccfh3uM+qPi6vcyzlWcrF5flvdE7jrSueio/qTt1zq3ce2/y8hP/Q8Bfe9n/hGIF32vDyPc9f2QAAAACMY8sGAAAAYBxbNgAAAADj2LIBAAAAGMeWDQAAAMA4tmwAAAAAxrFlAwAAADDOI0WN3966DIDvlN6EnDHkrH7HMoqP0NYytqz5qLoc4eM9zRjqU5PqwI+6+NWYbr06nZ+pyVH8M9Uz/NvV87riavyz6hCLO8tITa478FevuHHF11qv6qKn4vqG3FHcesbL8dw5eetbJRc3lv3+d++QDx1gtp/3qvBXNgAAAADj2LIBAAAAGMeWDQAAAMA4tmwAAAAAxrFlAwAAADCOLRsAAACAcR7pP5ThV5K/gZ/u5yX7bfNnX+EbDvztt00jirufFtxocl1Mcjke05o7YdJ5vJxxRzB2WRwitz/X/eudP49G8bPqvEKCeEwKP33grdzuVpj3at9j1WB4mMPD1Sje86oJB95r8vvf9r/+AP/ydxXo+v2Pi7+yAQAAABjHlg0AAADAOLZsAAAAAMaxZQMAAAAwji0bAAAAgHFs2QAAAACMY8sGAAAAYJxHq7oVen5rtQZgj3e/fVsfDdc5vuFjp56xdUKay66Kw3xl5+7FKutTk6P6D+kAX/XyGsXl4FrrddTjz+qfqZ5H/W9Xz6r5Zyj+rDqXg2nGWFx27iwjNckHXoy/QnF5CZ7h/L/KzvEh2nAnHNVK8hPXKG49XHWT5qMYyje89M53fv9HwJAPHa5RXl4/K38xD/T/zF/ZAAAAAIxjywYAAABgHFs2AAAAAOPYsgEAAAAYx5YNAAAAwDi2bAAAAADG6YV8t6TALiltwBvIDNxuzim9biXNfO53F4cU7UaTVmhxGk8p2r0A5tOdU5h0GVOdxssw7xUysGPnsvi4l8VlFHcsPr2MVF9Gbqcm6SyV4d+pcx3r3rnia8dd3X0EOlrB2I0DzE3+Ar8b4Mf5Gy+nc1qvNn9lAwAAADCOLRsAAACAcWzZAAAAAIxjywYAAABgHFs2AAAAAOPYsgEAAAAYx5YNAAAAwDiP909ZBrW3kskB/nflW+Vv+LPvzrMH3r9nzp/qukNYSWu61LkYTweex8smrRnr4tdRjL9ScfUvTOXgv8arGZ+h+LnuxeCRiovxz9S5ahKLq/HnUawtFacZU3F5CfKpblzE8k5oFafxI9y+5XDrEcjL6HTe8xF13Uuv0fn9n7a//vP91x/gJuk8/dlvQddxS/7nrrsd/ZUNAAAAwDi2bAAAAADGsWUDAAAAMI4tGwAAAIBxbNkAAAAAjGPLBgAAAGCcbwj5LrXyxOS5wd/0h4MH3/3aG3KqU8LulVpR3LXzy86hxY3c7q93+Nd41T3mdndSnFu53c9WPndVnOpbud1pxs8qdTsWn44Pb4V5xyads1Tmo69wcVux7u0b8rLU+WYieOsZP/vY/gf1P5Bv9/wH/uwvyD/wShjs/XeSv7IBAAAAGMeWDQAAAMA4tmwAAAAAxrFlAwAAADCOLRsAAACAcWzZAAAAAIxjywYAAABgnMd3L+A/kZLo35+RDlwkPebsNec8X7eSIZ1T8XEUn12xuB6sP/3K8VZxGn+1iqsDTE1S51f1L0yp+Bn+Oep5FOOp+LMq/kydq/GyQ38Z968Xp/HyEpSndMXr0ikOV7x5L9XC89JrEpz9Fnnsedds+DIbFtLo/P6PhjkfRvxe7jKi4dsI/soGAAAAYBxbNgAAAADj2LIBAAAAGMeWDQAAAMA4tmwAAAAAxrFlAwAAADDOjwz5TlrRbcOjvOCPkLj43/zZl9PphN0d013ZpHWAqfh8InitFcWdMpVDmHSjcxmAvdZ69opDYHYdxV2kaKcmz1Bc5nk/VyhuJYLXy2icpdgkJoI38tTL8O94e5y+l1bzrm7m3H+1Q55yw1vluuJJ/uzHHPA+v+lF469sAAAAAMaxZQMAAAAwji0bAAAAgHFs2QAAAACMY8sGAAAAYBxbNgAAAADj2LIBAAAAGOfx3Qv4NkcY/00R7jBHeuL+sHe/bIZcgu9YRnGqu8s4Tl+vI0xZdk7LK8fT2urOR6M4jb9C8av6d6BycK31PMriunNZ/Ox0TvWtJp+puBr/vGwZz3T+w4z1dQl3QnkJ4hWvmqTicKtvuCE3vU4bTTa8EK79CGgdy7sN+TACfpw/+zvdX9kAAAAAjGPLBgAAAGAcWzYAAAAA49iyAQAAABjHlg0AAADAOLZsAAAAAMZ55LSsP5rB1zrsP5s0Bv/wR98dY8w5/2ElG16cQ44xRnFXyccX5nbvKH6F5dWRz1vyoavxlHVdxlS3wrzXWs/j/t8HU2735yqK04x1cS+3Ox1L4yy1mrSuS7rizRuyGmw+zOF5ScWnn8TO8i59L133Or3S8OUB389r4n+Xzoa/sgEAAAAYx5YNAAAAwDi2bAAAAADGsWUDAAAAMI4tGwAAAIBxbNkAAAAAjGPLBgAAAGCcR/5PrZT04+xCfqbysMXL80f80cf+P/Fn3wpnD7x5j9XTtZocvSatA0zFxXhaczmeil9H0fkVD7AebzUpx59H/Y9Dz+ofjXrF4Z+dUpPP1ozVeNkhFW858FdV/IrF6SJWTcKxlE1az0UubnVuNald99hueau8mY9s4D1GvPLGu3Xeyv7KBgAAAGAcWzYAAAAA49iyAQAAABjHlg0AAADAOLZsAAAAAMaxZQMAAAAwji0bAAAAgHEem/qU+euNsPHfpHXYguv5Ef7ow9z27gd6yHX5jmUUp7q1jC1rPkKXo7O8crzsEDsfqbjR+RX+CedV1ZeDa63XUTRpFT/DMj6r4s9UvO7l+LOesS4um5cdVlh2OpawjFZx45Su5kUsb6d4EU93zrd6S+PdG5/EDS+FDR8BeRU/7wvjkE8o4Lyf9wL6DuEsbXgX+isbAAAAgHFs2QAAAACMY8sGAAAAYBxbNgAAAADj2LIBAAAAGMeWDQAAAMA4u0K+SykOTOrf/00iOKN4OL/mj4Z5Z5dG216kFy2cgrS/3qSX231ZcY7cboR/p0TwMnw6xle3grFPR26vkNtdxoenJik+vJVBXnZuhqw3itN4jns/fUOGpyg8F6m49Qj0mgSNzmG6KYa8ToEJPJ9nXJfb3eKvbAAAAADGsWUDAAAAMI4tGwAAAIBxbNkAAAAAjGPLBgAAAGAcWzYAAAAA49iyAQAAABjnkVLFr4xwL3u/O978J+qeoysvIj+GR4tdjh0vleYNWc/YalIuO3doHGPrhKQZy/FYfBQzvsIyyvFWcWxSLWOt9ar+Heh51P849GwVV+PPdS+LP2OTor5cRmqSinvHcro4n//0vDQuYnherire9FWl8yTu+VA8u+wtr6AhfM2AN/t5r4nxbjPeZOnK+isbAAAAgHFs2QAAAACMY8sGAAAAYBxbNgAAAADj2LIBAAAAGMeWDQAAAMA4j/Qfypyrtyd/JyNSuOZrnSZxcb+AB+MC734yhlzE71hGK4q7sGXNKQC4lRTeCjOuO8cU569PV0c+585hvJV1XRWXyd+puJWiHcO89zSpEsFbUdy9s9RKaq8754t4Pre71grM7uXZx+U1moT/0Ojca/x2Q17UwHkere3yKR3xCm9dcX9lAwAAADCOLRsAAACAcWzZAAAAAIxjywYAAABgHFs2AAAAAOPYsgEAAAAYx5YNAAAAwDiPVnUKMX97knyacETK+g91/ty9/Tb4Vdy73+3d9++QK56XseGEvP0Y6zWXyzj2HGCasRi/rviVio/iX2Ve4Z9q0vizav6MxcX4Z6u4WvNa63Pdv76Mz6MoTiuJx1KtZEtxeb1icX0R6yueLmLrtinHt9y95ROa3hJbXk07HvPr3oS+NLGWnzS/gof5bcKpnvK47LgT6mPxVzYAAAAA49iyAQAAABjHlg0AAADAOLZsAAAAAMaxZQMAAAAwji0bAAAAgHF6Id9JmUb1HYFn5ZxTcr9+vetO9PDwPHfYT/NH87yD94d5N6K484yNtOB6xlCdw4wbyuJXKq6q0zJaAcwhTDoUH43c6DJ5eoWk6jIAe+2I3G7Fh6cmzyo+PDVJx1KepWfvlDYuYo51b902tfrhCp1bT+KWrOuySfkQtTpsKb7UkJUMWQbsMvwnxl/wZ3O7W9P5KxsAAACAcWzZAAAAAIxjywYAAABgHFs2AAAAAOPYsgEAAAAYx5YNAAAAwDi2bAAAAADGeVzXuhWnviPzfFfvKTnw/P+5JPxHrnyv/EDHN5yQYsbW47zl2T+qLulspBnL+tykKj7q4ldVXA7G4lbno/6nmmf4J5xnVX9h8XEviz+rJqm47JyafHZOSDyW+iK2ihsXMd9Ltd7dWz8vDb3nIrZpva+2vNzONtl0LEP8xDX/Kq0L8Ge/J7tNf5Db7Pu0eS81jmXLXeqvbAAAAADGsWUDAAAAMI4tGwAAAIBxbNkAAAAAjGPLBgAAAGAcWzYAAAAA41wY8t2SkrK+I7ytnHN0LBn8eSNyHoe8Jq5bxnccYCOKe0uKeTOf+KriFPl8VInUOUy6LE750I1E6nIwzZgit8viMoc7NUnFrSbNwPJw4PV1Cae6cxFbGfBD7t4t0tumzCDvNjlfDKOM+A7E35NvvBEv1E3PRXEsWzqHJvWp81c2AAAAAOPYsgEAAAAYx5YNAAAAwDi2bAAAAADGsWUDAAAAMI4tGwAAAIBxbNkAAAAAjPNIsembkszPKpf3HWtrzTkiix5+oxFvpvFP+Iaz1DzGesZWk6NqkjtUM4bqZufUpPBKxVV1uYy11qteXqP4mYqPsrj+p5rnEcar+udx/3rxZ5jxs2qSlve5quLOmtP4szpLK5zVV5ixvC6t4nTFj2p5sbjzJJZ3aZJq686tZbTfV+cf5/e/IUd0bhmyDODN8vtxxFth08+AxrGcn/G2Yzp/ZQMAAAAwji0bAAAAgHFs2QAAAACMY8sGAAAAYBxbNgAAAADj2LIBAAAAGMeWDQAAAMA4j5T/fVQR4puy0M9K4eZDlpcXMiLQHn6IEQ/08If2ePdZSp8XDVvWfFRTps7l8nJxY/w46uJXVVwOps6vVudQ/Kz+VaYcXGs9w/KeR1H/GZp8VsVlh7SSskMqjsfSafKKJ6QxY9mkd8VDceteas3YfAS2vGoaTcpnfIsr3+ojPrb6fuiygS/JT/iIL7lXvoDqAzw/45ZTGprUHfyVDQAAAMA4tmwAAAAAxrFlAwAAADCOLRsAAACAcWzZAAAAAIxjywYAAABgnEf+T2Uk5FVZWVu0ksq+Y82tOUfkrsH1Rrw/hj9v14V5zznwOoo7xFc3O59PON7QJAYzV4nUreTpVFyOPzuJ4Gm8l9u97mVxnQgeissZW8uITdLZq69Luoinc7vDdQk3Xi3ekFXzfPd+dTA1ycUtjeV1m3zddW/IOe9e4Mdp5UMPsemLbHGMl/6QuG3I7Q7Ft0Znf2UDAAAAMI4tGwAAAIBxbNkAAAAAjGPLBgAAAGAcWzYAAAAA49iyAQAAABjHlg0AAADAOI9mfR03XqaKt2LM3y8t7tJo945yIaNPKfzTlGdr+FN03fKanYvr1V3b0WvSuENancvxVPxKTar/oVzGWutVL69RXA6utV5HMf4M//ry7BWH8XX/epPPqkmcsRovO6y1Psvi9rFUy4vFnetSdW5dxHR7hFs9FIfbuvXknn+4es/ylW/kK3tP+UQDfofwThn9pXXTe7BxjFtmPH+q8zKKJrdQ3VqGv7IBAAAAGMeWDQAAAMA4tmwAAAAAxrFlAwAAADCOLRsAAACAcWzZAAAAAIzzuC7rOoVQ/sTw7zFZjt2FjD7V/HxjnozKj737z57V82He3Sbpbd/SStEOud1pGa2Y5A1R3GXzMto5NSnzuVeKqQ7LaGZaN3K7Y7p2mQjeye1uZZC34sNXN4q7LA7Hcv72yCnaZxPBuzMGndzuHZ2vW15y3UfGj/0wAk75w7ndpfrAz8/YitzuN6mKb63OGw7cX9kAAAAAjGPLBgAAAGAcWzYAAAAA49iyAQAAABjHlg0AAADAOLZsAAAAAMaxZQMAAAAwziP9hzJAfEtU+1G1SYnlQ7QWd2WgfVdrLaMvAd9t0n393/zEe7d8E16smLF16rrnuaw/jvrA6+LYuTyWuvOrU9waLzuvtV5H8c8hr/BvJM9qPBWX4+V0qXM5uNb6PO6NJmHGz2r8Mx14NWNaXn2WOge+1npW91481XVx5x6Lt3qjc3iIytooPFyN5eUJG++x7rIbna9qvIZ/9m3xEz9AYZThP2OTHW+3+sCve2+2TnVzGeFYqi65c9EkFZfHkor9lQ0AAADAOLZsAAAAAMaxZQMAAAAwji0bAAAAgHFs2QAAAACMY8sGAAAAYJxHDpMqcqdSrNb5KK+cd9uIvxriurN0MYngf834W/K/+aG33XV53m8/IY0c4n/Ud4q3JIJXg6G6l+LcKc6J4MV4K6Y6F5ch340w79Tks9MkJYJvKO7kc6+Q5/3ccRF7GfCnU+e7EfXXve3rxyi+EVrx4Y0mLT/0c2SHn/ehD9/lJ+Z2b3rCr0vRbjXZsoxGbndu3tqLEPINAAAA8GfYsgEAAAAYx5YNAAAAwDi2bAAAAADGsWUDAAAAMI4tGwAAAIBxbNkAAAAAjPNIoee3Ohe8Li9HN4XDF22OlLK+Z8arNPLlxx/LjgW2zgdfNP7G+bKfeH8cl53/5tmol9FqsuVYjmrK1LlcXlpz2SR2PorxV1xGPV7Wv6rOqfgZip/Vv508wzLK4s+j/teXunMo/gz/hFOOpyb18ta9Lq6alB3axWE8XMRGcbw9qovbvNUbd29+LhrjreW1PlyuexOuaz8azi77J35swe+Tn+Sf94xuepk2Dvy61/ftsmXcbr0rG5o3thfSsTSLi/F04P7KBgAAAGAcWzYAAAAA49iyAQAAABjHlg0AAADAOLZsAAAAAMaxZQMAAAAwziP9h5DbnZKnivIcH35eI5mylSg2x+9KBC9tWfWPvLjBz7yMp/3ES3jpms/neV8X5h0jhEN89YW53ZcVx/DvKgr61QmZTsXleEqeDongIZ/7KNK1U5h3zO3uNCnjxuPyOvHh4Sylixia1Bcx3TaNGTfckOHBaEZuX/U5kp/9DU2CP/qZCPx3cru/ppE8vcWWX9m9yO3OwbSiuKeEfIfF+SsbAAAAgHFs2QAAAACMY8sGAAAAYBxbNgAAAADj2LIBAAAAGMeWDQAAAMA4tmwAAAAAxnm0qlP2+q2OIa/Ly9HrEuOP2Pvd2fXXydfl1/sDh/hbpLt0uLDsDTde84TUM7aanC8+ju6Bl/WNY0lrLsdfYXmvasZysDv+TMXVSp7h30ieveJiPBZX42WHtdbnuofxRpNn1SQXt46lcZZaFzF9SWgVh7s3Fbeei3IZtTzeatJ6zK/6FN7ykfFDP3fgL8jvjp/34F75a6RxNrYsY8t16X2K3FqdNyyjbNIsbmwjpAMsm3yEzv7KBgAAAGAcWzYAAAAA49iyAQAAABjHlg0AAADAOLZsAAAAAMaxZQMAAAAwziOlObYytEJudyNdu5XntilKLQVqNnK/hvuOs8of8vNiGLN8LL/pyWgE/bYOvJl83GgSO1fB2K1M5Vf4F4sUZF6GTL/2xFcX45+tzkcjn7scXLuiuDsZ5OXZy7npRXEM894R917eCc348NpR/YfW67T1xG1RrjkW93q//x27Ycbf9PEHu/ym3O5kxwvr3bndufl1K6l+TYcWYbiTot28wa4L+f6o8rxbnYV8AwAAAPwYtmwAAAAAxrFlAwAAADCOLRsAAACAcWzZAAAAAIxjywYAAABgHFs2AAAAAOM80n846hDxRkZ6Ski/9cpbnbco2uQZeznwk7WOZNOp5sf4PTd6PJYLb+rm2Wu8gsJ0G47lqKZMndPyyvrcpBh/9YprZZPXUXd+hn/JeFXjaXll8bMzY1rG53EvBkPx8yiXUXT454zF+OdqNCmXsdZ61telUxzPf2P8CNelvJ1ad++m4pZGk+aL4qo35JYPl9/0CfV+Z7+U8yv0n/Bfco9serU1zsZ1Xze3/CzNyyua3zoHk5bX/CiqltHsHJqE4ltjxo/O8prFAAAAAAxjywYAAABgHFs2AAAAAOPYsgEAAAAYx5YNAAAAwDi2bAAAAADGsWUDAAAAMM6jFeB+C5nlRyfevJwxde70SJ0vdNTtN8TOD9e6bUq/6Wz8UOcv4nD5AK+6+5qnNL1OWzM2jqXsfByNZaS1pWWU473isLxX3bn+R4hX1aTs8K/xqskzzPg8ivFm8b0urpqUHVKTz7CMz9ikNWN5LPUpLYvzdamKwzLK4tQ8zRju3lrr7g0dOp2bTVovvaPzDtry0uNd0uVyXX6nfF1/+RfAi2/oxm/e87ZcxNbybrctnRtn6dY7pWc7p/GPcOBl89T5o+y8XnXnasayw/JXNgAAAAAD2bIBAAAAGMeWDQAAAMA4tmwAAAAAxrFlAwAAADCOLRsAAACAcR4phLLMncrxgGUIZSMra0vWeNnmOyINN0Sh/3pbAgadPf6XcEKGhHmvciXNMO8Nxa3I4Rhr2JqxGqyjDkO0cEoEb6U11/nQMRF8S9Z1lQi+o3MZ0d3K7Y5Z4zGDvEgKz8XldWmc6tR5S1J7+Qhsiagv7/VWbnc3hrVTPCWK+zs+uf7s14SzGt+zuUbz3v3lV+bKJ7l36s6v5P253anzrdPlzVHczeLeAdYh361E8FbId4gPL8O/hXwDAAAA/Bi2bAAAAADGsWUDAAAAMI4tGwAAAIBxbNkAAAAAjGPLBgAAAGAcWzYAAAAA4zzSfziOIsr8FnLFy9FbiFQPxY0s+lQaZmysOenk1vfa5GU0Mur/rNZF5BfIV/yqh6N5jzVees0ZW50bZ+PodE4HkptUL73qw2Wt9aqKy8HU+bWj8zOOF//CUQ52iz+PezGYOh9V56pDXEbV4R/LC006ZynM2Lou52+PNN66q5udr3q4trwJj86LacubcIs3f+7/ja8Z4Yv5l0f/xlkqbLrR/+j5u/JHTeOUXvrbqvXzNnRIqt+JzYNpLa/snTq0ilu/eT+qvYi8jHr8o56xUVwuIxbHzq9OZwAAAACGsWUDAAAAMI4tGwAAAIBxbNkAAAAAjGPLBgAAAGAcWzYAAAAA4zxSCGWZdJXCWctQsaOT+9VaRtLKGm8lwOVY8euUuZ6NUwo/WnonXDZdy4Yw71bzVqZvyu59c8LxliYx8rn6MGrlQ+fk6TBepmvHKO4idfu5NkRxf1ZNUiL4Z9WkFR++Qup5PEvVeCtPvTzPqUm6l7bEvddR3PHharnq5fYzl8FMjW++f+NbqPv6f/Y3crtLVy3vFiKfm523RHE3ipuJ4I3A8lY+95W53UU+d+5cF997nQEAAAAYxpYNAAAAwDi2bAAAAADGsWUDAAAAMI4tGwAAAIBxbNkAAAAAjGPLBgAAAGCcR85eL/PQQ3k1fAuJ6kcdqJ4aF/+llRifSm//SK//cptytNW3KZ3SpHGq4VLNV82FGq+PtcrlNTukZXQf56q4qk4dyvG8jMaBpyavoxh/peJ6xlRc/HtDObjWeh5lcd25LF5rPavmqfizKv5sFXeOpVxbGm+tuT1jfcXr4lfnupR3QnmDreaN2nwEtjwvX+2Qm/Rem+WLIha3Wk/5TjFkGX/Blg9AfowrH63GvXTdMnLn65ZXdw6/m9OMG37itZqE4nAs5eAtFTc6f5wuXmt9VCuJxet1svO96pDGP251sb+yAQAAABjHlg0AAADAOLZsAAAAAMaxZQMAAAAwji0bAAAAgHFs2QAAAACM80i5kiEfupHbHaMpqxCzukNMGm+t+culWU4EL9o0O1+qlU8sEZw9tqTSnrclqvbSPO9Gh06Y8XU5xDGAOXRpxSTXud3hwMuVPDudW8HYK2Rg56zr08XH/evL6wWTdyK3uzOG8PUNKdqtDPhWYHYsPv3wn3/wc+eun/dhLkoaLvLrc7vzjBveK2HZV+V2t5bRPcALQ77rFO1G55SiXReH+PAynzs1z7ndRZN7JxH8HnK7y5DvlAjur2wAAAAAxrFlAwAAADCOLRsAAACAcWzZAAAAAIxjywYAAABgHFs2AAAAAOPYsgEAAAAY53HE/PVSykgvmsRo+KP6LzG5vpVR31hGK7s+NymXvaXz+3UuYvVfvmPNvEm4E6Zc88Yjt1bzVm8to3FC0ozH0TqrZXFrGXVxOZ7W9gpNyvFYXDV/hn9XKMdfRyquOsfiMF7Vf4bicvwzzPi57o3OnWWUa26d0rXWqzzVrSveuW1S5/Lea929cTw8ivUjEDs3hM5J53HurGPTSw/4fld+G9vwlF+3vNavudChWX9r/TItpV/TZ4vT2UjHWNbH4urA04wfdecdxdUyyg5rrY/1+vqM91unOHQum6RlPKrxvAwAAAAAhrFlAwAAADCOLRsAAACAcWzZAAAAAIxjywYAAABgHFs2AAAAAOM8tuRKBo0A61uIpiyzMFO+Wjmaw8Pr/3I+/Dskf7d69DLrJILzHxie2106H+bdb1J26J2lcsZWmHcrWjjGh/dCizdkKl8X+dxLBO9kXaco7rI+JoXXMxZh3t3l9ZZxunit9axOdZn8vULOeiw+HSS/JeS79Tg3H64Ntrz0dhj90QC/z9sfud7L5u3vmvPR2kkIxu506aZrf71J7tAIxm6FfJcp2qnJlbndjcjtFPK9Jbf7o2oSQ747ud1lcZn8vfyVDQAAAMBAtmwAAAAAxrFlAwAAADCOLRsAAACAcWzZAAAAAIxjywYAAABgHFs2AAAAAOM8jhjWXiacN4pvofjodC4j3I8jFFfDYbp/HEn5XxpN6tIgnaVWm+aM71fM2Vpz6/z/Bc2zN/o8NY8lOX+P1fIbsize0qRxLKG4nu7VKq5es2WHf4yXzV/hnwrK8ddRFz+r4nIwFnc6r7U+q/rPVLzuX+yQmqTi3oFXTeL5Dx+s9XXp3AnN26NVXIvPS/Ufmo9t63WajqVRXHfY9N5szDio8+hPNFiD7tHG43XpmkPz65YXfjV0uuTa8jdvatI6xrOd03TpwMv6j/ib63TxrS4um3yET/iy+H5rFK+17lXz1KQuDssrm6TiR6fYX9kAAAAAjGPLBgAAAGAcWzYAAAAA49iyAQAAABjHlg0AAADAOLZsAAAAAMaxZQMAAAAwzuN11FntH/Vwipcvq99eXA3fQhj9EZPkG8Nlk9Chlg97S5vWjJP1bscdrus9+0xfZtMJvepOODrXJU8XmlT/Q5qxbJ6Li/G0vHI8fQSEznXxK+z+l81fsUkx/gzFz2rGZziWsvgzrPl5hPF6xnujuDNjKv6sZoydW8tIF7G6BOm2qYvTFa+aNG/1RvFaq3xCm09ibccr6ELN5n/0Q6qUzsWl14uBvuOpOHuXXbrmxtedPZ1D8a0xY2vNeRmNn36364qrAy87/GO8/NTPxa8vLmOt9VE1KQdT53vsXBWnzreieK11r5uE4qrJo1OcOj/W84sdlr+yAQAAABjIlg0AAADAOLZsAAAAAMaxZQMAAAAwji0bAAAAgHFs2QAAAACM84ihrXVgdt2lHG5lmLWCFFMAdp3bHUOtN0Q3livJKdobori/voxu7+aMv94fOMTTLs3tvm7GHXne4RUUqptR3GdziFOm8lFnKtfOpzWvK7OuQ3GI3G6laIcmZbp2Sgr/rGaMseLVjDk+vIpC7wWTN/K5V0hwbyW1XxdRH4vjk9jSi5rtTNd5BXXzZ4H/atLTMjq3O8z47tzu+qdfp0XOur6uSaO4FcWdDrwMzO6HfDeiuMs87zJyOzVpRXGnYOy6cyfMO9XH4mr8EWcscrtzcSM+3F/ZAAAAAIxjywYAAABgHFs2AAAAAOPYsgEAAAAYx5YNAAAAwDi2bAAAAADGsWUDAAAAMM7jFQLm672cOk99rSqqPUbX12LrrxffquK85Pq/HFWTMos+NU+HXXbOx9KQjjF0zgu8ZsawDCY6f2Wz+ppfN2N44lJxUr1VQnWasSxvFjdmPI66c/m2f4WXfV18NIpzk7r4Wa3kGWZsFX+ue1VcDKbOm5ZXz/hZFqfrUs2Yi1tXfEOT1g1ZF8fnohxvFacZa60nseXKl96UJr9e4/sc58z+urjhmr//AG/fsOxixlunRWvNqXGzSeMHWq84/Kgsm6TOH6eL11of1UpicTX+sV6dznXxvep8v6XiYrwcbDcJxY96xmddXDVJy3vciiap2F/ZAAAAAIxjywYAAABgHFs2AAAAAOPYsgEAAAAYx5YNAAAAwDi2bAAAAADGeaQszDJgKm7wnM/4vlArPrwOSEsZpWVA3dHL7W4E/W6J5cun47pg8Qv+/7XWnFtsjLcnjG5J0W5oRXE3m4Qnsapu5XOn+mZxeFFXr6ZmwnGjcw7z3pBIXQdjp+JW56q4jNZea33GpPCyScjtrvK8m8fSiEJP5z8UN654qm/eNrVW51Z8eFOrSevZ73SOb7frPuh8hG6WTuifDf/+mXfYD83tLm0Ixs7O53Z3Ou8obuVzp7PXalJmXbcyyFPkdl0c4sNzbnfx+ZwTwRvFZVJ1LO4EY7eKy3zu3GRHbnc1XoZ5pyYpa9xf2QAAAACMY8sGAAAAYBxbNgAAAADj2LIBAAAAGMeWDQAAAMA4tmwAAAAAxrFlAwAAADDO4xXi4VNo/NeLP44QdH9r9S6bpA7ni1N9KK4n7M749UX0Op8/0cmt1/v8hJe14P/UuLKXnv98t5fFpzvHR7woTtOlNZf1ubgYTy/qVnE5nouLDf1nKH4eYbxuUv9TwfPoFFfjn6G4HC+n++fy7iebXFm85YpvaBKel9ZzUSu/UGx6EntNvq75XuLXOv+FiXM2PF7vv4hhxncfy+22ZcaiSfOXZuuXTqM4dw7jVZdU/FEf+I7i6rqUHf45/vpi57XWvSyuBtda96pJ2SGN328bih/r+fUmj86Mj9tlncOa/ZUNAAAAwDi2bAAAAADGsWUDAAAAMI4tGwAAAIBxbNkAAAAAjGPLBgAAAGCcxyuEg35Uw3VEVdj4ScVldlg5XbYjn7sTmJ0yreuM75BuXiaGpkDwo5cI3lje+xPBw3RbAhPlov4f3ppKu6nzZWHe4eUWOm/J5240b4V8p2M5n9udPgJC53qXP42XoeBbsq57ud3V+OcqQrvXWp9VmHeaMcWKhxk3pJu/quLWdSk7rG5ud/x0OX2rxxdCWZyWkcave0Oe/0S76k14aZPvUH/zevcqeJ+zt+qcm2NIbnf9S6fZIoVPh+LzHbbkdleDIdO6Ff6dTt6O3O7693TZOYZ5x5DpxowhXbsRH97Kuo6J4FW69iMWp/Dv8ljqdO26OMxYHuM9ZY1XxelY/JUNAAAAwDi2bAAAAADGsWUDAAAAMI4tGwAAAIBxbNkAAAAAjGPLBgAAAGAcWzYAAAAA4zxeadfmqJLkQ7x5qbUb9Kpj3eug+1s5morjnGHK+v9IxV/vsG5Vk+o0/6/qoknZYTWPJA2HJvVwPqtf1TqhyW3DQn6VLWf1vGPHdem8EvJzVHcumqQG54tzk7r4dRTjr1Rcjae3etn5GYrL8efRKE71reLPdQ/FxXgqLsc/Lz2W8uzF4sZFDMUb7qV49/aadDrX05W1vRdCy5Xvq1A85E0N3+bsMzDky1//J8aW5l+dMf1cak7X+iHQKE4zhh87rc6huKr+iJ0b460mH7e6uGzyseqf3mXxPfxOT8u7V80/QpN7OWNYXrmSVPy4PRvF1Xg68McqOqf6snMuDp07y2sV+ysbAAAAgHFs2QAAAACMY8sGAAAAYBxbNgAAAADj2LIBAAAAGMeWDQAAAMA4jxTVWbqFVNRVx5W10tE6Yqh1+R/eHwW4IT68PJSURZpi/I5OyuDkRPBEUurbnE/Y7V+sKhh7RwDwhhzibkxyGcxcZSen4i2dy9zoVj50KwA71afiz9PFrdzudCypSWt5ZfNXKC6vy3PLRWyly+9oEu7e5KrOV+Z2b/lAu/TNeXbGn6n1vYszrvrmNeRS/dnc7u75L3/NbZkxdG7kdl+Xz53qb73c7kZxK7c7JYL3QqZTInid293Iun7syLou07VbnVdMCr/sWHZEofsrGwAAAIBxbNkAAAAAjGPLBgAAAGAcWzYAAAAA49iyAQAAABjHlg0AAADAOLZsAAAAAMZ5PMOuzVGlgscNnjLB/VbXhuFG8ZZlZK0u1xV/vUMcLpvfwoRH7vKlvnm40TeUt68h/5t8/q86r60rfhyNZbSOJRUfG4rTrV6Pv6pjjMXVeDm41nodxeswFT+r8WfVYa1VfjSkz4s8fv/vg5+puFpJKv6sitMyyuLPam0rrDktLx54XRyuS30R68717REuYuteSk/i+UcgFnee/aDXIa2k06FT3Pzw4wJ7vhf+QG+9+Yaf0Ly8s2epf+DVV9wdp+92+jt8q0OqT4eyofhWL691LB/1MkJxnLEY/6h+NacZ73HG8qd3XXyvZrxXHf6xvLLJIzRpzVh3Tsu4PTvFZzun+k3HUl7E1ilNdy8AAAAAw9iyAQAAABjHlg0AAADAOLZsAAAAAMaxZQMAAAAwji0bAAAAgHEeZfrsWmvdyt2clABXhVeF2o9qwjr8KmwppeI6ETwt+cJMwsaUOQ+8DKLrpWiXQXQ5dvTNieCN8n4M43XLu0rzGN+96vPL6+bdtqK4v96h27kVWpxep2V9zO3upDiXudExH7oVU12GfIdlNKO46xTtMnX782hEcafisnP3WM6nnqfiZqx7J7e7fhJ7EfXlo5Gfl2owVIfOG5bRkpsM+XD45fK3oOtIX/8vfuKNPim3u2wSltHpfj63O52NEIy9ZRmNYOz0EyOkaHc6d0K+U5h3mrFukkK+q+Y58rlcXicfupM8nepb8dVpxjIp/L5C5Harc9UkHeCWY+kV98LXywz4VAwAAADAMLZsAAAAAMaxZQMAAAAwji0bAAAAgHFs2QAAAACMY8sGAAAAYBxbNgAAAADjPJ5h1+Y4qjj6kCsetn7qQPvjKJrEraPbF8dik7TiW726uvktVN/iWr4urKPunIqTRpO6NE5YlN/CyTjay/7CZPv+j7OLG2/TARanNN8erWXUF6vzgNbFx57iYvx1NIrXWq+ySSo+ijfZM3auiqsOa63ybZ8+AsomreK11mdV/9kpTjOWTcoOaXmt4rSSXFzeNnVxeRFzcfUkbrnxOo9A63n5T17h55qk5bX03pyNz9Ud0/0B6dw5Uf+7LY/WEPlY3v8dslQsI33vba7k9OtjrVu5vFjeKC47p/r8c6mcsS4uP/zyMorxj7CMj+pH4UdcRhivfgun4ntnxnvVueyQxssO/2pS1T96xc+vz/iIyyua5OLGgcdjOX32ynspNbnHe6zqHO9eAAAAAIaxZQMAAAAwji0bAAAAgHFs2QAAAACMY8sGAAAAYBxbNgAAAADj2LIBAAAAGOfxjLs2Vd74URcfVfFxq/t+1DPWIeTrKMZvITh9rWLKW8xC7wirqybsdTlCi7Ds3nxh3a0m9ZGXLaprFctvnVUc8QK8W/cC/HffcST1quuVdNaXa4sZuwdePhpbZiw7v9JZql5k6bFNTcrxV3hF1sXhjfWsmjzDMsq3/TO81T+r4s9YfA/LK8bLzmklqbgcbx1LKk6fiWV9Kn5Vxb3bY0dx6+5tjcfi6qlrPYm5uNE5CfXn3+tjPqL+gA3fYN5uwx32M+UDP3tlNp3S8ifGlmU0vjzn5o2zVBan6c4Xr7Vut/Mzph9oneJqGWWHtdZH9XM1Fled11r3ukn9y/ReNSk7pCaP8Ju3bHLvFK+1HnWTZ6f4ss7xlJ4tTvVbZixvp3R7NO9eAAAAAIaxZQMAAAAwji0bAAAAgHFs2QAAAACMY8sGAAAAYBxbNgAAAADjPFLQaRnU1staTpnPMaL7q25xzY3Audj8y4NhwhgQ2Orc1AghTzOGi7thgWUyX04EDz02DCeNpVyZGNpYdfPsNbRStJtNOlnj7QDgTvHp3O52yHcn8rmO4u4Vb4ipLvO5W51XTNeuE8HLpPDPUBziw1Px2bOU6lNSe+u6lE1ibnfVpLyl144U7TgeqnuJ4HWPZgzuaUNyoLnCn03XPu+6fO7+jC3vzu1uNbkut7s1Xd05fqPeEMVdjqcw4zI+PCcfd4rrtOZGIvhKId+dRPAtydNl+HcurtO1y/oycnuFZT9W6NxbXidye0e6ebiIqfh8bncrEVzINwAAAMAPYcsGAAAAYBxbNgAAAADj2LIBAAAAGMeWDQAAAMA4tmwAAAAAxrFlAwAAADDO45l2bapQ8CPEzocGdQj5cRQzps5lk7jPVPX4uNW19eLWKst7O1uNkxTmu1ZaX7GUtLrjwoMsOsf/P6yitbhvuQYXCQe+4QBbncvifFFaTVJxMX7E26MofoXO5Xhaxqt6ua21nvWMjeJn7FyMp+LPqvizV3wPxWG8al52TsXpE6pskoqf1fJap3SF65WLqyt+pHusaNK6Ibfcvb2H67LXZnpRtN7q3bdNo3Ns/Xs+R/it8j3a/NK0YcaW6mvhjta3zoGHCesOaXXnm6Q118Xht1XZpNV5rfXRafJRrSQW1z/9UnExfr/VP/Lq4vCL8CM0uXealCtpFT9S8Xp+scO/mtQzFp37y6sOPC6vcSzl+S/vmX826dxj9UVs3JC3dI+VxxIeW39lAwAAADCOLRsAAACAcWzZAAAAAIxjywYAAABgHFs2AAAAAOPYsgEAAAAY55GCTstUtyPnWxbFrWTmI4Ru38rlNWJAj9A5bVaVoeApQjXk5NXqBL4UnlxV5wS+HMZdDKVj+WqHf5X3mpzvvMVV8ZZN7w7NDRqR23nGMhW4UdzqvEJ8cmvGXqZyeG3mJmU+dF1cvpNjfHVdXEduny9u5XOvEP7dSgpPxa1087q4E7Kemqek9gujuMvPzx353M0Za9c9+78pRXvIRw4/S/MBkNv9leLrOl+W252K6xTtWivkOyUfl/U5n7gxYx3AHDs30pp7kdt7mpRZ13WKdhmY3YqpzmHeG2YMxSERvCzuRaE3bo8c696Je99xj133XPgrGwAAAIBxbNkAAAAAjGPLBgAAAGAcWzYAAAAA49iyAQAAABjHlg0AAADAOLZsAAAAAMZ5PMOuzVGFgqfE8q93WGutqkmqPaouR0hfL5t8xG2pes5yxhSofqtn3KFxpsM6Qo9QW5cfobwz4T/n/KrW6egqlnd+xeviRQfFwlvLyMXnO9cnNY5X3VtNXjuKX0enODz9z6pJeveW488jFd//++BnKP7cUFwMpuIVlv0Zz1J14KG4XMmWzq9wQsqLnm+bokm6e+vO1T2TiuMH6JcH/+n867DR4br3Vbc5XKr1ZexSO77whC/JO1q3vnC2Jrx1TnVZ3L2IZX1aRnn2YnE1/tFZRiwOv4A+VvFbLDUpx9Nvq1bne1V8D78Ty8731Dk0KWd83J6N5VWDa61HNWMuLmZsdU71jxWOpbO8XnF9ezTWHG+PeCeUz0ujOD8XrSexfKukzgAAAAAMY8sGAAAAYBxbNgAAAADj2LIBAAAAGMeWDQAAAMA4tmwAAAAAxnmkVNR7mQF31PFXdW2as87tDrVl4FYIYa3nDPleSSsatJWAHbL9dhTvWN75RPAro7G3dB7h/Smz35Hb3SkOD3+vyfmQ75ipXOZDN6KdU30M+a4TqUO6didFuxxP+dy9znuSwqsZ4yltRaFX57/TOY23orhbN2TrVt9THB7+5pP41cFsyts+nJApyzsvHYko9P9d83pPOXlyu9+zjBy5HWasvq+3crtT5zqHuBO53QotXt2Q76p5ylQ+H/m8KXl6QxR3OeMjFhfp2rG4zhpPxY3c7taMrSj0FOsecrs33B691PlOzn0rt1vINwAAAMDvZMsGAAAAYBxbNgAAAADj2LIBAAAAGMeWDQAAAMA4tmwAAAAAxrFlAwAAADDO43nc6/9SJZzXQeFrHdV/uN/SpEV12WGtteomdfVRRbgfR70tdYRw+JCd3ghUTwdT5rqn9PXyuLsbbGXrcNzhTIeLWHduLSPeTfG+OS3NeK50mw0H3lr20ZmxddeUT0CaLo8XXp0mr6MuflVPUupcjr/CW+UZntFn1eQZmxTv5M9Q/FnN+Fl1WGt9Vm/7skNqkj4v8oxF83iWqua5uNP59BVf4XYq76XUpNW59by0HqL0aducMTn/Vnl38d/Q+NBufRQNOdWbvjcMOZrajmNM307P9s0NGqe01SQVd7601sWtr+WxOHzVbs7Y+NVQ/nj5iMuoijud11of1e/EVHyvfqDF4qpz2SGNlx22FK+1Hp0mj/XsFJ/unM7SZSckz1jeY43O6e6tb/XQOd+9Zx+u1mObllG+e/MyAAAAABjGlg0AAADAOLZsAAAAAMaxZQMAAAAwji0bAAAAgHFs2QAAAACM80gJr3Vud2pTxtalJNEqWCt1DssIId/VMo5UHJJOj066eXnuYnH1H2JOYSd8sxVJmJxPBA+Ryq27Y0Nx/j+uiw9/tx353Ks8ISkAuLWMVipwalImIsfiTkxy2TnlQ5edY3EnkTq9e1vFZbp2LxE85Xa3EsHjjGeXl6PQy9zuxinN+dwbcrvrRyC8Iuso7rK0WRwjZS/TDEk+v5Lf81b/Dq3LtSERfJPRUdylK8/S78/tDp03pGj3OnfieJtpwY2VtJKPb53k41bkdm7SiHxO+dBlk14wdifrOoZ5x6zrIl07Fxfj5ZrXWvdbI7e7nLEVhb66eeqXxbq3AuN7Id87cu5bb5Vy/BaX0ejsr2wAAAAAxrFlAwAAADCOLRsAAACAcWzZAAAAAIxjywYAAABgHFs2AAAAAOPYsgEAAAAY5/F51Ls2R5UVXgeFr3VU1cetjBtf96POTv/6MtJCjmr4CFnoK0S4r+qEHGnGqnnKdS+XHXPdy4Pp5LqvtW7Vf8lJ8oV0JHWSfKguh9OayyseizsLTE2GSEcStB6MNGPjlJTF+UYviztrXutVvlVCk1c1/gpb0mXxM7wJ6+LQOTUp61Px57pXg6H4OF1cDaYmqfhZrXmFY4zF5YzpVHdOabiI4V4KnzrhHms0yY/AZcXlZ2JZ2n5Cz749cnFD87UZmmzp8vs5Tf/Fld8owpe001PmBo2L21pF+sLZKj4/Y+xcf+FMyyg719IPgdbyPjo/Mcrxj/BLp9m5bnK/FeOpyb1qUnaIxWEZj6pJKr7fnkWHVLxjxkc1Y15ep/PpU7rCxc1Nzt9jaRmdG7LqHB+izpPYe2zDw997BdU90rEAAAAAMIwtGwAAAIBxbNkAAAAAjGPLBgAAAGAcWzYAAAAA49iyAQAAABjHlg0AAADAOI/nutf/5ajCwlMKeRm0XseK141D7bpX1UdMjK86H6F3DENvRLWvo1hJeYArHGPaMyt7pEP5CAdzq7Pr04zFf0lJ8kenuD6WtIzqP8TiMF5KTXpaB3OZ8vzn4qR6uDozNovrNb/CA1PWv1KTsvioH6+y+BmexWen82dqchQvpzTjZ9X8M7yoyxk/q+licVxG0SQXN05IPkvF+CsWlxcx3R5V587tsZp3devuPd85fTS0OmeNZ/9K3WXPnXHGx8jfdeWd1LiM8Qt1R+7RWUlvxlbnDcsom8Ti+ntv49tpLm50Tr8ayuWl4nI8Fted659LZZN7Ko6/uRpN7lWTLcWP9fxih7XWo2rSKl5r3S+bsTzGXnHnlK7unVCNlzde6ty6IVPn8qGLT1zr2e/MuOd91Sn2VzYAAAAA49iyAQAAABjHlg0AAADAOLZsAAAAAMaxZQMAAAAwji0bAAAAgHEeKZy1juLek9tdjYa0w6PMMIuZz2UWaZ0olo6lTK07Qu5XndsdWpfx4alzGVeWI7dDk2q4DPNezUTwukNzfEdxKr8q3vJ8EGu/wdmE3ZwW/NXpUnErt7sVWrxSbndnxpgP3cntLqO4YyJ4K+s6zlila8fOZfF1ieCN4hWOJZ2l+lTHKO5GIni4PRo3XpyxE1Hfy+0uS+N/2BAXnGbc8rY53xn+l7eHvXceuT+R213qvZha0bZl8xSRfj6Ku7XmVg5xmnFL8nHZpPxFs0Jud8yHDk16ud29+OoiRfvRybqO+dxVkzIm/J/Lq5pcFsXdOqWtRPbVzO0uZ+zd6jt+8+4J+a6e89aMrRdFIuQbAAAA4GezZQMAAAAwji0bAAAAgHFs2QAAAACMY8sGAAAAYBxbNgAAAADj2LIBAAAAGOfxTLs2Zax4yAo/juI/HKH6qMLQjyMlybeWUSy67hCWsdY6qtj5e9V5rVUd9zo6ue7lmlOTWyfQPjZJy6uGb+Fc10ny4ZSGGTudy9Jwk6b/Ix14bnKJ9Fw0myTlk9hYSbgfQ3F6xqvBVyyuX0Gv6ulKTcrx9HJ7HcX4M3QumzyrDmutz/C+Kes/w/I+j6JJOpayuNU5FlfH8hkO/BkOvKxPx1KepVhcjZf3zFrrVRZ37qUVHo38CLSexOs6N4rzi7bs3NB8xzbeKr9J/2Nurg0fctcK34IuW3f60rqjSSpuNG8tryxuTZfq0/kPXwsby0tfkludP6pvya3Oqf7jVv8CKovv1Y+UWBw6l00+QufU5FH+XOrMmI7lUTXJxc+vF5fLKw9kW5POsZSXoHkR092bmjTu6nJ5reel9Vz0nvHwA7nVJL/0Gr9Mz79781kCAAAAYBhbNgAAAADj2LIBAAAAGMeWDQAAAMA4tmwAAAAAxrFlAwAAADDOI6XShgDgOtMqJZDVyqjUGLd1dhkpa2tLbvdHNZwj1qqQ704+90cnEXyFUPAtYYdl+PctLC8khac1N4Ism7mSKVb8KlvCWUOTRg5xKzS3FS2cHvu6uBPAvELWcgxmrvKhU3Ert7sujinajQzsVFymbpf53Kn42SlOHwEhn7tRvHac6nzFz2bAx1s93qiNwOzy0ejldofWrce2ZUsU93Q/MDH7Z57oS3UiVAfndneX9ubc7i0h383c7lZx46ts/N7b+ZJc53N3ilfIWm5lKt9DmHFdvCN5eksUdzMYu8jtbkVxN4PJi+lS5+6MZYp26yyl8PVWrPuWu/r8j8o9v0Dr+PBaK+S7+Y696nWa+CsbAAAAgHFs2QAAAACMY8sGAAAAYBxbNgAAAADj2LIBAAAAGMeWDQAAAMA4tmwAAAAAxnk8j3rXpgwQP1LaeFUdi6sY8nuoPqqE+RSbflRNPm6hcwq6r5rcQ/FRNU+dj+o0HeFojip2PnVOue7lsaQZQ9B9SpIvi+uFtIrLmykH16cZ69axTWfKk41zj8aUecLyijdmTMsox1+hdV0cNohfYcZy/BXeV8/OjOVL7xmKP6vx9NqMTY77Fzun5p+r6LDW+uwUh87pWKo1Nw+8darLK55OdX17dDq/0odO5xHYU1x+gPbeQbX6c7zZufV6O/8u3PE23aVcy4br8oc1Lm880ZddgfPfHJpLS9+vGvKXtLNNUod6xuoL5D+afNQzNpqk4o/OV9lyGR/p10FZHA681eRe/dKJxaFz2SQWV+NpGanJo9PksZ4nZ3y0iqvp0oytU5rq7/GGbMx4/vZo3Xhrx/PS+p2YllH+Um/+Ak02vFW2vJPr4vQVsuKvbAAAAADGsWUDAAAAMI4tGwAAAIBxbNkAAAAAjGPLBgAAAGAcWzYAAAAA4zxSwmsdEpuCq8qs6xihWuV2hwSteyf9tEwELzv8o0mZppaO+6NKZ713YsVjyHfV+RU6p9S0VytirU7XbiWC1zbkR+4p7sS07cntbulEvcUZT4d8h0exjElOD1EIYA7Fccbi1ZSatMKky/FWbncO8240KZO/V4wVbxTnZRRNWsfSCvNO9a2LmDPgO8XVPda6e1N9DvmuBuPD3Hg1tZ7xTXnI18VaC8z+QTZ8KIavMBf6Tbndrc7nc3DTxQqdLw35bqRrt6KFQ253K1O57hxTnOvI50aTZvL0hkTwnK5dBGm3orjzjFUieOdYygDyuIxmunl57+Xc7rOJ4L27N/5qa9zVOeS76tyK4o6Pbavz2eK1Vv3S21NcLSMceF0cxv2VDQAAAMA4tmwAAAAAxrFlAwAAADCOLRsAAACAcWzZAAAAAIxjywYAAABgHFs2AAAAAOM8nkfYtaliwY8QFn4cZdB93biMJj+O0PlWjN+rlPu11lG1LjustT46TT5Ck3IlR9lirY/q7H3c6mV8VOfpI3Q+Yhx9MX6EcPiyOAbdVytJSfK9zuFUf71zLh6hseL/q75YeGxSPUfpsS3vvPiMV+Ov1LlaxitsEMcm1Xh6X5XNn2HGskkq/uwUt5p8rvvXl/cZj6VoEovLsxRO6fni1b6IZ2+b8sZbzbu39Qjkx7lskjq3lvH16XpvmzTj+c6DlIfYO5hW9fs/dq66Mp1P5mt1vlq2OiTpK1NDa82tr1J5xup7V2gdvqS1ihtfLFf4insL307r4k7ncnCt9VHNmH5i1J3Dd/jU5F41uccfAsX4I3WumsRl3J6NzmH88d4Z01naUBzXnO6xRpPyHos3ZNWkdfe2notU3ypuvlUaL73zb7zUudukLo6tG6/TFn9lAwAAADCOLRsAAACAcWzZAAAAAIxjywYAAABgHFs2AAAAAOPYsgEAAAAY55HCWcvU7ZiuXSaCh7zVexmVGvLY6tzukJZVRnGXAeRpGSvFiocm5Uqaud2NDPIy0S11TuOvmEHeyFzcEQkZisPyvt45F7/blQm7W9KCO8nHdW53I587J4Kn3OhG5HMrt/t8fHUZ2v2PGcvU7TKfe8uMvdzuHac0XdyySb5tOsW9PPtG8Z6Hq15erfXshw5XFW+yIYP8Ouns71jeuw8x3kkzIrrP53M3OydDcrs3hHx/R273Vzus/K21FS0ccrvfnancStFOyytzo3PIdCfkexUp2inrumzSCsZeaz3eO2PrLLWK4xXfEffeS53v/PQ7n8+90uPc/LH59RmvDPluLCM26eR2d1fy9c6pg7+yAQAAABjHlg0AAADAOLZsAAAAAMaxZQMAAAAwji0bAAAAgHFs2QAAAACMY8sGAAAAYJzH86h3bY6Qyl4XH1WIeMioL/veyw5rHVVI+lEF16+17lXrssM/mnxUK0lNPqomZYcVAu3v8SwVTV5HfVHKZay1PqqLWC5jrfUqw+HjjIVb6HyrlpGD61vFtbSSCcor+8/6RpPySUznor7HUudO8at6q6TiZ9g4flXHkorL8fRyq4tT56pJKv7cMePncb9oec/qEqSz9OqsubxYK1z0snMsTh8Np+/e+BCVo6lJqG49tueX0bThFfSrhPPRuz92zPgThUPZcJqaJyl8/dgw45bOjS828QtM+UU7Lq8xY/m1sNW5/Lb5jybljOnbafgqG7731p3Dr4ZO53s1nr7Dl8XdJo/TM5YdUpP7etbL6MzYO5be8tIVb52lzr0U7ur6p1/rVo+dTz+JneK1VvnrdstbpfVDrNV5z+/Ezi7Hjm9dGz4T/ZUNAAAAwDi2bAAAAADGsWUDAAAAMI4tGwAAAIBxbNkAAAAAjGPLBgAAAGCcRwptParw0hC3WqempfTTe9XkCPFXzdzuYvzjaKRor7U+qmS4OsV8rY9yxk4sXO5cFKdMsnIZa62P6hq0EuDyjJ0gumo4Z8iVdqRsdhLdWmLsbmfCXrRwJ382FZeJyOmGDGnNqbgK+Q6dY250mUi9Jeu6F4x9Nj58hfDv62ZMKdrhwDdcxF5ud8yGb9zq9d3bC/muxSblh9GOJOhGom/TjkTwrl+UX1367ce35HZ/tfPpVNqYLn9Vwm4rcjt//Suna0Rup/EtmcplunMsPp2i3Yqp7jYp07VzcRHR3VpeTATfEcXdKz59XXJgfOv2SE9ia8bTT2LnfZUeoi1vlVa6drPzVzvE4tj6qjDvLbndrWX4KxsAAACAcWzZAAAAAIxjywYAAABgHFs2AAAAAOPYsgEAAAAYx5YNAAAAwDi2bAAAAADGeTzDrs1R5Y0fIYP8qHLg72HK4ygSx+9Vyv1a66iS1j+OUFwNfoSs9iPM+FEt77iF4io9PS2vLH71OtcX4COEw39U1+UjHHiZMF81iDOmjPpb1SXFzreC7nvFjdqeVuMjHnqjPs1YPlyvMGPZORWX46+jfnvUxaFzegWVzZ+dJs+wvA3FYc3XNXml4uqEpGWUTeJ1qe+l9HnRum023JDhuUjFX+2w/vEx13kSyzfcpuLClnfbZS/ICzvzf+h9uuy4MudnbHZI9eHrR6/z2a8f4ftmKG4u43yT8jthLO4sI38LTV9xy18NjeLU+d7pfK+axOJqvOzQnfERi5+N4s7yyib5WIpl5BnrO6EuTle8vi7pF1BjGWVx61Zf8bfVVQ/Xpb+tyvpWcfPd2ynuvE6TLZ+J3Q+pi5bhr2wAAAAAxrFlAwAAADCOLRsAAACAcWzZAAAAAIxjywYAAABgHFs2AAAAAOPYsgEAAAAY5/E86l2b41Zk1x9hi+c4GhnpR5Ukn/7/j6rzPaa9F//h4ygOJBWvtT6qA38ddfF9FcUft9C5LA6dP6pT8lGdutR5hbN3SwdezXiLM5ada7d6GY17JnbuNIldzgurCMP1OtKRlDdqq/iVZqzuvVT8qp79XFx1Dq+aZ2fG9L56lsWhc9mknC4Vl9P9a7zTpDx76cDr4k7ndF3q4s4VX817LNzqreJaXRwf29YTel1xrdX5/HR8u/h9590zJo0vA81lvPtLQlkcvtDt6ByKy+9XuUn4klZ9eYvF5RfOVnH1zXlXk/Kb9j0eeFnc6RyW8egUpxkf63myyaN3lorpUpN4lt5+XcrbIy0j3L2N4tYvndik89jGZ7x6KbQ6X/cm/EfzDTN2vhy9//Ol2bm0YTp/ZQMAAAAwji0bAAAAgHFs2QAAAACMY8sGAAAAYBxbNgAAAADj2LIBAAAAGOeRw1mL3Zwyn3utdVThVSnP6l5lqx4x4KvKGo/B2GUCX6PzCqnbKXowxIqnWLiycycBMQSWl51XCK5r5UemY+lly3XCJssm3TC2DVG6Hc2M3h0h3+ERCCHfqXMnt/t0IngM8+6Ef2+J4m6laIf48NC5cyytJulUl03Ki5Wa5MjtxjLSDVnee83c7sbzEour5eUnLmk0aRafXcZ3GLKM0c6nZe9y/mptiUTdktvdihXvfaN4b8JuK3J7hRzc3je6TvJx79tpp/MKgc1bIp9bud11inbMuq7yuVOYdy8Yu9Ek5Xa3TmkrsDxd3DLPu9WklQjeC4yPD9HZzqv5Vqmf/fBwbehclm57nV6V2x2ma7k0Ebzl7Gdlms5f2QAAAACMY8sGAAAAYBxbNgAAAADj2LIBAAAAGMeWDQAAAMA4tmwAAAAAxrFlAwAAADDO4xl2beo4+iOFjVfjMce8KD5C53L0CIH2RzXlcbzK4o8QGf+xivqPI+XLFyv5uIUZq+V9hAMvl3GLaw5Nqua3VFyN38KprovL0rVunWXUHZop9/HWu0ZeXHVDxibpHmsVF+OvcPe+6uU1il9H/fZ4lsXhVVN2Xms9q+axuGqel1eMl2tOTdJrc8uxlCcqFlcXt3WqU+ej7ty48Va4e1tNUuf6A2rHQ5SEJ7cx45bOLb33Jv9V8wK89WRv+YzrfrCGJqW6c2vZaXm9GavqVudc3PkWVH5XjGtufe9qFOdvp43lld9O73HN9Yz3ukkqLpqXHVKTXPz874OPzpofcc1XzZjP0uniuObOnRCafFRNWvdYLK46937pdIpTffh9ll4UG15BX58udV5rlW/O3os63B7NZZQ2fI60tM7Sps4N/soGAAAAYBxbNgAAAADj2LIBAAAAGMeWDQAAAMA4tmwAAAAAxrFlAwAAADDOI8XBHlXEWifiex0hYbeM6H7FiLVqMAVj1xlyIRg7hX+XUdwhw+zVSQRvZi62EsEbQXTpWOriMOP5eMuYWldHcjY6v18rLTic0V5ud47ibnRu5Xa38qHrmOqYNd5JCt8RK74hPryTCJ7qW03yqT4bxZ2uS128I+S7d6vHR7x8Lmq9JzT+l8aM5zu3XPwePLu89P9ft+xNIaByu/93F+Z2t4rP53anYylf382Q76tyiFN9K/k4R26fTQSP+dB7org7xZ2s6zJduxWM/VhFaHe7yZ4o7rPF6V6Kud3vDZLv/XjpPba1FF8dksKTN4d8p/FOk3gwZxPB44yXfdpuiRW/csYGf2UDAAAAMI4tGwAAAIBxbNkAAAAAjGPLBgAAAGAcWzYAAAAA49iyAQAAABjHlg0AAADAOI/XUSeI38rdnJBRf1RJ5uXgWus4ivEy5X6tVa6u1TkWh9z5Y72+2HmtdVT5669wlj6qzh/h/JcnJJ2lW5yxGL+FY6mLOzOmLPqySQyur5bXTblPy+71qLT6lsXlPfPP8Wow3Db1DdmZMRWXL4pX2PMtm6TOz6PVpC5+bljeVZ3jjPEiFs1bndO9VM6YbumyyZ67NxVX1an4/DKyq579/pvs7DKG23A6ohHnacsB7vg4SysJn++9zo3llZ1jh7CO8I2icSy5uPW9q9G59Y2u/K64wveu1ORefuFMM96K4nurc9UhFafxe2jy6BSXncsOqcl9Petl1MWNzmklrSapuLxt8lkq76XQufWjpvUItH68dDq3nvH0G2rLj5reL6Drijuv02aPZMOvuYuWcfGMV/FXNgAAAADj2LIBAAAAGMeWDQAAAMA4tmwAAAAAxrFlAwAAADCOLRsAAACAcR4pO7YcDfHQ66OO4q6F2NfarQoAPnZkjb9i1nVxMCm/sJmiXXbeEXEXZmwmVnZy8qrhXqheWRpcmXK6ofWlOcStrOU68jmESbeiuFsx1a0U7RwrfjbrOs5Y53anzp1l7DjVvaT2Ord7QyL4loj6sktzxqQRWN4sbmjNuKnzdYYEWSa/J7f7fPppK/Z1S5McYdtYRisYuxmae1nIdwwAbuV2d4o7Xzibud2Nzq0U7W7W9Y4o7pCi3cntLjvHZZzOGk/1W6LQ64vYSQRv/R6JTeINeVVud/k9L+d2n317rLWab8i3/gIq37FrrfhO7qzkutzu0paPuX7zCRq3OgAAAADfyZYNAAAAwDi2bAAAAADGsWUDAAAAMI4tGwAAAIBxbNkAAAAAjGPLBgAAAGCcxytkk9+q8RgOfzSC7tetGD9iCHlVXE2Xmryq6VLn2OSoz1LZ5CPM+CqLw7GUZy+t+dY5xnRdbp2LWJ6OWFxVx9vjy9OttVJ2/Q71nK35jqpJuOB1cZoxFxfj6Rk/qrs6FZfjr7DnWxeHhygdS9k8Lq93LFd1zsdS3QmdGVtXPN2loXjHjde8q3cUl7Y8tkljeeeXcaWzB9I35tArm05H+QG6QWjS+GjOnVufwhs+37ccS++LTVm84ytT/fUvFVczxuL1+vqM91sqLsbvYcayyT0soxxPy4hNqvpHLH5+vXPZpLW8R6e4dYDdJh/1dWncNun3SDlj/Imx467u/ahpPePVMtLbI3RuFOf6DT+X6g7xtb7lZ1RZfNVHdvMzccMy3v91J9jwke2vbAAAAADGsWUDAAAAMI4tGwAAAIBxbNkAAAAAjGPLBgAAAGAcWzYAAAAA4zxSTG8rCC10qJXRwjFyuw5va+Rzl9nVqXiFUPC0vA253SGBr5eHF2f8aufUPGXLhSZpeY1lhOne7XyYd7c4BjOfDpm+Lnm6tYxWfHiesX5f9WbspGiHdPNeyHfzInY6vzu3u3n3zsjtbnbY8L4ZE2p93btzzCFW5HafWEZo0vky0ArHbeVzpyZ7vkr1crsbMckhETzlQzdSnHPIdGfGt2ddP1aV2z0jirtc2wqnNOZzd2bMTcpfDenAzwbJl2nZ/2zSevY7T2L1kLc6d3O7W8UXvsDDJTjfecvyrlvGlTO+23UR6f7KBgAAAGAcWzYAAAAA49iyAQAAABjHlg0AAADAOLZsAAAAAMaxZQMAAAAwji0bAAAAgHEer6MOOC9HbzEMvfgPOdC+GE8h5mVE/UdYx1G1Scsoi9dat6NcXiiu1vcKxR/V+CscebnsskMqTstL17BsUp6N1KSVRR9vpT1NGuIVONthHVWTVvGWJkd4xl9150ZxOZhmjMUzZmwVt07pCtfrwiue7t6qS17GVwfXWul5aTzM/1h2o0Oy4VXROpYrbXnttYw49CsPu/ExlzQeuV6Hukn+HO80CdX1l4E4Y+N7V+ubQxr/qL/YNL53tYrL6dZaH+v1xQ5rrXtZHDqXxWnGe6fJ/VZ3rovDMh5Vk/t61p3DjI/OjGWTssNa634rVhI7985Scapba17d26Zqkorruzf+Omh0zs9Lo7h+UYS79/yLovW+SvY06Xw5Ov8x11pbv3lpw4zv/1rT1LjHruOvbAAAAADGsWUDAAAAMI4tGwAAAIBxbNkAAAAAjGPLBgAAAGAcWzYAAAAA49iyAQAAABjncbSSxUP4+qvqkQPty8G6vEyYP456HeX+0+1Wdz7CwZQzvkLxR7WSdOBlk9stHUu5jFq55rXWrbO80CR07lzx6wLt04GH+3TDnK3O9Qk9GsVrrfIJTY9tuIR18avsHJZXF8dlFONlh10zlo9Gb3nxupTLqLVm3HLFyy6t2yNprbnVpNkhua7z+215HZ434pRsOhfp8/2s6z7m0sdZqzh832k26RxLKG58sUlrbjUpB1OTj1v9ZarV+V597HyEb3R1cfhOdw/Lu1crKTunJq3ix3qGZVSdw5ofO5ZXriSfpcby6uJ043XWHG+bqkkqLu+QWFzde70nsVOc6luPc34FfbVDKm5+BPQ+L+LPyi+77qOhq/mZeHYlQ77rrLWu+2V6HX9lAwAAADCOLRsAAACAcWzZAAAAAIxjywYAAABgHFs2AAAAAOPYsgEAAAAY55GSd3tR3K0w6SoerZn01sjtLtf2jyYhEjKFSZ/OwgzLK2MDc67nhpy80CHFh5fD789VbdiSktcKLW5FbvcinztJ4XuiuFsh39U6Wp37yyuXseNYvtxhW5POYxQ6J2dzu8+Hdv9zxp+Y2z08FHK0Hefuqg+d1f6sLLU+mpMNwdjN0NzLQr5P5xCn8daMMV37dCJ4TNGuE8Eby1gxijs1ORuM/egFY4d87tjk9PI68eGtK56WUTZpJYKvZpB8HfIdv/A3Oree8daMrRdF8w254W3fe6uHAz+/jEk/l35Tbnfh0oj0N/NXNgAAAADj2LIBAAAAGMeWDQAAAMA4tmwAAAAAxrFlAwAAADCOLRsAAACAcWzZAAAAAIzzOHqR6nW8edkk9S1z7mPxrezcCLS/hd65SVF/HKm4GK+WnIs7x5LOf3lK84xJozic07COr3f4Do1Fr3UcjYWXnZtPXF2fmrRmfFXHsqXz+TV/x/KSRnGcsfOUn79tthxLS/euDk3e7Ceuecub892r3vK2T5+VzSal1qdw6tz6+Gt8c2hN1/x21GjykYqr1qlz2SR2Xq+6SfVNIzW51zOmzsV42SE1uVcd1lr3sjitOTZ5fr34Uc8YiusDD8uoih/V2rpN9hTX16VzEXfcePEeK2/IsLzek3i6+BaXUWv9xLjy90jn3dv7HppmbBVf9YHb+jjb0XmO1r30e/grGwAAAIBxbNkAAAAAjGPLBgAAAGAcWzYAAAAA49iyAQAAABjHlg0AAADAOI9WWm0v8jl2bnS4LhE8BlXX4WGd4pjo21pe1WFHAl9sXg+fTidu2pKHdz5y+LqY5G7WdRkr3lzeu4OxW/ncSWvG66LQy+ruRQzFWx6j8zdko3PXnlfCW6fjlPM3zZVh3ut8OGgrRTt+gHaCsc/nc29psiWHuByPieC9YObUpCjOieCXJU+fjqlOTcow7+6MdSL4rZHbXcaE/2PGEMXdKC6v7Ap3Qsztrm/1TnE35Lu6q+Mj0Hleyme89dg2X27dnxhXhXzX04UDr4s7ndda14VMN5v8pjzvP5rb3eKvbAAAAADGsWUDAAAAMI4tGwAAAIBxbNkAAAAAjGPLBgAAAGAcWzYAAAAA49iyAQAAABjncRwh+LyTh15Gw6cGraD10LnucTvKXPc0YR1of7sV9bdUXDdPxeVgXVx3qA4wdY5NWjPmpXTmbDh/e7RVj0Cr89FZdeqcmoT6RnHuXB14mK8urmu7J6SzvC0zVl22rDnXl1pXfEPnlk1P+IZHg/9d69P2upPavMM2fHKFDr0DDDO2PrI7K4lfr8rvKhu+OaTx8l/nbuFzvGzyETuXx/Kqi6sZW53XWveq+cetnvFeH0sorpqU06XxskMqfnSKtzS5355l8aMs3nKWWsuLTToXsb490j1WFXduyNx5w8PVe1FUK2m9r1qvoH/Uf71Jr0P8wGgceEv38yU0aTk745YD3+Sq6/Jn+SsbAAAAgHFs2QAAAACMY8sGAAAAYBxbNgAAAADj2LIBAAAAGMeWDQAAAMA4j5hWWwbh5oy1rzufCJ4y0FqRnEkZ2NdKCi9jwmNxCiz/codcXNuSWtfJM+20iLZkKm/p8NZE8FTfiuLude4Uf0Pnyw48F5euy+eOzc/POGM6Tnl/RuaOKO4NzgfK9puUn++XdQ7FIUW7GfJdBwC3cru3JB+X0c5pzSEpvKq/Loq7TJhea91XEZjdyrouo7W7y3tUy0hN0vLKpPDrTmmq78W6d26bnNvd6nz2uUj1sbh6KbSe/evehLl5p3jP67Rhzy+g3jJ+U2534bpT+he0Lq6/sgEAAAAYx5YNAAAAwDi2bAAAAADGsWUDAAAAMI4tGwAAAIBxbNkAAAAAjGPLBgAAAGCcR45TL8LCj1AdUtlD3HgnhbzTt1ec/ksrYf5WNbml01QWx96No7nFLkWTcs1/Wbypy+LG3bGOzqnuPYl7Zmx0bk0XzlIo3jFjs/P5A2907jdpOXss3yCs783LjpfqR74gN5y8Lcfd/AAtdV4fe5YRZqymjMWtzvVHc60s/ohrrsfL+rS8svjj9moUh873VTT5CGsui9daH9X4PSzvXq0kdS6b3NezLH7UxWkZxXjZ4Z/LK1by6MyYz9Lp4rjmdI81mpR3SNlhNW/I0Lnx2LaK03h6bN/8Vul+BPRes50vgM2VbDiWi5Zx8YzX2XJd/qjrzpK/sgEAAAAYx5YNAAAAwDi2bAAAAADGsWUDAAAAMI4tGwAAAIBxbNkAAAAAjPNohebmDLNG0GwZZheLq9U1FtEsTvWt4uviw6MYqiyR7f/2/mDmMYnUfzSKO8/4R6O4WxH1wQ99pRRHHk9G87X+N7U+zrKz4aytyO04Y6gOObhXxfHm3O5iMEcLvzv5OKQ1p86daOdObnQrtzsHYxcp2q2s61Zud1xGFea9ZcZmunmjuBXrvsLFTXHv4e7t3Oqdztc9tqnJprdK6briXm53q3MovupLU+vzYl/zN5Pb/Z8bcpb8lQ0AAADAOLZsAAAAAMaxZQMAAAAwji0bAAAAgHFs2QAAAACMY8sGAAAAYBxbNgAAAADjPI4YN15GuG/JJm90Pqra3opDdZ1QH9rf/lH+lf///5qx/C915+aJ3nBd2ufpGvmGvG7GUrghN0zXO8DyEdiyvNZKruu8pcmO2/SqK56EK5u8+7n4XVpnr74w9edLuoiDL1d/aWcfgvQB2lpJ2SR3CDNW/0NeXmPGj7q40fkj3Ex1cejcGv+4vRrFqy6+V8uOxdV42SEV5yYbih/r2VhG1eTRKy6m6864p7hzXcqLmzqnu7ps0rx7G8XpSTz/2N7iMpLWjF9dRipOej9q4ufc2WVs+gXU6nDVd7ohn/mtK/uXDbleLf7KBgAAAGAcWzYAAAAA49iyAQAAABjHlg0AAADAOLZsAAAAAMaxZQMAAAAwzqOZdV07eqFi56O1GsGqKUl3Q+hraHL0jq+uHpPT9hNz0DaETG8JYG4GY/ead5qMCCzfdENfFsUdWux4f2ww5HUwx5UXIH/QfWXony12GJ232ozi/mqH1CR+UIYpw/JaAcCNzim0uAzNjcWtkO9ObncMZu4kgt87nc8nT3ebPOrikK7dyu2uxsvp0ow5a/zd6ebX3R45t7sT8n36eek94zveHq0XRSuRutU5Ccd4VeT22vGjZsvH2ZUzXufC6/Lr/aaz5K9sAAAAAMaxZQMAAAAwji0bAAAAgHFs2QAAAACMY8sGAAAAYBxbNgAAAADj2LIBAAAAGOdxxMzyVqb9+eDzxnRpza1FpPlu9X+pe9dNOmcun/0tRzNZ4wAvPbzQ/Pwt3eu85RjPH0tzGRceS905tt5wveoZL+rLOW/+fMptrrpBrrqhN824ZXnlp23qXH803+rzn5p81DM2mqTij2olsbga/1ivRnE48FaT+61THDqXTWJxNZ6WkZo8ek2eRYfOjL3iarq0vNYpTfW5SeMeK5u07rHynvnXeL289CSeLW4+trXQpPcKar30yua9F3W4iHVxp/PKZ/W8sJIN073/gzXo3Tb87/7sWfJXNgAAAADj2LIBAAAAGMeWDQAAAMA4tmwAAAAAxrFlAwAAADCOLRsAAACAcWzZAAAAAIzzyP+pCD6vc+SjVvn7c9br5R2dlZxfdPOUJr88pf79Z2nLjKFJ72LtWEk94/nOR2xx1Q256U447/0XcbTm9X73+Sjn+00vzdax3JrnPzSvm5TFacZW8boV4+kfnfKMxXgqLpvfqmWstT7qzq+vF5eDa62PasZ7p/Na634r6j9Ck7J52WGtda9mvK/n15fxSMuIM7aaFCtpzdjqnK5LveZwgPG6dO6EcjzfY53iXufW87Kh+Fa9VpovhFqreMsbsu4Qp2wt76sd+k1qb/6SMOfzvfuB+zfNuV4zNL6TAAAAAPCdbNkAAAAAjGPLBgAAAGAcWzYAAAAA49iyAQAAABjHlg0AAADAOI/rwsdul0ULb8qI3ZApNjzG/DrDM+taSe3Nzuc7XHgjhNTtuWfjn4plD7/x/qzmdalvyFaY9B8w58Dfm4MbquvI7U52bxpP+cRlnndOPj6b252incsmMeS7F8XdyLreUvyowr9TmHczintHrHgn3bx1LHXnTuR2bpLuhLP3WPPu7T2JvdzucnkhPryZoj0i5Dt16YV/f730yuTp7gnZ0fn9zuep/35OyH+14YnzVzYAAAAA49iyAQAAABjHlg0AAADAOLZsAAAAAMaxZQMAAAAwji0bAAAAgHFs2QAAAACM8zg60em5tMgb3xBBHpdRL+S6GbcIy3v/qhtX/P2uPB3vvm2OuvWF5//NZ2/4E5f80GWfN/rJD9clve3PX8b0/7//LF03461zllJxWF7r/IfOVeu8jGK8VbzW+iib3BrF5WAsvr2+XnxfneLQOTUpx1tNWsWPtIzbs1Ncj5f1Zedc3OncWd6W8/8Rm5y9x/LdWzT56DwXm57Esrb77DeW0Xq5tTon4Ri3fFksz9IGrZ9+mzq/2du/rP9MTsh/9e4fE/7KBgAAAGAcWzYAAAAA49iyAQAAABjHlg0AAADAOLZsAAAAAMaxZQMAAAAwzqNVnfOsrkr+KvvmZfzE9N4fGprWWPb7r0pI107OXoKLD/DnpWu/P6mdrzguuzLXXZUcxd38cPgtzkduXxr7WsfxdrJ7Uw5xK+g3hhlXucWtJrlzGfncyFTOKdrlMnYEY/fiq0OKdifr+tEK+Y5Nrlrem6PQ072Uw78bd0JInd+Q270l5PtWrST9G3JoMiLkO7VohX+33sndWPHzM275YB3y1W3L2fv1hlys7zD69vBXNgAAAADj2LIBAAAAGMeWDQAAAMA4tmwAAAAAxrFlAwAAADCOLRsAAACAcWzZAAAAAIzzOI6Uv94IJ69b7Ah2byak/+Es+fc6ehfm3deledu0FMdy5XQ9O1byIx+iOZfgvCsvwPne9ZlO5/+6YylnvMUJh98gZ5fXPM/1dKnJraqPxdV/KDvkzo3ij1D8cdvRZL3+++A9dK6L4/LK4mIwjd+rDv9o8jjd5HF7nu/8WEWTXLxheefPXrwu9UW86l5a4UaNxVXz9HCVnXvF4Y3QfPYbTfJL72znJBxj73UabGnS6rDhM3HMl8XW7fFH/eETMvzrX4O/sgEAAAAYx5YNAAAAwDi2bAAAAADGsWUDAAAAMI4tGwAAAIBxbNkAAAAAjPPI/6mRCNbLfd3g90R2/dDktTEXoBO6+HY/MaJ+yKn7y667BDvusF6Kdoji5n+2JZy1bNLNu62bdFK0r4wWbiwjNUkpzmVMcspULtOdU5h0SAQPWdetRPBWunZvxhDyXRXHfO4qijsXdwLLOzPm4sbtUY63IrdbieCpSetWT3n2rce2fOj6b5URId/lf+glf3+9tN98x4w/NMxbbvf/7A+fkD/6S8Vf2QAAAACMY8sGAAAAYBxbNgAAAADj2LIBAAAAGMeWDQAAAMA4tmwAAAAAxrFlAwAAADDO4/3h5juS5Een0f+uvPjGqR5+4M3lvfseG372xiuvl5P6X7ROR/MBSOXFnGkZ5x+56zonrc5bltFssuURKJrcQudyeWnN5Xj6d6TbrZjxIywjjd/XqyiuOq+1Pqrie+p8K4uLwTRedkjFj9vz68VrrUfV/LEaTcoOqUk6lmbxVWevVxyWUd425T2zwg0Z797ODZmalE9oehLLGfMz3ujcelGk91V4qzRebrf43mwcy9c79JvUWmdpR+f3u/Ds/Rp/+Gz4Dv9f+CsbAAAAgHFs2QAAAACMY8sGAAAAYBxbNgAAAADj2LIBAAAAGMeWDQAAAMA4j/dPKbPrv/o9KdrJjmWL3D6jPnu/6xhLo7MR8+JGXJlyEf0T2ghf3zTje/XWd9WVbUZuN7J7V8jHbWX69nKIY8JxK+R7QzBzyIduxIfHkO8qojtmWtf53CnMu5XbXReXzfPyzh9LWPOOKO4Qvt6JdQ/LOH9D5jDvxvOy5eE6/6J4f8h3ko5xQ+dvSNc+O+N3fIDK7f6f/eGzMeL77Q/lr2wAAAAAxrFlAwAAADCOLRsAAACAcWzZAAAAAIxjywYAAABgHFs2AAAAAOPYsgEAAAAY53GEePjJofFS3c/YdPbeeoP82CtenKUfeyxslu+E4rbJz9tbb6g0WfN1kMr/6MNRno7bnrNRNEln/xb+Q7mStLxW8cetGP9Yr7q4alIOrrXuVefU/B5mLJvE4vX874OPWyouxnNxo/OjKk71j3jg1YyhuJzxHpdXnf/OWUrjrSbpHiuveO+GjDde57loPVw7Zjz/Cmp1bjevu/RmDBpvyJbrPrLn/Drb9CH1S8y5Lm/nNngTf2UDAAAAMI4tGwAAAIBxbNkAAAAAjGPLBgAAAGAcWzYAAAAA49iyAQAAABjHlg0AAADAOI/0H8Ssb3flKb1d2LvyM2+P+iz9zGPhb8l3aXFX59fBVTd76vvuF9O1GmfvugO/VctI04Xx+kDKznnGuvijU1yOlx3S+P32CsX1+L1uUs94r5rc17MsflQrKTuk4kevc6M4NW8Vp1P9uOwspRnL+nTblE3S7VE2iTdkddu0OrceorXWrXqem49trfWMhw6NzrFJeBJbM55fxpYZt3zavv0DNN1L/Bd/4IT4YTSRv7IBAAAAGMeWDQAAAMA4tmwAAAAAxrFlAwAAADCOLRsAAACAcWzZAAAAAIwTQ75/ou8IJRud9fYzU9qKU/ozD+QbOFH/sdFPclOO3C6P0l3zJteFr+eg32owrKOVFN7K9C3jkFc7U7kT7dyJfI4h071g7KrzLWRdV8WPy4rTSlJxK4O8jkLv5HO3itN4usdaieBlRHfr7m3ldsfHNswYnsTrQr6TTufYpTXjVzv0m7Q6/MTc7tqW1PNfY8hFuZgr/uP5KxsAAACAcWzZAAAAAIxjywYAAABgHFs2AAAAAOPYsgEAAAAYx5YNAAAAwDi2bAAAAADGefyuoPbbdy/gX37TqT5mn+o3+01X9s/achGHPxXlMd7iqjeckjBjUvyXIyzj/ae6OWPj7JWd03StZdyqZZSD3fFYfCvGP0Lxx3p1isvORYe11v0Wxqv6cnCt9aiaPOKMz68XP95bnOrLNafi8myste6raJJOaXld7uGKpyYfVZPePVbdpalJ64ZsPURpGbnJ2eL0Xmp1rjvEF9OGF3hrJeen2/LZ9x1fBsob8o/6swfuF8kv5q9sAAAAAMaxZQMAAAAwji0bAAAAgHFs2QAAAACMY8sGAAAAYBxbNgAAAADjPJr1vz837XfFo5Whub/f7zrG3//QXebdN0JrviHXNa15yPIGefcZaUVuB9V/6IZ818nHl8UklxHOK+Vzh+IY/l3HV9dZ12HGVjD2luIq5DsceFmc6nO6eSe3u5MIXl6XFPLduhPiPdZJnb91OpcPXSu3u5nPvafJ1zvH4nCMb15Gc8Yfmttd2HKWfpMh1+VKrjhr+SsbAAAAgIFs2QAAAACMY8sGAAAAYBxbNgAAAADj2LIBAAAAGMeWDQAAAMA4tmwAAAAAxnkMj7T/A2H09fn/AwdeG3Pgo58LvmbLRbzqlkx9x9x55ULGPKAzpItVjt/C2QvFnWXcUudivLXm3KSe8aMuftXF1bLLDmutezV+D53j+K0YT8WPqvgRi5+ni4vBvIxGcaovz8YKy95ySj/q4nQvhSZVfXkvpSbpHivv6lhcPTD5GW88RM0nMek8+53voc1PqC1NWh3Ofkh9x0fwVWfph/oDB+6rFG3+ygYAAABgHFs2AAAAAOPYsgEAAAAYx5YNAAAAwDi2bAAAAADGsWUDAAAAMM5jS5c/HFZWRNH9hbMx5hj/QBQg36x1j214MsoW77/RhyzjOpsyYre8C8smW9KCzxavZhR3OX4PAcxlFHQzH7qRz71C1vWm4iqKuwrzvngZjfDvVkR6SgRvFZeR2znkO92QRZNW6nwr5PsW7t7yBdLqnLTCv3ud41uvFR9+dhm5SUvrlE6x5UT9GnOuy5VccfbwVzYAAAAA49iyAQAAABjHlg0AAADAOLZsAAAAAMaxZQMAAAAwji0bAAAAgHFs2QAAAACM8/jDefG3cvTXn5AxB1iff/4PY67XWX/jeqejPHsZ0///9rN61QH+ZeU5TSf61rnkoXN9sT5u9XhZ/5GanC++vcri+yrG753iVN8qfqxnWfy4FeOP0PnR63y2eO04e62z9JGK69ujLk43ZHnbxLu6VVzNmIpbD1d4alvFsXldHM5ea8bzy2jOeF3n6/Qu4q/3Nw7cFx6+gb+yAQAAABjHlg0AAADAOLZsAAAAAMaxZQMAAAAwji0bAAAAgHFs2QAAAACM8/juBfxn5HN/u18e5DfpVP8Sl57S8bdjucANp6RsMfxsjAksr6UU7V4Udyda+OvTdTu38rlbMck5mLkT+Xw6crtVnOrLfO4VgrRT5zLPu9U55XO3ivMJaWSQl1e8dUpzrHtV3AnzTuPxEaiatx6BTU9iQ3xRVF22zFi+lbe8kHOTsx9/3/F5cdVZ+qF+0bH7ys8P4K9sAAAAAMaxZQMAAAAwji0bAAAAgHFs2QAAAACMY8sGAAAAYBxbNgAAAADj2LIBAAAAGOfxHZPe/vvQ8f5VzPAdB16c/x/qz942/B9ad8KYByAtZO59nVY25JRevIyrrsut6lwO5g61j7JzqC6L0/jHrVMcO7/+++A9FN/r4mJwrfVYz3r81mlSFefOxfhjR+d7Z81lcVpJui6tGcMVD8XVbdO68VZ4NNIN2Xq4yicjF5edk0ZxekJbTb7eodtky4w7Ol+l9e79TYac/4v90YvLD5K/0QEAAAAwjC0bAAAAgHFs2QAAAACMY8sGAAAAYBxbNgAAAADj2LIBAAAAGGdXyLfc7v+b3O6v+LO3B99ufCJ4OWdj1cOjuP+CkBbcKE7VdcJxKw65U9xtUud2h+TpOre7U1ymZafiFYK0y3zuFYKxWzPeQ+fWgZfLiCHfnSju1qluXfEtGfDphiybN6O4t4R8lzqdO2Heecba+aTq68K8+83P23BKf5PfdeB+TzDdlifOX9kAAAAAjGPLBgAAAGAcWzYAAAAA49iyAQAAABjHlg0AAADAOLZsAAAAAMaxZQMAAAAwziOFhf/ZmPsxB74lxP2txpw62Ka8q3/ew/lX1C+hcL02vLFusUkxvqW4PJZYfGt0/kjjVZOP9SqL71XxfU9xMZ6KH1Vxqn/EGZ9VcTGYZkydH1XneODVeHk2/tWkc/bKOyFd8XB7NO6xeENWnVN9fl7K4tr54uTW+aLd+nxpLSM3KV3X+VKNi/ib/IFj9CODQd7/xPkrGwAAAIBxbNkAAAAAjGPLBgAAAGAcWzYAAAAA49iyAQAAABjHlg0AAADAOLZsAAAAAMZ5/PqY+0kH+P4Q94ZJJwr4P6S3xy9/cK97ad5C6zRjayW94qr6Fq5sOZ6KP6rxcnCt9XF7hSbFeGxSF9ed79X4PSzj0SkuO6+1HlV9LC5njMfy7BR3OnfWnE7Ihot423CPlTdq6rzlESifxC3FdYdwLF+f7p/1Z9/2ecbrOl/l/Nn4oUb/kGj7oxeRmYY/XP7KBgAAAGAcWzYAAAAA49iyAQAAABjHlg0AAADAOLZsAAAAAMaxZQMAAAAwzuO7F7DTd4TFjU4Ek54HV0hP1ujXwRjjz9JVMb3N0OJaGZ+cooVDaHHKga6b3KvxnK5dFqfOrRTtYrzM4f7n8qomneJW5+uKcyJ74yKm8PUydTsngndS56vOrecijafn5Xxxcl2e95b46jDjdZ0vJM/7V/ijF5Hv9ZseIn9lAwAAADCOLRsAAACAcWzZAAAAAIxjywYAAABgHFs2AAAAAOPYsgEAAAAYx5YNAAAAwDiP717A/+D4hjnnhrh/x9kAvio9oUPeKeXyhqxtvlt9/uprXhbfwrkuh8N09Xj6t5eP9SoGb3XnjzDjx61q0im+V8tYa90vK36sZ2hSjN9j8XXHUpy9ezql1Xgurmcsm7TuhHTF6xsydK6fi07n1X5eGsV1h3AsX58uF2/4TpdnPNv8/R8NW07IcH/gA/f3X0QG+gNPlr+yAQAAAJjHlg0AAADAOLZsAAAAAMaxZQMAAAAwji0bAAAAgHFs2QAAAACM8w0h32/Pf/uRyV9S8oA/5qow3es6b8n07SUfV1M2E8GbId91kxBfXY2nrOuySZmWHTtXod3/mPGxYcZUXEVxh+I6fD2dpTpkvRHmnZtsuSHP5nanhyiPt5r8+jzv616bWzTC13+T33WAfpHwDX7XQ7SBv7IBAAAAGMeWDQAAAMA4tmwAAAAAxrFlAwAAADCOLRsAAACAcWzZAAAAAIxjywYAAABgnMd1rY/rWkejQ9y/44QAv1v50vujL5v0AXCr/kP306JVf+tcgrI4HktV/BGm+7i1il/1+CrG76H4XhWnzmVxOZhmbBXnGesT0puxPPDUuT6l4bqcvuIr3mONGbcUl3d1Lt7QpO4QztLXp8vFG969ecarlr1D42L9Jr/rAP/oNwfe5nc9L+/mr2wAAAAAxrFlAwAAADCOLRsAAACAcWzZAAAAAIxjywYAAABgHFs2AAAAAOP0Qr7ldv8fBOIB/6PyRTH61fYDnH/7pg6N8ZhDXMeKN3JwY4rz6QDmVucVMrBbIdN7crs7wdh5xs7yOongZZB56hzC11PIelXcSQRP46107RSMfT6ivvVcrLV6Wded9+yb87x/Zpj3Cm/C3+93HaOfL+zxu56L0fyVDQAAAMA4tmwAAAAAxrFlAwAAADCOLRsAAACAcWzZAAAAAIxjywYAAABgHFs2AAAAAOM80n843rmKtYYnu7/9bADwDdJHUTmei4sPjV7xrf7YKYs/QvFHXfwKxWG8qr+H4nL8HpYXijud0zLiCWnMWBans3cvT3X47lCOl1c2LSMV5/GvLmOtdauqm89FYxlZo0m55tSktYx0LC1hxuFfLbdcxJ/ndx3g8HuMn+R3PRo/j7+yAQAAABjHlg0AAADAOLZsAAAAAMaxZQMAAAAwji0bAAAAgHFs2QAAAACM8/iO/LfRMWEC8QDeb8cHQ+P9velzqJND3AlmzpHPZXGKqW4kT5cx1bFJK7c7JoIXTXJud6NzCiwv63NxK7e7kQjeui5l3Hssbt1j4Rlo3ZDNL02NyO34hNb/YUMi9fk87zzddUnhW5yNQv+hfuYx+pnCNj/zEfij/JUNAAAAwDi2bAAAAADGsWUDAAAAMI4tGwAAAIBxbNkAAAAAjGPLBgAAAGAcWzYAAAAA4zw29Zmb7H589wIAeIP0OXTrfA6k4nL8Fj/6quI0Y/UfWsv42FH8cavH7+v1xcHU5CMU329l58YyPqoO/1xe1SSekLK4deCd6xLOf/kPa+n2aM3Yu9XL0jCeOzeKk259ZcMXw3BCrut8obnf4Df5sQfoFwx7/NhHgP+bv7IBAAAAGMeWDQAAAMA4tmwAAAAAxrFlAwAAADCOLRsAAACAcWzZAAAAAIxjywYAAABgnEezfnSy+/HdCwBgo1t4r9cfRZ0PqNS5pbW8XFyMf4Tij1ur+FUVF4P/Gq9nrIvv1fg9La8qLte8wjHGztWaU5NececipiteNmndHulOj03CMYYmnc6d5bW+pt16j/NVxdd5/zK2vPSGG3Jxm37/dWG7n3mr85/zVzYAAAAA49iyAQAAABjHlg0AAADAOLZsAAAAAMaxZQMAAAAwji0bAAAAgHH+EfI9Nz5MGh7A+3U/FUL9u1/hWwKAW/m4OyKfG51jyPfpTOsVorivzO1OWeM7Asvrs9cp3pIIXjVppmiHuzHcN63c7uue0HoZ8flMJ6Q149lj2RNYfnIR/4lf/k157k+U/8Evvy5s92NvdTbzVzYAAAAA49iyAQAAABjHlg0AAADAOLZsAAAAAMaxZQMAAAAwji0bAAAAgHFs2QAAAACM8xie+H589wIA+P/5Pa/k9Ml3C8d4q/6HWFx3SMXF+EfoXI6nzmVx7vxqNLk1iu9hxnvVJC2vbNI6S2utj+pEpeLyuuQrfr64dr5zf8ZW50ZxsOWtsqFJOCHDX3qN6/JD/aZjgX9wq/MP/soGAAAAYBxbNgAAAADj2LIBAAAAGMeWDQAAAMA4tmwAAAAAxrFlAwAAADDO47sX8H8ZHqII8NMNyY/ctIxG8nH4hLkuFXhHivNlieCtAOzVTdeuU7Rb8eGdY+kUp5VsSQQP4etlbTe3uxqMnVve/c0rLbsubnVuruQi71/GkAPnv/Gzhshjy3/AX9kAAAAAjGPLBgAAAGAcWzYAAAAA49iyAQAAABjHlg0AAADAOLZsAAAAAMaxZQMAAAAwzuP9Ux7vnxKA2W5fHlxr3dJ/ON85jhefXeVgv/irHdZaH9V4ObjW+riVxa+vd15rfdyK+lhcNS+XkZqk5d3Kzt1jmXERm8XleKNznjEVlwfeKM62fAE82yS/PIZ/Px2+vLM6L/U5fvlF4aSfeVczkb+yAQD4/7Z3b8ut40C2AAmH//+PLc7DOTG3qNXtGoJSSc58pNFFALxod0kRCwBgHC0bAAAAgHG0bAAAAADG0bIBAAAAGEfLBgAAAGAcLRsAAACAcW4M+RZ8B3CzT3rR3pfd29IKZg4zqf6QkqfLmOQ4uArMbuVDtwKwtxTpJYJ3AsvjwkOseH0R4+BO1nU9vTC4UzlUSNJD9NQXRXn/x8Hd4s3xN3n+NIYsfIv3XMsnfdqy2Xve0rwTv7IBAAAAGEfLBgAAAGAcLRsAAACAcbRsAAAAAMbRsgEAAAAYR8sGAAAAYBwtGwAAAIBxvrdUObdUAeANrfiXER8O5fRWmNsKiynHxyL14NrXalT+qo6XB9PxVZ2uW+RrPRpnPOrB5UyaC6+ltbSuS+u2KW/1fHs0pBuyVbn3CNQ1Ws/ylgd/Q5Eda3m+4dODP631Aodd/MoGAAAAYBwtGwAAAIBxtGwAAAAAxtGyAQAAABhHywYAAABgHC0bAAAAgHF6Id+CBwH+gjExlltyiBufXTnFuWFHInhncMjtLgfHAOxebneI4q5m0osPv22XthTZktvdSgRvDr7LlhdCL918xxmve/40hix8i/dci//X4Tje9e7lY/mVDQAAAMA4WjYAAAAA42jZAAAAAIyjZQMAAAAwjpYNAAAAwDhaNgAAAADjaNkAAAAAjPOd/nA+cxYAzLLhQ2BdrxBLFNNLY8vj3bmt+oz1LoUztgY3pvEVKpfH0zRSkdYZwy7V6sGrs0udwfGMresSSl+/4mnSzRv12f92y08o/8o/tOH1vMN4C35lAwAAADCOlg0AAADAOFo2AAAAAONo2QAAAACMo2UDAAAAMI6WDQAAAMA4MeQbgHc0PLGyFcV9p24+dHUwBjNfzbr+CvHV5dcsMUy6KpLzuWvlTHJ8dWMt19PNW9M4wvVqnTG7Gti85f7flHP/1IVvybretBbgkw3/1xH8A7+yAQAAABhHywYAAABgHC0bAAAAgHG0bAAAAADG0bIBAAAAGEfLBgAAAGAcLRsAAACAcb7PV88AgL9kw8fO+vXB4zhWdcYVRpeDY5E4uDq4tlR+6uBcpPb17LX0XC/Su+K9Io3norvwm4vvr/AST572m+5S6W3X4v+BgPfgVzYAAAAA42jZAAAAAIyjZQMAAAAwjpYNAAAAwDhaNgAAAADjaNkAAAAAjKNlAwAAADDO96snAPDh1oYa54Yal+WFbJleUaS5dc/epdaGrDC9skgcvBq79FVOI4wuB6fi5TTi4Fi5sZbrlbPGWpo23JDNtbyfHfsM8F+8Vfg8fmUDAAAAMI6WDQAAAMA4WjYAAAAA42jZAAAAAIyjZQMAAAAwjpYNAAAAwDhCvgH+rjlZmGEmG1Kct+RDh+Mb8qHvzLpuDW5Fcaf9v3pdYoXOnrZ2L9uQFP57c57E2vT5AcDH8isbAAAAgHG0bAAAAADG0bIBAAAAGEfLBgAAAGAcLRsAAACAcbRsAAAAAMbRsgEAAAAY5/vVEwDghc7rJVbneBqc/3B1GuUau2drrWXVZ6y3esfgpFE5aZ2xt0vVH5pbukGrSGvhd9rw2LY8fYEAwP/nVzYAAAAA42jZAAAAAIyjZQMAAAAwjpYNAAAAwDhaNgAAAADjaNkAAAAAjCPkG+BdjU/evRpFfOcCU4p2UCdSN4psSjdvJYI3zrZW42K1orjzbXBfUvWOyjemzgOwgdcsf4Rf2QAAAACMo2UDAAAAMI6WDQAAAMA4WjYAAAAA42jZAAAAAIyjZQMAAAAwjpYNAAAAwDjfr54AwIdYe8qce8pcE9YyYm5HmN4K01vV6HyxQpHW2ldjcG8tjaO9OacNuX4nNJ+LW++x+6YNAHALv7IBAAAAGEfLBgAAAGAcLRsAAACAcbRsAAAAAMbRsgEAAAAYR8sGAAAAYBwh3wB/wpjQ4k58dbPIda1ptIKxc273XYng1093HEdvqy/fZHvu0h1VpgTa19IK75p1qjvmrQIAH8uvbAAAAADG0bIBAAAAGEfLBgAAAGAcLRsAAACAcbRsAAAAAMbRsgEAAAAYR8sGAAAAYJzvV08AgOc4L/73a8ckVqxSTG9dnvO+IqUNlTunu/GMHyVtUn3z3bWlcRY3ne9WH7UYAHgnfmUDAAAAMI6WDQAAAMA4WjYAAAAA42jZAAAAAIyjZQMAAAAwjpYNAAAAwDhCvgGmGx6km6Z337TLyq3TpeTvnAheZpA3ij//Ig6/bUqbArfT0jvlq7FnDHsv/tBKl/+kEO3WWuYsvJzJfdOYs3AAhvMrGwAAAIBxtGwAAAAAxtGyAQAAABhHywYAAABgHC0bAAAAgHG0bAAAAADG0bIBAAAAGOf71RMAeDNrT5lzT5m7tKY3ZC1DppHU0wu30/C1BOWsew9MY/SWPbpzo9NaGuc8qyLrzlk3r2H5lzl37/DpfYi0oZs+KwH+Or+yAQAAABhHywYAAABgHC0bAAAAgHG0bAAAAADG0bIBAAAAGEfLBgAAAGAcLRsAAACAcb5fPQEAdlq3Db7VfTPZUvnOjTpvrL3B1aXn5TUqt/YoDW4WKae34WKVlfNeFGdMk8hFir+coczT3wnphI2t7m/IBBsWDsBf4Fc2AAAAAONo2QAAAACMo2UDAAAAMI6WDQAAAMA4WjYAAAAA42jZAAAAAIwj5Bsgen5G7I4zbsiI3bTwYiabMsjvqtxc+Og43nJyW2YcArDT8TC4msoZg4+rmOq14T5Nawnp2mnhW6K4r2pd3FYieKpdHp0drQ3/qXGrA7yQX9kAAAAAjKNlAwAAADCOlg0AAADAOFo2AAAAAONo2QAAAACMo2UDAAAAMI6WDQAAAMA436+eAMDHO189gddY9eHWbtSD68rhfL0zxiIXx3bdV7uunPbo+r3bqnw2pxcGpyLl8bp2c+GNyvXpwtjVuxHS6NteQWXhzpPfnfGdD13h+dMYsvAtPmktAC/kVzYAAAAA42jZAAAAAIyjZQMAAAAwjpYNAAAAwDhaNgAAAADjaNkAAAAAjCPkG+BPuJ6rulqR2+1T/tEo9OY2lYMbwdgxcvsMRaqU6VaKdhrcWnionAY3jufbrjpjyOIuNy/vUlGkmXWdYu5bD1Fjer0Se7LGNxQJseJPj0LvGT49GCQ84/Bp/MoGAAAAYBwtGwAAAIBxtGwAAAAAxtGyAQAAABhHywYAAABgHC0bAAAAgHG0bAAAAADG+X71BABeb+0pcz75jKtzxuB6hT02XYLfV96w8C171ypSDs4VyqVv2Ol0xtb0wuDG9NLgWOQsjp8rFSkmmKfXuIxlkfTft+7eLTdkOGNj4WeYR9jpnrL2fW+P5PnTGLLwP2DDMw6wl1/ZAAAAAIyjZQMAAAAwjpYNAAAAwDhaNgAAAADjaNkAAAAAjKNlAwAAADCOkG8AeIJWfHVDylSuIq3/IXK7EZh9lqVDYHav8o5pNHPdW0nhKV27XEs9uFE3XdxmEHfznA3N5Omru5T/0gpZT3Xv2qVNhk+vQWA5QItf2QAAAACMo2UDAAAAMI6WDQAAAMA4WjYAAAAA42jZAAAAAIyjZQMAAAAwjpYNAAAAwDjfr54AwFOtV09glOfvRuuM4y9WY4Jn73hd+ayPNwaHCr3jeS3V4LOu/Fjl4LpyOTh96fQIa/mq11Kfsjwad6ma91nNOVVOd9L1wf/yl9+WryuszuDy4oZNektpl+9b4vPP+Ic1bnVey3PB5/ErGwAAAIBxtGwAAAAAxtGyAQAAABhHywYAAABgHC0bAAAAgHG0bAAAAADGEfIN0NXL9byeK7k6Z9wSY9ksMiTodMg0YhR00ErR/u3BODiMDkncx6OuHHK7fz2NVKSZz90LLC+Lr7DyVWaQx0Tw1kWsEsHD4DpYOCXD957bflb47zQTdtNFTLHi5eBwETtrCXnqd+3SJsOn1yCYGSDxKxsAAACAcbRsAAAAAMbRsgEAAAAYR8sGAAAAYBwtGwAAAIBxtGwAAAAAxtGyAQAAABjn+9UTALjFevUE5jkvDk5b2j1+XVm5ebo0vLVLG5Tny5Mopn2GtZTH0+BHKnJWRdaGMzYHF9Kcv6o5H2Ha903vDJcxVK5dfWi3DW88L6sz+AyzCLdYT1n7+R8Nz5/GkIVvMXstUz5H+I3Z9xL8E7+yAQAAABhHywYAAABgHC0bAAAAgHG0bAAAAADG0bIBAAAAGEfLBgAAAGAcId8A/6AR1fl3oyJnrHxLqmqKHC5jo1vBzCkfOuSOpsjt6uBtkdvpeCsR/BGm97g8+CsNTuHfrQ3ppZuX8dUpa7waXA5tptJuCswuR294vJoJu42tbhZprCUNDUHmc6Kdh0/v4wn/BnbyKxsAAACAcbRsAAAAAMbRsgEAAAAYR8sGAAAAYBwtGwAAAIBxtGwAAAAAxtGyAQAAABjn+9UTALhqvXoCG923llfs0vnc09VLPJ89jVqaxFlNuzyYB4fKZzH4sULlanAan6b3qI6XB1OR1sJT5a/O9NI3V+XgFSqXx5tXPA3+7cF/0BzfGl5uSF0h7F49+AyzKO/fNOPG+fa8IVON4px3TiNNIt29I96QLc/fPT6Ve4m34Fc2AAAAAONo2QAAAACMo2UDAAAAMI6WDQAAAMA4WjYAAAAA42jZAAAAAIyjZQMAAAAwzverJwAwwfnqCfynxkzWfbPgfys2+wxX4Dyr4ysM/vXBdMY4jV9XOI7jEYsUxx/lAo/jUa3xPOsvhx6rmGCuXBT5Out9KqdxHMdXVTwNLo/WVzYUWeEylkVC4eOsiqTbI70Q6n0Ko1fvvdIoXa4lDU67l65MGPz7Cr0inR3t6V7cm874ph8u49dy/e4F/iK/sgEAAAAYR8sGAAAAYBwtGwAAAIBxtGwAAAAAxtGyAQAAABhHywYAAABgHCHfwNsYEtU5ZBqZxNB/l/eolU+84Yyt3O5HfbCR290K895SZMv0WoO/OkVW+O5qVZudArDL473w9RBYfpa56eXQbUnX18vUFa6Hh6c/rJDUXq4lL7yR1B4q1MJtM+dFXd5jGy4ivK/8OMML+JUNAAAAwDhaNgAAAADjaNkAAAAAjKNlAwAAADCOlg0AAADAOFo2AAAAAONo2QAAAACM8/3qCQAU1o21zxtr/9qaMo23VO5dXkv5l3r/W1flDOcsj/cGh3mcqxwcKncGP6rBx3E8qvF5cPE90GPVi3lUC3+Eb5Ie1ZWJg8P2rWraX63dSxexs0vhdky3x28PHsdxdu7fdI8dYdrxnA2NJ3HFB7oYn9ZSLiWu+5cny4OT8L5qLDAXuU9vepNtuYh3+pytBm7iVzYAAAAA42jZAAAAAIyjZQMAAAAwjpYNAAAAwDhaNgAAAADjaNkAAAAAjCPkGyAaEgI6JvL872aOhgzmRhxvDmb+/el68eEhRbsxOB0/e1HcOxLBW1nj4YxlrHIcXJ2xjAlPlVOK86M8GqdRHDw7uelH+HYuPsxlYnZ4B+Wk6rp0XaMe2ohmXvFJbAz+/emOML3nf148P7766VnjNxq/llbuPE8y/rbhM/mVDQAAAMA4WjYAAAAA42jZAAAAAIyjZQMAAAAwjpYNAAAAwDhaNgAAAADjaNkAAAAAjPP96gkAf9q6sfZ5Y+2OO9d41+mePOdblffBGZZ4ptumOnzGbSr+cIbR5yqOP8L0vqrjjzCJcnBceJxe8dXO46x36VGu5ay/HHqsosgjTOPRm0Z9xlWNL+d8HMeqNqo8GAen+6M6nCrXFzdU/gonDI/ABvF5qZXzqyu0XkFpEtUtdhzhipdrydNorCWcLtVN57xafNNbvXyrbLiI/M6G24Pt8uMMe/iVDQAAAMA4WjYAAAAA42jZAAAAAIyjZQMAAAAwjpYNAAAAwDhaNgAAAADjCPkGnmF40uHN07svfVOu53+zJ2azEQDcTgqvB1+N4i5jwo8QmJ0yrVOseHk8D66iuMNuhMohEbwqkgb/xMjhcnq1VUY+h3TtVe1qTASvZpezxgv5xquVx9N1Kfc0LTzn3P9+IkkjzLgVjB2i4euL2IqvTtelvJeSZvj3kI+Az0meltYMDORXNgAAAADjaNkAAAAAjKNlAwAAADCOlg0AAADAOFo2AAAAAONo2QAAAACMo2UDAAAAMM73qycAfJr1gnOeLzgn90vXtbzHWoO7Zzw75ywHhwrHWRVJ03hUB1eo/FUdf4TB8fhZFVn19z2Ps5j4z6orr7Mosla99OuDj+N4VON/qgUex7GqNaZ7qZxIWPexqsuYLmJ5xeNE4jNQ/KG8PVKNRyj9VR6Oj9yWZ/Hq07/SY9u5iK119zapV6Rxzi1vyJb00kuXYLItF/FOreeCJxl/2/A2/MoGAAAAYBwtGwAAAIBxtGwAAAAAxtGyAQAAABhHywYAAABgHC0bAAAAgHGEfAP/R28a5n3ftLdUlv74HJsSwVu53WFwFTJ9hmzhVnx4mdv9FTKtY253tZaUCP5TZyq3Kt81+DiOn2p8uuJlYPlKgeX1wjuDw3Upa6TKMaK+vsfC4FCkMbiTNZ6fuA2R2eWTmAaXydPl1h05YD4oH9sQUd+p2zpj63Ld+klUvrI+Jvn7GPQ53nqIeJLxtw0T+ZUNAAAAwDhaNgAAAADjaNkAAAAAjKNlAwAAADCOlg0AAADAOFo2AAAAAONo2QAAAACM8/3qCQDcYr3gnOeTz9da4/UNecWW1sqNXmGCYXCj8nEcZ/WXM4w+V1G+rHAcx1nNpTwYB59hcDWNR6icdu9RfbXzE1a+VjH4EQY/Lg9uTeM4jlWNf1S7dBzHT7XwdNus6uKucF1WdcaVvkI7H7+s8I+uPrvp+73rr7xcIf2lXEu4Ey7PJFdoPOOhSPcl1Bh6/V19X+U5Z7zP+LU0HiKepvkK4m/xKxsAAACAcbRsAAAAAMbRsgEAAAAYR8sGAAAAYBwtGwAAAIBxtGwAAAAAxtGyAQAAABjn+9UTAN7AesE5zxec83fsxgDFRTjDLpXXK21ovrity16UP0OFctrnWQ9+rOL4CpXL43Fw74z19z2Pai2PMPinGpwqr87gR7i6P6sqEgav6g8/YZfWKmaS7pjyjGs90vCqQtilUKS8986w8HKJ6Xkpz/eVhlelw47+g7L41cfzCI9GequUh1f1sKQiacbpRVE+AknYo3zOq5W32LB7XLDh9gBu4lc2AAAAAONo2QAAAACMo2UDAAAAMI6WDQAAAMA4WjYAAAAA42jZAAAAAIwj5Bv4L28aX/30ab/jnI/3jOrckUpbx/H26pbH8+BGWnA5uEzLPmIOccjnrvKTUw7xVyrSSQp/VOHTZbT2EZKqH+G6/FRXMV3EFIdchoKXWeNpeumMP3X0ceMi5gDmMj68DvNuFQkJ3b0Y5zLPO92QZemUNR73+teVs8b0WsHYaS3lQ5deCK3w7+vJ30dc45zPiw27N8SdEel8pvzY8rf4lQ0AAADAOFo2AAAAAONo2QAAAACMo2UDAAAAMI6WDQAAAMA4WjYAAAAA42jZAAAAAIzz/eoJAK+xnn3C89knhOwMT0B6Lsrb9wx39VkdPlddO1QOg6vRj1B5VUXOsx4ci1TjVxj8U51xhS+HHmWFcoXHsVZR5BEG/1SDj+NY1fBVHj2On7JCfI9VCz/rafyUmxcLF/tUXtkjXKz/9x9cPBp2ur6ryyt7hC8J4wdDOmXvwa1Lh+O9C1MdarwQwjPUnVzajsYHbjk03WPl8NacN2m9qkd7xe61fM5Wf5Lw2PKx/MoGAAAAYBwtGwAAAIBxtGwAAAAAxtGyAQAAABhHywYAAABgHC0bAAAAgHGEfMOHe0Xm313Rj89fSysq9ePdmkV630aHtNoQxxuLlINbud21Mik8VS7jk1Mc76MTuZ0uYnk85GLXucU5irsxvZ9ql+JawvR+qj+kKO5yLTmDvBpczyJsabhByumlfOj4JJ5VUniIQn/EkO6ycnHOr058dSsRPFYJlzwnVf+2dHoSWxm75cU9QyJ7Sp1vZfpeDwDOb8K7XqdblNfrTT/Hx6c4N+4EnmN8YDz/d35lAwAAADCOlg0AAADAOFo2AAAAAONo2QAAAACMo2UDAAAAMI6WDQAAAMA4WjYAAAAA43y/egLANuvVE9joD6/lvFzhz6r36ay2tDx2HMcKe10OT9elHHyGM57VKcPYei2Ps57Iqio/QuXVKbLC0svjP+HLoVUt8qc8ehzrLIr8hAuwwv6taiY/dY1jVRdsrXotP9UJV1jLT7nVoXJ9xcLFirdvtfB1hnshFqlHdwYX0rLTI9AafYZLEJS3eqpQPraNwanyGZ/EYnw+X2N6Oz5fWu/C3uv0ujO+r3p32QSNO+w1encC8Et+ZQMAAAAwjpYNAAAAwDhaNgAAAADjaNkAAAAAjKNlAwAAADCOlg0AAADAOEK+4f2MSXPckNo4ZC3vGPb5DlpBs+W98OzrkuNgy8G11uA6ETyG4zYSwR91tHBj8BGuSwz5rvKJy5jw4zh+yspVmPcR8rxTgnNOCi+P1kXKd0Iv/jrlunfWUm5emkYuUuZ5p0D76hLUFerc9DPEh5fp1Sl1Pml901gWzxVa2dPl7VEPDo9zL3m6fPabn1yN6bWymvNz0Sjz/PjqHVs6xZNz0/tGfL7/WeOz4fl3fmUDAAAAMI6WDQAAAMA4WjYAAAAA42jZAAAAAIyjZQMAAAAwjpYNAAAAwDhaNgAAAADjfL96AsBxHMd69QT+zfnqCcB/VzwxZ+cuXfGZq4u0HoDe4Gr0uerplZXT6c6zKPII604b8qiKrFV/31OWWGF+5RJ/6rGhQlr6GaZXnXGVF+A4fqo1prWUf1idb8XiR0B5xjyPcLw1k8fvK8e7qVTN+qsx9jiOenJxefXDlUa3JlJWqQe3Tti5xcI9HV6GeRqN12lj2Zv+YVOve0fl35/u1jPeZ/xaep/CbPfkh4sr/MoGAAAAYBwtGwAAAIBxtGwAAAAAxtGyAQAAABhHywYAAABgHC0bAAAAgHGEfMNTjQ/Puytbcc7CYzDwhsr8D9fzI1uXakuEbY5FrbKuw9jrieAheDpFdDeyXFM89COkBZdf7ayzjFo+Vr1LIR+6jg9vDC5zuI98XcoE8ZwUXg2Ol7zKVE4J2FXln05u9+qkmB/hesWtLvcvnrEVul0NbgYRl6Xj41xOIqR8p7jx4K7XW3pe6jWGdZcXt5XbveX1lq9tKyu8Vfm6Lbnpo42Pdr56e3DF+Gz4P8qvbAAAAADG0bIBAAAAGEfLBgAAAGAcLRsAAACAcbRsAAAAAMbRsgEAAAAYR8sGAAAAYJzvV08APtZ69QT+zXlT3SELHzKN7K79/yzpMpa71xq84Yyb6hZ/yZXLadS1yyKPWLkuUh59pMFndXzd9eXQCtv0E/e6+A9WZ3rpjOUf0uBVTq+a23HUFyC+3OL0Lg9ON85Z7d5Kd1nnrdybXn00XNq6dH0F6quV3PWqOY5j1S+KcMbq8AprKYu03ler+XEWXt8bXuCtD4amxv6P/+cHtN35cPHv/MoGAAAAYBwtGwAAAIBxtGwAAAAAxtGyAQAAABhHywYAAABgHC0bAAAAgHGEfEPDe6bZfXiYdyZFe7sbU7RbQbPXJ9Fey+UA4GZudxpbBc2GWZRFUuUzVHl0MpVX/T1QHflcZvrWMeEhn/gnXcRYpJjeTydMOl/vxt1bh3+HSPDycEwx72hOL5UpLm66iCHuPd0etTKK+9F5EtN3leU8vmL4+vVr0IsxD/dY43WaTlg+XK346jL5+2iGf+ftaGXUtypf19n/2f9qunOXttjyzw82G3/bfA6/sgEAAAAYR8sGAAAAYBwtGwAAAIBxtGwAAAAAxtGyAQAAABhHywYAAABgHC0bAAAAgHH+Azfb0IgKZW5kc3RyZWFtCmVuZG9iago0NyAwIG9iago0MzMzMQplbmRvYmoKMTMgMCBvYmoKPDwgL0JCb3ggWyAtOCAtOCA4IDggXSAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDEzMSAvU3VidHlwZSAvRm9ybQovVHlwZSAvWE9iamVjdCA+PgpzdHJlYW0KeJxtkEEOhCAMRfc9RS/wSUtFZevSa7iZTOL9twNxQEzdNNC+PH5R/pLwTqXA+CQJS06z5HrTkNK6TIwY5tWyKMegUS3WznU4qM/QcGN0i7EUptTW6Hijm+k23pM/+rBZIUY/HA6vhHsWQyZcKTEGh98LL9vD/xGeXtTAH6KNfmNaQ/0KZW5kc3RyZWFtCmVuZG9iagoxNCAwIG9iago8PCAvQkJveCBbIC04IC04IDggOCBdIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTMxIC9TdWJ0eXBlIC9Gb3JtCi9UeXBlIC9YT2JqZWN0ID4+CnN0cmVhbQp4nG2QQQ6EIAxF9z1FL/BJS0Vl69JruJlM4v23A3FATN000L48flH+kvBOpcD4JAlLTrPketOQ0rpMjBjm1bIox6BRLdbOdTioz9BwY3SLsRSm1NboeKOb6Tbekz/6sFkhRj8cDq+EexZDJlwpMQaH3wsv28P/EZ5e1MAfoo1+Y1pD/QplbmRzdHJlYW0KZW5kb2JqCjIgMCBvYmoKPDwgL0NvdW50IDEgL0tpZHMgWyAxMCAwIFIgXSAvVHlwZSAvUGFnZXMgPj4KZW5kb2JqCjQ4IDAgb2JqCjw8IC9DcmVhdGlvbkRhdGUgKEQ6MjAyMjA2MDgyMTEzNDArMDInMDAnKQovQ3JlYXRvciAoTWF0cGxvdGxpYiB2My4zLjIsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcpCi9Qcm9kdWNlciAoTWF0cGxvdGxpYiBwZGYgYmFja2VuZCB2My4zLjIpID4+CmVuZG9iagp4cmVmCjAgNDkKMDAwMDAwMDAwMCA2NTUzNSBmIAowMDAwMDAwMDE2IDAwMDAwIG4gCjAwMDAwNTk5OTUgMDAwMDAgbiAKMDAwMDAxNTU4NyAwMDAwMCBuIAowMDAwMDE1NjQxIDAwMDAwIG4gCjAwMDAwMTU3ODMgMDAwMDAgbiAKMDAwMDAxNTgwNCAwMDAwMCBuIAowMDAwMDE1ODI1IDAwMDAwIG4gCjAwMDAwMDAwNjUgMDAwMDAgbiAKMDAwMDAwMDM5NSAwMDAwMCBuIAowMDAwMDAwMjA4IDAwMDAwIG4gCjAwMDAwMDUwMjcgMDAwMDAgbiAKMDAwMDAxNTkwMiAwMDAwMCBuIAowMDAwMDU5NDg3IDAwMDAwIG4gCjAwMDAwNTk3NDEgMDAwMDAgbiAKMDAwMDAwNTczNiAwMDAwMCBuIAowMDAwMDA1NTI4IDAwMDAwIG4gCjAwMDAwMDUyMTIgMDAwMDAgbiAKMDAwMDAwNjc4OSAwMDAwMCBuIAowMDAwMDA1MDQ4IDAwMDAwIG4gCjAwMDAwMDc4MDAgMDAwMDAgbiAKMDAwMDAwNzYwMCAwMDAwMCBuIAowMDAwMDA3Mjk0IDAwMDAwIG4gCjAwMDAwMDg4NTMgMDAwMDAgbiAKMDAwMDAwNjgyMSAwMDAwMCBuIAowMDAwMDA2OTczIDAwMDAwIG4gCjAwMDAwMTQzMzEgMDAwMDAgbiAKMDAwMDAxNDEzMSAwMDAwMCBuIAowMDAwMDEzNzM3IDAwMDAwIG4gCjAwMDAwMTUzODIgMDAwMDAgbiAKMDAwMDAwODg5OSAwMDAwMCBuIAowMDAwMDA5MjY0IDAwMDAwIG4gCjAwMDAwMDk1ODcgMDAwMDAgbiAKMDAwMDAxMDA5NyAwMDAwMCBuIAowMDAwMDEwNDI1IDAwMDAwIG4gCjAwMDAwMTA3NDQgMDAwMDAgbiAKMDAwMDAxMDg2NCAwMDAwMCBuIAowMDAwMDExMjExIDAwMDAwIG4gCjAwMDAwMTEzNzggMDAwMDAgbiAKMDAwMDAxMTU2NSAwMDAwMCBuIAowMDAwMDExOTE0IDAwMDAwIG4gCjAwMDAwMTIwMjggMDAwMDAgbiAKMDAwMDAxMjUwOSAwMDAwMCBuIAowMDAwMDEyNzIwIDAwMDAwIG4gCjAwMDAwMTI4MDkgMDAwMDAgbiAKMDAwMDAxMzA1MiAwMDAwMCBuIAowMDAwMDEzMzkyIDAwMDAwIG4gCjAwMDAwNTk0NjUgMDAwMDAgbiAKMDAwMDA2MDA1NSAwMDAwMCBuIAp0cmFpbGVyCjw8IC9JbmZvIDQ4IDAgUiAvUm9vdCAxIDAgUiAvU2l6ZSA0OSA+PgpzdGFydHhyZWYKNjAyMTIKJSVFT0YK\n", "image/svg+xml": [ "\n", "\n", "\n", "\n", " \n", " \n", " \n", " \n", " 2022-06-08T21:13:40.515338\n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.3.2, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n" ], "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "def visualize_classification(model, data, label):\n", " data_0 = data[label == 0]\n", " data_1 = data[label == 1]\n", " \n", " fig = plt.figure(figsize=(4,4), dpi=500)\n", " plt.scatter(data_0[:,0], data_0[:,1], edgecolor=\"#333\", label=\"Class 0\")\n", " plt.scatter(data_1[:,0], data_1[:,1], edgecolor=\"#333\", label=\"Class 1\")\n", " plt.title(\"Dataset samples\")\n", " plt.ylabel(r\"$x_2$\")\n", " plt.xlabel(r\"$x_1$\")\n", " plt.legend()\n", " \n", " # Let's make use of a lot of operations we have learned above\n", " c0 = np.array(to_rgba(\"C0\"))\n", " c1 = np.array(to_rgba(\"C1\"))\n", " x1 = jnp.arange(-0.5, 1.5, step=0.01)\n", " x2 = jnp.arange(-0.5, 1.5, step=0.01)\n", " xx1, xx2 = jnp.meshgrid(x1, x2, indexing='ij') # Meshgrid function as in numpy\n", " model_inputs = np.stack([xx1, xx2], axis=-1)\n", " logits = model(model_inputs)\n", " preds = nn.sigmoid(logits)\n", " output_image = (1 - preds) * c0[None,None] + preds * c1[None,None] # Specifying \"None\" in a dimension creates a new one\n", " output_image = jax.device_get(output_image) # Convert to numpy array. This only works for tensors on CPU, hence first push to CPU\n", " plt.imshow(output_image, origin='lower', extent=(-0.5, 1.5, -0.5, 1.5))\n", " plt.grid(False)\n", " return fig\n", "\n", "_ = visualize_classification(trained_model, dataset.data, dataset.label)\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The decision boundaries might not look exactly as in the figure in the preamble of this section, since this has been created with the PyTorch version of the tutorial. Nevertheless, the result on the accuracy metric should be the approximately the same. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Conclusion\n", "\n", "This concludes our tutorial on training a neural network with JAX. While the functional programming perspective of JAX may seem very different to PyTorch at first, it enables a considerable speedup in training, not only for tiny models like here. If you are interested in seeing more practical use cases of JAX, we recommend checking out our other JAX Tutorials, such as:\n", "\n", "* [Tutorial 5 (JAX): Inception, ResNet and DenseNet](https://uvadlc-notebooks.readthedocs.io/en/latest/tutorial_notebooks/JAX/tutorial5/Inception_ResNet_DenseNet.html) gives an intro to training convolutional classifiers on CIFAR10;\n", "* [Tutorial 6 (JAX): Transformers and Multi-Head Attention](https://uvadlc-notebooks.readthedocs.io/en/latest/tutorial_notebooks/JAX/tutorial6/Transformers_and_MHAttention.html) builds a Transformer from scratch with Flax;\n", "* [Tutorial 7 (JAX): Graph Neural Networks](https://uvadlc-notebooks.readthedocs.io/en/latest/tutorial_notebooks/JAX/tutorial7/GNN_overview.html) implements basic Graph Neural Network layers;\n", "* [Tutorial 9 (JAX): Deep Autoencoders](https://uvadlc-notebooks.readthedocs.io/en/latest/tutorial_notebooks/JAX/tutorial9/AE_CIFAR10.html) shows how to train autoencoders on CIFAR10;\n", "* [Tutorial 11 (JAX): Normalizing Flows for image modeling](https://uvadlc-notebooks.readthedocs.io/en/latest/tutorial_notebooks/JAX/tutorial11/NF_image_modeling.html) discusses Normalizing Flows as generative model on images;\n", "* [Tutorial 15 (JAX): Vision Transformers](https://uvadlc-notebooks.readthedocs.io/en/latest/tutorial_notebooks/JAX/tutorial15/Vision_Transformer.html) trains a Transformer on image classification for CIFAR10." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## ✨ The Fancy Bits ✨\n", "\n", "After reading this tutorial, you might wonder why we left out some key advertisement points of JAX: automatic vectorization, easy parallelization on multiple accelerators, etc. The reason why we did not include them in our previous discuss is that for building simple networks, and actual most models in our tutorials, you do not really need these methods. However, since they can be handy at some times, for instance, if you have access to a large cluster or are faced with functions that are annoying to vectorize, we review them here in a separate section: the Fancy Bits of JAX (the title is inspired by JAX's tutorial [🔪 JAX - The Sharp Bits 🔪](https://jax.readthedocs.io/en/latest/notebooks/Common_Gotchas_in_JAX.html))." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Automatic Vectorization with vmap\n", "\n", "In machine learning, we often vectorize methods to efficiently process multiple inputs or batch elements at the same time. Usually, we have to write the code ourselves to support additional dimensions to vectorize over. However, since JAX can already transform functions to run efficiently on accelerators or calculate gradients, it can also automatically vectorize a function. For instance, let's consider a simple linear layer where we write a function for a single input `x` of shape `[c_in]`, a weight matrix `[c_in, c_out]`, and a bias vector `[c_out]`: " ] }, { "cell_type": "code", "execution_count": 61, "metadata": {}, "outputs": [], "source": [ "def simple_linear(x, w, b):\n", " # We could already vectorize this function with matmul, but as an example,\n", " # let us use a non-vectorized function with same output\n", " return (x[:,None] * w).sum(axis=0) + b" ] }, { "cell_type": "code", "execution_count": 62, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "DeviceArray([-0.5393317, 1.4906642, 0.7108946], dtype=float32)" ] }, "execution_count": 62, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Example inputs\n", "rng, x_rng, w_rng, b_rng = jax.random.split(rng, 4)\n", "x_in = jax.random.normal(x_rng, (4,))\n", "w_in = jax.random.normal(w_rng, (4, 3))\n", "b_in = jax.random.normal(b_rng, (3,))\n", "\n", "simple_linear(x_in, w_in, b_in)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now, we would like the function to support a batch dimension on `x`, i.e. `[batch, c_in]`. Our naive implementation above does not support this, since we specialized the axis we sum over. So, let's make JAX do the work for us and vectorize the function by using `jax.vmap`:" ] }, { "cell_type": "code", "execution_count": 63, "metadata": {}, "outputs": [], "source": [ "vectorized_linear = jax.vmap(simple_linear,\n", " in_axes=(0, None, None), # Which axes to vectorize for each input\n", " out_axes=0 # Which axes to map to in the output\n", " )" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Specifying `None` for the in-axes of the input arguments `w` and `b` means that we do not want to vectorize any of their input dimensions. With this vmap specification, the function `vectorized_linear` now supports an extra batch dimension in `x`! Let's try it out:" ] }, { "cell_type": "code", "execution_count": 64, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "DeviceArray([[-0.5393317, 1.4906642, 0.7108946],\n", " [-0.5393317, 1.4906642, 0.7108946],\n", " [-0.5393317, 1.4906642, 0.7108946],\n", " [-0.5393317, 1.4906642, 0.7108946],\n", " [-0.5393317, 1.4906642, 0.7108946]], dtype=float32)" ] }, "execution_count": 64, "metadata": {}, "output_type": "execute_result" } ], "source": [ "x_vec_in = jnp.stack([x_in]*5, axis=0)\n", "\n", "vectorized_linear(x_vec_in, w_in, b_in)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The new function indeed vectorized our code, calculating $N$ applications of the weights and bias to the input. We can also vectorize the code to run multiple inputs `x` on multiple weights `w` and biases `b` by changing the input argument `in_axes` to `(0, 0, 0)`, or simply `0`. Morever, we can again stack multiple function transformations, such as jitting a vectorized function. Further details on `jax.vmap` can be found in this [tutorial](\n", "https://jax.readthedocs.io/en/latest/jax-101/03-vectorization.html\n", ") and its [documentation](https://jax.readthedocs.io/en/latest/_autosummary/jax.vmap.html?highlight=vmap)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Parallel evaluation with pmap\n", "\n", "`jax.vmap` vectorizes a function on a single accelerator. But what if we have multiple GPUs or TPUs available? In PyTorch, we can parallelize a model over multiple GPUs using `nn.DistributedDataParallel`. In JAX, this is yet another function transformation: `jax.pmap`. Similar to `jax.vmap`, we can specify over which axes each input should be parallelized. In a network training, we usually want to parallelize over an extra batch dimension in the data, while the parameters are identical for all devices. For more details on `jax.pmap`, see [Parallel Evaluation in JAX](https://jax.readthedocs.io/en/latest/jax-101/06-parallelism.html)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Working with PyTrees\n", "\n", "Network parameters in Flax are stored in a PyTree. We have visited them before, but what we haven't discuss yet is JAX's utilities to operate on pytrees! One common application is to obtain a list of all parameters in the network. This corresponds to extracting all leafs from a PyTree, for which JAX provides the function `jax.tree_leaves`:" ] }, { "cell_type": "code", "execution_count": 65, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "We have parameters with the following shapes: (8,), (2, 8), (1,), (8, 1)\n", "Overall parameter count: 33\n" ] } ], "source": [ "parameters = jax.tree_leaves(model_state.params)\n", "print('We have parameters with the following shapes:', ', '.join([str(p.shape) for p in parameters]))\n", "print('Overall parameter count:', sum([np.prod(p.shape) for p in parameters]))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can also create new PyTrees that are the result of applying a function on all elements in the tree using `jax.tree_map`. For instance, let's obtain a PyTree with all parameter shapes:" ] }, { "cell_type": "code", "execution_count": 66, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "FrozenDict({\n", " params: {\n", " linear1: {\n", " bias: (8,),\n", " kernel: (2, 8),\n", " },\n", " linear2: {\n", " bias: (1,),\n", " kernel: (8, 1),\n", " },\n", " },\n", "})" ] }, "execution_count": 66, "metadata": {}, "output_type": "execute_result" } ], "source": [ "jax.tree_map(lambda p: p.shape, model_state.params)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The nodes of PyTrees do not necessarily need to be NumPy or JAX arrays, but can be arbitrary objects. Overall, PyTrees provide a simple, structured representation of data useful in many applications. More details can be found in JAX's Tutorial [Working with PyTrees](https://jax.readthedocs.io/en/latest/jax-101/05.1-pytrees.html)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 🔪 The Sharp Bits 🔪\n", "\n", "Since JAX functions need to be written with certain constraints, there are situations where this can get annoying or difficult. A great overview of those, why they are needed, and most importantly, what to do about them, can be found in the original JAX tutorial [🔪 JAX - The Sharp Bits 🔪](https://jax.readthedocs.io/en/latest/notebooks/Common_Gotchas_in_JAX.html). In this final section of the tutorial, we want to visit a few of those points we have not explicitly discussed yet. Furthermore, we also focus on the combination with Flax, and what can be annoying when training models." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Dynamic shapes\n", "\n", "JAX has the great advantage of providing just-in-time compilation of functions to speed up the computation. For this, it uses its intermediate representation jaxpr, which is specialized to the shapes of the input arguments. However, this also means that a jitted function is specialized to a certain shape, and running the jitted function with a different input shape requires recompiling the function. For instance, consider the following simple function:" ] }, { "cell_type": "code", "execution_count": 67, "metadata": {}, "outputs": [], "source": [ "def my_function(x):\n", " print('Running the function with shape', x.shape)\n", " return x.mean()\n", "\n", "jitted_function = jax.jit(my_function)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The print statement is only executed once when the function is compiled, and for all consecutive function calls, this print statement will be ignored since it is not part of the jaxpr representation. Let's run the function now with multiple different input shapes:" ] }, { "cell_type": "code", "execution_count": 68, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Running the function with shape (1,)\n", "Running the function with shape (2,)\n", "Running the function with shape (3,)\n", "Running the function with shape (4,)\n", "Running the function with shape (5,)\n", "Running the function with shape (6,)\n", "Running the function with shape (7,)\n", "Running the function with shape (8,)\n", "Running the function with shape (9,)\n", "Running the function with shape (10,)\n" ] } ], "source": [ "for i in range(10):\n", " jitted_function(jnp.zeros(i+1,))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As we can see, the function is compiled for every different input we give it. This can become inefficient if we actually work with many different shapes. However, running the function again with one of the previous input shapes will not require another compilation:" ] }, { "cell_type": "code", "execution_count": 69, "metadata": {}, "outputs": [], "source": [ "# Running the functions a second time will not print out anything since\n", "# the functions are already jitted for the respective input shapes.\n", "for i in range(10):\n", " jitted_function(jnp.zeros(i+1,))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If we have a very limited set of different shapes, we do not see a big performance difference. For instance, in our evaluation, the last batch size is smaller than the previous since we have a limited size of the evaluation dataset. However, for other applications, we might encounter this problem much more often: NLP and time series, and graphs. In these cases, it is recommend to pad the batches to prevent many re-compilations (see Flax's [padding guide](https://flax.readthedocs.io/en/latest/howtos/full_eval.html) for details). We briefly review the two scenarios below.\n", "\n", "#### NLP and time series\n", "\n", "In Natural Language Processing, our data consist of sentences which greatly vary in size. Already for batching the elements, we need to apply padding, such that the shape of the batch is determined by the largest sentence in the batch. However, this largest length can vary between batches, especially when we shuffle the dataset before each epoch. In PyTorch, this is not a problem, since the dynamic computation graph allows us to stop the computation whenever we need to. In contrast, JAX would need to recompile the forward pass for every single largest sentence length, which can quickly become very expensive. Padding is needed to reduce the number of compilations, but at the same time introduces unnecessary computation. Hence, we have a tradeoff between number of compilations and extra compute per batch. In the extreme case, PyTorch may even become faster than JAX here.\n", "\n", "#### Graphs\n", "\n", "Similar to NLP, graphs can vary in their size. Often, graphs differ in their number of nodes, but especially in the number of edges. Furthermore, when we start batching the graphs, the variation of node sizes and edge count considerably increases. Again, padding is needed to reduce the number of compilations, and we will revisit this topic in Tutorial 7 (TBD)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Debugging in jitted functions\n", "\n", "During coding, we likely want to debug our model and sometimes print out intermediate values. In JAX, when jitting functions, this is not as straightforward. As we could see from the previous cells, a print statement is only executed once during compilation, and afterwards removed since it is not part of the jaxpr representation. Furthermore, there can be issues when tracking NaNs in your code (see the [sharp bits tutorial](https://jax.readthedocs.io/en/latest/notebooks/Common_Gotchas_in_JAX.html#debugging-nans)), and errors like out-of-bounds indexing are silently handled on accelerators by returning -1 instead of an error (see the corresponding section in the [sharp bits tutorial](https://jax.readthedocs.io/en/latest/notebooks/Common_Gotchas_in_JAX.html#out-of-bounds-indexing)). However, if necessary, one can either run the unjitted version of the forward pass first, and even introduce print statements to the jitted version where needed (see [here](https://github.com/google/jax/issues/196) for a great explanation). Still, it is not as straight-forward as in PyTorch, for example." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Modules with different train and evaluation functions\n", "\n", "Certain deep learning layers have different behaviors under evaluation than during training. For instance, dropout randomly masks out a number of neurons during training, but leaves the graph untouched during evaluation. In PyTorch, we can easily switch between the two states via `model.train()` and `model.eval()` without having to manually specify it in the dropout module instance. However, in JAX, we do not have global states in the model, and thus need to pass this information to every module in the forward pass that may need it. In our example above, this was not needed, since the forward pass of the simple classifier is identical during training and evaluation. Alternatively, since the parameters are not bound to a specific model during training, one can also create two models: one training model, and one evaluation model. Nonetheless, one still needs to add the information to every module with changing behaviors, which adds a certain overhead compared to PyTorch. We will discuss two common modules with such behaviors below: dropout and BatchNorm." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Dropout\n", "\n", "In Flax, [Dropout](https://flax.readthedocs.io/en/latest/_autosummary/flax.linen.Dropout.html) has an argument `deterministic` which turns off dropout when True, and otherwise applies the random masking as intended during training. This deterministic switch can either be defined in the constructor, or in every forward call. Furthermore, dropout has the special case that it is a random operation during training, meaning that it also needs a PRNG state. Fortunately, we do not have to pass this state in every PRNG state, but instead, can simply pass `rngs={'dropout': dropout_rng}` with `dropout_rng` being the PRNG state. For an example, see our [Tutorial 6](https://uvadlc-notebooks.readthedocs.io/en/latest/tutorial_notebooks/JAX/tutorial6/Transformers_and_MHAttention.html) on Transformers in which we use dropout in several occasions." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### BatchNorm\n", "\n", "Batch Normalization transforms the input in two different ways. During training, we determine the mean and standard deviation of the input, and normalize the data with it to a zero mean and standard deviation of one. During evaluation, on the other hand, we take a running statistic over the previous several batches, and use those to estimate the mean and standard deviation. This is necessary to keep the evaluation stable and invariant to the specific batches we choose. Still, in the Flax module ([documentation](https://flax.readthedocs.io/en/latest/_autosummary/flax.linen.BatchNorm.html)), we need to give the argument `use_running_average` (bool) to either the constructor or each forward pass. Furthermore, BatchNorm has a specific property we haven't discussed yet and is a bit tricky in JAX: keeping track of the running average. During every forward pass, we want to record the mean and standard deviation of the current batch, and update our current average over the past batches. However, this requires changing an input tensor, and returning this changed tensor again. In Flax, we can do this by defining the batch statistics as a mutable tensor. Check out our [Tutorial 5](https://uvadlc-notebooks.readthedocs.io/en/latest/tutorial_notebooks/JAX/tutorial5/Inception_ResNet_DenseNet.html) to see BatchNorm being used in practice with Flax." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "[![Star our repository](https://img.shields.io/static/v1.svg?logo=star&label=⭐&message=Star%20Our%20Repository&color=yellow)](https://github.com/phlippe/uvadlc_notebooks/) If you found this tutorial helpful, consider ⭐-ing our repository. \n", "[![Ask questions](https://img.shields.io/static/v1.svg?logo=star&label=❔&message=Ask%20Questions&color=9cf)](https://github.com/phlippe/uvadlc_notebooks/issues) For any questions, typos, or bugs that you found, please raise an issue on GitHub. \n", "\n", "---" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.2" } }, "nbformat": 4, "nbformat_minor": 4 }