Building a basic neural network
A simple task for a simple model
Our first artificial neural network will be simple, and it therefore needs a simple task to be trained on. We will train it to classify numbers between 0 and 1 as being either less than 0.5 (category 0) or equal to, or higher than, 0.5 (category 1). Phrased differently, we're going to teach the network to round numbers between 0 and 1.
Generating the data
We first generate an array of 10,000 random numbers (between 0 and 1), which we will use as the training data (or: input) for the model; that is, each training observation consists of a single number.
np.random.random(10000) returns a one-dimensional array of shape
(10000,). However, we need a two-dimensional array for our model. Specifically, the first axis should correspond to the individual observations, and should therefore be of length 10,000. The second axis should correspond to the values from each individual observation, and should therefore be of length 1. We can accomplish this by explicitly changing the shape of
import numpy as np data = np.random.random(10000) data.shape = 10000, 1 print(data[:5]) # Look at the first five observations
[[0.64670172] [0.6240171 ] [0.06398892] [0.27453739] [0.93943499]]
Next, we check which numbers in the training data are larger than, or equal to, 0.5. This results in an array of boolean values (
False), which we turn into an array of
0) values. This array will serve as the labels (or: expected output) for our model.
labels = np.array(data >= .5, dtype=int) print(labels[:5]) # Look at the first five labels
[    ]
Building a simple model
Sequential model class allows you to specify a model's architecture as a
list (or: sequence, hence the name) of layers. We specify two layers, both of which are
Dense, which means that each neuron in the layer is connected to all neurons in the layer above.
The first (non-input) layer, which consists of 8 neurons, specifies the shape of the input observations, which in our case is only a single number (i.e., an array of shape
(1, )). The
activation keyword specifies how the neurons translate their input to an output. Here we use the rectified-linear-unit function ('relu'), which essentially means that neurons cannot have a negative output.
The second layer, which consists of two neurons, is our output layer. The 'softmax' activation function essentially means that the neuron's output is squashed into the 0 - 1 range.
It is common to think of the input as a layer as well, even though it is not specified as such in Keras. We can therefore think of our model as a three-layer network, with one input layer (consisting of one neuron), one hidden layer (consisting of eight neurons), and one output layer (consisting of two neurons). A layer is referred to as 'hidden' if it's neither an input layer nor an output layer.
from keras import Sequential from keras.layers import Dense model = Sequential([ Dense(units=8, input_shape=(1,), activation='relu'), Dense(units=2, activation='softmax'), ])
Next, we compile the model so that it's ready to be trained. In doing so, we need to specify three things.
optimizer keyword specifies the algorithm that will be used to adjust the weights in the model. We will use Adam, but other algorithms are available as well, and the choice of one algorithm over another is largely one of preference and experience.
loss keyword specifies the loss function, which is the algorithm that determines how wrong the model's predictions are. The goal of training is to reduce the loss. We will use sparse categorical crossentropy, which assumes that label values of 0 mean that the first neuron in the output layer should be the most active, whereas label values of 1 mean that the second neuron should be the most active.
accuracy keyword specifies a metric that is printed during training so that you have some idea of how well the model is performing. This is just for visualization and doesn't affect the training process itself.
model.compile( optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'] )
Now let's look at the model summary:
Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= dense_2 (Dense) (None, 8) 16 _________________________________________________________________ dense_3 (Dense) (None, 2) 18 ================================================================= Total params: 34 Trainable params: 34 Non-trainable params: 0 _________________________________________________________________
Our model has 34 parameters, which correspond to all the weights of the connections between the neurons, as well as the bias parameter for each neuron (a kind of baseline level of activity that a neuron has in the absence of any input).
The first layer consists of eight neurons, which each have one connection to the input neuron plus one bias, so 8 × (1 + 1) = 16 parameters in total. The second layer consists of two neurons, which each have one connection all eight neurons in the preceding layer plus one bias, so 2 × (8 + 1) = 18 parameters in total.
Training the model
To train the model, we simply call the
fit() function. During training, the model's 34 parameters are adjusted to minimize the loss.
epochs keyword specifies how often the training data should be used to fit the model. In our case, we specify 10 epochs, which is similar to calling
fit() 10 times in a row with the same data and labels.
verbose keyword specifies how much information should be printed out during training.
model.fit(x=data, y=labels, epochs=10, verbose=2)
Epoch 1/10 313/313 - 1s - loss: 0.5330 - accuracy: 0.8056 Epoch 2/10 313/313 - 0s - loss: 0.3126 - accuracy: 0.9600 Epoch 3/10 313/313 - 0s - loss: 0.1968 - accuracy: 0.9855 Epoch 4/10 313/313 - 0s - loss: 0.1442 - accuracy: 0.9924 Epoch 5/10 313/313 - 0s - loss: 0.1157 - accuracy: 0.9940 Epoch 6/10 313/313 - 0s - loss: 0.0979 - accuracy: 0.9951 Epoch 7/10 313/313 - 0s - loss: 0.0855 - accuracy: 0.9950 Epoch 8/10 313/313 - 0s - loss: 0.0764 - accuracy: 0.9945 Epoch 9/10 313/313 - 0s - loss: 0.0692 - accuracy: 0.9976 Epoch 10/10 313/313 - 0s - loss: 0.0634 - accuracy: 0.9971
As you can see, during training, the loss goes down while the accuracy goes up until it reaches almost 100%. This means that the model has learned our simple task!
Testing the model
The true test of our model's performance is whether it is able to classify numbers that it wasn't trained on. In our case, we trained the model on no less than 10,000 randomly selected numbers, which means that our model has seen pretty much every number there is to see in the 0 - 1 range. Nevertheless, it is good practice to create a separate test set. So let's create a new array consisting of ten numbers:
test_set = np.array([0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9])
Now let's look at the model's predictions for our test set:
predictions = model.predict(test_set) print(predictions)
[[9.9989772e-01 1.0223002e-04] [9.9971110e-01 2.8888174e-04] [9.9918395e-01 8.1604376e-04] [9.9607420e-01 3.9257705e-03] [9.4276649e-01 5.7233561e-02] [5.1676810e-01 4.8323190e-01] [6.4919546e-02 9.3508047e-01] [4.4870418e-03 9.9551290e-01] [3.6606033e-04 9.9963391e-01] [6.7188019e-05 9.9993277e-01]]
What are we looking at here? The predictions are a 10 by 2 array, where the first axis corresponds to each of our ten test observations, and the second axis refers to the activity of the two neurons in the output layer.
The predictions are easier to interpret if we look at which of the two neurons was most active for each of the ten test observations. We can do this using
np.argmax() which returns the index of the highest element in the array.
[0 0 0 0 0 0 1 1 1 1]
Now we can easily tell that for the first five test observations (0, 0.1, 0.2, 0.3, 0.4), the first output neuron (0) was most active. This means that the network correctly classified these numbers as being less than 0.5. For the last five test observations (0.5, 0.6, 0.7, 0.8, 0.9), the second output neuron (1) was most active. This again means that the network correctly classified these numbers, this time as being equal to, or larger than, 0.5.
Validating the model during training
As mentioned above, it's important to verify a model's predictions using data that the model wasn't actually trained on. This is important because models are prone to overfitting, which occurs when a model has learned the specifics of the training data but is still unable to make (correct) predictions for new observations.
We will look at this in more detail in a later tutorial, but for now let's introduce the concept of validation data. By specifying the
validation_split keyword, we can tell the model to set apart some of the data (10% in the example blow) as validation data. This data will not be used for training, and you will see a separate accuracy metric (
val_accuracy) for this subset of the data.
We need to start with a fresh, untrained model. So let's first create a new model, copying the code above:
model = Sequential([ Dense(units=8, input_shape=(1,), activation='relu'), Dense(units=2, activation='softmax'), ]) model.compile( optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'] )
And now train our fresh model, this time setting 10% of the data apart for validation!
model.fit(x=data, y=labels, epochs=10, verbose=2, validation_split=.1)
Epoch 1/10 282/282 - 1s - loss: 0.5118 - accuracy: 0.7937 - val_loss: 0.3991 - val_accuracy: 0.9220 Epoch 2/10 282/282 - 1s - loss: 0.3085 - accuracy: 0.9547 - val_loss: 0.2421 - val_accuracy: 0.9490 Epoch 3/10 282/282 - 0s - loss: 0.1963 - accuracy: 0.9776 - val_loss: 0.1641 - val_accuracy: 1.0000 Epoch 4/10 282/282 - 0s - loss: 0.1432 - accuracy: 0.9894 - val_loss: 0.1265 - val_accuracy: 0.9920 Epoch 5/10 282/282 - 0s - loss: 0.1146 - accuracy: 0.9929 - val_loss: 0.1044 - val_accuracy: 1.0000 Epoch 6/10 282/282 - 0s - loss: 0.0970 - accuracy: 0.9958 - val_loss: 0.0898 - val_accuracy: 1.0000 Epoch 7/10 282/282 - 0s - loss: 0.0847 - accuracy: 0.9966 - val_loss: 0.0794 - val_accuracy: 0.9950 Epoch 8/10 282/282 - 0s - loss: 0.0758 - accuracy: 0.9966 - val_loss: 0.0715 - val_accuracy: 0.9950 Epoch 9/10 282/282 - 0s - loss: 0.0688 - accuracy: 0.9973 - val_loss: 0.0668 - val_accuracy: 0.9900 Epoch 10/10 282/282 - 0s - loss: 0.0636 - accuracy: 0.9953 - val_loss: 0.0603 - val_accuracy: 0.9950
In this case, the validation accuracy is just as high as the regular accuracy, which means that our model does not suffer from overfitting!