Alright so I rewrote some parts of your model such that it makes more sense for a classification problem. The first and most obvious reason your network was not working is due to the number of output nodes you selected. For a classification task the number of output nodes should be the same as the number of classes in your data. In this case we have 5 kinds of flowers, thus 5 labels which I reassigned to $y \in \{0, 1, 2, 3, 4\}$, thus we will have 5 output nodes.

So let's go through the code. First we bring the data into the notebook using the code you wrote.

from os import listdir import cv2 daisy_path = "flowers/daisy/" dandelion_path = "flowers/dandelion/" rose_path = "flowers/rose/" sunflower_path = "flowers/sunflower/" tulip_path = "flowers/tulip/" def iter_images(images,directory,size,label): try: for i in range(len(images)): img = cv2.imread(directory + images[i]) img = cv2.resize(img,size) img_data.append(img) labels.append(label) except: pass img_data = [] labels = [] size = 64,64 iter_images(listdir(daisy_path),daisy_path,size,0) iter_images(listdir(dandelion_path),dandelion_path,size,1) iter_images(listdir(rose_path),rose_path,size,2) iter_images(listdir(sunflower_path),sunflower_path,size,3) iter_images(listdir(tulip_path),tulip_path,size,4) 

We can visualize the data to get a better idea of the distribution of the classes.

import matplotlib.pyplot as plt %matplotlib inline n_classes = 5 training_counts = [None] * n_classes testing_counts = [None] * n_classes for i in range(n_classes): training_counts[i] = len(y_train[y_train == i])/len(y_train) testing_counts[i] = len(y_test[y_test == i])/len(y_test) # the histogram of the data train_bar = plt.bar(np.arange(n_classes)-0.2, training_counts, align='center', color = 'r', alpha=0.75, width = 0.41, label='Training') test_bar = plt.bar(np.arange(n_classes)+0.2, testing_counts, align='center', color = 'b', alpha=0.75, width = 0.41, label = 'Testing') plt.xlabel('Labels') plt.xticks((0,1,2,3,4)) plt.ylabel('Count (%)') plt.title('Label distribution in the training and test set') plt.legend(bbox_to_anchor=(1.05, 1), handles=[train_bar, test_bar], loc=2) plt.grid(True) plt.show() 

We will now transform the data and the labels to matrices.

import numpy as np data = np.array(img_data) data.shape data = data.astype('float32') / 255.0 labels = np.asarray(labels) 

Then we will split the data.. Notice that you do not need to shuffle the data yourself since sklearn can do it for you.

from sklearn.model_selection import train_test_split # Split the data x_train, x_test, y_train, y_test = train_test_split(data, labels, test_size=0.33, shuffle= True) 

Let's construct our model. I changed the last layer to use the softmax activation function. This will allow the outputs of the network to sum up to a total probability of 1. This is the usual activation function to use for classification tasks.

from keras.models import Sequential from keras.layers import Dense,Flatten,Convolution2D,MaxPool2D from __future__ import print_function import keras from keras.datasets import mnist from keras.models import Sequential from keras.layers import Dense, Dropout, Flatten from keras.layers import Conv2D, MaxPooling2D from keras.callbacks import ModelCheckpoint from keras.models import model_from_json from keras import backend as K model = Sequential() model.add(Convolution2D(32, (3,3),input_shape=(64, 64, 3),activation='relu')) model.add(MaxPool2D(pool_size=(2,2))) model.add(Flatten()) model.add(Dense(128,activation='relu')) model.add(Dense(5,activation='softmax')) model.compile(loss=keras.losses.categorical_crossentropy, optimizer=keras.optimizers.Adadelta(), metrics=['accuracy']) 

Then we can train our network. This will result in about 60% accuracy on the test set. This is pretty good considering the baseline for this task is 20%.

batch_size = 128 epochs = 10 model.fit(x_train, y_train_binary, batch_size=batch_size, epochs=epochs, verbose=1, validation_data=(x_test, y_test_binary)) 

After the model is trained you can predict instances using. Don't forget that the network needs to take the same shape in. Thus we must maintain the dimensionality of the matrix, that's why I use the [0:1].

print('Predict the classes: ') prediction = model.predict_classes(x_test[0:1]) print('Predicted class: ', prediction) print('Real class: ', y_test[0:1]) 

This gives

Predict the classes: 1/1
[==============================] - 0s 6ms/step
Predicted class: [4]
Real class: [4]

Some suggestions

The model you are currently using is the one that is most common for MNIST. However, that data only has a single channel thus we don't need as many layers. You can increase the performance by increasing the complexity of your model. Or by reducing the complexity of your data, for example you can train using the grayscale equivalent of the images, thus reducing the problem to a single channel.