1

I am facing a problem regarding Optimization of Hyperparameters in a Convolutional Neural Network for the analysis of text data. First, I will explain my process so far: With the help of various excellent Blog-Posts I was able to build a CNN that works for my project. In my project I am trying to predict the VIX and S&P 500 with the help of the FOMC meeting statements. So basically I habe text data on the one hand and financial data (returns) on the other hand. After preprocessing and applying Googles Word2Vec pre-trained Word-Embeddings I built the following Convolutional Network:

def ConvNet(embeddings, max_sequence_length, num_words, embedding_dim, trainable=False, extra_conv=True, lr=0.001, dropout=0.5): embedding_layer = Embedding(num_words, embedding_dim, weights=[embeddings], input_length=max_sequence_length, trainable=trainable) sequence_input = Input(shape=(max_sequence_length,), dtype='int32') embedded_sequences = embedding_layer(sequence_input) convs = [] filter_sizes = [3, 4, 5] for filter_size in filter_sizes: l_conv = Conv1D(filters=128, kernel_size=filter_size, activation='relu')(embedded_sequences) l_pool = MaxPooling1D(pool_size=3)(l_conv) convs.append(l_pool) l_merge = concatenate([convs[0], convs[1], convs[2]], axis=1) # add a 1D convnet with global maxpooling, instead of Yoon Kim model conv = Conv1D(filters=128, kernel_size=3, activation='relu')(embedded_sequences) pool = MaxPooling1D(pool_size=3)(conv) if extra_conv == True: x = Dropout(dropout)(l_merge) else: # Original Yoon Kim model x = Dropout(dropout)(pool) x = Flatten()(x) x = Dense(128, activation='relu')(x) preds = Dense(1, activation='linear')(x) model = Model(sequence_input, preds) sgd = SGD(learning_rate = lr, momentum= 0.8) model.compile(loss='mean_squared_error', optimizer= sgd, metrics=['mean_squared_error']) model.summary() return model model = ConvNet(train_embedding_weights, MAX_SEQUENCE_LENGTH, len(train_word_index)+1, EMBEDDING_DIM, False) #define callbacks early_stopping = EarlyStopping(monitor='val_loss', min_delta=0.01, patience=4, verbose=1) callbacks_list = [early_stopping] hist = model.fit(x_train, y_tr, epochs=5, batch_size=33, validation_split=0.2, shuffle=True, callbacks=callbacks_list)

My model architecture looks like this:

__________________________________________________________________________________________________ Layer (type) Output Shape Param # Connected to ================================================================================================== input_1 (InputLayer) (None, 1086) 0 __________________________________________________________________________________________________ embedding_1 (Embedding) (None, 1086, 300) 532500 input_1[0][0] __________________________________________________________________________________________________ conv1d_1 (Conv1D) (None, 1084, 128) 115328 embedding_1[0][0] __________________________________________________________________________________________________ conv1d_2 (Conv1D) (None, 1083, 128) 153728 embedding_1[0][0] __________________________________________________________________________________________________ conv1d_3 (Conv1D) (None, 1082, 128) 192128 embedding_1[0][0] __________________________________________________________________________________________________ max_pooling1d_1 (MaxPooling1D) (None, 361, 128) 0 conv1d_1[0][0] __________________________________________________________________________________________________ max_pooling1d_2 (MaxPooling1D) (None, 361, 128) 0 conv1d_2[0][0] __________________________________________________________________________________________________ max_pooling1d_3 (MaxPooling1D) (None, 360, 128) 0 conv1d_3[0][0] __________________________________________________________________________________________________ concatenate_1 (Concatenate) (None, 1082, 128) 0 max_pooling1d_1[0][0] max_pooling1d_2[0][0] max_pooling1d_3[0][0] __________________________________________________________________________________________________ dropout_2 (Dropout) (None, 1082, 128) 0 concatenate_1[0][0] __________________________________________________________________________________________________ flatten_1 (Flatten) (None, 138496) 0 dropout_2[0][0] __________________________________________________________________________________________________ dense_3 (Dense) (None, 128) 17727616 flatten_1[0][0] __________________________________________________________________________________________________ dense_4 (Dense) (None, 1) 129 dense_3[0][0] ================================================================================================== Total params: 18,721,429 Trainable params: 18,188,929 Non-trainable params: 532,500

Model architecture: enter image description here

So, now I am facing the next big problem, and I am really running out of ideas how to solve is: Optimization of hyperparameters

So my specific question is, how to perform the Optimization of hyperparameters?

My search code is: 

from hyperopt import fmin, hp, tpe, space_eval, Trials def train_and_score(args): # Train the model the fixed params plus the optimization args. # Note that this method should return the final History object. test = ConvNet(embeddings=train_embedding_weights, max_sequence_length= MAX_SEQUENCE_LENGTH, num_words=len(train_word_index)+1, embedding_dim= EMBEDDING_DIM, trainable=False, extra_conv=True, lr=args['lr'], dropout=args['dropout']) # Unpack and return the last validation loss from the history. return test['val_loss'][-1] # Define the space to optimize over. space = { 'lr': hp.loguniform('lr', np.log(0.01), np.log(0.1)), 'dropout': hp.uniform('dropout', 0, 0.5) } # Minimize the training score over the space. trials = Trials() best = fmin(fn=train_and_score, space=space, trials=trials, algo=tpe.suggest, max_evals=100) # Print details about the best results and hyperparameters. print(best) print(space_eval(space, best))

The specific error message is: 

__________________________________________________________________________________________________ 0%| | 0/100 [00:00<?, ?trial/s, best loss=?] job exception: 'Model' object is not subscriptable Traceback (most recent call last): File "/Users/lukaskoston/Desktop/MasterarbeitFOMCAnalysis/07_Regression/CNN regression neu.py", line 262, in <module> max_evals=100) File "/Users/lukaskoston/.local/lib/python3.7/site-packages/hyperopt/fmin.py", line 482, in fmin show_progressbar=show_progressbar, File "/Users/lukaskoston/.local/lib/python3.7/site-packages/hyperopt/base.py", line 686, in fmin show_progressbar=show_progressbar, File "/Users/lukaskoston/.local/lib/python3.7/site-packages/hyperopt/fmin.py", line 509, in fmin rval.exhaust() File "/Users/lukaskoston/.local/lib/python3.7/site-packages/hyperopt/fmin.py", line 330, in exhaust self.run(self.max_evals - n_done, block_until_done=self.asynchronous) File "/Users/lukaskoston/.local/lib/python3.7/site-packages/hyperopt/fmin.py", line 286, in run self.serial_evaluate() File "/Users/lukaskoston/.local/lib/python3.7/site-packages/hyperopt/fmin.py", line 165, in serial_evaluate result = self.domain.evaluate(spec, ctrl) File "/Users/lukaskoston/.local/lib/python3.7/site-packages/hyperopt/base.py", line 894, in evaluate rval = self.fn(pyll_rval) File "/Users/lukaskoston/Desktop/MasterarbeitFOMCAnalysis/07_Regression/CNN regression neu.py", line 248, in train_and_score return hist['val_loss'][-1] TypeError: 'Model' object is not subscriptable

Thanks in advance, Lukas

Improve this question
2
  • Where is your training/parameter search code? It's unclear to me how this error is related to hyperparameter optimization Commented May 7, 2020 at 17:31
  • I added it, Thanks, I think now it should be more clear Commented May 7, 2020 at 17:44

1 Answer 1

Reset to default
1

The error that you're getting is because you're trying to directly subset a model which is not directly subsettable like a list or a dictionary.

Your ConvNet function defines and compiles a model but it does not train it or evaluate it. You'll need to run model.fit() to train it and store the output of the training history like you did in the first script you posted hist = model.fit(...). You can then change the return statement of train_and_score to something like return hist.history['val_loss'][-1].

I would start by taking your first bit of training code 

early_stopping = EarlyStopping(monitor='val_loss', min_delta=0.01, patience=4, verbose=1) callbacks_list = [early_stopping] hist = model.fit(x_train, y_tr, epochs=5, batch_size=33, validation_split=0.2, shuffle=True, callbacks=callbacks_list)

and add it to your train_and_score function after the model definition. And then change the return statement.

Improve this answer
Sign up to request clarification or add additional context in comments.

9 Comments

Thanks for your fast response, I really appreciate that! I am really new to Python and Neural Networks, I don't get exactly what I have to change.. I don't know if that's too much to ask, but would you be so kind and show me what I have to change in the code?
I added a bit more explanation. In order to test or score a model, you must first train it. You already seem to know how to use *.fit, you just need to use that code to train your model during every call of test_and_score. Does it make sense now?
Yes indeed it helped, It is running now. I just don't unterstand, why I should change the return statement of train_and_score to something like return hist.history['val_loss'][-1] ? Could you explain that further? Thanks a lot!
What you had originally test['val_loss'][-1] is attempting to do a dictionary-like hash lookup in the object 'test' of the string 'val_loss'. test is an object of type tf.keras.Model and does not have data stored in a way that can be accessed like that. The error you received said it like this: TypeError: 'Model' object is not subscriptable. On the other hand, the output of model.fit() is a record of training loss values and metrics.
look at returns here tensorflow.org/api_docs/python/tf/keras/Model#fit
|

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.