Vfrdtyky

I want to fine tuning my L2 parameter in my last keras layer using a for loop approach. My target is build a Extreme Machine Learning model. Now, I'm using the code below:

#possible values for L2... 

k = 2^(seq(-20,-1,1)) 



#vectors with metrics 

acc_vector = vector('numeric',length(k)) 

loss_vector = vector('numeric',length(k))



for(i in seq_along(k)){



model0 = keras_model_sequential() %>% 

    layer_dense(units = 500,activation = 'relu',input_shape = c(784),

                trainable = F,name = 'dense1')  %>% 

    layer_dense(units = 10, activation = 'softmax',

                kernel_regularizer = regularizer_l2(k[i]),name='dense2') %>% 

    compile(loss = 'categorical_crossentropy',optimizer = optimizer_rmsprop(),

            metrics = c('accuracy'))



model0 %>% fit(

  x_train, y_train, 

  epochs = 5, batch_size = 512, 

  validation_split = 0.2,verbose=0)



eval = model0 %>% evaluate(x_test, y_test)



acc_vector[i] = eval$acc

loss_vector[i] = eval$loss



#I don't know why, but without the next 2 lines, my memory usage increase 2 times

rm(model0,eval)

gc()

}

So, here is my problem. With this aproach (run fast, at least), my weights start by random in each loop and the value of L2 doesn't make any sense. I tried other approachs like include weights = "weights" in the first layer and worked fine, except by the process time... it increased a lot! After this, I tried to pop the last layer and add a new layer with the new L2 as follow:

model0 = keras_model_sequential() %>% 

    layer_dense(units = 500,activation = 'relu',input_shape = c(784),

                trainable = F,name = 'dense1')  %>% 

    layer_dense(units = 10, activation = 'softmax',

                kernel_regularizer = regularizer_l2('any value'),name='dense2')



for(i in seq_along(k)){

  model0 %>% pop_layer() %>% 

  layer_dense(units = 10, activation = 'softmax',

              kernel_regularizer = regularizer_l2(k[i]),name='dense2')

}

But doesn't work. The behavior of the last approach makes the models have just the first layer. I just want change the value of L2 to retrain the last layer of my model. How can I do that in a simple way?