As suggested by the title, is it possible to fix some part of the neural network while training?
Since Mathematica provides a way to extract part of a neural network and combine it with some layers to make a new one:
newNet = NetChain[{Take[oldNet, 3], 10, Ramp, 10}] It would be very helpful to fix the layers taken from the old network. In this way one can reuse the neural network and investigate how transferrable are features in the neural network (c.f. https://arxiv.org/abs/1411.1792).
The LearningRateMultipliers supports using different learning rates with different layers. When setting the learning rate of a particular layer to None, the weight of that layer will be fixed during training.
You can have a look at the example in the documentation of LearningRateMultipliers.
In that example, net is constructed from part of the network trained for MNIST
net = NetChain[Join[Drop[Normal[lenet], -2], {LinearLayer[], SoftmaxLayer[]}], "Input" -> enc, "Output" -> dec] Then at the training time, only the newly added linear layer is allowed to change. All the other layers are fixed by setting the learning rate to None.
trained = NetTrain[net, trainingData, MaxTrainingRounds -> Quantity[10, "Seconds"], LearningRateMultipliers -> {-2 -> 1, _ -> None}, ValidationSet -> Scaled[0.1]] 
LearningRateMultipliers -> {-2 -> 1, _ -> None} and not LearningRateMultipliers -> { _ -> None, -2->1}? Aren't you undoing the first LearningRateMultipliers assigment? How can one transfer some layers without their trained values? $\endgroup$ {-2 -> 1, _ -> None} because the rules are matched with the order. If you use { _ -> None, -2->1} then the first rule is always matched so that the learning rate is 0 for all layers. $\endgroup$ NeuralNetworks`NetDeinitialize to remove the weights. See more here. $\endgroup$
LearningRateMultiplierswhat you are looking for? And here are examples of transfer learning. $\endgroup$