Verify that your code is bug free

For the love of all that is good, scale your data

Crawl Before You Walk; Walk Before You Run

Neural Network Training Is Like Lock Picking

Non-convex optimization is hard

Regularization

Keep a Logbook of Experiments

1. Verify that your code is bug free

2. For the love of all that is good, scale your data

3. Crawl Before You Walk; Walk Before You Run

4. Neural Network Training Is Like Lock Picking

5. Non-convex optimization is hard

6. Regularization

7. Keep a Logbook of Experiments

To achieve state of the art, or even merely good, results, you have to have to have set up all of the parts configured to work well together. Setting up a neural network configuration that actually learns is a lot like picking a lock: all of the pieces have to be lined up just right. Just as it is not sufficient to have a single tumbler in the right place, neither is it sufficient to have only the architecture, or only the optimizer, set up correctly.

To achieve state of the art, or even merely good, results, you have to set up all of the parts configured to work well together. Setting up a neural network configuration that actually learns is a lot like picking a lock: all of the pieces have to be lined up just right. Just as it is not sufficient to have a single tumbler in the right place, neither is it sufficient to have only the architecture, or only the optimizer, set up correctly.

• Variables are created but never used (usually because of copy-paste errors);
• Expressions for gradient updates are incorrect;
• Weight updates are not applied;
• Loss functions are not measured on the correct scale (for example, cross-entropy loss can be expressed in terms of probability or logits)
• The loss is not appropriate for the task (for example, using categorical cross-entropy loss for a regression task).
• Dropout is used during testing, instead of only being used for training.
• Make sure you're minimizing the loss function $L(x)$, instead of minimizing $-L(x)$.

• Variables are created but never used (usually because of copy-paste errors);
• Expressions for gradient updates are incorrect;
• Weight updates are not applied;
• Loss functions are not measured on the correct scale (for example, cross-entropy loss can be expressed in terms of probability or logits)
• The loss is not appropriate for the task (for example, using categorical cross-entropy loss for a regression task).
• Dropout is used during testing, instead of only being used for training.
• Make sure you're minimizing the loss function $L(x)$, instead of minimizing $-L(x)$.
• Make sure your loss is computed correctly.