This repository was archived by the owner on Jun 19, 2025. It is now read-only.
Closed
Conversation
No Taskcluster jobs started for this pull requestThe `allowPullRequests` configuration for this repository (in `.taskcluster.yml` on the default branch) does not allow starting tasks for this pull request. |
Collaborator
| @DanBmh Thanks, can you elaborate a little bit? My mind is kind of somewhere else, so I'm unsure I get the point here. |
Contributor Author
| Currently training looks like this: The old approach did still work well but I think we can make it even better by reloading the weights from the best_dev checkpoint. |
Contributor Author
| Not ready yet! |
DanBmh changed the title Contributor Author
| Working again:) |
DanBmh changed the title 
lissyx approved these changes Aug 18, 2020
Collaborator
lissyx left a comment
lissyx left a comment There was a problem hiding this comment.
LGTM, I'd like Reuben's opinion.
reuben suggested changes Aug 19, 2020
Comment on lines +648 to +649
| # Reload checkpoint that we use the best_dev weights again | ||
| load_or_init_graph_for_training(session, allow_drop_layers=False) |
Contributor
There was a problem hiding this comment.
Please keep this function unchanged and add a new explicitly load_best_checkpoint function, we don't want this call to load last silently for example.
reuben approved these changes Aug 19, 2020
Contributor
| Re-opened as #3261 to run tests. |
reuben added a commit that referenced this pull request Aug 20, 2020
Tests #3245 Reload weights after plateau
Contributor
| Merged in #3261 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reload checkpoint weights after reaching a plateau that we use the best_dev weights again