Improve training/evaluation pipeline for SubRep neural skill generator - #47
Open
miruts-code wants to merge 5 commits into
Open
miruts-code wants to merge 5 commits into
miruts-code wants to merge 5 commits into
Conversation
- Added a real train/val/test split (previously training and evaluation used the same data with no held-out set); split is persisted to a manifest so evaluation always reuses the exact split from training instead of recomputing it. - Training now runs only on the train split; validation loss is tracked each epoch and used for model selection (best-validation checkpoint, not just the final epoch) and early stopping. - Replaced hardcoded hyperparameters (batch size, epochs, learning rate, hidden dim, seed, split ratios) with CLI arguments. - Added mid-training checkpointing, so a crash mid-run doesn't lose all progress. - Added explicit device selection (GPU if available, else CPU). - Extended evaluation beyond raw loss: added a certification-focused report covering candidate success/admission rate, rejection reasons, payoff/motive improvement over baseline, comparison against the non-neural candidate policies, and generalization to seeds never used in training -- reusing the existing certification and baseline logic unmodified. - Updated README.md, generator/README.md, and data/README.md to reflect the above
… by giving the argument a default value.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
What does this PR do? (e.g., "Implements 2-head MLP forward pass")
Checklist
tests/test_*.py)Testing Notes
How should the reviewer test this?