Skip to content

Improve training/evaluation pipeline for SubRep neural skill generator - #47

Open
miruts-code wants to merge 5 commits into
iCog-Labs-Dev:mainfrom
miruts-code:improve-training-pipeline
Open

miruts-code wants to merge 5 commits into
iCog-Labs-Dev:mainfrom
miruts-code:improve-training-pipeline

Conversation

@miruts-code

Copy link
Copy Markdown
  • Added a real train/val/test split (previously training and evaluation used the same data with no held-out set); split is persisted to a manifest so evaluation always reuses the exact split from training instead of recomputing it.
  • Training now runs only on the train split; validation loss is tracked each epoch and used for model selection (best-validation checkpoint, not just the final epoch) and early stopping.
  • Replaced hardcoded hyperparameters (batch size, epochs, learning rate, hidden dim, seed, split ratios) with CLI arguments.
  • Added mid-training checkpointing, so a crash mid-run doesn't lose all progress.
  • Added explicit device selection (GPU if available, else CPU).
  • Extended evaluation beyond raw loss: added a certification-focused report covering candidate success/admission rate, rejection reasons, payoff/motive improvement over baseline, comparison against the non-neural candidate policies, and generalization to seeds never used in training -- reusing the existing certification and baseline logic unmodified.
  • Updated README.md, generator/README.md, and data/README.md to reflect the above

Description

What does this PR do? (e.g., "Implements 2-head MLP forward pass")

Checklist

  • Code follows project style guidelines
  • Tests added/updated (tests/test_*.py)
  • README updated (if interface changed)
  • Verified locally (all tests pass)

Testing Notes

How should the reviewer test this?

- Added a real train/val/test split (previously training and evaluation
  used the same data with no held-out set); split is persisted to a
  manifest so evaluation always reuses the exact split from training
  instead of recomputing it.
- Training now runs only on the train split; validation loss is tracked
  each epoch and used for model selection (best-validation checkpoint,
  not just the final epoch) and early stopping.
- Replaced hardcoded hyperparameters (batch size, epochs, learning rate,
  hidden dim, seed, split ratios) with CLI arguments.
- Added mid-training checkpointing, so a crash mid-run doesn't lose all
  progress.
- Added explicit device selection (GPU if available, else CPU).
- Extended evaluation beyond raw loss: added a certification-focused
  report covering candidate success/admission rate, rejection reasons,
  payoff/motive improvement over baseline, comparison against the
  non-neural candidate policies, and generalization to seeds never used
  in training -- reusing the existing certification and baseline logic unmodified.
- Updated README.md, generator/README.md, and data/README.md to reflect the above
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant