Skip to content

Dimred methods preprocessing - #301

Merged
dburkhardt merged 10 commits into
openproblems-bio:masterfrom
lazappi:dimred-methods-preprocessing
Apr 7, 2021
Merged

Dimred methods preprocessing#301
dburkhardt merged 10 commits into
openproblems-bio:masterfrom
lazappi:dimred-methods-preprocessing

Conversation

@lazappi

@lazappi lazappi commented Mar 31, 2021

Copy link
Copy Markdown
Member

Submission type

Take 2 at adding standard pre-processing to the dimensionality (2D) task, replacing #289.

This submission modifies the dimensionality reduction (2D) task to include a script with standard pre-processing functions for use by methods (See #279). These should store a transformed expression matrix in adata.X and a higher dimensional embedding in adata.obsm['X_input'] which is then used as the input for (most) methods.

  • Add a preprocessing.py function that has pre-processing functions (one currently)
  • Added that pre-processing function to the existing methods
    • Created a new variant of PHATE that uses the standard pre-processing
  • Updated the README to describe the pre-processing step

Testing

  • This submission was written on a forked copy of SingleCellOpenProblems
  • GitHub Actions "Run Benchmark" tests are passing on this base branch of this pull request (include link to passed test: ) PENDING
  • If this pull request is not ready for review (including passing the "Run Benchmark" tests), I will open this PR as a draft (click on the down arrow next to the "Create Pull Request" button)

Submission guidelines

  • This submission follows the guidelines in our Contributing document
  • I have checked to ensure there aren't other open Pull Requests for the same update/change

lazappi and others added 5 commits March 31, 2021 14:02
@lazappi
Add preprocessing.py for dim red methods
96b3483 
This file should contain standard pre-processing functions used to
create variations on different methods
@lazappi
Add pre-processing function existing dim red methods
8fda5eb 
Create a new PHATE variant using Scanpy pre-processing
@lazappi
Update dim red README to describe pre-processing
df178d4
@lazappi
Merge branch 'master' into dimred-methods-preprocessing
ec2bf87 
* master:
  Update alra.py
  Update alra.R
  pre-commit
  Create alra.R
  pre-commit
  Update alra.py
  Create alra.py
  Delete alra.py
  Create alra.py
@github-actions
pre-commit
2f4181c
@lazappi

lazappi commented Mar 31, 2021 •
edited
Loading

Copy link
Copy Markdown
Member Author

Hmmm...looks like the actions are failing on my repo because phate isn't available https://github.com/lazappi/SingleCellOpenProblems/runs/2237020126?check_suite_focus=true. Not sure why that is because I haven't modified any of the containers but I'll see if I can fix it before making any updates here.

@lazappi

lazappi commented Mar 31, 2021

Copy link
Copy Markdown
Member Author

Ok, think I have fixed the PHATE issue (now I understand how things are run). Seems to be running but I'll wait for it to finish before I update here.

@dburkhardt
dburkhardt self-requested a review April 5, 2021 18:59
dburkhardt
dburkhardt approved these changes Apr 5, 2021
View reviewed changes

@dburkhardt dburkhardt left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, I like this contribution. We might like to in the long term think about a way to do more comparisons between available preprocessing methods as described in #310

@dburkhardt dburkhardt mentioned this pull request Apr 5, 2021
Closed
@dburkhardt
dburkhardt merged commit 2b2cc67 into openproblems-bio:master Apr 7, 2021
lazappi added a commit to lazappi/openproblems that referenced this pull request Apr 16, 2021
@lazappi
Merge branch 'master' into dimred-datasets
790c167 
* master:
  fix region
  fix nf wkdir
  use env.BRANCH
  rm echo
  BRANCH -> WKDIR for s3
  set branch variable in S3 setup job
  Fix s3 bucket clash
  Dimred methods preprocessing (openproblems-bio#301)
  Add trustworthiness score for dimred task (openproblems-bio#258)
  Add Method SCOT to Multi-modal data Integration Task (openproblems-bio#298)
  Update README.md
  Update README.md
  Update README.md
lazappi added a commit to lazappi/openproblems that referenced this pull request Apr 16, 2021
@lazappi
Merge branch 'master' into dimred-methods
042ce11 
* master:
  fix region
  fix nf wkdir
  use env.BRANCH
  rm echo
  BRANCH -> WKDIR for s3
  set branch variable in S3 setup job
  Fix s3 bucket clash
  Dimred methods preprocessing (openproblems-bio#301)
  Add trustworthiness score for dimred task (openproblems-bio#258)
  Add Method SCOT to Multi-modal data Integration Task (openproblems-bio#298)
  Update README.md
  Update README.md
  Update README.md
lazappi added a commit to lazappi/openproblems that referenced this pull request Apr 16, 2021
@lazappi
Merge branch 'master' into dimred-metrics
6f72c75 
* master:
  fix region
  fix nf wkdir
  use env.BRANCH
  rm echo
  BRANCH -> WKDIR for s3
  set branch variable in S3 setup job
  Fix s3 bucket clash
  Dimred methods preprocessing (openproblems-bio#301)
  Add trustworthiness score for dimred task (openproblems-bio#258)
  Add Method SCOT to Multi-modal data Integration Task (openproblems-bio#298)
  Update README.md
  Update README.md
  Update README.md
@lazappi lazappi mentioned this pull request Apr 19, 2021
Closed
lazappi added a commit to lazappi/openproblems that referenced this pull request May 12, 2021
@lazappi
Merge branch 'dimred-methods' into method-densmap-dimensionality-redu…
9718e93 
…ction

* dimred-methods: (34 commits)
  Update alra.py (openproblems-bio#304)
  updated template for PR with PR evaluation checks (openproblems-bio#314)
  pre-commit
  Fix preprocessing
  Fix preprocessing
  Add new preprocessing
  Add new preprocessing
  fix region
  fix nf wkdir
  use env.BRANCH
  rm echo
  BRANCH -> WKDIR for s3
  set branch variable in S3 setup job
  Fix s3 bucket clash
  Dimred methods preprocessing (openproblems-bio#301)
  Add trustworthiness score for dimred task (openproblems-bio#258)
  Add Method SCOT to Multi-modal data Integration Task (openproblems-bio#298)
  Change batch size to 1k cells for aff. matrix
  Add preprocessing
  Fix some small bugs
  ...
@lazappi
lazappi deleted the dimred-methods-preprocessing branch April 11, 2022 08:48
rcannood pushed a commit that referenced this pull request Sep 4, 2024
Kai Waldrant
Update process datasets wf (#301)
cc1fd42 
* Add nf-tower cli for dataset loader

* add mising directive labels for dataset loader

* add missing directive labels process datasets

* remove space in file name

* update s3 bucket

* increase yaml limit to 5mb

* Fix dataset schema validation and remove
unnecessary code to fix meta  file size

* Update dataset schema file path in config.vsh.yaml
and main.nf

* Add script for processing datasets on nf-tower
 in bat_int

* Remove dataset_schema input from config.vsh.yaml

* Add output_task_info to workflow configuration

* Update publish directory in process_datasets.sh for bat_int

* Update denoising process_datasets wf

* Add dimensionality reduction dataset processing
script and configuration files for nf-tower

* Add label projection nf-tower scripts

* add nf-tower scripts match_modalities

* Add nf-tower scripts and configuration files for
processing and running benchmark in predict_modality workflow

* Add dataset info fields to benchmark API files
rcannood pushed a commit that referenced this pull request Sep 4, 2024
Kai Waldrant
Update process datasets wf (#301)
f4cc583 
* Add nf-tower cli for dataset loader

* add mising directive labels for dataset loader

* add missing directive labels process datasets

* remove space in file name

* update s3 bucket

* increase yaml limit to 5mb

* Fix dataset schema validation and remove
unnecessary code to fix meta  file size

* Update dataset schema file path in config.vsh.yaml
and main.nf

* Add script for processing datasets on nf-tower
 in bat_int

* Remove dataset_schema input from config.vsh.yaml

* Add output_task_info to workflow configuration

* Update publish directory in process_datasets.sh for bat_int

* Update denoising process_datasets wf

* Add dimensionality reduction dataset processing
script and configuration files for nf-tower

* Add label projection nf-tower scripts

* add nf-tower scripts match_modalities

* Add nf-tower scripts and configuration files for
processing and running benchmark in predict_modality workflow

* Add dataset info fields to benchmark API files

Former-commit-id: cc1fd42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dburkhardt dburkhardt dburkhardt approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@lazappi @dburkhardt