Optimize ui lint pre-commits - #51725
Conversation
Running pre-commits for typescript compilation, linting formatting is currently very slow. The pre-commit that is doing it took more than 2.5 minutes - always, independently which files changed. Also, no matter whether UI or simple auth manager UI changed, both were compiled. This is the first iteration of speeding up the pre-commit especially for incremental changes. The following changes were implemented: * the scripts were split into ui and simple-auth-manager-ui * timing information and verbose commands displayed so that we can measure which commands take how much time * the pre-commmit now passes all the files as input (and pre-commit is serialized) - which means that some commands can only be run on the subset of files that changed * tsc step is modified to only run on a subset of .ts and .tsx files that were changed (via dynamic extension of the project) Overall when just few files change, the pre-commit should now complete in less than 10-20 seconds Still some optimisations are possible in the future.
|
We could - potentially already refactor the code in both pre-commits - because they are essentially the same (minus directory) - but I think maybe coupling those two might not make sense eventually - we might want to have different steps in both (but happy to hear your opinion). |
Backport failed to create: v3-0-test. View the failure log Run details
You can attempt to backport this manually by running: cherry_picker 52be0dd v3-0-testThis should apply the commit to the v3-0-test branch and leave the commit in conflict state marking After you have resolved the conflicts, you can continue the backport process by running: cherry_picker --continue |
Running pre-commits for typescript compilation, linting formatting is currently very slow. The pre-commit that is doing it took more than 2.5 minutes - always, independently which files changed. Also, no matter whether UI or simple auth manager UI changed, both were compiled. This is the first iteration of speeding up the pre-commit especially for incremental changes. The following changes were implemented: * the scripts were split into ui and simple-auth-manager-ui * timing information and verbose commands displayed so that we can measure which commands take how much time * the pre-commmit now passes all the files as input (and pre-commit is serialized) - which means that some commands can only be run on the subset of files that changed * tsc step is modified to only run on a subset of .ts and .tsx files that were changed (via dynamic extension of the project) Overall when just few files change, the pre-commit should now complete in less than 10-20 seconds Still some optimisations are possible in the future. (cherry picked from commit 52be0dd) Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Running pre-commits for typescript compilation, linting formatting is currently very slow. The pre-commit that is doing it took more than 2.5 minutes - always, independently which files changed. Also, no matter whether UI or simple auth manager UI changed, both were compiled. This is the first iteration of speeding up the pre-commit especially for incremental changes. The following changes were implemented: * the scripts were split into ui and simple-auth-manager-ui * timing information and verbose commands displayed so that we can measure which commands take how much time * the pre-commmit now passes all the files as input (and pre-commit is serialized) - which means that some commands can only be run on the subset of files that changed * tsc step is modified to only run on a subset of .ts and .tsx files that were changed (via dynamic extension of the project) Overall when just few files change, the pre-commit should now complete in less than 10-20 seconds Still some optimisations are possible in the future. (cherry picked from commit 52be0dd)
|
Yeah - it should take seconds now for translators :) |
|
Thank you! I've been meaning to look into why it was so slow. |
You are most welcome .... Also It would be good to see if it works "perfectly" - I think I got all the types of files in and trigger the right things when the "source" ones change - but I might have missed "some" edge cases file types. But it should be easy to update the pre-commits to fix any changes so feel free in case you see anythign missed. We could also potentially add a manual pre-commit similar to what was before to retrigger it "for sure" and without passing the files to pre-commit in case we want to be absolutely sure. We had similar setup for |
apache#68919 changed _private_ui.yaml without regenerating the TypeScript client and CI stayed green: selective checks decide whether to skip the ts-compile-lint-* hooks from file groups that do not cover the UI codegen input specs, while the hooks themselves trigger on any yaml under the openapi directories. The stale client then masked a type error that broke main for every PR running full static checks. The same gap applied to v2-simple-auth-manager-generated.yaml, so a spec-only change also skipped the simple auth manager UI hook. Track the codegen-input specs in their own file group so spec-only changes run the UI hooks without forcing the full test matrix, and drop the main UI hook trigger for the simple auth manager spec: that pattern was copied in when the hook was created (apache#51725), a month after the spec was renamed v1 to v2 (apache#50705), so it has never matched a file - apache#53636 fixed the sibling hook's copy but missed this one. The main UI codegen does not consume that spec either.
#68919 changed _private_ui.yaml without regenerating the TypeScript client and CI stayed green: selective checks decide whether to skip the ts-compile-lint-* hooks from file groups that do not cover the UI codegen input specs, while the hooks themselves trigger on any yaml under the openapi directories. The stale client then masked a type error that broke main for every PR running full static checks. The same gap applied to v2-simple-auth-manager-generated.yaml, so a spec-only change also skipped the simple auth manager UI hook. Track the codegen-input specs in their own file group so spec-only changes run the UI hooks without forcing the full test matrix, and drop the main UI hook trigger for the simple auth manager spec: that pattern was copied in when the hook was created (#51725), a month after the spec was renamed v1 to v2 (#50705), so it has never matched a file - #53636 fixed the sibling hook's copy but missed this one. The main UI codegen does not consume that spec either.
…nges (#70643) #68919 changed _private_ui.yaml without regenerating the TypeScript client and CI stayed green: selective checks decide whether to skip the ts-compile-lint-* hooks from file groups that do not cover the UI codegen input specs, while the hooks themselves trigger on any yaml under the openapi directories. The stale client then masked a type error that broke main for every PR running full static checks. The same gap applied to v2-simple-auth-manager-generated.yaml, so a spec-only change also skipped the simple auth manager UI hook. Track the codegen-input specs in their own file group so spec-only changes run the UI hooks without forcing the full test matrix, and drop the main UI hook trigger for the simple auth manager spec: that pattern was copied in when the hook was created (#51725), a month after the spec was renamed v1 to v2 (#50705), so it has never matched a file - #53636 fixed the sibling hook's copy but missed this one. The main UI codegen does not consume that spec either. (cherry picked from commit b43a87e) Co-authored-by: Shahar Epstein <60007259+shahar1@users.noreply.github.com>
…nges (#70643) #68919 changed _private_ui.yaml without regenerating the TypeScript client and CI stayed green: selective checks decide whether to skip the ts-compile-lint-* hooks from file groups that do not cover the UI codegen input specs, while the hooks themselves trigger on any yaml under the openapi directories. The stale client then masked a type error that broke main for every PR running full static checks. The same gap applied to v2-simple-auth-manager-generated.yaml, so a spec-only change also skipped the simple auth manager UI hook. Track the codegen-input specs in their own file group so spec-only changes run the UI hooks without forcing the full test matrix, and drop the main UI hook trigger for the simple auth manager spec: that pattern was copied in when the hook was created (#51725), a month after the spec was renamed v1 to v2 (#50705), so it has never matched a file - #53636 fixed the sibling hook's copy but missed this one. The main UI codegen does not consume that spec either. (cherry picked from commit b43a87e) Co-authored-by: Shahar Epstein <60007259+shahar1@users.noreply.github.com>
…nges (#70643) #68919 changed _private_ui.yaml without regenerating the TypeScript client and CI stayed green: selective checks decide whether to skip the ts-compile-lint-* hooks from file groups that do not cover the UI codegen input specs, while the hooks themselves trigger on any yaml under the openapi directories. The stale client then masked a type error that broke main for every PR running full static checks. The same gap applied to v2-simple-auth-manager-generated.yaml, so a spec-only change also skipped the simple auth manager UI hook. Track the codegen-input specs in their own file group so spec-only changes run the UI hooks without forcing the full test matrix, and drop the main UI hook trigger for the simple auth manager spec: that pattern was copied in when the hook was created (#51725), a month after the spec was renamed v1 to v2 (#50705), so it has never matched a file - #53636 fixed the sibling hook's copy but missed this one. The main UI codegen does not consume that spec either. (cherry picked from commit b43a87e) Co-authored-by: Shahar Epstein <60007259+shahar1@users.noreply.github.com>
…nges (#70643) #68919 changed _private_ui.yaml without regenerating the TypeScript client and CI stayed green: selective checks decide whether to skip the ts-compile-lint-* hooks from file groups that do not cover the UI codegen input specs, while the hooks themselves trigger on any yaml under the openapi directories. The stale client then masked a type error that broke main for every PR running full static checks. The same gap applied to v2-simple-auth-manager-generated.yaml, so a spec-only change also skipped the simple auth manager UI hook. Track the codegen-input specs in their own file group so spec-only changes run the UI hooks without forcing the full test matrix, and drop the main UI hook trigger for the simple auth manager spec: that pattern was copied in when the hook was created (#51725), a month after the spec was renamed v1 to v2 (#50705), so it has never matched a file - #53636 fixed the sibling hook's copy but missed this one. The main UI codegen does not consume that spec either. (cherry picked from commit b43a87e) Co-authored-by: Shahar Epstein <60007259+shahar1@users.noreply.github.com>
…#70643) apache#68919 changed _private_ui.yaml without regenerating the TypeScript client and CI stayed green: selective checks decide whether to skip the ts-compile-lint-* hooks from file groups that do not cover the UI codegen input specs, while the hooks themselves trigger on any yaml under the openapi directories. The stale client then masked a type error that broke main for every PR running full static checks. The same gap applied to v2-simple-auth-manager-generated.yaml, so a spec-only change also skipped the simple auth manager UI hook. Track the codegen-input specs in their own file group so spec-only changes run the UI hooks without forcing the full test matrix, and drop the main UI hook trigger for the simple auth manager spec: that pattern was copied in when the hook was created (apache#51725), a month after the spec was renamed v1 to v2 (apache#50705), so it has never matched a file - apache#53636 fixed the sibling hook's copy but missed this one. The main UI codegen does not consume that spec either.
Running pre-commits for typescript compilation, linting formatting is currently very slow. The pre-commit that is doing it took more than 2.5 minutes - always, independently which files changed. Also, no matter whether UI or simple auth manager UI changed, both were compiled.
This is the first iteration of speeding up the pre-commit especially for incremental changes.
The following changes were implemented:
Overall when just few files change, the pre-commit should now complete in less than 10-20 seconds
Still some optimisations are possible in the future.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in airflow-core/newsfragments.