Skip to content

[Build] Fix skipping tests when there are only documentation changes and run centos7 & windows cpp build only when there are cpp changes #10376

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
codelipenghui merged 2 commits into from
Apr 28, 2021
Merged

[Build] Fix skipping tests when there are only documentation changes and run centos7 & windows cpp build only when there are cpp changes #10376

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
339a1e7
Fix skipping tests for documentation changes
lhotari Apr 26, 2021
e28ceb5
Move filters to .github/changes-filter.yaml file
lhotari Apr 28, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions .github/changes-filter.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# contains pattern definitions used in workflows "changes" step
# pattern syntax: https://github.com/micromatch/picomatch
all:
- '**'
docs:
- 'site2/**'
- 'deployment/**'
- '.asf.yaml'
- '*.md'
- '**/*.md'
16 changes: 14 additions & 2 deletions .github/workflows/ci-build-macos.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,11 +22,9 @@ on:
pull_request:
branches:
- master
paths-ignore: ['site2/**', 'deployment/**']

@lhotari lhotari Apr 28, 2021

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

paths-ignore would be fine here, but it's removed to be consistent with other workflows in apache/pulsar where paths-filter action (copy of https://github.com/dorny/paths-filter) is used.

GitHub provides a paths-ignore feature out of the box.
However, GitHub's paths-ignore has some limitations:

  • GitHub's paths-ignore fails to look at previous commits. This means that the outcome depends on how often you push changes.
  • Consequently, GitHub's paths-ignore does not work for required checks.
    If you path-ignore a required check, then pull requests will block forever without being mergeable.

(source)

paths-ignore was once used in apache/pulsar (introduced by #9527), but it had to be reverted in #9606 because of the limitations.

push:
branches:
- branch-*
paths-ignore: ['site2/**', 'deployment/**']

env:
MAVEN_OPTS: -Dhttp.keepAlive=false -Dmaven.wagon.http.pool=false -Dmaven.wagon.http.retryHandler.class=standard -Dmaven.wagon.http.retryHandler.count=3
Expand All @@ -45,7 +43,18 @@ jobs:
- name: Tune Runner VM
uses: ./.github/actions/tune-runner-vm

- name: Detect changed files
id: changes
uses: apache/pulsar-test-infra/paths-filter@master
with:
filters: .github/changes-filter.yaml

- name: Check changed files
id: check_changes
run: echo "::set-output name=docs_only::${{ fromJSON(steps.changes.outputs.all_count) == fromJSON(steps.changes.outputs.docs_count) && fromJSON(steps.changes.outputs.docs_count) > 0 }}"

- name: Cache Maven dependencies
if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
uses: actions/cache@v2
with:
path: |
Expand All @@ -56,12 +65,15 @@ jobs:
${{ runner.os }}-m2-dependencies-all-

- name: Set up JDK 1.8
if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
uses: actions/setup-java@v1
with:
java-version: 1.8

- name: Replace maven's wagon-http version
if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
run: sudo ./build/replace_maven-wagon-http-version.sh

- name: build package
if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
run: mvn -B clean install -DskipTests
28 changes: 14 additions & 14 deletions .github/workflows/ci-cpp-build-centos7.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,10 +22,15 @@ on:
pull_request:
branches:
- master
paths:
- '.github/workflows/**'
- 'pulsar-client-cpp/**'
push:
branches:
- branch-*

paths:
- '.github/workflows/**'
- 'pulsar-client-cpp/**'
jobs:

cpp-build-centos7:
Expand All @@ -40,23 +45,18 @@ jobs:
- name: Tune Runner VM
uses: ./.github/actions/tune-runner-vm

- name: Changed files check
- name: Detect changed files
id: changes
uses: apache/pulsar-test-infra/paths-filter@master
with:
filters: |
# pattern syntax: https://github.com/micromatch/picomatch
all:
- 'pulsar-client-cpp/**'
docs:
- 'site2/**'
- 'deployment/**'
- '.asf.yaml'
- '*.md'
- '**/*.md'
filters: .github/changes-filter.yaml

- name: Check changed files
id: check_changes
run: echo "::set-output name=docs_only::${{ fromJSON(steps.changes.outputs.all_count) == fromJSON(steps.changes.outputs.docs_count) && fromJSON(steps.changes.outputs.docs_count) > 0 }}"

- name: build cpp client on centos 7
if: ${{ steps.changes.outputs.all_count }} > ${{ steps.changes.outputs.docs_count }}
if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
run: |
echo "Build C++ client library on CentOS 7"
pulsar-client-cpp/docker-build-centos7.sh
pulsar-client-cpp/docker-build-centos7.sh
21 changes: 20 additions & 1 deletion .github/workflows/ci-cpp-build-windows.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,15 @@ on:
pull_request:
branches:
- master
paths:
- '.github/workflows/**'
- 'pulsar-client-cpp/**'
push:
branches:
- branch-*
paths:
- '.github/workflows/**'
- 'pulsar-client-cpp/**'

env:
VCPKG_FEATURE_FLAGS: manifests
Expand All @@ -51,12 +57,24 @@ jobs:
- name: checkout
uses: actions/checkout@v2

- name: Detect changed files
id: changes
uses: apache/pulsar-test-infra/paths-filter@master
with:
filters: .github/changes-filter.yaml

- name: Check changed files
id: check_changes
run: echo "::set-output name=docs_only::${{ fromJSON(steps.changes.outputs.all_count) == fromJSON(steps.changes.outputs.docs_count) && fromJSON(steps.changes.outputs.docs_count) > 0 }}"

- name: Install vcpkg packages
if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
shell: bash
run: |
cd pulsar-client-cpp && vcpkg install --triplet ${{ matrix.triplet }}

- name: Configure
if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
shell: bash
run: |
if [ "$RUNNER_OS" == "Windows" ]; then
Expand All @@ -71,9 +89,10 @@ jobs:
fi

- name: Compile
if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
shell: bash
run: |
if [ "$RUNNER_OS" == "Windows" ]; then
cd pulsar-client-cpp && \
cmake --build ./build
fi
fi
31 changes: 13 additions & 18 deletions .github/workflows/ci-cpp.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,23 +43,18 @@ jobs:
- name: Tune Runner VM
uses: ./.github/actions/tune-runner-vm

- name: Changed files check
- name: Detect changed files
id: changes
uses: apache/pulsar-test-infra/paths-filter@master
with:
filters: |
# pattern syntax: https://github.com/micromatch/picomatch
all:
- '**'
docs:
- 'site2/**'
- 'deployment/**'
- '.asf.yaml'
- '*.md'
- '**/*.md'
filters: .github/changes-filter.yaml

- name: Check changed files
id: check_changes
run: echo "::set-output name=docs_only::${{ fromJSON(steps.changes.outputs.all_count) == fromJSON(steps.changes.outputs.docs_count) && fromJSON(steps.changes.outputs.docs_count) > 0 }}"

- name: Cache local Maven repository
if: ${{ steps.changes.outputs.all_count }} > ${{ steps.changes.outputs.docs_count }}
if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
uses: actions/cache@v2
with:
path: |
Expand All @@ -71,12 +66,12 @@ jobs:

- name: Set up JDK 1.8
uses: actions/setup-java@v1
if: ${{ steps.changes.outputs.all_count }} > ${{ steps.changes.outputs.docs_count }}
if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
with:
java-version: 1.8

- name: clean disk
if: ${{ steps.changes.outputs.all_count }} > ${{ steps.changes.outputs.docs_count }}
if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
run: |
sudo swapoff -a
sudo rm -f /swapfile
Expand All @@ -85,22 +80,22 @@ jobs:
df -h

- name: Replace maven's wagon-http version
if: ${{ steps.changes.outputs.all_count }} > ${{ steps.changes.outputs.docs_count }}
if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
run: sudo ./build/replace_maven-wagon-http-version.sh

- name: build package
if: ${{ steps.changes.outputs.all_count }} > ${{ steps.changes.outputs.docs_count }}
if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
run: mvn -B -ntp -q install -Pcore-modules,-main -DskipTests

- name: build cpp artifacts
if: ${{ steps.changes.outputs.all_count }} > ${{ steps.changes.outputs.docs_count }}
if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
run: |
echo "Build C++ client library"
export CMAKE_ARGS="-DCMAKE_BUILD_TYPE=Debug -DBUILD_DYNAMIC_LIB=OFF -DPYTHON_INCLUDE_DIR=/usr/include/python2.7 -DPYTHON_LIBRARY=/usr/lib/x86_64-linux-gnu/libpython2.7.so"
pulsar-client-cpp/docker-build.sh

- name: run c++ tests
if: ${{ steps.changes.outputs.all_count }} > ${{ steps.changes.outputs.docs_count }}
if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
run: pulsar-client-cpp/docker-tests.sh

- name: Upload test-logs
Expand Down
25 changes: 10 additions & 15 deletions .github/workflows/ci-go-functions-style.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,43 +51,38 @@ jobs:
- name: Tune Runner VM
uses: ./.github/actions/tune-runner-vm

- name: Changed files check
- name: Detect changed files
id: changes
uses: apache/pulsar-test-infra/paths-filter@master
with:
filters: |
# pattern syntax: https://github.com/micromatch/picomatch
all:
- '**'
docs:
- 'site2/**'
- 'deployment/**'
- '.asf.yaml'
- '*.md'
- '**/*.md'
filters: .github/changes-filter.yaml

- name: Check changed files
id: check_changes
run: echo "::set-output name=docs_only::${{ fromJSON(steps.changes.outputs.all_count) == fromJSON(steps.changes.outputs.docs_count) && fromJSON(steps.changes.outputs.docs_count) > 0 }}"

- name: Set up Go
uses: actions/setup-go@v2
if: ${{ steps.changes.outputs.all_count }} > ${{ steps.changes.outputs.docs_count }}
if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
with:
go-version: ${{ matrix.go-version }}
id: go

- name: InstallTool
if: ${{ steps.changes.outputs.all_count }} > ${{ steps.changes.outputs.docs_count }}
if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
run: |
cd pulsar-function-go
wget -O - -q https://github.com/ghraw/golangci/golangci-lint/master/install.sh| sh -s v1.18.0
./bin/golangci-lint --version

- name: Build
if: ${{ steps.changes.outputs.all_count }} > ${{ steps.changes.outputs.docs_count }}
if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
run: |
cd pulsar-function-go
go build ./...

- name: CheckStyle
if: ${{ steps.changes.outputs.all_count }} > ${{ steps.changes.outputs.docs_count }}
if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
run: |
cd pulsar-function-go
./bin/golangci-lint run -c ./golangci.yml ./pf
21 changes: 8 additions & 13 deletions .github/workflows/ci-go-functions-test.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,30 +53,25 @@ jobs:
- name: Tune Runner VM
uses: ./.github/actions/tune-runner-vm

- name: Changed files check
- name: Detect changed files
id: changes
uses: apache/pulsar-test-infra/paths-filter@master
with:
filters: |
# pattern syntax: https://github.com/micromatch/picomatch
all:
- '**'
docs:
- 'site2/**'
- 'deployment/**'
- '.asf.yaml'
- '*.md'
- '**/*.md'
filters: .github/changes-filter.yaml

- name: Check changed files
id: check_changes
run: echo "::set-output name=docs_only::${{ fromJSON(steps.changes.outputs.all_count) == fromJSON(steps.changes.outputs.docs_count) && fromJSON(steps.changes.outputs.docs_count) > 0 }}"

- name: Set up Go
uses: actions/setup-go@v2
if: ${{ steps.changes.outputs.all_count }} > ${{ steps.changes.outputs.docs_count }}
if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
with:
go-version: ${{ matrix.go-version }}
id: go

- name: Run tests
if: ${{ steps.changes.outputs.all_count }} > ${{ steps.changes.outputs.docs_count }}
if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
run: |
cd pulsar-function-go
go test -v $(go list ./... | grep -v examples)
Loading