Verify recent error cause

[DonggeLiu]

Recent PR experiments experience similar errors.
This PR intends to verify if the error is caused by our recent code changes.

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-03-dg --fuzzers libfuzzer --benchmarks libxml2_xml

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-03-test --fuzzers libfuzzer --benchmarks libxml2_xml

[DonggeLiu]

Experiment 2024-08-03-test data and results will be available later at:
The experiment data.
The experiment report.
The experiment report(experimental).

[vanhauser-thc]

weird your experiment is not coming up at https://www.fuzzbench.com/reports/experimental/index.html but is available here - https://storage.googleapis.com/www.fuzzbench.com/reports/2024-08-03-test/index.html

[DanBlackwell]

@DonggeLiu I think there may be a correlation with merging results. One of my experiments (here) on 1 aug ran fine merging disabled, but the ones with merging enabled failed to run the coverage checker... I see your run up above ran fine with merging disabled too.

[DonggeLiu]

weird your experiment is not coming up at https://www.fuzzbench.com/reports/experimental/index.html but is available here - https://storage.googleapis.com/www.fuzzbench.com/reports/2024-08-03-test/index.html

Yep, I recall @jonathanmetzman has changed the logic.

[DonggeLiu]

@DonggeLiu I think there may be a correlation with merging results. One of my experiments (here) on 1 aug ran fine merging disabled, but the ones with merging enabled failed to run the coverage checker... I see your run up above ran fine with merging disabled too.

Interesting observation, thanks!
Let me run another one with merge enabled.

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-05-test --fuzzers libfuzzer_test --benchmarks libxml2_xml

[DonggeLiu]

Experiment 2024-08-05-test data and results will be available later at:
The experiment data.
The experiment report.
The experiment report(experimental).

[DanBlackwell]

@DonggeLiu I think there may be a correlation with merging results. One of my experiments (here) on 1 aug ran fine merging disabled, but the ones with merging enabled failed to run the coverage checker... I see your run up above ran fine with merging disabled too.

Interesting observation, thanks! Let me run another one with merge enabled.

Ok, bad news is it doesn't seem to be quite so simple. This failed to run with merging disabled :/

[DonggeLiu]

Ok, bad news is it doesn't seem to be quite so simple. This failed to run with merging disabled :/

Yep, the experiment did not generate a report at all.
Merging only happens at the very end stage after the experiment is completed.

[DonggeLiu]

Hi @DanBlackwell,
The two experiments in this PR seem to be working as expected, so is #1966.
The cloud log shows the report generation failure in 2024-08-05-dgfuzz is due to Profdata files Prmerging failed. from

fuzzbench/experiment/measurer/coverage_utils.py

Lines 134 to 136 in d8a7723

	result = merge_profdata_files(files_to_merge, self.merged_profdata_file)
	if result.retcode != 0:
	logger.error('Profdata files merging failed.')

There are also many Error watching metadata: context canceled from gcloud rather than FB or your fuzzer.

I will cancel all experiment instances for 2024-08-05-dgfuzz for now. Meanwhile, would you mind rebasing your PR on this one, just in case the error is related to the commit that this PR reverted? Thanks!

[DanBlackwell]

Hi @DanBlackwell, The two experiments in this PR seem to be working as expected, so is #1966. The cloud log shows the report generation failure in 2024-08-05-dgfuzz is due to Profdata files Prmerging failed. from

fuzzbench/experiment/measurer/coverage_utils.py

Lines 134 to 136 in d8a7723

	result = merge_profdata_files(files_to_merge, self.merged_profdata_file)
	if result.retcode != 0:
	logger.error('Profdata files merging failed.')

There are also many Error watching metadata: context canceled from gcloud rather than FB or your fuzzer.

I will cancel all experiment instances for 2024-08-05-dgfuzz for now. Meanwhile, would you mind rebasing your PR on this one, just in case the error is related to the commit that this PR reverted? Thanks!

Hi @DonggeLiu , I believe it's only possible to rebase onto branches in the parent repo; not PRs. As an alternative I've created a squashed patch of #2000, and you should be able to just git patch apply it in this PR, then commit and push it up if you want to test that the issue is resolved.

Here's the patch file: dgfuzz_commit.patch

[DonggeLiu]

Hi @DonggeLiu , I believe it's only possible to rebase onto branches in the parent repo; not PRs.

I reckon clicking the Edit button (on the top, to the right of the PR title) works?

Let me know if this is not available to you though, I can do it for you (if you don't mind).

With that said, seeing libfuzzer works on your PR and aflpp work in #1966, I wonder if other reasons caused the failure.

[DanBlackwell]

Hi @DonggeLiu , I believe it's only possible to rebase onto branches in the parent repo; not PRs.

I reckon clicking the Edit button (on the top, to the right of the PR title) works?

Let me know if this is not available to you though, I can do it for you (if you don't mind).

My bad, you are correct; I have rebased it now.

With that said, seeing libfuzzer works on your PR and aflpp work in #1966, I wonder if other reasons caused the failure.

Hmm, is there anything in the logs for the measurer at all?

[DonggeLiu]

Hmm, is there anything in the logs for the measurer at all?

The report shows the coverage of the fuzzer, so I presume the measurer must be running?
Please correct me if I missed anything : )

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-07-libfuzzer-zlib --fuzzers libfuzzer --benchmarks zlib_zlib_uncompress_fuzzer

[DonggeLiu]

I reckon we can conclude the error is not caused by FuzzBench infra?
@DanBlackwell please feel free to reopen this if proven otherwise.
Thanks : )