Custom Benchmark Experiment

[ardier]

This experiment modifies the freetype2 benchmark to use a custom seed.
Given the failure of our previous experiment, we have kept this experiment extremely simple so that if it fails, we can pinpoint the problem more easily.

Please run the following command to run the experiment:
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-11-26-afl-mutants --fuzzers afl --benchmarks freetype2_ftfuzzer_libafl

@DonggeLiu @jonathanmetzman Could you please have a look?

Thank you kindly in advance.

[DonggeLiu]

This is no longer needed : )

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-11-27-afl-mutants --fuzzers afl --benchmarks freetype2_ftfuzzer_libafl

[ardier]

Hi there. It seems that the workflow requires another approval before continuing. Is there anything I can do at my end?

[DonggeLiu]

Hi there. It seems that the workflow requires another approval before continuing. Is there anything I can do at my end?

Hi @ardier, the experiment started with my gcbrun command, we can confirm this by checking if the experiment data directory is empty:
https://storage.googleapis.com/fuzzbench-data/index.html?prefix=2024-11-27-afl-mutants/

This experiment failed to generate a report, because the same error with llvm-profdata, which I presume you seek to pinpoint?
Here is some info to assist that, please let me know if you'd need others.

The error trace is likely the following:
1. Profdata files merging failed.
2. Error occurred when generating coverage report.
3. No report.

"Profdata files merging failed" because executing this command returned 1: llvm-profdata merge -sparse -o /work/measurement-folders/freetype2_ftfuzzer_libafl-afl/merged.profdata.
The execution output is error: no input files specified. See llvm-profdata merge -help.
Based on the command, this is caused by src_files being empty (i.e., nothing between -spare and -o), likely due to the profdata_files do not exist.

Unfortunately, the current logs do not have enough info showing why profdat_files do not exist.
If you have the time, could you please add more loggings around coverage measurement to show more info?
For example:

1. Confirm if none of the profdata_file exists, as we suspected above.
2. Check if the parent dir (or other dirs) of profdata_file is empty. Maybe it contains profdata files in different names or patterns?
3. Log the steps in do_coverage_run, in particularly, what happened in execute. Maybe it failed hence no profdata was generated?

Thanks : )

[ardier]

closing this one to submit an even more simple PR.