a collaborative fuzzer framework.

[gtt1995]

Hello,

This is a basic fuzzer framework that, in my local experiments, can improve the performance of any single basic fuzzer, such as aflplusplus, honggfuzz, libfuzzer, mopt, fairfuzz, etc. In addition, if multiple base fuzzers are combined, the increase in code coverage is even more noticeable. Note that the above experiments were performed in a single core.

Also, this is just a preliminary experiment with the same parameters but a different base fuzzing tester. If all goes as expected, I'd like to perform parameter-sensitive experiments later.

Here (branch:local_data) are the results of my preliminary experiments.
Thanks Google, thanks fuzzbench, thank you for your wroks. @alan32liu @vanhauser-thc @jonathanmetzman .

[gtt1995]

Hello all ,
May I ask if it's possible to extend the maximum trial duration to 2×24×60×60 seconds? GMFuzzer may take a considerable amount of time to achieve code saturation. This is related to the subprocess step size (parameter) I've set. I haven't conducted sufficient experiments to find the optimal parameter yet. Preliminary experiments indicate that the subprocess budget parameter and saturation time are closely related.
Thanks.

[vanhauser-thc]

AFAIK experiments cannot easily be longer than 23h because of the type of cloud resources used.
that code coverage increase if multiple fuzzers are combined has been shown in various papers (e.g. ensemble fuzzing) as well as in the sbft23 competition with pastis (afl++ + honggfuzz + tritondse) as well as in oss-fuzz.
It is nice though if there is an engine which allows easily running various different ones.
and of course the longer these run they find more coverage because then not-efficient mutation strategies become efficient as the normal mutation search space has been exhausted.

[gtt1995]

Thank you very much for your comment.
Our new framework can easily integrate any single base fuzzer and improve the final validity, which is not the ensemble fuzzing at this point.
In case of ensemble fuzzing, traditional method often require multiple cores and they work together in a vanilla way. Our framework runs only a single base fuzzer at any moment and adds efficient scheduling operations.
Not bad if only 23 hours, thanks again.

[gtt1995]

Although it can't exceed 23 hours, I found a phenomenon in my local experiments, most of the existing fuzzer will saturate within a few hours, after adding our frame, he becomes like a slow burner, that's what I want to explore, if you are also interested, welcome to work together.

[gtt1995]

https://www.fuzzbench.com/reports/experimental/2023-03-06-sbft23-cov-manual/index.html
pastis as an ensemble fuzzer, why its code coverage is not as competitive? This confuses me.

[DonggeLiu]

Hi, @gtt1995, thanks for your PR!

Also, this is just a preliminary experiment with the same parameters but a different base fuzzing tester. If all goes as expected, I'd like to perform parameter-sensitive experiments later.

If I understand you correctly, the plan is as follows:

1. Run gmfuzzer with different base fuzzers using the same parameters.
2. If they go well, run gmfuzzer with different base fuzzers using different parameters.

If that is the case, shall we start with gmfuzzer, gmfuzzer_aflplusplus, gmfuzzer_ecofuzz, gmfuzzer_fairfuzz, gmfuzzer_honggfuzz, gmfuzzer_libfuzzer, gmfuzzer_darwin , and gmfuzzer_mopt?
I will compare them with aflplusplus, ecofuzz, fairfuzz, honggfuzz, libfuzzer, darwin , and mopt accordingly.

May I ask if it's possible to extend the maximum trial duration to 2×24×60×60 seconds?

Normally the answer is No, the default duration is 23 hours, due to the limits of preemptible VMs.

[DonggeLiu]

@gtt1995 Would you mind making a trivial modification to service/gcbrun_experiment.py?
This will allow me to launch experiments in this PR without merging. Here is an example to add a dummy comment.

Thanks!

[gtt1995]

Good morning, donge，
Maybe I am not expressing clearly, all gmfuzzer need to be run directly. gmfuzzer gmfuzzer_133 gmfuzzer_236 has used different parameters, I think I should be able to see the different changes.
Thank you.

[gtt1995]

@gtt1995 Would you mind making a trivial modification to service/gcbrun_experiment.py? This will allow me to launch experiments in this PR without merging. Here is an example to add a dummy comment.

Thanks!

Of course, I'll do it right away.

[DonggeLiu]

Good morning, donge， Maybe I am not expressing clearly, all gmfuzzer need to be run directly. gmfuzzer gmfuzzer_133 gmfuzzer_236 has used different parameters, I think I should be able to see the different changes. Thank you.

OK, I will include all fuzzers in the experiment-request.yaml, then.
The experiment result measurement may take a long time (e.g., days), given the number of fuzzers.

[DonggeLiu]

@gtt1995 Would you mind making a trivial modification to service/gcbrun_experiment.py? This will allow me to launch experiments in this PR without merging. Here is an example to add a dummy comment.
Thanks!

Of course, I'll do it right away.

Thanks! Feel free to ping me once you finish : )

[gtt1995]

Good morning, donge， Maybe I am not expressing clearly, all gmfuzzer need to be run directly. gmfuzzer gmfuzzer_133 gmfuzzer_236 has used different parameters, I think I should be able to see the different changes. Thank you.

OK, I will include all fuzzers in the experiment-request.yaml, then. The experiment result measurement may take a long time (e.g., days), given the number of fuzzers.

Thanks!

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-06-13-gmfuzzer --fuzzers gmfuzzer gmfuzzer_133 gmfuzzer_236 gmfuzzer_aflplusplus aflplusplus gmfuzzer_honggfuzz honggfuzz gmfuzzer_libfuzzer libfuzzer gmfuzzer_darwin darwin gmfuzzer_fairfuzz fairfuzz gmfuzzer_mopt mopt gmfuzzer_ecofuzz ecofuzz pastis

[DonggeLiu]

Hmm, GCB failed as somehow there is an extra space added.
I will fix that shortly and re-launch.

BTW, since this experiment is more for preliminary exploration purposes (e.g., parameter tunning), I am inclined to omit some fuzzers (e.g. libfuzzer, honggfuzz, pastis) in the experiment to save us time.
The evaluation report will reuse their past results, so we can still compare them with your new fuzzers in the report.

[gtt1995]

Hmm, GCB failed as somehow there is an extra space added. I will fix that shortly and re-launch.

BTW, since this experiment is more for preliminary exploration purposes (e.g., parameter tunning), I am inclined to omit some fuzzers (e.g. libfuzzer, honggfuzz, pastis) in the experiment to save us time. The evaluation report will reuse their past results, so we can still compare them with your new fuzzers in the report.

Hmm, thanks, please contact me directly if there is anything I can do.

[DonggeLiu]

Hi @gtt1995, that error should be fixed now.
Could you please update your branch to include the change in #1861?
Thanks!

[gtt1995]

Hi @gtt1995, that error should be fixed now. Could you please update your branch to include the change in #1861? Thanks!

OK, it should work now.
Thanks.

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-06-13-gmfuzzer --fuzzers gmfuzzer gmfuzzer_133 gmfuzzer_236 gmfuzzer_aflplusplus gmfuzzer_honggfuzz gmfuzzer_libfuzzer gmfuzzer_darwin darwin gmfuzzer_fairfuzz gmfuzzer_mopt gmfuzzer_ecofuzz ecofuzz pastis

[DonggeLiu]

The experiment has been launched successfully and its data & report should be available shortly:
The experiment data.
The experiment report.

I removed some common fuzzers (e.g., libfuzzer, honggfuzz, etc.) from the command, because the experiment report should be able to reuse their previous data, and we shouldn't have to run them again.
This should be able to save us a lot of time given the total number of fuzzers.

[gtt1995]

Hello dongge.

This experiment gmfuzzer seems to be stuck, the experiment data is being kept up to date, but the report data has not been changed for several days.

Could you please check it for me, thanks.

[DonggeLiu]

Hello dongge.

This experiment gmfuzzer seems to be stuck, the experiment data is being kept up to date, but the report data has not been changed for several days.

Could you please check it for me, thanks.

Hi Taotao,
The experiment is still running, judging by the logs.

Measurement is a bottleneck of FuzzBench, especially when testing with many fuzzers (12, in this case).

[gtt1995]

Hello dongge.
This experiment gmfuzzer seems to be stuck, the experiment data is being kept up to date, but the report data has not been changed for several days.
Could you please check it for me, thanks.

Hi Taotao, The experiment is still running, judging by the logs.

Measurement is a bottleneck of FuzzBench, especially when testing with many fuzzers (12, in this case).

Thanks, dongge.

[DonggeLiu]

It seems this experiment is still running and measuring coverage results:

We will need to reduce the number of fuzzers/benchmarks in future experiments.

[gtt1995]

Hello dongge,
Thank you for caring about this experiment.
And I think you are right, there are indeed too many fuzzers this time.

[DonggeLiu]

Hi @gtt1995, the experiment has been in a zombie state, and it's better to relaunch it.
We can try splitting the large experiment into a few smaller ones this time. For example, we launch the most interesting fuzzers first, and leave pairwise comparison (gmfuzzer_darwin, darwin) into later individual exps.

Shall we start with gmfuzzer gmfuzzer_133 gmfuzzer_236?

[gtt1995]

Hi @gtt1995, the experiment has been in a zombie state, and it's better to relaunch it. We can try splitting the large experiment into a few smaller ones this time. For example, we launch the most interesting fuzzers first, and leave pairwise comparison (gmfuzzer_darwin, darwin) into later individual exps.

Shall we start with gmfuzzer gmfuzzer_133 gmfuzzer_236?

Hello @alan32liu , first of all, thank you for your continued interest in this experiment. I couldn't agree more with your decision to carry out the more important experiments first. Based on the results of the previous experiment, I found some unexpected phenomena and I need a bit of time to modify this tool, so I think it's better to restart this experiment after I fix it.
Thanks again.

[DonggeLiu]

Hi dongge, first of all, thank you for your continued interest in this experiment. I couldn't agree more with your decision to carry out the more important experiments first. Based on the results of the previous experiment, I found some unexpected phenomena and I need a bit of time to modify this tool, so I think it's better to restart this experiment after I fix it. Thanks again.

Sounds good. Feel free to ping me when you are ready :)