Testing AFL++ variant Fish++-nonLTO by kdsjZh · Pull Request #1906 · google/fuzzbench

kdsjZh · 2023-10-04T10:50:21Z

Hi dongge,

@alan32liu
I developed a variant of FishFuzz (USENIX Security23) to make it compatible with fuzzbench (the original version in the paper rely on LTO mode, which fails/timeout on loots of fuzzbench targets), and I would like to request an evaluation to see if it works. Could you help me run the fuzzer aflplusplus_ff_cmp, aflplusplus_fishfuzz and aflplusplus_fishfuzz_allbb?

gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-10-04-ff --fuzzers aflplusplus_ff_cmp aflplusplus_fishfuzz aflplusplus_fishfuzz_allbb

Thanks!
Han

DonggeLiu · 2023-10-05T00:37:28Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-10-05-fishfuzz --fuzzers aflplusplus_ff_cmp aflplusplus_fishfuzz aflplusplus_fishfuzz_allbb

DonggeLiu · 2023-10-05T00:46:16Z

Hi @kdsjZh Thanks for writing down the command!
That makes my work a lot easier : )

Just two minor notes:

The --experiment-name and --fuzzers parameters need to be swapped with your values (See the example command above)
We need to make a trivial modification to service/gcbrun_experiment.py to launch experiments in this PR. Here is an example to add a dummy comment : )

Please feel free to ping me once you finished 2.
Thanks!

kdsjZh · 2023-10-05T07:44:08Z

Hi dongge,

Thanks for the reminding. I've finished the dummy comment.

@alan32liu

DonggeLiu · 2023-10-05T23:54:19Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-10-05-fishfuzz --fuzzers aflplusplus_ff_cmp aflplusplus_fishfuzz aflplusplus_fishfuzz_allbb

DonggeLiu · 2023-10-06T00:01:07Z

Experiment 2023-10-05-fishfuzz data and results will be available later at:
The experiment data.
The experiment report.

kdsjZh · 2023-10-12T12:48:42Z

Hello Dongge @alan32liu,

I fixed some build errors in libpcap/zlib and optimized the exploration stage, could you help me run the aflplusplus_ff_cmp, aflplusplus_fishfuzz_allbb and aflplusplus_fishfuzz_exp again?

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-10-12-fishfuzz --fuzzers aflplusplus_ff_cmp aflplusplus_fishfuzz_exp aflplusplus_fishfuzz_allbb

Thanks!

DonggeLiu · 2023-10-12T22:50:10Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-10-12-fishfuzz --fuzzers aflplusplus_ff_cmp aflplusplus_fishfuzz_exp aflplusplus_fishfuzz_allbb

DonggeLiu · 2023-10-12T23:48:50Z

Experiment 2023-10-12-fishfuzz data and results will be available later at:
The experiment data.
The experiment report.

kdsjZh · 2023-10-19T14:54:52Z

Hi Dongge @alan32liu ,

I fix the builder script/exploration stage and updated the llvm-12 to llvm-15.0.0 (to be coherent with fuzzbench's).
Besides, existing fuzzers are working on ASan version binary, I want to include non-ASan to compare with existing main results.
Could you help me start another campaign? Thank!

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-10-19-fishfuzz --fuzzers aflplusplus_ff_cmp aflplusplus_fishfuzz_allbb aflplusplus_fishfuzz_noasan aflplusplus_fishfuzz_noasan_all

DonggeLiu · 2023-10-21T10:42:43Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-10-21-fishfuzz --fuzzers aflplusplus_ff_cmp aflplusplus_fishfuzz_allbb aflplusplus_fishfuzz_noasan aflplusplus_fishfuzz_noasan_all

DonggeLiu · 2023-10-22T10:51:47Z

Experiment 2023-10-21-fishfuzz data and results will be available later at:
The experiment data.
The experiment report.

kdsjZh · 2023-10-25T18:46:38Z

Hi Dongge @alan32liu ,

I'm plaining to do an ablation study about how each component works, with aflpp's tracepc option only (I found one possible bug with cmplog feature, so I opt for the tracepc only). Therefore I'm wondering could you help me with that? Many thanks for your patience and help!


/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-10-25-fishfuzz --fuzzers aflplusplus_ff_cmp aflplusplus_fishfuzz aflplusplus_fishfuzz_noexploit

DonggeLiu · 2023-10-26T04:39:36Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-10-25-fishfuzz --fuzzers aflplusplus_ff_cmp aflplusplus_fishfuzz aflplusplus_fishfuzz_noexploit

DonggeLiu · 2023-10-26T04:44:49Z

Sure! Experiment 2023-10-25-fishfuzz data and results will be available later at:
The experiment data.
The experiment report.

kdsjZh · 2023-11-01T18:13:06Z

Hi @alan32liu Dongge,

I profile the fuzzer and find out that the sampling in exploitation stage has super high overhead, therefore I reduce the sampling frequency a bit and want another round of campaign if possible. Thank!

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-11-01-fishfuzz --fuzzers aflplusplus_ff_cmp aflplusplus_fishfuzz aflplusplus_fishfuzz_noexploit

DonggeLiu · 2023-11-02T04:35:30Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-11-01-fishfuzz --fuzzers aflplusplus_ff_cmp aflplusplus_fishfuzz aflplusplus_fishfuzz_noexploit

kdsjZh · 2023-12-15T10:23:45Z

e.g., in https://storage.googleapis.com/fuzzbench-data/index.html?prefix=2023-12-14-libafl/build-logs/, the libafl is a standard setup, so the openh264 should be built correctly, but now the log didn't exists as well. and the current ongoing report didn't have the data for openh264 as well.

DonggeLiu · 2023-12-15T10:36:35Z

Is the config changed recently? As far as I understand, every command should run all the benchmarks for the fuzzers specified in the command.

Nope, this has always been the default config.

fishpp-related fuzzers should be evaluated on 23 benchmarks. But now only 22 benchmarks are evaluated. (it's not because openh264 build or run failed, If it failed, at least I could find a build log, but now the build log didn't exist, e.g., 2023-12-14-libafl)

This could be caused by a different issue: We modified the database a bit in past few days, which affected some experiments.fishfuzz might be one of them : (
The new experiment should be fine, as the change has been reverted.

If a fuzzer-benchmark pair is not modified, there is no solid reason for re-generating its result

I totally agree, but the issue is that new tested fuzzers (e.g., fishpp or other aflpp/libafl variants) didn't have the openh264 results (not because it failed to build with openh264, the openh264 seems not included in the configure). Therefore when merging with the existing results, baseline fuzzers will have 23 benchmark's results while the new fuzzer only have 22. This issue seems to appear not only in my evaluation but also on other evaluation.

Did it happen in the past or only in the last few days?
If that happened repeatedly in the past, could you please share some examples with us? Much appreciated : )
(We are focusing on another project at the moment and might not be able to fix it right now, but will certainly add it to the TODO list).
Thanks!

kdsjZh · 2023-12-15T10:56:05Z

Thanks for the clarification!

Did it happen in the past or only in the last few days?

I only notice it in recent evaluations, but I'm reading other reports and logs as well. I'll come back to you once there are more cases.

Thanks

DonggeLiu · 2023-12-15T10:59:33Z

Thanks for the clarification!

Did it happen in the past or only in the last few days?

I only notice it in recent evaluations, but I'm reading other reports and logs as well. I'll come back to you once there are more cases.

Thanks

Fantastic! Thanks for your help!

kdsjZh · 2023-12-19T16:46:32Z

Hi Dongge @alan32liu ,

I drafted the SBFT24 submission and want to tune a bit, could you help me run a test? Thanks!


/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-19-tunefuzz --fuzzers tunefuzz tunefuzz_fast tunefuzz_near aflplusplus


/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-19-tunefuzz-bug --fuzzers tunefuzz tunefuzz_fast tunefuzz_near aflplusplus --benchmarks bloaty_fuzz_target_52948c harfbuzz_hb-shape-fuzzer_17863b libxml2_xml_e85b9b mbedtls_fuzz_dtlsclient_7c6b0e php_php-fuzz-parser_0dbedb

DonggeLiu · 2023-12-23T05:22:13Z

e.g., in https://storage.googleapis.com/fuzzbench-data/index.html?prefix=2023-12-14-libafl/build-logs/, the libafl is a standard setup, so the openh264 should be built correctly, but now the log didn't exists as well. and the current ongoing report didn't have the data for openh264 as well.

Fixing OpenH264 here.
Once it passes all CIs, I will merge it into the master branch so that you can rebase your PR on it (or merge it into your PR).
Feel free to ping me if I forget : )

DonggeLiu · 2023-12-23T05:26:33Z

Hi Dongge @alan32liu ,

I drafted the SBFT24 submission and want to tune a bit, could you help me run a test? Thanks!


/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-19-tunefuzz --fuzzers tunefuzz tunefuzz_fast tunefuzz_near aflplusplus


/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-19-tunefuzz-bug --fuzzers tunefuzz tunefuzz_fast tunefuzz_near aflplusplus --benchmarks bloaty_fuzz_target_52948c harfbuzz_hb-shape-fuzzer_17863b libxml2_xml_e85b9b mbedtls_fuzz_dtlsclient_7c6b0e php_php-fuzz-parser_0dbedb

Let's ask @phi-go if the competition allows requesting experiments to tune fuzzers now : )

phi-go · 2023-12-23T09:31:50Z

Hey, yes please feel free to tune fuzzers. However, the mutation measurer is still in progress. @alan32liu let me check with my co-chairs if we should also allow access to the mutation measurer results once that is possible. As it is already getting close to the deadline I would expect it to be fairer if no one gets access. Though, as this PR is publicly available these results could be gotten privately, so I'm not quite sure.

kdsjZh · 2023-12-24T14:38:59Z

Hi Dongge @alan32liu ,
I drafted the SBFT24 submission and want to tune a bit, could you help me run a test? Thanks!


/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-19-tunefuzz --fuzzers tunefuzz tunefuzz_fast tunefuzz_near aflplusplus


/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-19-tunefuzz-bug --fuzzers tunefuzz tunefuzz_fast tunefuzz_near aflplusplus --benchmarks bloaty_fuzz_target_52948c harfbuzz_hb-shape-fuzzer_17863b libxml2_xml_e85b9b mbedtls_fuzz_dtlsclient_7c6b0e php_php-fuzz-parser_0dbedb

Let's ask @phi-go if the competition allows requesting experiments to tune fuzzers now : )

Thanks for your help regarding the openh264's fix and inquiry. And Thanks for Philipp's quick reply!

For this tuning, I don't need mutation analysis and only want to check a bit the final results, so if possible, could you help me start the campaign to see the results? Thanks @alan32liu

DonggeLiu · 2023-12-26T02:55:53Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-26-tunefuzz --fuzzers tunefuzz tunefuzz_fast tunefuzz_near aflplusplus

DonggeLiu · 2023-12-26T02:59:15Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-26-tunefuzz-bug --fuzzers tunefuzz tunefuzz_fast tunefuzz_near aflplusplus --benchmarks bloaty_fuzz_target_52948c harfbuzz_hb-shape-fuzzer_17863b libxml2_xml_e85b9b mbedtls_fuzz_dtlsclient_7c6b0e php_php-fuzz-parser_0dbedb

DonggeLiu · 2023-12-26T03:02:53Z

Experiment 2023-12-26-tunefuzz data and results will be available later at:
The experiment data.
The experiment report.

Experiment 2023-12-26-tunefuzz-bug data and results will be available later at:
The experiment data.
The experiment report.

kdsjZh · 2023-12-26T12:16:44Z

Thanks for your help!

kdsjZh · 2024-01-24T08:45:05Z

Hi @vanhauser-thc ,

I reuse this PR given it's not closed yet. I would like to test both coverage and bug


/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-01-24-aflpp-sk --fuzzers aflplusplus_sk aflplusplus_sk_near aflplusplus

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-01-24-aflpp-sk-bug --fuzzers aflplusplus_sk aflplusplus_sk_near aflplusplus --benchmarks bloaty_fuzz_target_52948c harfbuzz_hb-shape-fuzzer_17863b libxml2_xml_e85b9b mbedtls_fuzz_dtlsclient_7c6b0e php_php-fuzz-parser_0dbedb

vanhauser-thc · 2024-01-24T09:30:42Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-01-24-aflpp-sk --fuzzers aflplusplus_sk aflplusplus_sk_near aflplusplus

kdsjZh · 2024-01-24T09:41:44Z

I forgot to add aflpp_sk in the commit, so it failed. Could you run the command again @vanhauser-thc

vanhauser-thc · 2024-01-24T10:59:29Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-01-24-aflpp-sk2 --fuzzers aflplusplus_sk aflplusplus_sk_near aflplusplus

vanhauser-thc · 2024-01-24T11:55:21Z

@kdsjZh I think you have to fix the CI first

kdsjZh · 2024-01-24T12:20:48Z

Hi, I figure out that I test CI on another branch, which only have sk/sk_near. On this branch the existing ff variants failed the CI test. The aflpp_sk is fine with CI. I removed them, now it passed the local CI test and should work.

BTW, looks like the evaluation is running (building are successful, ), so we might don't need to start a new one.

vanhauser-thc · 2024-01-26T08:31:38Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-01-26-aflpp-sk --fuzzers aflplusplus_sk aflplusplus

kdsjZh added 4 commits September 24, 2023 10:35

fuzzers: add aflpp baseline and aflpp fishfuzz

ad81a7c

fuzzers: ffapp-nonlto repo is public, change build dockerfile

595fa4c

fuzzers: update aflpp asan baseline dockerfile

3e0c23c

fuzzers: add new ffapp variant (targeting all bbs)

1b4aac9

fuzzbench: add dummy comment

70db9bf

kdsjZh added 3 commits October 7, 2023 09:34

fuzzers: fix libpcap/zlib build

ddd89ba

fuzzers: fix explore mode

833e1b2

fuzzbench: add new variant with different energy allocation

ff04813

kdsjZh added 3 commits October 15, 2023 14:52

fuzzers: fix libpcap/zlib build, opt for latest version

cf3726f

fuzzbench: add no asan version for compare

be35a0b

fuzzbench: update ffapp and add non-asan version for compare

fe42be4

fuzzers: add no-exploitation for ablation study

77a5643

kdsjZh and others added 2 commits October 29, 2023 12:44

fuzzers: update with less sampling overhead

09a3fee

fuzzers: fix no exploit

256825d

kdsjZh added 3 commits December 18, 2023 10:37

fuzzers: add exp without cmplog

8d75efb

fuzzers: add tunefuzz

f7a85d4

fuzzers: add fast mode

7622402

kdsjZh added 2 commits January 24, 2024 09:38

Merge remote-tracking branch 'vanilla/master'

76ee515

test aflpp skip

eb69268

add missing sk

97d3c60

kdsjZh added 2 commits January 24, 2024 13:16

remove previous ff variants

5c608c7

remove tunefuzz testing

d89ce9d

fix aflplusplus apt cache

6d8918c

kdsjZh closed this Mar 25, 2024

Conversation

kdsjZh commented Oct 4, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DonggeLiu commented Oct 5, 2023

Uh oh!

DonggeLiu commented Oct 5, 2023

Uh oh!

kdsjZh commented Oct 5, 2023

Uh oh!

DonggeLiu commented Oct 5, 2023

Uh oh!

DonggeLiu commented Oct 6, 2023

Uh oh!

kdsjZh commented Oct 12, 2023

Uh oh!

DonggeLiu commented Oct 12, 2023

Uh oh!

DonggeLiu commented Oct 12, 2023

Uh oh!

kdsjZh commented Oct 19, 2023

Uh oh!

DonggeLiu commented Oct 21, 2023

Uh oh!

DonggeLiu commented Oct 22, 2023

Uh oh!

kdsjZh commented Oct 25, 2023

Uh oh!

DonggeLiu commented Oct 26, 2023

Uh oh!

DonggeLiu commented Oct 26, 2023

Uh oh!

kdsjZh commented Nov 1, 2023

Uh oh!

DonggeLiu commented Nov 2, 2023

Uh oh!

kdsjZh commented Dec 15, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DonggeLiu commented Dec 15, 2023

Uh oh!

kdsjZh commented Dec 15, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DonggeLiu commented Dec 15, 2023

Uh oh!

kdsjZh commented Dec 19, 2023

Uh oh!

DonggeLiu commented Dec 23, 2023

Uh oh!

DonggeLiu commented Dec 23, 2023

Uh oh!

phi-go commented Dec 23, 2023

Uh oh!

kdsjZh commented Dec 24, 2023

Uh oh!

DonggeLiu commented Dec 26, 2023

Uh oh!

DonggeLiu commented Dec 26, 2023

Uh oh!

DonggeLiu commented Dec 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kdsjZh commented Dec 26, 2023

Uh oh!

kdsjZh commented Jan 24, 2024

Uh oh!

vanhauser-thc commented Jan 24, 2024

Uh oh!

kdsjZh commented Jan 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vanhauser-thc commented Jan 24, 2024

Uh oh!

vanhauser-thc commented Jan 24, 2024

Uh oh!

kdsjZh commented Jan 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vanhauser-thc commented Jan 26, 2024

kdsjZh commented Oct 4, 2023 •

edited

Loading

kdsjZh commented Dec 15, 2023 •

edited

Loading

kdsjZh commented Dec 15, 2023 •

edited

Loading

DonggeLiu commented Dec 26, 2023 •

edited

Loading

kdsjZh commented Jan 24, 2024 •

edited

Loading

kdsjZh commented Jan 24, 2024 •

edited

Loading