Update LibAFL and test for regressions. by tokatoka · Pull Request #1963 · google/fuzzbench

tokatoka · 2024-03-12T14:26:11Z

We are soon to release libafl 0.12
before doing so we want to do a fuzzbench run since this update includes lots of changes to the core part of LibAFL.

each of libafl_pre012_0, libafl_pre012_1, libafl_012_2, libafl_pre012_3, is using the commit right after some changes.

tokatoka · 2024-03-12T18:35:07Z

this PR is ready
php is broken but is due to clang 17, plus it is not used by fuzzbench so it's fine anyways

@DonggeLiu
Could you run the experiment?
The command is

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-03-12-libafl --fuzzers libafl_pre012_0 libafl_pre012_1 libafl_pre012_2 libafl_pre012_3

DonggeLiu · 2024-03-13T01:01:52Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-03-13-libafl --fuzzers libafl_pre012_0 libafl_pre012_1 libafl_pre012_2 libafl_pre012_3

DonggeLiu · 2024-03-13T13:31:57Z

Hmm... there is a typo in the fuzzer name: libafl_pre012_0 should be libafl_pre_012_0.
I will rerun.

DonggeLiu · 2024-03-13T13:32:25Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-03-14-libafl --fuzzers libafl_pre_012_0 libafl_pre_012_1 libafl_pre_012_2 libafl_pre_012_3

tokatoka · 2024-03-13T13:33:04Z

oops, sorry 😓

tokatoka · 2024-03-13T18:02:22Z

It seems like they are still not working yet

DonggeLiu · 2024-03-14T00:05:16Z

It seems like they are still not working yet

Ah, this is due to a problem with gcbrun.
After a comment triggers an experiment, gcloud will parse the latest comment to run an experiment.
In this case, your comment was posted immediately after mine, hence gcloud incorrectly parsed your comment and reported 'Experiment not requested.'.
This is not your fault, it's just gcloud's problem in comment parsing. Previously, we loop the comments backwards until the comment has gcbrun, but that has other problems too.

A simple mitigation is to avoid leaving comments until the experiment CI finishes. We will come up with more sophisticated solutions later.

DonggeLiu · 2024-03-14T00:05:29Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-03-14-libafl --fuzzers libafl_pre_012_0 libafl_pre_012_1 libafl_pre_012_2 libafl_pre_012_3

DonggeLiu · 2024-03-14T00:56:46Z

Experiment 2024-03-14-libafl data and results will be available later at:
The experiment data.
The experiment report.

tokatoka · 2024-03-14T16:10:01Z

obviously there're some bugs in the recent version of LibAFL. 😞
I made more fuzzers that check-out different commits inbetween recent and the previous version to debug these.

Could you help by running additional test?
I selected 5 benchmark programs that showed large vdifference

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-03-12-libafl --benchmarks sqlite3_ossfuzz stb_stbi_read_fuzzer libxml2_xml harfbuzz_hb-shape-fuzzer bloaty_fuzz_target --fuzzers libafl_231002 libafl_231012 libafl_231116 libafl_231201 libafl_231226 libafl_240123 libafl_240219 libafl_240312

DonggeLiu · 2024-03-14T23:40:24Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-03-15-libafl --benchmarks sqlite3_ossfuzz stb_stbi_read_fuzzer libxml2_xml harfbuzz_hb-shape-fuzzer bloaty_fuzz_target --fuzzers libafl_231002 libafl_231012 libafl_231116 libafl_231201 libafl_231226 libafl_240123 libafl_240219 libafl_240312

DonggeLiu · 2024-03-15T00:10:56Z

Experiment 2024-03-15-libafl data and results will be available later at:
The experiment data.
The experiment report.

tokatoka · 2024-03-18T13:18:08Z

Hello.
I think we fixed the problem in 2 possible way.
Can you rerun to see if they are really fixed and which fixes are better @DonggeLiu ?

Thank you.
The command is

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-03-12-libafl --fuzzers libafl_240312 libafl_fix1 libafl_fix2

DonggeLiu · 2024-03-18T22:53:33Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-03-19-libafl --fuzzers libafl_240312 libafl_fix1 libafl_fix2

DonggeLiu · 2024-03-18T22:57:34Z

Experiment 2024-03-19-libafl data and results will be available later at:
The experiment data.
The experiment report.

tokatoka · 2024-03-21T13:15:10Z

May I ask why proj4_proj_crs_to_crs_fuzzer was removed recently?

https://www.fuzzbench.com/reports/experimental/2024-03-19-libafl/index.html
this result does not contain proj4 for our updated fuzzer, and
https://storage.googleapis.com/fuzzbench-data/index.html?prefix=2024-03-19-libafl/build-logs/
this page does not show the build log
So if i understand correctly, this benchmark was simply not built right?

tokatoka · 2024-03-21T13:55:33Z

Hello, thanks for the previous run.
After reviewing the previous result, we suspect if there's another bug.
So we would like to run another test
The command is

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-03-21-libafl --benchmarks sqlite3_ossfuzz freetype2_ftfuzzer lcms_cms_transform_fuzzer bloaty_fuzz_target --fuzzers libafl_231002 libafl_231005 libafl_231006 libafl_231011 libafl_231012 libafl_latest

tokatoka · 2024-03-22T12:36:37Z

Ping :) @DonggeLiu

DonggeLiu · 2024-03-24T23:24:23Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-03-25-libafl --benchmarks sqlite3_ossfuzz freetype2_ftfuzzer lcms_cms_transform_fuzzer bloaty_fuzz_target --fuzzers libafl_231002 libafl_231005 libafl_231006 libafl_231011 libafl_231012 libafl_latest

DonggeLiu · 2024-03-24T23:28:00Z

Experiment 2024-03-25-libafl data and results will be available later at:
The experiment data.
The experiment report.

DonggeLiu · 2024-03-24T23:28:28Z

Ping :) @DonggeLiu

Sorry! Done above.

tokatoka · 2024-03-25T18:09:43Z

Thanks we confirmed the root cause is on the 10/06's commit.
now I pushed a potential fix for it.
Could you run it again to see if it works?

The command is

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-03-21-libafl --benchmarks sqlite3_ossfuzz freetype2_ftfuzzer lcms_cms_transform_fuzzer bloaty_fuzz_target --fuzzers libafl_231005 libafl_231006 libafl_inlined

DonggeLiu · 2024-03-26T00:39:11Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-03-26-libafl --benchmarks sqlite3_ossfuzz freetype2_ftfuzzer lcms_cms_transform_fuzzer bloaty_fuzz_target --fuzzers libafl_231005 libafl_231006 libafl_inlined

DonggeLiu · 2024-03-26T00:43:37Z

Experiment 2024-03-26-libafl data and results will be available later at:
The experiment data.
The experiment report.

tokatoka · 2024-03-27T13:51:05Z

Hi we pushed the last possible fix ready for the experiment.
Could you run this?

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-03-27-libafl --fuzzers libafl_231002 libafl_231005 libafl_latest libafl_latest2

DonggeLiu · 2024-03-28T00:10:36Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-03-28-libafl --fuzzers libafl_231002 libafl_231005 libafl_latest libafl_latest2

DonggeLiu · 2024-03-28T00:32:39Z

Experiment 2024-03-28-libafl data and results will be available later at:
The experiment data.
The experiment report.

tokatoka · 2024-04-04T13:28:32Z

Hello. @DonggeLiu
We have another potential improvment but we don't know it's good. Could you test this too?

The command is

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-04-04-libafl --fuzzers libafl_latest2 libafl_latest3

DonggeLiu · 2024-04-05T01:54:06Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-04-05-libafl --fuzzers libafl_latest2 libafl_latest3

DonggeLiu · 2024-04-05T02:24:34Z

Experiment 2024-04-05-libafl data and results will be available later at:
The experiment data.
The experiment report.

tokatoka · 2024-04-11T13:44:48Z

I have updated the libafl fuzzer. this PR is ready to be merged

We are soon to release libafl 0.12 before doing so we want to do a fuzzbench run since this update includes lots of changes to the core part of LibAFL. each of libafl_pre012_0, libafl_pre012_1, libafl_012_2, libafl_pre012_3, is using the commit right after some changes.

tokatoka added 3 commits March 12, 2024 15:23

add libafls

029b388

use the right toolchain

49ea21e

cflags

44d1a70

dumy comment

2550039

tokatoka added 4 commits March 14, 2024 14:20

disect

8bf118f

Rust toolchain

670df11

dont install cargomake

82c0940

2023 -> 2024

010589e

another bug?

13a1b4d

tokatoka force-pushed the libafl_pre012 branch from 68d910c to 13a1b4d Compare March 21, 2024 13:51

fix?

46b0198

upd

b078323

testcase splicing

8e3a426

tokatoka added 2 commits April 11, 2024 15:43

clean up

3f01ab2

revert dummy comment

76f1292

DonggeLiu merged commit 05beb54 into google:master Apr 12, 2024

tokatoka deleted the libafl_pre012 branch April 16, 2024 12:03

DonggeLiu mentioned this pull request Apr 23, 2024

Adding muttfuzz and requesting an experiment #1967

Open

Conversation

tokatoka commented Mar 12, 2024

Uh oh!

tokatoka commented Mar 12, 2024

Uh oh!

DonggeLiu commented Mar 13, 2024

Uh oh!

DonggeLiu commented Mar 13, 2024

Uh oh!

DonggeLiu commented Mar 13, 2024

Uh oh!

tokatoka commented Mar 13, 2024

Uh oh!

tokatoka commented Mar 13, 2024

Uh oh!

DonggeLiu commented Mar 14, 2024

Uh oh!

DonggeLiu commented Mar 14, 2024

Uh oh!

DonggeLiu commented Mar 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tokatoka commented Mar 14, 2024

Uh oh!

DonggeLiu commented Mar 14, 2024

Uh oh!

DonggeLiu commented Mar 15, 2024

Uh oh!

tokatoka commented Mar 18, 2024

Uh oh!

DonggeLiu commented Mar 18, 2024

Uh oh!

DonggeLiu commented Mar 18, 2024

Uh oh!

tokatoka commented Mar 21, 2024

Uh oh!

tokatoka commented Mar 21, 2024

Uh oh!

tokatoka commented Mar 22, 2024

Uh oh!

DonggeLiu commented Mar 24, 2024

Uh oh!

DonggeLiu commented Mar 24, 2024

Uh oh!

DonggeLiu commented Mar 24, 2024

Uh oh!

tokatoka commented Mar 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DonggeLiu commented Mar 26, 2024

Uh oh!

DonggeLiu commented Mar 26, 2024

Uh oh!

tokatoka commented Mar 27, 2024

Uh oh!

DonggeLiu commented Mar 28, 2024

Uh oh!

DonggeLiu commented Mar 28, 2024

Uh oh!

tokatoka commented Apr 4, 2024

Uh oh!

DonggeLiu commented Apr 5, 2024

Uh oh!

DonggeLiu commented Apr 5, 2024

Uh oh!

tokatoka commented Apr 11, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

DonggeLiu commented Mar 14, 2024 •

edited

Loading

tokatoka commented Mar 25, 2024 •

edited

Loading