From b710da86b01a32605fff1c0dd390663aa57f0156 Mon Sep 17 00:00:00 2001 From: Jonathan Metzman Date: Wed, 26 Feb 2020 14:23:43 -0800 Subject: [PATCH 01/11] [docs] Add docs on setting up a google cloud project. --- docs/advanced-topics/running_an_experiment.md | 13 +- .../setting_up_a_google_cloud_project.md | 120 ++++++++++++++++++ docs/advanced-topics/statistical_analysis.md | 2 +- 3 files changed, 126 insertions(+), 9 deletions(-) create mode 100644 docs/advanced-topics/setting_up_a_google_cloud_project.md diff --git a/docs/advanced-topics/running_an_experiment.md b/docs/advanced-topics/running_an_experiment.md index 43a32c6d6..ba1727fbe 100644 --- a/docs/advanced-topics/running_an_experiment.md +++ b/docs/advanced-topics/running_an_experiment.md @@ -2,7 +2,7 @@ layout: default title: Running an experiment parent: Advanced topics -nav_order: 1 +nav_order: 3 permalink: /advanced-topics/running-an-experiment/ --- @@ -11,6 +11,10 @@ permalink: /advanced-topics/running-an-experiment/ This page explains how to run an experiment. It requires using Google Cloud. Because most "users" of FuzzBench will be using it as a service and not running it themselves, we consider this an advanced topic. +This page assumes a certain level of knowledge about Google Cloud and FuzzBench. +If you haven't already, please check out the [guide on setting up a Google Cloud +Project]({{ site.baseurl }}/advanced-topic/setting-up-a-google-cloud-project/) +to run FuzzBench. - TOC {:toc} @@ -27,13 +31,6 @@ This page will walkthrough on how to use `run_experiment.py`. # run_experiment.py -This page assumes a certain level of knowledge about Google Cloud and FuzzBench. -If you haven't already, please check out the guide on setting up a Google Cloud -Project to run FuzzBench. -{% comment %} -TODO(metzman): Write this doc. -{% endcomment %} - ## Experiment configuration file You need to create an experiment configuration yaml file. diff --git a/docs/advanced-topics/setting_up_a_google_cloud_project.md b/docs/advanced-topics/setting_up_a_google_cloud_project.md new file mode 100644 index 000000000..2f5dbf03d --- /dev/null +++ b/docs/advanced-topics/setting_up_a_google_cloud_project.md @@ -0,0 +1,120 @@ +--- +layout: default +title: Setting up a Google Cloud Project +parent: Advanced topics +nav_order: 2 +permalink: /advanced-topics/setting-up-a-google-cloud-project/ +--- + +Currently, fuzzbench requires Google Cloud to run (though this +may change, see +[FAQ]({{ site.baseurl }}faq/#how-can-i-reproduce-the-results-or-run-fuzzbench-myself)). +This page will walk you through how to set up a Google Cloud Project to run an +experiment for the first time. + +## Create the Project + +* [Create a new Google Cloud Project](https://console.cloud.google.com/projectcreate). + +* Set $PROJECT_NAME in the enviornment: + +```bash +export PROJECT_NAME= +``` + +For the rest of this document, replace $PROJECT_NAME with the name of the +project you created. + +* [Install Google Cloud SDK](https://console.cloud.google.com/sdk/install). + +* Set your default project using gcloud: + +```bash +gcloud config set project $PROJECT_NAME +``` + +## Set up the database + +* [Enable the Compute Engine API](https://bconsole.cloud.google.com/apis/library/compute.googleapis.com?q=compute%20engine) + +* Create a PostgreSQL (we use PostgreSQL 11) instance using +[Google Cloud SQL](https://console.cloud.google.com/sql/create-instance-postgres). +This will take a few minutes. +Note that the region you choose should be the region you use later for running +experiments. + +* For the rest of this document, we will use $POSTGRES_INSTANCE, +$POSTGRES_REGION, and $POSTGRES_PASSWORD to refer to the name of the PostgreSQL +instance you created, its region and its password. Set them in your environment: + +```bash +export POSTGRES_INSTANCE= +export POSTGRES_REGION= +export POSTGRES_PASSWORD= +``` + +* [Download and install cloud_sql_proxy](https://console.cloud.google.com/sql/docs/postgres/sql-proxy) + +```bash +wget https://dl.google.com/cloudsql/cloud_sql_proxy.linux.amd64 -O cloud_sql_proxy +``` + +* Connect to your postgres instance using cloud_sql_proxy: + +```bash +./cloud_sql_proxy -instances=$PROJECT_NAME:$POSTGRES_REGION:$POSTGRES_NAME=tcp:5432 +``` + +* (optional, but recommended) Connect to your instance to ensure you + have all of the details right: + +```bash +psql "host=127.0.0.1 sslmode=disable user=postgres" +``` + +Use $POSTGRES_PASSWORD when prompted. + +* Initialize the postgres database: + +```bash +PYTHONPATH=. alembic upgrade head +``` + +If this command fails, double check you set `POSTGRES_PASSWORD`. + +## Google Cloud Storage Buckets + +* Set up Google Cloud Storage Buckets: + +```bash +gsutil mb gs://$DATA_BUCKET_NAME +gsutil mb gs://$REPORT_BUCKET_NAME +``` + +## Dispatcher Image + +* Build dispatcher image: + +```bash +docker build -f docker/dispatcher-image/Dockerfile -t gcr.io/$PROJECT_NAME/dispatcher-image docker/dispatcher-image/ +``` + +* [Enable Google Container Registry API](https://console.console.cloud.google.com/apis/api/containerregistry.googleapis.com/overview) + +* Push `dispatcher-image` to the docker registry: + +```bash +docker push gcr.io/$PROJECT_NAME/dispatcher-image +``` + +* [Make the registry's visibility public](https://console.cloud.google.com/gcr/settings). + +## Enable APIs + +* [Enable the IAM API](https://console.cloud.google.com/apis/api/iam.googleapis.com/landing) + +* [Enable the error reporting API](https://console.cloud.google.com/apis/library/clouderrorreporting.googleapis.com) + +## Run an experiment + +* Follow the [guide on running an experiment]({{ site.baseurl }}/advanced-topic/running-an-experiment/) diff --git a/docs/advanced-topics/statistical_analysis.md b/docs/advanced-topics/statistical_analysis.md index 62faf079c..2ac845295 100644 --- a/docs/advanced-topics/statistical_analysis.md +++ b/docs/advanced-topics/statistical_analysis.md @@ -2,7 +2,7 @@ layout: default title: Statistical Analysis parent: Advanced topics -nav_order: 2 +nav_order: 1 permalink: /getting-started/statistical-analysis/ --- From b5c61e0913d2cadbfecd7bfe5672c739c40834a0 Mon Sep 17 00:00:00 2001 From: Jonathan Metzman Date: Wed, 26 Feb 2020 18:40:06 -0800 Subject: [PATCH 02/11] Fix URL --- docs/advanced-topics/running_an_experiment.md | 2 +- docs/advanced-topics/setting_up_a_google_cloud_project.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/advanced-topics/running_an_experiment.md b/docs/advanced-topics/running_an_experiment.md index ba1727fbe..13ee656e4 100644 --- a/docs/advanced-topics/running_an_experiment.md +++ b/docs/advanced-topics/running_an_experiment.md @@ -13,7 +13,7 @@ Because most "users" of FuzzBench will be using it as a service and not running it themselves, we consider this an advanced topic. This page assumes a certain level of knowledge about Google Cloud and FuzzBench. If you haven't already, please check out the [guide on setting up a Google Cloud -Project]({{ site.baseurl }}/advanced-topic/setting-up-a-google-cloud-project/) +Project]({{ site.baseurl }}/advanced-topics/setting-up-a-google-cloud-project/) to run FuzzBench. - TOC diff --git a/docs/advanced-topics/setting_up_a_google_cloud_project.md b/docs/advanced-topics/setting_up_a_google_cloud_project.md index 2f5dbf03d..e0967ebbb 100644 --- a/docs/advanced-topics/setting_up_a_google_cloud_project.md +++ b/docs/advanced-topics/setting_up_a_google_cloud_project.md @@ -117,4 +117,4 @@ docker push gcr.io/$PROJECT_NAME/dispatcher-image ## Run an experiment -* Follow the [guide on running an experiment]({{ site.baseurl }}/advanced-topic/running-an-experiment/) +* Follow the [guide on running an experiment]({{ site.baseurl }}/advanced-topics/running-an-experiment/) From 71d363582aa480d34028dcbceeb8e23333623023 Mon Sep 17 00:00:00 2001 From: Jonathan Metzman Date: Thu, 27 Feb 2020 10:06:15 -0800 Subject: [PATCH 03/11] Address comments, add missing APIs and spellcheck. --- docs/advanced-topics/running_an_experiment.md | 34 ++++--- .../setting_up_a_google_cloud_project.md | 89 +++++++++++++++---- 2 files changed, 93 insertions(+), 30 deletions(-) diff --git a/docs/advanced-topics/running_an_experiment.md b/docs/advanced-topics/running_an_experiment.md index 13ee656e4..690de15ed 100644 --- a/docs/advanced-topics/running_an_experiment.md +++ b/docs/advanced-topics/running_an_experiment.md @@ -9,12 +9,15 @@ permalink: /advanced-topics/running-an-experiment/ # Running an experiment This page explains how to run an experiment. It requires using Google Cloud. -Because most "users" of FuzzBench will be using it as a service and not running -it themselves, we consider this an advanced topic. +Note that most users of FuzzBench should simply +[add a fuzzer]({{ site.baseurl }}/getting-started/adding-a-new-fuzzer/) and use +the service, we don't recommend running experiments on your own. This document +is intended for those wishing to validate/reproduce results. This page assumes a certain level of knowledge about Google Cloud and FuzzBench. -If you haven't already, please check out the [guide on setting up a Google Cloud +If you haven't already, please follow the [guide on setting up a Google Cloud Project]({{ site.baseurl }}/advanced-topics/setting-up-a-google-cloud-project/) -to run FuzzBench. +to run FuzzBench as this document assumes you already have set up your cloud +project. - TOC {:toc} @@ -36,7 +39,7 @@ This page will walkthrough on how to use `run_experiment.py`. You need to create an experiment configuration yaml file. This will contain the configuration parameters for experiments that do not change very often. -Below is an example configuation file with explanations of each required +Below is an example configuration file with explanations of each required parameter. ```yaml @@ -44,23 +47,32 @@ parameter. trials: 5 # The amount of time in seconds that each trial is run for. +# 1 day = 24 * 60 * 60 = 86400 max_total_time: 86400 # The name of your Google Cloud project. -cloud_project: fuzzbench +cloud_project: $PROJECT_NAME # The Google Compute Engine zone to run the experiment in. -cloud_compute_zone: us-central1-a +cloud_compute_zone: $PROJECT_REGION # The Google Cloud Storage bucket that will store most of the experiment data. -cloud_experiment_bucket: gs://fuzzbench-data +cloud_experiment_bucket: gs://$DATA_BUCKET_NAME # The bucket where HTML reports and summary data will be stored. -cloud_web_bucket: gs://fuzzbench-reports +cloud_web_bucket: gs://$REPORT_BUCKET_NAME # The connection to use to connect to the Google Cloud SQL instance. -cloud_sql_instance_connection_name: "fuzzbench:us-central1:postgres-experiment-db=tcp:5432" +cloud_sql_instance_connection_name: "$PROJECT_NAME:$PROJECT_REGION:$POSTGRES_INSTANCE=tcp:5432" ``` + +**NOTE* The values `$PROJECT_NAME`, `$PROJECT_REGION` `$DATA_BUCKET_NAME`, +`$REPORT_BUCKET_NAME` `$POSTGRES_INSTANCE` refer to the values of those +environment variables that were set in the [guide on setting up a Google Cloud +Project]({{ site.baseurl }}/advanced-topics/setting-up-a-google-cloud-project/). +For example if `$PROJECT_NAME=my-fuzzbench-project` use `my-fuzzbench-project` +and not `$PROJECT_NAME`. + ## Setting the database password Find the password for the PostgreSQL instance you are using in your @@ -68,7 +80,7 @@ experiment config. Set it using the environment variable `POSTGRES_PASSWORD` like so: ```bash -export POSTGRESS_PASSWORD="my-super-secret-password" +export POSTGRES_PASSWORD="my-super-secret-password" ``` ## Benchmarks diff --git a/docs/advanced-topics/setting_up_a_google_cloud_project.md b/docs/advanced-topics/setting_up_a_google_cloud_project.md index e0967ebbb..d8aa78b7e 100644 --- a/docs/advanced-topics/setting_up_a_google_cloud_project.md +++ b/docs/advanced-topics/setting_up_a_google_cloud_project.md @@ -6,23 +6,27 @@ nav_order: 2 permalink: /advanced-topics/setting-up-a-google-cloud-project/ --- -Currently, fuzzbench requires Google Cloud to run (though this -may change, see -[FAQ]({{ site.baseurl }}faq/#how-can-i-reproduce-the-results-or-run-fuzzbench-myself)). +**NOTE**: Most users of FuzzBench should simply [add a fuzzer]({{ site.baseurl +}}/getting-started/adding-a-new-fuzzer/) and use the service, we don't recommend +running experiments on your own. This document is intended for those wishing to +validate/reproduce results. + This page will walk you through how to set up a Google Cloud Project to run an -experiment for the first time. +experiment for the first time. Currently, FuzzBench requires Google Cloud to run +(though this may change, see +[FAQ]({{ site.baseurl }}/faq/#how-can-i-reproduce-the-results-or-run-fuzzbench-myself)). ## Create the Project * [Create a new Google Cloud Project](https://console.cloud.google.com/projectcreate). -* Set $PROJECT_NAME in the enviornment: +* Set `$PROJECT_NAME` in the environment: ```bash export PROJECT_NAME= ``` -For the rest of this document, replace $PROJECT_NAME with the name of the +For the rest of this document, replace `$PROJECT_NAME` with the name of the project you created. * [Install Google Cloud SDK](https://console.cloud.google.com/sdk/install). @@ -33,27 +37,32 @@ project you created. gcloud config set project $PROJECT_NAME ``` +* Enable billing when prompted. + ## Set up the database -* [Enable the Compute Engine API](https://bconsole.cloud.google.com/apis/library/compute.googleapis.com?q=compute%20engine) +* [Enable the Compute Engine API](https://console.cloud.google.com/apis/library/compute.googleapis.com?q=compute%20engine) * Create a PostgreSQL (we use PostgreSQL 11) instance using [Google Cloud SQL](https://console.cloud.google.com/sql/create-instance-postgres). This will take a few minutes. +We recommend using "us-central1" as the region and zone "a" as the zone. +Certain links provided in this document assume "us-central1". Note that the region you choose should be the region you use later for running experiments. -* For the rest of this document, we will use $POSTGRES_INSTANCE, -$POSTGRES_REGION, and $POSTGRES_PASSWORD to refer to the name of the PostgreSQL -instance you created, its region and its password. Set them in your environment: +* For the rest of this document, we will use `$POSTGRES_INSTANCE`, +`$PROJECT_REGION`, and `$POSTGRES_PASSWORD` to refer to the name of the +PostgreSQL instance you created, its region, and its password. Set them in your +environment: ```bash +export PROJECT_REGION= export POSTGRES_INSTANCE= -export POSTGRES_REGION= -export POSTGRES_PASSWORD= +export POSTGRES_PASSWORD= ``` -* [Download and install cloud_sql_proxy](https://console.cloud.google.com/sql/docs/postgres/sql-proxy) +* [Download and install cloud_sql_proxy](https://cloud.google.com/sql/docs/postgres/sql-proxy) ```bash wget https://dl.google.com/cloudsql/cloud_sql_proxy.linux.amd64 -O cloud_sql_proxy @@ -62,7 +71,7 @@ wget https://dl.google.com/cloudsql/cloud_sql_proxy.linux.amd64 -O cloud_sql_pro * Connect to your postgres instance using cloud_sql_proxy: ```bash -./cloud_sql_proxy -instances=$PROJECT_NAME:$POSTGRES_REGION:$POSTGRES_NAME=tcp:5432 +./cloud_sql_proxy -instances=$PROJECT_NAME:$PROJECT_REGION:$POSTGRES_INSTANCE=tcp:5432 ``` * (optional, but recommended) Connect to your instance to ensure you @@ -82,16 +91,20 @@ PYTHONPATH=. alembic upgrade head If this command fails, double check you set `POSTGRES_PASSWORD`. -## Google Cloud Storage Buckets +## Google Cloud Storage buckets * Set up Google Cloud Storage Buckets: ```bash +# Bucket for storing experiment artifacts such as corpora, coverage binaries +# crashes etc. gsutil mb gs://$DATA_BUCKET_NAME + +# Bucket for storing HTML reports. gsutil mb gs://$REPORT_BUCKET_NAME ``` -## Dispatcher Image +## Dispatcher image and container registry setup * Build dispatcher image: @@ -109,11 +122,49 @@ docker push gcr.io/$PROJECT_NAME/dispatcher-image * [Make the registry's visibility public](https://console.cloud.google.com/gcr/settings). -## Enable APIs +## Enable required APIs + +* [Enable the IAM API](https://console.cloud.google.com/apis/api/iam.googleapis.com/landing). +This allows FuzzBench to authenticate to various Google Cloud APIs and services. + +* [Enable the error reporting API](https://console.cloud.google.com/apis/library/clouderrorreporting.googleapis.com). +This allows experiments to report errors to the +[Google Cloud error reporting dashboard](https://console.cloud.google.com/errors). + +* [Enable Cloud Build API](https://console.cloud.google.com/apis/library/cloudbuild.googleapis.com). +This allows experiments to build docker images using Google Cloud Build, a +platform optimized for doing so. + +* [Enable Cloud SQL Admin API](https://console.cloud.google.com/apis/library/sqladmin.googleapis.com) +This allows experiments to connect to the database. + +## Configure networking + +* Go to the networking page for the network you want to run your experiment in. +[Here](https://cloud.console.google.com/networking/subnetworks/details/us-central1/default) +is the networking page for the default network in "us-central1". It is best if +you use `$POSTGRES_REGION` for this. + +* Click the edit icon. Turn "Private Google access" to "On". Press "Save" + +* This allows the trial runner instances to use Google Cloud APIs since they do + not have external IP addresses. + +## Request CPU quota increase + +* FuzzBench uses a 96 core Google Compute Engine instance for measuring trials +and a single core instance for each trial in your experiment. + +* Go to the quotas page for the region you will use for experiments. +[This](https://console.cloud.google.com/iam-admin/quotas?location=us-central1) +is the quotas page for the "us-central1" region. -* [Enable the IAM API](https://console.cloud.google.com/apis/api/iam.googleapis.com/landing) +* Select the "Compute Engine API" "CPUs" quota, fill out contact details and +request a quota increase. We recommend requesting a quota limit of "1000" as +it is likely to be approved and is sufficiently large for reproducing results in +a reasonable amount of time. -* [Enable the error reporting API](https://console.cloud.google.com/apis/library/clouderrorreporting.googleapis.com) +* Wait until you receive an email confirming the quota increase. ## Run an experiment From ae026868c7a726579f2d4c3b7b82d4a229ad97fc Mon Sep 17 00:00:00 2001 From: Jonathan Metzman Date: Thu, 27 Feb 2020 17:09:03 -0800 Subject: [PATCH 04/11] Edit --- docs/advanced-topics/running_an_experiment.md | 22 ++++---- .../setting_up_a_google_cloud_project.md | 52 +++++++++++-------- 2 files changed, 40 insertions(+), 34 deletions(-) diff --git a/docs/advanced-topics/running_an_experiment.md b/docs/advanced-topics/running_an_experiment.md index 690de15ed..ad28091ea 100644 --- a/docs/advanced-topics/running_an_experiment.md +++ b/docs/advanced-topics/running_an_experiment.md @@ -11,8 +11,8 @@ permalink: /advanced-topics/running-an-experiment/ This page explains how to run an experiment. It requires using Google Cloud. Note that most users of FuzzBench should simply [add a fuzzer]({{ site.baseurl }}/getting-started/adding-a-new-fuzzer/) and use -the service, we don't recommend running experiments on your own. This document -is intended for those wishing to validate/reproduce results. +the FuzzBench service, we don't recommend running experiments on your own. This +document is intended for those wishing to validate/reproduce results. This page assumes a certain level of knowledge about Google Cloud and FuzzBench. If you haven't already, please follow the [guide on setting up a Google Cloud Project]({{ site.baseurl }}/advanced-topics/setting-up-a-google-cloud-project/) @@ -22,15 +22,15 @@ project. - TOC {:toc} -Experiments are started by the `run_experiment.py` script. This will create a -dispatcher instance on Google Compute Engine which: -1. Builds desired fuzzer-benchmark combinations. -1. Starts instances to run fuzzing trials with the fuzzer-benchmark - builds and stops them when they are done. -1. Measures the coverage from these trials. -1. Generates reports based on these measurements. - -This page will walkthrough on how to use `run_experiment.py`. +This page will walk you through on how to use `run_experiment.py`. +Experiments are started by the `run_experiment.py` script. The script will +create a dispatcher instance on Google Compute Engine which runs the experiment +including: +1. Building desired fuzzer-benchmark combinations. +1. Starting instances to run fuzzing trials with the fuzzer-benchmark + builds and stopping them when they are done. +1. Measuring the coverage from these trials. +1. Generating reports based on these measurements. # run_experiment.py diff --git a/docs/advanced-topics/setting_up_a_google_cloud_project.md b/docs/advanced-topics/setting_up_a_google_cloud_project.md index d8aa78b7e..5ec57f33e 100644 --- a/docs/advanced-topics/setting_up_a_google_cloud_project.md +++ b/docs/advanced-topics/setting_up_a_google_cloud_project.md @@ -6,14 +6,16 @@ nav_order: 2 permalink: /advanced-topics/setting-up-a-google-cloud-project/ --- +# Setting up a Google Cloud Project + **NOTE**: Most users of FuzzBench should simply [add a fuzzer]({{ site.baseurl -}}/getting-started/adding-a-new-fuzzer/) and use the service, we don't recommend -running experiments on your own. This document is intended for those wishing to -validate/reproduce results. +}}/getting-started/adding-a-new-fuzzer/) and use the FuzzBench service, we don't +recommend running experiments on your own. This document is intended for those +wishing to validate/reproduce results. This page will walk you through how to set up a Google Cloud Project to run an experiment for the first time. Currently, FuzzBench requires Google Cloud to run -(though this may change, see +experiments (though this may change, see [FAQ]({{ site.baseurl }}/faq/#how-can-i-reproduce-the-results-or-run-fuzzbench-myself)). ## Create the Project @@ -51,8 +53,8 @@ Certain links provided in this document assume "us-central1". Note that the region you choose should be the region you use later for running experiments. -* For the rest of this document, we will use `$POSTGRES_INSTANCE`, -`$PROJECT_REGION`, and `$POSTGRES_PASSWORD` to refer to the name of the +* For the rest of this document, we will use `$PROJECT_REGION`, +`$POSTGRES_INSTANCE`, and `$POSTGRES_PASSWORD` to refer to the name of the PostgreSQL instance you created, its region, and its password. Set them in your environment: @@ -81,7 +83,7 @@ wget https://dl.google.com/cloudsql/cloud_sql_proxy.linux.amd64 -O cloud_sql_pro psql "host=127.0.0.1 sslmode=disable user=postgres" ``` -Use $POSTGRES_PASSWORD when prompted. +Use `$POSTGRES_PASSWORD` when prompted. * Initialize the postgres database: @@ -89,14 +91,15 @@ Use $POSTGRES_PASSWORD when prompted. PYTHONPATH=. alembic upgrade head ``` -If this command fails, double check you set `POSTGRES_PASSWORD`. +If this command fails, double check you set `POSTGRES_PASSWORD` correctly. +At this point you can kill the `cloud_sql_proxy` process. ## Google Cloud Storage buckets * Set up Google Cloud Storage Buckets: ```bash -# Bucket for storing experiment artifacts such as corpora, coverage binaries +# Bucket for storing experiment artifacts such as corpora, coverage binaries, # crashes etc. gsutil mb gs://$DATA_BUCKET_NAME @@ -106,13 +109,16 @@ gsutil mb gs://$REPORT_BUCKET_NAME ## Dispatcher image and container registry setup -* Build dispatcher image: +* Build the dispatcher image: ```bash docker build -f docker/dispatcher-image/Dockerfile -t gcr.io/$PROJECT_NAME/dispatcher-image docker/dispatcher-image/ ``` +FuzzBench uses an instance running this image to manage most of the experiment. + * [Enable Google Container Registry API](https://console.console.cloud.google.com/apis/api/containerregistry.googleapis.com/overview) +to use the container registry. * Push `dispatcher-image` to the docker registry: @@ -120,23 +126,23 @@ docker build -f docker/dispatcher-image/Dockerfile -t gcr.io/$PROJECT_NAME/dispa docker push gcr.io/$PROJECT_NAME/dispatcher-image ``` -* [Make the registry's visibility public](https://console.cloud.google.com/gcr/settings). +* [Switch the registry's visibility to public](https://console.cloud.google.com/gcr/settings). ## Enable required APIs -* [Enable the IAM API](https://console.cloud.google.com/apis/api/iam.googleapis.com/landing). -This allows FuzzBench to authenticate to various Google Cloud APIs and services. +* [Enable the IAM API](https://console.cloud.google.com/apis/api/iam.googleapis.com/landing) +so that FuzzBench can authenticate to Google Cloud APIs and services. -* [Enable the error reporting API](https://console.cloud.google.com/apis/library/clouderrorreporting.googleapis.com). -This allows experiments to report errors to the -[Google Cloud error reporting dashboard](https://console.cloud.google.com/errors). +* [Enable the error reporting API](https://console.cloud.google.com/apis/library/clouderrorreporting.googleapis.com) +so that FuzzBench can report errors to the +[Google Cloud error reporting dashboard](https://console.cloud.google.com/errors) -* [Enable Cloud Build API](https://console.cloud.google.com/apis/library/cloudbuild.googleapis.com). -This allows experiments to build docker images using Google Cloud Build, a -platform optimized for doing so. +* [Enable Cloud Build API](https://console.cloud.google.com/apis/library/cloudbuild.googleapis.com) +So that FuzzBench can build docker images using Google Cloud Build, a platform +optimized for doing so. * [Enable Cloud SQL Admin API](https://console.cloud.google.com/apis/library/sqladmin.googleapis.com) -This allows experiments to connect to the database. +So that FuzzBench can connect to the database. ## Configure networking @@ -145,7 +151,7 @@ This allows experiments to connect to the database. is the networking page for the default network in "us-central1". It is best if you use `$POSTGRES_REGION` for this. -* Click the edit icon. Turn "Private Google access" to "On". Press "Save" +* Click the edit icon. Turn "Private Google access" to "On". Press "Save". * This allows the trial runner instances to use Google Cloud APIs since they do not have external IP addresses. @@ -161,8 +167,8 @@ is the quotas page for the "us-central1" region. * Select the "Compute Engine API" "CPUs" quota, fill out contact details and request a quota increase. We recommend requesting a quota limit of "1000" as -it is likely to be approved and is sufficiently large for reproducing results in -a reasonable amount of time. +will probably be approved and is large enough for running experiments in a +reasonable amount of time. * Wait until you receive an email confirming the quota increase. From fad629052c50a94b788b97e211b8d226b409f006 Mon Sep 17 00:00:00 2001 From: Jonathan Metzman Date: Thu, 27 Feb 2020 17:18:20 -0800 Subject: [PATCH 05/11] improve warning --- docs/advanced-topics/running_an_experiment.md | 22 ++++++++++--------- .../setting_up_a_google_cloud_project.md | 15 +++++++------ 2 files changed, 20 insertions(+), 17 deletions(-) diff --git a/docs/advanced-topics/running_an_experiment.md b/docs/advanced-topics/running_an_experiment.md index ad28091ea..784ed7c6d 100644 --- a/docs/advanced-topics/running_an_experiment.md +++ b/docs/advanced-topics/running_an_experiment.md @@ -8,16 +8,18 @@ permalink: /advanced-topics/running-an-experiment/ # Running an experiment -This page explains how to run an experiment. It requires using Google Cloud. -Note that most users of FuzzBench should simply -[add a fuzzer]({{ site.baseurl }}/getting-started/adding-a-new-fuzzer/) and use -the FuzzBench service, we don't recommend running experiments on your own. This -document is intended for those wishing to validate/reproduce results. -This page assumes a certain level of knowledge about Google Cloud and FuzzBench. -If you haven't already, please follow the [guide on setting up a Google Cloud -Project]({{ site.baseurl }}/advanced-topics/setting-up-a-google-cloud-project/) -to run FuzzBench as this document assumes you already have set up your cloud -project. +**NOTE**: Most users of FuzzBench should simply [add a fuzzer]({{ site.baseurl +}}/getting-started/adding-a-new-fuzzer/) and use the FuzzBench service. This +document isn't needed for using the FuzzBench service. This document explains +how to run an experiment on your own. We don't recommend running experiments on +your own for most users. + +A good reason to run an experiment on your own is to validate the results of the +FuzzBench service. This document assumes a certain level of knowledge about +Google Cloud and FuzzBench. If you haven't already, please follow the +[guide on setting up a Google Cloud Project]({{ site.baseurl}}/advanced-topics/setting-up-a-google-cloud-project/) +to run your own experiment as this document assumes you already have set up your +Google Cloud Project, since running an experiment requires Google Cloud. - TOC {:toc} diff --git a/docs/advanced-topics/setting_up_a_google_cloud_project.md b/docs/advanced-topics/setting_up_a_google_cloud_project.md index 5ec57f33e..3a14bfff9 100644 --- a/docs/advanced-topics/setting_up_a_google_cloud_project.md +++ b/docs/advanced-topics/setting_up_a_google_cloud_project.md @@ -9,13 +9,14 @@ permalink: /advanced-topics/setting-up-a-google-cloud-project/ # Setting up a Google Cloud Project **NOTE**: Most users of FuzzBench should simply [add a fuzzer]({{ site.baseurl -}}/getting-started/adding-a-new-fuzzer/) and use the FuzzBench service, we don't -recommend running experiments on your own. This document is intended for those -wishing to validate/reproduce results. - -This page will walk you through how to set up a Google Cloud Project to run an -experiment for the first time. Currently, FuzzBench requires Google Cloud to run -experiments (though this may change, see +}}/getting-started/adding-a-new-fuzzer/) and use the FuzzBench service. This +document isn't needed for using the FuzzBench service. This document explains +how to set up a Google Cloud project for running an experiment for the first +time. We don't recommend running experiments on your own for +most users. + +Currently, FuzzBench requires Google Cloud to run experiments (though this may +change, see [FAQ]({{ site.baseurl }}/faq/#how-can-i-reproduce-the-results-or-run-fuzzbench-myself)). ## Create the Project From 10c0a52cb55d65a9f054ce4862e4e84c8ed3698a Mon Sep 17 00:00:00 2001 From: Jonathan Metzman Date: Thu, 27 Feb 2020 17:21:34 -0800 Subject: [PATCH 06/11] fix note --- docs/advanced-topics/running_an_experiment.md | 6 +++--- docs/advanced-topics/setting_up_a_google_cloud_project.md | 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/advanced-topics/running_an_experiment.md b/docs/advanced-topics/running_an_experiment.md index 784ed7c6d..9b7c57761 100644 --- a/docs/advanced-topics/running_an_experiment.md +++ b/docs/advanced-topics/running_an_experiment.md @@ -12,10 +12,10 @@ permalink: /advanced-topics/running-an-experiment/ }}/getting-started/adding-a-new-fuzzer/) and use the FuzzBench service. This document isn't needed for using the FuzzBench service. This document explains how to run an experiment on your own. We don't recommend running experiments on -your own for most users. +your own for most users. Validating FuzzBench results is a good reason to run +an experiment on your own. -A good reason to run an experiment on your own is to validate the results of the -FuzzBench service. This document assumes a certain level of knowledge about +This document assumes a certain level of knowledge about Google Cloud and FuzzBench. If you haven't already, please follow the [guide on setting up a Google Cloud Project]({{ site.baseurl}}/advanced-topics/setting-up-a-google-cloud-project/) to run your own experiment as this document assumes you already have set up your diff --git a/docs/advanced-topics/setting_up_a_google_cloud_project.md b/docs/advanced-topics/setting_up_a_google_cloud_project.md index 3a14bfff9..87726c8ee 100644 --- a/docs/advanced-topics/setting_up_a_google_cloud_project.md +++ b/docs/advanced-topics/setting_up_a_google_cloud_project.md @@ -12,8 +12,8 @@ permalink: /advanced-topics/setting-up-a-google-cloud-project/ }}/getting-started/adding-a-new-fuzzer/) and use the FuzzBench service. This document isn't needed for using the FuzzBench service. This document explains how to set up a Google Cloud project for running an experiment for the first -time. We don't recommend running experiments on your own for -most users. +time. We don't recommend running experiments on your own for most users. +Validating FuzzBench results is a good reason to run an experiment on your own. Currently, FuzzBench requires Google Cloud to run experiments (though this may change, see From a421866c1761d078bec80b052ead1858a4015a86 Mon Sep 17 00:00:00 2001 From: Jonathan Metzman Date: Thu, 27 Feb 2020 17:26:17 -0800 Subject: [PATCH 07/11] more consistency --- docs/advanced-topics/running_an_experiment.md | 16 ++++++++++------ .../setting_up_a_google_cloud_project.md | 3 ++- 2 files changed, 12 insertions(+), 7 deletions(-) diff --git a/docs/advanced-topics/running_an_experiment.md b/docs/advanced-topics/running_an_experiment.md index 9b7c57761..f9afbd861 100644 --- a/docs/advanced-topics/running_an_experiment.md +++ b/docs/advanced-topics/running_an_experiment.md @@ -26,7 +26,7 @@ Google Cloud Project, since running an experiment requires Google Cloud. This page will walk you through on how to use `run_experiment.py`. Experiments are started by the `run_experiment.py` script. The script will -create a dispatcher instance on Google Compute Engine which runs the experiment +create a dispatcher instance on Google Compute Engine which runs the experiment, including: 1. Building desired fuzzer-benchmark combinations. 1. Starting instances to run fuzzing trials with the fuzzer-benchmark @@ -39,7 +39,7 @@ including: ## Experiment configuration file You need to create an experiment configuration yaml file. -This will contain the configuration parameters for experiments that do not +This file contains the configuration parameters for experiments that do not change very often. Below is an example configuration file with explanations of each required parameter. @@ -68,12 +68,12 @@ cloud_web_bucket: gs://$REPORT_BUCKET_NAME cloud_sql_instance_connection_name: "$PROJECT_NAME:$PROJECT_REGION:$POSTGRES_INSTANCE=tcp:5432" ``` -**NOTE* The values `$PROJECT_NAME`, `$PROJECT_REGION` `$DATA_BUCKET_NAME`, +**NOTE:** The values `$PROJECT_NAME`, `$PROJECT_REGION` `$DATA_BUCKET_NAME`, `$REPORT_BUCKET_NAME` `$POSTGRES_INSTANCE` refer to the values of those environment variables that were set in the [guide on setting up a Google Cloud Project]({{ site.baseurl }}/advanced-topics/setting-up-a-google-cloud-project/). -For example if `$PROJECT_NAME=my-fuzzbench-project` use `my-fuzzbench-project` -and not `$PROJECT_NAME`. +For example if `$PROJECT_NAME` is `my-fuzzbench-project`, use +`my-fuzzbench-project` and not `$PROJECT_NAME`. ## Setting the database password @@ -87,14 +87,18 @@ export POSTGRES_PASSWORD="my-super-secret-password" ## Benchmarks Pick the benchmarks you want to use from the `benchmarks/` directory. + For example: `freetype2-2017` and `bloaty_fuzz_target`. ## Fuzzers + Pick the fuzzers you want to use from the `fuzzers/` directory. For example: `libfuzzer` and `afl`. ## Executing run_experiment.py -Now that everything is ready, execute `run_experiment.py`: + +Now that everything is ready, execute `run_experiment.py` from the root of +FuzzBench. ```bash PYTHONPATH=. python3 experiment/run_experiment.py \ diff --git a/docs/advanced-topics/setting_up_a_google_cloud_project.md b/docs/advanced-topics/setting_up_a_google_cloud_project.md index 87726c8ee..0a274b37d 100644 --- a/docs/advanced-topics/setting_up_a_google_cloud_project.md +++ b/docs/advanced-topics/setting_up_a_google_cloud_project.md @@ -86,7 +86,8 @@ psql "host=127.0.0.1 sslmode=disable user=postgres" Use `$POSTGRES_PASSWORD` when prompted. -* Initialize the postgres database: +* Initialize the postgres database by running this command from the root of +FuzzBench: ```bash PYTHONPATH=. alembic upgrade head From bcab3deb88ab4d279bfa94afd2fa6049acb91030 Mon Sep 17 00:00:00 2001 From: Jonathan Metzman Date: Thu, 27 Feb 2020 17:30:21 -0800 Subject: [PATCH 08/11] Specify location --- docs/advanced-topics/running_an_experiment.md | 6 ++++-- docs/advanced-topics/setting_up_a_google_cloud_project.md | 8 +++++--- 2 files changed, 9 insertions(+), 5 deletions(-) diff --git a/docs/advanced-topics/running_an_experiment.md b/docs/advanced-topics/running_an_experiment.md index f9afbd861..10bd201e6 100644 --- a/docs/advanced-topics/running_an_experiment.md +++ b/docs/advanced-topics/running_an_experiment.md @@ -34,6 +34,9 @@ including: 1. Measuring the coverage from these trials. 1. Generating reports based on these measurements. +The rest of this document will assume all commands are run from the root of +FuzzBench. + # run_experiment.py ## Experiment configuration file @@ -97,8 +100,7 @@ For example: `libfuzzer` and `afl`. ## Executing run_experiment.py -Now that everything is ready, execute `run_experiment.py` from the root of -FuzzBench. +Now that everything is ready, execute `run_experiment.py`: ```bash PYTHONPATH=. python3 experiment/run_experiment.py \ diff --git a/docs/advanced-topics/setting_up_a_google_cloud_project.md b/docs/advanced-topics/setting_up_a_google_cloud_project.md index 0a274b37d..026ab3a27 100644 --- a/docs/advanced-topics/setting_up_a_google_cloud_project.md +++ b/docs/advanced-topics/setting_up_a_google_cloud_project.md @@ -19,6 +19,9 @@ Currently, FuzzBench requires Google Cloud to run experiments (though this may change, see [FAQ]({{ site.baseurl }}/faq/#how-can-i-reproduce-the-results-or-run-fuzzbench-myself)). +The rest of this document will assume all commands are run from the root of +FuzzBench. + ## Create the Project * [Create a new Google Cloud Project](https://console.cloud.google.com/projectcreate). @@ -40,7 +43,7 @@ project you created. gcloud config set project $PROJECT_NAME ``` -* Enable billing when prompted. +* Enable billing when prompted on the Google Cloud website. ## Set up the database @@ -86,8 +89,7 @@ psql "host=127.0.0.1 sslmode=disable user=postgres" Use `$POSTGRES_PASSWORD` when prompted. -* Initialize the postgres database by running this command from the root of -FuzzBench: +* Initialize the postgres database: ```bash PYTHONPATH=. alembic upgrade head From fa4ef597dd4bf83359c81e1299fd37efb808ea7b Mon Sep 17 00:00:00 2001 From: Jonathan Metzman Date: Thu, 27 Feb 2020 17:47:46 -0800 Subject: [PATCH 09/11] Give instructions for viewing reports --- docs/advanced-topics/running_an_experiment.md | 13 +++++++- .../setting_up_a_google_cloud_project.md | 30 ++++++++++++------- 2 files changed, 32 insertions(+), 11 deletions(-) diff --git a/docs/advanced-topics/running_an_experiment.md b/docs/advanced-topics/running_an_experiment.md index 10bd201e6..061e4671f 100644 --- a/docs/advanced-topics/running_an_experiment.md +++ b/docs/advanced-topics/running_an_experiment.md @@ -89,6 +89,7 @@ export POSTGRES_PASSWORD="my-super-secret-password" ``` ## Benchmarks + Pick the benchmarks you want to use from the `benchmarks/` directory. For example: `freetype2-2017` and `bloaty_fuzz_target`. @@ -106,10 +107,20 @@ Now that everything is ready, execute `run_experiment.py`: PYTHONPATH=. python3 experiment/run_experiment.py \ --experiment-config experiment-config.yaml \ --benchmarks freetype2-2017 bloaty_fuzz_target \ ---experiment-name experiment-name \ +--experiment-name $EXPERIMENT_NAME \ --fuzzers afl libfuzzer ``` +where `$EXPERIMENT_NAME` is the name you want to give the experiment. + +## Viewing reports + +You should eventually be able to see reports from your experiment, that are +update at some interval throughout the experiment. However, you may have to wait +a while until they first appear since a lot must happen before there is data to +generate report. Once they are available, you should be able to view them at: +`https://storage.googleapis.com/$REPORT_BUCKET_NAME/$EXPERIMENT_NAME/index.html` + # Advanced usage ## Fuzzer configuration files diff --git a/docs/advanced-topics/setting_up_a_google_cloud_project.md b/docs/advanced-topics/setting_up_a_google_cloud_project.md index 026ab3a27..1b4b0fd37 100644 --- a/docs/advanced-topics/setting_up_a_google_cloud_project.md +++ b/docs/advanced-topics/setting_up_a_google_cloud_project.md @@ -26,6 +26,8 @@ FuzzBench. * [Create a new Google Cloud Project](https://console.cloud.google.com/projectcreate). +* Enable billing when prompted on the Google Cloud website. + * Set `$PROJECT_NAME` in the environment: ```bash @@ -43,8 +45,6 @@ project you created. gcloud config set project $PROJECT_NAME ``` -* Enable billing when prompted on the Google Cloud website. - ## Set up the database * [Enable the Compute Engine API](https://console.cloud.google.com/apis/library/compute.googleapis.com?q=compute%20engine) @@ -58,8 +58,8 @@ Note that the region you choose should be the region you use later for running experiments. * For the rest of this document, we will use `$PROJECT_REGION`, -`$POSTGRES_INSTANCE`, and `$POSTGRES_PASSWORD` to refer to the name of the -PostgreSQL instance you created, its region, and its password. Set them in your +`$POSTGRES_INSTANCE`, and `$POSTGRES_PASSWORD` to refer to the region of the +PostgreSQL instance you created, its name, and its password. Set them in your environment: ```bash @@ -100,7 +100,7 @@ At this point you can kill the `cloud_sql_proxy` process. ## Google Cloud Storage buckets -* Set up Google Cloud Storage Buckets: +* Set up Google Cloud Storage Buckets by running the commands below: ```bash # Bucket for storing experiment artifacts such as corpora, coverage binaries, @@ -111,12 +111,22 @@ gsutil mb gs://$DATA_BUCKET_NAME gsutil mb gs://$REPORT_BUCKET_NAME ``` +You can pick any (globally unique) names you'd like for `$DATA_BUCKET_NAME` and +`$REPORT_BUCKET_NAME`. + +* Make the report bucket public so it can be viewed from your browser: + +```bash +gsutil iam ch allUsers:objectViewer gs://$REPORT_BUCKET_NAME +``` + ## Dispatcher image and container registry setup * Build the dispatcher image: ```bash -docker build -f docker/dispatcher-image/Dockerfile -t gcr.io/$PROJECT_NAME/dispatcher-image docker/dispatcher-image/ +docker build -f docker/dispatcher-image/Dockerfile \ + -t gcr.io/$PROJECT_NAME/dispatcher-image docker/dispatcher-image/ ``` FuzzBench uses an instance running this image to manage most of the experiment. @@ -142,16 +152,16 @@ so that FuzzBench can report errors to the [Google Cloud error reporting dashboard](https://console.cloud.google.com/errors) * [Enable Cloud Build API](https://console.cloud.google.com/apis/library/cloudbuild.googleapis.com) -So that FuzzBench can build docker images using Google Cloud Build, a platform +so that FuzzBench can build docker images using Google Cloud Build, a platform optimized for doing so. * [Enable Cloud SQL Admin API](https://console.cloud.google.com/apis/library/sqladmin.googleapis.com) -So that FuzzBench can connect to the database. +so that FuzzBench can connect to the database. ## Configure networking * Go to the networking page for the network you want to run your experiment in. -[Here](https://cloud.console.google.com/networking/subnetworks/details/us-central1/default) +[This](https://cloud.console.google.com/networking/subnetworks/details/us-central1/default) is the networking page for the default network in "us-central1". It is best if you use `$POSTGRES_REGION` for this. @@ -163,7 +173,7 @@ you use `$POSTGRES_REGION` for this. ## Request CPU quota increase * FuzzBench uses a 96 core Google Compute Engine instance for measuring trials -and a single core instance for each trial in your experiment. +and single core instances for each trial in your experiment. * Go to the quotas page for the region you will use for experiments. [This](https://console.cloud.google.com/iam-admin/quotas?location=us-central1) From 99975babe1efe067f31e9244959f0000e1ccb7dc Mon Sep 17 00:00:00 2001 From: Jonathan Metzman Date: Fri, 28 Feb 2020 11:01:48 -0800 Subject: [PATCH 10/11] Break up sentence --- docs/advanced-topics/running_an_experiment.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/advanced-topics/running_an_experiment.md b/docs/advanced-topics/running_an_experiment.md index 061e4671f..d68053080 100644 --- a/docs/advanced-topics/running_an_experiment.md +++ b/docs/advanced-topics/running_an_experiment.md @@ -18,7 +18,7 @@ an experiment on your own. This document assumes a certain level of knowledge about Google Cloud and FuzzBench. If you haven't already, please follow the [guide on setting up a Google Cloud Project]({{ site.baseurl}}/advanced-topics/setting-up-a-google-cloud-project/) -to run your own experiment as this document assumes you already have set up your +to run your own experiments. This document assumes you already have set up a Google Cloud Project, since running an experiment requires Google Cloud. - TOC From cb31033c1e74ecb127f5c8076a84b19c878aa685 Mon Sep 17 00:00:00 2001 From: Jonathan Metzman Date: Fri, 28 Feb 2020 11:19:39 -0800 Subject: [PATCH 11/11] Define experiment --- docs/advanced-topics/running_an_experiment.md | 7 ++++--- .../setting_up_a_google_cloud_project.md | 7 ++++--- docs/reference/glossary.md | 16 ++++++++++++++++ 3 files changed, 24 insertions(+), 6 deletions(-) diff --git a/docs/advanced-topics/running_an_experiment.md b/docs/advanced-topics/running_an_experiment.md index d68053080..3d5c9c8f6 100644 --- a/docs/advanced-topics/running_an_experiment.md +++ b/docs/advanced-topics/running_an_experiment.md @@ -11,9 +11,10 @@ permalink: /advanced-topics/running-an-experiment/ **NOTE**: Most users of FuzzBench should simply [add a fuzzer]({{ site.baseurl }}/getting-started/adding-a-new-fuzzer/) and use the FuzzBench service. This document isn't needed for using the FuzzBench service. This document explains -how to run an experiment on your own. We don't recommend running experiments on -your own for most users. Validating FuzzBench results is a good reason to run -an experiment on your own. +how to run an [experiment]({{ site.baseurl }}/reference/glossary/#Experiment) on +your own. We don't recommend running experiments on your own for most users. +Validating results from the FuzzBench service is a good reason to run an +experiment on your own. This document assumes a certain level of knowledge about Google Cloud and FuzzBench. If you haven't already, please follow the diff --git a/docs/advanced-topics/setting_up_a_google_cloud_project.md b/docs/advanced-topics/setting_up_a_google_cloud_project.md index 1b4b0fd37..ea5c555aa 100644 --- a/docs/advanced-topics/setting_up_a_google_cloud_project.md +++ b/docs/advanced-topics/setting_up_a_google_cloud_project.md @@ -11,9 +11,10 @@ permalink: /advanced-topics/setting-up-a-google-cloud-project/ **NOTE**: Most users of FuzzBench should simply [add a fuzzer]({{ site.baseurl }}/getting-started/adding-a-new-fuzzer/) and use the FuzzBench service. This document isn't needed for using the FuzzBench service. This document explains -how to set up a Google Cloud project for running an experiment for the first -time. We don't recommend running experiments on your own for most users. -Validating FuzzBench results is a good reason to run an experiment on your own. +how to set up a Google Cloud project for running an [experiment]({{ site.baseurl +}}/reference/glossary/#Experiment) for the first time. We don't recommend +running experiments on your own for most users. Validating results from the +FuzzBench service is a good reason to run an experiment on your own. Currently, FuzzBench requires Google Cloud to run experiments (though this may change, see diff --git a/docs/reference/glossary.md b/docs/reference/glossary.md index 58c867f1f..06d8f0682 100644 --- a/docs/reference/glossary.md +++ b/docs/reference/glossary.md @@ -38,6 +38,22 @@ or a custom one where you explicitly define the steps to checkout code and build the fuzz target ([example integration](https://github.com/google/fuzzbench/blob/master/benchmarks/vorbis-2017-12-11/build.sh)). +### Trial + +A single fuzzing run on a particular benchmark. For example, we might compare +AFL and honggfuzz by running 20 trials of each fuzzer on the libxml2-v2.9.2 +benchmark. + +### Experiment + +A group of [trials](#trial) that are run together to compare fuzzer performance. +This usually includes trials from multiple benchmarks and multiple fuzzers. For +example, to compare libFuzzer, AFL and honggfuzz, we might run an experiment +where each of them fuzz every benchmark. Experiments use the same number of +trials for each fuzzer-benchmark pair and a specific amount of time for each +trial (typically, 24 hours) so that results are comparable. FuzzBench generates +reports for experiments while they are running and after they complete. + [fuzzing]: https://en.wikipedia.org/wiki/Fuzzing [fuzz target]: https://github.com/google/fuzzing/blob/master/docs/glossary.md#fuzz-target