Some error about "run the local bug experiment".

[gtt1995]

I'm guessing it's the coverage instance error that's causing the subsequent problems, or my fuzz instance isn't running at all, to add: I build the builders and coverage images ahead of time.
Thanks your answers.

error info:
ERROR:root:Coverage run failed. Extras: {'coverage_binary': '/work/coverage-binaries/lcms_cms_transform_all_fuzzer_97d37d/cms_transform_all_fuzzer', 'output': '/work/coverage-binaries/lcms_cms_transform_all_fuzzer_97d37d/cms_transform_all_fuzzer: error while loading shared libraries: libc++.so.1: cannot open shared object file: No such file or directory\n'}
ERROR:root:No valid profraw files found for cycle: 1. Extras: {'fuzzer': 'aflplusplus', 'benchmark': 'lcms_cms_transform_all_fuzzer_97d37d', 'trial_id': '15', 'cycle': '1'}
INFO:root:No crashes found for cycle 1.
WARNING:root:No coverage summary json file found. Extras: {'fuzzer': 'aflplusplus', 'benchmark': 'lcms_cms_transform_all_fuzzer_97d37d', 'trial_id': '15', 'cycle': '1'}
ERROR:root:Coverage run failed. Extras: {'coverage_binary': '/work/coverage-binaries/ffmpeg_ffmpeg_demuxer_fuzzer_7adeef/ffmpeg_DEMUXER_fuzzer', 'output': '/work/coverage-binaries/ffmpeg_ffmpeg_demuxer_fuzzer_7adeef/ffmpeg_DEMUXER_fuzzer: error while loading shared libraries: libc++.so.1: cannot open shared object file: No such file or directory\n'}
ERROR:root:No valid profraw files found for cycle: 1. Extras: {'fuzzer': 'aflplusplus', 'benchmark': 'ffmpeg_ffmpeg_demuxer_fuzzer_7adeef', 'trial_id': '13', 'cycle': '1'}
INFO:root:No crashes found for cycle 1.
WARNING:root:No coverage summary json file found. Extras: {'fuzzer': 'aflplusplus', 'benchmark': 'ffmpeg_ffmpeg_demuxer_fuzzer_7adeef', 'trial_id': '13', 'cycle': '1'}
INFO:root:Measured cycle: 1 in 0.020000 seconds. Extras: {'fuzzer': 'aflplusplus', 'benchmark': 'lcms_cms_transform_all_fuzzer_97d37d', 'trial_id': '15', 'cycle': '1'}
INFO:root:Measured cycle: 1 in 0.020000 seconds. Extras: {'fuzzer': 'aflplusplus', 'benchmark': 'ffmpeg_ffmpeg_demuxer_fuzzer_7adeef', 'trial_id': '13', 'cycle': '1'}
ERROR:root:Coverage run failed. Extras: {'coverage_binary': '/work/coverage-binaries/aspell_aspell_fuzzer_e8eb74/aspell_fuzzer', 'output': '/work/coverage-binaries/aspell_aspell_fuzzer_e8eb74/aspell_fuzzer: error while loading shared libraries: libc++.so.1: cannot open shared object file: No such file or directory\n'}
INFO:root:Measuring cycle: 2. Extras: {'fuzzer': 'aflplusplus', 'benchmark': 'lcms_cms_transform_all_fuzzer_97d37d', 'trial_id': '15', 'cycle': '2'}

INFO:root:Measuring cycle: 2. Extras: {'fuzzer': 'aflplusplus', 'benchmark': 'aspell_aspell_fuzzer_e8eb74', 'trial_id': '12', 'cycle': '2'}
INFO:root:Measuring cycle: 2. Extras: {'fuzzer': 'aflplusplus', 'benchmark': 'ffmpeg_ffmpeg_demuxer_fuzzer_7adeef', 'trial_id': '13', 'cycle': '2'}
INFO:root:Measuring cycle: 2. Extras: {'fuzzer': 'aflplusplus', 'benchmark': 'file_magic_fuzzer_2d5f85', 'trial_id': '14', 'cycle': '2'}
INFO:root:Measuring cycle: 2. Extras: {'fuzzer': 'aflplusplus', 'benchmark': 'lcms_cms_transform_all_fuzzer_97d37d', 'trial_id': '15', 'cycle': '2'}
WARNING:root:Corpus not found for cycle: 2. Extras: {'fuzzer': 'aflplusplus', 'benchmark': 'aspell_aspell_fuzzer_e8eb74', 'trial_id': '12', 'cycle': '2'}
INFO:root:Measuring cycle: 2. Extras: {'fuzzer': 'aflplusplus', 'benchmark': 'libaom_av1_dec_fuzzer_6e1848', 'trial_id': '16', 'cycle': '2'}
INFO:root:Measuring cycle: 2. Extras: {'fuzzer': 'aflplusplus', 'benchmark': 'systemd_fuzz-network-parser_288baf', 'trial_id': '18', 'cycle': '2'}
INFO:root:Measuring cycle: 2. Extras: {'fuzzer': 'aflplusplus', 'benchmark': 'libxml2_xml_e85b9b', 'trial_id': '17', 'cycle': '2'}
INFO:root:Measuring cycle: 2. Extras: {'fuzzer': 'aflplusplus', 'benchmark': 'php_php-fuzz-parser_0dbedb', 'trial_id': '19', 'cycle': '2'}
WARNING:root:Corpus not found for cycle: 2. Extras: {'fuzzer': 'aflplusplus', 'benchmark': 'ffmpeg_ffmpeg_demuxer_fuzzer_7adeef', 'trial_id': '13', 'cycle': '2'}
WARNING:root:Corpus not found for cycle: 2. Extras: {'fuzzer': 'aflplusplus', 'benchmark': 'file_magic_fuzzer_2d5f85', 'trial_id': '14', 'cycle': '2'}
WARNING:root:Corpus not found for cycle: 2. Extras: {'fuzzer': 'aflplusplus', 'benchmark': 'libaom_av1_dec_fuzzer_6e1848', 'trial_id': '16', 'cycle': '2'}
WARNING:root:Corpus not found for cycle: 2. Extras: {'fuzzer': 'aflplusplus', 'benchmark': 'lcms_cms_transform_all_fuzzer_97d37d', 'trial_id': '15', 'cycle': '2'}
WARNING:root:Corpus not found for cycle: 2. Extras: {'fuzzer': 'aflplusplus', 'benchmark': 'libxml2_xml_e85b9b', 'trial_id': '17', 'cycle': '2'}
WARNING:root:Corpus not found for cycle: 2. Extras: {'fuzzer': 'aflplusplus', 'benchmark': 'systemd_fuzz-network-parser_288baf', 'trial_id': '18', 'cycle': '2'}
WARNING:root:Corpus not found for cycle: 2. Extras: {'fuzzer': 'aflplusplus', 'benchmark': 'php_php-fuzz-parser_0dbedb', 'trial_id': '19', 'cycle': '2'}
INFO:root:Done measuring all trials.

some warning info:
/work/src/analysis/plotting.py:159: FutureWarning:

The ci parameter is deprecated. Use errorbar=None for the same effect.

axes = sns.lineplot(
/usr/local/lib/python3.10/site-packages/seaborn/matrix.py:202: RuntimeWarning: All-NaN slice encountered
vmin = np.nanmin(calc_data)
/usr/local/lib/python3.10/site-packages/seaborn/matrix.py:207: RuntimeWarning: All-NaN slice encountered
vmax = np.nanmax(calc_data)
/usr/local/lib/python3.10/site-packages/seaborn/matrix.py:202: RuntimeWarning: All-NaN slice encountered
vmin = np.nanmin(calc_data)
/usr/local/lib/python3.10/site-packages/seaborn/matrix.py:207: RuntimeWarning: All-NaN slice encountered
vmax = np.nanmax(calc_data)
/work/src/analysis/plotting.py:159: FutureWarning:

The ci parameter is deprecated. Use errorbar=('ci', 95) for the same effect.

[gtt1995]

gcr.io/fuzzbench/runners/aflplusplus/ffmpeg_ffmpeg_demuxer_fuzzer_7adeef latest 60554fb54562 3 hours ago 4GB
gcr.io/fuzzbench/builders/aflplusplus/ffmpeg_ffmpeg_demuxer_fuzzer_7adeef latest b378c42f2c9e 3 hours ago 14.5GB
gcr.io/fuzzbench/builders/coverage/ffmpeg_ffmpeg_demuxer_fuzzer_7adeef latest cd3b4ee97768 4 hours ago 11.4GB
gcr.io/fuzzbench/builders/coverage/ffmpeg_ffmpeg_demuxer_fuzzer_7adeef-intermediate latest 8f431ebab65e 36 hours ago 5.22GB
gcr.io/fuzzbench/builders/aflplusplus/ffmpeg_ffmpeg_demuxer_fuzzer_7adeef-intermediate latest 860adc9b90eb 2 days ago 6.03GB
gcr.io/fuzzbench/builders/benchmark/ffmpeg_ffmpeg_demuxer_fuzzer_7adeef latest 9307945cc40b 2 days ago 5.22GB
gcr.io/fuzzbench/runners/aflplusplus/ffmpeg_ffmpeg_demuxer_fuzzer_7adeef-intermediate latest 4634e7ff4a3a 2 days ago 3.02GB

[gtt1995]

@DonggeLiu Excuse me, thanks.

[DonggeLiu]

Hi @gtt1995,

I suspect the main common is libc++.so.1: cannot open shared object file: No such file or directory from:

ERROR:root:Coverage run failed. Extras: {'coverage_binary': '/work/coverage-binaries/lcms_cms_transform_all_fuzzer_97d37d/cms_transform_all_fuzzer', 'output': '/work/coverage-binaries/lcms_cms_transform_all_fuzzer_97d37d/cms_transform_all_fuzzer: error while loading shared libraries: libc++.so.1: cannot open shared object file: No such file or directory\n'}
ERROR:root:Coverage run failed. Extras: {'coverage_binary': '/work/coverage-binaries/aspell_aspell_fuzzer_e8eb74/aspell_fuzzer', 'output': '/work/coverage-binaries/aspell_aspell_fuzzer_e8eb74/aspell_fuzzer: error while loading shared libraries: libc++.so.1: cannot open shared object file: No such file or directory\n'}

Others are likely either the side effects of this, or benign warnings.

However, it's hard to further debug the error by looking at the messages above.
Here are some suggestions to help:

1. To check if the error above is caused by missing the required libraries, you can run make-<fuzzer-name>-<benchmark-name> locally. fuzzer-name can be afl or libfuzzer, because they are default fuzzers; benchmark-name can be lcms_cms_transform_all_fuzzer_97d37d, ffmpeg_ffmpeg_demuxer_fuzzer_7adeef, and aspell_aspell_fuzzer_e8eb74, given they are erroneous in your experiment.
2. Similarly, you can test your fuzzer out on those benchmarks independently to see if your fuzzer can compile/run on them.
3. Depending on your local experiment config, the build logs of your local experiments above can be found here. It should follow the same structure as cloud experiments. Files in build-logs/ might show some clues too (e.g., benchmark-lcms_cms_transform_fuzzer-coverage.txt).

[gtt1995]

Hi dongge, previous issue has addressed, now a new issue appeared:
INFO:root:Done building fuzzer benchmarks.
ERROR:root:Error conducting experiment. Extras: {'traceback': 'Traceback (most recent call last):\n File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\nsqlite3.OperationalError: table trial has no column named trial_group_num\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File "/work/src/experiment/dispatcher.py", line 201, in main\n dispatcher_main()\n File "/work/src/experiment/dispatcher.py", line 154, in dispatcher_main\n _initialize_trials_in_db(trials)\n File "/work/src/experiment/dispatcher.py", line 90, in _initialize_trials_in_db\n db_utils.bulk_save(trials)\n File "/work/src/database/utils.py", line 96, in bulk_save\n scoped_session.bulk_save_objects(entities)\n File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 3646, in bulk_save_objects\n self.bulk_save_mappings(\n File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 3861, in bulk_save_mappings\n with util.safe_reraise():\n File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 70, in exit\n compat.raise(\n File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 208, in raise\n raise exception\n File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 3850, in _bulk_save_mappings\n persistence._bulk_insert(\n File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/persistence.py", line 107, in _bulk_insert\n _emit_insert_statements(\n File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements\n c = connection._execute_20(\n File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20\n return meth(self, args_10style, kwargs_10style, execution_options)\n File "/usr/local/lib/python3.10/site-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection\n return connection._execute_clauseelement(\n File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement\n ret = self._execute_context(\n File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context\n self.handle_dbapi_exception(\n File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2124, in handle_dbapi_exception\n util.raise(\n File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 208, in raise\n raise exception\n File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\nsqlalchemy.exc.OperationalError: (sqlite3.OperationalError) table trial has no column named trial_group_num\n[SQL: INSERT INTO trial (fuzzer, experiment, benchmark, preemptible, preempted, trial_group_num) VALUES (?, ?, ?, ?, ?, ?)]\n[parameters: ('aflplusplus', 'buglcms', 'lcms_cms_transform_all_fuzzer_97d37d', 0, 0, 0)]\n(Background on this error at: https://sqlalche.me/e/14/e3q8)\n'}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
self.dialect.do_execute(
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
cursor.execute(statement, parameters)
sqlite3.OperationalError: table trial has no column named trial_group_num

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/work/src/experiment/dispatcher.py", line 219, in
sys.exit(main())
File "/work/src/experiment/dispatcher.py", line 204, in main
raise error
File "/work/src/experiment/dispatcher.py", line 201, in main
dispatcher_main()
File "/work/src/experiment/dispatcher.py", line 154, in dispatcher_main
_initialize_trials_in_db(trials)
File "/work/src/experiment/dispatcher.py", line 90, in _initialize_trials_in_db
db_utils.bulk_save(trials)
File "/work/src/database/utils.py", line 96, in bulk_save
scoped_session.bulk_save_objects(entities)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 3646, in bulk_save_objects
self.bulk_save_mappings(
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 3861, in bulk_save_mappings
with util.safe_reraise():
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 70, in exit
compat.raise(
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 208, in raise
raise exception
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 3850, in _bulk_save_mappings
persistence._bulk_insert(
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/persistence.py", line 107, in _bulk_insert
_emit_insert_statements(
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements
c = connection._execute_20(
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20
return meth(self, args_10style, kwargs_10style, execution_options)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection
return connection._execute_clauseelement(
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement
ret = self._execute_context(
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context
self.handle_dbapi_exception(
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2124, in handle_dbapi_exception
util.raise(
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 208, in raise
raise exception
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
self.dialect.do_execute(
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) table trial has no column named trial_group_num
[SQL: INSERT INTO trial (fuzzer, experiment, benchmark, preemptible, preempted, trial_group_num) VALUES (?, ?, ?, ?, ?, ?)]
[parameters: ('aflplusplus', 'buglcms', 'lcms_cms_transform_all_fuzzer_97d37d', 0, 0, 0)]
(Background on this error at: https://sqlalche.me/e/14/e3q8)

@DonggeLiu i am sorry to bother you again, thanks

[renatahodovan]

@gtt1995 What was the solution for the last issue with "table trial has no column named trial_group_num" ?