DAOS-6159 test: Run fault injection tests in parallel and prepare for IL testing. by ashleypittman · Pull Request #3902 · daos-stack/daos

ashleypittman · 2020-11-18T12:03:01Z

Update to the fault injection testing to run multiple instances in
parallel, up to the number of cores per node. This dramatically
increases the speed of testing, and cuts down time-to-result.

Add the ability to run a read through the interception library under
fault injection, but do not enable this yet as it still causes a number
of issues.

Resolve the worst/easiest of the issues with the interception library
and the array object code.

Signed-off-by: Ashley Pittman ashley.m.pittman@intel.com

This changes rather than simply adds testing so isn't ready for merge, however it does find a large number of issues, including some assertion failures, double-frees and deadlocks. Resolve the worst of the issues. Skip-func-hw-test: true Quick-build: true Skip-checkpatch: true Signed-off-by: Ashley Pittman <ashley.m.pittman@intel.com>

daosbuild1 · 2020-11-18T17:03:11Z

Test stage NLT completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-3902/1/execution/node/526/log

Skip-checkpath: true Quick-build: true Skip-func-hw-test: true Signed-off-by: Ashley Pittman <ashley.m.pittman@intel.com>

daosbuild1

Style warning(s) for job https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-3902/2/
Please review https://wiki.hpdd.intel.com/display/DC/Coding+Rules

FYI: Errors found in lines not modified in the patch:

src/tests/ftest/cart/util/cart_logtest.py:480:
(pylint-line-too-long) Line too long (82/80)

src/tests/ftest/cart/util/cart_logtest.py:485:
(pylint-line-too-long) Line too long (96/80)

utils/node_local_test.py:421:
(lint) Remove this block when daos_server shutdown works.

utils/node_local_test.py:469:
(lint) Enable memleak checking when server shutdown works.

utils/node_local_test.py:1207:
(lint) This doesn't work with two pools, partly related to

utils/node_local_test.py:1252:
(lint) change this to something else, md5sum uses fread which isn't

utils/node_local_test.py:1465:
(pylint-line-too-long) Line too long (81/80)

utils/node_local_test.py:493:
(pylint-protected-access) Access to a protected member _daos of a client class

daosbuild1 · 2020-11-19T14:12:58Z

Test stage checkpatch completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-3902/2/execution/node/57/log

Signed-off-by: Ashley Pittman <ashley.m.pittman@intel.com>

Skip-func-hw-test: true Signed-off-by: Ashley Pittman <ashley.m.pittman@intel.com>

Updated patch

daosbuild1

LGTM. No errors found by checkpatch.

FYI: Errors found in lines not modified in the patch:

src/tests/ftest/cart/util/cart_logtest.py:480:
(pylint-line-too-long) Line too long (82/80)

src/tests/ftest/cart/util/cart_logtest.py:485:
(pylint-line-too-long) Line too long (96/80)

utils/node_local_test.py:421:
(lint) Remove this block when daos_server shutdown works.

utils/node_local_test.py:469:
(lint) Enable memleak checking when server shutdown works.

utils/node_local_test.py:1223:
(lint) This doesn't work with two pools, partly related to

utils/node_local_test.py:1267:
(lint) change this to something else, md5sum uses fread which isn't

utils/node_local_test.py:1480:
(pylint-line-too-long) Line too long (81/80)

utils/node_local_test.py:500:
(pylint-protected-access) Access to a protected member _daos of a client class

daosbuild1 · 2020-11-20T11:19:30Z

Test stage Build DEB on Ubuntu 20.04 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-3902/3/execution/node/302/log

daosbuild1 · 2020-11-20T11:21:21Z

Test stage Build RPM on CentOS 7 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-3902/3/execution/node/238/log

daosbuild1 · 2020-11-20T11:24:25Z

Test stage Build RPM on Leap 15 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-3902/3/execution/node/266/log

Skip-func-hw-test: true Signed-off-by: Ashley Pittman <ashley.m.pittman@intel.com>

daosbuild1 · 2020-11-20T20:43:42Z

Test stage checkpatch completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-3902/4/execution/node/57/log

daosbuild1 · 2020-11-20T20:50:08Z

Test stage Build DEB on Ubuntu 20.04 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-3902/4/execution/node/255/log

daosbuild1 · 2020-11-20T20:51:24Z

Test stage Build RPM on CentOS 7 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-3902/4/execution/node/266/log

daosbuild1 · 2020-11-20T20:55:36Z

Test stage Build RPM on Leap 15 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-3902/4/execution/node/275/log

daosbuild1

Style warning(s) for job https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-3902/5/
Please review https://wiki.hpdd.intel.com/display/DC/Coding+Rules

FYI: Errors found in lines not modified in the patch:

src/tests/ftest/cart/util/cart_logtest.py:480:
(pylint-line-too-long) Line too long (82/80)

src/tests/ftest/cart/util/cart_logtest.py:485:
(pylint-line-too-long) Line too long (96/80)

utils/node_local_test.py:421:
(lint) Remove this block when daos_server shutdown works.

utils/node_local_test.py:469:
(lint) Enable memleak checking when server shutdown works.

utils/node_local_test.py:1223:
(lint) This doesn't work with two pools, partly related to

utils/node_local_test.py:1267:
(lint) change this to something else, md5sum uses fread which isn't

utils/node_local_test.py:1482:
(pylint-line-too-long) Line too long (81/80)

utils/node_local_test.py:500:
(pylint-protected-access) Access to a protected member _daos of a client class

daosbuild1 · 2020-11-23T15:44:19Z

Test stage checkpatch completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-3902/5/execution/node/57/log

daosbuild1 · 2020-11-23T15:49:54Z

Test stage Build RPM on CentOS 7 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-3902/5/execution/node/244/log

daosbuild1 · 2020-11-23T15:50:56Z

Test stage Build DEB on Ubuntu 20.04 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-3902/5/execution/node/309/log

daosbuild1 · 2020-11-23T15:54:24Z

Test stage Build RPM on Leap 15 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-3902/5/execution/node/260/log

daosbuild1 · 2020-11-23T15:58:42Z

Test stage Build on CentOS 7 debug completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-3902/5/execution/node/328/log

daosbuild1 · 2020-11-23T15:58:56Z

Test stage Build on CentOS 7 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-3902/5/execution/node/332/log

daosbuild1 · 2020-11-23T15:59:17Z

Test stage Build on CentOS 7 release completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-3902/5/execution/node/342/log

daosbuild1 · 2020-11-23T16:00:32Z

Test stage Build on Ubuntu 20.04 with Clang completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-3902/5/execution/node/263/log

daosbuild1 · 2020-11-23T16:01:00Z

Test stage Build on Leap 15 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-3902/5/execution/node/293/log

daosbuild1 · 2020-11-23T16:02:43Z

Test stage Build on Leap 15 with Intel-C and TARGET_PREFIX completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-3902/5/execution/node/312/log

daosbuild1

LGTM. No errors found by checkpatch.

FYI: Errors found in lines not modified in the patch:

utils/node_local_test.py:1:
(pylint-missing-docstring) Missing module docstring

utils/node_local_test.py:3:
(pylint-pointless-string-statement) String statement has no effect

daosbuild1

Style warning(s) for job https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-3902/40/
Please review https://wiki.hpdd.intel.com/display/DC/Coding+Rules

daosbuild1 · 2021-01-25T22:34:12Z

Test stage checkpatch completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-3902/40/execution/node/67/log

Signed-off-by: Ashley Pittman <ashley.m.pittman@intel.com>

Updated patch

daosbuild1

LGTM. No errors found by checkpatch.

Signed-off-by: Ashley Pittman <ashley.m.pittman@intel.com>

daosbuild1

LGTM. No errors found by checkpatch.

ashleypittman · 2021-02-01T08:37:50Z

I've updated the title and first comment of this PR and it should now be ready for review. This PR improves run-time and allows us to enable more testing, however if that testing is enabled them multiple errors are encountered which causes failures so enabling this new test and fixing these errors is being handled separately.

jolivier23 · 2021-02-01T16:26:35Z

+
+        fatal_errors = False
+
+        while not finished or active:


I personally think this is easier to read if you add parenthesis around "not finished" so one doesn't need to remember order of operations rules

I think pylint would warn if I tried that, but I'll address if I need to revise this.

daltonbohning

I had a few questions, but otherwise LGTM.

ashleypittman added 2 commits November 19, 2020 13:52

Merge branch 'master' into nlt-fi-il

8fed144

Increase log-file size.

bb4c2c9

Skip-checkpath: true Quick-build: true Skip-func-hw-test: true Signed-off-by: Ashley Pittman <ashley.m.pittman@intel.com>

daosbuild1 previously requested changes Nov 19, 2020

View reviewed changes

Comment thread utils/node_local_test.py Outdated

Comment thread utils/node_local_test.py

ashleypittman added 2 commits November 19, 2020 15:21

Fix checkpatch warnings.

5c21e55

Signed-off-by: Ashley Pittman <ashley.m.pittman@intel.com>

Merge branch 'master' into nlt-fi-il

4588066

Skip-func-hw-test: true Signed-off-by: Ashley Pittman <ashley.m.pittman@intel.com>

daosbuild1 reviewed Nov 20, 2020

View reviewed changes

Avoid deadlock at startup.

9179f04

Skip-func-hw-test: true Signed-off-by: Ashley Pittman <ashley.m.pittman@intel.com>

daosbuild1 previously requested changes Nov 23, 2020

View reviewed changes

Comment thread src/client/array/dc_array.c Outdated

Comment thread src/client/dfuse/il/int_posix.c Outdated

ashleypittman force-pushed the nlt-fi-il branch from b31ae64 to 1d215a5 Compare November 23, 2020 16:03

daosbuild1 reviewed Jan 25, 2021

View reviewed changes

Merge branch 'master' into nlt-fi-il

3d55b93

daosbuild1 previously requested changes Jan 25, 2021

View reviewed changes

Comment thread utils/node_local_test.py Outdated

ashleypittman added 3 commits January 26, 2021 09:25

Merge branch 'master' into nlt-fi-il

37d1f98

Fix bad merge.

359a309

Signed-off-by: Ashley Pittman <ashley.m.pittman@intel.com>

Merge branch 'master' into nlt-fi-il

8359502

daosbuild1 reviewed Jan 28, 2021

View reviewed changes

ashleypittman requested review from daltonbohning, jolivier23, liuxuezhao, mchaarawi and phender January 29, 2021 17:21

ashleypittman added 2 commits January 29, 2021 17:24

Merge branch 'master' into nlt-fi-il

82b78ce

Complete active commands.

c0c998f

Signed-off-by: Ashley Pittman <ashley.m.pittman@intel.com>

jolivier23 previously approved these changes Jan 29, 2021

View reviewed changes

Merge branch 'master' into nlt-fi-il

edf8302

ashleypittman dismissed jolivier23’s stale review via edf8302 January 30, 2021 14:30

daosbuild1 reviewed Jan 30, 2021

View reviewed changes

ashleypittman requested a review from jolivier23 February 1, 2021 08:33

ashleypittman changed the title ~~DAOS-6159 test: Add IL to fault injection testing~~ DAOS-6159 test: Run fault injection tests in parallel and prepare for IL testing. Feb 1, 2021

jolivier23 approved these changes Feb 1, 2021

View reviewed changes

ashleypittman requested a review from mjmac February 2, 2021 15:41

daltonbohning approved these changes Feb 2, 2021

View reviewed changes

Comment thread src/client/array/dc_array.c

Comment thread src/client/dfuse/il/int_posix.c

Comment thread utils/node_local_test.py

Comment thread utils/node_local_test.py

ashleypittman requested a review from a team February 2, 2021 17:20

ashleypittman merged commit ff8acc8 into master Feb 2, 2021

ashleypittman deleted the nlt-fi-il branch February 2, 2021 17:34

Conversation

ashleypittman commented Nov 18, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

daosbuild1 commented Nov 18, 2020

Uh oh!

daosbuild1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

daosbuild1 commented Nov 19, 2020

Uh oh!

daosbuild1 left a comment

Choose a reason for hiding this comment

Uh oh!

daosbuild1 commented Nov 20, 2020

Uh oh!

daosbuild1 commented Nov 20, 2020

Uh oh!

daosbuild1 commented Nov 20, 2020

Uh oh!

daosbuild1 commented Nov 20, 2020

Uh oh!

daosbuild1 commented Nov 20, 2020

Uh oh!

daosbuild1 commented Nov 20, 2020

Uh oh!

daosbuild1 commented Nov 20, 2020

Uh oh!

daosbuild1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

daosbuild1 commented Nov 23, 2020

Uh oh!

daosbuild1 commented Nov 23, 2020

Uh oh!

daosbuild1 commented Nov 23, 2020

Uh oh!

daosbuild1 commented Nov 23, 2020

Uh oh!

daosbuild1 commented Nov 23, 2020

Uh oh!

daosbuild1 commented Nov 23, 2020

Uh oh!

daosbuild1 commented Nov 23, 2020

Uh oh!

daosbuild1 commented Nov 23, 2020

Uh oh!

daosbuild1 commented Nov 23, 2020

Uh oh!

daosbuild1 commented Nov 23, 2020

Uh oh!

daosbuild1 left a comment

Choose a reason for hiding this comment

Uh oh!

daosbuild1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

daosbuild1 commented Jan 25, 2021

Uh oh!

daosbuild1 left a comment

Choose a reason for hiding this comment

Uh oh!

daosbuild1 left a comment

Choose a reason for hiding this comment

Uh oh!

ashleypittman commented Feb 1, 2021

Uh oh!

jolivier23 Feb 1, 2021

Choose a reason for hiding this comment

Uh oh!

ashleypittman Feb 1, 2021

Choose a reason for hiding this comment

Uh oh!

daltonbohning left a comment

Choose a reason for hiding this comment

ashleypittman commented Nov 18, 2020 •

edited

Loading