DAOS-5415 test: System reformat with restart of all system ranks by amjustin13 · Pull Request #3263 · daos-stack/daos

amjustin13 · 2020-08-14T19:44:23Z

Start and format daos_server instances which result in io_server instances running
Create first pool at 90% capacity
Try to create second pool at 90% and verify that pool_create fails with DER_NOSPACE
dmg system stop –force (stop all system ranks' processes)
dmg storage format –reformat (reformat and restart all system ranks)
o The MS leader/access point daos_server instance (all daos_server instances are still running at this point) will reformat storage for all registered ranks (that have previously been started) by using information stored in the membership.
o The reformat and restart is automatic from this point on , and the end result should be that io_servers that were stopped get started with clean storage.
Verify that pools have been wiped, create a new pool to verify we have our space back.

In addition, I'm removing the dmg_storage_reformat test as it is a duplicate of a test added in DAOS-3596.

Added @skipForTicket("DAOS-6004") tag on test case until DAOS-6004 is resolved and can remove skip tag.

Signed-off-by: Justiniano-pagn amanda.justiniano-pagn@intel.com

Signed-off-by: Justiniano-pagn <amanda.justiniano-pagn@intel.com>

daosbuild1

Style warning(s) for job https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-3263/1/
Please review https://wiki.hpdd.intel.com/display/DC/Coding+Rules

daosbuild1 · 2020-08-14T20:03:57Z

Test stage checkpatch completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-3263/1/execution/node/60/log

Signed-off-by: Justiniano-pagn <amanda.justiniano-pagn@intel.com>

Updated patch

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1 · 2020-08-15T04:01:49Z

Test stage Functional_Hardware_Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-3263/2/execution/node/912/log

Signed-off-by: Justiniano-pagn <amanda.justiniano-pagn@intel.com>

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1 · 2020-08-17T15:20:12Z

Test stage Functional_Hardware_Small completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-3263/3/execution/node/780/log

daosbuild1 · 2020-08-17T17:29:45Z

Test stage Functional_Hardware_Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-3263/3/execution/node/859/log

Signed-off-by: Justiniano-pagn <amanda.justiniano-pagn@intel.com>

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1 · 2020-08-17T20:54:55Z

Test stage Functional_Hardware_Small completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-3263/5/execution/node/818/log

amjustin13 · 2020-08-19T13:39:45Z

@tanabarr , here's the info I'm getting on the log of the test, does it look right? After this, the server restarts as expected.

Running '/home/amjustin/daos/install/bin/dmg -o /home/amjustin/daos/install/tmp/test_daos_server_dmg.yaml storage format --reformat'                                                        
[stdout] [1,0]<stdout>:DEBUG 15:43:11.427776 ctl_system.go:187: Received SystemQuery RPC                                                                                                    
[stdout] [1,0]<stdout>:DEBUG 15:43:11.427951 system.go:688: DAOS system ping-ranks request: &{unaryRequest:{request:{HostList:[10.7.1.10:10001]} rpc:0x9055c0} Ranks:0 Force:false}         
[stdout] [1,0]<stdout>:DEBUG 15:43:11.428161 rpc.go:183: request hosts: [10.7.1.10:10001]                                                                                                   
[stdout] [1,0]<stdout>:DEBUG 15:43:11.474221 mgmt_system.go:313: MgmtSvc.PingRanks dispatch, req:{Force:false Ranks:0 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}          
[stdout] [1,0]<stdout>:DEBUG 15:43:11.474351 mgmt_system.go:325: MgmtSvc.PingRanks dispatch, resp:{Results:[msg:"/home/amjustin/daos/install/bin/daos_io_server (instance 0) exited: signal: aborted" state:6 ] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}                                                                                                            
[stdout] [1,0]<stdout>:DEBUG 15:43:11.474751 ctl_system.go:213: Responding to SystemQuery RPC: members:<addr:"10.7.1.10:10001" uuid:"2941f0c6-7cd7-41bd-a9ea-214ecb6cf390" state:6 info:"system stop" >                                                                                                                                                                                 
[stdout] [1,0]<stdout>:DEBUG 15:43:11.547967 ctl_system.go:342: Received SystemResetFormat RPC                                                                                              
[stdout] [1,0]<stdout>:DEBUG 15:43:11.548127 system.go:646: DAOS system reset-format-ranks request: &{unaryRequest:{request:{HostList:[10.7.1.10:10001]} rpc:0x9053a0} Ranks:0 Force:false} 
[stdout] [1,0]<stdout>:DEBUG 15:43:11.548321 rpc.go:183: request hosts: [10.7.1.10:10001]                                                                                                   
[stdout] [1,0]<stdout>:DEBUG 15:43:11.599272 mgmt_system.go:346: MgmtSvc.ResetFormatRanks dispatch, req:{Force:false Ranks:0 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}   
[stdout] [1,0]<stdout>:DEBUG 15:43:11.599391 instance_exec.go:49: instance 0: checking if storage is formatted                                                                              
[stdout] [1,0]<stdout>:Waiting for DAOS I/O Server instance 0 storage to be ready...                                                                                                        
[stdout] [1,0]<stdout>:DEBUG 15:43:11.599464 instance_storage.go:91: /mnt/daos: checking formatting                                                                                         
[stdout] [1,0]<stdout>:DEBUG 15:43:11.627148 instance_storage.go:107: /mnt/daos (ram) needs format: false                                                                                   
[stdout] [1,0]<stdout>:DEBUG 15:43:11.627182 instance_storage.go:138: instance 0: no SCM format required; checking for superblock                                                           
[stdout] [1,0]<stdout>:DEBUG 15:43:11.627201 instance_superblock.go:109: /mnt/daos: checking superblock                                                                                     
[stdout] [1,0]<stdout>:DEBUG 15:43:11.627362 instance_superblock.go:113: /mnt/daos: needs superblock (doesn't exist)                                                                        
[stdout] [1,0]<stdout>:Metadata format required on instance 0                                                                                                                               
[stdout] [1,0]<stdout>:DEBUG 15:43:12.099611 mgmt_system.go:394: MgmtSvc.ResetFormatRanks dispatch, resp:{Results:[state:1 ] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}   
[stdout] [1,0]<stdout>:DEBUG 15:43:12.100014 ctl_system.go:368: Responding to SystemResetFormat RPC: results:<action:"reset format" state:1 addr:"10.7.1.10:10001" >                        
[stdout] [1,0]<stdout>:DEBUG 15:43:12.169630 ctl_storage_rpc.go:201: received StorageFormat RPC reformat:true ; proceeding to instance storage format                                       
[stdout] [1,0]<stdout>:Formatting SCM storage for DAOS I/O Server instance 0 (reformat: true)                                                                                               
[stdout] [1,0]<stdout>:Instance 0: starting format of SCM (ram:/mnt/daos)                                                                                                                   
[stdout] [1,0]<stdout>:Instance 0: finished format of SCM (ram:/mnt/daos)                                                                                                                   
[stdout] [1,0]<stdout>:Formatting NVMe storage for DAOS I/O Server instance 0                                                                                                               
[stdout] [1,0]<stdout>:DEBUG 15:43:12.629963 instance_superblock.go:109: /mnt/daos: checking superblock                                                                                     
[stdout] [1,0]<stdout>:DEBUG 15:43:12.630190 instance_superblock.go:113: /mnt/daos: needs superblock (doesn't exist)                                                                        
[stdout] [1,0]<stdout>:Instance 0: starting format of nvme block devices [0000:81:00.0]                                                                                                     
[stdout] [1,0]<stdout>:Instance 0: finished format of nvme block devices [0000:81:00.0]                                                                                                     
[stdout] [1,0]<stdout>:DAOS I/O Server instance 0 storage ready                                                                                                                             
[stdout] Format Summary:                                                                                                                                                                    
[stdout] [1,0]<stdout>:DEBUG 15:43:13.062652 instance_superblock.go:109: /mnt/daos: checking superblock                                                                                     
[stdout]   Hosts     SCM Devices NVMe Devices                                                                                                                                               
[stdout]   -----     ----------- ------------                                                                                                                                               
[stdout] [1,0]<stdout>:DEBUG 15:43:13.063088 instance_superblock.go:113: /mnt/daos: needs superblock (doesn't exist)                                                                        
[stdout]   10.7.1.10 1           1                                                                                                                                                          
Command '/home/amjustin/daos/install/bin/dmg -o /home/amjustin/daos/install/tmp/test_daos_server_dmg.yaml storage format --reformat' finished with 0 after 1.7140610218s

Signed-off-by: Justiniano-pagn <amanda.justiniano-pagn@intel.com>

daosbuild1

Style warning(s) for job https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-3263/7/
Please review https://wiki.hpdd.intel.com/display/DC/Coding+Rules

daosbuild1 · 2020-08-19T19:48:46Z

Test stage checkpatch completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-3263/7/execution/node/61/log

Signed-off-by: Justiniano-pagn <amanda.justiniano-pagn@intel.com>

Updated patch

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1 · 2020-08-19T22:25:56Z

Test stage Functional_Hardware_Small completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-3263/8/execution/node/818/log

Signed-off-by: Justiniano-pagn <amanda.justiniano-pagn@intel.com>

daosbuild1

LGTM. No errors found by checkpatch.

Signed-off-by: Justiniano-pagn <amanda.justiniano-pagn@intel.com>

daosbuild1 · 2020-08-20T15:48:48Z

Test stage Functional_Hardware_Small completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-3263/9/display/redirect

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1 · 2020-08-20T19:14:35Z

Test stage Functional_Hardware_Small completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-3263/10/execution/node/818/log

daosbuild1 · 2020-11-13T08:02:09Z

Test stage Functional_Hardware_Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-3263/48/execution/node/1061/log

daosbuild1 · 2020-11-13T13:39:56Z

Test stage Functional_Hardware_Large completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-3263/48/execution/node/1125/log

Signed-off-by: Justiniano-pagn <amanda.justiniano-pagn@intel.com>

daosbuild1

LGTM. No errors found by checkpatch.

FYI: Errors found in lines not modified in the patch:

src/tests/ftest/util/control_test_base.py:28:
(pylint-unused-import) Unused get_host_data imported from general_utils

daosbuild1 · 2020-11-16T17:55:34Z

Test stage Functional_Hardware_Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-3263/49/execution/node/1076/log

shimizukko

LGTM

Signed-off-by: Justiniano-pagn <amanda.justiniano-pagn@intel.com>

daosbuild1

LGTM. No errors found by checkpatch.

FYI: Errors found in lines not modified in the patch:

src/tests/ftest/util/control_test_base.py:28:
(pylint-unused-import) Unused get_host_data imported from general_utils

Signed-off-by: Justiniano-pagn <amanda.justiniano-pagn@intel.com>

daosbuild1

Style warning(s) for job https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-3263/51/
Please review https://wiki.hpdd.intel.com/display/DC/Coding+Rules

FYI: Errors found in lines not modified in the patch:

src/tests/ftest/util/control_test_base.py:28:
(pylint-unused-import) Unused get_host_data imported from general_utils

daosbuild1 · 2020-11-17T21:12:48Z

Test stage checkpatch completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-3263/51/execution/node/58/log

Signed-off-by: Justiniano-pagn <amanda.justiniano-pagn@intel.com>

Updated patch

daosbuild1

LGTM. No errors found by checkpatch.

FYI: Errors found in lines not modified in the patch:

src/tests/ftest/util/control_test_base.py:28:
(pylint-unused-import) Unused get_host_data imported from general_utils

daosbuild1 · 2020-11-18T10:30:31Z

Test stage Functional_Hardware_Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-3263/52/execution/node/1010/log

shimizukko

LGTM

amjustin13 · 2020-11-18T22:56:52Z

Test failures for:

https://jira.hpdd.intel.com/browse/DAOS-6033: rebuild ec tests failing with vos assertion

Functional_Hardware_Medium / DAOS rebuild ec tests.REBUILD9: rebuild2p partial update with data tgt fail
Functional_Hardware_Medium / DAOS rebuild ec tests.REBUILD10: rebuild2p partial update with data/parity tgts fail
Functional_Hardware_Medium / DAOS rebuild ec tests.REBUILD11: rebuild2p partial update with parity tgt fail
Functional_Hardware_Medium / DAOS rebuild ec tests.REBUILD12: rebuild2p partial update with 2 parity tgt fail
Functional_Hardware_Medium / DAOS rebuild ec tests.REBUILD13: rebuild with mixed partial/full stripe

https://jira.hpdd.intel.com/browse/DAOS-5807: OSAOnlineReintegration test failure - "Pool Version Error: After exclude"

Test / Functional_Hardware_Large / 1-./osa/osa_online_reintegration.py:OSAOnlineReintegration.test_osa_online_reintegration

mjmac

Looks good, will land. Just a question for a follow-on PR, perhaps.

mjmac · 2020-11-19T17:29:46Z

+        # To verify that we are using the membership information instead of the
+        # dmg config explicit hostlist
+        # Uncomment below after DAOS-5979 is resolved
+        # self.assertTrue(
+        #     self.server_managers[-1].dmg.set_config_value("hostlist", None))


Should this be removed/reworked?

Add system reformat test.

d4ced7f

Signed-off-by: Justiniano-pagn <amanda.justiniano-pagn@intel.com>

daosbuild1 previously requested changes Aug 14, 2020

View reviewed changes

Fix pylint issues.

3c59c2b

Signed-off-by: Justiniano-pagn <amanda.justiniano-pagn@intel.com>

daosbuild1 reviewed Aug 14, 2020

View reviewed changes

Modify yaml file, modifying test tag.

35a16fb

Signed-off-by: Justiniano-pagn <amanda.justiniano-pagn@intel.com>

daosbuild1 reviewed Aug 17, 2020

View reviewed changes

Justiniano-pagn added 2 commits August 17, 2020 14:03

Rename files to appropriate test name.

48d9664

Signed-off-by: Justiniano-pagn <amanda.justiniano-pagn@intel.com>

Merge branch 'master' into amjustin/DAOS-5415

ab17639

daosbuild1 reviewed Aug 17, 2020

View reviewed changes

Justiniano-pagn added 2 commits August 19, 2020 15:26

Fix for test do that it searched for the io_server start up correctly.

2f2f172

Signed-off-by: Justiniano-pagn <amanda.justiniano-pagn@intel.com>

Add system stop force flag.

f164442

Signed-off-by: Justiniano-pagn <amanda.justiniano-pagn@intel.com>

daosbuild1 previously requested changes Aug 19, 2020

View reviewed changes

Comment thread src/tests/ftest/control/dmg_system_reformat.py Outdated

Comment thread src/tests/ftest/control/dmg_system_reformat.py Outdated

Comment thread src/tests/ftest/util/apricot/apricot/test.py Outdated

Fix pylint issues.

6bd20fa

Signed-off-by: Justiniano-pagn <amanda.justiniano-pagn@intel.com>

daosbuild1 reviewed Aug 19, 2020

View reviewed changes

Fix typo on test yaml file.

69d353e

Signed-off-by: Justiniano-pagn <amanda.justiniano-pagn@intel.com>

daosbuild1 reviewed Aug 20, 2020

View reviewed changes

Match test yaml file with local env.

cee2d57

Signed-off-by: Justiniano-pagn <amanda.justiniano-pagn@intel.com>

daosbuild1 reviewed Aug 20, 2020

View reviewed changes

Justiniano-pagn added 2 commits November 16, 2020 08:54

Fix list_pools.py test failure, sync with PR2868

c1048f2

Signed-off-by: Justiniano-pagn <amanda.justiniano-pagn@intel.com>

Merge branch 'master' into amjustin/DAOS-5415

ac0ca91

daosbuild1 reviewed Nov 16, 2020

View reviewed changes

amjustin13 requested review from phender, shimizukko and tanabarr November 16, 2020 19:16

shimizukko previously approved these changes Nov 16, 2020

View reviewed changes

tanabarr previously approved these changes Nov 16, 2020

View reviewed changes

Merge branch 'master' into amjustin/DAOS-5415

5f31d64

Signed-off-by: Justiniano-pagn <amanda.justiniano-pagn@intel.com>

amjustin13 dismissed stale reviews from tanabarr and shimizukko via 5f31d64 November 17, 2020 19:56

daosbuild1 reviewed Nov 17, 2020

View reviewed changes

Fix docstring on server_utils on feedback.

9ed85a4

Signed-off-by: Justiniano-pagn <amanda.justiniano-pagn@intel.com>

daosbuild1 previously requested changes Nov 17, 2020

View reviewed changes

Comment thread src/tests/ftest/util/server_utils.py Outdated

Fix typo.

b0e6db0

Signed-off-by: Justiniano-pagn <amanda.justiniano-pagn@intel.com>

daosbuild1 reviewed Nov 17, 2020

View reviewed changes

phender approved these changes Nov 18, 2020

View reviewed changes

amjustin13 requested a review from shimizukko November 18, 2020 19:46

shimizukko approved these changes Nov 18, 2020

View reviewed changes

amjustin13 requested a review from a team November 18, 2020 22:57

mjmac approved these changes Nov 19, 2020

View reviewed changes

mjmac merged commit b8d568d into master Nov 19, 2020

mjmac deleted the amjustin/DAOS-5415 branch November 19, 2020 17:41

Conversation

amjustin13 commented Aug 14, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

daosbuild1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

daosbuild1 commented Aug 14, 2020

Uh oh!

daosbuild1 left a comment

Choose a reason for hiding this comment

Uh oh!

daosbuild1 commented Aug 15, 2020

Uh oh!

daosbuild1 left a comment

Choose a reason for hiding this comment

Uh oh!

daosbuild1 commented Aug 17, 2020

Uh oh!

daosbuild1 commented Aug 17, 2020

Uh oh!

daosbuild1 left a comment

Choose a reason for hiding this comment

Uh oh!

daosbuild1 commented Aug 17, 2020

Uh oh!

amjustin13 commented Aug 19, 2020

Uh oh!

daosbuild1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

daosbuild1 commented Aug 19, 2020

Uh oh!

daosbuild1 left a comment

Choose a reason for hiding this comment

Uh oh!

daosbuild1 commented Aug 19, 2020

Uh oh!

daosbuild1 left a comment

Choose a reason for hiding this comment

Uh oh!

daosbuild1 commented Aug 20, 2020

Uh oh!

daosbuild1 left a comment

Choose a reason for hiding this comment

Uh oh!

daosbuild1 commented Aug 20, 2020

Uh oh!

daosbuild1 commented Nov 13, 2020

Uh oh!

daosbuild1 commented Nov 13, 2020

Uh oh!

daosbuild1 left a comment

Choose a reason for hiding this comment

Uh oh!

daosbuild1 commented Nov 16, 2020

Uh oh!

shimizukko left a comment

Choose a reason for hiding this comment

Uh oh!

daosbuild1 left a comment

Choose a reason for hiding this comment

Uh oh!

daosbuild1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

daosbuild1 commented Nov 17, 2020

Uh oh!

daosbuild1 left a comment

Choose a reason for hiding this comment

Uh oh!

amjustin13 commented Aug 14, 2020 •

edited

Loading