DAOS-12706 object: Some memory corruption issue by wangdi1 · Pull Request #11624 · daos-stack/daos

wangdi1 · 2023-03-07T00:51:11Z

Fix some memory corruption issuse for csum in
obj_get_iods_offs_by_oid().

Required-githooks: true

Before requesting gatekeeper:

Two review approvals and any prior change requests have been resolved.
Testing is complete and all tests passed or there is a reason documented in the PR why it should be force landed and forced-landing tag is set.
Features: (or Test-tag*) commit pragma was used or there is a reason documented that there are no appropriate tags for this PR.
Commit messages follows the guidelines outlined here.
Any tests skipped by the ticket being addressed have been run and passed in the PR.

Gatekeeper:

github-actions · 2023-03-07T00:51:33Z

Bug-tracker data:
Ticket title is 'soak regression: daos_racer causes daos server to segfault'
Status is 'In Review'
Labels: 'soak,triaged'
Job should run at elevated priority (3)
https://daosio.atlassian.net/browse/DAOS-12706

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1 · 2023-03-07T16:34:04Z

Test stage Functional Hardware Large completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-11624/1/execution/node/1256/log

daosbuild1 · 2023-03-07T17:50:37Z

Test stage Functional Hardware Medium Verbs Provider completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-11624/1/testReport/(root)/

Fix some memory corruption issuse for csum in obj_get_iods_offs_by_oid(). Required-githooks: true Signed-off-by: Di Wang <di.wang@intel.com>

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1 · 2023-03-09T00:29:12Z

Test stage Functional Hardware Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-11624/2/execution/node/1209/log

wangdi1 · 2023-03-10T18:25:24Z

failure due to DAOS-12857

wangdi1 · 2023-03-14T16:44:49Z

@Nasf-Fan @liuxuezhao please review.

Nasf-Fan · 2023-03-15T11:05:27Z

 					   orw->orw_iod_array.oia_iods,
-					   orw->orw_iod_array.oia_iod_nr,
-					   ioc->ioc_coc->sc_csummer,
+					   iods_nr, ioc->ioc_coc->sc_csummer,


why "orw->orw_iod_array.oia_iods" instead of the input "iods"? that is the pairs of "iods_nr".

ah, this is for fetch only, so it will never do leader forward, so obj_get_iods_offs_by_oid() will never be called. i.e.
orw->orw_iod_array.ioa_iods should be always == iods, probably this part does not need change, but harmless for now. @Nasf-Fan

So we may add "D_ASSERT(orw->orw_iod_array.ioa_iods should be always == iods)" ? if has other chance to refresh the patch?

Anyway, your patches seems not resolve the issue:

https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-11624/2/artifact/Functional%20Hardware%20Medium/scrubber/target_auto_eviction.py/stacktraces.wolf-138/core.daos_io_1.1678316154.114222.stacktrace/*view*/

Thread 1 (Thread 0x7f176168f700 (LWP 114471)):
#0 0x00007f17dfe5fa9f in raise () from /lib64/libc.so.6
No symbol table info available.
#1 0x00007f17dfe32e05 in abort () from /lib64/libc.so.6
No symbol table info available.
#2 0x00007f17dfe32cd9 in __assert_fail_base.cold.0 () from /lib64/libc.so.6
No symbol table info available.
#3 0x00007f17dfe583f6 in __assert_fail () from /lib64/libc.so.6
No symbol table info available.
#4 0x0000559e25a607cd in proc_struct_dcs_csum_info_adv (proc=proc@entry=0x7f175ea29f10, proc_op=proc_op@entry=CRT_PROC_ENCODE, csum=0x7f175ed16810, idx=idx@entry=0, nr=) at src/object/rpc_csum.c:65
buf_len = 2
rc = 0
func = "proc_struct_dcs_csum_info_adv"
PRETTY_FUNCTION = "proc_struct_dcs_csum_info_adv"
#5 0x0000559e25a60d4b in proc_struct_dcs_csum_info (proc=proc@entry=0x7f175ea29f10, proc_op=proc_op@entry=CRT_PROC_ENCODE, csum=) at src/object/rpc_csum.c:95
No locals.
#6 0x0000559e25a60de3 in proc_struct_dcs_csum_info (csum=, proc_op=CRT_PROC_ENCODE, proc=0x7f175ea29f10) at src/object/rpc_csum.c:92
No locals.
#7 crt_proc_struct_dcs_csum_info (proc=proc@entry=0x7f175ea29f10, proc_op=, p_csum=p_csum@entry=0x7f175ea8b958) at src/object/rpc_csum.c:110
csum_enabled = true
rc = 0
func = "crt_proc_struct_dcs_csum_info"
#8 0x0000559e25a5190e in crt_proc_obj_rw_in (proc=0x7f175ea29f10, ptr=0x7f175ea8b8b0) at src/object/obj_rpc.c:1079
proc_op = CRT_PROC_ENCODE
rc = 0
func = "crt_proc_obj_rw_in"

This csum assertion is different issue DAOS-12857.

liuxuezhao

just test "orterun -np 1 ./daos_test -I -u 17 -n ./daos_control.yml --csum_type "crc32" " cannot work even after this PR, probably still some problem in singv csum handling.

liuxuezhao · 2023-03-20T09:43:54Z

+				(*csums)[idx].ic_nr = siod->siod_nr;
 			}
 		} else {
 			if (iod_parent->iod_recxs != NULL)


could you please remind me why the iod_parent->iod_recxs can be NON-NULL for single value? seems for singv it should be always NULL, no?

ah, hmm, I checked the original obj_ec_rw_req_split(), which does not include this iod_recxs check under SINGLE branch, so I had thought iod_recxs is being reused for some cases. ok, I can remove this check if the patch needs to be refreshed.

wangdi1 · 2023-03-20T20:50:07Z

just test "orterun -np 1 ./daos_test -I -u 17 -n ./daos_control.yml --csum_type "crc32" " cannot work even after this PR, probably still some problem in singv csum handling.

Just checked a bit, I think the failure is due to the incorrect the iod_size handling for single value, instead of these get_offs changes.

wangdi1 · 2023-03-21T15:41:05Z

failure due to daos-12706

* DAOS-11939 test: Adding a test to identify bad node on basis of performance (#10730) Adding a test to deployment dir to identify nodes with bad performance. In this test ior job is run for each rank individually and the performance is compared for each rank with expected values for different transfer sizes. Signed-off-by: Saurabh Tandan <saurabh.tandan@intel.com> * DAOS-12938 build: Update VERSION to 2.3.106 (#11728) Updated to match the RPM version. Signed-off-by: Kris Jacque <kristin.jacque@intel.com> * DAOS-623 test: Remove bandit checks B309,b325 . (#11742) This check is no longer used and has been removed from bandit so do not tell bandit to use this. There is no change in coverage here, bandit has been updated to remove a no-longer-valid check so daos needs updating for the new bandit version. Signed-off-by: Ashley Pittman <ashley.m.pittman@intel.com> * DAOS-12561 control: Consistently format pool tier ratios (#11724) Use a common helper when printing tier ratio values to ensure consistent output. Also adds more comprehensive unit testing for the tierRatioFlag type. Signed-off-by: Michael MacDonald <mjmac.macdonald@intel.com> * DAOS-12723 test: fix fio command string (#11710) Fix fio command string, broken by 7d624c0. Instead of overriding __str__, override with_bind so all of these properties work as expected: - __str__ - with_bind - with_sudo - with_exports Required-githooks: true Signed-off-by: Dalton Bohning <dalton.bohning@intel.com> * DAOS-12677 test: Update dmg storage query commands to use json output (#11651) Updating the dmg storage query set of commands to produce json output to avoid the pitfalls of searching command text output. Signed-off-by: Phil Henderson <phillip.henderson@intel.com> * DAOS-12948 test: launch.py: use full config path to storage (#11751) When looking for "storage: auto", instead of matching on generic: ["storage"] Match on the more precise: ["server_config", "engines", "*", "storage"] This allows "storage" to be used in yaml paths other than just server_config Signed-off-by: Dalton Bohning <dalton.bohning@intel.com> * DAOS-12577 dfuse: Update dfuse cache on lookup. (#11741) When dfuse replies to a lookup request then save the data it passes to the kernel. Previously only getattr requests were updating the cache so it was possible to populate the cache, have a remote update and then have loopup mark the cache as valid without updating it's contents. This was particuarly visable for writeback cache where the kernel queries dfuse to check the file-size before read and dfuse could return known-stale cache data. Signed-off-by: Ashley Pittman <ashley.m.pittman@intel.com> * DAOS-12638 test: Increase number of cores for dfuse (#11730) The dfuse threads were oversubscribed in soak and appears to have caused dfuse to segfault. Increase the numb of core that dfuse uses so that num cores = num threads. Signed-off-by: Maureen Jean <maureen.jean@intel.com> * DAOS-12699 control: Prevent simultaneous daos_agent processes (#11718) - Detect if another daos_agent is running on startup, and if so exit with error. - Add debug output to daos_agent startup process. This will make it easier to debug startup hangs. Signed-off-by: Kris Jacque <kristin.jacque@intel.com> * DAOS-9721 test: use container label by default with most tests (#11717) - Use a container label by default with all tests. - Except for create through the pydaos.raw API - Except for edge-case tests not using get_container() - Also rename the pool label of TestLabel to TestPool Signed-off-by: Dalton Bohning <dalton.bohning@intel.com> * DAOS-5936 client: Don't allow creating cont with owner (#11701) - Disable creating containers via the API with an owner besides the current user set in the properties. - Update daos_test. Signed-off-by: Kris Jacque <kristin.jacque@intel.com> * DAOS-10079 control: Include owner/group in JSON get-acl output (#11665) - Update the JSON output for ACLs to include the owner user and group. - Modify internal translations of the ACL data. Signed-off-by: Kris Jacque <kristin.jacque@intel.com> * DAOS-12706 object: Some memory corruption issue (#11624) Fix some memory corruption issuse for csum in obj_get_iods_offs_by_oid(). Signed-off-by: Di Wang <di.wang@intel.com> * DAOS-12869 object: shrink layout version to 16 bits (#11674) shrink object layout version to 16 bits. Signed-off-by: Di Wang <di.wang@intel.com> * DAOS-7615 test: Include DM job in soak (#11708) Enable datamover job in soak Signed-off-by: Maureen Jean <maureen.jean@intel.com> * DAOS-12925 test: MOFED 5.8-1.1.2.1 daos_test/dfs.py failures (#11748) Disable openmpi's hardware collectives since there seems to be issues with those on MOFED 5.8-x Signed-off-by: Ding Ho ding-hwa.ho@intel.com * DAOS-12065 security: Add CONT_CAPA_{OPEN_EX,EVICT_ALL} (#11729) Add CONT_CAPA_OPEN_EX and CONT_CAPA_EVICT_ALL and remove the is_owner parameter of ds_sec_cont_get_capabilities. Signed-off-by: Li Wei <wei.g.li@intel.com> * DAOS-12854 control: Allow system query to retry in PoolCreate (#11684) When the pool create size is specified as a percentage, the PoolCreate handler needs to scan available storage across the servers in order to determine how much space can be allocated for the pool. A previous commit added a pre-scan SystemQuery in order to avoid running the scan before the MS was up and running, but that change did not allow the query to be retried, leading to problems when creating pools on larger systems. This commit enables the normal retry mechanism for the pre-scan SystemQuery so that it behaves like most other RPCs. In particular, it will properly handle an ErrNotReplica and retry with the replica hosts set in the error. Signed-off-by: Michael MacDonald <mjmac.macdonald@intel.com> * DAOS-12727 cart: Update WAIT_OBJ default settings (#11641) No longer set "FI_OFI_RXM_DEF_TCP_WAIT_OBJ" to pollfd by default. Setting this causes performance issues on some tests Signed-off-by: Alexander A Oganezov <alexander.a.oganezov@intel.com> * DAOS-12924 build: Add numactl dep to agent and server in spec (#11734) Add numactl dep to agent and server in spec Signed-off-by: Tom Nabarro <tom.nabarro@intel.com> * DAOS-12956 test: Add unique container label to soak (#11765) Update method to use get_container() Signed-off-by: Maureen Jean <maureen.jean@intel.com> * DAOS-10035 test: Include VMD led check in soak harassers (#11750) Add dmg storage led identify and check to soak harassers Signed-off-by: Maureen Jean <maureen.jean@intel.com> * DAOS-12872 build: don't ship protobuf-c RPMs (#11706) update .rpmignore to not ship locally built protobuf-c RPMs Signed-off-by: Michael Hennecke <michael.hennecke@intel.com> * DAOS-6102 test: Check free space information (#11689) Check the total free space changes while performing exclude, drain and extend operations. Signed-off-by: Padmanabhan <ravindran.padmanabhan@intel.com> * DAOS-12959 gurt: Replace strtok with strtok_r (#11775) strtok_r() should be used to be safe. Otherwise, calling daos API could break the applications calling strtok(). Signed-off-by: Lei Huang <lei.huang@intel.com> * DAOS-12576 mgmt: Log client interface choice at INFO level (#11784) Log the client's final interface/domain selection after processing the GetAttachInfo results and manually-set environment variables at INFO level instead of DEBUG. Signed-off-by: Kris Jacque <kristin.jacque@intel.com> * DAOS-12823 pool: fix pool upgrading unexpectedly (#11654) 1. ds_pool_mark_upgrade_completed_internal() should be only called if pool upgrading is needed. 2. let cont_iv_update() tolerate retryable errors like pool_iv_update() as this might make resumed upgrading fail. Signed-off-by: Wang Shilong <shilong.wang@intel.com> * DAOS-12967 dtx: reduce the consume of HLC (#11793) For convenience, some DTX logic directly uses d_hlc_get for local sequence timestamp, that is unnecessary but consumes more HLC and may cause the risk of HLC overflow (in the future) if we may have higher physical clock resolution. The patch replaces HLC with local timestamp for those unnecessary cases of using HLC. Signed-off-by: Fan Yong <fan.yong@intel.com> * DAOS-12960 engine: accurate error message for dss_srv_init (#11778) Log distinguishable error message for different cases when failed inside dss_srv_init(). Signed-off-by: Fan Yong <fan.yong@intel.com> * DAOS-12828 object: a few refine of dc_shard_update_size (#11737) A few refine and add a few logs. Signed-off-by: Xuezhao Liu <xuezhao.liu@intel.com> * DAOS-9163 pool: Do map_dist for restarted ranks (#7678) Request pool map distributions for potentially restarted ranks, so that any rank who has restarted without triggering a SWIM DEAD event will eventually receive maps of the pools that this rank actively belongs to. Introduce POOL_GROUP_MAP_STATUS to ensure that a consistent set of map status is used to determine the up ranks in various cases. Signed-off-by: Li Wei <wei.g.li@intel.com> * DAOS-12910 tools: Move daos cont list-snap logic to Go (#11766) In order to support JSON output for this command, the listing logic needed to be ported to Go. Signed-off-by: Michael MacDonald <mjmac.macdonald@intel.com> * DAOS-12311 control: Enable socket affinity in daos_server scm cmds (#11642) Use ISetID to map ndctl regions to PMem regions generated by ipmctl to avoid confusion when ndctl reports numa_node that doesn't match SocketID reported by ipmctl. Use command structures to apply specific arg strings when calling exec.Command to shell out to ndctl and ipmctl binaries. This helps resolve odd situations where for example ipmctl requires -o nvxml to be specified as one (rather than two) string arguments. Signed-off-by: Tom Nabarro <tom.nabarro@intel.com> * DAOS-12899 test: Reduce nvme size for pool/create_capacity.py (#11764) Moving the pool/create_capacity.py test to run on the Hardware Large clusters with 6 hosts running 1 engine per host to ensure there is enough NVMe capacity for 200 pools. Signed-off-by: Phil Henderson <phillip.henderson@intel.com> * DAOS-12025 control: Make connection timeout retryable (#11533) Properly exposes a gRPC connection timeout as a new client fault, and adds it to the list of retryable errors during a MS request. A connection timeout may occur when a previously-reachable endpoint goes away (e.g. server crashed, cable unplugged, etc). Signed-off-by: Michael MacDonald <mjmac.macdonald@intel.com> * DAOS-12857 csum: Initialize RPC object for CSUM Report (#11798) The structure for sending an RPC for the CSUM error report wasn't completely initialized and contained garbage data. This change fixes that. A test was also added to reproduce the issue. Signed-off-by: Ryon Jensen <ryon.jensen@intel.com> * DAOS-12595 dtx: support CPU yield during local TX commit (#11703) Under original PMDK based local TX model, we assume that current ULT will not yield inside the local PMDK TX. But such assumption is not guaranteed under other backend local TX model, such as on SSD based system. The patch introduces three temporary status for active DTX entry in DRAM: 'preparing', 'committing' and 'aborting'. Base on these, It adjusts related DTX logic to allow CPU yield during DTX status changes, including: 1. 'init' => 'preparing' => 'prepared' 2. 'init' => 'committing' => 'committed' 3. 'prepared' => 'committing' => 'committed' 4. 'prepared' => 'aborting' => 'aborted' It works for both PMDK based local TX and metadata on SSD model. It does not change on-wire RPC protocol. As for the on-disk data layout, it is also compatible with old release. In the patch, we reuse the DTX entry local index (a 32-bits integer) in the data record (vos_irec_df::ir_dtx, evt_desc::dc_dtx, ilog_id::id_tx_id) to specially indicate whether related DTX is for solo transaction (single modification against single replicated object) or not via the highest (31th) one bit. For old release, it will not find any DTX entry if the DTX local index with such bit. Then according to old logic, related DTX will be regarded as 'committed', that just is correct and expected. Signed-off-by: Fan Yong <fan.yong@intel.com> * DAOS-12904 tools: Close files in cleanup closure (#11768) Instead of closing them in the reader goroutine, close them in the cleanup closure that is invoked by the caller. This ensures that the file handles are released synchronously. Also adds context-aware cleanup to avoid leaking goroutines. Signed-off-by: Michael MacDonald <mjmac.macdonald@intel.com> * DAOS-12708 cart: Mark HG_FAULT as DER_INVAL. (#11758) This avoids a retry on non-recoverable errors and therefore an infinite loop. Signed-off-by: Ashley Pittman <ashley.m.pittman@intel.com> * DAOS-12847 container: cache nhandles in batch close rdb tx (#11736) Before this change, when closing container handles from a pool handle (e.g., as done for a crashed client, initiated by daos_agent), the cont_close_hdls() ... get_nhandles() execution for a container decremented its ds_cont_prop_nhandles and also read it back within the same (uncommitted) rdb tx. This code violates the rule that an rdb tx shall not read back its own updates. For containers with more than one handle, this value was not decremented on close. With this change, an rdb tx-scoped hash table is added to track by container UUID, the number of open handles. get_nhandles() is updated to, in this use case, search for the cached nhandles value in the cache before updating the value in rdb. After an rdb tx is ended, the table is destroyed and inserted records freed. Also with this change the test_ior_crash functional test adds checks of the container nhandles values after the test abruptly terminates ior. The expected value is 1 to reflect that the ior open handles have been closed as a result of daos_agent-based cleanup. The final ior run (that gracefully finishes) is also checked to confirm nhandles == 1. Signed-off-by: Kenneth Cain <kenneth.c.cain@intel.com> * DAOS-12866 tests: disable rebuild for degrade test (#11732) Since we allow multiple shards in the same target during rebuild, so RP_XPF in degrade test will not avoid rebuild anymore, so let's disable rebuild explicitly. Signed-off-by: Di Wang <di.wang@intel.com> * DAOS-12911 dtx: handle stale pool map when prepare TX commit (#11723) dc_tx_commit_prepare() may return -DER_STALE because of stale pool map. Under such case, we need to refresh the pool map to avoid subsequent TX restart falling into dead loop. Some other code cleanup related with task completion. Signed-off-by: Fan Yong <fan.yong@intel.com> * DAOS-12184 docker: Run DAOS application in a unique container (#11629) Defines how to use docker containers for running an application accessing a DAOS file system. The DAOS Agent service and the DAOS client are running in the same docker container. Signed-off-by: Cedric Koch-Hofer <cedric.koch-hofer@intel.com> * DAOS-12707 cart: Automatically clamp ports to 0-511 range for CXI (#11537) Some cart tools and utilities use random port numbers that can be outside of the range required by CXI. Instead of modifying each tool to abide by CXI limitations, CART will clamp ports using '% 512' operation and will display a warning for CXI provider if the supplied port falls outside of the required range. Signed-off-by: Alexander A Oganezov <alexander.a.oganezov@intel.com> * DAOS-11881 dfuse: Use many queues and batch complete events. (#10526) Fetch up to 128 complete data events rather than one at a time. Thin out the dfuse debug logs to focus on quality not quantity and make the logs more readable. Signed-off-by: Ashley Pittman <ashley.m.pittman@intel.com> * DAOS-12674 test: nvme/enospace.py test not waiting for thread completion (#11799) Update script enospace.py to complete the job before exit with an ior read event indicator to stop ior read thread. Signed-off-by: Ding Ho ding-hwa.ho@intel.com * DAOS-9721 test: update tests to use get_container (#11769) - Update tests to use get_container instead of TestContainer directly - Misc avocado tag cleanup - Misc code cleanup Signed-off-by: Dalton Bohning <dalton.bohning@intel.com> * DAOS-10192 vos: Add storage usage metrics (#11572) This PR introduce two new gauge metrics defining the SCM and NVME storage used for individual target. It also add a new functional tests validating the value returned by these new gauge metrics. Signed-off-by: Cedric Koch-Hofer <cedric.koch-hofer@intel.com> * DAOS-12990 src/client: Fix event cb registration for multiple cb (#11804) The function daos_event_register_comp_cb() adds the event’s call back completion list to the tail of the newly allocated daos_event_comp_list. This results in only the last registered completion callback being run when an event completes. To fix this, the newly allocated daos_event_comp_list is now added to the tail of the event’s callback completion list to accumulate all callbacks and upon event completion run all registered callbacks. Signed-off-by: Mike Carlin <mike.carlin@hpe.com> * DAOS-12972 control: Add ofi prefix to all libfabric providers (#11808) All libfabric providers should be labeled with the prefix, since we now also have UCX providers. Trying to differentiate between tested and untested providers in code seems unmaintainable. Signed-off-by: Kris Jacque <kristin.jacque@intel.com> * DAOS-12717 cart: pass correct cio_ctx_max_num to crt_init_opt (#11777) Fix an issue of did not pass correct cio_ctx_max_num to crt_init_opt, and refined a few details in crt init process. Signed-off-by: Xuezhao Liu <xuezhao.liu@intel.com> * DAOS-11260 dfuse: Apply timeouts to dfuse data cache. (#10540) Use the information we keep on when files are opened to track how old any data in the kernel page cache might be and only set the keep_cache flag if it's recent. Extend the current container setting from being an on/off to also accepting a time value. Signed-off-by: Ashley Pittman <ashley.m.pittman@intel.com> * DAOS-9313 tests: more log for osa_offline_reintegration_with_rf (#11567) Enable IO debug for more logs. Signed-off-by: Fan Yong <fan.yong@intel.com> * DAOS-12577 dfuse: Update NLT to test cache on lookup. (#11599) Add a test for cache-on-lookup, the fix has already landed. Code cleanup for the pool and container create test code. Resolve pylint warnings in code. Signed-off-by: Ashley Pittman <ashley.m.pittman@intel.com> * DAOS-11863 dfuse: Cache EOF from read and avoid round-trip read. (#11781) When a file is read linearly an EOF is reached then avoid a round-trip for the final read. Signed-off-by: Ashley Pittman <ashley.m.pittman@intel.com> * DAOS-12971 dfuse: Increase hash table size for dfuse inodes. (#11838) At high file counts the hash table becomes a real performance bottleneck so increase the size to accomodate. With 1m files this can result in a 10x performance improvement for some workflows. Signed-off-by: Ashley Pittman <ashley.m.pittman@intel.com> * DAOS-623 test: Simplify VOS unit tests a little (#11837) Since the nested iterator option is specific to a small subset of VOS IO tests, remove it and handle running those internally both ways. Remove related dead code from vos_perf. Don't run tests that are not affected by oclass more than once Make some minor naming adjustments. We don't support VOS over vmem anymore so no need to run evtree test with vmem Signed-off-by: Jeff Olivier <jeffrey.v.olivier@intel.com> * DAOS-12672 control: Make dRPC calls context aware (#11760) - Use context to close connection on cancel - Introduce mutex to protect connection reference - Remove unused dRPC retry logic Signed-off-by: Tom Nabarro <tom.nabarro@intel.com> * DAOS-11925 test: Error running /usr/lib/daos/TESTING/ftest/cart/cart_logtest.py timeout (#11827) Increase run_remote logger timeout to 4800. Signed-off-by: Ding Ho ding-hwa.ho@intel.com * DAOS-11394 test: Save dmg log for daos_test suite (#11783) Save dmg logs for all daos_test cases. Signed-off-by: Kris Jacque <kristin.jacque@intel.com> * DAOS-12984 test: Use NodeSet to provide client hostnames to run IOR (#11809) We need to provide hostnames with NodeSet to run IOR using job manager because the tear down code expects the hostnames to be provided from the tests as NodeSet. Signed-off-by: Makito Kano <makito.kano@intel.com> * DAOS-12985 test: nvme/fault.py 'dict' object has no attribute 'stdout_text" (#11805) Update nvme_utils.py set_device_faulty result to parse json output. Signed-off-by: Ding Ho ding-hwa.ho@intel.com * DAOS-12986 test: offline_parallel_test.py - Update kwargs to be passe… (#11817) We need to pass in a dmg subcommand string (e.g., drain, exclude) to dmg_thread() along with the necessary parameters and the results queue. We used to pass in parameters for all actions (dmg subcommand string) to dmg_thread() even though we only need to pass in parameters for a particular action. This has caused an error in dmg_thread(). Update kwargs to store only the necessary parameters for a particular action, along with the action string and the results queue. Also make minor update to dmg_thread() to accompany with the change in kwargs structure. Signed-off-by: Makito Kano <makito.kano@intel.com> * DAOS-12739 test: update to MOFED 5.8-1.1.2.1 (#11695) Update to MOFED 5.8-1.1.2.1 Update util/job_manager_utils.py --mca "coll": "^hcoll" Signed-off-by: Ding Ho ding-hwa.ho@intel.com * DAOS-13019 test: Kill one ranks twice in multiple_failure.py (#11826) Update server_utils stop_ranks with dmg copy option. Signed-off-by: Ding Ho ding-hwa.ho@intel.com * DAOS-13006 test: resolve all flake issues in ftest (#11812) Resolve all flake issues in ftest. Signed-off-by: Dalton Bohning <dalton.bohning@intel.com> * DAOS-12964 control: Refine dmg hostlist help (#11815) - Associate -l/--host-list options only with commands that accept a host-list parameter. - Deprecate and hide top-level version of the -l/--host-list option. This is to avoid breaking users' scripts if they put the -l option immediately after the dmg invocation. The value passed via this parameter is copied to the new one if it is a hostlist command. - Add a single host version of the hostlist parameter with appropriate help text. For now this is used for telemetry commands. Signed-off-by: Kris Jacque <kristin.jacque@intel.com> * DAOS-10637 control: Scan only for specified provider (#11807) Adjust fabric scanning logic to filter at the libfabric level, to avoid initializing providers we are not using during routine server and agent operations. Network scan commands will still scan for all providers in the following cases: - The user has explicitly specified "--provider all" - The user has not specified a provider on the command line for the daos_agent version of the command. - The user has not specified a provider for the daos_server version of the command, and there is no provider specified in the server config supplied to the command. Signed-off-by: Kris Jacque <kristin.jacque@intel.com> * DAOS-12895 object: handle fetch with non-zero iod_size and NULL sgl (#11794) return -DER_INVAL if obj_fetch with non-zero iod_size and NULL sgl, to avoid assertion in obj_ec_recx_reasb(). Signed-off-by: Xuezhao Liu <xuezhao.liu@intel.com> * DAOS-13087 build: Use ar instead of xiar for Intel builds (#11859) xiar seems to have a serious bug on newer Leap 15. Use the system archive tool instead for now. Signed-off-by: Jeff Olivier <jeffrey.v.olivier@intel.com> * DAOS-12943 test: Set device as faulty before replace. (#11759) Summary: - Added the workaround in the test to address the following issues: 1. If the device is not in a faulty state, storage_replace_nvme command returns host_errors (DER_INVAL) 2. Also, the storage_replace_nvme command will return DER_BUSY error even if the nvme is in faulty state. 3. The test needs to retry storage_replace_nvme command few times before the DER_BUSY error disappears. - Update storage_set_faulty command to return json output. Signed-off-by: Padmanabhan <ravindran.padmanabhan@intel.com> * DAOS-12628 control: Add non-debug event logging for daos_server (#11848) When a rank exits unexpectedly, a RAS event is raised and should be logged at ERROR level. When a pool's service ranks are updated, the event should be logged at INFO level. Signed-off-by: Michael MacDonald <mjmac.macdonald@intel.com> * DAOS-12942 bio: enable SSD auto faulty (#11776) - Enable SSD auto faulty by default. - Fixed client retry code to retry update on -DER_NVME_IO error. Signed-off-by: Niu Yawei <yawei.niu@intel.com> * DAOS-13000 cart: Cleanup messages with hg returns (#11806) - Added DF_HG_RC/DP_HG_RC pair to print mercury error codes and the string version. - Renamed crt_req_force_timeout -> crt_req_force_completion - Changed crt_req_hg_send() to return void. - Revert a change that returned a cart error on HG_Finalize() failure Signed-off-by: Alexander A Oganezov <alexander.a.oganezov@intel.com> * DAOS-13066 test: disable failing test in CI (#11874) Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com> * DAOS-13051 build: Use os.environ.copy() for SCONS_ENV option (#11853) The env["ENV"] object must be implemented with some copy on write or such because replacing it with os.environ means that environments created with Clone() just get a reference to the environment. I tested this with a small SConscript. When replacing env["ENV"] with os.environ wholesale, any changes in cloned environments appear in the original environment as well. With this change, the proper behavior is restored. Signed-off-by: Jeff Olivier <jeffrey.v.olivier@intel.com> * DAOS-12410 object: fix a bug in recov_task_cb (#11810) original code possibly set iod_size from invalid memory address when there are multiple iods in the fetch request and hit -DER_FETCH_AGAIN in data recovery process. Signed-off-by: Xuezhao Liu <xuezhao.liu@intel.com> * DAOS-623 test: Create python script for unit tests (#11869) It's difficult to add new features and new tests that don't fit the standard in bash. This gives a little more flexibility. Preemptively add placeholders for tests that are on feature branches so that post merge, they will still run. Fix a bug in srv_checksum_test.c. It wasn't actually running under valgrind due to placement within run_test.sh, something that was unintentional and another reason to use python. Signed-off-by: Jeff Olivier <jeffrey.v.olivier@intel.com> * DAOS-11964 container: retry for cont_iv_ec_agg_eph_update (#11852) For cont_iv_ec_agg_eph_update(), retry for failure case, to avoid the EC agg epoch boundary IV update failure. Signed-off-by: Xuezhao Liu <xuezhao.liu@intel.com> * DAOS-12934 placement: fix typo in the test (#11839) fix typo in jump_map_place_obj.c Signed-off-by: Di Wang <di.wang@intel.com> * DAOS-3871 test: Remove disable=too-many-ancestors from tests (#11800) The maximum number of ancestors is 9 for our CI. This should cover most of the cases, so we shouldn't have to disable it on the tests. Signed-off-by: Makito Kano <makito.kano@intel.com> * DAOS-623 test: Fix a couple of bugs with utest (#11883) The log name is not correct The --mount option doesn't work without changing a bunch of tests, so remove it for now. Remove run_test.sh Need daos_ prefix for logs to be captured in artifacts Remove references to run_utest.sh Signed-off-by: Jeff Olivier <jeffrey.v.olivier@intel.com> * DAOS-13060 tools: Port more daos cmd hdlrs to Go (#11847) Removed the following C command handlers in favor of cgo implementations in order to add JSON support and remove asynchronous output issues: * cont_create_snap_hdlr() * cont_list_snaps_hdlr() * cont_destroy_snap_hdlr() * cont_rollback_hdlr() * obj_query_hdlr() Signed-off-by: Michael MacDonald <mjmac.macdonald@intel.com> * DAOS-13125 test: Remove redundant control/dmg_pool_evict test (#11901) The pool/evict test is more effective at actually testing that a revoked pool handle results in a client application being unable to continue working with a pool. The test being removed just does a `daos pool list-cont` after the eviction, but the daos tool obtains a fresh pool handle for each invocation. Signed-off-by: Michael MacDonald <mjmac.macdonald@intel.com> * DAOS-623 test: Improve run_utest.py (#11886) Save stdout and stderr in logs Print to output where to find logs for a failed test Fix regression in run_go_test cmocka results (UTEST_control. prefix) Signed-off-by: Jeff Olivier <jeffrey.v.olivier@intel.com> * DAOS-10034 test: Add VMD specific testing for fault + reintegration (#11771) Update fault_reintegration.yaml for build error on Leap15. Update set_device_faulty to nvme_utils and dmg_utils.py Update with fix for pylint error. Resolve nume_utils.py conflict Signed-off-by: Ding Ho ding-hwa.ho@intel.com * DAOS-13077 dtx: trace container reference for DTX logic (#11879) More clear trace for DTX register/resync/reindex enter and exit. Signed-off-by: Fan Yong <fan.yong@intel.com> * DAOS-623 test: Unit tests write directly to log file (#11917) Rather than capturing output and then printing when the command is finished, have subprocess.run write directly to output file. This makes for a better experience when running locally so we can monitor a long running command. Signed-off-by: Jeff Olivier <jeffrey.v.olivier@intel.com> * SRE-415 test: Add some diagnostics (#11930) When trying to create the job status directory during Launchable analysis fails. Signed-off-by: Brian J. Murrell <brian.murrell@intel.com> * DAOS-12163 test: increase rf test timeout (#11909) increase rf test timeout Signed-off-by: Wang Shilong <shilong.wang@intel.com> * DAOS-7857 test: Enhance control/config_generate_output.py (#11850) Enhance control/config_generate_output.py by considering numa node The test used to verify NVMe PCI addresses, bdev name (e.g., pmem0), and the network interface against the values obtained from dmg storage scan and dmg network scan without considering numa node. This means as long as the PCI addr, bdev, or interface is found in the dmg output somewhere, we would pass the test. Each address, dev name, and interface is mapped to a numa node (pinned_numa_node in dmg config generate output). Enhance the test so that it verifies the numa node mapping is also accurate. Update the dictionaries that store expected values so that key would be numa node and value would be a set of addresses, devices, or interfaces. Then use these dictionaries to verify dmg config generate output. Signed-off-by: Makito Kano <makito.kano@intel.com> * DAOS-12398 object: Check punch epoch before migration punch (#11667) Since object enumeration might return next punch epoch of the current object/key/rec, so let's compare the punch epoch with rebuild/reintegration stable epoch before migrate punch. Do not allow reint/drain/extend if there are other rebuild job going on to avoid screwing up the object layout. Fix converting task error reset in dc_tx_check_update() Retry UPDATE_AGAIN with delay, similar as INPROGRESS. Signed-off-by: Di Wang <di.wang@intel.com> Co-authored-by: Fan Yong <fan.yong@intel.com> * DAOS-13088 control: Support fabric scan with multiple providers (#11867) - Add internal support for fabric scan with multiple providers specified. Signed-off-by: Kris Jacque <kristin.jacque@intel.com> * DAOS-13121 container: read not set properties for IV (#11898) Currently IV update need all properties be set, don't ignore not set properties(for upgrading case). Signed-off-by: Wang Shilong <shilong.wang@intel.com> * DAOS-623 test: Only use open remote PR for base (#11960) I hit a situation where I resused a remote branch from a PR that had been merged and the branch deleted. This caused my new PR to use a testbuild branch for the base calculation. Add a check to find_base.sh that the branch it finds is an open PR. Signed-off-by: Jeff Olivier <jeffrey.v.olivier@intel.com> * DAOS-13157 test: nvme_fault_reintegration - not in expected NORMAL, OFF state (#11961) Move this testcase to manual Signed-off-by: Ding Ho ding-hwa.ho@intel.com * DAOS-13152 githooks: accept "git --version" output with MacOs (#11934) Handle unexpected extra “ (Apple Git-133)" pattern from “git --version" output on MacOs Signed-off-by: Bruno Faccini <bruno.faccini@intel.com> * DAOS-13153 control: Use libfabric method to alloc hint (#11941) Allocating the hint manually wasn't working for some providers. Signed-off-by: Kris Jacque <kristin.jacque@intel.com> * DAOS-12407 cart: Update libfabric to v1.17.1 (#11951) - Update to libfabric to v1.17.1 - Apply DAOS 12407 patch to ofi Signed-off-by: Alexander A Oganezov <alexander.a.oganezov@intel.com> * Revert "DAOS-12949 pipeline: add layout version to OID (#11903)" This reverts commit be48a67. Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com> * DAOS-12949 pipeline: add layout version to OID Add layout version to OID. Required-githooks: true Signed-off-by: Di Wang <di.wang@intel.com> --------- Signed-off-by: Saurabh Tandan <saurabh.tandan@intel.com> Signed-off-by: Kris Jacque <kristin.jacque@intel.com> Signed-off-by: Ashley Pittman <ashley.m.pittman@intel.com> Signed-off-by: Michael MacDonald <mjmac.macdonald@intel.com> Signed-off-by: Dalton Bohning <dalton.bohning@intel.com> Signed-off-by: Phil Henderson <phillip.henderson@intel.com> Signed-off-by: Maureen Jean <maureen.jean@intel.com> Signed-off-by: Di Wang <di.wang@intel.com> Signed-off-by: Ding Ho ding-hwa.ho@intel.com Signed-off-by: Li Wei <wei.g.li@intel.com> Signed-off-by: Alexander A Oganezov <alexander.a.oganezov@intel.com> Signed-off-by: Tom Nabarro <tom.nabarro@intel.com> Signed-off-by: Michael Hennecke <michael.hennecke@intel.com> Signed-off-by: Padmanabhan <ravindran.padmanabhan@intel.com> Signed-off-by: Lei Huang <lei.huang@intel.com> Signed-off-by: Wang Shilong <shilong.wang@intel.com> Signed-off-by: Fan Yong <fan.yong@intel.com> Signed-off-by: Xuezhao Liu <xuezhao.liu@intel.com> Signed-off-by: Ryon Jensen <ryon.jensen@intel.com> Signed-off-by: Kenneth Cain <kenneth.c.cain@intel.com> Signed-off-by: Cedric Koch-Hofer <cedric.koch-hofer@intel.com> Signed-off-by: Mike Carlin <mike.carlin@hpe.com> Signed-off-by: Jeff Olivier <jeffrey.v.olivier@intel.com> Signed-off-by: Makito Kano <makito.kano@intel.com> Signed-off-by: Niu Yawei <yawei.niu@intel.com> Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com> Signed-off-by: Brian J. Murrell <brian.murrell@intel.com> Signed-off-by: Bruno Faccini <bruno.faccini@intel.com> Co-authored-by: saurabhtandan <saurabh.tandan@intel.com> Co-authored-by: Kris Jacque <kristin.jacque@intel.com> Co-authored-by: Ashley Pittman <ashley.m.pittman@intel.com> Co-authored-by: Michael MacDonald <mjmac.macdonald@intel.com> Co-authored-by: Dalton Bohning <dalton.bohning@intel.com> Co-authored-by: Phil Henderson <phillip.henderson@intel.com> Co-authored-by: mjean308 <48688872+mjean308@users.noreply.github.com> Co-authored-by: wangdi <di.wang@intel.com> Co-authored-by: dinghwah <48604964+dinghwah@users.noreply.github.com> Co-authored-by: Li Wei <wei.g.li@intel.com> Co-authored-by: Alexander Oganezov <alexander.a.oganezov@intel.com> Co-authored-by: Tom Nabarro <tom.nabarro@intel.com> Co-authored-by: Michael Hennecke <michael.hennecke@intel.com> Co-authored-by: Ravindran Padmanabhan <ravindran.padmanabhan@intel.com> Co-authored-by: wiliamhuang <lei.huang@intel.com> Co-authored-by: Wang Shilong <shilong.wang@intel.com> Co-authored-by: Nasf-Fan <fan.yong@intel.com> Co-authored-by: Liu Xuezhao <xuezhao.liu@intel.com> Co-authored-by: Ryon Jensen <ryon.jensen@intel.com> Co-authored-by: Ken Cain <kenneth.c.cain@intel.com> Co-authored-by: Cedric Koch-Hofer <94527853+knard-intel@users.noreply.github.com> Co-authored-by: mikecarlin1 <129101571+mikecarlin1@users.noreply.github.com> Co-authored-by: Jeff Olivier <jeffrey.v.olivier@intel.com> Co-authored-by: Makito Kano <makito.kano@intel.com> Co-authored-by: Niu Yawei <yawei.niu@intel.com> Co-authored-by: Brian J. Murrell <brian.murrell@intel.com> Co-authored-by: Bruno Faccini <bruno.faccini@intel.com>

wangdi1 requested review from Nasf-Fan and liuxuezhao March 7, 2023 00:51

github-actions Bot added priority Ticket has high priority (automatically managed) release-2.4.0 Targeted for release 2.4.0 labels Mar 7, 2023

daosbuild1 reviewed Mar 7, 2023

View reviewed changes

DAOS-12706 object: Some memory corruption issue

7b8fcbc

Fix some memory corruption issuse for csum in obj_get_iods_offs_by_oid(). Required-githooks: true Signed-off-by: Di Wang <di.wang@intel.com>

wangdi1 force-pushed the wangdi/daos_12706 branch from 477ddeb to 7b8fcbc Compare March 8, 2023 01:33

daosbuild1 reviewed Mar 8, 2023

View reviewed changes

Nasf-Fan reviewed Mar 15, 2023

View reviewed changes

Nasf-Fan self-requested a review March 15, 2023 11:06

liuxuezhao reviewed Mar 20, 2023

View reviewed changes

wangdi1 requested a review from liuxuezhao March 20, 2023 14:41

liuxuezhao approved these changes Mar 20, 2023

View reviewed changes

Nasf-Fan approved these changes Mar 21, 2023

View reviewed changes

wangdi1 requested a review from a team March 21, 2023 15:40

johannlombardi merged commit 0ac4744 into master Mar 21, 2023

johannlombardi deleted the wangdi/daos_12706 branch March 21, 2023 16:01

Conversation

wangdi1 commented Mar 7, 2023

Before requesting gatekeeper:

Gatekeeper:

Uh oh!

github-actions Bot commented Mar 7, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

daosbuild1 left a comment

Choose a reason for hiding this comment

Uh oh!

daosbuild1 commented Mar 7, 2023

Uh oh!

daosbuild1 commented Mar 7, 2023

Uh oh!

daosbuild1 left a comment

Choose a reason for hiding this comment

Uh oh!

daosbuild1 commented Mar 9, 2023

Uh oh!

wangdi1 commented Mar 10, 2023

Uh oh!

wangdi1 commented Mar 14, 2023

Uh oh!

Nasf-Fan Mar 15, 2023

Choose a reason for hiding this comment

Uh oh!

wangdi1 Mar 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Nasf-Fan Mar 20, 2023

Choose a reason for hiding this comment

Uh oh!

wangdi1 Mar 20, 2023

Choose a reason for hiding this comment

Uh oh!

liuxuezhao left a comment

Choose a reason for hiding this comment

Uh oh!

liuxuezhao Mar 20, 2023

Choose a reason for hiding this comment

Uh oh!

wangdi1 Mar 20, 2023

Choose a reason for hiding this comment

Uh oh!

wangdi1 commented Mar 20, 2023

Uh oh!

wangdi1 commented Mar 21, 2023

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Milestone

Development

Uh oh!

5 participants

github-actions Bot commented Mar 7, 2023 •

edited

Loading

wangdi1 Mar 17, 2023 •

edited

Loading