Somewhat of an alternative way for
I was thinking on if we would like to make duct collected metadata expandable, we might want actually underlying tools to become duct aware and populate whatever info they want into duct info record, may be under "external" key . To make it non competing between tools (as multiple might decide to write into it at the same time), I think we should
- set for the child process the
CON_DUCT_EXTERNAL_PREFIX env var which would be the prefix to all the other files with only _external- suffix.
- Then any external tool could write into
{CON_DUCT_EXTERNAL_PREFIX}{name}-{pid}.json a dict with key "{name}" (or multiple)
- Upon finalization
duct would collect all the json files and populate with their content the "external" attribute within _info.json.
This way we can e.g. datalad to provide the entire "datalad-wtf" or alike (may be all the sections separately under "datalad:{section}" keys) within there if datalad configured to do so on the machine (or in a given dataset)!
The question would be what con-duct ls etc should do to enable their listing. But it could be quite cool to be able to interrogate for internal tool specific details across runs!
Somewhat of an alternative way for
I was thinking on if we would like to make
ductcollected metadata expandable, we might want actually underlying tools to become duct aware and populate whatever info they want into duct info record, may be under"external"key . To make it non competing between tools (as multiple might decide to write into it at the same time), I think we shouldCON_DUCT_EXTERNAL_PREFIXenv var which would be the prefix to all the other files with only_external-suffix.{CON_DUCT_EXTERNAL_PREFIX}{name}-{pid}.jsona dict with key"{name}"(or multiple)ductwould collect all the json files and populate with their content the"external"attribute within_info.json.This way we can e.g.
dataladto provide the entire"datalad-wtf"or alike (may be all the sections separately under"datalad:{section}"keys) within there if datalad configured to do so on the machine (or in a given dataset)!The question would be what
con-duct lsetc should do to enable their listing. But it could be quite cool to be able to interrogate for internal tool specific details across runs!