Shared arguments for tasks #395

Let's say I have task1 and task2 and they both want to support the same few arguments, say --noop and --envs=dev,test. Let's also say I don't want to repeat myself with the declaration, handling and documentation of these arguments. Regarding handling, let's say that each argument may require some data fetching and munging before the task gets into its stride.

I considered a few approaches and tried to get invoke's existing automatic help support to play nicely and do as much as possible. Using kwargs for this is easy but there was no invoke help for kwargs. It could be that adding some docstring parser looking for sphinx-style param and type hints to invoke would be a better way to go, but I was bored and curious.

The approach that has borne the most fruit was the decorator approach but this has now reached some limits (mostly readability!) and I'd appreciate any advice. Untested pseudo-code follows:

def handle_noop(func):
    # Update the docstring with the new info 
    func.__doc__ += "\nIf --noop is given, do a dry run."

    # --- SNIP --- for clarity: building argspec_with_annotations and argspec_no_annotations

    # The rabbit hole starts here
    fn_body = """
@wraps(func)
def noop_fn({0}):
    # Update kwargs with noop value
    return func({1})
    """.format(
        argspec_with_annotations, 
        argspec_no_annotations)
    exec(
        fn_body, 
        {"func": func, "wraps": wraps}, 
        locals())
    decorated = locals()["noop_fn"]
    return decorated


@task
@handle_noop
def task1(ctx, foo=True, *args, **kwargs)
    """Do the needful for task 1."""
    print("Performing task1")

$ inv -h task1
Usage: inv[oke] [--core-opts] task1 [--options] [other tasks here ...]

Docstring:
  Do the needful for task 1.
  If --noop is given, do a dry run.

Options:
  -f, --foo
  -n, --noop

$ inv task1 --noop
$ inv task1 --not-an-argument
No idea what '--not-an-argument' is!

Using functools.wraps in the typical way works, but inv -h task1 shows only the args from the wrapper, it does not merge them with those of the underlying task (obviously, since the task's function is replaced). A little playing with the inspect module early on in the decorator and I created a formatted merged argspec. I then create the wrapper by exec-ing a string with the formatted, merged argspec interpolated in. I did this twice, once for the wrapper args with annotations and once for the call of the wrapped task without annotations.

Invoke now correctly shows help (as above pseudo-output), the options in the task help are ordered and the task handles the correct arguments (inv task1 --noop works fine but it has no idea what inv task1 --not-an-argument is).

Annotations seem to be important for getting invoke to offer the --[no]-noop for bools and that was working a few iterations ago, though I've broken that at the moment as I play with annotations in other decorators. Hmm, can't re-create what I thought was the trivial case of a task with an arg of noop:bool=False, I'm now just getting ValueError: Function has keyword-only arguments or annotations, use getfullargspec() API which can support them.

Obviously we can't easily refer to the args themselves within the task and I couldn't seem to populate args in the task function easily. Keeping track of args, their defaults and values through multiple decorators was getting hairy so the decorator is just moving the arg to kwargs for the task itself and later decorators to use. It's important to me that using introspection should not be necessary in the task body to get merged in arguments.

The --envs argument is more interesting as it is a string on the commandline so handle_envs() would coerce to an array and store in the kwargs. I include it here only as food for thought about argument types more complex than bool.

I like the decorator syntax for this applying this type of thing but not the exec-ing for implementing it, it's fugly and IDEs hate it. Debugging it sucks. I'm keen for something more pythonic.

I hope to hear that I'm over-complicating this and the'res a simpler way that already exists or is on the roadmap but I've been through the github issues and can't find anything. I haven't dived into the invoke code as much as I should have done so maybe I can do that and suggest a simple PR for kwarg documentation. I know you've put a lot of thought into argument handling so if you know any non-obvious reasons why this is insane please let me know. I know this feature/use case will not be able to be supported by great changes in invoke.

One last thing, any chance of a copy of Fabric2 please? If I can't do remote tasks at all I'm just going to keep poking and playing with invoke without being able to use it for anything useful.