Cannot use sys.stdout in create_subprocess_exec.

uvloop version: 0.9.1
Python version: 3.6.5
Platform: macOS 10.13.3
Can you reproduce the bug with PYTHONASYNCIODEBUG in env?: Same behavior irregardless of PYTHONASYNCIODEBUG.

Code for reproduction:

import asyncio
import sys

import uvloop


async def run_subprocess():
    proc = await asyncio.create_subprocess_exec(
        sys.executable, '-c', 'print(1)',
        stdout=sys.stdout,
    )
    await proc.communicate()


def run(loop_factory):
    loop = loop_factory()
    asyncio.set_event_loop(loop)
    try:
        loop.run_until_complete(run_subprocess())
    finally:
        loop.close()


def main():
    print('asyncio')
    run(asyncio.new_event_loop)
    print('uvloop')
    run(uvloop.new_event_loop)


if __name__ == '__main__':
    main()

Output:

asyncio
1
uvloop
Traceback (most recent call last):
  File "hoge.py", line 32, in <module>
    main()
  File "hoge.py", line 28, in main
    run(uvloop.new_event_loop)
  File "hoge.py", line 19, in run
    loop.run_until_complete(run_subprocess())
  File "uvloop/loop.pyx", line 1364, in uvloop.loop.Loop.run_until_complete
  File "hoge.py", line 10, in run_subprocess
    stdout=sys.stdout,
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/coroutines.py", line 110, in __next__
    return self.gen.send(None)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/subprocess.py", line 225, in create_subprocess_exec
    stderr=stderr, **kwds)
  File "uvloop/loop.pyx", line 2368, in __subprocess_run
  File "uvloop/handles/process.pyx", line 564, in uvloop.loop.UVProcessTransport.new
  File "uvloop/handles/process.pyx", line 49, in uvloop.loop.UVProcess._init
  File "uvloop/handles/process.pyx", line 38, in uvloop.loop.UVProcess._init
  File "uvloop/handles/process.pyx", line 254, in uvloop.loop.UVProcess._init_options
  File "uvloop/handles/process.pyx", line 456, in uvloop.loop.UVProcessTransport._init_files
ValueError: invalid stdout argument value 1

I'm facing the same/similar error in my Travis CI environment.
My uvloop version is 0.8.1 and pytest version is 3.5.0, and I'm passing stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL.

        proc = await asyncio.create_subprocess_exec(*[
            'python', '-m', 'ai.backend.agent.stats',
            f'tcp://127.0.0.1:{stats_port}',
            cid,
            '--type', collection_type,
>       ], **output_opts)
tests/test_stats.py:53: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/local/lib/python3.6/asyncio/subprocess.py:225: in create_subprocess_exec
    stderr=stderr, **kwds)
uvloop/loop.pyx:2164: in __subprocess_run (uvloop/loop.c:39419)
    ???
uvloop/handles/process.pyx:549: in uvloop.loop.UVProcessTransport.new (uvloop/loop.c:93810)
    ???
uvloop/handles/process.pyx:49: in uvloop.loop.UVProcess._init (uvloop/loop.c:84790)
    ???
uvloop/handles/process.pyx:38: in uvloop.loop.UVProcess._init (uvloop/loop.c:84510)
    ???
uvloop/handles/process.pyx:239: in uvloop.loop.UVProcess._init_options (uvloop/loop.c:88007)
    ???
uvloop/handles/process.pyx:415: in uvloop.loop.UVProcessTransport._init_files (uvloop/loop.c:91046)
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <_pytest.capture.DontReadFromInput object at 0x7f4717c82860>
    def fileno(self):
>       raise UnsupportedOperation("redirected stdin is pseudofile, "
                                   "has no fileno()")
E       io.UnsupportedOperation: redirected stdin is pseudofile, has no fileno()
../../../virtualenv/agent/lib/python3.6/site-packages/_pytest/capture.py:583: UnsupportedOperation

Allow file objects to be passed to loop.subprocess* functions.

Fixed in master. Please test!

In 0.14.0 still present

platform darwin -- Python 3.6.10, pytest-5.4.2, py-1.8.1, pluggy-0.13.1

Code failed with same issue:

    def _parse_list(output, files_patt):
        patt = r'(\d+)\s+.{16}\s+(%s)' % files_patt
        return re.findall(patt, output)

    async def list_files(cls, rar_file: str, files_patt):
        process = await asyncio.create_subprocess_exec(
            'unrar', 'l', rar_file,
            stdout=asyncio.subprocess.PIPE,
        )
        output, _ = await process.communicate()
        if process.returncode != os.EX_OK:
            raise RuntimeError(
                f'unrar failed: return_code={process.returncode!r}')

        return [(int(size), name) for size, name in _parse_list(output.decode(), files_patt)]

csv_in_rar_count = await list_files(rar_path, '(?i).*CSV')

@1st1

@1st1

    process = await asyncio.create_subprocess_exec(*cmd)
../../../../../.pyenv/versions/3.7.8/lib/python3.7/asyncio/subprocess.py:217: in create_subprocess_exec
    stderr=stderr, **kwds)
uvloop/loop.pyx:2749: in subprocess_exec
    ???
uvloop/loop.pyx:2707: in __subprocess_run
    ???
uvloop/handles/process.pyx:596: in uvloop.loop.UVProcessTransport.new
    ???
uvloop/handles/process.pyx:60: in uvloop.loop.UVProcess._init
    ???
uvloop/handles/process.pyx:49: in uvloop.loop.UVProcess._init
    ???
uvloop/handles/process.pyx:274: in uvloop.loop.UVProcess._init_options
    ???
uvloop/handles/process.pyx:458: in uvloop.loop.UVProcessTransport._init_files
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <_pytest.capture.DontReadFromInput object at 0x114fc6590>

    def fileno(self) -> int:
>       raise UnsupportedOperation("redirected stdin is pseudofile, has no fileno()")
E       io.UnsupportedOperation: redirected stdin is pseudofile, has no fileno()

../../.venv/lib/python3.7/site-packages/_pytest/capture.py:230: UnsupportedOperation

Whatever the underlying problem is here, it definitely has not been closed out. I have found that it can be worked around in tests where it pops up by using the default event loop policy via a pytest mark at the top of my test files that hit the issue. Hopefully, this helps until we can nail down the underlying issue:

# NOTE: These tests are brittle when run under uvloop. We run these under the default
# asyncio event loop policy to avoid exceptions relating to pytest output capture.
# The exception is: `io.UnsupportedOperation: redirected stdin is pseudofile, has no fileno()`
pytestmark = [pytest.mark.asyncio, pytest.mark.event_loop_policy("default")]

redirected stdin is pseudofile, has no fileno()

This error message is from pytest. If you try to read from stdin, it also says:

pytest: reading from stdin while output is captured! Consider using -s.

I'm checking why asyncio doesn't fail this test.

Use FD=0/1/2 for subprocess(stdio=None)

Navigation Menu

Repository navigation

Cannot use `sys.stdout` in `create_subprocess_exec`. #136

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Cannot use sys.stdout in create_subprocess_exec. #136

Description

Activity

achimnol commented on Apr 24, 2018

1st1 commented on May 24, 2018

spumer commented on May 20, 2020

spumer commented on Nov 26, 2020

blakewatters commented on Dec 8, 2020

fantix commented on Feb 8, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Cannot use `sys.stdout` in `create_subprocess_exec`. #136