Skip to content

Prevent powershell from adding a BOM to test input - #3595

Merged
martin-frbg merged 10 commits into
OpenMathLib:developfrom
martin-frbg:issue3594
Apr 7, 2022
Merged

Prevent powershell from adding a BOM to test input#3595
martin-frbg merged 10 commits into
OpenMathLib:developfrom
martin-frbg:issue3594

Conversation

@martin-frbg

@martin-frbg martin-frbg commented Apr 5, 2022

Copy link
Copy Markdown
Collaborator

fixes #3594 as suggested by mmuetzel

@mmuetzel

mmuetzel commented Apr 6, 2022

Copy link
Copy Markdown
Contributor

I'm not sure I understand what the actual issue is. But the following PowerShell command should read a file (with or without BOM) and write it without BOM (as long as those files only contain ASCII characters):

Get-Content .\test_file.txt -Encoding UTF8 | Set-Content .\test_file_no_BOM.txt -Encoding ASCII

@martin-frbg

Copy link
Copy Markdown
Collaborator Author

The problem is that on "some" Windows hosts in Azure, powershell´s Get-Content prepends a BOM to anything it pipes into an external program, and none of the various internet recipes I tried got it to stop doing that. This behaviour breaks some of the BLAS tests where the name of the output file to create is written as the first parameter in the input file for the test. As the (Fortran) test code does not support UTF, it simply adds the character pattern of the BOM to the filename, the test succeeds but the subsequent check does not find the expected file and reports a failure.
What I tried here is also destined to fail, as the BOM is not there (yet) when piping to the built-in replace command, but gets added in the next stage. #3598 has a different approach where I just try to replace the non-portable file command with something equally external but always available on Windows.

@mmuetzel

mmuetzel commented Apr 7, 2022

Copy link
Copy Markdown
Contributor

By default on the system I'm testing, the input encoding is set to ibm850. That might be the reason I didn't see this locally.
I think I can reproduce this if I set the input encoding of the console to UTF-8 with BOM with this PowerShell command:

[Console]::InputEncoding = New-Object Text.UTF8Encoding $true

It might be a good idea to unconditionally set the console input encoding to UTF-8 without BOM in that .ps1 script. That way the pipes should behave pretty similarly to what they do on modern Linux:

[Console]::InputEncoding = New-Object Text.UTF8Encoding $false

@martin-frbg

martin-frbg commented Apr 7, 2022

Copy link
Copy Markdown
Collaborator Author

Thank you for the suggestion - it seems this was not among the many variations I tried in #3563.

@martin-frbg martin-frbg changed the title Strip any spurious BOM from test input Prevent powershell from adding a BOM to test input Apr 7, 2022
@martin-frbg
martin-frbg merged commit 145ccf8 into OpenMathLib:develop Apr 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CTest failure with windows build

2 participants