Support cabal test-suite with shelltestrunner

When I develop executables as part of a Haskell project, I have to do and repeat some manual tests to ensure that the executable behaviour is correct (e.g. status code, error message, output ...). This is tedious, and I think it could be automated nicely with shelltestrunner.

My proposal would be to support shelltestrunner in Cabal file as a test-suite.

Example of a Cabal test-suite with shelltestrunner support:

test-suite shelltest
hs-source-dirs: test
main-is: shelltest.hs
type: exitcode-stdio-1.0
default-language: Haskell2010
build-depends: base, shelltestrunner

test/shelltest.hs

module Main where

import Test.ShellTest (shelltest)

main :: IO ()
main = shelltest ["test/shelltest/"]

Shelltestrunner test files would be gathered in test/shelltest.

Shelltestrunner configuration could be set directly in shelltest.hs or in an external configuration file.

Sounds nice! Would you like to make a start on it ?

At the moment, the whole project is in one file.

I would need to split it into an executable and a library.

Do you think it makes sense ? Is it something acceptable ?

Sure, that’s fine.

FWIW, I do basically the same thing now (this is my ShellTests.hs):

module Main
where

import System.Process (
   CreateProcess( env ),
   createProcess,
   shell,
   waitForProcess )
import System.Exit ( exitWith )

main :: IO ()
main = do
  -- Get a CreateProcess object corresponding to our shell command.
  let createproc = shell "shelltest test/shell/*.test"

  -- But clear its environment before running the command.
  let empty_env_createproc = createproc { env = Just [] }

  -- Ignore stdin/stdout/stderr...
  (_,_,_,hproc) <- createProcess empty_env_createproc

  -- Now run the ProcessHandle and exit with its result.
  result <- waitForProcess hproc
  exitWith result

Granted that yours looks a lot nicer. Depending on libraries with cabal works a lot better than depending on executables, too. For example, hackage can't tell you that my project depends on "the shelltest executable being in $PATH".

Thanks for this idea back in 2017. Are you still using it ? Is there anything we need to do here ?

Perhaps add as a reproducible cookbook example to the https://github.com/simonmichael/shelltestrunner/blob/master/README.md ?

Yep, still using it, but struggling to remember what this issue is about =)

I think all the original report is asking for is a way to "do what the shelltest program would do," but within a library function. If there was a shelltest function provided by a library under Test.ShellTest, then you could use the sample code in the first comment rather than the real code in my previous comment.

Here's a sketch:

Refactor the bulk of shelltest's main function into Test.ShellTest.shelltest
Update shelltest's main to call that new function.
Ship Test.ShellTest as a public library.

Then anyone who wants to use shelltest as part of a cabal test suite can call the library function instead of the executable (which is a bit more trouble).

Ah, so make shelltestrunner usable as a haskel library ? That I can understand, thanks for clarifying. I agree this could be useful.

In general, an executable can be used in a testsuite by cabal test if it is declared as build dependency.
For an example, testing "myself" (agda2lagda) using goldplate, see: https://github.com/andreasabel/agda2lagda/blob/93d317c87bddf973e3fa6abd5bf24dcfa9a797f0/agda2lagda.cabal#L104-L115

So, add build-tool-depends: shelltestrunner:shelltestrunner to the testsuite stanza. Then you can system "shelltestrunner" ... in the Main of the testsuite. (Example: https://github.com/andreasabel/agda2lagda/blob/93d317c87bddf973e3fa6abd5bf24dcfa9a797f0/test/Tests.hs#L5)

Ah, build-tool-depends does indeed solve half of the problem. I am somehow just learning about it today. But keep in mind that this issue predates cabal-2.0 =)

Using system "shelltestrunner" isn't a great solution though. It's not really portable -- you have to feed it a command that will work on every possible shell, including Windows' cmd.exe -- and it's also a bit inefficient in that it creates a whole new shell process (sourcing .rc files and whatnot) to launch the command.

With shelltestrunner, I usually have multiple .test files in some subdirectory of tests/. What I'd like to do is to use a portable solution to find them all...

import System.FilePath.Find ((==?), always, extension, find)
find_sources :: IO [FilePath]
find_sources = find always (extension ==? ".test") "tests/shell"

and then test them within the same process, without involving an OS shell at all, e.g. (in pseudocode):

main :: IO ()
main = do
  sources <- find_sources
  shelltest $ sources

Ah, build-tool-depends does indeed solve half of the problem.

This was premature ☠️

The "configure" phase happily proceeds when the required build-tool-depends are missing. In this case...

$ runghc Setup.hs configure --enable-tests -v3
...
Searching for shelltestrunner in path.
Cannot find shelltestrunner on the path

returns success (zero) on the command-line, rather than dying immediately as it would if a required library were absent.

$ runghc Setup.hs configure --enable-tests -v3

I have not tested this since I am always using the cabal command. Is there an issue on this problem in the cabal issue tracker?

I have not tested this since I am always using the cabal command. Is there an issue on this problem in the cabal issue tracker?

This one is similar in spirit: haskell/cabal#5464

Navigation Menu

Repository navigation

Support cabal test-suite with shelltestrunner #11

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Support cabal test-suite with shelltestrunner #11

Description

Activity

simonmichael commented on Dec 30, 2017

fmind commented on Jan 10, 2018

simonmichael commented on Jan 10, 2018

orlitzky commented on Jan 26, 2018

simonmichael commented on Oct 19, 2020

simonmichael commented on Oct 19, 2020

orlitzky commented on Oct 19, 2020

simonmichael commented on Oct 19, 2020

andreasabel commented on Nov 13, 2021

orlitzky commented on Nov 13, 2021

orlitzky commented on Nov 29, 2021

andreasabel commented on Nov 29, 2021

orlitzky commented on Nov 29, 2021

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions