Bug: recursively copies artifact dir when it's in source dir

[julianzq]

Recursively copies artifact dir when it's in source dir until it errors out:

JSON-RPC input:

{
...
"source_dir":    "path/to/source",
"artifacts_dir":  "path/to/source/.build/artifact",
...
}

Then running lambda-builders <JSON INPUT>

Results in

PythonPipBuilder:CopySource - [Errno 63] File name too long: '/path/to/source/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/python_dateutil-2.8.2.dist-info/top_level.txt'

Setting artifact_dir outside the source_dir solves this issue.

---

Is this a bug, or is this expected? Perhaps we should add ignore: {} interface? I can contribute a PR for this if you guys can point me in the right direction.

---

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: Mac
2. If using SAM CLI, sam --version:
3. AWS region: us-east-1 (shouldn't matter)

Add --debug flag to any SAM CLI commands you are running

[jfuss]

@israelg99 artifact_dir and source_dir should be different directories unless you wanted to embed (and copy them) for some reason. To my knowledge, we don't want to restrict the behavior here so we don't want to fail the run. Maybe we should? But not sure what that could break now, if we do.

On the ignore interface, I am not sure. It seems like a good idea and something we might be able to leverage in SAM CLI to ignore different directories or files. I can check with the team on the support. My concern is more about making sure we have the right scope and not locking ourselves into something we couldn't expand. For example, we would need to make sure its something we can support in SAM CLI, as I know customers have wanted support for ignoring files in their source. Your ask seems slightly different (maybe it isn't) but my point is we just need to really think through all the use-cases for something like this.

In the mean time, my suggestion is to use separate directories for each as this was how Lambda Builders (and SAM CLI) was designed.

[pkit]

@jfuss

In the mean time, my suggestion is to use separate directories for each as this was how Lambda Builders (and SAM CLI) was designed.

Err... how exactly it should be done?
Copying all the files three times or what?
Let's think about a typical project:

/home/user/my_project/
/home/user/my_project/main.py
/home/user/my_project/requirements.txt
/home/user/my_project/my_module/
/home/user/my_project/my_module/module.py

What's the proposed solution for building this one?
Typical solution:

/home/user/my_project/.build

will fail miserably because of that strange behavior.

---

The solution that I would do will have something like:

{
  ....
  ignore_file: ".ignore"
}

Where:

cat .ignore
.build/

[pkit]

Ah, this one is so cute!

aws-lambda-builders/aws_lambda_builders/workflows/python_pip/workflow.py

Line 26 in cefc331

".aws-sam",

aws-lambda-builders/aws_lambda_builders/workflows/ruby_bundler/workflow.py

Line 26 in 75927e3

EXCLUDED_FILES = (".aws-sam", ".git")

aws-lambda-builders/aws_lambda_builders/workflows/java_maven/workflow.py

Line 29 in 75927e3

EXCLUDED_FILES = (".aws-sam", ".git")

aws-lambda-builders/aws_lambda_builders/workflows/nodejs_npm/workflow.py

Line 41 in 75927e3

EXCLUDED_FILES = (".aws-sam", ".git")

So, aws-sam does suffer from the same bug, but you've just hardcoded the escape hatch...
Nice!

[julianzq]

@pkit Agreed. I think exposing this escape hatch with some ignore: interface would be helpful.

[torresxb1]

We have a few issues asking for a similar thing in ignore: or a .samignore file. Closing so that we can track here: #185