Error: Text file busy (os error 26) - (GKE COS w/ v1.20.11-gke.1300)

Hello,

Context

We're deploying core-dump-handler. So far it has worked with our dev cluster which is using the version 1.21.9-gke.1002 of Kubernetes. No luck with our prod cluster. Relevant specs are below:

GKE control plane version: 1.20.15-gke.3400
GKE worker nodes version:: 1.20.11-gke.1300
GKE workers image type: Container-Optimized OS with containerd (cos_containerd)

Otherwise, the parameters used to configure both our dev and prod cluster are the same.

Helm non-sensitive values:

daemonset:
  hostDirectory: "/home/kubernetes/bin"
  coreDirectory: "/home/kubernetes/cores"
  extraEnvVars: |-
    - name: S3_ENDPOINT 
      value: "https://storage.googleapis.com"
image:
  tag: v8.0.3 ***************(also tried with v8.3.1)***************
composer:
  logLevel: "Error"

Description of the issue.

Some pod for 2 to 3 of our 14 workers nodes in prod (again using the version 1.20.11-gke.1300 for the worker nodes) falls into the CrashLoopBackOff state. Here's logs when the issue happens:

[2022-04-26T20:58:18Z INFO  core_dump_agent] no .env file found 
     That's ok if running in kubernetes
[2022-04-26T20:58:18Z INFO  core_dump_agent] Setting host location to: /home/kubernetes/bin
[2022-04-26T20:58:18Z INFO  core_dump_agent] Current Directory for setup is /app
[2022-04-26T20:58:18Z INFO  core_dump_agent] Copying the composer from ./vendor/default/cdc to /home/kubernetes/bin/cdc
Error: Text file busy (os error 26)
Stream closed EOF for observe/core-dump-handler-2kbvh (coredump-container)

Let me know if you require additional info from my end.

Hi - Thanks for using the project and the detailed bug report.

Given the Error: Text file busy (os error 26) which is usually caused when the OS is trying to run a binary when it's being copied and the fact it's only happening on 2-3 nodes it looks like it's some sort of race condition on the deploy.

This is unusual as the core agent deploys the composer before changing the sysctls

A couple of questions:

It seems very unlikely but is it possible that there are core dumps happening in the production environment as the chart is being deployed?

Did this happen on the very first install or part of an upgrade?

If it is an upgrade can you confirm that you are removing the chart and installing it when upgrading as outlined here

I'll try and replicate this week but it's likely to be Friday before I get to do an install.

It seems very unlikely but is it possible that there are core dumps happening in the production environment as the chart is being deployed?

Other than this project, we have nothing else that is in place that interacts with the kernel to generate coredumps on our GKE workers. I'm not sure if they are even enabled by default.

Did this happen on the very first install or part of an upgrade?

It happened on the second install (I initially tried to upgrade v8.0.3 to v8.3.1 due to the agents to picking up on the segfaulter test container crashing.)

If it is an upgrade can you confirm that you are removing the chart and installing it when upgrading as outlined here

I didn't upgraded this way. I've used the helm upgrade command.
I've just tried it using the instructions without success (tried with v8.0.3 and v8.3.1)

OK the fact it happened on the second try and upgrade was used is starting to give a little insight.
When testing helm upgrade previously it didn't work hence the update note in the README.

Specifically the patch strategy of helm/k8s seemed to create order issues between the termination of the old container cleanup code and the new container deployment code.

Current recommendation is to remove the chart and then install as per docs.

The long term fix would be to put the deploy of the host config and the agent into two separate containers and manage them through an operator but that is not in the current road map.

As an aside: For the question on core dumps happening I was looking to find out if you have other containers crashing apart from not have you installed

For the question on core dumps happening I was looking to find out if you have other containers crashing apart from not have you installed

Yes, I confirm that some containers were crashing before and during the installation of coredump helm chart.

OK other containers crashing during the install will cause the text file busy error.
The code tries to mitigate against it but it doesn't guarantee that it won't happen.
If the uninstall and reinstall doesn't work during a period where there are no crashing containers then please update this issue

Just tried again (following the install/update notes) in the prod cluster when there's no container crashing and the result is the same. It's actually the same three nodes that has this issue.

Is there some element holding a "state" in the faulty nodes that could be cleared/reset?

It's likely the kernel thinks that something is using the cdc exe.
From where the deploy process has got to the error is raised when copying the cdc file to the host machine
https://github.com/IBM/core-dump-handler/blob/main/core-dump-agent/src/main.rs#L435
If you can get onto the host machine and run lsof | grep cdc you might be able to see if a process is holding on to the cdc file.

I've used toolbox for CoreOs in order to troubleshoot the issue after connecting inside one of the GKE nodes. The command ps was used to find processes using /home/kubernetes/bin/cdc

The issue I'm seeing is that some segfaulter process that I ran weeks ago gets stuck when I executed by /home/kubernetes/bin/cdc.

Examples:

# ps aux | grep cdc
root     2070765  0.0  0.0   3080   896 ?        S+   15:54   0:00 grep cdc
root     3423665  0.0  0.0  13584  2480 ?        S    Apr26   0:00 /home/kubernetes/bin/cdc -c=18446744073709551615 -e=segfaulter -p=1 -s=4 -t=1650983080 -d=/home/kubernetes/cores -h=segfaulter -E=!usr!local!bin!segfaulter
root     3425383  0.0  0.0  13584  2492 ?        S    Apr26   0:00 /home/kubernetes/bin/cdc -c=18446744073709551615 -e=segfaulter -p=1 -s=4 -t=1650983162 -d=/home/kubernetes/cores -h=segfaulter -E=!usr!local!bin!segfaulter
root     3427276  0.0  0.0  13584  2360 ?        S    Apr26   0:00 /home/kubernetes/bin/cdc -c=18446744073709551615 -e=segfaulter -p=1 -s=4 -t=1650983257 -d=/home/kubernetes/cores -h=segfaulter -E=!usr!local!bin!segfaulter

Some production workloads that crashed were also stuck.

Killing the processes with kill <pid> then restarting the core-dump pod for that worker node fixes the issue related to the pod not starting.

Since this issue is resolved, should we close this thread and open a new one for the issue regarding core-dumps freezing when being executed by the cdc process?

Glad you have it resolved.
It would be great to know what is causing cdc to hang. Does this issue reoccur now you have upgraded?
I think the mitigation for this would be to have cdc spawn a supervised task that has a timeout.
Please feel free to close this and open up an issue for cdc freezing issues.

Closing this and linking it to the systemd requirement as that would provide supervised process support if required.

Navigation Menu

Repository navigation

Error: Text file busy (os error 26) - (GKE COS w/ v1.20.11-gke.1300) #86

Context

Description of the issue.

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Error: Text file busy (os error 26) - (GKE COS w/ v1.20.11-gke.1300) #86

Description

Context

Description of the issue.

Activity

No9 commented on Apr 27, 2022

dsaintilma-flinks commented on Apr 27, 2022

No9 commented on Apr 28, 2022

dsaintilma-flinks commented on Apr 29, 2022

No9 commented on Apr 29, 2022

dsaintilma-flinks commented on Apr 29, 2022

No9 commented on Apr 29, 2022

dsaintilma-flinks commented on May 3, 2022

No9 commented on May 4, 2022

No9 commented on May 21, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions