AWS Region not found in k8s cluster

[tdrozdowski]

Cannot find a valid AWS region when running in k8s cluster:

time="2017-01-30T23:19:15Z" level=debug msg="[Synchronize] Sleeping for 1m0s..." 
time="2017-01-30T23:19:15Z" level=info msg="[AWS] Listening for events..." 
time="2017-01-30T23:19:15Z" level=info msg="ADDED: kube-system/kubernetes-dashboard" 
time="2017-01-30T23:19:15Z" level=warning msg="[Service] The load balancer of service 'kube-system/kubernetes-dashboard' does not have any ingress." 
time="2017-01-30T23:19:15Z" level=info msg="ADDED: xcdn-staging/dns-auth" 
time="2017-01-30T23:19:15Z" level=info msg="[AWS] Processing (dns-auth.staging.xcdn.io, , ad50df968e73211e6bfeb023f1e3e2b4-1855914691.us-west-2.elb.amazonaws.com)\n" 
time="2017-01-30T23:19:15Z" level=info msg="ADDED: kube-system/collectd" 
time="2017-01-30T23:19:15Z" level=warning msg="[Service] The load balancer of service 'kube-system/collectd' does not have any ingress." 
time="2017-01-30T23:19:15Z" level=info msg="ADDED: default/kubernetes" 
time="2017-01-30T23:19:15Z" level=warning msg="[Service] The load balancer of service 'default/kubernetes' does not have any ingress." 
time="2017-01-30T23:19:15Z" level=info msg="ADDED: kube-system/kube-dns" 
time="2017-01-30T23:19:15Z" level=warning msg="[Service] The load balancer of service 'kube-system/kube-dns' does not have any ingress." 
time="2017-01-30T23:19:15Z" level=error msg="Error getting LBs: MissingRegion: could not find region configuration. Skipping..." 
time="2017-01-30T23:19:15Z" level=error msg="Error getting LBs: MissingRegion: could not find region configuration. Skipping..." 
time="2017-01-30T23:19:15Z" level=error msg="Canonical Zone ID for endpoint: ad50df968e73211e6bfeb023f1e3e2b4-1855914691.us-west-2.elb.amazonaws.com is not found" 
panic: runtime error: index out of range

goroutine 19 [running]:
panic(0x14d7a00, 0xc420014060)
	/usr/local/go/src/runtime/panic.go:500 +0x1a1
github.com/zalando-incubator/mate/consumers.(*awsConsumer).Process(0xc42039ed00, 0xc42026f4d0, 0xc420266e78, 0x3)
	/home/master/workspace/teabag_mate_master-5HM4GNJPGTJMNQYAZYN6JFXPZ2TWYP4JPPHRCHUMWCZK6LHVMNQA/mate/_jenkins_build/go/src/github.com/zalando-incubator/mate/consumers/aws.go:190 +0x45a
github.com/zalando-incubator/mate/consumers.(*awsConsumer).Consume(0xc42039ed00, 0xc4203cd080, 0xc4203cd0e0, 0xc4203cd140, 0xc4201a7d50)

[tdrozdowski]

k8s version is 1.4.7 - am upgrading to 1.5.2 to see if that makes any difference.

Using fleet to manage k8s - so wondering if perhaps I missing something there (a flag I need to set on the kubelets?)

[ideahitme]

@tdrozdowski as a quick fix I would suggest u to include the region as an environment variable AWS_REGION=us-west-2 into the deployment. For example:

      containers:
      - name: mate
        image: registry.opensource.zalan.do/teapot/mate:v0.5.1
        env:
        - name: AWS_REGION
          value: us-west-2
        args:
        - --producer=kubernetes
...

Meanwhile we will look for a better way to handle the issue.

[tdrozdowski]

I can confirm that helps - created the records almost immediately!

thank you!

[snoby]

I'm running into the same issue.