Docker should avoid re-using IPs too quickly

[PaulFurtado]

When allocating an IP for a container, libnetwork always chooses the lowest IP. This can be a big problem when a container exits and a different container running the same type of service starts on the same IP. Ex:

• MySQL client configured to retry on connection error
• A MySQL container for one cluster exits, and another MySQL container for a different cluster starts up with the same IP
• The client reconnects to the new IP, but it's a totally different database so now the errors are query errors for tables not existing, etc, so the connection is not retried.

This is just one example, but this can happen for just about any service, especially HTTP services in a microservice environment due to their quantity.

[fcrisciani]

There is a PR open that aims to allocate IPs sequentially: #1788

Regarding in a microservice application you would like to spawn different mysql clusters as different services, and you will use DNS in the application to always point to the correct instance. Also on a single network there should never be use of the same IP for 2 running containers.

Closing this one, feel free to follow up if you have further questions

[PaulFurtado]

Thanks @fcrisciani that PR looks like it will solve this!

Regarding in a microservice application you would like to spawn different mysql clusters as different services, and you will use DNS in the application to always point to the correct instance. Also on a single network there should never be use of the same IP for 2 running containers.

Just to follow up and clarify the problems... This is already what we do, but collisions can still happen relatively easily. We're running hundreds of MySQL clusters on Kubernetes among other data services. Ingress traffic from clients is generally never the problem because it comes through virtual service IPs, the problem is generally with direct communication between replicas. Things like redis-sentinel, zookeeper, hbase, kafka, and etcd all require direct container-to-container communication within their clusters. We've seen independent redis-sentinel clusters accidentally discover each other and "merge", or MySQL hosts accidentally trying to replicate from a different MySQL cluster, etc. Each of these individual cases has other solutions, eg: ensuring every cluster of a given service has different auth credentials so connections fail, etc; but avoiding rapid IP reuse is a simple change that really avoids a lot of edge cases in production so this is a very welcome change and will probably reduce pain for a lot of people. Thanks!

[fcrisciani]

sure @PaulFurtado no problem, thanks for clarifying the case. This commit mitigate for sure the condition but still when you will get close to finish the address pool you will still experience situation like this.