https://www.webratz.de/tags/scout24/Recent content in Scout24 on webratz.deHugoenWed, 10 Oct 2018 21:44:13 +0200https://www.webratz.de/article/nfs-bug-hunting/Wed, 10 Oct 2018 21:44:13 +0200https://www.webratz.de/article/nfs-bug-hunting/<p><em>This post appeared initially on the <a href="https://medium.com/scout24-engineering/ecs-docker-nfs-issue-hunt-down-81d44adac042">Scout24 Engineering Blog</a></em></p> <p>Who doesn’t know this situation: Sometime something weird is going on in your setup, but everything is so vague that you don’t know where to start. This is a story about a bug we experienced in a production setup and how we found out what was the root cause.</p> <h2 id="the-problem">The problem</h2> <p>Imagine a ECS cluster with multiple tasks running with an NFS (EFS in this case) backed persistent root volume. Launching new tasks works just fine but as soon you are doing a cluster upgrade complete hosts seem to be unable to start tasks or can only start a few and then break. But also some hosts just work fine. So scaling up or down your cluster is a really scary thing that could not be done anymore without manual intervention. That’s an unacceptable situation, but what to do?</p>