Codex infrastructure issue

Incident Report for GjirafaTech

Resolved

One of the nodes on the distributed infrastructure of Codex experienced memory issues causing some applications running on it to have latencies. This caused requests timeouts to users visiting Codex admin or Zeny website. Due to the nature of the distributed architecture we can't know the exact percentage of requests that experienced problems, because it depends on the instances of applications running on that specific node and multiple requests happening in the background, but we suspect that it might impact up to 40% of the requests (mainly Codex Admin and Codex Edge).

The issue started to happen during the night (early morning) and was resolved in the morning after it was seen by the infrastructure team. The team is also analyzing how to prevent such cases in the future.

Posted Feb 05, 2023 - 09:30 CET