[RESOLVED] The faulty hardware has been replaced and everything is back online.

WordCamp.org is currently offline due to a hardware issue. Should hopefully be resolved shortly.

[RESOLVED] We are currently investigating some reachability issues to wordpress.org. The WordPress.org API is not affected.

Network issues are resolved and access to WordPress.org is restored.

[RESOLVED] On Nov 12th 2024 beginning at 21:20 UTC, WordPress.org experienced intermittent elevated latency and increased error rates. These affected both the WordPress.org website and API. This initial incident was quickly resolved by the team, but reoccurred briefly on Nov 13 at 0110 UTC, 0900 UTC, and 1400 UTC. The underlying cause was network saturation of the hosting provider’s network that hosts some of the WordPress.org infrastructure. Although all of the servers and switch interfaces that our team has visibility into were within normal levels of utilization, there were some upstream network bottlenecks that created increased latency and packet loss that affected the WordPress.org environment. Timeline below:

Nov 12 2120 UTC – WordPress 6.7 released.
Nov 12 2220 UTC – Alerts received and adjustments made to reduce overall network utilization.
Nov 13 0110 UTC – Additional alerts received and more adjustments made to reduce overall network utilization even further.
Nov 13 0850 UTC – Attempt to increase the WordPress 6.7 rollout speed.
Nov 13 0900 UTC – Rollout percentage decreased due to additional increased latency and error rates.
Nov 13 1400 UTC – Another attempted to increase WordPress 6.7 rollout speed.
Nov 13 1410 UTC – Rollout percentage decreased due to additional increased latency and error rates.
Nov 13 1830 UTC – Workaround implemented for upstream provider network congestion.
Nov 13 2300 UTC – WordPress 6.7 rollout completed.

While we are still working with our provider to resolve the underlying issues within their network, the changes implemented during this incident should make future WordPress updates faster and more robust.

[RESOLVED] All WordPress.org services are currently offline. We are working with the hosting provider to determine the scope of the problem and how to resolve it.

0043 UTC – This is related to a INAP data center outage in Chicago. They are currently investigating.

0200 UTC – INAP says the root cause is a fiber cut and the ETA is 1-2 hours for partial restoration of services.

0210 UTC – We have updated DNS for wordpress.org, api.wordpress.org, and downloads.wordpress.org so that requests no longer timeout and an error page is returned quickly instead. Hopefully this should prevent slowdowns on WordPress sites checking for auto-updates, etc.

0300 UTC – Network services have been restored to the data center and WordPress.org services are back online.

UPDATE The switch maintenance is now complete.

UPDATE: Internap has announced a maintenance window starting Saturday May 16 at 2100 CDT to upgrade the software on the affected switches. They believe that the outage was caused by a software problem which is fixed in a more recent release. There should be no downtime as part of this maintenance.

Between 2000 and 2120 UTC today there were some intermittent connectivity issues which affected all of WordPress.org. In some cases this would have resulted in complete unreachability and in other cases it might have meant a 5XX error or higher than normal response time. The underlying cause appears to be a faulty network switch which has since been removed from production and we are working with the hosting provider to identify the exact issue before attempting to return the switch to production.