On September 28th the Facebook website and app went down for most users.
This was the third Facebook outage over the past few weeks. Facebook users took refuge in other social media websites to get their fix of viral content and to vent their frustration.
While Facebook is back up and running smoothly again now, the absences make us curious about what happened and whether it’s likely to happen again.
Luckily, Facebook wrote a blog post explaining some of the technical reasons for the outage.
The problem started last week, during an earlier shut down. At the root of the problem was a “configuration issue.” When an error in the configuration occurred, Facebook’s effort to fix the problem accidentally made it worse. Facebook uses an automated system to verify configuration values.
When the system sees an invalid configuration, it replaces it with a valid one. However, in this case, Facebook’s store of “valid” values was also corrupted, so the system was replacing incorrect configurations with more incorrect ones.
This created a loop of repeated queries attempting to correct the invalid configurations. Basically, the automated system not only didn’t fix the configuration problem, but clogged up Facebook’s database.
In order to shut down this endless loop, Facebook had to turn off the site. Facebook then fixed the root of the problem and slowly allowed users back onto the site. As a temporary fix, Facebook disabled its automated system for fixing configuration values. During the most recent shutdown, a configuration problem was once again the culprit.
At the moment, Facebook is working on developing a new system that will be better able to deal with configuration problems.
Their blog post stated, “We’re exploring new designs for this configuration system following design patterns of other systems at Facebook that deal more gracefully with feedback loops and transient spikes.”
Hopefully this new system will be in place soon, and it will prevent future outages.
Written by content manager Meghan Woolley
Please subscribe to our newsletter about social media news and strategies for social media marketing in your inbox every week.