LiveJournal (livejournal) wrote,
LiveJournal
livejournal

О недоступности ЖЖ


Dear users,

As of writing LiveJournal has returned to regular functioning and we can give you some details as to what happened and the consequences.

On 15 May at 03:22 pm PST, the LiveJournal status monitoring system informed us of an error that occurred in the 7th user cluster (cluster name is Bratwurst¹). Analysis showed that several database structures were corrupted, which required correction and subsequent data recovery from a mirroring server. Initial repair estimates were 5-6 hours, so in an effort to avoid data loss in the event of the mirroring server malfunction, we made the decision to switch off the whole cluster. While the 7th cluster was being restored (which in itself is a resource-intesive process, as it is necessary to send large volumes of data over a local network which is already occupied by processes related to the correct functioning of LiveJournal) our monitoring system informed us of an analogous error on the 9th cluster (cluster name is ChickenTikka²). This required the same procedures that were implemented for the 7th cluster, but given that our network was already operating at full capacity due to the work being done to fix the cluster, the 5-6 hour fix estimate increased significantly.

All users located in these cluster lost all access to LiveJournal, irrespective of which pages they tried to visit. Given that some of the information on users and their posts is collected from clusters in real time, and both the 7th and 9th clusters were temporarily turned off, other users (who were not from these clusters) also experienced situational problems with LiveJournal. For example, if an entry from one of the affected clusters was due to appear in a friends feed, the feed would not load and give a 500 or 503 error. Similar behaviour was experienced if your inbox contained a message from someone from the 7th or 9th clusters. If a user has a friend from the affected cluster in one or more of his friends groups, the post creation page would not be accessible. As a result of this incident LiveJournal was working with varying degrees of stability for different users.

As of 09:12 pm PST 35% of the 7th cluster is restored, and 1.5% of the 9th cluster.
10:51 pm PST: UC7 — 50%, UC9 — 5%.
To optimise the network load, the restoration of the 9th cluster was postponed until the 7th cluster was fully restored. The 7th cluster was fully restored at 03:41 am PST and work on the 9th cluster was resumed. At 08:09 am PST the system informed us that some problems were being experienced in the restoration of the 9th cluster, which required intervention. As a result, the 9th cluster was only restored by 05:42 pm PST, while LiveJournal functionality returned to normal at 06:40 pm on 16th May.

Some users from the 9th cluster may still experience difficulties in seeing posts created in the last couple of days. We are working on restoring them from the archives. If any of your entries have gone missing, we would appreciate it very much if you bring it to the attention of the LiveJournal Support staff (http://www.livejournal.com/support/submit.bml), as it will help us to complete the process significantly faster.

It is worth noting that scheduled entry and notification functionality was disabled during the down time. After restoration these services are once again functioning, so scheduled entries were posted with a delay and the notifications queue has grown quite large, meaning it will be another few hours before the system catches up.

We have yet to identify the reason for the almost simultaneous malfunction of two clusters and it will take some time to investigate. We will inform you the results of our investigation once it is completed.

All users of paid services will be compensated with a one-week extension due the downtime.

We sincerely apologise for any inconvenience this has caused.

¹ Bratwurst — a veal and/or pork sausage prepared with a mixture of herbs.
² ChickenTikka — an Afghan national dish popular worldwide from chicken breast marinated in yoghurt with spices.

Tags: status
Subscribe
  • Post a new comment

    Error

    default userpic

    Your IP address will be recorded 

    When you submit the form an invisible reCAPTCHA check will be performed.
    You must follow the Privacy Policy and Google Terms of use.
  • 1072 comments
Уважаемые! Как раз 16-го в 1 час ночи написал несколько постов. Они исчезли. Когда их восстановят?
Что за загогулина эти кластеры??

strangeleopard

4 years ago

reklammer

4 years ago

sagetator

4 years ago

reklammer

4 years ago

sagetator

4 years ago

reklammer

4 years ago

coroner_spb

4 years ago

elsreference

4 years ago

livejournal

4 years ago

savage3000

4 years ago

livejournal

4 years ago

savage3000

4 years ago

icluzo

May 17 2013, 07:21:05 UTC 4 years ago Edited:  May 17 2013, 07:21:14 UTC

как страшно жить(с)
А не жить еще страшнее

andreygorckakov

4 years ago

wladsmart

4 years ago

mie_n_sport

4 years ago

metler

4 years ago

youloved.ru

4 years ago

Хочется ВАС поддержать!!! А то критики наверное много было. Держитесь и все у НАС будет хорошо!!! Я с ВАМИ!!! Успехов ребята!!!
спасибо вам большое!

rusiank

4 years ago

muzeta

4 years ago

livejournal

4 years ago

muzeta

4 years ago

alex987654321

4 years ago

killa_gramm

4 years ago

mie_n_sport

4 years ago

Suspended comment

to4ka_olya

4 years ago

Ну ладно хоть сделали, и то хорошо!
Стоит задуматься, почему во время судилища над Навальным, вылетел именно этот кластер...

Deleted comment

i_aga_5

4 years ago

pvn123

4 years ago

maximlancelot

4 years ago

pvn123

4 years ago

trifol

4 years ago

pvn123

4 years ago

trifol

4 years ago

maximlancelot

4 years ago

pvn123

4 years ago

maximlancelot

4 years ago

ivantama

4 years ago

pvn123

4 years ago

Зря решили опубликовать отложенные записи. Это произошло ночью и моментально. Кому это было нужно и кто это будет читать?!
когда в Киеве ночь, в других частях света утро или вечер прошлого дня....

Я в курсе

carabaas

4 years ago

sue888

4 years ago

carabaas

4 years ago

sue888

4 years ago

livejournal

4 years ago

carabaas

4 years ago

Spammed comment

carabaas

4 years ago

dance_45

4 years ago

carabaas

4 years ago

dance_45

4 years ago

ziptop

4 years ago

ya_snezhana

4 years ago

strangeleopard

4 years ago


Спасибо за разъяснения.

спасибо вам за понимание!

ru_deep_space

4 years ago

evgensemenihin

4 years ago

m4a3e8

4 years ago

ru_deep_space

4 years ago

evgensemenihin

4 years ago

Спасибо за разъяснения.
спасибо за поддержку!

wat

4 years ago

livejournal

4 years ago

Я нашел причину аварии: педофилы и политика. Всё просто!

Deleted comment

darkheavy

4 years ago

doctor_li

4 years ago

Deleted comment

doctor_li

4 years ago

vuntean

4 years ago

doctor_li

4 years ago

belkino

4 years ago

darkheavy

4 years ago

belkino

4 years ago

Главное, что уведомили о трудностях вовремя, хотя и так уже ясно, что если что-то не работает, значит будут чинить. Но уведомление своевременное всегда радует. ) Спасибо за работу, и за разъяснения.
А меня никто не уведомлял, только после обеда когда удалось загрузить ЖЖ, узнала о сбоях.

ovaleeva

4 years ago

bigmir81

4 years ago

ovaleeva

4 years ago

livejournal

4 years ago

ovaleeva

4 years ago

livejournal

4 years ago

doctor_li

4 years ago

ovaleeva

4 years ago

evgensemenihin

4 years ago

wfladjmjr

4 years ago

Спасибо за оперативное информирование и исправление недостатков.

alnikol

4 years ago

mr_espozito

4 years ago

alnikol

4 years ago

а в каком кластере я? и все ли с этим кластером хорошо?

skinny_bodycare

4 years ago

doctor_li

4 years ago

katinka83

4 years ago

shemberlen

4 years ago

Успехов в работе
спасибо!
Не сдавайтесь.
мы и так :)

Deleted comment

Поддержу вас коментом верю что вы которые работали, пахали как лошади.
снова здорова. упахались. опять.

будет ли восстановлена потеря СК из-за простоя?
Дался вам этот СК.

inna1903gr

4 years ago

belkino

4 years ago

delfin_1975

4 years ago

belkino

4 years ago

delfin_1975

4 years ago

belkino

4 years ago

delfin_1975

4 years ago

belkino

4 years ago

delfin_1975

4 years ago

belkino

4 years ago

delfin_1975

4 years ago

belkino

4 years ago

delfin_1975

4 years ago

belkino

4 years ago

delfin_1975

4 years ago

ingwar333

4 years ago

wfladjmjr

4 years ago

belkino

4 years ago

doctor_li

4 years ago

abra_akbar

4 years ago

inna1903gr

4 years ago

abra_akbar

4 years ago

mr_espozito

4 years ago

inna1903gr

4 years ago

mr_espozito

4 years ago