Contributors mailing list archives


Re: runbot is down

Acsone SA/NV, Stéphane Bidoul
- 11/10/2016 09:33:04
Hi Alexandre,

"internal postgres corruption" looks pretty scary. Any idea on the root cause or lesson we can learn here?


On Tue, Oct 11, 2016 at 9:23 AM Alexandre Fayolle <> wrote:
On 10/10/2016 18:40, Alexandre Fayolle wrote:
> On 10/10/2016 18:27, Alexandre Fayolle wrote:
>> Hello,
>> I just diagnosed a bad case of postgresql crash on the runbot servers.
>> I'm repairing, but current state of affairs is that our runbots are
>> down. I'm on it and will keep you posted.
> OK, runbots are back up, it took some rebooting and persuasion to get
> postgresql back up on both servers. Consequence is that all running
> instances are now down. I've triggered some rebuilds but you may need to
> commit --amend and push -f on your PR to trigger one if I missed your
> branch.
> Sorry for the inconvenience. I'll keep an eye on this tomorrow.

Well well well... What I did yesterday did not work, because resetting a
build expects the build directory to exist, and I had made some cleanup.
Things should be mostly back to normal this morning, but the consequence
of this is that if you need a build for a branch for which none is
available, you cannot just ask for a rebuild, a new commit on the PR
(and a push) is needed.

Once again, I'm sorry for the inconvenience.

Alexandre Fayolle
Chef de Projet
Tel : +33 4 58 48 20 30

Camptocamp France SAS
Savoie Technolac, BP 352
73377 Le Bourget du Lac Cedex

Post to: