1. Mailing Lists
  2. Contributors
  3. Re: Runbot long term to install packages and other features

Contributors mailing list archives

contributors@odoo-community.org

Browse archives

Re: Runbot long term to install packages and other features

Re: Runbot long term to install packages and other features

Avatar

Re: Runbot long term to install packages and other features

by
Camptocamp France SAS, Alexandre Fayolle
- 10/08/2016 11:01:14 
Hello Daniel,

The OCA's runbot is currently running something very close to Odoo's
runbot. The main difference is that we have disabled the tests from the
addons in order to save some CPU, because these are run in travis.

Among the constraints that we have in the current installment of the OCA
runbot:

* all the builds share a common environment, including external python
dependencies. We could add a step to process the requirements.txt of a
repository before starting a build to make sure these are processed
correctly

* same thing for external non-python dependencies, only installing these
require root priviledges, and I'm quite conservative in whom I grant
these on the servers (and I admit I've not been reactive enough in
handling these issues in the past few months. I'm getting organized to
improve this)

These limitations come from the design of the runbot, which is meant to
test odoo core where the external dependencies are known in advance and
rarely change.

Having docker based builds means the second point could be solved with
non root priviledge, but at a cost which is not clear to me regarding
server ressource consumption.

Currently, the OCA runbot infrastructure is uses 3 servers:

* runbot1
* runbot2
* oca-db

Each runbot runs 350 odoo instances and the related PostgreSQL
databases. Each instance uses 2 databases (one "base" and one "all").
The CPU load is typically quite low (2-3) and the memory consumption is
~60GB. Currently, the tree for the files of the runbot instance + the
build instances consumes ~300GB on each server.

The oca-db server hosts the runbot's instance database, which is used to
receive the logs from the runs (so it is tuned to accept a lot of
simultaneous connections). That server also hosts the main
odoo-community instance (odoo + database).




On 10/08/2016 09:53, Daniel Reis wrote:
> I must say I'm not familliar with Runbot, so forgive me if I say
> something stupid :-)
> 
> Why doesn't Runbot simply reuses the scripts from MQT?
> They provide already all the "machinery" to have a repos properly
> installed (and tested).
> And today pip provides us caching that can make these builds lighter.
> We can even foresee to optimize MQT for this specific use case.
> 
> 
> Regards
> Daniel
> 
>  
> 
> Citando Moises Lopez :
> 
>> ** History*
>>  
>> *A) Install packages*
>> The current process to install packages for our runbot for OCA is:
>> 1. Request to install pip and apt packages.
>> 2. Wait manual installing by server-admin.
>> 3. Receive the confirmation of package installed.
>> 4. Rebuild manually.
>>  
>> Between 2 and 3 we could wait a pair of days or weeks
>> The worst example is
>> https://github.com/OCA/report-print-send/issues/54 (1 month).
>>  
>> If we need of one person this delay is normal (anyone who is).
>>  
>> But we have a requirements.txt file that we can re-use to install
>> dependencies on runbot.
>>  
>>  
>> *B) Reproduce travis errors*
>> Currently we have errors from travis that we can't reproduce on local
>> and we don't have access to the build of travis to debug that environment.
>>  
>> Example: https://github.com/OCA/maintainer-tools/pull/209
>> Other example: https://github.com/OCA/runbot-addons/pull/53
>> We can't reproduce this error from other environment the last one have
>> a video of how to detect them with the same environment:
>> https://youtu.be/r6-IE87UZQc
>>  
>> *C) Repositories dependencies*
>> Currently we have a runbot where we need configure manually the
>> repositories dependencies, but we have a oca_dependencies.txt.
>> Runbot have a method to process them and MQT have other one for travis.
>> We can re-use it too (similar case from requirements.txt) and avoid
>> duplicated code and re-configurate 2 times the same thing.
>>  
>>  
>> ** Proposal*
>>  
>> *- Pieces*
>> The tool travis2docker 
>> generates a Dockerfile base on .travis.yml (86% of coverage)
>>  
>> Travis has the docker image available
>> 
>> (Our use a custom image with custom packages pre-installed and other
>> things to make caching
>> https://hub.docker.com/r/vauxoo/odoo-80-image-shippable-auto/)
>>  
>> All repos of OCA are using a .travis.yml with MQT centralized.
>>  
>> We have a odoo module runbot_travis2docker
>>  for
>> version 9.0 and 94% of coverage
>>  
>> *- Join them*
>> We have all pieces to join them to fix A) B) and C).
>>  
>> With this pieces if you add a package from .travis.yml,
>> requirements.txt or oca_dependencies we re-use the same code of MQT to
>> generate the instance with all apt and pip packages, and all
>> repositories dependencies recursively.
>> We don't need extra custom configuration from runbot instance or
>> server commands. 
>>  
>> FYI we have a stable runbot using on http://runbot.vauxoo.com/runbot
>> (Offtopic with a isolated environment we have a ssh conexion feature
>> to debug direclty from container if a error is hardly to reproduce
>> locally).
>>  
>> It's 100% is compatible with others CI like as @lasley told us here
>> 
>>  
>> *Conclusion*
>> My proposal is migrate our current runbot to dockerized one.
>> @Lasley @alan196 and me give you our help to configure and fix new
>> issues.
>>  
>> FYI @Alan196 send us a video of how configure it here
>> 
>>  
>> Note: We have other cool tools, but many of them require a extra file
>> or extra configuration to work.
>>  
>> What do you think?
>>  
>>  
>> -- 
>> Moisés López Calderón
>> Vauxoo - OpenERP's Gold Partner
>> Mobile: (+521) 477-752-22-30
>> Office: (+52) 477-773-33-46
>> web: http://www.vauxoo.com
>> twitter: @vauxoo
>>            @moylop260           
>> hangout: moylop260@vauxoo.com <mailto:moylop260@vauxoo.com>
>>  
>>
>> _______________________________________________
>> Mailing-List: http://odoo-community.org/groups/contributors-15
>> Post to: mailto:contributors@odoo-community.org
>> Unsubscribe: http://odoo-community.org/groups?unsubscribe
>>
> 
>  
> 
> _______________________________________________
> Mailing-List: http://odoo-community.org/groups/contributors-15
> Post to: mailto:contributors@odoo-community.org
> Unsubscribe: http://odoo-community.org/groups?unsubscribe
> 


-- 
Alexandre Fayolle
Chef de Projet
Tel : +33 4 58 48 20 30

Camptocamp France SAS
Savoie Technolac, BP 352
73377 Le Bourget du Lac Cedex
http://www.camptocamp.com

Reference