1. Mailing Lists
  2. Contributors
  3. Re: initialization of qweb template caches on website

Contributors mailing list archives

contributors@odoo-community.org

Browse archives

Re: OCA Repositories in Odoo Apps Store

Re: OCA Repositories in Odoo Apps Store

Avatar

Re: initialization of qweb template caches on website

by
Camptocamp France SAS, Alexandre Fayolle
- 09/06/2020 09:15:31 
Thanks for jumping on this Moises :)

Yes I am able to reproduce locally, and I did run some profiling before
asking :)

Attached are 2 profiling sessions loading the same page with py-spy
(https://pypi.org/project/py-spy/), once with an uninitialized worker,
and once with an initialized worker.

What I see is that a significant amount of time is spent in the flatten
method (odoo/http.py:1270) in the uninitialized worker run, and this is
not present in the initialized worker run (meaning the result of flatten
is cached, and we see this in the call stack).

The database is not on the critical path here, as far as I can tell,
it's a pure python issue, and caching cannot directly help, because the
performance issue is directly caused by the cache initialization.

So what I would like to do is preload the cache when the worker starts.
With a "normal" system, this would be done by browsing the site after it
starts, but with Odoo's recycling workers, it is harder to do from
outside. And besides, I found out that the caches are being invalidated,
and we had to disable the event reminder cron which is invalidating all
caches every 30min.

I'll check with the customer if he is ok with sharing a database dump,
and also with our GDPR guys here at Camptocamp.

Alexandre


On 09/06/2020 06:32, Moises Lopez wrote:

> Hi Alexandre,

> 

>> ...issue related to qweb templates on the website. *The first time a

> worker serves these*...

> 

> I agree with you

>  - If it is only the first time so it confirms that there is a slow

> cached process.

>  - Since the first time was slow (caching) and the second one it is not.

> (If the second time is using the same worker).

> 

> 1) Can you reproduce it even if you are not using a proxy with cache

> (e.g. nginx)?

> 

> If we analyze Odoo we can look for the cached methods:

>  - rgrep "\@.*cache(" . --include=*.py --exclude-dir=test* -A2 |grep " 

>   def "

> 

> The following method are cached in Odoo and enterprise projects:

>  - https://gist.github.com/moylop260/a3394674e570da8ca457ef4fe2eb4176

> 

> In our experience slowest are:

>  - ir_rule._compute_domain:

>    - computing a lot of records

>    - computing relational fields (field_o2m.field_id.name

> ) 

>    - Using a computed field

>  - ir_translation.__get_source:

>    - A lot of terms to translate in the qweb view

>    - A lot of fields(translate=True) renderized

> 

> But if you like to find the culprit... Are you able to run profilers?

> 

> Check the following cases detected before using profilers:

>  - https://github.com/odoo/odoo/pull/49690

>  - https://github.com/odoo/odoo/pull/25196

> 

> 

> - Enable advanced logs in postgresql to get a pgbadger report to detect:

>    - most executed queries

>    - slowest executed queries

>    - missing index

>    - locked records

> - Run py-spy to get a flame to detect:

>   - slowest python methods

> 

> Running both at the same time you can cross information and look for the

> culprit easier.

> 

> 

> El lun., 8 jun. 2020 a las 8:22, Gustavo Valverde

> (<gustavo@iterativo.do>) escribió:

> 

>     Hi Alexandre, 

> 

>     This is an interesting subject, as it's kind of hard to find a sweet

>     spot.

> 

>     I've been dealing with Odoo performance improvements in different

>     scenarios for a few months now. I thought I had most of it figured

>     out until I started dealing with ecommerce and fighting with

>     Google's pagespeed. 

> 

>     I found out multiple things have to be taken into consideration when

>     dealing with websites:

> 

>       * If you have a high amount of workers, restarting or redeploying

>         new changes make things hellish slow.

>       * Mixed setups (multiprocess for the backend + multithread for the

>         website) make things better; but it depends on the load and

>         amount of online visitors that you could have at the same time.

>       * Nginx caching and CDNs are your friends. Don't be afraid of

>         caching /web/content, but do not cache it for excessive times.

> 

>     Regards,

> 

>     On Mon, Jun 8, 2020 at 8:22 AM Alexandre Fayolle

>     <alexandre.fayolle@camptocamp.com

>     <mailto:alexandre.fayolle@camptocamp.com>> wrote:

> 

>         Hello,

> 

>         A customer (odoo 12.0) here is facing a performance issue related to

>         qweb templates on the website. The first time a worker serves these, we

>         get terrible performance. I suspect we are not the only ones facing

>         this. Using an external service to GET /shop for instance on a regular

>         basis sort of alleviates this, but this is really unelegant.

> 

>         Is there a killer module I missed which is able to preload the cache for

>         a number of views each time a worker is started? Or some other strategy?

> 

>         Thanks for your feedback

> 

>         -- Alexandre Fayolle Chef de Projet Tel : +33 4 58 48 20 30

>         Camptocamp France SAS 18 rue du Lac Saint André 73 370 Le

>         Bourget-du-Lac France http://www.camptocamp.com

> 

>         _______________________________________________

>         Mailing-List: https://odoo-community.org/groups/contributors-15

>         Post to: mailto:contributors@odoo-community.org

>         <mailto:contributors@odoo-community.org>

>         Unsubscribe: https://odoo-community.org/groups?unsubscribe

> 

> 

> 

>     -- 

> 

>     Gustavo Valverde

> 

>     Chief Technology Officer

> 

>     M(829) 686 8270 D  (829) 956 5191 ext. 202

> 

>     E  gustavo@iterativo.do

> 

> 

>     	

> 

>     ITERATIVO SRL

> 

>     Roble Corporate Center, Piso 7

> 

>     Rafael Augusto Sánchez 86, Piantini, Santo Domingo, R. Dominicana

> 

> 

>     T(829) 956 5191| Einfo@iterativo.do | www.iterativo.do

>     
> 
> 
>     Member firm of ITERATIVO LLC & Progressa Corporate Group
> 
> 
>     Este mensaje y sus anexos pueden contener información confidencial y
>     privilegiada con la intención de que sea utilizada por las personas
>     u organizaciones a quienes esta dirigida, por lo que su uso es
>     exclusivo para su destinatario. Si usted ha recibido este mensaje
>     por error, favor de eliminarlo e informar al remitente del mensaje a
>     través de un correo de respuesta. Si este es el caso, le notificamos
>     que queda estrictamente prohibida la distribución o reproducción de
>     este e-mail y/o sus anexos. iterativo no se hace responsable de las
>     opiniones vertidas en esta comunicación que no estén acordes con su
>     quehacer y fines, y que no se revistan de un carácter oficial. 
> 
>     This message and its enclosures may contain confidential and
>     privileged information intended for the use of people and
>     organizations to which it is directed and its use is thus limited to
>     its addressee. If you have received this message by mistake, please
>     eliminate it and inform the sender through a reply message. Should
>     this be the case, you are advised that the distribution or
>     reproduction of this e-mail and/or any attachments contained herein
>     is strictly forbidden. iterativo is not liable for opinions
>     expressed in this message which may not coincide with its
>     responsibilities and purpose and which may not express official
>     matters. 
> 
>     __ __
> 
>     https://iterativo.do/web/image/39802/hoja-verde.png____
> 
>     /No imprimas este correo si no es necesario. Cuidemos del ambiente!/____
> 
>     /Do not print this mail if it's not necessary. Let's take care of
>     the environment!/
> 
>     _______________________________________________
>     Mailing-List: https://odoo-community.org/groups/contributors-15
>     Post to: mailto:contributors@odoo-community.org
>     <mailto:contributors@odoo-community.org>
>     Unsubscribe: https://odoo-community.org/groups?unsubscribe
> 
> 
> 
> -- 
> Moisés López Calderón
> Mobile: (+521) 477-752-22-30
> Twitter: @moylop260
> hangout: moylop260@vauxoo.com <mailto:moylop260@vauxoo.com>
> http://www.vauxoo.com - Odoo Gold Partner
> Twitter: @vauxoo
> 
> _______________________________________________
> Mailing-List: https://odoo-community.org/groups/contributors-15
> Post to: mailto:contributors@odoo-community.org
> Unsubscribe: https://odoo-community.org/groups?unsubscribe
> 


-- 
Alexandre Fayolle
Chef de Projet
Tel : +33 4 58 48 20 30

Camptocamp France SAS
18 rue du Lac Saint André
73 370 Le Bourget-du-Lac
France

http://www.camptocamp.com

Reference