Contributors mailing list archives
contributors@odoo-community.org
Browse archives
Re: initialization of qweb template caches on website
by
Vauxoo, Moisés López Calderón
Hi Alexandre
I just checked the svg files attached in your previous email.
The cache process is consuming 1/4 of the total time:
In your profiling you are using 4 workers and the cache process was computed for 2 workers (pid-126 and pid-130)
The issue is worse if you are using 20 workers so this cache will be computed 20 times.
And even worse if you have a method clearing cache so often since that it will clear the cache of all methods in all workers.
The following methods are clearing cache in Odoo:
It means that if you create/unlink/modify a new crm.reveal.rule so the website will be slow again even if the crm.reveal.rule is not affecting the website.
The same case for:
The worst cases are cron of minutes clearing cache:
- Check the following e.g. of a fixed issue related https://github.com/odoo/odoo/issues/25029
But you still can have a cron calling a method clearing caches.
Because of all this I have created the following issue:
Check the Olivier Dony answer.
Regards!
El mar., 9 jun. 2020 a las 2:17, Alexandre Fayolle (<alexandre.fayolle@camptocamp.com>) escribió:
Thanks for jumping on this Moises :) Yes I am able to reproduce locally, and I did run some profiling before asking :) Attached are 2 profiling sessions loading the same page with py-spy (https://pypi.org/project/py-spy/), once with an uninitialized worker, and once with an initialized worker. What I see is that a significant amount of time is spent in the flatten method (odoo/http.py:1270) in the uninitialized worker run, and this is not present in the initialized worker run (meaning the result of flatten is cached, and we see this in the call stack). The database is not on the critical path here, as far as I can tell, it's a pure python issue, and caching cannot directly help, because the performance issue is directly caused by the cache initialization. So what I would like to do is preload the cache when the worker starts. With a "normal" system, this would be done by browsing the site after it starts, but with Odoo's recycling workers, it is harder to do from outside. And besides, I found out that the caches are being invalidated, and we had to disable the event reminder cron which is invalidating all caches every 30min. I'll check with the customer if he is ok with sharing a database dump, and also with our GDPR guys here at Camptocamp. Alexandre On 09/06/2020 06:32, Moises Lopez wrote: > Hi Alexandre, > >> ...issue related to qweb templates on the website. *The first time a > worker serves these*... > > I agree with you > - If it is only the first time so it confirms that there is a slow > cached process. > - Since the first time was slow (caching) and the second one it is not. > (If the second time is using the same worker). > > 1) Can you reproduce it even if you are not using a proxy with cache > (e.g. nginx)? > > If we analyze Odoo we can look for the cached methods: > - rgrep "\@.*cache(" . --include=*.py --exclude-dir=test* -A2 |grep " > def " > > The following method are cached in Odoo and enterprise projects: > - https://gist.github.com/moylop260/a3394674e570da8ca457ef4fe2eb4176 > > In our experience slowest are: > - ir_rule._compute_domain: > - computing a lot of records > - computing relational fields (field_o2m.field_id.name > ) > - Using a computed field > - ir_translation.__get_source: > - A lot of terms to translate in the qweb view > - A lot of fields(translate=True) renderized > > But if you like to find the culprit... Are you able to run profilers? > > Check the following cases detected before using profilers: > - https://github.com/odoo/odoo/pull/49690 > - https://github.com/odoo/odoo/pull/25196 > > > - Enable advanced logs in postgresql to get a pgbadger report to detect: > - most executed queries > - slowest executed queries > - missing index > - locked records > - Run py-spy to get a flame to detect: > - slowest python methods > > Running both at the same time you can cross information and look for the > culprit easier. > > > El lun., 8 jun. 2020 a las 8:22, Gustavo Valverde > (<gustavo@iterativo.do>) escribió: > > Hi Alexandre, > > This is an interesting subject, as it's kind of hard to find a sweet > spot. > > I've been dealing with Odoo performance improvements in different > scenarios for a few months now. I thought I had most of it figured > out until I started dealing with ecommerce and fighting with > Google's pagespeed. > > I found out multiple things have to be taken into consideration when > dealing with websites: > > * If you have a high amount of workers, restarting or redeploying > new changes make things hellish slow. > * Mixed setups (multiprocess for the backend + multithread for the > website) make things better; but it depends on the load and > amount of online visitors that you could have at the same time. > * Nginx caching and CDNs are your friends. Don't be afraid of > caching /web/content, but do not cache it for excessive times. > > Regards, > > On Mon, Jun 8, 2020 at 8:22 AM Alexandre Fayolle > <alexandre.fayolle@camptocamp.com > <mailto:alexandre.fayolle@camptocamp.com>> wrote: > > Hello, > > A customer (odoo 12.0) here is facing a performance issue related to > qweb templates on the website. The first time a worker serves these, we > get terrible performance. I suspect we are not the only ones facing > this. Using an external service to GET /shop for instance on a regular > basis sort of alleviates this, but this is really unelegant. > > Is there a killer module I missed which is able to preload the cache for > a number of views each time a worker is started? Or some other strategy? > > Thanks for your feedback > > -- Alexandre Fayolle Chef de Projet Tel : +33 4 58 48 20 30 > Camptocamp France SAS 18 rue du Lac Saint André 73 370 Le > Bourget-du-Lac France http://www.camptocamp.com > > _______________________________________________ > Mailing-List: https://odoo-community.org/groups/contributors-15 > Post to: mailto:contributors@odoo-community.org > <mailto:contributors@odoo-community.org> > Unsubscribe: https://odoo-community.org/groups?unsubscribe > > > > -- > > Gustavo Valverde > > Chief Technology Officer > > M(829) 686 8270 D (829) 956 5191 ext. 202 > > E gustavo@iterativo.do > > > > > ITERATIVO SRL > > Roble Corporate Center, Piso 7 > > Rafael Augusto Sánchez 86, Piantini, Santo Domingo, R. Dominicana > > > T(829) 956 5191| Einfo@iterativo.do | www.iterativo.do > > > > Member firm of ITERATIVO LLC & Progressa Corporate Group > > > Este mensaje y sus anexos pueden contener información confidencial y > privilegiada con la intención de que sea utilizada por las personas > u organizaciones a quienes esta dirigida, por lo que su uso es > exclusivo para su destinatario. Si usted ha recibido este mensaje > por error, favor de eliminarlo e informar al remitente del mensaje a > través de un correo de respuesta. Si este es el caso, le notificamos > que queda estrictamente prohibida la distribución o reproducción de > este e-mail y/o sus anexos. iterativo no se hace responsable de las > opiniones vertidas en esta comunicación que no estén acordes con su > quehacer y fines, y que no se revistan de un carácter oficial. > > This message and its enclosures may contain confidential and > privileged information intended for the use of people and > organizations to which it is directed and its use is thus limited to > its addressee. If you have received this message by mistake, please > eliminate it and inform the sender through a reply message. Should > this be the case, you are advised that the distribution or > reproduction of this e-mail and/or any attachments contained herein > is strictly forbidden. iterativo is not liable for opinions > expressed in this message which may not coincide with its > responsibilities and purpose and which may not express official > matters. > > __ __ > > https://iterativo.do/web/image/39802/hoja-verde.png____ > > /No imprimas este correo si no es necesario. Cuidemos del ambiente!/____ > > /Do not print this mail if it's not necessary. Let's take care of > the environment!/ > > _______________________________________________ > Mailing-List: https://odoo-community.org/groups/contributors-15 > Post to: mailto:contributors@odoo-community.org > <mailto:contributors@odoo-community.org> > Unsubscribe: https://odoo-community.org/groups?unsubscribe > > > > -- > Moisés López Calderón > Mobile: (+521) 477-752-22-30 > Twitter: @moylop260 > hangout: moylop260@vauxoo.com <mailto:moylop260@vauxoo.com> > http://www.vauxoo.com - Odoo Gold Partner > Twitter: @vauxoo > > _______________________________________________ > Mailing-List: https://odoo-community.org/groups/contributors-15 > Post to: mailto:contributors@odoo-community.org > Unsubscribe: https://odoo-community.org/groups?unsubscribe > -- Alexandre Fayolle Chef de Projet Tel : +33 4 58 48 20 30 Camptocamp France SAS 18 rue du Lac Saint André 73 370 Le Bourget-du-Lac France http://www.camptocamp.com_______________________________________________
Mailing-List: https://odoo-community.org/groups/contributors-15
Post to: mailto:contributors@odoo-community.org
Unsubscribe: https://odoo-community.org/groups?unsubscribe
Moisés López Calderón
Mobile: (+521) 477-752-22-30
Twitter: @moylop260
Twitter: @vauxoo
Reference
-
initialization of qweb template caches on website
byCamptocamp France SAS, Alexandre Fayolle- 08/06/2020 14:20:20 - 0-
Re: initialization of qweb template caches on website
byVauxoo, Moisés López Calderón- 09/06/2020 17:31:52 - 0 -
Re: initialization of qweb template caches on website
byCamptocamp France SAS, Alexandre Fayolle- 09/06/2020 09:15:31 - 2 -
Re: initialization of qweb template caches on website
byVauxoo, Moisés López Calderón- 09/06/2020 06:29:22 - 0 -
Re: initialization of qweb template caches on website
by Gustavo Valverde <gustavo@iterativo.do> - 08/06/2020 15:20:29 - 0 -
-
Re: initialization of qweb template caches on website
byAgent ERP GmbH, Georg Notter- 08/06/2020 14:38:36 - 0
-
