1. Mailing Lists
  2. Contributors
  3. Re: Product variant import (performance issue)

Contributors mailing list archives

contributors@odoo-community.org

Browse archives

Re: Product variant import (performance issue)

Re: Product variant import (performance issue)

Avatar

Re: Product variant import (performance issue)

by
Tecnativa. S. L., Pedro M. Baeza
- 26/01/2016 10:58:20
Well, that's true, but you can mount that customizations on top of that module. As it's said here in Spain, "you can make omelette without breaking the eggs" ;)

I don't find for now a case where I can't dig for using this principle and looking for template, not product, but it requires to adapt process, of course.

Regards.

2016-01-26 10:53 GMT+01:00 Graeme Gellatly <gdgellatly@gmail.com>:

I tried it a long time ago. From memory the first issue I met was name search broke customizations because when you have hundreds of thousands products, and the way the ORM filters in realtime if you had another override of name search you never got the expected results or in the right order.

But it has been a long time since I looked. All I meant really was those modules have a specific use case they were built for and so Odoo mrp extends from there.. Modules that also affect product in a deep way but no nothing of that module can become incompatible.

Many customizations of product expect the product to be there, and for all the overrides there were cases I couldn't make work. Especially true when dealing with a large number of products.


On Tue, 26 Jan 2016 9:08 PM  <Pedro@pad.odoo-community.org> wrote:
Graeme, you only need 3/4 modules for all the works with variants, not the full set of modules: one for inhibiting the creation, one for sale, one for purchase and another for MRP. No more. These modules work in every possible use case we have found for now. If not, just tell me one that doesn't fit.

Regards.

2016-01-26 5:38 GMT+01:00 Graeme Gellatly <gdgellatly@gmail.com>:
I did loads of work on this years ago for the old product variant.

Our largest product variant was 22,000 products and would take a very long time.  500,000 variants in all.  It would go at roughly 3000 per hour.  Creation and updates.

On examination there were many things to speed the process and by the end we got to 500/minute for creation and 5000 per minute for updates which was acceptable.  Of course none of this applies to v8 variants as implemented by Odoo which is quite broken, but the general principles remain.

What we did in essence was
use ORM for the Create calls.  This ensured required fields, log_access fields and defaults were correctly filled.  It also returned the ids.
We then just ran an UPSERT SQL statement essentially for all the records.

But in your case, I'd advise heavily against the SQL method or even csv and go with the usual interface if it is just one template.  I've done a bit of investigation and the Odoo implementation is very very limited.  Even csv is going to suck, but at least you don't have to deal with calculating stored functional fields, m2m relations etc.  Even then you are going to have 16,000 create calls, and probably 16,000 write calls as well.  And if you don't get the variant ordering exactly right, any change to the template will delete them all anyway.  Some of my comments are in product-attribute repo pull requests which show what is not working and no way to override.  (Sorting, sequencing, ordering, optional variants to name a few)

I know Pedro's module as part of OdooMRP, but it really doesn't work for many use cases outside of it, especially for people who do need all the variants.  OdooMRP is kind of this whole system of modules which work around other limitations, but really works best as a whole system rather than individual modules, mainly due to the Odoo function itself create_variant_ids.

I'd just start a second server, not in worker mode, on a different port so you don't hit timeouts, memory limits etc and let it go overnight.  8000 is not so many, will only take a few hours, and a lot less time than writing the csv files.  Normally you can monitor the process just using top anyway and see when its done after browser timed out. 


On Tue, Jan 26, 2016 at 11:53 AM, Maxime Chambreuil <maxime.chambreuil@savoirfairelinux.com> wrote:
Hello,

I have to create/import one product with 8692 variants. I left the creation runs for more than hour, no success.

Before I create a csv file with all the different combinations, I would like to know if you have a better suggestion.

Thanks!
--
Maxime Chambreuil
Ring ID: 239e01f6262afbf3c7d276d2c661173ddddc0340

_______________________________________________
Mailing-List: http://odoo-community.org/groups/contributors-15
Post to: mailto:contributors@odoo-community.org
Unsubscribe: http://odoo-community.org/groups?unsubscribe

_______________________________________________
Mailing-List: http://odoo-community.org/groups/contributors-15
Post to: mailto:contributors@odoo-community.org
Unsubscribe: http://odoo-community.org/groups?unsubscribe

_______________________________________________
Mailing-List: http://odoo-community.org/groups/contributors-15
Post to: mailto:contributors@odoo-community.org
Unsubscribe: http://odoo-community.org/groups?unsubscribe

_______________________________________________
Mailing-List: http://odoo-community.org/groups/contributors-15
Post to: mailto:contributors@odoo-community.org
Unsubscribe: http://odoo-community.org/groups?unsubscribe


Reference