• chevron_right

      Upgrading to Odoo 12

      jnanar · Saturday, 23 February, 2019 - 11:00 · 5 minutes

    Odoo 12.0 is out since october. I am currently investigating the differences with previous versions to update the instance of the association Les Compagnons du CEP . A lot of changes have been made in a few years but the workflow stays about the same. This article describes my workflow, the backup policy, how a module was used and fixed to restore a missing feature. Finally, the changes in my custom product import function are presented.

    Production and debug setup

    Agayon.be instance of Odoo runs inside a Docker container on a small VPS. Postgresql is installed as a core package. This setup is great in production but it is difficult to debug some python code with this configuration. The first step is to run Odoo with an IDE (I personally use the excellent Pycharm ).

    Running and debugging Odoo

    The first step is the creation of the virtualenv. Once the Postgresql instance is ready , you can build the environment.

    git clone https://www.github.com/odoo/odoo --depth 1 --branch 12.0$ virtualenv myenv$ source myenv/bin/activate(myenv)$ pip install -r odoo/requirements.txt(myenv)$ mkdir custom-addons(myenv)$ chown odoo: odoo11-custom-addons

    Edit odoo.conf and then run odoo:

    odoo/odoo-bin -c odoo.conf

    Merge purchase order

    UPDATE: 23/02/2019: The following paragraph is not needed anymore. Odoo 12 restored the automatic merge of purchase orders.

    Unfortunately, a major feature has disappeared in version 12.0: automatic merge of purchase order. I don't know why since it seems unrealistic to send each quotation separately to your vendor. Fortunately a free module can be used to perform the merge but it has a critical bug. Some quotation lines are merged even if they concerns different products .

    Fix

    After forking it, I decided to start by refactoring it. The module is quite small but a lot of code is duplicated. As I try to avoid spaghetti code , it needed to be refactored .I think the new code may be improved but no line is duplicated. Finally, the bug has been fixed .

    Backup management

    Since a few version, the filestore is mandatory in Odoo. If it is incoherent with the database, some really annoying errors are raised and the solution is quite tedious. My backup procedure has been updated to avoid losing any data. It is based on the article from zeroheure . The backups are saved with the auto_backup module. Restoring the data is not possible with the web interface because the process reaches the memory limit but it can be performed with the following shell script.

    #!/bin/bashBACKUPLOCATION="/path/to/backup.zip"DBNAME="db_name"FILESTORE_DIR="/path/to/filestore"if[ -z "$FILESTORE_DIR"]||[ -z "$DBNAME"]||[ -z "$BACKUPLOCATION"]thenecho"verify your variables"elsecd$BACKUPLOCATION    rm filestore     unzip -q $DBNAME.zip    cp -r filestore $DBNAME    sudo rm -rf $FILESTORE_DIR/$DBNAME    sudo mv $DBNAME$FILESTORE_DIR    sudo chown -R odoo:odoo $FILESTORE_DIR/$DBNAME    dropdb -U odoo $DBNAME    createdb -U odoo $DBNAME    psql $DBNAME --quiet < dump.sqlfi

    Wine import with Django website

    Version 12.0 needs some minor changes in the code displayed in the previous article .These modifications includes:

    • removing state property in the product template.
    • Adding the invoice_policy and purchase_method in the product template. 1
    • Add a reference to the standard price in the product_supplierinfo dictionary. This value is used in the orders when purchasing wines to suppliers.
    defresearch(default_code,supplier_code,wine_name):# 1) search if default code is used?)# 2) search if suppliers is in the database# 3) search if the name is already used# Retrieve the dataframes. This example comes from a jupyter nootebook.# df_product and df_sellers are already defined.# In a real case, we should use class variables.n_code,df_code=search_df(df=df_product,col_name='default_code',search_item=default_code,search_int=False)n_supplier,df_suppliers=search_df(df=df_suppliers,col_name='function',search_item=supplier_code,search_int=False)n_name,df_name=search_df(df=df_product,col_name='name',search_item=wine_name,search_int=False)try:# ids_product = list(df_code['id'])ids_product=df_code['id'].tolist()exceptAttributeError:ids_product=[]ifn_code==0andn_name==0andn_supplier!=0:return'success',ids_product,n_supplier,n_nameifn_code!=0andn_name!=0:# Another product uses the same name with another codereturn'e_code_used_same_name',ids_product,n_supplier,n_nameifn_code!=0:# the code is already usedreturn'e_code_used',ids_product,n_supplier,n_nameifn_supplier==0:# Cannot find the supplierreturn'e_missing_seller',ids_product,n_supplier,n_nameifn_name!=0:# A product with the same name and another code exists.return'e_code_used_different_name',ids_product,n_supplier,n_namedefimport2odoo():route_warehouse0_mto=1route_warehouse0_manufacture=5[...]# iterate over all rows, read the cells and assign the wine parameters to variables# each row correspond to one wineforrowinrows:seller_name=row[0]default_code=row[1]name=row[2]do_import=row[2]comment=row[3]name=row[4]default_code=row[5]standard_price=row[6]list_price=row[7]seller_code=row[8]res_search,ids_product,n_supplier,n_name=research(default_code,seller_code,name)ifres_search=='success'anddo_import=="1":product_template={'name':name,'active':True,'standard_price':standard_price,'list_price':list_price,'description':comment,'default_code':default_code,'purchase_ok':1,'sale_ok':1,'uom_id':1,'uom_po_id':1,'type':'product','cost_method':'standard','invoice_policy':'order',# ordered quantities'purchase_method':'receive',# control received quantities (or ordered ones, test to delivery)'route_ids':[(6,0,[route_warehouse0_mto,route_warehouse0_manufacture])]}# For each wine, a template must be createdtemplate_id=sock.execute(dbname,uid,pwd,'product.template','create',product_template)# Create the supplier information for the wineproduct_supplierinfo={'name':name,'product_code':row[1],# code for supplier'product_name':row[15],# name for supplier'min_qty':1,'delay':300,'product_tmpl_id':template_id,'price':standard_price,}# Create the supplier information for the wineproduct_supplierinfo_id=sock.execute(dbname,uid,pwd,'product.supplierinfo','create',product_supplierinfo)logging.info("Wine {} : {} has been added".format(default_code,name))[...]# Here we take into account the exceptions and several cases: the wine is already present, the seller is missing etc.

    Conclusions

    After using Odoo in production for 4 years, I still love it. My users are happy to use it daily and the improvements over the year are impressive. Like any other alive project, some changes may break the workflow but Odoo and it's community make it relatively simple to adapt with plugins. Odoo 12 is strong and I look forward to use it for the next 3 years :-).

    Links

    1. See this post and this issue
    • chevron_right

      Fail2Ban analysis

      jnanar · Wednesday, 5 December, 2018 - 20:00 · 3 minutes

    Running a server on the internet is quite an adventure but it's not always easy to keep track of log files, security and potential threads. This article describes some actions carried out on this server to mitigate script kiddies .

    Among the good practices, I have

    • a subscription to the Debian security mailing list,
    • automatic security updates (enabled by default with the Scaleway Debian images)
    • A monitoring tool ( Munin ) which provides useful graphs to watch the activity of the server.
    • Fail2Ban
    • Backups

    The following paragraphs describe how I analyze the country IP banned by Fail2Ban.

    As explained on their website, Fail2Ban scans log files and bans IPs that show the malicious signs: too many password failures, seeking for exploits, etc. Generally Fail2Ban is then used to update firewall rules to reject the IP addresses for a specified amount of time, although any arbitrary other action (e.g. sending an email) could also be configured. Out of the box Fail2Ban comes with filters for various services (apache, mail, ssh, etc).

    Recently, I had the need to check if Belgian IP were blacklisted. Most of my users are Belgian and one of my Fail2Ban rules was too strict. I decided to log the IP in a file to perform a geolocalisation analysis to detect and prevent false positives.

    The fail2ban-blacklist script was used to log blacklisted IP into a CSV file. The analysis is performed on another computer.

    Scripts

    The following script is called generate_report.py . It read a CSV file that has several information about bans: the date, time and IP. The country IP are discovered with the whois information thanks to a script. Finally, a barplot is generated to visualize the amount of hits per country. The whole process is launched with the report.sh script.

    generate_report.py

    importmatplotlib.pyplotaspltimportpandasaspdimportnumpyasnpimportsysimportmatplotlibimportosimportsubprocessimportreimportlogginglogging.basicConfig(level=logging.DEBUG,format="[%(asctime)s] %(levelname)s%(message)s",datefmt="%H:%M:%S",stream=sys.stdout)defget_country(ip):whois=subprocess.Popen(['whois',ip],stdout=subprocess.PIPE)whois.wait()# Prevent problems if output is not utf8str_whois=whois.communicate()[0].decode("utf-8","replace")find_country=re.search(r'country:(.*)',str_whois)iffind_country:returnfind_country.group(1).strip()else:return"NONE"defplot_graph(df=None):plt.figure(figsize=(16,10),)df.country.value_counts().plot.bar(figsize=(16,10))plt.xlabel('Country',fontsize=16)plt.ylabel('Counts',fontsize=16)plt.title('Counts of blacklisted countries',fontsize=20)plt.savefig("fail2ban_report.png",dpi=150,facecolor='w',edgecolor='w',orientation='portrait',papertype=None,format=None,transparent=False,bbox_inches=None,pad_inches=0.1,frameon=None,metadata=None)defreport():filename='blacklist.pkl'df_backup=Nonedf_csv=Nonedf=Nonedf_concat=Noneifos.path.exists(filename):df_backup=pd.read_pickle(filename)filename=r'blacklist.csv'df_csv=pd.read_csv(filename,encoding="UTF-8",sep=',',engine='python',)iflen(df_backup)<len(df_csv):df_tmp=df_backup[['DATE','TIME','IP_ADDRESS']]df_concat=pd.concat([df_csv,df_tmp]).drop_duplicates(keep=False)df_concat['country']=""# df = df_backup.append(df_concat,sort=False)logging.info("Append {} lines".format(len(df_concat)))else:logging.info("No difference between backup and CSV")df=df_backup.copy()friend_list=[]# df_friends = pd.DataFrame(columns=list(df_backup.columns.values))df_friends=pd.read_pickle("blacklist_friends.pkl")ifdf_concatisnotNone:foridx,rowindf_concat.iterrows():# Do not process the dataframe multiples times.ifnotrow['country']:logging.info("Process IP : {}".format(str(row['IP_ADDRESS'])))country=get_country(str(row['IP_ADDRESS']))df_concat.loc[idx,'country']=countryelse:country=row['country']# country == 'Be' do not workcountry=str(country)ifcountry.lower()in['be','other_friendly_country']:logging.info(row)row['country']=countryfriend_list.append(row)df=df_backup.append(df_concat,sort=False)try:logging.info("Append friends")df_friends=df_friends.append(friend_list)df_friends.to_pickle("./blacklist_friends.pkl")exceptIndexError:logging.info("No friend to append")df.to_pickle("./blacklist.pkl")plot_graph(df)if__name__=="__main__":report()

    report.sh

    This script copy the Fail2Ban CSV file from the server (whois requests are forbidden on my VPS), generate the data and display the bar plot with the help of typop , a built-in function of terminology , a great terminal emulator for Linux/BSD/UNIX systems.

    #!/bin/shscp user@agayon.be:/etc/fail2ban/report.py .python generate_report.pytypop fail2ban_report.pngexit0

    Graphs

    This graph was generated on 5 of december 2018. Some countries are more represented but the threat is global.

    Fail2Ban report graph

    The following graphs is generated by Munin. It display the number of ban per jail.

    Fail2Ban graph week

    Links

    • chevron_right

      3615 MyLife

      jnanar · Thursday, 19 July, 2018 - 08:21 · 1 minute

    This year, the holidays and city trip was the occasion to meet Damien Accorsi, founder of Algoo SAS and his team.

    This summer, I had the opportunity to meet Damien Accorsi in Moirans near Grenoble. He is the founder of Algoo SAS , a company that provides software development services and Tracim. Tracim is a collaborative software designed to allow people to share and work on various data and document types .

    Everything started from a post on LinuxFR . I stayed near Grenoble for a week in the beginning of July and therefore, I asked the community about nice activities to do in the region. Damien answered quite quickly and made some useful suggestions. We never talked before. He also suggested meeting in his startup in Moirans. I happily accepted and we have met in his quarter. We talked about his activities. If you speak French, I suggest his instructive posts in LinuxFR . We talked about our projects, we laugh and had a really good time. His employees are really nice and fun. When I left, I promised myself to make more IRL meetings in the future.

    Yet It was not the first time I visited a software developer. Two years ago, I have met Goffi from the Salut-à-Toi (SàT) project in Prague (Czech Republic). We had nice conversations and exchange about XMPP, the link between communication tools and politics, the struggle of developers to take part in open source project during free time, building a community with limited resources, etc. From this exchange started a nice collaboration on his tool. To this day, I write the PKGBUILD (packages) of SàT for Archlinux .

    I hope to meet other people during holidays and events like FOSDEM. It is nice to put a face on a nickname. Maybe next time it will be an inventor or an artist.

    In the meantime, if you go to Grenoble, according to Damien and myself, you should try:

    • chevron_right

      Mixing Pandas with Odoo

      jnanar · Tuesday, 13 March, 2018 - 18:00 · 10 minutes

    This article describes the use of XML-RPC API provided by Odoo , a well-known ERP system. Upgrading to version 11.0 is the occasion to update my python scripts to reduce considerably the number of requests. The improvements were done with the help of pandas , the famous data structures and data analysis library.

    Photo credit: Panda_3956, Ken_from_MD
    Photo credit: Panda_3956, Ken_from_MD

    Introduction

    In my spare time, I help a small association, Les Compagnons du CEP , a joint buying organization who buys French wines for its members directly from producers. Since 2014, I set up an Odoo instance to manage the quotations, purchase order and the members. Odoo fulfills all their needs and we are happy to use it daily.

    In addition to the user-friendly Web interface, the following management tasks are performed within a custom Django website.

    • Calculate the price of wines based on the seller price, taxes, transport cost, VAT, Fost+ .
    • Import wines to database. The list of sellable wines is updated two times a year in order to adapt to the seasonal dishes 1 . Of course, the vintages are changed each year depending of the wines. The association sells approximately 600 different wines (up to 10000 bottles a year). As a result, I wrote a massive import script. This program reads large Excel files containing the several parameters (name, vintage, seller name, description, etc) and it uses the XML-RPC API offered by Odoo to create the items in the PostgreSQL database. This task is the main subject if this article.
    • Generate the price list based on the newly added wines. A price list Excel file is uploaded by the user. The file is transferred on another machine with the help of Errol . A LaTeX document is generated with python from the Excel file and it is compiled to PDF with TeX Live . Afterward, the PDF is automatically copied on the Django website with Errol.

    The Django website is successfully used since 4 years. Unfortunately, I observed slowdown in the process since the implementation of product update from the Excel file.

    In order to prepare the upgrade to Odoo 11.0 , I decided to update the XML-RPC calls in order to reduce their number and therefore accelerate the import process.

    Some code to get your teeth into

    Photo credit: Panda, Sue Cantan https://www.flickr.com/photos/suecan/4349221370/
    Photo credit: Panda, Sue Cantan

    To chose the more suitable strategy, I decided to compare the current code base to a new scenario.

    Description of the current code

    Each row of the Excel file correspond to a wine. The name, seller code 2 and default code are searched in the Odoo database in order to avoid duplicates. If no collision is found with the older products, a product template is created and the supplier information are updated. Since 2014, version 8.0 is used in production with product variant support. Unfortunately, the variant requires to perform an additional search to update the price of the product. In version 11.0, we will get rid of the product variant.

    The python code can be summarized as follows.

    importxmlrpc.clientasxmlrpclibimportpandasaspdusername='user'pwd='password'dbname='database_name'sock_common=xmlrpclib.ServerProxy('http://localhost:8069/xmlrpc/common')uid=sock_common.login(database,username,pwd)sock=xmlrpclib.ServerProxy('http://localhost:8069/xmlrpc/object')defresearch(default_code,supplier_code,wine_name):supplier_code=int(float(supplier_code))args_p=[('default_code','=',default_code)]args_s=[('function','=',supplier_code)]args_n=[('name','=',wine_name)]ids_product=Nonen_supplier=Nonen_name=Nonetry:ids_product=sock.execute(dbname,uid,pwd,'product.product','search',args_p)# Rather than retrieve a possibly gigantic list of records and count them, search_count()# can be used to retrieve only the number of records matching the query.# It takes the same domain filter as search() and no other parameter.# https://www.odoo.com/documentation/8.0/api_integration.htmln_supplier=sock.execute(dbname,uid,pwd,'res.partner','search_count',args_s)n_name=sock.execute(dbname,uid,pwd,'product.product','search_count',args_n)n_supplier=int(n_supplier)n_name=int(n_name)exceptAttributeError:start_session()return"e_initialization",ids_product,n_supplier,n_nameiflen(ids_product)==0andn_name==0andn_supplier!=0:return'success',ids_product,n_supplier,n_nameiflen(ids_product)!=0andn_name!=0:# Le code est déjà utilisé et un produit du même nom existe.return'e_code_used_same_name',ids_product,n_supplier,n_nameiflen(ids_product)!=0:# Le code est déjà utiliséreturn'e_code_used',ids_product,n_supplier,n_name# if len(ids_product) == 0:ifn_supplier==0:# Le fournisseur n existe pasreturn'e_missing_seller',ids_product,n_supplier,n_nameifn_name!=0:# un produit du même nom mais pas le même code existereturn'e_code_used_different_name',ids_product,n_supplier,n_namedefimport2odoo():route_warehouse0_mto=1route_warehouse0_manufacture=5[...]# iterate over all rows, read the cells and assign the wine parameters to variables# each row correspond to one wineforrowinrows:seller_name=row[0]default_code=row[1]name=row[2]do_import=row[2]comment=row[3]name=row[4]default_code=row[5]standard_price=row[6]list_price=row[7]seller_code=row[8]res_search,ids_product,ids_supplier,ids_name=research(default_code,seller_code,name)ifres_search=='success'anddo_import=="1":# Success, the wine may be addedresults_dict['added_wines'].append([default_code,name,standard_price,list_price,comment])product_template={'name':name,'active':True,'state':'sellable','standard_price':standard_price,'list_price':list_price,'description':comment,'purchase_ok':1,'sale_ok':1,'uom_id':1,'uom_po_id':1,'type':'product','cost_method':'standard','route_ids':[(6,0,[route_warehouse0_mto,route_warehouse0_manufacture])]}# For each wine, a template must be createdtemplate_id=sock.execute(dbname,uid,pwd,'product.template','create',product_template)# Create the supplier information for the wineproduct_supplierinfo={'name':name,'product_code':default_code,# code chez le fournisseur'product_name':name,# name chez le producteur'min_qty':1,'delay':300,'product_tmpl_id':template_id,}# Create the supplier information for the wineproduct_supplierinfo_id=sock.execute(dbname,uid,pwd,'product.supplierinfo','create',product_supplierinfo)# The product id must be obtained to set the default codeproduct_product={'product_tmpl_id':template_id,'default_code':default_code,}args=[('name','=',name),]product_id=sock.execute(dbname,uid,pwd,'product.product','search',args)result=sock.execute(dbname,uid,pwd,'product.product','write',product_id,product_product)[...]# Here we take into account the exceptions and several cases: the wine is already present, the seller is missing etc.

    Metrics

    If N is the number of wines to add, x , the number of milliseconds to research an item and y the number of milliseconds to write an item into the database, the previous code is made of:

    • 3 researches per wine : 3 N * x
    • templates creation: N y
    • supplierinfo creation: N y
    • research products: N x
    • product update: N y

    Total: 4N x + 3N y

    This code is not optimized. It runs slowly even if it achieves what we expect from it. Now, we can try to do enhance it.

    The new version

    As previously mentioned, the product variant suppression will remove one search but there is more room for improvements.

    The API call for each row is the main bottleneck of the script. I decided to replace them by a search of all product at the beginning of the script. The list if wine is return in a list of dictionary. Fortunately, a list of dicts is the easiest object to convert to a Pandas Dataframe.

    importpandasaspddefdataframes_generator():# Use a search function with empty args to get all ids :# https://www.odoo.com/fr_FR/forum/aide-1/question/is-it-possible-to-retrieve-2-fields-of-all-entry-within-a-model-thru-xml-rpc-6886# 1st => search all ids 2nd => read the selected fields on the whole list of ids.# Only 2 rpc-xml requests :o)#1args_product=[('default_code','!=','foo')]ids_product=sock.execute(dbname,uid,pwd,'product.product','search',args_product)# 2fields_products=['name','id','default_code']recordset_products=sock.execute(dbname,uid,pwd,'product.product','read',ids_product,fields_products)args_seller=[('name','!=','foo')]fields_seller=['name','id','function']ids_sellers=sock.execute(dbname,uid,pwd,'res.partner','search',args_seller)recordset_sellers=sock.execute(dbname,uid,pwd,'res.partner','read',ids_sellers,fields_seller)df_product=pd.DataFrame(recordset_products)df_sellers=pd.DataFrame(recordset_sellers)returnTrue,ids_product,ids_sellers

    For now, the ids list is generated by searching all product with a dummy code ('foo' in the example). My trials to get rid of the args_product variable failed.

    As the dataframes are generated, it is easy to search the product and sellers among their respective dataframes:

    code_to_find='A18'df_product[df_product['default_code'].str.contains(code_to_find)]default_codeidname0A18/99921Firstwine1A18/99922Secondwine

    A small function search_string_df can be written to facilitate the future searches in dataframes. It returns the dataframe and its lenght:

    defsearch_df(df=None,col_name='',search_item=None,search_int=None):#  if we search an int.ifsearch_int:df_res=df.loc[df[col_name]==search_item]# if we search a string. We must ignore NaN values.else:df_res=df[df[col_name].str.contains(search_item,na=False)]n_res=len(df_res)returnn_res,df_resn,df=search_string_df(df=df_product,col_name='default_code',search_str='A18')

    Final cut

    When the research function is modified, the whole code can also be adapted:

    defresearch(default_code,supplier_code,wine_name):# 1) search if default code is used?)# 2) search if suppliers is in the database# 3) search if the name is already used# Retrieve the dataframes. This example comes from a jupyter nootebook.# df_product and df_sellers are already defined.# In a real case, we should use class variables.n_code,df_code=search_df(df=df_product,col_name='default_code',search_item=default_code,search_int=False)n_supplier,df_suppliers=search_df(df=df_suppliers,col_name='function',search_item=supplier_code,search_int=False)n_name,df_name=search_df(df=df_product,col_name='name',search_item=wine_name,search_int=False)try:# ids_product = list(df_code['id'])ids_product=df_code['id'].tolist()exceptAttributeError:ids_product=[]ifn_code==0andn_name==0andn_supplier!=0:return'success',ids_product,n_supplier,n_nameifn_code!=0andn_name!=0:# Another product uses the same name with another codereturn'e_code_used_same_name',ids_product,n_supplier,n_nameifn_code!=0:# the code is already usedreturn'e_code_used',ids_product,n_supplier,n_nameifn_supplier==0:# Cannot find the supplierreturn'e_missing_seller',ids_product,n_supplier,n_nameifn_name!=0:# A product with the same name and another code exists.return'e_code_used_different_name',ids_product,n_supplier,n_namedefimport2odoo():route_warehouse0_mto=1route_warehouse0_manufacture=5[...]# iterate over all rows, read the cells and assign the wine parameters to variables# each row correspond to one wineforrowinrows:seller_name=row[0]default_code=row[1]name=row[2]do_import=row[2]comment=row[3]name=row[4]default_code=row[5]standard_price=row[6]list_price=row[7]seller_code=row[8]res_search,ids_product,n_supplier,n_name=research(default_code,seller_code,name)ifres_search=='success'anddo_import=="1":product_template={'name':name,'active':True,'standard_price':standard_price,'list_price':list_price,'description':comment,'default_code':default_code,'purchase_ok':1,'sale_ok':1,'uom_id':1,'uom_po_id':1,'type':'product','cost_method':'standard','route_ids':[(6,0,[route_warehouse0_mto,route_warehouse0_manufacture])]}# For each wine, a template must be createdtemplate_id=sock.execute(dbname,uid,pwd,'product.template','create',product_template)# Create the supplier information for the wineproduct_supplierinfo={'name':name,'product_code':default_code,# code chez le fournisseur'product_name':name,# name chez le producteur'min_qty':1,'delay':300,'product_tmpl_id':template_id,}# Create the supplier information for the wineproduct_supplierinfo_id=sock.execute(dbname,uid,pwd,'product.supplierinfo','create',product_supplierinfo)logging.info("Wine {} : {} has been added".format(default_code,name))[...]# Here we take into account the exceptions and several cases: the wine is already present, the seller is missing etc.

    Metrics

    If we follow the same conventions than before:

    The new code is made of:

    • 3 researches: 3 * x
    • templates creation: N y
    • supplierinfo creation: N y

    Total: 3 x + 2N y

    This result is a huge improvement. The data search in Dataframe is really fast and therefore, the impact of the bottleneck is decreased.

    Conclusions

    After several tests, the new implementation seems solid. It is not as fast as I would have expected but I am working on it. Some helpful resources are available on the web.

    The import campaign of spring is already finished but the new algorithm will be tested with the aim to be ready for the wines coming in autumn. Nevertheless, reducing the number of request from 3N to 3 where N is the number of wines can only have beneficial effects.

    The next step will be to watch and analyse the databases requests . The option log-level=debug_rpc will probably be crucial.

    The results of these investigations will be shared here.

    Stay tuned !

    Photo credit: Panda in China, George Lu https://www.flickr.com/photos/gzlu/7708872342/
    Photo credit: Panda in China, George Lu

    Links

    1. The covered regions are: Champagne, Alsace, Loire, Bourgogne, Beaujolais - Maconnais, Rhône, Provence, Languedoc
    2. The seller code is saved in the field 'function'