To complete our current article on how to setup a CRON task to automate CSV imports, you can also configure multiple CRON tasks to import large CSV files multiple times.
To do so:
- In the CRON php file, configure the option 'olderthan' to 120 minutes for example. This way files that have been modified less than 120 minutes ago will not be processed.
- The option "deletetodofile' should be set to '0' allowing the CRON tasks to continue the same import instead of starting from the beginning of your original file.
- Within the Import panel in Store Commander (top panel), modify the number of lines to import by a quarter of the total lines existing in your CSV file. For instance, if the file contains 1956 lines, configure the option 'number of lines' to 500 for 4 CRON tasks (500*4=2000 lines to be processed in total).
- Then run the CRON tasks multiple times in a row (in our example: 4 times every 20 minutes. You need to keep a 20 minutes interval to avoid importing the same data multiple times).
This way, your large CSV files will be processed in 4 parts rather than one. You can of course modify the above settings to suit your needs, depending on the size of the CSV files.