Hi David,
How to speed up the import of products (manual or CRON)?
I have a large dedicated server, that's to do tests with 300,000 products.
:O)
Best regards,
Raoul (France)
Hi David,
I have a VPS (2 cores, 8go RAM, SSD). At this moment, for 105,000 products, I am more than 24 hours of import.
As I rectified earlier, I test version 3.
Cheers,
Raoul
Thanks Raoul,
For comparative purposes, how long does the Price Tapestry (backend / cron.php) import take for the same 105K products?
Thanks,
David.
--
PriceTapestry.com
Directly with Price Tapestry (directory / TP), about 3 minutes.
Via the VPS CRON, the same time as the manual import (about 24 hours).
Cheers,
Raoul
> Via the VPS CRON, the same time as the manual import (about 24 hours).
Just to confirm, VPS CRON is running Price Tapestry cron.php and WordPress wp-cron.php - and it is that which takes 24 hours?
But if you go to /pt/admin/ and click Import All, then it is just 3 minutes for 105K products?
Thanks,
David.
--
PriceTapestry.com
At this moment (9h39).
The cron wp-cron.php:
Out of 52 693 imported products (by manual import) in 18h00.
Table wp_postmeta, the cron deleted 20,000 fields.
Table wp_posts, nothing changed, always 52,693.
20 minutes of CRON
Before, I imported (manual) 52 693 products out of 105 000 (table pt_products) for approx. 17:00.
Since 9:38 am (about 2:00), in cron with wp-cron.php:
The CRON cleared the wp_postmeta table (currently 15).
Wp_posts still at 52,693.
Just wp_options increases very slowly (836).
No new products have been put into DB.
Raoul
Hi Raoul,
Could you perhaps try an experiment once cron has finished (or terminated), you can increase the block size of the trial import from the Imported Product Manager page (wp-admin > Settings > PriceTapestry.org > Imported Product Manager) at line 310 in the plugin file pto.pto.php:
$block = 10;If you could try changing that to 100, run the Delete function and then once complete run the Trial Import and if you could time that and let me know how long the 100 products take to import...
Thanks,
David.
--
PriceTapestry.com
Thanks David.
The erasing is also very slow. Currently there are still 51858 products in the process of being erased.
It is as slow as import.
(Of course, the installation of WP is blank, up-to-date).
The only plugins installed are:
Akismet Anti-Spam
Facebook Open Graph, Google+ and Twitter Card Tags
Google XML Sitemaps (disabled)
Multi-Column Taxonomy List
PriceTapestry.org
SF Author Url Control
WP Ajaxify Comments (disabled)
WP Rocket
WPS Hide Login
Raoul
Passing $block=10; To $block=100;, in "trial". 2:30 min.
I have done an import test with pt / admin (in the same DB), 2mn47 for 105 006 products ...
Raoul
Thanks Raoul,
I'll do some experiments on my test server for comparison and follow up with you...
Cheers,
David.
--
PriceTapestry.com
Hello David
I just looked in the PT panel (pt/admin), I have 1330 clicks on the only merchant in reference.
It is not logical, the site/domain is new and not referenced.
Cheers,
Raoul
Hello David
On the other server (php7), the CRON import task does not work.
The first:
Cd / home / xxxx / www / pt / scripts /; / usr / bin / php cron.php;
Works well and makes the products import.
The second:
Cd / home / lemegaphone / www /; / usr / bin / php wp-cron.php
Does nothing, stops immediately without error message, but no products.
Cheers,
Raoul
Hello Raoul,
Regarding clicks being recorded, there is nothing in the basic click tracking to ignore robot clicks (although it is suggested that /pt/jump.php should be excluded via your top level robots.txt) however if none should have discovered your site check whether any other plugin functionality e.g. caching is automating clicks on the links on any of the imported product pages...
Regarding wp-cron.php not doing anything on the new server make sure that the following line is added to wp-config.php:
define('DISABLE_WP_CRON', true);Will report back regarding performance as soon as I've had a chance to make further tests...
Cheers,
David.
--
PriceTapestry.com
Hello David,
Thank you for your reply.
For my part, after several tests, manual import is still extremely slow on both servers.
> 72h to import 234000 products out of a total of 487000 (I stopped before).
The update by CRON server on both does not work (wp-cron.php).
Only (cron.php) works.
The "virtual" mode works very well.
Cheers,
Raoul
Hello Raoul,
Thanks - my test server imports at just over 90s / 1000 posts, so that would equate to around 6h for 234000 - that's with no other plugins installed, server is only 1 CPU / 512MB RAM (but not under any traffic load) - what is the spec of your server / cloud instance?
(I did consider the impact of key generation so tried with keys disabled on most of the WordPress tables but that made no difference - and would not be ideal if the site was receiving traffic at the same time)
Regarding wp-cron.php exiting immediately, are you able to invoke it from the command prompt incase there is an error being reported e.g.
cd public_html
php wp-cron.php
Cheers,
David.
--
PriceTapestry.com
Hi David.
Here are the latest tests.
VPS1, 2 Hearts 3.50GHz, 12GB RAM, 2x80GB SSD.
VPS2, 1 core 3.50GHz, 2GB RAM, 1x80GB SSD.
Wordpress blank, without and with plugins (like).
In the PT (and DB) panel, 103,758 products.
Launch the import.
2000 products = 40sec.
7000 products = 6mn
10,000 = 11 min
15,000 = 21 min
17,000 = 27 min
18,000 = 32 min
26,000 = 57 min
33 000 = 1h40mn
47 000 = 3h42mn
Next, for 100 products added = 1mn12sec
76 500 = 24h00
77 300 = + 30h. Script planted this morning. The meter no longer rotates.
Can you try on your side any longer?
For the CRON, the path is good, the cron "cron.php" works well.
Not that of WP.
Cheers,
Raoul
Hello Raoul,
I'm slightly confused can you confirm, as you say wp-cron.php is not working, how are you initiating the WordPress import during testing?
Thanks,
David.
--
PriceTapestry.com
Hi David,
Manual import made by the plugin panel.
On the VPS2 (PhP7), the CRON import "wp-cron.php" does not work.
No error message in the log. Just nothing ...
On the VPS1 (PhP5.5), it works, but with the same timing.
The plugin at the beginning is fast, then slowed down and becomes very slow.
Cheers,
Raoul
Hello Raoul,
Apologies for the inconvenience, would it be possible for you to create a test WordPress installation in a sub-directory and then run the import into that installation in order to measure the import speed without any other plugins running (I know that plugins can be disabled but this wouldn't impact on your site at all and would be a genuine test of the import speed against a WordPress installation unmodified in any way...
Thanks,
David.
--
PriceTapestry.com
894/5000
Hi David.
I did the tests without and with plugin, precisely on a new installation and that, on both servers at the same time.
In fact, in the beginning, it works very well, but the more the WP DB is filled (more than 30,000), the slower it gets.
---
Total Products: 103758
Imported Products: 77300
---
If I subsequently launch a full import manual through the plugin, I expect 4/5mn per 100 products.
Currently,
Total Products: 103758
Imported Products: 77300 (800) since 34mn ...
I start over a blank WP installation on this server and I come back to you.
-
Operating system CentOS Linux 6.9 PHP 5.4
Processor Intel (R) Xeon (R) CPU information E5-1650 v3 @ 3.50GHz, 2 cores
Process in progress Average load CPU 0.63 (1 min) 0.78 (5 mins) 0.76 (15 mins)
Actual memory 2.45 GB used, 12 GB total
Virtual Memory 280.14 MB used, 2 GB total
Cheers,
Raoul
Hello Raoul,
It's really down to the WordPress post creation time - how long are you finding it to take?
Are you running on a vps or cloud instance with root access and would be able to find tune MySQL configuration?
Cheers,
David.
--
PriceTapestry.com