Discovering VERY slow and sync Speed VERY slow

Ask all your questions regarding OC 8.0 and 8.1 Please read the Support Forum Rules
Forum rules
The forums were migrated over to https://central.owncloud.org which is based on the forum software Discourse. The forums here is put into read-only mode starting from today.

More background information about this move and the reasoning behind it is available in this blogpost:

https://daniel.molkentin.net/2016/07/20 ... d-central/
RealRancor
ownCloud master
Posts: 17381
Joined: Sat May 26, 2012 3:00 pm
ownCloud version: 9.0.2
Webserver: nginx
Database: MySQL
OS: Linux
PHP version: 7.0.x

Re: Discovering VERY slow and sync Speed VERY slow

Postby RealRancor » Thu Mar 19, 2015 12:08 pm

Yes, there are some reports that disc drivers of virtual machines are also causing this.

If thats not the case just follow the links posted above.
*gone*

tflidd
Forum Moderator
Posts: 7159
Joined: Sat Dec 07, 2013 7:27 pm
ownCloud version: 8.2.3
Webserver: Apache
Database: MySQL
OS: Linux

Re: Discovering VERY slow and sync Speed VERY slow

Postby tflidd » Thu Mar 19, 2015 12:36 pm

You can use the wget-command to download a big test file (e.g. http://www.thinkbroadband.com/download/).

Ian Lewis
Newbie
Posts: 6
Joined: Fri Apr 17, 2015 1:59 pm
ownCloud version: 8.0.2
Webserver: Apache
Database: MySQL
OS: Linux
PHP version: PHP Version 5.5.9-1u

Re: Discovering VERY slow and sync Speed VERY slow

Postby Ian Lewis » Fri Apr 17, 2015 2:28 pm

created an account just for this...

Fresh install of latest production server (see sig) and desktop client (Windows 7, client V 1.8.0 (build 4893)).

Simple ethernet connection of desktop and server on same subnet, other file transfers (non-OC) can saturate the network easily enough. No virtual machines.

Syncing a desktop folder 510MB, 12500 files, to a new alias on the OC server, total time 1 hour 30, approx transfer rate 94kB/s.

Network Utilization: 0.1%
Desktop CPU: a couple of percent, owncloud.exe using <1%, no page faults.
Server CPU/mem:

Code: Select all

top - 13:10:18 up 9 days, 55 min,  3 users,  load average: 2.35, 2.42, 2.35
Tasks: 221 total,   2 running, 219 sleeping,   0 stopped,   0 zombie
%Cpu(s): 13.0 us,  4.7 sy,  0.0 ni, 39.2 id, 43.0 wa,  0.0 hi,  0.2 si,  0.0 st
KiB Mem:   3979812 total,  3839564 used,   140248 free,   286496 buffers
KiB Swap:  4124668 total,     1732 used,  4122936 free.  2101196 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 6183 mysql     20   0  952580  84940  11256 S   8.6  2.1   6:10.99 mysqld
12658 www-data  20   0  464476  44508  29504 S   8.6  1.1   0:49.96 apache2
12674 www-data  20   0  464080  44800  30208 S   8.0  1.1   0:37.51 apache2
21086 www-data  20   0  464208  41344  26700 S   6.6  1.0   0:03.47 apache2
  177 root      20   0       0      0      0 D   1.0  0.0   0:33.71 jbd2/dm-0-8


So the server's not particularly busy, but it is spending most of its time in MySQL...

At a guess there's still an exceptional database workload for every file upload and that's crippling the server when you sync a new typical 'end-user' folder where the many files are small (I've seen reports of 500 SQL queries per file upload, but I haven't turned MySQL logging on to confirm).

RealRancor
ownCloud master
Posts: 17381
Joined: Sat May 26, 2012 3:00 pm
ownCloud version: 9.0.2
Webserver: nginx
Database: MySQL
OS: Linux
PHP version: 7.0.x

Re: Discovering VERY slow and sync Speed VERY slow

Postby RealRancor » Fri Apr 17, 2015 2:32 pm

Hi,

if this is not the expected behaviour its probably the best to post such infos to the developers at the bugtracker as already mentioned in this thread.
*gone*

Ian Lewis
Newbie
Posts: 6
Joined: Fri Apr 17, 2015 1:59 pm
ownCloud version: 8.0.2
Webserver: Apache
Database: MySQL
OS: Linux
PHP version: PHP Version 5.5.9-1u

Re: Discovering VERY slow and sync Speed VERY slow

Postby Ian Lewis » Fri Apr 17, 2015 2:49 pm

The problem is these threads get closed (you'll probably close this one) so there's no clear place where this issue is being discussed. E.g. a thread you suggested (I think) : https://github.com/owncloud/client/issues/2559
is closed with "There is work in progress on the server to speed up this use case. I am closing this hence."

or this one:
https://github.com/owncloud/core/issues/7072

"karlitschek closed this 24 days ago"

I'm guessing there is a server design issue of inefficient processing of file uploads, perhaps with very naive handling of the SQL code associated with each upload, and this severely impacts first-time sync of typical end-user folders with many small files. Maybe, I'm guessing, but I've spent a long time searching.

Is there a link to one thread specific to this issue?

Thanks for your help.

tflidd
Forum Moderator
Posts: 7159
Joined: Sat Dec 07, 2013 7:27 pm
ownCloud version: 8.2.3
Webserver: Apache
Database: MySQL
OS: Linux

Re: Discovering VERY slow and sync Speed VERY slow

Postby tflidd » Fri Apr 17, 2015 3:23 pm

These are the right topics you found. They are not very recent and to improve the performance of OC, they have applied some patches. So in your case, I would open a new thread (you can reference the old ones) to open a new round of improvements ;-)

It's also good, to add some data for comparison, like you did 12500 files and the upload time (compared to ftp->rules out errors in your network, slow disk, ...). Also interesting if you can take a different set of slightly larger files (500 MB in 100 files, 1 file of 500 MB).

If you are up for some testing, you can try the OC 8.1 alpha (only for testing purposes), I think they tried to optimize the SQL queries. In case it is much better, you will only have to wait and better performance is coming, if not, you can help to give some feedback.

Ian Lewis
Newbie
Posts: 6
Joined: Fri Apr 17, 2015 1:59 pm
ownCloud version: 8.0.2
Webserver: Apache
Database: MySQL
OS: Linux
PHP version: PHP Version 5.5.9-1u

Re: Discovering VERY slow and sync Speed VERY slow

Postby Ian Lewis » Fri Apr 17, 2015 3:57 pm

quick answer to the above (I'll do some other testing and create a thread as suggested):

sftp transfer (used FileZilla, drag-and-drop) of the same 12500 files between the same Windows PC and the same Ubuntu server, on the same network, took *about* (I was slightly surprised by how quick it was):

* 8 parallel xfers: 3 mins 30 seconds (+/- 15 seconds)

* no parallel xfer, i.e. only 1 file at time: 8 mins 30 seconds

(vs the initial OC sync which was at least an hour and a half).

RealRancor
ownCloud master
Posts: 17381
Joined: Sat May 26, 2012 3:00 pm
ownCloud version: 9.0.2
Webserver: nginx
Database: MySQL
OS: Linux
PHP version: 7.0.x

Re: Discovering VERY slow and sync Speed VERY slow

Postby RealRancor » Fri Apr 17, 2015 4:15 pm

Hi,

Ian Lewis wrote:The problem is these threads get closed (you'll probably close this one)


please note:

viewtopic.php?f=14&t=23696

I'm also not closing anything in here or at the issue tracker as i'm just a user as you are.
*gone*


  • Similar Topics
    Replies
    Views
    Last post

Return to “ownCloud Server 8.0 and 8.1”

Who is online

Users browsing this forum: Google [Bot] and 2 guests