oc8 seems to drag apache down on high client sync loads

Ask all your questions regarding OC 8.0 and 8.1 Please read the Support Forum Rules
Forum rules
The forums were migrated over to https://central.owncloud.org which is based on the forum software Discourse. The forums here is put into read-only mode starting from today.

More background information about this move and the reasoning behind it is available in this blogpost:

https://daniel.molkentin.net/2016/07/20 ... d-central/
User avatar
finius
Starter
Posts: 95
Joined: Thu Jul 18, 2013 10:32 am
ownCloud version: 8.0.2
Webserver: Apache
Database: MySQL
OS: Linux
PHP version: Ubuntu 14.04 BuildIn
Location: Frankfurt / Main

oc8 seems to drag apache down on high client sync loads

Postby finius » Fri Apr 17, 2015 8:25 am

Hi,
I have a huge problem with my owncloud which might is caused by a misconfiguration of apache.
Since we have a user with a huge data dir (he uploaded 75GB), the owncloud crashes now every 1-2 hours. The snyc client gets an "socket operation timeout", webusers see a white page or a server error and our monitoring gives the following:

agent name: cloud
Process: processes_apache2@cloud
Datum/Uhrzeit: 17-4-2015 08:30:01 (GMT + 3)
Current Working set size: 1533.05 MB / Max value: 1024
Current Page file usage: 58187.38 MB / Max value: 15360


Summarized you can say, that owncloud seems to stop operating on the server side after a while when huge sync loads are proceeded on the sync client over a longer period of time (some hours) - even if the server system itself seems to continue operating normal. Just a quick reboot puts the server back to normal operation - until a critcal amout of sync load had been processed again.

Let me give some facts on the situation to give an impression of what's going on:
oc8 runs virtualized on a single ubuntu 14.04 LTS server with 4 cores on a new Dell PowerEdge R730xd, 4GB on a 20GB hdd with about 50% free space left. storage of the data dir is mounted into linux from a Windows network share. CPU and RAM are operating in the lowest 10% of their given server ressources.
oc8 is performing fine when operating in regular daily business (normal sync loads - about 2-3GB a day by 50 users)
performance problems always seems to occurs when users are uploading/downloading huge amounts of personal data (>5-10GB), in a stronly nested folder hierarchy, at a time using the owncloud sync client for Windows (even in version 1.8).
The sync client gets disconnected from the server after a while with a "socket operation timeout" error. When trying to login to the owncloud server, you might get a blackscreen (hang) OR you're facing network HTTP problems concerning the apace/oc8, although the performance key indicators (RAM, SWAP, HDD, CPU) seem to be within acceptable ranges. system parameters are unsuspicious at any time (much free RAM, much free SWAP, etc) - except HTTP response times on the apache port of oc8 which exceed 1500ms by far in case of the fault which is described here.

Does anyone have an idea what this can be and how it can be solved?
"Der Anfang jeder Katastrophe ist ne beschissene Vermutung!"

tflidd
Forum Moderator
Posts: 7159
Joined: Sat Dec 07, 2013 7:27 pm
ownCloud version: 8.2.3
Webserver: Apache
Database: MySQL
OS: Linux

Re: oc8 seems to drag apache down on high client sync loads

Postby tflidd » Fri Apr 17, 2015 9:17 am

Did you change your default apache configuration? The default settings are probably not suitable and the number of threads and child processes is not optimized.
Do you use Apache MPM prefork or MPM worker (default is prefork in most systems)?
http://httpd.apache.org/docs/2.4/en/mod/prefork.html
http://httpd.apache.org/docs/2.4/en/mod/worker.html

Have a look here: https://www.digitalocean.com/community/ ... erformance

User avatar
finius
Starter
Posts: 95
Joined: Thu Jul 18, 2013 10:32 am
ownCloud version: 8.0.2
Webserver: Apache
Database: MySQL
OS: Linux
PHP version: Ubuntu 14.04 BuildIn
Location: Frankfurt / Main

Re: oc8 seems to drag apache down on high client sync loads

Postby finius » Fri Apr 17, 2015 9:38 am

Hi,
thanks for your quick response! I'm working on Ubuntu 14.04 LTS with the default LAMP configuration, that's right. Further we use the owncloud repositories to install and update our owncloud. Apache shows that prefork is currently activated. But when I try to switch to worker using this documentation (http://askubuntu.com/questions/524770/a ... worker-mpm) I get this error which I don't know how to solve:

Code: Select all

[Fri Apr 17 09:27:27.651670 2015] [:crit] [pid 4285:tid 139724709111680] Apache is running a threaded MPM, but your PHP Module is not compiled to be threadsafe.  You need to recompile PHP.
AH00013: Pre-configuration failed
Action 'configtest' failed.
The Apache error log may have more information.
administrator@CLOUD-LINUX:~$


Further I have no feeling for what might be appropriate settings / sizings for the Apache variables according to our needs. (about 50 user, some of them with a huge Folder hierarchy and more than 20GB)

What would you suggest?
"Der Anfang jeder Katastrophe ist ne beschissene Vermutung!"

tflidd
Forum Moderator
Posts: 7159
Joined: Sat Dec 07, 2013 7:27 pm
ownCloud version: 8.2.3
Webserver: Apache
Database: MySQL
OS: Linux

Re: oc8 seems to drag apache down on high client sync loads

Postby tflidd » Fri Apr 17, 2015 9:49 am

Then you have also to switch to php over fastcgi (packages: libapache2-mod-fastcgi php5-fpm). There are plenty of guides.

I would first try to optimize apache-prefork. Maybe not the best overall performance but perhaps it's already sufficient.

RealRancor
ownCloud master
Posts: 17381
Joined: Sat May 26, 2012 3:00 pm
ownCloud version: 9.0.2
Webserver: nginx
Database: MySQL
OS: Linux
PHP version: 7.0.x

Re: oc8 seems to drag apache down on high client sync loads

Postby RealRancor » Fri Apr 17, 2015 9:56 am

Hi,

when i'm reading "crashing apache" then could you have a look if you have the package php5-apcu installed?

Version 4.0.2 of php5-apcu shipped in 14.04 LTS is known for such problems. There is an workaround build into ownCloud to disable this broken version, but according to the changelog of the upcoming OC 8.0.3:

https://owncloud.org/releases/Changelog

there is a fix for "Fix disabling of APCu".
*gone*

tflidd
Forum Moderator
Posts: 7159
Joined: Sat Dec 07, 2013 7:27 pm
ownCloud version: 8.2.3
Webserver: Apache
Database: MySQL
OS: Linux

Re: oc8 seems to drag apache down on high client sync loads

Postby tflidd » Fri Apr 17, 2015 10:00 am

I forgot, besides of the comment of RealRancor, have a look into your apache error-logfile (and php). There should be some information when apache reaches its limits.

User avatar
finius
Starter
Posts: 95
Joined: Thu Jul 18, 2013 10:32 am
ownCloud version: 8.0.2
Webserver: Apache
Database: MySQL
OS: Linux
PHP version: Ubuntu 14.04 BuildIn
Location: Frankfurt / Main

Re: oc8 seems to drag apache down on high client sync loads

Postby finius » Fri Apr 17, 2015 10:44 am

Thanks for your hints!

My /etc/apache2/apache2.conf now got this entry:

Code: Select all

<Directory /var/www/owncloud/ >
   Options Indexes SymLinksIfOwnerMatch
   AllowOverride All
</Directory>


To /etc/apache2/sites-enabled/000-default.conf I added this entry:

Code: Select all

<IfModule prefork.c>
        StartServers 10
        MinSpareServers 10
        MaxSpareServers 500
        ServerLimit 125
        MaxClients 125
        MaxRequestsPerChild 100
</IfModule>


And in the /var/www/owncloud/.htaccess I modified this:

Code: Select all

php_value memory_limit 32M


The only thing I don't get is, why there was so much swapping, even if the servers physical memory wasn't reached yet.
"Der Anfang jeder Katastrophe ist ne beschissene Vermutung!"

User avatar
finius
Starter
Posts: 95
Joined: Thu Jul 18, 2013 10:32 am
ownCloud version: 8.0.2
Webserver: Apache
Database: MySQL
OS: Linux
PHP version: Ubuntu 14.04 BuildIn
Location: Frankfurt / Main

Re: oc8 seems to drag apache down on high client sync loads

Postby finius » Fri Apr 17, 2015 10:57 am

Thanks again,
just removed apc. I'll keep an eye on the error logs.

We will now keep watching if the service keep running from now on.
Regards
"Der Anfang jeder Katastrophe ist ne beschissene Vermutung!"

User avatar
Akhénaton
Helpful Elf
Posts: 172
Joined: Wed Jul 23, 2014 6:39 pm
ownCloud version: 9.0.2
Webserver: Apache
Database: MySQL
OS: Linux
PHP version: 7.0.7
Location: Paris - FR

Re: oc8 seems to drag apache down on high client sync loads

Postby Akhénaton » Fri Apr 17, 2015 11:26 am

Hi,

[quote="finius"]And in the /var/www/owncloud/.htaccess I modified this:

Code: Select all

php_value memory_limit 32M


Hum, that's really too short... Try increasing to 128M or even more.

post_max_size integer
Sets max size of post data allowed. This setting also affects file upload. To upload large files, this value must be larger than upload_max_filesize. If memory limit is enabled by your configure script, memory_limit also affects file uploading. Generally speaking, memory_limit should be larger than post_max_size.

You must also consider
php_value max_execution_time 3600
php_value max_input_time 3600

Regards,

P.S. I rather think that your problem is not with Apache2 but in your PHP config.
Last edited by Akhénaton on Fri Apr 17, 2015 11:46 am, edited 1 time in total.
ownCloud 9.0.2 + Redis server 3.0.7 / 2.2.7 + PHP 7.0.7 running at home.

User avatar
finius
Starter
Posts: 95
Joined: Thu Jul 18, 2013 10:32 am
ownCloud version: 8.0.2
Webserver: Apache
Database: MySQL
OS: Linux
PHP version: Ubuntu 14.04 BuildIn
Location: Frankfurt / Main

Re: oc8 seems to drag apache down on high client sync loads

Postby finius » Fri Apr 17, 2015 11:42 am

Akhénaton wrote:Hum, that's really too short... Try increasing to 128M or even more.


If I would set this value to 128M, I guess it would also reeffect these values:

Code: Select all

<IfModule prefork.c>
        StartServers 10
        MinSpareServers 10
        MaxSpareServers 500
        ServerLimit 125
        MaxClients 125
        MaxRequestsPerChild 100
</IfModule>

So, If the server has got 4GB of RAM, I would guess I have to set down MaxClients to something below 32 - also the other value depending on this, right?
So, what happens if the 33th user tries to sync his cloud? Or, if there is just one active user/client but with a huge sync load (extensive folder hierarchy, many files, big data dir about 75GB)?
"Der Anfang jeder Katastrophe ist ne beschissene Vermutung!"


  • Similar Topics
    Replies
    Views
    Last post

Return to “ownCloud Server 8.0 and 8.1”

Who is online

Users browsing this forum: No registered users and 2 guests