Relatively slow upload speed

Ask all your questions regarding OC 8.0 and 8.1 Please read the Support Forum Rules
Forum rules
The forums were migrated over to https://central.owncloud.org which is based on the forum software Discourse. The forums here is put into read-only mode starting from today.

More background information about this move and the reasoning behind it is available in this blogpost:

https://daniel.molkentin.net/2016/07/20 ... d-central/
Stamper
Newbie
Posts: 2
Joined: Wed Aug 26, 2015 6:51 pm
ownCloud version: 8.1.1
Webserver: nginx
Database: MySQL
OS: Linux
PHP version: 5.6.12

Relatively slow upload speed

Postby Stamper » Wed Aug 26, 2015 7:45 pm

Hi everyone!

My first post and it is one of those "arggg my owncloud is slow and I don't know how to fix it :(" posts. Sorry about that! I have been trying to figure out for myself what could be wrong but I cannot seem to find any issue with the configuration which might be the bottleneck here. I am looking for the magic speed switch :P

Problem
So first things first, what is the problem: I have used the Arch linux wiki to install Owncloud 8.1.1 on my Arch server (https://wiki.archlinux.org/index.php/OwnCloud). While uploading a large file, my upload speed is "only" ~7MBps while sitting on the same gigabit LAN as the server. Is this expected performance or is something the matter?

Questions
- Can someone tell me which configuration options I can try to increase my upload/download speed?
- Could someone give me a basic overview on which components do what so I could try to self-debug further? (e.g. which component does the actual writing of the file, are there buffers involved between php and nginx for file transfer)
- Feel free to post any other comments you might think relevant to help with this problem

Thank you in advance!!

Configuration
Arch linux 4.1.4
Nginx 1.8.0
MySQL/MariaDB 15.1 / 10.0.20
Php 5.6.12 (using php-fpm)
Php APCu 4.0.7

System
Intel G3220
Raid 1 2x 1TB WD Red drives
4gb DDR3

What have I looked at
- I have looked at the administrators guide for server tuning and done some of the steps I thought necassary
- Drive benchmark, read and write +150MB/s
- Sync client windows and linux and browser upload all report same speeds
- Server tuning for nginx, php and mariadb (mysqltuner)
EDIT: - htop doesn't report anything odd. CPU is only at 10%, total mem usage is at ~25%
- iotop reports nginx writing when browser upload, php and nginx writing when sync client upload

Experience
This is my first personal server and my first owncloud setup. So feel free to ask me the most basic questions as I might have forgotten something really simple.

Nginx config

Code: Select all

user nginx http;
worker_processes  2;

events {
    worker_connections  64;
}

http {
    include       mime.types;   
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;
   
    access_log /var/log/nginx/access.log;
    error_log  /var/log/nginx/error.log  notice;   

    # TLS/SSL settings
    ssl_session_timeout          5m;
    ssl_session_cache            shared:SSL:5m;
    ssl_protocols                TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers                  "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH";
    ssl_prefer_server_ciphers    on;
    ssl_session_tickets          off;
    ssl_stapling                 on;
    ssl_stapling_verify          on;
    resolver                     8.8.8.8 8.8.4.4 valid=300s;
    resolver_timeout             5s;

    add_header Strict-Transport-Security "max-age=31536000; includeSubdomains;";

    #Error pages
    error_page   500 502 503 504  /50x.html;

    # Redirect all http to https
    server {
        listen       80;
        return       301 https://$host$request_uri;
    }

    upstream php-handler {
      server unix:/run/php-fpm/php-fpm.sock;
    }

    server {
      listen 443 ssl spdy;
      server_name owncloud.;

      ssl_certificate     validpath;
      ssl_certificate_key validpath;

      # Add headers to serve security related headers
      add_header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload;";
      add_header X-Content-Type-Options nosniff;
      add_header X-Frame-Options "SAMEORIGIN";
      add_header X-XSS-Protection "1; mode=block";
      add_header X-Robots-Tag none;

      # Path to the root of your installation
      root                  /www/owncloud/;
      # set max upload size
      client_max_body_size  16G;
      fastcgi_buffers 128   1M;
      sendfile              on;

      # Disable gzip to avoid the removal of the ETag header
      gzip off;

      # Uncomment if your server is build with the ngx_pagespeed module
      # This module is currently not supported.
      #pagespeed off;

      rewrite ^/caldav(.*)$ /remote.php/caldav$1 redirect;
      rewrite ^/carddav(.*)$ /remote.php/carddav$1 redirect;
      rewrite ^/webdav(.*)$ /remote.php/webdav$1 redirect;

      index index.php;

      location = /robots.txt {
        allow all;
        log_not_found off;
        access_log off;
      }

      location ~ ^/(?:\.htaccess|data|config|db_structure\.xml|README){
        deny all;
      }

      location / {
       # The following 2 rules are only needed with webfinger
         rewrite ^/.well-known/host-meta /public.php?service=host-meta last;
         rewrite ^/.well-known/host-meta.json /public.php?service=host-meta-json last;

         rewrite ^/.well-known/carddav /remote.php/carddav/ redirect;
         rewrite ^/.well-known/caldav /remote.php/caldav/ redirect;

         rewrite ^(/core/doc/[^\/]+/)$ $1/index.html;

         try_files $uri $uri/ /index.php;
       }

       location ~ \.php(?:$|/) {
         fastcgi_split_path_info ^(.+\.php)(/.+)$;
         include fastcgi_params;
         fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
         fastcgi_param PATH_INFO $fastcgi_path_info;
         fastcgi_param HTTPS on;
         fastcgi_param modHeadersAvailable true; #Avoid sending the security headers twice
         fastcgi_pass php-handler;
       }

       # Optional: set long EXPIRES header on static assets
       location ~* \.(?:jpg|jpeg|gif|bmp|ico|png|css|js|swf)$ {
           expires 30d;
           # Optional: Don't log access to assets
             access_log off;
       }

      }
}


mysql config

Code: Select all

# The MariaDB server
[mysqld]
port      = ....
bind-address    = 127.0.0.1
socket      = /run/mysqld/mysqld.sock
skip-external-locking
key_buffer_size = 16M
max_allowed_packet = 1M
table_open_cache = 64
sort_buffer_size = 512K
net_buffer_length = 8K
read_buffer_size = 256K
read_rnd_buffer_size = 512K
myisam_sort_buffer_size = 8M

query_cache_type = 1
query_cache_limit = 2M
query_cache_min_res_unit = 2K
query_cache_size = 64M

tmp_table_size = 128M
max_heap_table_size = 64M
table_open_cache = 128

log-bin=mysql-bin

# binary logging format - mixed recommended
binlog_format=mixed

innodb_buffer_pool_size = 256M
innodb_buffer_pool_instance = 1

thread_cache_size = 4

[mysqldump]
quick
max_allowed_packet = 32M

[mysql]
no-auto-rehash

[myisamchk]
key_buffer_size = 20M
sort_buffer_size = 20M
read_buffer = 2M
write_buffer = 2M

[mysqlhotcopy]
interactive-timeout


APCu

Code: Select all

extension=apcu.so

apc.ttl=7200
apc.enable_cli=1
apc.enabled=1
apc.shm_size=64M


php-fpm

Code: Select all

pid = /run/php-fpm/php-fpm.pid
error_log = /var/log/php/php-fpm-error.log

[www]
listen = /run/php-fpm/php-fpm.sock
listen.owner = php-fpm
listen.group = http
listen.mode = 0660

pm = dynamic
pm.max_children = 5
pm.start_servers = 2

env[HOSTNAME] = $HOSTNAME
env[PATH] = /usr/local/bin:/usr/bin:/bin


php

Code: Select all

[PHP]
safe_mode = off
engine = On
short_open_tag = Off
asp_tags = Off
precision = 14
output_buffering = 4096
zlib.output_compression = Off
implicit_flush = Off
unserialize_callback_func =
serialize_precision = 17
open_basedir = <stuff>
disable_functions =
disable_classes =
realpath_cache_size = 1M
zend.enable_gc = On
expose_php = On
max_execution_time = 30
max_input_time = 60
memory_limit = 128M
error_reporting = E_ALL & ~E_DEPRECATED & ~E_STRICT
display_errors = Off
display_startup_errors = Off
log_errors = On
log_errors_max_len = 1024
ignore_repeated_errors = Off
ignore_repeated_source = Off
report_memleaks = On
track_errors = Off
html_errors = On
variables_order = "GPCS"
request_order = "GP"
register_argc_argv = Off
auto_globals_jit = On
post_max_size = 16G
auto_prepend_file =
auto_append_file =
default_mimetype = "text/html"
default_charset = "UTF-8"
include_path = ".:/usr/share/pear"
doc_root =
user_dir =
extension_dir = "/usr/lib/php/modules/"
enable_dl = Off
file_uploads = On
upload_max_filesize = 16G
max_file_uploads = 20
allow_url_fopen = On
allow_url_include = Off
default_socket_timeout = 60
extension=bz2.so
extension=curl.so
extension=exif.so
extension=gd.so
extension=gettext.so
extension=iconv.so
extension=imap.so
extension=intl.so
extension=mcrypt.so
extension=mysql.so
extension=mysqli.so
zend_extension=opcache.so
extension=openssl.so
extension=pdo_mysql.so
extension=posix.so
extension=xmlrpc.so
extension=zip.so

[CLI Server]
cli_server.color = On

[Date]
date.timezone = Europe/Amsterdam

[Pdo_mysql]
pdo_mysql.cache_size = 2000
pdo_mysql.default_socket=

[mail function]
SMTP = localhost
smtp_port = 25
mail.add_x_header = On

[SQL]
sql.safe_mode = Off

[ODBC]
odbc.allow_persistent = On
odbc.check_persistent = On
odbc.max_persistent = -1
odbc.max_links = -1
odbc.defaultlrl = 4096
odbc.defaultbinmode = 1

[Interbase]
ibase.allow_persistent = 1
ibase.max_persistent = -1
ibase.max_links = -1
ibase.timestampformat = "%Y-%m-%d %H:%M:%S"
ibase.dateformat = "%Y-%m-%d"
ibase.timeformat = "%H:%M:%S"

[MySQL]
mysql.allow_local_infile = On
mysql.allow_persistent = On
mysql.cache_size = 2000
mysql.max_persistent = -1
mysql.max_links = -1
mysql.default_port =
mysql.default_socket =
mysql.default_host =
mysql.default_user =
mysql.default_password =
mysql.connect_timeout = 60
mysql.trace_mode = Off

[MySQLi]
mysqli.max_persistent = -1
mysqli.allow_persistent = On
mysqli.max_links = -1
mysqli.cache_size = 2000
mysqli.default_port = 3306
mysqli.default_socket =
mysqli.default_host =
mysqli.default_user =
mysqli.default_pw =
mysqli.reconnect = Off

[mysqlnd]
mysqlnd.collect_statistics = On
mysqlnd.collect_memory_statistics = Off

[PostgreSQL]
pgsql.allow_persistent = On
pgsql.auto_reset_persistent = Off
pgsql.max_persistent = -1
pgsql.max_links = -1
pgsql.ignore_notice = 0
pgsql.log_notice = 0

[Sybase-CT]
sybct.allow_persistent = On
sybct.max_persistent = -1
sybct.max_links = -1
sybct.min_server_severity = 10
sybct.min_client_severity = 10

[bcmath]
bcmath.scale = 0

[Session]
session.save_handler = files
session.save_path = "/tmp"
session.use_strict_mode = 0
session.use_cookies = 1
session.use_only_cookies = 1
session.name = PHPSESSID
session.auto_start = 0
session.cookie_lifetime = 0
session.cookie_path = /
session.cookie_domain =
session.cookie_httponly =
session.serialize_handler = php
session.gc_probability = 1
session.gc_divisor = 1000
session.gc_maxlifetime = 1440
session.referer_check =
session.cache_limiter = nocache
session.cache_expire = 180
session.use_trans_sid = 0
session.hash_function = 0
session.hash_bits_per_character = 5
url_rewriter.tags = "a=href,area=href,frame=src,input=src,form=fakeentry"

[MSSQL]
mssql.allow_persistent = On
mssql.max_persistent = -1
mssql.max_links = -1
mssql.min_error_severity = 10
mssql.min_message_severity = 10
mssql.compatibility_mode = Off
mssql.secure_connection = Off

[Tidy]
tidy.clean_output = Off

[soap]
soap.wsdl_cache_enabled=1
soap.wsdl_cache_dir="/tmp"
soap.wsdl_cache_ttl=86400
soap.wsdl_cache_limit = 5

tflidd
Forum Moderator
Posts: 7159
Joined: Sat Dec 07, 2013 7:27 pm
ownCloud version: 8.2.3
Webserver: Apache
Database: MySQL
OS: Linux

Re: Relatively slow upload speed

Postby tflidd » Sat Aug 29, 2015 11:46 pm

Can you try to upload a file via scp as a reference? The number of server and children for php and nginx differ, I don't know if that matters and how many connections you open during your transfer.

If you want to dive deeper into the oc code and track errors, please ask via IRC/Mailinglist for further advice.

Stamper
Newbie
Posts: 2
Joined: Wed Aug 26, 2015 6:51 pm
ownCloud version: 8.1.1
Webserver: nginx
Database: MySQL
OS: Linux
PHP version: 5.6.12

Re: Relatively slow upload speed

Postby Stamper » Wed Sep 02, 2015 10:20 pm

Thanks for thinking with me!

Took me a while, but I have been trying to find info and debug when possible. My internet connection on the server is fine. I can copy a large file through sFTP with 50 MiB/s.

However, I have found a lead which looks very promising! My first view on my iotop log was a bit too hastly. It seems the read and write are indeed not so high but the IO is through the roof! I am using ext4 on the raid 1 partition which is targetted by Owncloud. The journalling process jbd2 is causing a near 100% IO according to iotop. This is only when using Owncloud. During a sFTP transfer the IO % is near zero and something else is causing the bottleneck.

While the jbd2 is hitting my disk, mysqld is slowly but steadily also at ~1-5% disk IO. So my guesse is that mysql has been configured to completely sync all operations which cause the bottleneck. My next step is too find out if that is possible and which options could cause this behavior.

To be continued!

UPDATE:

So yes, we actually went a step forward! Mysql was indeed the bottleneck. Mariadb was continously syncing any commits to disk immediatly instead of saving them up a bit and then mass writing them to disk. If you are in the same situation AND your server is not data critical (if your server powers off during write, will you not wish to undo it by giving away your firstborn? :P), you can adjust the following setting:

innodb_flush_log_at_trx_commit = 2

Default is one which is flush after each sync. 2 is safe them up for about a second (which means you could lose up to a second of transactions).

I had another issue. My Nginx was giving off upstream header too big errors due to long filenames. I had to define bigger fastcgi header buffers:

fastcgi_buffers 16 64k;
fastcgi_buffer_size 128k;

These are relatively enormous!

UPDATE 2:
Reads are happening at a comfortable 45 -50 MB/s with iotop at nearly 0% and htop with nginx at ~25% and php-fpm at 15%. So reads are on par.

UPDATE 3:
SO I have been tweaking some more and found some more usefull settings. Please take care in copying these! If you do not know what they do, find out if they have side effects which you can live with!

So Mysql seems to continue to be the bottleneck. I have found a "burst" option by increasing the size of the log file where commits are written too and increasing the amount of tables that may be dirty.

innodb_log_buffer_size = 32M
innodb_max_dirty_pages_pct = 90

My writes have increased by bursting to 68MB/s at start and steadily declining till about 21 MB/s. My idea is that the logs heap up and cannot be written to disk quickly enough. I am going to look for options to improve this and than lower this buffer as 32 MB of writes lost on powerloss is quit a bit. Will get back to here!

UPDATE 4:
So I also tried some other stuff (e.g. mount with barriers=0 and different flush methods for innodb). Nothing really made a difference. I guess this is it for performance with a mysql heavy program like owncloud and harddrives.


My htop is now filled with 4 php-fpm processes each at ~16% - ~25%. So my CPU can get filled between 60% ~ 90%. My iotop is with jbd2 at ~15% and spikes to 100% every 10 seconds or so (probably the flush kicking in multiple times).

This is for a system with an Intel G3220 pentium with Raid 1 WD Red 1TB drives

kekesiv
Newbie
Posts: 1
Joined: Mon Nov 02, 2015 4:02 pm
ownCloud version: 8.1.3
Webserver: Apache
Database: MySQL
OS: Linux
PHP version: 5.6.14-0+deb8u1

Re: Relatively slow upload speed

Postby kekesiv » Mon Nov 02, 2015 4:11 pm

Stamper wrote:innodb_flush_log_at_trx_commit = 2
You are the man! this is what finally made my owncloud transfer small files 1000 times faster! It synced 3000 files in 1 min instead of the 3 hours initially shown.
Thank you!

sven.glueckspilz
Newbie
Posts: 1
Joined: Mon Jun 27, 2016 10:52 am
ownCloud version: 9.0.2
Webserver: Apache
Database: MySQL
OS: Linux
PHP version: 7

Re: Relatively slow upload speed

Postby sven.glueckspilz » Mon Jun 27, 2016 11:01 am

I have the same trouble with OC902 server on xubuntu1604 using PHP7 (as well as on Raspberry Debian Jessy with PHP 5.6).
I'd like to story my entire Documents folder (128GB, lots of pics plus lots of small files) in OC. With initial setup, I hardly see any sync progress with the OC client. Transfer rates below 5kB/s, estimated duration varies up to 1000 years... MySQL Workbench monitoring states low transfer rates and low number of queries. Server cpu load is rather low.
Eventually I transformed oc_activity, oc_file_cache and oc_file_locks to MYISAM tables (using phpmyadmin). Now sync is flying :-)

tflidd
Forum Moderator
Posts: 7159
Joined: Sat Dec 07, 2013 7:27 pm
ownCloud version: 8.2.3
Webserver: Apache
Database: MySQL
OS: Linux

Re: Relatively slow upload speed

Postby tflidd » Mon Jun 27, 2016 1:24 pm

sven.glueckspilz wrote:Eventually I transformed oc_activity, oc_file_cache and oc_file_locks to MYISAM tables (using phpmyadmin). Now sync is flying :-)

If you have workarounds for possible bottlenecks, it would be great to let the developers know. Either it is an interesting option for everybody or it could have negative consequences that are not obvious.


  • Similar Topics
    Replies
    Views
    Last post

Return to “ownCloud Server 8.0 and 8.1”

Who is online

Users browsing this forum: No registered users and 2 guests