Poor performance with many small files???

Ask all your questions on desktop sync clients. Please read the Support Forum Rules
Forum rules
The forums were migrated over to https://central.owncloud.org which is based on the forum software Discourse. The forums here is put into read-only mode starting from today.

More background information about this move and the reasoning behind it is available in this blogpost:

https://daniel.molkentin.net/2016/07/20 ... d-central/
srfreeman
ownCloud professional
Posts: 1625
Joined: Sun Apr 21, 2013 10:34 am
ownCloud version: 6.0.4
Webserver: Apache
Database: MySQL
OS: Linux
PHP version: 5.3.10
Location: US

Poor performance with many small files???

Postby srfreeman » Wed Oct 09, 2013 5:54 pm

Hello All;

I have seen several posts noting poor performance when synchronizing several small files, providing little information used to define the term 'poor'. So I decided to test using publicly available infrastructure in an attempt to see what the problem is. I purchased a cheap VPS with the only criteria being that I could run Ubuntu 12.04 on it.

Specs are:
Operating system: Ubuntu Linux 12.04
Kernel and CPU: Linux 2.6.32-042stab076.8 on i686
Processor information: Intel(R) Xeon(R) CPU E3-1240 V2 @ 3.40GHz, 4 cores
Real memory: 134.33 MB used, 2 GB total
Local disk space: 2.69 GB used, 30 GB total (RAID6 they tell me)

ownCloud 5.0.12 installed (via apt get)

Server Location: Data center, somewhere on the East coast, US
Client Location: Residential, West Tennessee, central US

The results do not seem bad to me. I would appreciate some conversation on whether the community considers this 'poor' performance:

Here are some times I see for upload to an ownCloud instance running on a VPS with Internet access - Shared 1Gbps port. Local ISP gives me 5mbps upload speed.

Using a file set consisting of 1933 files in 334 folders (sizes from <1KB to 20KB) totaling ~11MB. Avg. file size ~0.6KB

Upload through WebDAV client (NetDrive) - Total upload in ~12 minutes. No bandwidth throttling. Avg. of ~0.4 seconds per file.

Upload through ownCloud client v. 1.4.1 - Total upload in ~30 minutes. Upload bandwidth throttled to 10KBps, download bandwidth throttled to 80KBps. Avg. of ~0.9 seconds per file. Without throttling I can exceed the shared port limits of the VPS provider, presumably because of the two way communications required for the synchronization process.

Upload to FTP server through FileZilla - Total upload in ~10 minutes, after resending some failed transfers. No bandwidth throttling. Avg. of ~0.3 seconds per file.

Certainly, performance could be improved by caching the web server, indexing the database, separating the database server and web server, adjusting the 'bandwidth throttling', etc..., but I cannot see the current performance of ownCloud server or client here as anything that could be considered 'poor'.

Kartoffelfreund
Beginner
Posts: 10
Joined: Wed Oct 09, 2013 7:00 pm
ownCloud version: 5.0.12
Webserver: Apache
Database: SQLite
OS: Linux Embedded
PHP version: 5

Re: Poor performance with many small files???

Postby Kartoffelfreund » Wed Oct 09, 2013 7:08 pm

I had the problem with poor performance.

My System is as follows:

Raspberry PI Rev. B
Apache
PHP5
SQLite
Raspbian netinstall

My definition of 'slow' is this one:
[*] Fetching foto of ~300KB: 4 Seconds
[*] Transfer: fast (few miliseconds)
[*] Storing: 5-7 Seconds.

The transfer is fast with larger files (~3-4 MB/s) but the fetching and saving process are incredibly slow...

this server has only one user: Me.

I try to keep my Laptop and my smartphone actual.

srfreeman
ownCloud professional
Posts: 1625
Joined: Sun Apr 21, 2013 10:34 am
ownCloud version: 6.0.4
Webserver: Apache
Database: MySQL
OS: Linux
PHP version: 5.3.10
Location: US

Re: Poor performance with many small files???

Postby srfreeman » Wed Oct 09, 2013 7:35 pm

Hello Kartoffelfreund,

So, do you consider the performance of my posted test 'poor' or not?

Is the point of your post to illustrate the RPI does not provide the performance you need?

Without knowing the number and size of files, the network type and speeds, the storage type, etc... it is impossible to judge the relative performance of the noted system.

Kartoffelfreund
Beginner
Posts: 10
Joined: Wed Oct 09, 2013 7:00 pm
ownCloud version: 5.0.12
Webserver: Apache
Database: SQLite
OS: Linux Embedded
PHP version: 5

Re: Poor performance with many small files???

Postby Kartoffelfreund » Wed Oct 09, 2013 8:07 pm

For me there's no difference if I try to transmit a single file with 4sec read, few millisec transfer, 6-7 sec write or if it's a folder with roughly 1000 Pictures... this lasts 1000 times the first time ;)

My laptop and the rPI are connected to the local network, the rPI with cable and my thinkpad via wireless.

My Thinkpad is a T400 with a OCZ Vertex2 SSD as system disk, Ubuntu 13.04 OS. The above shown values com from the values the ubuntu-oc-client shows me.

The owncloud DB is saved on a 320GB, 5k rpm samsung HDD, connected via USB2.

I recenty made a backup of my (laptop) home-folder with ubuntu's backup program "déja vu" and that was MUCH faster. The backup lasted rougly two hours for ~30GB (~150.000 files).

The OC client just transferred 8GB of 16GB in TWO DAYS

srfreeman
ownCloud professional
Posts: 1625
Joined: Sun Apr 21, 2013 10:34 am
ownCloud version: 6.0.4
Webserver: Apache
Database: MySQL
OS: Linux
PHP version: 5.3.10
Location: US

Re: Poor performance with many small files???

Postby srfreeman » Wed Oct 09, 2013 8:22 pm

Hello Kartoffelfreund,

I can see one bottle neck already.
Kartoffelfreund wrote:The owncloud DB is saved on a 320GB, 5k rpm samsung HDD, connected via USB2.
The slow hard drive on a slow connection will mean that the several reads and writes to the data base required for synchronization will be very slow due to disk IO waits. The restrictions of the ARM processor will just exacerbate the situation.

The database usage is not required for your backup, so, of course it would be faster.

If you were to move to a 7.2K rpm disk on a USB3 connection, you would see much better performance.

Kartoffelfreund
Beginner
Posts: 10
Joined: Wed Oct 09, 2013 7:00 pm
ownCloud version: 5.0.12
Webserver: Apache
Database: SQLite
OS: Linux Embedded
PHP version: 5

Re: Poor performance with many small files???

Postby Kartoffelfreund » Wed Oct 09, 2013 8:31 pm

Well... I know that USB2 and the HDD are not the fastest but ~12seconds for only writing a single bitmap with a size of 300kbit??? I can't believe this.

That the webinterface is not the fastest is clear and that is not the point wich annoys me.

srfreeman
ownCloud professional
Posts: 1625
Joined: Sun Apr 21, 2013 10:34 am
ownCloud version: 6.0.4
Webserver: Apache
Database: MySQL
OS: Linux
PHP version: 5.3.10
Location: US

Re: Poor performance with many small files???

Postby srfreeman » Wed Oct 09, 2013 8:52 pm

Hello Kartoffelfreund,

The actual write of the file to disk has little to do with it. The hundreds of reads to the database per file and the several writes for logging from the several server elements (web server, data base server, PHP, ownCloud itself, etc...) have much more of an effect on performance - all dependent on the speed of the hard drive and its interface with the system at large.

Note also, the relatively small buffer available to your particular hard drive.

The web interface is going to be as fast as the underlying infrastructure and the network connection allows it to be. Again, we are back to the hard drive, when speaking of your noted system.

Sorry that you are annoyed, but, server administration does have its requirements.

Kartoffelfreund
Beginner
Posts: 10
Joined: Wed Oct 09, 2013 7:00 pm
ownCloud version: 5.0.12
Webserver: Apache
Database: SQLite
OS: Linux Embedded
PHP version: 5

Re: Poor performance with many small files???

Postby Kartoffelfreund » Wed Oct 09, 2013 9:07 pm

srfreeman wrote:Hello Kartoffelfreund,

Hello srfreeman

srfreeman wrote:The actual write of the file to disk has little to do with it. The hundreds of reads to the database per file and the several writes for logging from the several server elements (web server, data base server, PHP, ownCloud itself, etc...) have much more of an effect on performance - all dependent on the speed of the hard drive and its interface with the system at large.

Hm... I thought it is easy as 1-2-3: rad from client, transfer to server, storing the file. I have no clue why it should have/has "hundreds of reads to the database" but okay...My pictures (clientside) are saved via backup so this isn't that important.

srfreeman wrote:The web interface is going to be as fast as the underlying infrastructure and the network connection allows it to be. Again, we are back to the hard drive, when speaking of your noted system.
Here I see the problem of having a heavy load on PI's small shoulders and I accept it.

srfreeman wrote:Sorry that you are annoyed, but, server administration does have its requirements.

I thougt it should work with ease 'just' running a webserver with nothing on it and OC, where I am the one and only user. I do not need connection via internet or so... I just want to keep a calendar (okay four), my contacts, my photos and my music synced... :/

srfreeman
ownCloud professional
Posts: 1625
Joined: Sun Apr 21, 2013 10:34 am
ownCloud version: 6.0.4
Webserver: Apache
Database: MySQL
OS: Linux
PHP version: 5.3.10
Location: US

Re: Poor performance with many small files???

Postby srfreeman » Wed Oct 09, 2013 9:37 pm

Hello Kartoffelfreund,

Yep, it is a bit more complicated than many seem to think.

From testing and our conversation here, it seems that a $7.00 per month VPS and a cheap, 5 year old PC with an Internet connection will trump an RPI and a business class notebook for your noted needs.

Add in the cost of time and aggravation and you can afford to run the VPS for ever. :)

Kartoffelfreund
Beginner
Posts: 10
Joined: Wed Oct 09, 2013 7:00 pm
ownCloud version: 5.0.12
Webserver: Apache
Database: SQLite
OS: Linux Embedded
PHP version: 5

Re: Poor performance with many small files???

Postby Kartoffelfreund » Wed Oct 09, 2013 10:19 pm

Hey and thanks for your answer. :D

I always want to learn stuff so this does not matter. The power of the PI is enough for me as I have found my solution (big stuff via "deja vu" and small stuff with OC).

And I can't get replace my laptop because I have to work with it ;)


  • Similar Topics
    Replies
    Views
    Last post

Return to “ownCloud desktop sync clients”

Who is online

Users browsing this forum: Bing [Bot], Yahoo [Bot] and 3 guests