Idea to make desktop sync client more efficient

Ask all your questions on desktop sync clients. Please read the Support Forum Rules
Forum rules
Questions on desktop sync clients only.
You want to report a bug or check for feature requests?
Please visit https://github.com/owncloud/mirall/issues

Idea to make desktop sync client more efficient

Postby jblack » Mon Apr 29, 2013 6:11 pm

There has been a lot of progress made in ownCloud lately, but there is still a serious issue regarding desktop client efficiency that has been alarmingly written off here in the forums as "ownCloud working as intended". By default, the sync client checks the server every 30 seconds for changes. Anecdotally, on my laptop it takes 20 seconds at 50%+ CPU to check the server on the local network for my 8 GB Dropbox folder even if no files have changed. This might be relatively unnoticeable on a desktop, but on a laptop this is absolutely horrible for battery life and performance. Currently the "fix" is to increase the poll interval. However, this ignores the larger issue at hand and undermines the reason for having a sync client in the first place. This issue is so bad that I have had to stop using ownCloud altogether. My goal was to have ownCloud handle thousands of files and many GB of data on a number of computers, but I can't trust it with this going on. Since there is a ton of potential for ownCloud and I want it to succeed, I hope my idea will be helpful.

Ever since the desktop client enabled inotify the sync performance increased dramatically. Why can't inotify be used on the server? Instead of the client polling the server for changes, the server would use inotify to detect changes and push them to the client. I am not sure if this idea is either already implemented or in the pipeline, but I thought it was worth mentioning. I realize inotify may not be available in all configurations, but it should still be the server's job to detect changes on the server, not the client.
jblack
Newbie
 
Posts: 3
Joined: Mon Apr 29, 2013 5:13 pm

Re: Idea to make desktop sync client more efficient

Postby srfreeman » Tue Apr 30, 2013 8:17 am

Hello jblack,

It seems to me that you have a non-issue here...

You have 8GB of files that you want to keep synced between your account on the server and your laptop. 8GB seems quite large to me, but let's say for grins that this is comprised of several large spreadsheets, could be any type of files, that you will be updating while offsite for the day, could be week or month I guess.

You connect the laptop to the network, turn on the sync client and in a few seconds you have the latest versions of the files. Presumably you were connected to AC power so there is no battery life hit. The sync client should now be turned off.

You are ready to go, you disconnect from the network and go on your merry way. You use your laptop as normal and update the required files as needed during your time offsite.

When you get back to the office, you connect the laptop to the network and turn the sync client on. In a few minutes the new files are on the server and you are back in sync. Turn the sync client back off there is no need for it until the next trip.

During this entire process the logs show when you transferred the files to your laptop so when matched with travel logs the IT department knows when the files left the premises, what they contained, where you were going with them, when they came back, etc... And, with the versioning turned on, the whole mess can be rolled back if you completely screwed up while you were gone.

Syncing of files has been going on for years, it is relatively easy to keep track of thousands of files of whatever size on numerous machines with the correct logging procedures and corporate policies.

None of this has any bearing on the poll interval of the sync client or the use of inotify or any need to trust anything. Client control of the sync process seems most economical. The server detecting changes on the server process can be helpful when creating a mirror on another server, ala rsync, but, that is not ownCloud's lot in life.
srfreeman
OwnCloud professional
 
Posts: 1625
Joined: Sun Apr 21, 2013 9:34 am
Location: US
ownCloud version: 6.0.4
Webserver: Apache
Database: MySQL
OS: Linux
PHP version: 5.3.10

Re: Idea to make desktop sync client more efficient

Postby jblack » Tue Apr 30, 2013 8:35 pm

Sure, I can micro-manage the sync client on my own, but that goes completely against ownCloud's current mantra of "solving the Dropbox problem." For it to replace Dropbox, ordinary users don't need to worry about keeping their computer usable by turning off and on the client to keep their files synchronized. No sane IT department would want to entrust their users to do anything more than they have to when Dropbox can be left on and do it automatically without penalty. Shouldn't that be a sign that ownCloud can do things better? I don't understand where you're coming from when you say that it's not ownCloud Server's job of doing necessary ownCloud server duties. Having an efficient and usable sync process when ownCloud is touting itself to be a Dropbox replacement is absolutely essential and should very much be their "lot in life."
jblack
Newbie
 
Posts: 3
Joined: Mon Apr 29, 2013 5:13 pm

Re: Idea to make desktop sync client more efficient

Postby srfreeman » Tue Apr 30, 2013 9:46 pm

Hello jblack,

Okay, keep the sync client on. It just seems unnecessary, when you are out of the office it will not connect to the server behind the firewall and there is no need to. When you are connected to the corporate network, files can be edited in place, no real need for sync. There is a Pause button on the sync client interface and there seems to be a reason for it.

Remember that the 'Dropbox Problem' is simply that corporate files are being stored outside the firewall on servers and devices that are not under control of the IT department.

The relative sanity of using in house sync technology as opposed to many other ways of solving the problem is certainly up for debate, it just seems to be the soup of the day and ownCloud does provide a way to do it while allowing the employees to play safely. Usage and control policies are still up to the IT department and the corporate training system.
srfreeman
OwnCloud professional
 
Posts: 1625
Joined: Sun Apr 21, 2013 9:34 am
Location: US
ownCloud version: 6.0.4
Webserver: Apache
Database: MySQL
OS: Linux
PHP version: 5.3.10

Re: Idea to make desktop sync client more efficient

Postby RandolphCarter » Tue Apr 30, 2013 10:23 pm

To me the original intention of the OP definitely makes sense. Being able to leave it running all the time without having to worry about it is one of the main Dropbox advantages I would say.
And indeed there is an issue about it: https://github.com/owncloud/mirall/issues/29
The client side part is implemented; the part for synchronizing server changes is still waiting on such an update.
RandolphCarter
Inventory
 
Posts: 4299
Joined: Wed May 30, 2012 7:42 am
ownCloud version: 7.0.1
Webserver: Apache
Database: MySQL
OS: Linux
PHP version: 5.5.x

Re: Idea to make desktop sync client more efficient

Postby srfreeman » Tue Apr 30, 2013 10:50 pm

Hello RandolphCarter,

Certainly there is room for improvement in the ownCloud sync processes. There is no byte or block level comparrison, no delta encoding, etc... CPU usage has been an issue since the inception of rsync, solve that one and we have something.

The current code does what is required at this point, deciding as a coommunity member to drop out of the project and or complain about the effectiveness of the community based on an issue that can be solved with a small usage adjustment seems out of place to me.
srfreeman
OwnCloud professional
 
Posts: 1625
Joined: Sun Apr 21, 2013 9:34 am
Location: US
ownCloud version: 6.0.4
Webserver: Apache
Database: MySQL
OS: Linux
PHP version: 5.3.10

Re: Idea to make desktop sync client more efficient

Postby jblack » Wed May 01, 2013 6:27 pm

RandolphCarter, thank you for the link; I'll definitely keep an eye on any updates to that issue specifically. I'm curious as to what the developers are afraid of in regards to inotify / folder watchers being unreliable. If that's the case then it really makes me wonder what Dropbox uses. I know Dropbox has to occasionally "check the index" for changes, so maybe that's their fallback solution. I am familiar with C++ / Qt so I might see if there's anything I can do to help development.

Srfreeman, I understand where you're coming from where a corporate environment holds data security paramount. For your application it seems that ownCloud is "working as intended." However, I've never understood ownCloud to be meant for only enterprise solutions. I realize coming in the forums and saying I can't use the product makes it seem like I'm dropping out of the community or complaining, but it's really quite the contrary. If I honestly didn't care then I would have said nothing at all. I have been testing ownCloud on and off for over a year to see how it's progressing and I was hoping that by coming into the forums with some feedback it may help in some way. I've spent many, many hours reading the forums seeing other people have this and similar issues with no real solution other than "change how you use it." If it's meant to replace Dropbox then why would you want to use it differently by sacrificing features and usability? Even people in that issue thread claim this to be an absolute showstopper for deployment until it's fixed. I still have a lot of hope for this project as the conflicting files issue that I got surprised by a couple months ago has been recently fixed. I feel it's only a matter of time before this is fixed as well.
jblack
Newbie
 
Posts: 3
Joined: Mon Apr 29, 2013 5:13 pm

Re: Idea to make desktop sync client more efficient

Postby srfreeman » Wed May 01, 2013 11:17 pm

Hello jblack,

Yourself and many others are confused in the usage disscussions, taking 'use it as it is intended' as 'change how you use it'.

When you take the enterprise need for endpoint security and control out of the mix, ownCloud simply becomes another of the many web based file manager softwares.

Since ownCloud was intended to replace the 'use' of Dropbox in the enterprise environment, not as a competing product, feature comparison with Dropbox is really a futile endeavor. Without the enterprise 'Dropbox problem' it will be hard to fit ownCloud into any usage model, and I see that as an expected result.

Since we have here a forum for discussion amongst community members, not developers, the fact that you feel slighted because the development has not matched your usage is really a moot point. The only thing we 'can' discuss is the usage of the code in its current form.

Joining the development team and bringing your skills to bear in moving the project to where you want it to go would certainly be a more productive outlet for your concerns.

BTW; If you get into development and would like to add functionallity to make the sync portion of ownCloud more like the Dropbox product, look into the other open source project librsync.
srfreeman
OwnCloud professional
 
Posts: 1625
Joined: Sun Apr 21, 2013 9:34 am
Location: US
ownCloud version: 6.0.4
Webserver: Apache
Database: MySQL
OS: Linux
PHP version: 5.3.10

Re: Idea to make desktop sync client more efficient

Postby jospoortvliet » Mon Jul 28, 2014 8:52 am

srfreeman: I really don't know what the FSCK you are talking about. ownCloud was most definately build to be a product to feature everything dropbox does - and more. No offence, but this is absolute nonsense:
BTW; If you get into development and would like to add functionallity to make the sync portion of ownCloud more like the Dropbox product, look into the other open source project librsync.


Things are rather the other way around. ownCloud was and is and always has aimed to be able to replace dropbox for normal home users as much as possible and we go through extreme lengths to make it as easy as possible. That is why it is written in PHP in the first place (runs EVERYWHERE), that is why it still defaults to SQLite despite its performance issues (pissing off many advanced users!) and why ease of use is one of the central development goals.

If you have ever sat together with Frank Karlitschek (our benevolent dictator :D ) and Jan-c Borchart (lead designer), you would know that the day ownCloud ceases to be useful for normal home users and exclusively targets enterprise usage is the day they will fork ownCloud to bring it back to its original goal: free the data of NORMAL home users.

@jblack: I can guarantee you that ANY improvement to the performance of either the client or the server, or their interaction, would be more than appreciated. If you don't believe it, I hereby invite you to come to the ownCloud Contributor Conference in the last week of August in Berlin (which I am organizing) to work directly with Klaas and Daniel and the others on the client and/or server.

I hope you have tried ownCloud 7 and have noticed we worked real hard to make it faster in many ways. As Randolph said, the server part of using change notifications is afaik now also implemented and that should help keep cpu usage down. And I believe we now do parallel, chunked uploads - still not 'only changes', we didn't get to that but help is welcome. Also - it still kicks off a full sync ever 5 minutes, I've asked why that is:
ogoffart: the problem is that changes can happen durring the sync
ogoffart: and we disabled the watcher during the sync
ogoffart: for the obvious reason that the sync is modifying all the files
jospoortvliet: ok so you have to re-scan everything after the sync? sync modifies ALL files?
ogoffart: Well, all the files that it downloads
ogoffart: + the journal
jospoortvliet: just for my info: is it possible to do this better and did we simply not have time to do this, or is this the best we'll ever do?
ogoffart: the problem is also that the watcher does not tell *which files* were changed, just the directories in which something was changed
ogoffart: When the watcher tells us exactly which file change, it is possible to ignore the journal or file that we are currently syncing


So, this would need work on the watcher. You say you have Qt and C++ coding skills - if you're interested in helping to fix the issues you have I'm certain that would be welcomed with open arms. Ping me if you need any help - google should give you a mail address ;-)
jospoortvliet
Beginner
 
Posts: 19
Joined: Fri Apr 11, 2014 1:20 pm
ownCloud version: 6.0.2
Webserver: Apache
Database: MySQL
OS: Linux
PHP version: 5.5


Return to ownCloud desktop sync clients

Who is online

Users browsing this forum: No registered users and 5 guests