Tahoe-LAFS integration idea

Ask all your questions regarding OC 5.x Please read the Support Forum Rules
Forum rules
Version 5 is not supported anymore! Only security issues are fixed. Please upgrade your ownCloud.
Before you post; make sure you are using at least PHP Version 5.3.x - Also read Support Forum - Read this before posting

Re: Tahoe-LAFS integration idea

Postby anaqreon » Wed Sep 11, 2013 10:24 pm

Now that you mention it, I guess I didn't need admin rights. I could have made a local folder as a regular user and simply given it full read/write permissions for everybody. Instead, I made a folder owned by the "www-data" user directly, not writable by other users, which is only possible if you have admin rights (isn't it?).

The installation of Tahoe itself and the mounting of the grid folder via sshfs did not require special permissions, you are correct. I did not test anything with the public test grid. I only tried it with a three node grid I set up myself (one Rackspace virtual server, one virtual machine using NAT, and one regular machine). Maybe there is some complication due to the fact you are using the public test grid?
anaqreon
Beginner
 
Posts: 36
Joined: Mon Mar 26, 2012 5:15 pm

Re: Tahoe-LAFS integration idea

Postby srfreeman » Wed Sep 11, 2013 10:56 pm

Hello tilllt,

I have not played with the public grid for a year or two, but the guide that I posted a link to earlier provided a working, mountable system as late as march of this year.

Mounting via FUSE is certainly doable and though I have not tried it, I cannot see any reason why the SFTP connector in the ownCloud External Storage app would not work for this.

Admin rights are going to be required to, well, administer the server. I do not understand the problem here...

Normal users of ownCloud can be given the ability to mount an SFTP account as external storage, but somehow I do not think this is what you are after.
srfreeman
OwnCloud professional
 
Posts: 1625
Joined: Sun Apr 21, 2013 10:34 am
Location: US
ownCloud version: 6.0.4
Webserver: Apache
Database: MySQL
OS: Linux
PHP version: 5.3.10

Re: Tahoe-LAFS integration idea

Postby tilllt » Wed Sep 11, 2013 11:27 pm

anaqreon & srfreeman:

First another tip, for local testing. There is a script called grid-in-a-box which lets you quickly set up a bunch of test nodes for local testing of a tahoe grid:
https://github.com/nejucomo/lafs-giab

@srfreeman: about Owncloud and Tahoe Interoperability... lets talk about it once you have tried yourself mounting the testgrid in owncloud. it's kind of meaningless to discuss the theoretical possibility of mounting a tahoe grid in owncloud, when my practical experience was that it didnt work. i know its supposed to work and like i said, i had no problem to access the testgrid via FTP, SFTP or SSHFS via Fuse. The part that didnt work was getting the testgrid to be used as an external storage in owncloud. so i'd appreciate if we could investigate IF it works and IF NOT, find a way to make it work. In my tests it didnt work.

Since my main interest would be a friendnet kind of grid, the tahoe testgrid is pretty close to what i would want to get running.

BTW, as a reply to something mentioned earlier: the tahoe devs mentioned that it wouldnt be a good idea to have a high number of appearing and disappearing storage nodes as it will create too much overhead on the grid. so running storage nodes on the Desktop Machines (that are running the owncloud sync clients) would not be a good idea. what would makes sense though is to run tahoe as a gateway on the desktop system to take off the encryption workload from the storage nodes, since a lot of people also run owncloud on NAS' and Raspberry Pi's atc, which would take forever to encrypt bigger files.
tilllt
Starter
 
Posts: 66
Joined: Fri Jul 06, 2012 9:59 am

Re: Tahoe-LAFS integration idea

Postby srfreeman » Thu Sep 12, 2013 1:52 pm

Hello tilllt,

I have no need or desire to play with Tahoe-LAFS's pub net. Been there, done that, they do not even offer a t-shirt. Note that from the beginning of my involvement in this thread, I have had generally negative input. I do not really like the ideas posed here, but, my mind is open enough to see if you (those who do like it) can provide a compelling reason to revisit the use of distributed file systems.

The combination of the two projects;

ownCloud:
An application used to limit and track the flow of files within a group or organization.

Tahoe-LAFS:
An application used to create a distributed file / storage system that can make use of currently wasted storage and network resources.

May be useful in some ways, but, it does not create any 'new' usage possibilities and the competing file usage (one wants to track files and one wants to ensure that files cannot be tracked) may even make the combination detrimental to both parts.

From my desire to decipher what is actually wanted as an end result here; Is it possible that what is wanted is a decentralized networking model that makes it possible to move encrypted data around the Internet (or a subset thereof), without depending on a centralized infrastructure. Such as: https://projectmeshnet.org/
srfreeman
OwnCloud professional
 
Posts: 1625
Joined: Sun Apr 21, 2013 10:34 am
Location: US
ownCloud version: 6.0.4
Webserver: Apache
Database: MySQL
OS: Linux
PHP version: 5.3.10

Re: Tahoe-LAFS integration idea

Postby tilllt » Thu Sep 12, 2013 3:05 pm

srfreeman,

sorry man, but i really cannot follow your line of thought anymore. It is quite obvious why Tahoe-LAFS (a decentralized & secure storage grid) and Owncloud - a promising but still half baked - service / gui for storing / retrieving / versioning files, could perfectly make sense as a combination, once security related problems can be overcome. It doesnt matter if this combination would be run in a decentralized darknet or on the internet. I don't see Tahoe as a replacement for anonymous file sharing, like Freenet - and from what read of the devs of Tahoe-LAFS, this absolutely not what they are aiming for. The point of Tahoe is NOT "to ensure that files cannot be tracked", but to ensure "files cannot be tracked by anyone else then you - or the people that YOU share the decryption keys" with.

The intermediate goal of this combination would be a replacement of a service like Dropbox - i guess that was obvious from the beginning. For the reasons mentioned earlier, you can have no insight on what is happening at companies like Dropbox, securitywise. They are acting under US legislation and what this means has been partly revealed by Edward Snowdens whistleblowing efforts. So basically, putting your files on Dropbox most probably means sharing your files with US authorities as well.

Owncloud on the other hand, on a superficial level seems to provide a replacement to Dropbox: You throw your files in a folder and they get synced with a server (and they are versioned). On the other hand, Owncloud by itself is not at all providing the same type of service as Dropbox: Unless you take care of it yourself, there is no backup and the security features are probably much weaker than even what a company like Dropbox is providing.

So, whats so difficult in seeing the strong advantage of this combination? Once Tahoe-LAFS and Owncloud would be smoothly integrated, the shortcomings of both pieces of software would be minimized and the goal of having "more trustable" replacement for a service like Dropbox (or Sugarsync etc. for that matter) could come closer.

And there is more: Firefox Sync is based on the concept of Tahoe-LAFS ( http://docs.services.mozilla.com/sync/overview.html, https://tahoe-lafs.org/trac/tahoe-lafs/ ... edProjects ) so even when not backing up the "big data" part of Owncloud, i.e. the files from sync manager, it could be also very useful to backup the database at least - like Firefox sync does it. So plenty of reasons why this could be a winning combination in my eyes.






So none of your remarks actually provided constructive feedback to the OP's original question.
tilllt
Starter
 
Posts: 66
Joined: Fri Jul 06, 2012 9:59 am

Re: Tahoe-LAFS integration idea

Postby anaqreon » Thu Sep 12, 2013 3:17 pm

Thanks for the link about Project Meshnet. It looks like an interesting idea I'm sure we'll have fun exploring.

Some meta-discussion (skip to final paragraph for technical info):

You are correct, you have generally been a Negative Nancy (perhaps Debbie Downer?), but that's okay. I think I understand that the primary reason is your perspective. Your description of ownCloud as
an application used to limit and track the flow of files within a group or organization
is not at all how I view the purpose of the project, simply because I am not using it in a business context. From a business perspective, ownCloud is promising as a competitor to services like Dropbox and Google Drive, which helps the business regain control over their data and perhaps even enhance their ability to track and monitor the activity of employees. This is why ownCloud has a commercial side through which I hope they will make massive profits from their efforts.

In contrast, my description of ownCloud would be something more like "an application that provides me personal cloud services in a way that gives me control over data access and protects my privacy". I use it to keep my family's calendar and contact information synced between our phones and devices. I use it to have access to my files from anywhere. I use it to easily share photos and videos with people and provide them a (sufficiently) secure way to upload files they want to share with me.

Back to the technical issues:

I was able to use the SFTP external storage type to access my grid yesterday directly, without sshfs or any local folder involvement. The problem was that I could only view and download existing files. When I tried to create new ones, ownCloud would just hang and I actually had to restart Apache to access it again. The URI for the folder I specified in the "accounts" file in the Tahoe node "private" folder is a read/write URI, not the read-only, so that shouldn't be the problem. To be fair, ownCloud's interface displayed a red box next to the external storage configuration instead of the green circle. I guess I'll have to look through the log files to see if there are clues about what happened.
anaqreon
Beginner
 
Posts: 36
Joined: Mon Mar 26, 2012 5:15 pm

Re: Tahoe-LAFS integration idea

Postby srfreeman » Thu Sep 12, 2013 4:24 pm

Hello anaqreon,

The red box in the ownCloud display for the SFTP connector, usually signifies a problem with the SFTP (ssh2) implementation in PHP, which can be different for various OS's and web servers. I find that getting an SFTP server running on the system serving ownCloud seems to be a prerequisite to getting the SFTP connector working.

Beyond that, the SFTP implementation in Tahoe-LAFS should / could work. There may still be some issues surrounding path length and file / folder naming that may have to be addressed in the ownCloud database. The need for ownCloud to retain exclusive use of the storage location may become an issue. The web server owning the mount location may also give Tahoe-LAFS some heart burn.
srfreeman
OwnCloud professional
 
Posts: 1625
Joined: Sun Apr 21, 2013 10:34 am
Location: US
ownCloud version: 6.0.4
Webserver: Apache
Database: MySQL
OS: Linux
PHP version: 5.3.10

Re: Tahoe-LAFS integration idea

Postby anaqreon » Thu Sep 12, 2013 5:47 pm

I didn't check the SFTP server directly with 'get' and 'put' kinds of commands, but I did have sshfs successfully connect to SFTP server and mount that local folder. Since ownCloud had no trouble reading and writing to that folder, I assume that means the SFTP server must be capable of executing all those operations. I did double check through the Tahoe WebUI that the files were indeed being changed in the grid and that it wasn't just some internal ownCloud thing. That means that the Tahoe node SFTP server is fine, but that as you suggested perhaps the PHP connections to it are failing.

If that is true, then the fault might lie in the ownCloud code compatibility with varieties of SFTP servers or something. I get rapidly out of my depth with that stuff. Is it worth posting as an issue for the developers to consider?
anaqreon
Beginner
 
Posts: 36
Joined: Mon Mar 26, 2012 5:15 pm

Re: Tahoe-LAFS integration idea

Postby srfreeman » Thu Sep 12, 2013 8:08 pm

Hello anaqreon,

Why would you consider contacting the ownCloud team about PHP configuration issues? The ownCloud code has nothing to do with this.

Remember, once you get ownCloud connected, you are done with the Tahoe-LAFS interfaces for affecting the files in the attached directory. Per the ownCloud documentation, ownCloud must retain exclusive control of the storage location. Problems with ignoring this rule are well documented in many places on this board.
srfreeman
OwnCloud professional
 
Posts: 1625
Joined: Sun Apr 21, 2013 10:34 am
Location: US
ownCloud version: 6.0.4
Webserver: Apache
Database: MySQL
OS: Linux
PHP version: 5.3.10

Re: Tahoe-LAFS integration idea

Postby tilllt » Thu Sep 12, 2013 8:15 pm

@anaqreon:


did you read this, maybe there are some hints
https://tahoe-lafs.org/trac/tahoe-lafs/ ... tpFrontend

Thats another reason why using the Tahoe API would be very useful because then it would be possible to integrate progress indicators directly in owncloud, which does not seem possible via SFTP

General compatibility issues

Before uploading a file to a Tahoe filesystem, the whole file has to be available. This means that the upload can only start when the file has been closed in the SFTP session. Particularly when writing large files, the client may time out between sending the close request and receiving the response (ticket #1041). This is known to be a problem for at least the WinSCP client, which has a default close timeout of 15 seconds. In the case of WinSCP this can be worked around by setting WinSCP -> Connection -> Timeouts to 6000 seconds (the maximum allowed); other clients with this problem may have similar settings.

In the period after the close but before the upload has finished, the closed file may not appear in directory listings, or may appear with an incorrect modification time.

Since Tahoe uses capability access control rather than Unix-style permissions, the permission bits seen by SFTP clients are only an approximation chosen to avoid confusing client programs. In particular the 'user', 'group' and 'world' permissions on a Tahoe file will always be the same. It is possible to clear all of the 'w' bits on a file, which will prevent that file from being opened for writing, but note that its directory entry can still be replaced via a write cap to the directory.

See the last section of ​docs/frontends/FTP-and-SFTP.rst for information on how the SFTP frontend treats immutable and mutable files.

Deleting a directory via the SFTP frontend will not check that it is empty. The directory will be unlinked from its parent, but its contents will remain accessible via any other capabilities to it.

The 'ctime' and 'mtime' attributes will always be the same, and are set from the Tahoe linkmotime timestamp, which is changed only when the link from the parent directory is modified (see the ​'About the metadata' section of webapi.rst). These fields are not updated when the contents of a mutable file are changed. The SFTP protocol and the server are able to represent dates up to the year 2106, but some clients may print dates incorrectly after 2037.

Versions of Twisted before v11.0 have ​a bug in support for rekeying. Tahoe-LAFS v1.10.0 requires a later version of Twisted and so will not be affected by this problem. For earlier versions of Tahoe-LAFS, it is recommended to install a version of Twisted between v11.0 and v12.2 inclusive (and use tahoe --version to check that it is being used); otherwise, this bug might cause a hang or 100% CPU usage by the gateway when a client tries to rekey. Depending on the client, rekeying may be triggered based on a time interval or the amount of data sent (for example, 1 GiB to 4 GiB for the openssh client), so this problem may not happen immediately when testing.

Version v12.3 of Twisted is incompatible with the SFTP frontend (#1926, #1525). Upgrade to Tahoe-LAFS v1.10.0 to fix this problem.
tilllt
Starter
 
Posts: 66
Joined: Fri Jul 06, 2012 9:59 am

PreviousNext

Return to ownCloud Community Edition 5.x

Who is online

Users browsing this forum: No registered users and 17 guests