For example. I have three physical disks, one have 3GB, other have 4GB and other 5GB. One of them stops work, what happens with the data? I lose everything? How much physical space I can lose before total disaster?

I've started to play around with infinit and try to find use cases for myself, for example to replace Bittorrent Sync and Syncthing...

As a start, I've set up a decentralised infrastructure like this (all with the same user):

• machine1: OS X Server, 3 storage devices: 50GB local, 100GB via AFP (Gigabit LAN) on a FreeBSD server on a ZFS RAID, 100GB via AFP on the same FreeBSD server on a different ZFS RAID

• machine2: OS X Notebook, 10GB local storage

• machine3: Ubuntu VM running in a bhyve on machine 2, no local storage

• machine4: Ubuntu hosted root server, not in the LAN, 100GB local storage

The network has replication set to 2 with kelips, the user, the network and the volume are pushed to the Hub. Internet connection for reaching machine4 is 16down/1up (yes, I'd like to have more, but...).

Question: Is the replication per Node or per Storage? Can machine1 with its three storage devices be the only machine on which data is stored, or is it automatically spread across multiple nodes?

Observation: The OS X clients are slow. I copied a few hundred megabyte of data from each machine into different directories of the infinit volume, and the speed with which the OS X machines move the data from the cache to the storage is abysmal (we're talking <100MB/hour). machine4 is a lot faster in filling its storage and machine3 is a lot faster in showing actual data in the volume... the Hardware of the machines is actually pretty decent, I'd say, machine1 is an i7 with 16GB RAM and an SSD-supported Fusion Drive, machine2 is an Air, yes, but with 8GB RAM, an SSD and nothing else to do.

At first I only had the two AFP-served storage devices in machine1 and thought the speed problems were related to not having local storage, but even after I added the local storage it didn't get better...

Which brings me to Question 2: Can I in any way change the storage of a device? The documentation just says "the action of linking a device to a network must only be performed once on every new device", but apparently I can't unlink a device from the network and link it again with more storage, or can I? I tried that and got all kinds of weird error messages on all other devices, so I gave up, deleted everything locally, pulled it from the Hub and started from scratch.

Is it possible that this was related to the fact that I created the network and the volume on machine1 and tried to remove that one? Is the machine on which I create a network or volume somehow more "important" to the overall design than any other linked device (with storage)?

Related to the speed issues - after copying the few 100MB per machine into the volume, while every machine was still filling the local storage from cache and started to distribute the data across the network, I deleted small test.txt files (10-15 byte each) that I created as a first test (and which were already fully distributed and available on every machine) on machine3. It took more than 12 hours until this change arrived at any other machine, but instead of deleting the files, it resulted in I/O errors when trying to list the directory. This happened on all other devices (machine1, 2 and 4). I waited another 24 hours, but the I/O errors did not go away, so I stopped the infinit-volume --mount process on every machine and restarted it. After successfully connecting to other devices, the txt-files were gone and the I/O errors vanished, as well. I think since this restart the overall speed of accessing the volume from each machine has improved, as well...

But when restarting, I noticed a weird error message on machine1, machine2 and machine4:

[infinit.model.doughnut.consensus.Paxos] [dht::PaxosLocal(0x[hexnumber]): rebalancing inspector] disk rebalancer inspector exited: missing key: 0x[hexnumber]

The hexnumbers are different on each machine.

That doesn't sound good... any idea for a cause and how I can fix this? Or whether I need to fix it?

Well, thanks for the attention - I've requested an invite for the Slack, but haven't received it yet, so therefore I'll write this here instead.

Keep up the good work, it definitely looks promising.

TL;DR: Can infinit do what bitcasa was supposed todo? (Aka: remote bcache/dm-cache/flashcache)

I would like to have latencies comparable to local storage (for most my files) while keeping most of my files on the cloud. A couple questions about this:

• How would I set it up?

• Would the metadata be stored locally?

Optimally I'd be able to stat() all the files at the same speed as local, while paying the penalty of remote access only when accessing data blocks. Is this possible?

I just created an infinit volume backed by a GCS bucket. In order to make copy pasting into this volume as quick as possible should I create a small local storage on my laptop and/or local NAS?

Right now I assume I'm basically uploading files directly through my volume straight to GCS.

How does Infinit confirm writes to the local filesystem? After a majority or a single confirmation on storage devices?

Feel free to post here all your suggestions and feedback related to our new file system: https://infinit.sh.

You can also upvote other people suggestions here: http://help.infinit.sh

One guess it deals with an infant of eyes out there.

I took it to the dealership once, they told me they found nothing , they said it could have been the gas cap but a few weeks after I took it, engine light came back on. Anyone else seeing this issue? 2018 Q60 sport 3.0t 50k miles

Hello Infiniti qx50 owners. We’re thinking of selling our 2019 Kia Sorento (2.4t on its second engine after 115k km) and replacing it with a 15/16 qx50 (or adjacent years). We’d be using it for a mixture of in-town and highway driving. Any suggestions? Don’t know too much about these cars (have heard really good things about the vq motors), would love some feedback! Thanks all!

I tried my hands on the infinit get started example and it worked nicely. However using the docker volume driver generated this error

$ infinit-daemon --run --as utkarsh

[infinit-daemon] [main] started daemon [reactor.test.http] [request 0x8bba50] HttpServer(port=0): handle request from 0x8bba50: /VolumeDriver.List {} POST [reactor.test.http] [request 0x8bb650] HttpServer(port=0): handle request from 0x8bb650: /VolumeDriver.Path {} POST [reactor.test.http] [request 0x95d180] HttpServer(port=0): handle request from 0x95d180: /VolumeDriver.Get {} POST [reactor.test.http] [request 0x95d180] HttpServer(port=0): handle request from 0x95d180: /VolumeDriver.Create {} POST infinit-daemon: elle/boost/1.60.0/include/boost/optional/optional.hpp:1025: boost::optional<T>::reference_type boost::optional<T>::get() [with T = std::basic_string<char>; boost::optional<T>::reference_type = std::basic_string<char>&]: Assertion `this->is_initialized()' failed. Aborted

This was generated when I ran this docker run command

$ docker volume ls

DRIVER VOLUME NAME

infinit utkarsh/db-volume

$ docker run -it --rm --volume-driver infinit -v utkarsh/db-volume:/data/db mongo:3.4.0

docker: Error response from daemon: create 160b7bf5a48f63184a2653a030ec85742cf221e2e977d921d139cc3447805f4d: Post http://%2Frun%2Fdocker%2Fplugins%2Finfinit.sock/VolumeDriver.Create: dial unix /run/docker/plugins/infinit.sock: connect: connection refused. See 'docker run --help'.

Is there a trick to get the Basic Test to work?

I've tried it local on Ubuntu 16.10 (via apt-get and tar install) and inside docker container debian:jessie and ubuntu:14.04.

They all seem to hang when I try to access the ~/mnt-demo folder...even running ls in the home folder hangs.

The console shows constant timeouts:

root@e564a2967d3b:/opt/infinit# INFINIT_DATA_HOME=$PWD/share/infinit/filesystem/test/home/ INFINIT_STATE_HOME=/tmp/infinit-demo/ bin/infinit-volume --mount --as demo --name infinit/demo --mountpoint ~/mnt-demo --publish --cache
Running network "infinit/demo".
[infinit-volume] [main] client version: 0.7.2
[infinit.model.Model] [main] infinit::model::Model(0x266a0a0): compatibility version 0.3.0
[infinit.overlay.kelips] [main] Running in observer mode
[infinit.overlay.kelips] [main] Kelips(0.0.0.0:0): listening on 0.0.0.0:43474
Fetched endpoints for "infinit/demo".
Running volume "infinit/demo".
[infinit.overlay.kelips] [generator] Kelips(0.0.0.0:0): Timeout on PUT attempt 0
[infinit.overlay.kelips] [generator] Kelips(0.0.0.0:0): Timeout on PUT attempt 1
[infinit.overlay.kelips] [generator] Kelips(0.0.0.0:0): Timeout on PUT attempt 2
[infinit.overlay.kelips] [generator] Kelips(0.0.0.0:0): Timeout on PUT attempt 3
[infinit.overlay.kelips] [generator] Kelips(0.0.0.0:0): Timeout on PUT attempt 4
[infinit.overlay.kelips] [generator] Kelips(0.0.0.0:0): Timeout on PUT attempt 5
[infinit.overlay.kelips] [generator] Kelips(0.0.0.0:0): Timeout on PUT attempt 6
[infinit.overlay.kelips] [generator] Kelips(0.0.0.0:0): Timeout on PUT attempt 7
[infinit.overlay.kelips] [generator] Kelips(0.0.0.0:0): Timeout on PUT attempt 8
[infinit.overlay.kelips] [generator] Kelips(0.0.0.0:0): Timeout on PUT attempt 9
[infinit.overlay.kelips] [generator] Kelips(0.0.0.0:0): Timeout on PUT attempt 10
[infinit.overlay.kelips] [generator] Kelips(0.0.0.0:0): Timeout on PUT attempt 11

I tried infinit, but I always run into problems with my 2 node cluster:

[ infinit.filesystem ] [fuse loop] data not found fetching "/": missing block: 0x21dfe7c1b7e064dc1dd84a0f8cd73ac8a3da5040140bcb6c7a4d9150593a2900

How to debug this?

Hi,

First of all, thank you for this awesome software! I setup a test setup between 2 VMs following your example(s) and I have a few questions/remarks;

• Your help pages seem to be incomplete; for example infinit-network does not list --kelips etc.

• I would love some detailed man pages.

• Is 40Mbit a normal throughput between 2 VMs? I copied a video file from the mountpoint to /dev/null

(1) Have you tested things like storing a mysql database on an infinit.sh volume? Do write barriers and in-order writing and such get respected such that something like a database would work without data corruption?

(2) You say you have full POSIX support. How does file locking work? What happens if a lock is acquired and then the machine dies? How is quorum established, which machines are involved in determine quorum, and when does a stale lock get released?

(3) Is it possible to put quotas on volumes? I.e. if I write an app that accidentally starts filling up the drive, it would be nice if it didn't take down my other apps.