[cdc_rsync] Enable local syncing (#75)

Adds support for local syncs of files and folders on the same Windows
machine, e.g. cdc_rsync C:\source C:\dest. The two main changes are

- Skip the check whether the port is available remotely with PortManager.
- Do not deploy cdc_rsync_server.
- Run cdc_rsync_server directly, not through an SSH tunnel.

The current implementation is not optimal as it starts
cdc_rsync_server as a separate process and communicates to it via a
TCP port.
This commit is contained in:
Lutz Justen
2023-01-26 09:57:19 +01:00
committed by GitHub
parent 9cf71cae65
commit f8c10ce7bd
12 changed files with 168 additions and 73 deletions

View File

@@ -1,8 +1,8 @@
# CDC File Transfer
Born from the ashes of Stadia, this repository contains tools for syncing and
streaming files from Windows to Linux. They are based on Content Defined
Chunking (CDC), in particular
streaming files from Windows to Windows or Linux. The tools are based on Content
Defined Chunking (CDC), in particular
[FastCDC](https://www.usenix.org/conference/atc16/technical-sessions/presentation/xia),
to split up files into chunks.
@@ -132,9 +132,9 @@ difference operation. It does not involve a per-byte hash map lookup.
## CDC Stream
`cdc_stream` is a tool to stream files and directories from a Windows machine to a
Linux device. Conceptually, it is similar to [sshfs](https://github.com/libfuse/sshfs),
but it is optimized for read speed.
`cdc_stream` is a tool to stream files and directories from a Windows machine to
a Linux device. Conceptually, it is similar to
[sshfs](https://github.com/libfuse/sshfs), but it is optimized for read speed.
* It caches streamed data on the Linux device.
* If a file is re-read on Linux after it changed on Windows, only the
differences are streamed again. The rest is read from the cache.
@@ -161,6 +161,34 @@ In one case, the game is streamed via `sshfs`, in the other case we use
<img src="docs/cdc_stream_vs_sshfs.png" alt="Comparison of cdc_stream and sshfs" width="752" />
</p>
# Supported Platforms
| `cdc_rsync` | From | To |
|:-----------------------------|:--------------------:|:--------------------:|
| Windows x86_64 | &check; | &check; <sup>1</sup> |
| Ubuntu 22.04 x86_64 | &cross; <sup>2</sup> | &check; |
| Ubuntu 22.04 aarch64 | &cross; | &cross; |
| macOS 13 x86_64 <sup>3</sup> | &cross; | &cross; |
| macOS 13 aarch64 <sup>3</sup>| &cross; | &cross; |
| `cdc_stream` | From | To |
|:-----------------------------|:--------------------:|:--------------------:|
| Windows x86_64 | &check; | &cross; |
| Ubuntu 22.04 x86_64 | &cross; | &check; |
| Ubuntu 22.04 aarch64 | &cross; | &cross; |
| macOS 13 x86_64 <sup>3</sup> | &cross; | &cross; |
| macOS 13 aarch64 <sup>3</sup>| &cross; | &cross; |
<span style="font-size: 0.8rem">
<sup>1</sup> Only local syncs, e.g. `cdc_rsync C:\src\* C:\dst`. Support for
remote syncs is being added, see
[#61](https://github.com/google/cdc-file-transfer/issues/61).
<sup>2</sup> See [#56](https://github.com/google/cdc-file-transfer/issues/56).
<sup>3</sup> See [#62](https://github.com/google/cdc-file-transfer/issues/62).
</span>
# Getting Started
Download the precompiled binaries from the
@@ -190,7 +218,7 @@ To build the tools from source, the following steps have to be executed on
git submodule update --init --recursive
```
Finally, install an SSH client on the Windows device if not present.
Finally, install an SSH client on the Windows machine if not present.
The file transfer tools require `ssh.exe` and `sftp.exe`.
## Building
@@ -304,6 +332,10 @@ To get per file progress, add `-v`:
```
cdc_rsync C:\path\to\assets\* user@linux.device.com:~/assets -vr
```
The tool also supports local syncs:
```
cdc_rsync C:\path\to\assets\* C:\path\to\destination -vr
```
### CDC Stream