Instead of calling netstat locally to find out available ports in a
tight range, call bind() with port zero to find an available ephemeral
port. This is faster and much simpler, and will eventually help
getting rid of PortManager.
Also fixes issues with running SSH commands on Windows when the remote
shell is Powershell (aka Backslash Bingo).
There are no real changes, just moving files around. Sockets will be
used in the future to find available ports in cdc_stream. Therefore,
they need to be in common.
Instead of calling netstat on the remote device to detect available
ports, simply call bind with port 0 to bind to any available port.
Since the port is not yet known when cdc_rsync_server.exe is called,
port forwarding needs to be started AFTER the server reports its port.
Fixes an issue with port forwarding when localhost on the remote
system maps to the IPV6 localhost. In that case the server would time
out on accept() since it creates an IPV4 socket, so the connection
is never established.
Also allows passing in port 0, so that it will auto-detect an
available port. This will be used in a future CL to remove the
necessity of running netstat/ss.
On Windows, fclose() seems to be very expensive for large files, where
closing a 1 GB file takes up to 5 seconds. This CL calls fclose() in
background threads. This tremendously improves local syncs, e.g.
copying a 4.5 GB, 300 files data set takes only 7 seconds instead of
30 seconds.
Also increases the buffer size for copying from 16K to 128K (better
throughput for local copies), and adds a timestamp to debug and
verbose console logs (useful when comparing client and server logs).
Build id is an optional unique identifier specified during cdc_rsync build via CDC_BUILD_VERSION definition.
If build id specified on both client and server components it will be used to check the version of server component instead of file size + modified time.
Adds a ServerArch class whose job it is to encapsulate differences
between Windows and Linux cdc_rsync_servers. It detects the type
based on a heuristic in the destination path. This is not fool proof
and will probably require further work, like falling back to the other
type if the detected one doesn't work.
Uses the ServerArch class to determine the different commands to start
the server and to deploy the server.
Note that the functionality is not well tested on Windows yet, but
copying plain files works.
Fixes an issue in UnzstdStream where the Read() method always tries to
read new input data if no input data is available, instead of first
trying to uncompress. Since zstd maintains internal buffers,
uncompression might succeed even without reading more input, so this
is faster. This bug can lead to pipeline stalls in cdc_rsync.
Makes ServerSocket multi-platform, mainly by working around some small
API differences. The code is largely the same, there should be no
differences on Linux.
Also moves WSAStartup() and WSACleanup() up to the Socket level as
static methods because it's used by both ClientSocket and ServerSocket,
and because it doesn't make sense to do that in the socket class as
that would prevent one from using several sockets.
Expand path variables for sync destination
Running commands like cdc_rsync C:\assets\* host:~/assets -vr would create a directory called ~assets. This CL expands path variables properly.
Implements the cdc_stream client and adjusts asset streaming in
various places to work better outside of a GGP environment.
This CL tries to get quoting for SSH commands right. It also brings
back the ability to start a streaming session from
asset_stream_manager.
Also cleans up Bazel targets setup. Since the sln file is now in root,
it is no longer necessary to prepend ../ to relative filenames to
make clicking on errors work.
* Remove dependencies of cdc_sync from GGP
Allows overriding the SSH and SCP commands via command line flags.
Hence, strict host checking, SSH config etc. can be removed since it
is passed in by command line flags for GGP. Also deploys
cdc_rsync_server to ~/.cache/cdc_file_transfer/ and creates that dir
if it does not exist.
* Tweak RemoteUtil
Replaces localhost: by //./ in the workaround for scp since localhost:
had two disadvantages: 1) It required 2 gnubby touches for gLinux and
2) it didn't work for ggp. //./ works for both. Also tweaks quoting,
which didn't quite work for ggp.
* Don't check remote ports in cdc_rsync
Turns off checking remote ports in PortManager. In the future, the
server should return available ports after failing to connect to the
provided port.
Since now the first remote connection is running cdc_rsync_server,
the timeout check has to be done when running that process.
* Remove now-unused kInstancePickerNotAvailableInQuietMode enum
* Add more details to the readme
* [cdc_rsync] Accept [user@]host:destination
Removes the --ip command line argument and assumes user/host are
passed in along with the destination, so it works in the same way as
other popular tools.
* [ggp_rsync] Combine server deploy commands
Combines two chmod and one mv command into one ssh command. This makes
deploy a bit quicker, especially if each ssh command involves touching
your gnubby.
* Remove GGP specific stuff from VS build commands
* [cdc_rsync] Get rid of cdc_rsync.dll
Compile the CDC RSync client as a static library instead. This removes
quite a bit of boiler plate and makes string handling easier since
we can now pass std::strings instead of const chars.
Also fixes an issue where we were sometimes trying to assign nullptr
to std::strings, which is forbidden.
* Allow specifying ssh/scp commands with env vars
* Rename GgpRsync* to CdcRsync*
* Merge ggp_rsync_cli into ggp_rsync
* [cdc_rsync] Refactor cdc_rsync.cc/h
Merges cdc_rsync.cc/h with main.cc and CdcRsyncClient since code is
closer to where it's being used and should be more readable.
The tools allow efficient and fast synchronization of large directory
trees from a Windows workstation to a Linux target machine.
cdc_rsync* support efficient copy of files by using content-defined
chunking (CDC) to identify chunks within files that can be reused.
asset_stream_manager + cdc_fuse_fs support efficient streaming of a
local directory to a remote virtual file system based on FUSE. It also
employs CDC to identify and reuse unchanged data chunks.
