Uses the abstractions written for ServerSocket in ClientSocket, so
that it builds on Linux. Also adds a method to poll for connections
and uses that in cdc_rsync. Similar code will be used in cdc_fuse_fs
to wait for a connection in a future CL.
There are no real changes, just moving files around. Sockets will be
used in the future to find available ports in cdc_stream. Therefore,
they need to be in common.
Improves ServerArch so that it can detect the remote architecture by
running uname and checking %PROCESSOR_ARCHITECTURE%. So far, only
x64 Linux and x64 Windows are supported, but in the future it is easy
to add support for others, e.g. aarch64, as well.
Before the detection is run, the remote architecture is guessed first
based on the destination. For instance, if the destination directory
starts with "C:\", it pretty much means Windows. If cdc_rsync_server
exists and runs fine, there's no need for detection.
Since also PortManager depends on the remote architecture, it has to
be adjusted as well. So far, PortManager assumeed that "local" means
Windows and "remote" means Linux. This is no longer the case for
syncing to Windows devices, so this CL adds the necessary abstractions
to PortManager.
Also refactors ArchType into a separate class in common, since it is
used now from several places. It is also expanded to handle future
changes that add support for different processor architectures, e.g.
aarch64.
On Windows, fclose() seems to be very expensive for large files, where
closing a 1 GB file takes up to 5 seconds. This CL calls fclose() in
background threads. This tremendously improves local syncs, e.g.
copying a 4.5 GB, 300 files data set takes only 7 seconds instead of
30 seconds.
Also increases the buffer size for copying from 16K to 128K (better
throughput for local copies), and adds a timestamp to debug and
verbose console logs (useful when comparing client and server logs).
Build id is an optional unique identifier specified during cdc_rsync build via CDC_BUILD_VERSION definition.
If build id specified on both client and server components it will be used to check the version of server component instead of file size + modified time.
Adds a function to filter ANSI escape sequences from a string.
Executing SSH commands on Windows yields output that is full of ANSI
escape sequences if the "-tt" (forced TTY) argument is used. One
particular escape sequence sets the window title to
"c:\windows\system32\cmd.exe". This string is null terminated and
messes with parsing the actual output later in that string.
The filter function removes those escape sequences.
The outout is still a bit messed up, even after removing escape
sequences. Some sequences delete rows and move the cursor. Without
properly interpreting these sequences it doesn't seem possible to
retrieve the proper output.
In a future CL the -tt argument is removed on Windows, which removes
the necessity to filter ANSI codes. However, sometimes the target
architecture is not known (yet), so that it is still useful to filter
ANSI codes in that case to print useful debug output.
Adds a flag to set the SSH forwarding port or port range used for
'cdc_stream start-service' and 'cdc_rsync'.
If a single number is passed, e.g. --forward-port 12345, then this
port is used without checking availability of local and remote ports.
If the port is taken, this results in an error when trying to connect.
Note that this restricts the number of connections that stream can
make to one.
If a range is passed, e.g. --forward-port 45000-46000, the tools
search for available ports locally and remotely in that range. This is
more robust, but a bit slower due to the extra overhead.
Optimizes port_manager_win as it was very slow for a large port range.
It's still not optimal, but the time needed to scan 30k ports is
<< 1 seconds now.
Fixes#12
Fixes a couple of issues with the FUSE:
- Creates the mount directory if it does not exist.
This assumes the mount dir to be the last arg. Ideally, we'd parse the
command line and then create the directory, but unfortunately
fuse_parse_cmdline already verifies that the dir exists.
- Expands the cache_dir (e.g. ~).
- Fixes a compile issue in manifest_iterator.
Improve cdc_fuse_fs and path
Improves the error handling in path so that std:error_codes are not
assumed to be of system category, and also that their messages are
displayed. Also improves debug messages in GameletComponent.
The tools allow efficient and fast synchronization of large directory
trees from a Windows workstation to a Linux target machine.
cdc_rsync* support efficient copy of files by using content-defined
chunking (CDC) to identify chunks within files that can be reused.
asset_stream_manager + cdc_fuse_fs support efficient streaming of a
local directory to a remote virtual file system based on FUSE. It also
employs CDC to identify and reuse unchanged data chunks.
