On Windows, fclose() seems to be very expensive for large files, where
closing a 1 GB file takes up to 5 seconds. This CL calls fclose() in
background threads. This tremendously improves local syncs, e.g.
copying a 4.5 GB, 300 files data set takes only 7 seconds instead of
30 seconds.
Also increases the buffer size for copying from 16K to 128K (better
throughput for local copies), and adds a timestamp to debug and
verbose console logs (useful when comparing client and server logs).
Adds a function to filter ANSI escape sequences from a string.
Executing SSH commands on Windows yields output that is full of ANSI
escape sequences if the "-tt" (forced TTY) argument is used. One
particular escape sequence sets the window title to
"c:\windows\system32\cmd.exe". This string is null terminated and
messes with parsing the actual output later in that string.
The filter function removes those escape sequences.
The outout is still a bit messed up, even after removing escape
sequences. Some sequences delete rows and move the cursor. Without
properly interpreting these sequences it doesn't seem possible to
retrieve the proper output.
In a future CL the -tt argument is removed on Windows, which removes
the necessity to filter ANSI codes. However, sometimes the target
architecture is not known (yet), so that it is still useful to filter
ANSI codes in that case to print useful debug output.
Adds a flag to set the SSH forwarding port or port range used for
'cdc_stream start-service' and 'cdc_rsync'.
If a single number is passed, e.g. --forward-port 12345, then this
port is used without checking availability of local and remote ports.
If the port is taken, this results in an error when trying to connect.
Note that this restricts the number of connections that stream can
make to one.
If a range is passed, e.g. --forward-port 45000-46000, the tools
search for available ports locally and remotely in that range. This is
more robust, but a bit slower due to the extra overhead.
Optimizes port_manager_win as it was very slow for a large port range.
It's still not optimal, but the time needed to scan 30k ports is
<< 1 seconds now.
Fixes#12
Fixes a couple of issues with the FUSE:
- Creates the mount directory if it does not exist.
This assumes the mount dir to be the last arg. Ideally, we'd parse the
command line and then create the directory, but unfortunately
fuse_parse_cmdline already verifies that the dir exists.
- Expands the cache_dir (e.g. ~).
- Fixes a compile issue in manifest_iterator.
Improve cdc_fuse_fs and path
Improves the error handling in path so that std:error_codes are not
assumed to be of system category, and also that their messages are
displayed. Also improves debug messages in GameletComponent.
The tools allow efficient and fast synchronization of large directory
trees from a Windows workstation to a Linux target machine.
cdc_rsync* support efficient copy of files by using content-defined
chunking (CDC) to identify chunks within files that can be reused.
asset_stream_manager + cdc_fuse_fs support efficient streaming of a
local directory to a remote virtual file system based on FUSE. It also
employs CDC to identify and reuse unchanged data chunks.
