Remove GGP dependencies from CDC RSync (#1)

* Remove dependencies of cdc_sync from GGP

Allows overriding the SSH and SCP commands via command line flags.
Hence, strict host checking, SSH config etc. can be removed since it
is passed in by command line flags for GGP. Also deploys
cdc_rsync_server to ~/.cache/cdc_file_transfer/ and creates that dir
if it does not exist.

* Tweak RemoteUtil

Replaces localhost: by //./ in the workaround for scp since localhost:
had two disadvantages: 1) It required 2 gnubby touches for gLinux and
2) it didn't work for ggp. //./ works for both. Also tweaks quoting,
which didn't quite work for ggp.

* Don't check remote ports in cdc_rsync

Turns off checking remote ports in PortManager. In the future, the
server should return available ports after failing to connect to the
provided port.

Since now the first remote connection is running cdc_rsync_server,
the timeout check has to be done when running that process.

* Remove now-unused kInstancePickerNotAvailableInQuietMode enum

* Add more details to the readme

* [cdc_rsync] Accept [user@]host:destination

Removes the --ip command line argument and assumes user/host are
passed in along with the destination, so it works in the same way as
other popular tools.

* [ggp_rsync] Combine server deploy commands

Combines two chmod and one mv command into one ssh command. This makes
deploy a bit quicker, especially if each ssh command involves touching
your gnubby.

* Remove GGP specific stuff from VS build commands

* [cdc_rsync] Get rid of cdc_rsync.dll

Compile the CDC RSync client as a static library instead. This removes
quite a bit of boiler plate and makes string handling easier since
we can now pass std::strings instead of const chars.

Also fixes an issue where we were sometimes trying to assign nullptr
to std::strings, which is forbidden.

* Allow specifying ssh/scp commands with env vars

* Rename GgpRsync* to CdcRsync*

* Merge ggp_rsync_cli into ggp_rsync

* [cdc_rsync] Refactor cdc_rsync.cc/h

Merges cdc_rsync.cc/h with main.cc and CdcRsyncClient since code is
closer to where it's being used and should be more readable.
This commit is contained in:
ljusten
2022-11-15 12:48:09 +01:00
committed by GitHub
parent 4326e972ac
commit 9fdccb3548
43 changed files with 952 additions and 1120 deletions

View File

@@ -32,8 +32,7 @@
<BazelSedCommand>| sed -r &quot;s/^([^:\(]+[:\(][[:digit:]]+(,[[:digit:]]+)?[:\)])/$(BazelSourcePathPrefix)\\1/&quot;</BazelSedCommand> <BazelSedCommand>| sed -r &quot;s/^([^:\(]+[:\(][[:digit:]]+(,[[:digit:]]+)?[:\)])/$(BazelSourcePathPrefix)\\1/&quot;</BazelSedCommand>
<!-- Clang prints errors to stderr, so pipe it into stdout. --> <!-- Clang prints errors to stderr, so pipe it into stdout. -->
<BazelSedCommand Condition="'$(Platform)'=='GGP'">2&gt;&amp;1 $(BazelSedCommand)</BazelSedCommand> <BazelSedCommand Condition="'$(Platform)'=='GGP'">2&gt;&amp;1 $(BazelSedCommand)</BazelSedCommand>
<!-- The workspace_status_command saves about 8 seconds. "exit 0" is roughly equivalent to 'true'. --> <BazelArgs>--config=$(BazelPlatform)</BazelArgs>
<BazelArgs>--config=$(BazelPlatform) --workspace_status_command=&quot;exit 0&quot; --bes_backend=</BazelArgs>
<BazelArgs Condition="'$(Configuration)|$(Platform)'=='Release|GGP'">$(BazelArgs) --linkopt=-Wl,--strip-all</BazelArgs> <BazelArgs Condition="'$(Configuration)|$(Platform)'=='Release|GGP'">$(BazelArgs) --linkopt=-Wl,--strip-all</BazelArgs>
<!-- Prevent protobuf recompilation (protobuf is written to "host" by default for both x84 and ggp, causing recompiles). --> <!-- Prevent protobuf recompilation (protobuf is written to "host" by default for both x84 and ggp, causing recompiles). -->
<BazelArgs Condition="'$(Platform)'=='x64'">$(BazelArgs) --distinct_host_configuration=false</BazelArgs> <BazelArgs Condition="'$(Platform)'=='x64'">$(BazelArgs) --distinct_host_configuration=false</BazelArgs>

114
README.md
View File

@@ -6,9 +6,113 @@ on Content Defined Chunking (CDC), in particular
to split up files into chunks. to split up files into chunks.
## CDC RSync ## CDC RSync
Tool to sync files to a remote machine, similar to the standard Linux
[rsync](https://linux.die.net/man/1/rsync). It supports fast compression and
uses a higher performing remote diffing approach based on CDC.
## Asset Streaming CDC RSync is a tool to sync files from a Windows machine to a Linux device,
Tool to stream assets from a Windows machine to a Linux device. similar to the standard Linux [rsync](https://linux.die.net/man/1/rsync). It is
basically a copy tool, but optimized for the case where there is already an old
version of the files available in the target directory.
* It skips files quickly if timestamp and file size match.
* It uses fast compression for all data transfer.
* If a file changed, it determines which parts changed and only transfers the
differences.
The remote diffing algorithm is based on CDC. In our tests, it is up to 30x
faster than the one used in rsync (1500 MB/s vs 50 MB/s).
## CDC Stream
CDC Stream is a tool to stream files and directories from a Windows machine to a
Linux device. Conceptually, it is similar to [sshfs](https://github.com/libfuse/sshfs),
but it is optimized for read speed.
* It caches streamed data on the Linux device.
* If a file is re-read on Linux after it changed on Windows, only the
differences are streamed again. The rest is read from cache.
* Stat operations are very fast since the directory metadata (filenames,
permissions etc.) is provided in a streaming-friendly way.
To efficiently determine which parts of a file changed, the tool uses the same
CDC-based diffing algorithm as CDC RSync. Changes to Windows files are almost
immediately reflected on Linux, with a delay of roughly (0.5s + 0.7s x total
size of changed files in GB).
The tool does not support writing files back from Linux to Windows; the Linux
directory is readonly.
# Getting Started
The project has to be built both on Windows and Linux.
## Prerequisites
The following steps have to be executed on **both Windows and Linux**.
* Download and install Bazel from https://bazel.build/install.
* Clone the repository.
```
git clone https://github.com/google/cdc-file-transfer
```
* Initialize submodules.
```
cd cdc-file-transfer
git submodule update --init --recursive
```
Finally, install an SSH client on the Windows device if not present.
The file transfer tools require `ssh.exe` and `scp.exe`.
## Building
The two tools can be built and used independently.
### CDC Sync
* Build Linux components
```
bazel build --config linux --compilation_mode=opt //cdc_rsync_server
```
* Build Windows components
```
bazel build --config windows --compilation_mode=opt //cdc_rsync
```
* Copy the Linux build output file `cdc_rsync_server` from
`bazel-bin/cdc_rsync_server` on the Linux system to `bazel-bin\cdc_rsync`
on the Windows machine.
### CDC Stream
* Build Linux components
```
bazel build --config linux --compilation_mode=opt //cdc_fuse_fs
```
* Build Windows components
```
bazel build --config windows --compilation_mode=opt //asset_stream_manager
```
* Copy the Linux build output files `cdc_fuse_fs` and `libfuse.so` from
`bazel-bin/cdc_fuse_fs` on the Linux system to `bazel-bin\asset_stream_manager`
on the Windows machine.
## Usage
### CDC Sync
To copy the contents of the Windows directory `C:\path\to\assets` to `~/assets`
on the Linux device `linux.machine.com`, run
```
cdc_rsync --ssh-command=C:\path\to\ssh.exe --scp-command=C:\path\to\scp.exe C:\path\to\assets\* user@linux.machine.com:~/assets -vr
```
Depending on your setup, you may have to specify additional arguments for the
ssh and scp commands, including proper quoting, e.g.
```
cdc_rsync --ssh-command="\"C:\path with space\to\ssh.exe\" -F ssh_config_file -i id_rsa_file -oStrictHostKeyChecking=yes -oUserKnownHostsFile=\"\"\"known_hosts_file\"\"\"" --scp-command="\"C:\path with space\to\scp.exe\" -F ssh_config_file -i id_rsa_file -oStrictHostKeyChecking=yes -oUserKnownHostsFile=\"\"\"known_hosts_file\"\"\"" C:\path\to\assets\* user@linux.machine.com:~/assets -vr
```
Lengthy ssh/scp commands that rarely change can also be put into environment
variables `CDC_SSH_COMMAND` and `CDC_SCP_COMMAND`, e.g.
```
set CDC_SSH_COMMAND="C:\path with space\to\ssh.exe" -F ssh_config_file -i id_rsa_file -oStrictHostKeyChecking=yes -oUserKnownHostsFile="""known_hosts_file"""
set CDC_SCP_COMMAND="C:\path with space\to\scp.exe" -F ssh_config_file -i id_rsa_file -oStrictHostKeyChecking=yes -oUserKnownHostsFile="""known_hosts_file"""
cdc_rsync C:\path\to\assets\* user@linux.machine.com:~/assets -vr
```
### CDC Stream

View File

@@ -96,10 +96,8 @@
<ClCompile Include="$(MSBuildThisFileDirectory)cdc_rsync\base\message_pump.cc" /> <ClCompile Include="$(MSBuildThisFileDirectory)cdc_rsync\base\message_pump.cc" />
<ClCompile Include="$(MSBuildThisFileDirectory)cdc_rsync\base\message_pump_test.cc" /> <ClCompile Include="$(MSBuildThisFileDirectory)cdc_rsync\base\message_pump_test.cc" />
<ClCompile Include="$(MSBuildThisFileDirectory)cdc_rsync\client_socket.cc" /> <ClCompile Include="$(MSBuildThisFileDirectory)cdc_rsync\client_socket.cc" />
<ClCompile Include="$(MSBuildThisFileDirectory)cdc_rsync\dllmain.cc" />
<ClCompile Include="$(MSBuildThisFileDirectory)cdc_rsync\file_finder_and_sender.cc" /> <ClCompile Include="$(MSBuildThisFileDirectory)cdc_rsync\file_finder_and_sender.cc" />
<ClCompile Include="$(MSBuildThisFileDirectory)cdc_rsync\file_finder_and_sender_test.cc" /> <ClCompile Include="$(MSBuildThisFileDirectory)cdc_rsync\file_finder_and_sender_test.cc" />
<ClCompile Include="$(MSBuildThisFileDirectory)cdc_rsync\cdc_rsync.cc" />
<ClCompile Include="$(MSBuildThisFileDirectory)cdc_rsync\cdc_rsync_client.cc" /> <ClCompile Include="$(MSBuildThisFileDirectory)cdc_rsync\cdc_rsync_client.cc" />
<ClCompile Include="$(MSBuildThisFileDirectory)cdc_rsync\parallel_file_opener.cc" /> <ClCompile Include="$(MSBuildThisFileDirectory)cdc_rsync\parallel_file_opener.cc" />
<ClCompile Include="$(MSBuildThisFileDirectory)cdc_rsync\parallel_file_opener_test.cc" /> <ClCompile Include="$(MSBuildThisFileDirectory)cdc_rsync\parallel_file_opener_test.cc" />
@@ -107,9 +105,9 @@
<ClCompile Include="$(MSBuildThisFileDirectory)cdc_rsync\progress_tracker_test.cc" /> <ClCompile Include="$(MSBuildThisFileDirectory)cdc_rsync\progress_tracker_test.cc" />
<ClCompile Include="$(MSBuildThisFileDirectory)cdc_rsync\zstd_stream.cc" /> <ClCompile Include="$(MSBuildThisFileDirectory)cdc_rsync\zstd_stream.cc" />
<ClCompile Include="$(MSBuildThisFileDirectory)cdc_rsync\zstd_stream_test.cc" /> <ClCompile Include="$(MSBuildThisFileDirectory)cdc_rsync\zstd_stream_test.cc" />
<ClCompile Include="$(MSBuildThisFileDirectory)cdc_rsync_cli\main.cc" /> <ClCompile Include="$(MSBuildThisFileDirectory)cdc_rsync\main.cc" />
<ClCompile Include="$(MSBuildThisFileDirectory)cdc_rsync_cli\params.cc" /> <ClCompile Include="$(MSBuildThisFileDirectory)cdc_rsync\params.cc" />
<ClCompile Include="$(MSBuildThisFileDirectory)cdc_rsync_cli\params_test.cc" /> <ClCompile Include="$(MSBuildThisFileDirectory)cdc_rsync\params_test.cc" />
<ClCompile Include="$(MSBuildThisFileDirectory)cdc_rsync_server\file_deleter_and_sender.cc" /> <ClCompile Include="$(MSBuildThisFileDirectory)cdc_rsync_server\file_deleter_and_sender.cc" />
<ClCompile Include="$(MSBuildThisFileDirectory)cdc_rsync_server\file_deleter_and_sender_test.cc" /> <ClCompile Include="$(MSBuildThisFileDirectory)cdc_rsync_server\file_deleter_and_sender_test.cc" />
<ClCompile Include="$(MSBuildThisFileDirectory)cdc_rsync_server\file_diff_generator.cc" /> <ClCompile Include="$(MSBuildThisFileDirectory)cdc_rsync_server\file_diff_generator.cc" />
@@ -202,14 +200,12 @@
<ClInclude Include="$(MSBuildThisFileDirectory)cdc_rsync\base\socket.h" /> <ClInclude Include="$(MSBuildThisFileDirectory)cdc_rsync\base\socket.h" />
<ClInclude Include="$(MSBuildThisFileDirectory)cdc_rsync\client_file_info.h" /> <ClInclude Include="$(MSBuildThisFileDirectory)cdc_rsync\client_file_info.h" />
<ClInclude Include="$(MSBuildThisFileDirectory)cdc_rsync\client_socket.h" /> <ClInclude Include="$(MSBuildThisFileDirectory)cdc_rsync\client_socket.h" />
<ClInclude Include="$(MSBuildThisFileDirectory)cdc_rsync\error_messages.h" />
<ClInclude Include="$(MSBuildThisFileDirectory)cdc_rsync\file_finder_and_sender.h" /> <ClInclude Include="$(MSBuildThisFileDirectory)cdc_rsync\file_finder_and_sender.h" />
<ClInclude Include="$(MSBuildThisFileDirectory)cdc_rsync\cdc_rsync.h" />
<ClInclude Include="$(MSBuildThisFileDirectory)cdc_rsync\cdc_rsync_client.h" /> <ClInclude Include="$(MSBuildThisFileDirectory)cdc_rsync\cdc_rsync_client.h" />
<ClInclude Include="$(MSBuildThisFileDirectory)cdc_rsync\parallel_file_opener.h" /> <ClInclude Include="$(MSBuildThisFileDirectory)cdc_rsync\parallel_file_opener.h" />
<ClInclude Include="$(MSBuildThisFileDirectory)cdc_rsync\progress_tracker.h" /> <ClInclude Include="$(MSBuildThisFileDirectory)cdc_rsync\progress_tracker.h" />
<ClInclude Include="$(MSBuildThisFileDirectory)cdc_rsync\zstd_stream.h" /> <ClInclude Include="$(MSBuildThisFileDirectory)cdc_rsync\zstd_stream.h" />
<ClInclude Include="$(MSBuildThisFileDirectory)cdc_rsync_cli\params.h" /> <ClInclude Include="$(MSBuildThisFileDirectory)cdc_rsync\params.h" />
<ClInclude Include="$(MSBuildThisFileDirectory)cdc_rsync_server\file_deleter_and_sender.h" /> <ClInclude Include="$(MSBuildThisFileDirectory)cdc_rsync_server\file_deleter_and_sender.h" />
<ClInclude Include="$(MSBuildThisFileDirectory)cdc_rsync_server\file_diff_generator.h" /> <ClInclude Include="$(MSBuildThisFileDirectory)cdc_rsync_server\file_diff_generator.h" />
<ClInclude Include="$(MSBuildThisFileDirectory)cdc_rsync_server\file_finder.h" /> <ClInclude Include="$(MSBuildThisFileDirectory)cdc_rsync_server\file_finder.h" />
@@ -247,7 +243,6 @@
<None Include="$(MSBuildThisFileDirectory)cdc_rsync\BUILD" /> <None Include="$(MSBuildThisFileDirectory)cdc_rsync\BUILD" />
<None Include="$(MSBuildThisFileDirectory)cdc_rsync\protos\messages.proto" /> <None Include="$(MSBuildThisFileDirectory)cdc_rsync\protos\messages.proto" />
<None Include="$(MSBuildThisFileDirectory)cdc_rsync\README.md" /> <None Include="$(MSBuildThisFileDirectory)cdc_rsync\README.md" />
<None Include="$(MSBuildThisFileDirectory)cdc_rsync_cli\BUILD" />
<None Include="$(MSBuildThisFileDirectory)cdc_rsync_server\BUILD" /> <None Include="$(MSBuildThisFileDirectory)cdc_rsync_server\BUILD" />
<None Include="$(MSBuildThisFileDirectory)cdc_rsync_tests\BUILD" /> <None Include="$(MSBuildThisFileDirectory)cdc_rsync_tests\BUILD" />
<None Include="$(MSBuildThisFileDirectory)manifest\BUILD" /> <None Include="$(MSBuildThisFileDirectory)manifest\BUILD" />
@@ -259,6 +254,7 @@
<None Include="$(MSBuildThisFileDirectory)proto\BUILD" /> <None Include="$(MSBuildThisFileDirectory)proto\BUILD" />
<None Include="$(MSBuildThisFileDirectory)proto\local_assets_stream_manager.proto" /> <None Include="$(MSBuildThisFileDirectory)proto\local_assets_stream_manager.proto" />
<None Include="$(MSBuildThisFileDirectory)proto\manifest.proto" /> <None Include="$(MSBuildThisFileDirectory)proto\manifest.proto" />
<None Include="$(MSBuildThisFileDirectory)README.md" />
<None Include="$(MSBuildThisFileDirectory)rm_bazel_out_dir.bat" /> <None Include="$(MSBuildThisFileDirectory)rm_bazel_out_dir.bat" />
<None Include="$(MSBuildThisFileDirectory)tests_asset_streaming_30\BUILD" /> <None Include="$(MSBuildThisFileDirectory)tests_asset_streaming_30\BUILD" />
<None Include="$(MSBuildThisFileDirectory)tests_common\BUILD" /> <None Include="$(MSBuildThisFileDirectory)tests_common\BUILD" />

View File

@@ -29,7 +29,7 @@ namespace {
constexpr char kFuseFilename[] = "cdc_fuse_fs"; constexpr char kFuseFilename[] = "cdc_fuse_fs";
constexpr char kLibFuseFilename[] = "libfuse.so"; constexpr char kLibFuseFilename[] = "libfuse.so";
constexpr char kFuseStdoutPrefix[] = "cdc_fuse_fs_stdout"; constexpr char kFuseStdoutPrefix[] = "cdc_fuse_fs_stdout";
constexpr char kRemoteToolsBinDir[] = "/opt/developer/tools/bin/"; constexpr char kRemoteToolsBinDir[] = "~/.cache/cdc_file_transfer/";
// Mount point for FUSE on the gamelet. // Mount point for FUSE on the gamelet.
constexpr char kMountDir[] = "/mnt/workstation"; constexpr char kMountDir[] = "/mnt/workstation";

View File

@@ -51,7 +51,7 @@ Session::Session(std::string instance_id, std::string instance_ip,
/*forward_output_to_logging=*/true), /*forward_output_to_logging=*/true),
metrics_recorder_(std::move(metrics_recorder)) { metrics_recorder_(std::move(metrics_recorder)) {
assert(metrics_recorder_); assert(metrics_recorder_);
remote_util_.SetIpAndPort(instance_ip, instance_port); remote_util_.SetUserHostAndPort(instance_ip, instance_port);
} }
Session::~Session() { Session::~Session() {

View File

@@ -1,12 +1,16 @@
load(
"//tools:windows_cc_library.bzl",
"cc_windows_shared_library",
)
package(default_visibility = [ package(default_visibility = [
"//:__subpackages__", "//:__subpackages__",
]) ])
cc_binary(
name = "cdc_rsync",
srcs = ["main.cc"],
deps = [
":cdc_rsync_client",
":params",
],
)
cc_library( cc_library(
name = "client_file_info", name = "client_file_info",
hdrs = ["client_file_info.h"], hdrs = ["client_file_info.h"],
@@ -57,25 +61,16 @@ cc_test(
], ],
) )
cc_windows_shared_library( cc_library(
name = "cdc_rsync", name = "cdc_rsync_client",
srcs = [ srcs = ["cdc_rsync_client.cc"],
"cdc_rsync.cc", hdrs = ["cdc_rsync_client.h"],
"cdc_rsync_client.cc",
"dllmain.cc",
],
hdrs = [
"cdc_rsync.h",
"cdc_rsync_client.h",
"error_messages.h",
],
linkopts = select({ linkopts = select({
"//tools:windows": [ "//tools:windows": [
"/DEFAULTLIB:Ws2_32.lib", # Sockets, e.g. recv, send, WSA*. "/DEFAULTLIB:Ws2_32.lib", # Sockets, e.g. recv, send, WSA*.
], ],
"//conditions:default": [], "//conditions:default": [],
}), }),
local_defines = ["COMPILING_DLL"],
target_compatible_with = ["@platforms//os:windows"], target_compatible_with = ["@platforms//os:windows"],
deps = [ deps = [
":client_socket", ":client_socket",
@@ -128,6 +123,28 @@ cc_test(
], ],
) )
cc_library(
name = "params",
srcs = ["params.cc"],
hdrs = ["params.h"],
deps = [
":cdc_rsync_client",
"@com_github_zstd//:zstd",
"@com_google_absl//absl/status",
],
)
cc_test(
name = "params_test",
srcs = ["params_test.cc"],
data = ["testdata/root.txt"] + glob(["testdata/params/**"]),
deps = [
":params",
"//common:test_main",
"@com_google_googletest//:gtest",
],
)
cc_library( cc_library(
name = "progress_tracker", name = "progress_tracker",
srcs = ["progress_tracker.cc"], srcs = ["progress_tracker.cc"],

View File

@@ -1,125 +0,0 @@
// Copyright 2022 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#include "cdc_rsync/cdc_rsync.h"
#include <vector>
#include "cdc_rsync/cdc_rsync_client.h"
#include "cdc_rsync/error_messages.h"
#include "common/log.h"
#include "common/path_filter.h"
#include "common/status.h"
namespace cdc_ft {
namespace {
ReturnCode TagToMessage(Tag tag, const Options* options, std::string* msg) {
msg->clear();
switch (tag) {
case Tag::kSocketEof:
*msg = kMsgConnectionLost;
return ReturnCode::kConnectionLost;
case Tag::kAddressInUse:
*msg = kMsgAddressInUse;
return ReturnCode::kAddressInUse;
case Tag::kDeployServer:
*msg = kMsgDeployFailed;
return ReturnCode::kDeployFailed;
case Tag::kInstancePickerNotAvailableInQuietMode:
*msg = kMsgInstancePickerNotAvailableInQuietMode;
return ReturnCode::kInstancePickerNotAvailableInQuietMode;
case Tag::kConnectionTimeout:
*msg =
absl::StrFormat(kMsgFmtConnectionTimeout, options->ip, options->port);
return ReturnCode::kConnectionTimeout;
case Tag::kCount:
return ReturnCode::kGenericError;
}
// Should not happen (TM). Will fall back to status message in this case.
return ReturnCode::kGenericError;
}
PathFilter::Rule::Type ToInternalType(FilterRule::Type type) {
switch (type) {
case FilterRule::Type::kInclude:
return PathFilter::Rule::Type::kInclude;
case FilterRule::Type::kExclude:
return PathFilter::Rule::Type::kExclude;
}
assert(false);
return PathFilter::Rule::Type::kInclude;
}
} // namespace
ReturnCode Sync(const Options* options, const FilterRule* filter_rules,
size_t num_filter_rules, const char* sources_dir,
const char* const* sources, size_t num_sources,
const char* destination, const char** error_message) {
LogLevel log_level = Log::VerbosityToLogLevel(options->verbosity);
Log::Initialize(std::make_unique<ConsoleLog>(log_level));
PathFilter path_filter;
for (size_t n = 0; n < num_filter_rules; ++n) {
path_filter.AddRule(ToInternalType(filter_rules[n].type),
filter_rules[n].pattern);
}
std::vector<std::string> sources_vec;
for (size_t n = 0; n < num_sources; ++n) {
sources_vec.push_back(sources[n]);
}
// Run rsync.
GgpRsyncClient client(*options, std::move(path_filter), sources_dir,
std::move(sources_vec), destination);
absl::Status status = client.Run();
if (status.ok()) {
*error_message = nullptr;
return ReturnCode::kOk;
}
std::string msg;
ReturnCode code = ReturnCode::kGenericError;
absl::optional<Tag> tag = GetTag(status);
if (tag.has_value()) {
code = TagToMessage(tag.value(), options, &msg);
}
// Fall back to status message.
if (msg.empty()) {
msg = std::string(status.message());
} else if (options->verbosity >= 2) {
// In verbose mode, log the status as well, so nothing gets lost.
LOG_ERROR("%s", status.ToString().c_str());
}
// Store error message in static buffer (don't use std::string through DLL
// boundary!).
static char buf[1024] = {0};
strncpy_s(buf, msg.c_str(), _TRUNCATE);
*error_message = buf;
return code;
}
} // namespace cdc_ft

View File

@@ -1,107 +0,0 @@
/*
* Copyright 2022 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
#ifndef CDC_RSYNC_CDC_RSYNC_H_
#define CDC_RSYNC_CDC_RSYNC_H_
#ifdef COMPILING_DLL
#define CDC_RSYNC_API __declspec(dllexport)
#else
#define CDC_RSYNC_API __declspec(dllimport)
#endif
namespace cdc_ft {
#ifdef __cplusplus
extern "C" {
#endif
struct Options {
const char* ip = nullptr;
int port = 0;
bool delete_ = false;
bool recursive = false;
int verbosity = 0;
bool quiet = false;
bool whole_file = false;
bool relative = false;
bool compress = false;
bool checksum = false;
bool dry_run = false;
bool existing = false;
bool json = false;
const char* copy_dest = nullptr;
int compress_level = 6;
int connection_timeout_sec = 10;
// Compression level 0 is invalid.
static constexpr int kMinCompressLevel = -5;
static constexpr int kMaxCompressLevel = 22;
};
// Rule for including/excluding files.
struct FilterRule {
enum class Type {
kInclude,
kExclude,
};
Type type;
const char* pattern;
FilterRule(Type type, const char* pattern) : type(type), pattern(pattern) {}
};
enum class ReturnCode {
// No error. Will match the tool's exit code, so OK must be 0.
kOk = 0,
// Generic error.
kGenericError = 1,
// Server connection timed out.
kConnectionTimeout = 2,
// Connection to the server was shut down unexpectedly.
kConnectionLost = 3,
// Binding to the forward port failed, probably because there's another
// instance of cdc_rsync running.
kAddressInUse = 4,
// Server deployment failed. This should be rare, it means that the server
// components were successfully copied, but the up-to-date check still fails.
kDeployFailed = 5,
// Gamelet selection asks for user input, but we are in quiet mode.
kInstancePickerNotAvailableInQuietMode = 6,
};
// Calling Sync() a second time overwrites the data in |error_message|.
CDC_RSYNC_API ReturnCode Sync(const Options* options,
const FilterRule* filter_rules,
size_t filter_num_rules, const char* sources_dir,
const char* const* sources, size_t num_sources,
const char* destination,
const char** error_message);
#ifdef __cplusplus
} // extern "C"
#endif
} // namespace cdc_ft
#endif // CDC_RSYNC_CDC_RSYNC_H_

View File

@@ -42,12 +42,12 @@
<Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" /> <Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
</ImportGroup> </ImportGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'"> <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">
<OutDir>$(SolutionDir)bazel-out\x64_windows-dbg\bin\cdc_rsync_cli\</OutDir> <OutDir>$(SolutionDir)bazel-out\x64_windows-dbg\bin\cdc_rsync\</OutDir>
<AdditionalOptions>/std:c++17</AdditionalOptions> <AdditionalOptions>/std:c++17</AdditionalOptions>
<NMakePreprocessorDefinitions>UNICODE</NMakePreprocessorDefinitions> <NMakePreprocessorDefinitions>UNICODE</NMakePreprocessorDefinitions>
</PropertyGroup> </PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'"> <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'">
<OutDir>$(SolutionDir)bazel-out\x64_windows-opt\bin\cdc_rsync_cli\</OutDir> <OutDir>$(SolutionDir)bazel-out\x64_windows-opt\bin\cdc_rsync\</OutDir>
<NMakePreprocessorDefinitions>UNICODE</NMakePreprocessorDefinitions> <NMakePreprocessorDefinitions>UNICODE</NMakePreprocessorDefinitions>
<AdditionalOptions>/std:c++17</AdditionalOptions> <AdditionalOptions>/std:c++17</AdditionalOptions>
</PropertyGroup> </PropertyGroup>
@@ -66,7 +66,7 @@
</ItemDefinitionGroup> </ItemDefinitionGroup>
<!-- Bazel setup --> <!-- Bazel setup -->
<PropertyGroup> <PropertyGroup>
<BazelTargets>//cdc_rsync_cli:cdc_rsync</BazelTargets> <BazelTargets>//cdc_rsync</BazelTargets>
<BazelOutputFile>cdc_rsync.exe</BazelOutputFile> <BazelOutputFile>cdc_rsync.exe</BazelOutputFile>
<BazelIncludePaths>..\;..\third_party\absl;..\third_party\blake3\c;..\bazel-stadia-file-transfer\external\com_github_zstd\lib;..\third_party\googletest\googletest\include;..\third_party\protobuf\src;$(VC_IncludePath);$(WindowsSDK_IncludePath)</BazelIncludePaths> <BazelIncludePaths>..\;..\third_party\absl;..\third_party\blake3\c;..\bazel-stadia-file-transfer\external\com_github_zstd\lib;..\third_party\googletest\googletest\include;..\third_party\protobuf\src;$(VC_IncludePath);$(WindowsSDK_IncludePath)</BazelIncludePaths>
<BazelSourcePathPrefix>..\/</BazelSourcePathPrefix> <BazelSourcePathPrefix>..\/</BazelSourcePathPrefix>

View File

@@ -47,7 +47,7 @@ constexpr int kExitCodeNotFound = 127;
constexpr int kForwardPortFirst = 44450; constexpr int kForwardPortFirst = 44450;
constexpr int kForwardPortLast = 44459; constexpr int kForwardPortLast = 44459;
constexpr char kGgpServerFilename[] = "cdc_rsync_server"; constexpr char kGgpServerFilename[] = "cdc_rsync_server";
constexpr char kRemoteToolsBinDir[] = "/opt/developer/tools/bin/"; constexpr char kRemoteToolsBinDir[] = "~/.cache/cdc_file_transfer/";
SetOptionsRequest::FilterRule::Type ToProtoType(PathFilter::Rule::Type type) { SetOptionsRequest::FilterRule::Type ToProtoType(PathFilter::Rule::Type type) {
switch (type) { switch (type) {
@@ -94,14 +94,12 @@ absl::Status GetServerExitStatus(int exit_code, const std::string& error_msg) {
} // namespace } // namespace
GgpRsyncClient::GgpRsyncClient(const Options& options, PathFilter path_filter, CdcRsyncClient::CdcRsyncClient(const Options& options,
std::string sources_dir,
std::vector<std::string> sources, std::vector<std::string> sources,
std::string destination) std::string user_host, std::string destination)
: options_(options), : options_(options),
path_filter_(std::move(path_filter)),
sources_dir_(std::move(sources_dir)),
sources_(std::move(sources)), sources_(std::move(sources)),
user_host_(std::move(user_host)),
destination_(std::move(destination)), destination_(std::move(destination)),
remote_util_(options.verbosity, options.quiet, &process_factory_, remote_util_(options.verbosity, options.quiet, &process_factory_,
/*forward_output_to_log=*/false), /*forward_output_to_log=*/false),
@@ -109,24 +107,26 @@ GgpRsyncClient::GgpRsyncClient(const Options& options, PathFilter path_filter,
kForwardPortFirst, kForwardPortLast, &process_factory_, kForwardPortFirst, kForwardPortLast, &process_factory_,
&remote_util_), &remote_util_),
printer_(options.quiet, Util::IsTTY() && !options.json), printer_(options.quiet, Util::IsTTY() && !options.json),
progress_(&printer_, options.verbosity, options.json) {} progress_(&printer_, options.verbosity, options.json) {
if (!options_.ssh_command.empty()) {
remote_util_.SetSshCommand(options_.ssh_command);
}
if (!options_.scp_command.empty()) {
remote_util_.SetScpCommand(options_.scp_command);
}
}
GgpRsyncClient::~GgpRsyncClient() { CdcRsyncClient::~CdcRsyncClient() {
message_pump_.StopMessagePump(); message_pump_.StopMessagePump();
socket_.Disconnect(); socket_.Disconnect();
} }
absl::Status GgpRsyncClient::Run() { absl::Status CdcRsyncClient::Run() {
absl::Status status = remote_util_.GetInitStatus();
if (!status.ok()) {
return WrapStatus(status, "Failed to initialize critical components");
}
// Initialize |remote_util_|. // Initialize |remote_util_|.
remote_util_.SetIpAndPort(options_.ip, options_.port); remote_util_.SetUserHostAndPort(user_host_, options_.port);
// Start the server process. // Start the server process.
status = StartServer(); absl::Status status = StartServer();
if (HasTag(status, Tag::kDeployServer)) { if (HasTag(status, Tag::kDeployServer)) {
// Gamelet components are not deployed or out-dated. Deploy and retry. // Gamelet components are not deployed or out-dated. Deploy and retry.
status = DeployServer(); status = DeployServer();
@@ -166,7 +166,7 @@ absl::Status GgpRsyncClient::Run() {
return status; return status;
} }
absl::Status GgpRsyncClient::StartServer() { absl::Status CdcRsyncClient::StartServer() {
assert(!server_process_); assert(!server_process_);
// Components are expected to reside in the same dir as the executable. // Components are expected to reside in the same dir as the executable.
@@ -187,8 +187,8 @@ absl::Status GgpRsyncClient::StartServer() {
std::string component_args = GameletComponent::ToCommandLineArgs(components); std::string component_args = GameletComponent::ToCommandLineArgs(components);
// Find available local and remote ports for port forwarding. // Find available local and remote ports for port forwarding.
absl::StatusOr<int> port_res = absl::StatusOr<int> port_res = port_manager_.ReservePort(
port_manager_.ReservePort(options_.connection_timeout_sec); /*check_remote=*/false, /*remote_timeout_sec unused*/ 0);
constexpr char kErrorMsg[] = "Failed to find available port"; constexpr char kErrorMsg[] = "Failed to find available port";
if (absl::IsDeadlineExceeded(port_res.status())) { if (absl::IsDeadlineExceeded(port_res.status())) {
// Server didn't respond in time. // Server didn't respond in time.
@@ -205,9 +205,11 @@ absl::Status GgpRsyncClient::StartServer() {
std::string(kRemoteToolsBinDir) + kGgpServerFilename; std::string(kRemoteToolsBinDir) + kGgpServerFilename;
// Test existence manually to prevent misleading bash output message // Test existence manually to prevent misleading bash output message
// "bash: .../cdc_rsync_server: No such file or directory". // "bash: .../cdc_rsync_server: No such file or directory".
std::string remote_command = absl::StrFormat( // Also create the bin dir because otherwise scp below might fail.
"if [ ! -f %s ]; then exit %i; fi; %s %i %s", remote_server_path, std::string remote_command =
kExitCodeNotFound, remote_server_path, port, component_args); absl::StrFormat("mkdir -p %s; if [ ! -f %s ]; then exit %i; fi; %s %i %s",
kRemoteToolsBinDir, remote_server_path, kExitCodeNotFound,
remote_server_path, port, component_args);
ProcessStartInfo start_info = ProcessStartInfo start_info =
remote_util_.BuildProcessStartInfoForSshPortForwardAndCommand( remote_util_.BuildProcessStartInfoForSshPortForwardAndCommand(
port, port, false, remote_command); port, port, false, remote_command);
@@ -225,16 +227,25 @@ absl::Status GgpRsyncClient::StartServer() {
} }
// Wait until the server process is listening. // Wait until the server process is listening.
auto detect_listening = [is_listening = &is_server_listening_]() -> bool { Stopwatch timeout_timer;
return *is_listening; bool is_timeout = false;
auto detect_listening_or_timeout = [is_listening = &is_server_listening_,
timeout = options_.connection_timeout_sec,
&timeout_timer, &is_timeout]() -> bool {
is_timeout = timeout_timer.ElapsedSeconds() > timeout;
return *is_listening || is_timeout;
}; };
status = process->RunUntil(detect_listening); status = process->RunUntil(detect_listening_or_timeout);
if (!status.ok()) { if (!status.ok()) {
// Some internal process error. Note that this does NOT mean that // Some internal process error. Note that this does NOT mean that
// cdc_rsync_server does not exist. In that case, the ssh process exits with // cdc_rsync_server does not exist. In that case, the ssh process exits with
// code 127. // code 127.
return status; return status;
} }
if (is_timeout) {
return SetTag(absl::DeadlineExceededError("Timeout while starting server"),
Tag::kConnectionTimeout);
}
if (process->HasExited()) { if (process->HasExited()) {
// Don't re-deploy for code > kServerExitCodeOutOfDate, which means that the // Don't re-deploy for code > kServerExitCodeOutOfDate, which means that the
@@ -263,7 +274,7 @@ absl::Status GgpRsyncClient::StartServer() {
return absl::OkStatus(); return absl::OkStatus();
} }
absl::Status GgpRsyncClient::StopServer() { absl::Status CdcRsyncClient::StopServer() {
assert(server_process_); assert(server_process_);
// Close socket. // Close socket.
@@ -282,7 +293,7 @@ absl::Status GgpRsyncClient::StopServer() {
return absl::OkStatus(); return absl::OkStatus();
} }
absl::Status GgpRsyncClient::HandleServerOutput(const char* data) { absl::Status CdcRsyncClient::HandleServerOutput(const char* data) {
// Note: This is called from a background thread! // Note: This is called from a background thread!
// Handle server error messages. Unfortunately, if the server prints to // Handle server error messages. Unfortunately, if the server prints to
@@ -319,7 +330,7 @@ absl::Status GgpRsyncClient::HandleServerOutput(const char* data) {
return absl::OkStatus(); return absl::OkStatus();
} }
absl::Status GgpRsyncClient::Sync() { absl::Status CdcRsyncClient::Sync() {
absl::Status status = SendOptions(); absl::Status status = SendOptions();
if (!status.ok()) { if (!status.ok()) {
return WrapStatus(status, "Failed to send options to server"); return WrapStatus(status, "Failed to send options to server");
@@ -377,7 +388,7 @@ absl::Status GgpRsyncClient::Sync() {
return status; return status;
} }
absl::Status GgpRsyncClient::DeployServer() { absl::Status CdcRsyncClient::DeployServer() {
assert(!server_process_); assert(!server_process_);
std::string exe_dir; std::string exe_dir;
@@ -409,34 +420,26 @@ absl::Status GgpRsyncClient::DeployServer() {
return WrapStatus(status, "Failed to copy cdc_rsync_server to instance"); return WrapStatus(status, "Failed to copy cdc_rsync_server to instance");
} }
// Make cdc_rsync_server executable. // Do 3 things in one SSH command, to save time:
status = remote_util_.Chmod("a+x", remoteServerTmpPath); // - Make the old cdc_rsync_server writable (if it exists).
// - Make the new cdc_rsync_server executable.
// - Replace the old cdc_rsync_server by the new one.
std::string old_path = RemoteUtil::EscapeForWindows(
std::string(kRemoteToolsBinDir) + kGgpServerFilename);
std::string new_path = RemoteUtil::EscapeForWindows(remoteServerTmpPath);
std::string replace_cmd = absl::StrFormat(
" ([ ! -f %s ] || chmod u+w %s) && chmod a+x %s && mv %s %s", old_path,
old_path, new_path, new_path, old_path);
status = remote_util_.Run(replace_cmd, "chmod && chmod && mv");
if (!status.ok()) { if (!status.ok()) {
return WrapStatus(status, return WrapStatus(status,
"Failed to set executable flag on cdc_rsync_server"); "Failed to replace old cdc_rsync_server by new one");
}
// Make old file writable. Mv might fail to overwrite it, e.g. if someone made
// it read-only.
std::string remoteServerPath =
std::string(kRemoteToolsBinDir) + kGgpServerFilename;
status = remote_util_.Chmod("u+w", remoteServerPath, /*quiet=*/true);
if (!status.ok()) {
LOG_DEBUG("chmod u+w %s failed (expected if file does not exist): %s",
remoteServerPath, status.ToString());
}
// Replace old file by new file.
status = remote_util_.Mv(remoteServerTmpPath, remoteServerPath);
if (!status.ok()) {
return WrapStatus(status, "Failed to replace '%s' by '%s'",
remoteServerPath, remoteServerTmpPath);
} }
return absl::OkStatus(); return absl::OkStatus();
} }
absl::Status GgpRsyncClient::SendOptions() { absl::Status CdcRsyncClient::SendOptions() {
LOG_INFO("Sending options"); LOG_INFO("Sending options");
SetOptionsRequest request; SetOptionsRequest request;
@@ -448,7 +451,7 @@ absl::Status GgpRsyncClient::SendOptions() {
request.set_compress(options_.compress); request.set_compress(options_.compress);
request.set_relative(options_.relative); request.set_relative(options_.relative);
for (const PathFilter::Rule& rule : path_filter_.GetRules()) { for (const PathFilter::Rule& rule : options_.filter.GetRules()) {
SetOptionsRequest::FilterRule* filter_rule = request.add_filter_rules(); SetOptionsRequest::FilterRule* filter_rule = request.add_filter_rules();
filter_rule->set_type(ToProtoType(rule.type)); filter_rule->set_type(ToProtoType(rule.type));
filter_rule->set_pattern(rule.pattern); filter_rule->set_pattern(rule.pattern);
@@ -457,7 +460,7 @@ absl::Status GgpRsyncClient::SendOptions() {
request.set_checksum(options_.checksum); request.set_checksum(options_.checksum);
request.set_dry_run(options_.dry_run); request.set_dry_run(options_.dry_run);
request.set_existing(options_.existing); request.set_existing(options_.existing);
if (options_.copy_dest) { if (!options_.copy_dest.empty()) {
request.set_copy_dest(options_.copy_dest); request.set_copy_dest(options_.copy_dest);
} }
@@ -470,13 +473,13 @@ absl::Status GgpRsyncClient::SendOptions() {
return absl::OkStatus(); return absl::OkStatus();
} }
absl::Status GgpRsyncClient::FindAndSendAllSourceFiles() { absl::Status CdcRsyncClient::FindAndSendAllSourceFiles() {
LOG_INFO("Finding and sending all sources files"); LOG_INFO("Finding and sending all sources files");
Stopwatch stopwatch; Stopwatch stopwatch;
FileFinderAndSender file_finder(&path_filter_, &message_pump_, &progress_, FileFinderAndSender file_finder(&options_.filter, &message_pump_, &progress_,
sources_dir_, options_.recursive, options_.sources_dir, options_.recursive,
options_.relative); options_.relative);
progress_.StartFindFiles(); progress_.StartFindFiles();
@@ -497,7 +500,7 @@ absl::Status GgpRsyncClient::FindAndSendAllSourceFiles() {
return absl::OkStatus(); return absl::OkStatus();
} }
absl::Status GgpRsyncClient::ReceiveFileStats() { absl::Status CdcRsyncClient::ReceiveFileStats() {
LOG_INFO("Receiving file stats"); LOG_INFO("Receiving file stats");
SendFileStatsResponse response; SendFileStatsResponse response;
@@ -517,7 +520,7 @@ absl::Status GgpRsyncClient::ReceiveFileStats() {
return absl::OkStatus(); return absl::OkStatus();
} }
absl::Status GgpRsyncClient::ReceiveDeletedFiles() { absl::Status CdcRsyncClient::ReceiveDeletedFiles() {
LOG_INFO("Receiving path of deleted files"); LOG_INFO("Receiving path of deleted files");
std::string current_directory; std::string current_directory;
@@ -548,7 +551,7 @@ absl::Status GgpRsyncClient::ReceiveDeletedFiles() {
return absl::OkStatus(); return absl::OkStatus();
} }
absl::Status GgpRsyncClient::ReceiveFileIndices( absl::Status CdcRsyncClient::ReceiveFileIndices(
const char* file_type, std::vector<uint32_t>* file_indices) { const char* file_type, std::vector<uint32_t>* file_indices) {
LOG_INFO("Receiving indices of %s files", file_type); LOG_INFO("Receiving indices of %s files", file_type);
@@ -582,7 +585,7 @@ absl::Status GgpRsyncClient::ReceiveFileIndices(
return absl::OkStatus(); return absl::OkStatus();
} }
absl::Status GgpRsyncClient::SendMissingFiles() { absl::Status CdcRsyncClient::SendMissingFiles() {
if (missing_file_indices_.empty()) { if (missing_file_indices_.empty()) {
return absl::OkStatus(); return absl::OkStatus();
} }
@@ -653,7 +656,7 @@ absl::Status GgpRsyncClient::SendMissingFiles() {
return absl::OkStatus(); return absl::OkStatus();
} }
absl::Status GgpRsyncClient::ReceiveSignaturesAndSendDelta() { absl::Status CdcRsyncClient::ReceiveSignaturesAndSendDelta() {
if (changed_file_indices_.empty()) { if (changed_file_indices_.empty()) {
return absl::OkStatus(); return absl::OkStatus();
} }
@@ -731,7 +734,7 @@ absl::Status GgpRsyncClient::ReceiveSignaturesAndSendDelta() {
return absl::OkStatus(); return absl::OkStatus();
} }
absl::Status GgpRsyncClient::StartCompressionStream() { absl::Status CdcRsyncClient::StartCompressionStream() {
assert(!compression_stream_); assert(!compression_stream_);
// Notify server that data is compressed from now on. // Notify server that data is compressed from now on.
@@ -762,7 +765,7 @@ absl::Status GgpRsyncClient::StartCompressionStream() {
return absl::OkStatus(); return absl::OkStatus();
} }
absl::Status GgpRsyncClient::StopCompressionStream() { absl::Status CdcRsyncClient::StopCompressionStream() {
assert(compression_stream_); assert(compression_stream_);
// Finish writing to |compression_process_|'s stdin and change back to // Finish writing to |compression_process_|'s stdin and change back to

View File

@@ -22,7 +22,6 @@
#include "absl/status/status.h" #include "absl/status/status.h"
#include "cdc_rsync/base/message_pump.h" #include "cdc_rsync/base/message_pump.h"
#include "cdc_rsync/cdc_rsync.h"
#include "cdc_rsync/client_socket.h" #include "cdc_rsync/client_socket.h"
#include "cdc_rsync/progress_tracker.h" #include "cdc_rsync/progress_tracker.h"
#include "common/path_filter.h" #include "common/path_filter.h"
@@ -34,13 +33,38 @@ namespace cdc_ft {
class Process; class Process;
class ZstdStream; class ZstdStream;
class GgpRsyncClient { class CdcRsyncClient {
public: public:
GgpRsyncClient(const Options& options, PathFilter filter, struct Options {
std::string sources_dir, std::vector<std::string> sources, int port = RemoteUtil::kDefaultSshPort;
std::string destination); bool delete_ = false;
bool recursive = false;
int verbosity = 0;
bool quiet = false;
bool whole_file = false;
bool relative = false;
bool compress = false;
bool checksum = false;
bool dry_run = false;
bool existing = false;
bool json = false;
std::string copy_dest;
int compress_level = 6;
int connection_timeout_sec = 10;
std::string ssh_command;
std::string scp_command;
std::string sources_dir; // Base dir for files loaded for --files-from.
PathFilter filter;
~GgpRsyncClient(); // Compression level 0 is invalid.
static constexpr int kMinCompressLevel = -5;
static constexpr int kMaxCompressLevel = 22;
};
CdcRsyncClient(const Options& options, std::vector<std::string> sources,
std::string user_host, std::string destination);
~CdcRsyncClient();
// Deploys the server if necessary, starts it and runs the rsync procedure. // Deploys the server if necessary, starts it and runs the rsync procedure.
absl::Status Run(); absl::Status Run();
@@ -93,11 +117,9 @@ class GgpRsyncClient {
absl::Status StopCompressionStream(); absl::Status StopCompressionStream();
Options options_; Options options_;
PathFilter path_filter_;
const std::string sources_dir_;
std::vector<std::string> sources_; std::vector<std::string> sources_;
const std::string user_host_;
const std::string destination_; const std::string destination_;
WinProcessFactory process_factory_; WinProcessFactory process_factory_;
RemoteUtil remote_util_; RemoteUtil remote_util_;
PortManager port_manager_; PortManager port_manager_;

View File

@@ -1,29 +0,0 @@
// Copyright 2022 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#define WIN32_LEAN_AND_MEAN
#include <windows.h>
BOOL APIENTRY DllMain(HMODULE /* hModule */, DWORD ul_reason_for_call,
LPVOID /* lpReserved */
) {
switch (ul_reason_for_call) {
case DLL_PROCESS_ATTACH:
case DLL_THREAD_ATTACH:
case DLL_THREAD_DETACH:
case DLL_PROCESS_DETACH:
break;
}
return TRUE;
}

View File

@@ -1,54 +0,0 @@
/*
* Copyright 2022 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
#ifndef CDC_RSYNC_ERROR_MESSAGES_H_
#define CDC_RSYNC_ERROR_MESSAGES_H_
namespace cdc_ft {
// Server connection timed out. SSH probably stale.
constexpr char kMsgFmtConnectionTimeout[] =
"Server connection timed out. Please re-run 'ggp ssh init' and verify that "
"the IP '%s' and the port '%i' are correct.";
// Server connection timed out and IP was not passed in. Probably network error.
constexpr char kMsgConnectionTimeoutWithIp[] =
"Server connection timed out. Please check your network connection.";
// Receiving pipe end was shut down unexpectedly.
constexpr char kMsgConnectionLost[] =
"The connection to the instance was shut down unexpectedly.";
// Binding to the port failed.
constexpr char kMsgAddressInUse[] =
"Failed to establish a connection to the instance. All ports are already "
"in use. This can happen if another instance of this command is running. "
"Currently, only 10 simultaneous connections are supported.";
// Deployment failed even though gamelet components were copied successfully.
constexpr char kMsgDeployFailed[] =
"Failed to deploy the instance components for unknown reasons. "
"Please report this issue.";
// Picking an instance is not allowed in quiet mode.
constexpr char kMsgInstancePickerNotAvailableInQuietMode[] =
"Multiple gamelet instances are reserved, but the instance picker is not "
"available in quiet mode. Please specify --instance or remove -q resp. "
"--quiet.";
} // namespace cdc_ft
#endif // CDC_RSYNC_ERROR_MESSAGES_H_

147
cdc_rsync/main.cc Normal file
View File

@@ -0,0 +1,147 @@
// Copyright 2022 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#define WIN32_LEAN_AND_MEAN
#include <windows.h>
#include <string>
#include <vector>
#include "cdc_rsync/cdc_rsync_client.h"
#include "cdc_rsync/params.h"
#include "common/log.h"
#include "common/status.h"
#include "common/util.h"
namespace {
enum class ReturnCode {
// No error. Will match the tool's exit code, so OK must be 0.
kOk = 0,
// Generic error.
kGenericError = 1,
// Server connection timed out.
kConnectionTimeout = 2,
// Connection to the server was shut down unexpectedly.
kConnectionLost = 3,
// Binding to the forward port failed, probably because there's another
// instance of cdc_rsync running.
kAddressInUse = 4,
// Server deployment failed. This should be rare, it means that the server
// components were successfully copied, but the up-to-date check still fails.
kDeployFailed = 5,
};
ReturnCode TagToMessage(cdc_ft::Tag tag,
const cdc_ft::params::Parameters& params,
std::string* msg) {
msg->clear();
switch (tag) {
case cdc_ft::Tag::kSocketEof:
// Receiving pipe end was shut down unexpectedly.
*msg = "The connection to the instance was shut down unexpectedly.";
return ReturnCode::kConnectionLost;
case cdc_ft::Tag::kAddressInUse:
*msg =
"Failed to establish a connection to the instance. All ports are "
"already in use. This can happen if another instance of this command "
"is running. Currently, only 10 simultaneous connections are "
"supported.";
return ReturnCode::kAddressInUse;
case cdc_ft::Tag::kDeployServer:
*msg =
"Failed to deploy the instance components for unknown reasons. "
"Please report this issue.";
return ReturnCode::kDeployFailed;
case cdc_ft::Tag::kConnectionTimeout:
// Server connection timed out. SSH probably stale.
*msg = absl::StrFormat(
"Server connection timed out. Verify that host '%s' and port '%i' "
"are correct, or specify a larger timeout with --contimeout.",
params.user_host, params.options.port);
return ReturnCode::kConnectionTimeout;
case cdc_ft::Tag::kCount:
return ReturnCode::kGenericError;
}
// Should not happen (TM). Will fall back to status message in this case.
return ReturnCode::kGenericError;
}
} // namespace
int wmain(int argc, wchar_t* argv[]) {
// Convert args from wide to UTF8 strings.
std::vector<std::string> utf8_str_args;
utf8_str_args.reserve(argc);
for (int i = 0; i < argc; i++) {
utf8_str_args.push_back(cdc_ft::Util::WideToUtf8Str(argv[i]));
}
// Convert args from UTF8 strings to UTF8 c-strings.
std::vector<const char*> utf8_args;
utf8_args.reserve(argc);
for (const auto& utf8_str_arg : utf8_str_args) {
utf8_args.push_back(utf8_str_arg.c_str());
}
// Read parameters from the environment and the command line.
cdc_ft::params::Parameters parameters;
if (!cdc_ft::params::Parse(argc, utf8_args.data(), &parameters)) {
return 1;
}
// Initialize logging.
cdc_ft::LogLevel log_level =
cdc_ft::Log::VerbosityToLogLevel(parameters.options.verbosity);
cdc_ft::Log::Initialize(std::make_unique<cdc_ft::ConsoleLog>(log_level));
// Run rsync.
cdc_ft::CdcRsyncClient client(parameters.options, parameters.sources,
parameters.user_host, parameters.destination);
absl::Status status = client.Run();
if (status.ok()) {
return static_cast<int>(ReturnCode::kOk);
}
// Get an error message from the tag associated with the status.
std::string error_message;
ReturnCode code = ReturnCode::kGenericError;
absl::optional<cdc_ft::Tag> tag = cdc_ft::GetTag(status);
if (tag.has_value()) {
code = TagToMessage(tag.value(), parameters, &error_message);
}
// Fall back to status message if there was no tag.
if (error_message.empty()) {
error_message = status.message();
} else if (parameters.options.verbosity >= 2) {
// In verbose mode, log the status as well, so nothing gets lost.
LOG_ERROR("%s", status.ToString());
}
if (!error_message.empty()) {
fprintf(stderr, "Error: %s\n", error_message.c_str());
}
return static_cast<int>(code);
}

View File

@@ -12,12 +12,13 @@
// See the License for the specific language governing permissions and // See the License for the specific language governing permissions and
// limitations under the License. // limitations under the License.
#include "cdc_rsync_cli/params.h" #include "cdc_rsync/params.h"
#include <cassert> #include <cassert>
#include "absl/status/status.h" #include "absl/status/status.h"
#include "absl/strings/str_format.h" #include "absl/strings/str_format.h"
#include "absl/strings/str_split.h"
#include "common/path.h" #include "common/path.h"
#include "lib/zstd.h" #include "lib/zstd.h"
@@ -25,6 +26,8 @@ namespace cdc_ft {
namespace params { namespace params {
namespace { namespace {
using Options = CdcRsyncClient::Options;
template <typename... Args> template <typename... Args>
void PrintError(const absl::FormatSpec<Args...>& format, Args... args) { void PrintError(const absl::FormatSpec<Args...>& format, Args... args) {
std::cerr << "Error: " << absl::StrFormat(format, args...) << std::endl; std::cerr << "Error: " << absl::StrFormat(format, args...) << std::endl;
@@ -39,11 +42,13 @@ Synchronizes local files and files on a gamelet. Matching files are skipped.
For partially matching files only the deltas are transferred. For partially matching files only the deltas are transferred.
Usage: Usage:
cdc_rsync [options] source [source]... destination cdc_rsync [options] source [source]... [user@]host:destination
Parameters: Parameters:
source Local file or folder to be copied source Local file or directory to be copied
destination Destination folder on the gamelet user Remote SSH user name
host Remote host or IP address
destination Remote destination directory
Options: Options:
--ip string Gamelet IP. Required. --ip string Gamelet IP. Required.
@@ -54,7 +59,7 @@ Options:
--json Print JSON progress --json Print JSON progress
-n, --dry-run Perform a trial run with no changes made -n, --dry-run Perform a trial run with no changes made
-r, --recursive Recurse into directories -r, --recursive Recurse into directories
--delete Delete extraneous files from destination folder --delete Delete extraneous files from destination directory
-z, --compress Compress file data during the transfer -z, --compress Compress file data during the transfer
--compress-level num Explicitly set compression level (default: 6) --compress-level num Explicitly set compression level (default: 6)
-c, --checksum Skip files based on checksum, not mod-time & size -c, --checksum Skip files based on checksum, not mod-time & size
@@ -68,13 +73,29 @@ Options:
-R, --relative Use relative path names -R, --relative Use relative path names
--existing Skip creating new files on instance --existing Skip creating new files on instance
--copy-dest dir Use files from dir as sync base if files are missing --copy-dest dir Use files from dir as sync base if files are missing
from destination folder --ssh-command Path and arguments of SSH command to use, e.g.
C:\path\to\ssh.exe -F config -i id_rsa -oStrictHostKeyChecking=yes -oUserKnownHostsFile="""known_hosts"""
Can also be specified by the CDC_SSH_COMMAND environment variable.
--scp-command Path and arguments of SSH command to use, e.g.
C:\path\to\scp.exe -F config -i id_rsa -oStrictHostKeyChecking=yes -oUserKnownHostsFile="""known_hosts"""
Can also be specified by the CDC_SCP_COMMAND environment variable.
-h --help Help for cdc_rsync -h --help Help for cdc_rsync
)"; )";
constexpr char kSshCommandEnvVar[] = "CDC_SSH_COMMAND";
constexpr char kScpCommandEnvVar[] = "CDC_SCP_COMMAND";
// Populates some parameters from environment variables.
void PopulateFromEnvVars(Parameters* parameters) {
path::GetEnv(kSshCommandEnvVar, &parameters->options.ssh_command)
.IgnoreError();
path::GetEnv(kScpCommandEnvVar, &parameters->options.scp_command)
.IgnoreError();
}
// Handles the --exclude-from and --include-from options. // Handles the --exclude-from and --include-from options.
OptionResult HandleFilterRuleFile(const std::string& option_name, OptionResult HandleFilterRuleFile(const std::string& option_name,
const char* path, FilterRule::Type type, const char* path, PathFilter::Rule::Type type,
Parameters* params) { Parameters* params) {
if (!path) { if (!path) {
PrintError("Option '%s' needs a value", option_name); PrintError("Option '%s' needs a value", option_name);
@@ -92,7 +113,7 @@ OptionResult HandleFilterRuleFile(const std::string& option_name,
} }
for (std::string& pattern : patterns) { for (std::string& pattern : patterns) {
params->filter_rules.emplace_back(type, std::move(pattern)); params->options.filter.AddRule(type, std::move(pattern));
} }
return OptionResult::kConsumedKeyValue; return OptionResult::kConsumedKeyValue;
} }
@@ -143,11 +164,6 @@ bool LoadFilesFrom(const std::string& files_from,
OptionResult HandleParameter(const std::string& key, const char* value, OptionResult HandleParameter(const std::string& key, const char* value,
Parameters* params, bool* help) { Parameters* params, bool* help) {
if (key == "ip") {
params->options.ip = value;
return OptionResult::kConsumedKeyValue;
}
if (key == "port") { if (key == "port") {
if (value) { if (value) {
params->options.port = atoi(value); params->options.port = atoi(value);
@@ -181,27 +197,29 @@ OptionResult HandleParameter(const std::string& key, const char* value,
} }
if (key == "include") { if (key == "include") {
params->filter_rules.emplace_back(FilterRule::Type::kInclude, value); params->options.filter.AddRule(PathFilter::Rule::Type::kInclude, value);
return OptionResult::kConsumedKeyValue; return OptionResult::kConsumedKeyValue;
} }
if (key == "include-from") { if (key == "include-from") {
return HandleFilterRuleFile(key, value, FilterRule::Type::kInclude, params); return HandleFilterRuleFile(key, value, PathFilter::Rule::Type::kInclude,
params);
} }
if (key == "exclude") { if (key == "exclude") {
params->filter_rules.emplace_back(FilterRule::Type::kExclude, value); params->options.filter.AddRule(PathFilter::Rule::Type::kExclude, value);
return OptionResult::kConsumedKeyValue; return OptionResult::kConsumedKeyValue;
} }
if (key == "exclude-from") { if (key == "exclude-from") {
return HandleFilterRuleFile(key, value, FilterRule::Type::kExclude, params); return HandleFilterRuleFile(key, value, PathFilter::Rule::Type::kExclude,
params);
} }
if (key == "files-from") { if (key == "files-from") {
// Implies -R. // Implies -R.
params->options.relative = true; params->options.relative = true;
params->files_from = value; params->files_from = value ? value : std::string();
return OptionResult::kConsumedKeyValue; return OptionResult::kConsumedKeyValue;
} }
@@ -250,7 +268,7 @@ OptionResult HandleParameter(const std::string& key, const char* value,
} }
if (key == "copy-dest") { if (key == "copy-dest") {
params->options.copy_dest = value; params->options.copy_dest = value ? value : std::string();
return OptionResult::kConsumedKeyValue; return OptionResult::kConsumedKeyValue;
} }
@@ -259,13 +277,23 @@ OptionResult HandleParameter(const std::string& key, const char* value,
return OptionResult::kConsumedKey; return OptionResult::kConsumedKey;
} }
if (key == "ssh-command") {
params->options.ssh_command = value ? value : std::string();
return OptionResult::kConsumedKeyValue;
}
if (key == "scp-command") {
params->options.scp_command = value ? value : std::string();
return OptionResult::kConsumedKeyValue;
}
PrintError("Unknown option: '%s'", key); PrintError("Unknown option: '%s'", key);
return OptionResult::kError; return OptionResult::kError;
} }
bool CheckParameters(const Parameters& params, bool help) { bool ValidateParameters(const Parameters& params, bool help) {
if (help) { if (help) {
printf("%s", kHelpText); std::cout << kHelpText;
return false; return false;
} }
@@ -274,13 +302,7 @@ bool CheckParameters(const Parameters& params, bool help) {
return false; return false;
} }
if (!params.options.ip || params.options.ip[0] == '\0') { if (params.options.port <= 0 || params.options.port > UINT16_MAX) {
PrintError("--ip must specify a valid IP address");
return false;
}
if (!params.options.port || params.options.port <= 0 ||
params.options.port > UINT16_MAX) {
PrintError("--port must specify a valid port"); PrintError("--port must specify a valid port");
return false; return false;
} }
@@ -301,9 +323,10 @@ bool CheckParameters(const Parameters& params, bool help) {
// Warn that any include rules not followed by an exclude rule are pointless // Warn that any include rules not followed by an exclude rule are pointless
// as the files would be included, anyway. // as the files would be included, anyway.
for (int n = static_cast<int>(params.filter_rules.size()) - 1; n >= 0; --n) { const std::vector<PathFilter::Rule>& rules = params.options.filter.GetRules();
const Parameters::FilterRule& rule = params.filter_rules[n]; for (int n = static_cast<int>(rules.size()) - 1; n >= 0; --n) {
if (rule.type == FilterRule::Type::kExclude) { const PathFilter::Rule& rule = rules[n];
if (rule.type == PathFilter::Rule::Type::kExclude) {
break; break;
} }
std::cout << "Warning: Include pattern '" << rule.pattern std::cout << "Warning: Include pattern '" << rule.pattern
@@ -311,6 +334,31 @@ bool CheckParameters(const Parameters& params, bool help) {
<< std::endl; << std::endl;
} }
if (params.sources.empty() && params.destination.empty()) {
PrintError("Missing source and destination");
return false;
}
if (params.destination.empty()) {
PrintError("Missing destination");
return false;
}
if (params.sources.empty()) {
// If one arg was passed on the command line, it is not clear whether it
// was supposed to be a source or destination.
PrintError("Missing source or destination");
return false;
}
if (params.user_host.empty()) {
PrintError(
"No remote host specified in destination '%s'. "
"Expected [user@]host:destination.",
params.destination);
return false;
}
return true; return true;
} }
@@ -335,6 +383,25 @@ bool CheckOptionResult(OptionResult result, const std::string& name,
return true; return true;
} }
// Removes the user/host part of |destination| and puts it into |user_host|,
// e.g. if |destination| is initially "user@foo.com:~/file", it is "~/file"
// afterward and |user_host| is |user@foo.com|. Does not touch Windows drives,
// e.g. C:\foo.
void PopUserHost(std::string* destination, std::string* user_host) {
std::vector<std::string> parts =
absl::StrSplit(*destination, absl::MaxSplits(':', 1));
if (parts.size() < 2) return;
// Don't mistake the C part of C:\foo as user/host.
if (parts[0].size() == 1 && toupper(parts[0][0]) >= 'A' &&
toupper(parts[0][0]) <= 'Z') {
return;
}
*user_host = parts[0];
*destination = parts[1];
}
} // namespace } // namespace
const char* HelpText() { return kHelpText; } const char* HelpText() { return kHelpText; }
@@ -349,6 +416,9 @@ bool Parse(int argc, const char* const* argv, Parameters* parameters) {
return false; return false;
} }
// Before applying args, populate parameters from env vars.
PopulateFromEnvVars(parameters);
bool help = false; bool help = false;
for (int index = 1; index < argc; ++index) { for (int index = 1; index < argc; ++index) {
// Handle '--key [value]' and '--key=value' options. // Handle '--key [value]' and '--key=value' options.
@@ -404,34 +474,15 @@ bool Parse(int argc, const char* const* argv, Parameters* parameters) {
// Load files-from file (can't do it when --files-from is handled since not // Load files-from file (can't do it when --files-from is handled since not
// all sources might have been read at that point. // all sources might have been read at that point.
if (parameters->files_from && if (!parameters->files_from.empty() &&
!LoadFilesFrom(parameters->files_from, &parameters->sources, !LoadFilesFrom(parameters->files_from, &parameters->sources,
&parameters->sources_dir)) { &parameters->options.sources_dir)) {
return false; return false;
} }
if (!CheckParameters(*parameters, help)) { PopUserHost(&parameters->destination, &parameters->user_host);
return false;
}
if (parameters->sources.empty() && parameters->destination.empty()) { if (!ValidateParameters(*parameters, help)) {
PrintError("Missing source and destination");
return false;
}
if (parameters->destination.empty()) {
PrintError("Missing destination");
return false;
}
if (parameters->sources.empty()) {
// If one arg was passed on the command line, it is not clear whether it
// was supposed to be a source or destination. Try to infer that, e.g.
// cdc_rsync *.txt -> Missing destination
// cdc_rsync /mnt/developer -> Missing source
bool missing_src = parameters->destination[0] == '/';
PrintError("Missing %s", missing_src ? "source" : "destination");
return false; return false;
} }

View File

@@ -14,34 +14,24 @@
* limitations under the License. * limitations under the License.
*/ */
#ifndef CDC_RSYNC_CLI_PARAMS_H_ #ifndef CDC_RSYNC_PARAMS_H_
#define CDC_RSYNC_CLI_PARAMS_H_ #define CDC_RSYNC_PARAMS_H_
#include <string> #include <string>
#include <vector> #include <vector>
#include "cdc_rsync/cdc_rsync.h" #include "cdc_rsync/cdc_rsync_client.h"
namespace cdc_ft { namespace cdc_ft {
namespace params { namespace params {
// All cdc_rsync command line parameters. // All cdc_rsync command line parameters.
struct Parameters { struct Parameters {
// Copy of cdc_ft::FilterRule with std::string instead of const char*. CdcRsyncClient::Options options;
struct FilterRule {
using Type = ::cdc_ft::FilterRule::Type;
FilterRule(Type type, std::string pattern)
: type(type), pattern(std::move(pattern)) {}
Type type;
std::string pattern;
};
Options options;
std::vector<FilterRule> filter_rules;
std::vector<std::string> sources; std::vector<std::string> sources;
std::string user_host;
std::string destination; std::string destination;
const char* files_from = nullptr; std::string files_from;
std::string sources_dir; // Base directory for files loaded for --files-from.
}; };
// Parses sources, destination and options from the command line args. // Parses sources, destination and options from the command line args.
@@ -51,4 +41,4 @@ bool Parse(int argc, const char* const* argv, Parameters* parameters);
} // namespace params } // namespace params
} // namespace cdc_ft } // namespace cdc_ft
#endif // CDC_RSYNC_CLI_PARAMS_H_ #endif // CDC_RSYNC_PARAMS_H_

View File

@@ -12,11 +12,12 @@
// See the License for the specific language governing permissions and // See the License for the specific language governing permissions and
// limitations under the License. // limitations under the License.
#include "cdc_rsync_cli/params.h" #include "cdc_rsync/params.h"
#include "absl/strings/match.h" #include "absl/strings/match.h"
#include "common/log.h" #include "common/log.h"
#include "common/path.h" #include "common/path.h"
#include "common/status_test_macros.h"
#include "common/test_main.h" #include "common/test_main.h"
#include "gtest/gtest.h" #include "gtest/gtest.h"
@@ -24,6 +25,13 @@ namespace cdc_ft {
namespace params { namespace params {
namespace { namespace {
using Options = CdcRsyncClient::Options;
constexpr char kSrc[] = "source";
constexpr char kUserHostDst[] = "user@host:destination";
constexpr char kUserHost[] = "user@host";
constexpr char kDst[] = "destination";
class TestLog : public Log { class TestLog : public Log {
public: public:
explicit TestLog() : Log(LogLevel::kInfo) {} explicit TestLog() : Log(LogLevel::kInfo) {}
@@ -44,9 +52,15 @@ std::string NeedsValueError(const char* option_name) {
class ParamsTest : public ::testing::Test { class ParamsTest : public ::testing::Test {
public: public:
void SetUp() override { prev_stderr_ = std::cerr.rdbuf(errors_.rdbuf()); } void SetUp() override {
prev_stdout_ = std::cout.rdbuf(output_.rdbuf());
prev_stderr_ = std::cerr.rdbuf(errors_.rdbuf());
}
void TearDown() override { std::cerr.rdbuf(prev_stderr_); } void TearDown() override {
std::cout.rdbuf(prev_stdout_);
std::cerr.rdbuf(prev_stderr_);
}
protected: protected:
void ExpectNoError() const { void ExpectNoError() const {
@@ -54,6 +68,12 @@ class ParamsTest : public ::testing::Test {
<< "Expected empty stderr but got\n'" << errors_.str() << "'"; << "Expected empty stderr but got\n'" << errors_.str() << "'";
} }
void ExpectOutput(const std::string& expected) const {
EXPECT_TRUE(absl::StrContains(output_.str(), expected))
<< "Expected stdout to contain '" << expected << "' but got\n'"
<< output_.str() << "'";
}
void ExpectError(const std::string& expected) const { void ExpectError(const std::string& expected) const {
EXPECT_TRUE(absl::StrContains(errors_.str(), expected)) EXPECT_TRUE(absl::StrContains(errors_.str(), expected))
<< "Expected stderr to contain '" << expected << "' but got\n'" << "Expected stderr to contain '" << expected << "' but got\n'"
@@ -68,16 +88,16 @@ class ParamsTest : public ::testing::Test {
path::Join(base_dir_, "empty_source_files.txt"); path::Join(base_dir_, "empty_source_files.txt");
Parameters parameters_; Parameters parameters_;
std::stringstream output_;
std::stringstream errors_; std::stringstream errors_;
std::streambuf* prev_stdout_;
std::streambuf* prev_stderr_; std::streambuf* prev_stderr_;
}; };
TEST_F(ParamsTest, ParseSucceedsDefaults) { TEST_F(ParamsTest, ParseSucceedsDefaults) {
const char* argv[] = {"cdc_rsync.exe", "--ip=1.2.3.4", "--port=1234", const char* argv[] = {"cdc_rsync.exe", kSrc, kUserHostDst, NULL};
"source", "destination", NULL};
EXPECT_TRUE(Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_)); EXPECT_TRUE(Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_));
EXPECT_STREQ("1.2.3.4", parameters_.options.ip); EXPECT_EQ(RemoteUtil::kDefaultSshPort, parameters_.options.port);
EXPECT_EQ(1234, parameters_.options.port);
EXPECT_FALSE(parameters_.options.delete_); EXPECT_FALSE(parameters_.options.delete_);
EXPECT_FALSE(parameters_.options.recursive); EXPECT_FALSE(parameters_.options.recursive);
EXPECT_EQ(0, parameters_.options.verbosity); EXPECT_EQ(0, parameters_.options.verbosity);
@@ -86,19 +106,19 @@ TEST_F(ParamsTest, ParseSucceedsDefaults) {
EXPECT_FALSE(parameters_.options.compress); EXPECT_FALSE(parameters_.options.compress);
EXPECT_FALSE(parameters_.options.checksum); EXPECT_FALSE(parameters_.options.checksum);
EXPECT_FALSE(parameters_.options.dry_run); EXPECT_FALSE(parameters_.options.dry_run);
EXPECT_EQ(parameters_.options.copy_dest, nullptr); EXPECT_TRUE(parameters_.options.copy_dest.empty());
EXPECT_EQ(6, parameters_.options.compress_level); EXPECT_EQ(6, parameters_.options.compress_level);
EXPECT_EQ(10, parameters_.options.connection_timeout_sec); EXPECT_EQ(10, parameters_.options.connection_timeout_sec);
EXPECT_EQ(1, parameters_.sources.size()); EXPECT_EQ(1, parameters_.sources.size());
EXPECT_EQ(parameters_.sources[0], "source"); EXPECT_EQ(parameters_.sources[0], kSrc);
EXPECT_EQ(parameters_.destination, "destination"); EXPECT_EQ(parameters_.user_host, kUserHost);
EXPECT_EQ(parameters_.destination, kDst);
ExpectNoError(); ExpectNoError();
} }
TEST_F(ParamsTest, ParseSucceedsWithOptionFromTwoArguments) { TEST_F(ParamsTest, ParseSucceedsWithOptionFromTwoArguments) {
const char* argv[] = { const char* argv[] = {
"cdc_rsync.exe", "--ip=1.2.3.4", "--port=1234", "--compress-level", "2", "cdc_rsync.exe", "--compress-level", "2", kSrc, kUserHostDst, NULL};
"source", "destination", NULL};
EXPECT_TRUE(Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_)); EXPECT_TRUE(Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_));
EXPECT_EQ(parameters_.options.compress_level, 2); EXPECT_EQ(parameters_.options.compress_level, 2);
ExpectNoError(); ExpectNoError();
@@ -106,68 +126,104 @@ TEST_F(ParamsTest, ParseSucceedsWithOptionFromTwoArguments) {
TEST_F(ParamsTest, TEST_F(ParamsTest,
ParseSucceedsWithOptionFromOneArgumentWithEqualityWithValue) { ParseSucceedsWithOptionFromOneArgumentWithEqualityWithValue) {
const char* argv[] = { const char* argv[] = {"cdc_rsync.exe", "--compress-level=2", kSrc,
"cdc_rsync.exe", "--ip=1.2.3.4", "--port=1234", "--compress-level=2", kUserHostDst, NULL};
"source", "destination", NULL};
EXPECT_TRUE(Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_)); EXPECT_TRUE(Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_));
ASSERT_EQ(parameters_.sources.size(), 1); ASSERT_EQ(parameters_.sources.size(), 1);
EXPECT_EQ(parameters_.options.compress_level, 2); EXPECT_EQ(parameters_.options.compress_level, 2);
EXPECT_EQ(parameters_.sources[0], "source"); EXPECT_EQ(parameters_.sources[0], kSrc);
EXPECT_EQ(parameters_.destination, "destination"); EXPECT_EQ(parameters_.user_host, kUserHost);
EXPECT_EQ(parameters_.destination, kDst);
ExpectNoError(); ExpectNoError();
} }
TEST_F(ParamsTest, ParseFailsOnCompressLevelEqualsNoValue) { TEST_F(ParamsTest, ParseFailsOnCompressLevelEqualsNoValue) {
const char* argv[] = {"cdc_rsync.exe", "--compress-level=", "source", const char* argv[] = {"cdc_rsync.exe", "--compress-level=", kSrc,
"destination", NULL}; kUserHostDst, NULL};
EXPECT_FALSE( EXPECT_FALSE(
Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_)); Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_));
ExpectError(NeedsValueError("compress-level")); ExpectError(NeedsValueError("compress-level"));
} }
TEST_F(ParamsTest, ParseFailsOnPortEqualsNoValue) { TEST_F(ParamsTest, ParseFailsOnPortEqualsNoValue) {
const char* argv[] = {"cdc_rsync.exe", "--port=", "source", "destination", const char* argv[] = {"cdc_rsync.exe", "--port=", kSrc, kUserHostDst, NULL};
NULL};
EXPECT_FALSE( EXPECT_FALSE(
Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_)); Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_));
ExpectError(NeedsValueError("port")); ExpectError(NeedsValueError("port"));
} }
TEST_F(ParamsTest, ParseFailsOnContimeoutEqualsNoValue) { TEST_F(ParamsTest, ParseFailsOnContimeoutEqualsNoValue) {
const char* argv[] = {"cdc_rsync.exe", "--contimeout=", "source", const char* argv[] = {"cdc_rsync.exe", "--contimeout=", kSrc, kUserHostDst,
"destination", NULL}; NULL};
EXPECT_FALSE( EXPECT_FALSE(
Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_)); Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_));
ExpectError(NeedsValueError("contimeout")); ExpectError(NeedsValueError("contimeout"));
} }
TEST_F(ParamsTest, ParseFailsOnIpEqualsNoValue) { TEST_F(ParamsTest, ParseSucceedsWithSshScpCommands) {
const char* argv[] = {"cdc_rsync.exe", "--ip=", "source", "destination", const char* argv[] = {"cdc_rsync.exe", kSrc,
kUserHostDst, "--ssh-command=sshcmd",
"--scp-command=scpcmd", NULL};
EXPECT_TRUE(Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_));
EXPECT_EQ(parameters_.options.scp_command, "scpcmd");
EXPECT_EQ(parameters_.options.ssh_command, "sshcmd");
}
TEST_F(ParamsTest, ParseSucceedsWithSshScpCommandsByEnvVars) {
EXPECT_OK(path::SetEnv("CDC_SSH_COMMAND", "sshcmd"));
EXPECT_OK(path::SetEnv("CDC_SCP_COMMAND", "scpcmd"));
const char* argv[] = {"cdc_rsync.exe", kSrc, kUserHostDst, NULL};
EXPECT_TRUE(Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_));
EXPECT_EQ(parameters_.options.scp_command, "scpcmd");
EXPECT_EQ(parameters_.options.ssh_command, "sshcmd");
}
TEST_F(ParamsTest, ParseSucceedsWithNoSshCommand) {
const char* argv[] = {"cdc_rsync.exe", kSrc, kUserHostDst,
"--ssh-command=", NULL};
EXPECT_FALSE(
Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_));
ExpectError(NeedsValueError("ssh-command"));
}
TEST_F(ParamsTest, ParseSucceedsWithNoScpCommand) {
const char* argv[] = {"cdc_rsync.exe", kSrc, kUserHostDst, "--scp-command",
NULL}; NULL};
EXPECT_FALSE( EXPECT_FALSE(
Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_)); Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_));
ExpectError(NeedsValueError("ip")); ExpectError(NeedsValueError("scp-command"));
}
TEST_F(ParamsTest, ParseFailsOnNoUserHost) {
const char* argv[] = {"cdc_rsync.exe", kSrc, kDst, NULL};
EXPECT_FALSE(
Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_));
ExpectError("No remote host specified");
}
TEST_F(ParamsTest, ParseDoesNotThinkCIsAHost) {
const char* argv[] = {"cdc_rsync.exe", kSrc, "C:\\foo", NULL};
EXPECT_FALSE(
Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_));
ExpectError("No remote host specified");
} }
TEST_F(ParamsTest, ParseWithoutParametersFailsOnMissingSourceAndDestination) { TEST_F(ParamsTest, ParseWithoutParametersFailsOnMissingSourceAndDestination) {
const char* argv[] = {"cdc_rsync.exe", "--ip=1.2.3.4", "--port=1234", NULL}; const char* argv[] = {"cdc_rsync.exe", NULL};
EXPECT_FALSE( EXPECT_FALSE(
Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_)); Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_));
ExpectError("Missing source"); ExpectOutput("Usage:");
} }
TEST_F(ParamsTest, ParseWithSingleParameterFailsOnMissingDestination) { TEST_F(ParamsTest, ParseWithSingleParameterFailsOnMissingDestination) {
const char* argv[] = {"cdc_rsync.exe", "--ip=1.2.3.4", "--port=1234", const char* argv[] = {"cdc_rsync.exe", kSrc, NULL};
"source", NULL};
EXPECT_FALSE( EXPECT_FALSE(
Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_)); Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_));
ExpectError("Missing destination"); ExpectError("Missing source or destination");
} }
TEST_F(ParamsTest, ParseSuccessedsWithMultipleLetterKeyConsumed) { TEST_F(ParamsTest, ParseSucceedsWithMultipleLetterKeyConsumed) {
const char* argv[] = { const char* argv[] = {"cdc_rsync.exe", "-rvqWRzcn", kSrc, kUserHostDst, NULL};
"cdc_rsync.exe", "--ip=1.2.3.4", "--port=1234", "-rvqWRzcn",
"source", "destination", NULL};
EXPECT_TRUE(Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_)); EXPECT_TRUE(Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_));
EXPECT_TRUE(parameters_.options.recursive); EXPECT_TRUE(parameters_.options.recursive);
EXPECT_EQ(parameters_.options.verbosity, 1); EXPECT_EQ(parameters_.options.verbosity, 1);
@@ -182,17 +238,15 @@ TEST_F(ParamsTest, ParseSuccessedsWithMultipleLetterKeyConsumed) {
TEST_F(ParamsTest, TEST_F(ParamsTest,
ParseFailsOnMultipleLetterKeyConsumedOptionsWithUnsupportedOne) { ParseFailsOnMultipleLetterKeyConsumedOptionsWithUnsupportedOne) {
const char* argv[] = {"cdc_rsync.exe", "-rvqaWRzcn", "source", "destination", const char* argv[] = {"cdc_rsync.exe", "-rvqaWRzcn", kSrc, kUserHostDst,
NULL}; NULL};
EXPECT_FALSE( EXPECT_FALSE(
Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_)); Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_));
ExpectError("Unknown option: 'a'"); ExpectError("Unknown option: 'a'");
} }
TEST_F(ParamsTest, ParseSuccessedsWithMultipleLongKeyConsumedOptions) { TEST_F(ParamsTest, ParseSucceedsWithMultipleLongKeyConsumedOptions) {
const char* argv[] = {"cdc_rsync.exe", const char* argv[] = {"cdc_rsync.exe",
"--ip=1.2.3.4",
"--port=1234",
"--recursive", "--recursive",
"--verbosity", "--verbosity",
"--quiet", "--quiet",
@@ -204,8 +258,8 @@ TEST_F(ParamsTest, ParseSuccessedsWithMultipleLongKeyConsumedOptions) {
"--dry-run", "--dry-run",
"--existing", "--existing",
"--json", "--json",
"source", kSrc,
"destination", kUserHostDst,
NULL}; NULL};
EXPECT_TRUE(Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_)); EXPECT_TRUE(Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_));
EXPECT_TRUE(parameters_.options.recursive); EXPECT_TRUE(parameters_.options.recursive);
@@ -223,51 +277,42 @@ TEST_F(ParamsTest, ParseSuccessedsWithMultipleLongKeyConsumedOptions) {
} }
TEST_F(ParamsTest, ParseFailsOnUnknownKey) { TEST_F(ParamsTest, ParseFailsOnUnknownKey) {
const char* argv[] = {"cdc_rsync.exe", "-unknownKey", "source", "destination", const char* argv[] = {"cdc_rsync.exe", "-unknownKey", kSrc, kUserHostDst,
NULL}; NULL};
EXPECT_FALSE( EXPECT_FALSE(
Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_)); Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_));
ExpectError("Unknown option: 'u'"); ExpectError("Unknown option: 'u'");
} }
TEST_F(ParamsTest, ParseSuccessedsWithSupportedKeyValue) { TEST_F(ParamsTest, ParseSucceedsWithSupportedKeyValue) {
const char* argv[] = { const char* argv[] = {
"cdc_rsync.exe", "--compress-level", "11", "--port=4086", "cdc_rsync.exe", "--compress-level", "11", "--contimeout", "99", "--port",
"--ip=127.0.0.1", "--contimeout", "99", "--copy-dest=dest", "4086", "--copy-dest=dest", kSrc, kUserHostDst, NULL};
"source", "destination", NULL};
EXPECT_TRUE(Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_)); EXPECT_TRUE(Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_));
EXPECT_EQ(parameters_.options.compress_level, 11); EXPECT_EQ(parameters_.options.compress_level, 11);
EXPECT_EQ(parameters_.options.connection_timeout_sec, 99); EXPECT_EQ(parameters_.options.connection_timeout_sec, 99);
EXPECT_EQ(parameters_.options.port, 4086); EXPECT_EQ(parameters_.options.port, 4086);
EXPECT_STREQ(parameters_.options.ip, "127.0.0.1"); EXPECT_EQ(parameters_.options.copy_dest, "dest");
EXPECT_STREQ(parameters_.options.copy_dest, "dest");
ExpectNoError(); ExpectNoError();
} }
TEST_F(ParamsTest, TEST_F(ParamsTest, ParseSucceedsWithSupportedKeyValueWithoutEqualityForChars) {
ParseSuccessedsWithSupportedKeyValueWithoutEqualityForChars) { const char* argv[] = {"cdc_rsync.exe", "--copy-dest", "dest", kSrc,
const char* argv[] = {"cdc_rsync.exe", "--port", "4086", "--ip", kUserHostDst, NULL};
"127.0.0.1", "--copy-dest", "dest", "source",
"destination", NULL};
EXPECT_TRUE(Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_)); EXPECT_TRUE(Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_));
EXPECT_EQ(parameters_.options.port, 4086); EXPECT_EQ(parameters_.options.copy_dest, "dest");
EXPECT_STREQ(parameters_.options.ip, "127.0.0.1");
EXPECT_STREQ(parameters_.options.copy_dest, "dest");
ExpectNoError(); ExpectNoError();
} }
TEST_F(ParamsTest, ParseFailsOnGameletIpNeedsPort) { TEST_F(ParamsTest, ParseFailsOnInvalidPort) {
const char* argv[] = {"cdc_rsync.exe", "--ip=127.0.0.1", "source", const char* argv[] = {"cdc_rsync.exe", "--port=0", kSrc, kUserHostDst, NULL};
"destination", NULL};
EXPECT_FALSE( EXPECT_FALSE(
Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_)); Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_));
ExpectError("--port must specify a valid port"); ExpectError("--port must specify a valid port");
} }
TEST_F(ParamsTest, ParseFailsOnDeleteNeedsRecursive) { TEST_F(ParamsTest, ParseFailsOnDeleteNeedsRecursive) {
const char* argv[] = { const char* argv[] = {"cdc_rsync.exe", "--delete", kSrc, kUserHostDst, NULL};
"cdc_rsync.exe", "--ip=1.2.3.4", "--port=1234", "--delete",
"source", "destination", NULL};
EXPECT_FALSE( EXPECT_FALSE(
Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_)); Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_));
ExpectError("--delete does not work without --recursive (-r)"); ExpectError("--delete does not work without --recursive (-r)");
@@ -281,10 +326,10 @@ TEST_F(ParamsTest, ParseChecksCompressLevel) {
for (int n = 0; n < std::size(levels); ++n) { for (int n = 0; n < std::size(levels); ++n) {
std::string level = "--compress-level=" + std::to_string(levels[n]); std::string level = "--compress-level=" + std::to_string(levels[n]);
const char* argv[] = {"cdc_rsync.exe", "--ip=1.2.3.4", "--port=1234", const char* argv[] = {"cdc_rsync.exe", level.c_str(), kSrc, kUserHostDst,
level.c_str(), "source", "destination"}; NULL};
EXPECT_TRUE(Parse(static_cast<int>(std::size(argv)) - 1, argv, EXPECT_EQ(Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_),
&parameters_) == valid[n]); valid[n]);
if (valid[n]) { if (valid[n]) {
ExpectNoError(); ExpectNoError();
} else { } else {
@@ -295,94 +340,95 @@ TEST_F(ParamsTest, ParseChecksCompressLevel) {
} }
TEST_F(ParamsTest, ParseFailsOnUnknownKeyValue) { TEST_F(ParamsTest, ParseFailsOnUnknownKeyValue) {
const char* argv[] = {"cdc_rsync.exe", "--unknownKey=5", "source", const char* argv[] = {"cdc_rsync.exe", "--unknownKey=5", kSrc, kUserHostDst,
"destination", NULL}; NULL};
EXPECT_FALSE( EXPECT_FALSE(
Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_)); Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_));
ExpectError("unknownKey"); ExpectError("unknownKey");
} }
TEST_F(ParamsTest, ParseFailsWithHelpOption) { TEST_F(ParamsTest, ParseFailsWithHelpOption) {
const char* argv[] = {"cdc_rsync.exe", "--ip=1.2.3.4", "--port=1234", const char* argv[] = {"cdc_rsync.exe", kSrc, kUserHostDst, NULL};
"source", "destination", NULL};
EXPECT_TRUE(Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_)); EXPECT_TRUE(Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_));
const char* argv2[] = { const char* argv2[] = {"cdc_rsync.exe", kSrc, kUserHostDst, "--help", NULL};
"cdc_rsync.exe", "--ip=1.2.3.4", "--port=1234", "source",
"destination", "--help", NULL};
EXPECT_FALSE( EXPECT_FALSE(
Parse(static_cast<int>(std::size(argv2)) - 1, argv2, &parameters_)); Parse(static_cast<int>(std::size(argv2)) - 1, argv2, &parameters_));
ExpectNoError(); ExpectNoError();
const char* argv3[] = { const char* argv3[] = {"cdc_rsync.exe", kSrc, kUserHostDst, "-h", NULL};
"cdc_rsync.exe", "--ip=1.2.3.4", "--port=1234", "source",
"destination", "-h", NULL};
EXPECT_FALSE( EXPECT_FALSE(
Parse(static_cast<int>(std::size(argv3)) - 1, argv3, &parameters_)); Parse(static_cast<int>(std::size(argv3)) - 1, argv3, &parameters_));
ExpectNoError(); ExpectNoError();
} }
TEST_F(ParamsTest, ParseSucceedsWithIncludeExclude) { TEST_F(ParamsTest, ParseSucceedsWithIncludeExclude) {
const char* argv[] = { const char* argv[] = {"cdc_rsync.exe",
"cdc_rsync.exe", "--ip=1.2.3.4", "--port=1234", "--include=*.txt", "--include=*.txt",
"--exclude", "*.dat", "--include", "*.exe", "--exclude",
"source", "destination", NULL}; "*.dat",
"--include",
"*.exe",
kSrc,
kUserHostDst,
NULL};
EXPECT_TRUE(Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_)); EXPECT_TRUE(Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_));
ASSERT_EQ(parameters_.filter_rules.size(), 3); const std::vector<PathFilter::Rule>& rules =
ASSERT_EQ(parameters_.filter_rules[0].type, FilterRule::Type::kInclude); parameters_.options.filter.GetRules();
ASSERT_EQ(parameters_.filter_rules[0].pattern, "*.txt"); ASSERT_EQ(rules.size(), 3);
ASSERT_EQ(parameters_.filter_rules[1].type, FilterRule::Type::kExclude); ASSERT_EQ(rules[0].type, PathFilter::Rule::Type::kInclude);
ASSERT_EQ(parameters_.filter_rules[1].pattern, "*.dat"); ASSERT_EQ(rules[0].pattern, "*.txt");
ASSERT_EQ(parameters_.filter_rules[2].type, FilterRule::Type::kInclude); ASSERT_EQ(rules[1].type, PathFilter::Rule::Type::kExclude);
ASSERT_EQ(parameters_.filter_rules[2].pattern, "*.exe"); ASSERT_EQ(rules[1].pattern, "*.dat");
ASSERT_EQ(rules[2].type, PathFilter::Rule::Type::kInclude);
ASSERT_EQ(rules[2].pattern, "*.exe");
ExpectNoError(); ExpectNoError();
} }
TEST_F(ParamsTest, FilesFrom_NoFile) { TEST_F(ParamsTest, FilesFrom_NoFile) {
const char* argv[] = { const char* argv[] = {"cdc_rsync.exe", kSrc, kUserHostDst, "--files-from",
"cdc_rsync.exe", "--ip=1.2.3.4", "--port=1234", "source", NULL};
"destination", "--files-from", NULL};
EXPECT_FALSE( EXPECT_FALSE(
Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_)); Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_));
ExpectError(NeedsValueError("files-from")); ExpectError(NeedsValueError("files-from"));
} }
TEST_F(ParamsTest, FilesFrom_ImpliesRelative) { TEST_F(ParamsTest, FilesFrom_ImpliesRelative) {
const char* argv[] = { const char* argv[] = {"cdc_rsync.exe", "--files-from",
"cdc_rsync.exe", "--ip=1.2.3.4", "--port=1234", "--files-from", sources_file_.c_str(), base_dir_.c_str(),
sources_file_.c_str(), base_dir_.c_str(), "destination", NULL}; kUserHostDst, NULL};
EXPECT_TRUE(Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_)); EXPECT_TRUE(Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_));
EXPECT_TRUE(parameters_.options.relative); EXPECT_TRUE(parameters_.options.relative);
ExpectNoError(); ExpectNoError();
} }
TEST_F(ParamsTest, FilesFrom_WithoutSourceArg) { TEST_F(ParamsTest, FilesFrom_WithoutSourceArg) {
const char* argv[] = { const char* argv[] = {"cdc_rsync.exe", "--files-from", sources_file_.c_str(),
"cdc_rsync.exe", "--ip=1.2.3.4", "--port=1234", "--files-from", kUserHostDst, NULL};
sources_file_.c_str(), "destination", NULL};
EXPECT_TRUE(Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_)); EXPECT_TRUE(Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_));
EXPECT_TRUE(parameters_.sources_dir.empty()); EXPECT_TRUE(parameters_.options.sources_dir.empty());
EXPECT_EQ(parameters_.destination, "destination"); EXPECT_EQ(parameters_.user_host, kUserHost);
EXPECT_EQ(parameters_.destination, kDst);
ExpectNoError(); ExpectNoError();
} }
TEST_F(ParamsTest, FilesFrom_WithSourceArg) { TEST_F(ParamsTest, FilesFrom_WithSourceArg) {
const char* argv[] = { const char* argv[] = {"cdc_rsync.exe", "--files-from",
"cdc_rsync.exe", "--ip=1.2.3.4", "--port=1234", "--files-from", sources_file_.c_str(), base_dir_.c_str(),
sources_file_.c_str(), base_dir_.c_str(), "destination", NULL}; kUserHostDst, NULL};
EXPECT_TRUE(Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_)); EXPECT_TRUE(Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_));
std::string expected_sources_dir = base_dir_; std::string expected_sources_dir = base_dir_;
path::EnsureEndsWithPathSeparator(&expected_sources_dir); path::EnsureEndsWithPathSeparator(&expected_sources_dir);
EXPECT_EQ(parameters_.sources_dir, expected_sources_dir); EXPECT_EQ(parameters_.options.sources_dir, expected_sources_dir);
EXPECT_EQ(parameters_.destination, "destination"); EXPECT_EQ(parameters_.user_host, kUserHost);
EXPECT_EQ(parameters_.destination, kDst);
ExpectNoError(); ExpectNoError();
} }
TEST_F(ParamsTest, FilesFrom_ParsesFile) { TEST_F(ParamsTest, FilesFrom_ParsesFile) {
const char* argv[] = { const char* argv[] = {"cdc_rsync.exe", "--files-from", sources_file_.c_str(),
"cdc_rsync.exe", "--ip=1.2.3.4", "--port=1234", "--files-from", kUserHostDst, NULL};
sources_file_.c_str(), "destination", NULL};
EXPECT_TRUE(Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_)); EXPECT_TRUE(Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_));
std::vector<const char*> expected = {"file1", "file2", "file3"}; std::vector<const char*> expected = {"file1", "file2", "file3"};
@@ -394,13 +440,8 @@ TEST_F(ParamsTest, FilesFrom_ParsesFile) {
} }
TEST_F(ParamsTest, FilesFrom_EmptyFile_WithoutSourceArg) { TEST_F(ParamsTest, FilesFrom_EmptyFile_WithoutSourceArg) {
const char* argv[] = {"cdc_rsync.exe", const char* argv[] = {"cdc_rsync.exe", "--files-from",
"--ip=1.2.3.4", empty_sources_file_.c_str(), kUserHostDst, NULL};
"--port=1234",
"--files-from",
empty_sources_file_.c_str(),
"destination",
NULL};
EXPECT_FALSE( EXPECT_FALSE(
Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_)); Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_));
ExpectError(empty_sources_file_); ExpectError(empty_sources_file_);
@@ -408,14 +449,9 @@ TEST_F(ParamsTest, FilesFrom_EmptyFile_WithoutSourceArg) {
} }
TEST_F(ParamsTest, FilesFrom_EmptyFile_WithSourceArg) { TEST_F(ParamsTest, FilesFrom_EmptyFile_WithSourceArg) {
const char* argv[] = {"cdc_rsync.exe", const char* argv[] = {
"--ip=1.2.3.4", "cdc_rsync.exe", "--files-from", empty_sources_file_.c_str(),
"--port=1234", base_dir_.c_str(), kUserHostDst, NULL};
"--files-from",
empty_sources_file_.c_str(),
base_dir_.c_str(),
"destination",
NULL};
EXPECT_FALSE( EXPECT_FALSE(
Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_)); Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_));
ExpectError(empty_sources_file_); ExpectError(empty_sources_file_);
@@ -423,17 +459,16 @@ TEST_F(ParamsTest, FilesFrom_EmptyFile_WithSourceArg) {
} }
TEST_F(ParamsTest, FilesFrom_NoDestination) { TEST_F(ParamsTest, FilesFrom_NoDestination) {
const char* argv[] = {"cdc_rsync.exe", "--ip=1.2.3.4", "--port=1234", const char* argv[] = {"cdc_rsync.exe", "--files-from", sources_file_.c_str(),
"--files-from", sources_file_.c_str(), NULL}; NULL};
EXPECT_FALSE( EXPECT_FALSE(
Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_)); Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_));
ExpectError("Missing destination"); ExpectError("Missing destination");
} }
TEST_F(ParamsTest, IncludeFrom_NoFile) { TEST_F(ParamsTest, IncludeFrom_NoFile) {
const char* argv[] = { const char* argv[] = {"cdc_rsync.exe", kSrc, kUserHostDst, "--include-from",
"cdc_rsync.exe", "--ip=1.2.3.4", "--port=1234", "source", NULL};
"destination", "--include-from", NULL};
EXPECT_FALSE( EXPECT_FALSE(
Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_)); Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_));
ExpectError(NeedsValueError("include-from")); ExpectError(NeedsValueError("include-from"));
@@ -441,20 +476,22 @@ TEST_F(ParamsTest, IncludeFrom_NoFile) {
TEST_F(ParamsTest, IncludeFrom_ParsesFile) { TEST_F(ParamsTest, IncludeFrom_ParsesFile) {
std::string file = path::Join(base_dir_, "include_files.txt"); std::string file = path::Join(base_dir_, "include_files.txt");
const char* argv[] = { const char* argv[] = {"cdc_rsync.exe", "--include-from",
"cdc_rsync.exe", "--ip=1.2.3.4", "--port=1234", "--include-from", file.c_str(), kSrc,
file.c_str(), "source", "destination", NULL}; kUserHostDst, NULL};
EXPECT_TRUE(Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_)); EXPECT_TRUE(Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_));
ASSERT_EQ(parameters_.filter_rules.size(), 1); const std::vector<PathFilter::Rule>& rules =
ASSERT_EQ(parameters_.filter_rules[0].type, FilterRule::Type::kInclude); parameters_.options.filter.GetRules();
ASSERT_EQ(parameters_.filter_rules[0].pattern, "file3"); ASSERT_EQ(rules.size(), 1);
ASSERT_EQ(rules[0].type, PathFilter::Rule::Type::kInclude);
ASSERT_EQ(rules[0].pattern, "file3");
ExpectNoError(); ExpectNoError();
} }
TEST_F(ParamsTest, ExcludeFrom_NoFile) { TEST_F(ParamsTest, ExcludeFrom_NoFile) {
const char* argv[] = {"cdc_rsync.exe", "source", "destination", const char* argv[] = {"cdc_rsync.exe", kSrc, kUserHostDst, "--exclude-from",
"--exclude-from", NULL}; NULL};
EXPECT_FALSE( EXPECT_FALSE(
Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_)); Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_));
ExpectError(NeedsValueError("exclude-from")); ExpectError(NeedsValueError("exclude-from"));
@@ -462,16 +499,18 @@ TEST_F(ParamsTest, ExcludeFrom_NoFile) {
TEST_F(ParamsTest, ExcludeFrom_ParsesFile) { TEST_F(ParamsTest, ExcludeFrom_ParsesFile) {
std::string file = path::Join(base_dir_, "exclude_files.txt"); std::string file = path::Join(base_dir_, "exclude_files.txt");
const char* argv[] = { const char* argv[] = {"cdc_rsync.exe", "--exclude-from",
"cdc_rsync.exe", "--ip=1.2.3.4", "--port=1234", "--exclude-from", file.c_str(), kSrc,
file.c_str(), "source", "destination", NULL}; kUserHostDst, NULL};
EXPECT_TRUE(Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_)); EXPECT_TRUE(Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_));
ASSERT_EQ(parameters_.filter_rules.size(), 2); const std::vector<PathFilter::Rule>& rules =
EXPECT_EQ(parameters_.filter_rules[0].type, FilterRule::Type::kExclude); parameters_.options.filter.GetRules();
EXPECT_EQ(parameters_.filter_rules[0].pattern, "file1"); ASSERT_EQ(rules.size(), 2);
EXPECT_EQ(parameters_.filter_rules[1].type, FilterRule::Type::kExclude); EXPECT_EQ(rules[0].type, PathFilter::Rule::Type::kExclude);
EXPECT_EQ(parameters_.filter_rules[1].pattern, "file2"); EXPECT_EQ(rules[0].pattern, "file1");
EXPECT_EQ(rules[1].type, PathFilter::Rule::Type::kExclude);
EXPECT_EQ(rules[1].pattern, "file2");
ExpectNoError(); ExpectNoError();
} }
@@ -479,31 +518,31 @@ TEST_F(ParamsTest, IncludeExcludeMixed_ProperOrder) {
std::string exclude_file = path::Join(base_dir_, "exclude_files.txt"); std::string exclude_file = path::Join(base_dir_, "exclude_files.txt");
std::string include_file = path::Join(base_dir_, "include_files.txt"); std::string include_file = path::Join(base_dir_, "include_files.txt");
const char* argv[] = {"cdc_rsync.exe", const char* argv[] = {"cdc_rsync.exe",
"--ip=1.2.3.4",
"--port=1234",
"--include-from", "--include-from",
include_file.c_str(), include_file.c_str(),
"--exclude=excl1", "--exclude=excl1",
"source", kSrc,
"--exclude-from", "--exclude-from",
exclude_file.c_str(), exclude_file.c_str(),
"destination", kUserHostDst,
"--include", "--include",
"incl1", "incl1",
NULL}; NULL};
EXPECT_TRUE(Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_)); EXPECT_TRUE(Parse(static_cast<int>(std::size(argv)) - 1, argv, &parameters_));
ASSERT_EQ(parameters_.filter_rules.size(), 5); const std::vector<PathFilter::Rule>& rules =
EXPECT_EQ(parameters_.filter_rules[0].type, FilterRule::Type::kInclude); parameters_.options.filter.GetRules();
EXPECT_EQ(parameters_.filter_rules[0].pattern, "file3"); ASSERT_EQ(rules.size(), 5);
EXPECT_EQ(parameters_.filter_rules[1].type, FilterRule::Type::kExclude); EXPECT_EQ(rules[0].type, PathFilter::Rule::Type::kInclude);
EXPECT_EQ(parameters_.filter_rules[1].pattern, "excl1"); EXPECT_EQ(rules[0].pattern, "file3");
EXPECT_EQ(parameters_.filter_rules[2].type, FilterRule::Type::kExclude); EXPECT_EQ(rules[1].type, PathFilter::Rule::Type::kExclude);
EXPECT_EQ(parameters_.filter_rules[2].pattern, "file1"); EXPECT_EQ(rules[1].pattern, "excl1");
EXPECT_EQ(parameters_.filter_rules[3].type, FilterRule::Type::kExclude); EXPECT_EQ(rules[2].type, PathFilter::Rule::Type::kExclude);
EXPECT_EQ(parameters_.filter_rules[3].pattern, "file2"); EXPECT_EQ(rules[2].pattern, "file1");
EXPECT_EQ(parameters_.filter_rules[4].type, FilterRule::Type::kInclude); EXPECT_EQ(rules[3].type, PathFilter::Rule::Type::kExclude);
EXPECT_EQ(parameters_.filter_rules[4].pattern, "incl1"); EXPECT_EQ(rules[3].pattern, "file2");
EXPECT_EQ(rules[4].type, PathFilter::Rule::Type::kInclude);
EXPECT_EQ(rules[4].pattern, "incl1");
ExpectNoError(); ExpectNoError();
} }

View File

@@ -1,3 +0,0 @@
x64/*
*.log
*.user

View File

@@ -1,44 +0,0 @@
package(default_visibility = [
"//:__subpackages__",
])
cc_binary(
name = "cdc_rsync",
srcs = ["main.cc"],
deps = [
":params",
"//cdc_rsync",
],
)
cc_library(
name = "params",
srcs = ["params.cc"],
hdrs = ["params.h"],
deps = [
"//cdc_rsync",
"@com_github_zstd//:zstd",
"@com_google_absl//absl/status",
],
)
cc_test(
name = "params_test",
srcs = ["params_test.cc"],
data = ["testdata/root.txt"] + glob(["testdata/params/**"]),
deps = [
":params",
"//common:test_main",
"@com_google_googletest//:gtest",
],
)
filegroup(
name = "all_test_sources",
srcs = glob(["*_test.cc"]),
)
filegroup(
name = "all_test_data",
srcs = glob(["testdata/**"]),
)

View File

@@ -1,72 +0,0 @@
// Copyright 2022 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#define WIN32_LEAN_AND_MEAN
#include <windows.h>
#include <string>
#include <vector>
#include "cdc_rsync/cdc_rsync.h"
#include "cdc_rsync_cli/params.h"
#include "common/util.h"
int wmain(int argc, wchar_t* argv[]) {
cdc_ft::params::Parameters parameters;
// Convert args from wide to UTF8 strings.
std::vector<std::string> utf8_str_args;
utf8_str_args.reserve(argc);
for (int i = 0; i < argc; i++) {
utf8_str_args.push_back(cdc_ft::Util::WideToUtf8Str(argv[i]));
}
// Convert args from UTF8 strings to UTF8 c-strings.
std::vector<const char*> utf8_args;
utf8_args.reserve(argc);
for (const auto& utf8_str_arg : utf8_str_args) {
utf8_args.push_back(utf8_str_arg.c_str());
}
if (!cdc_ft::params::Parse(argc, utf8_args.data(), &parameters)) {
return 1;
}
// Convert sources from string-vec to c-str-vec.
std::vector<const char*> sources_ptr;
sources_ptr.reserve(parameters.sources.size());
for (const std::string& source : parameters.sources) {
sources_ptr.push_back(source.c_str());
}
// Convert filter rules from string-structs to c-str-structs.
std::vector<cdc_ft::FilterRule> filter_rules;
filter_rules.reserve(parameters.filter_rules.size());
for (const cdc_ft::params::Parameters::FilterRule& rule :
parameters.filter_rules) {
filter_rules.emplace_back(rule.type, rule.pattern.c_str());
}
const char* error_message = nullptr;
cdc_ft::ReturnCode code = cdc_ft::Sync(
&parameters.options, filter_rules.data(), parameters.filter_rules.size(),
parameters.sources_dir.c_str(), sources_ptr.data(),
parameters.sources.size(), parameters.destination.c_str(),
&error_message);
if (error_message) {
fprintf(stderr, "Error: %s\n", error_message);
}
return static_cast<int>(code);
}

View File

View File

@@ -146,14 +146,14 @@ PathFilter::Rule::Type ToInternalType(
} // namespace } // namespace
GgpRsyncServer::GgpRsyncServer() = default; CdcRsyncServer::CdcRsyncServer() = default;
GgpRsyncServer::~GgpRsyncServer() { CdcRsyncServer::~CdcRsyncServer() {
message_pump_.reset(); message_pump_.reset();
socket_.reset(); socket_.reset();
} }
bool GgpRsyncServer::CheckComponents( bool CdcRsyncServer::CheckComponents(
const std::vector<GameletComponent>& components) { const std::vector<GameletComponent>& components) {
// Components are expected to reside in the same dir as the executable. // Components are expected to reside in the same dir as the executable.
std::string component_dir; std::string component_dir;
@@ -172,7 +172,7 @@ bool GgpRsyncServer::CheckComponents(
return true; return true;
} }
absl::Status GgpRsyncServer::Run(int port) { absl::Status CdcRsyncServer::Run(int port) {
socket_ = std::make_unique<ServerSocket>(); socket_ = std::make_unique<ServerSocket>();
absl::Status status = socket_->StartListening(port); absl::Status status = socket_->StartListening(port);
if (!status.ok()) { if (!status.ok()) {
@@ -205,7 +205,7 @@ absl::Status GgpRsyncServer::Run(int port) {
return absl::OkStatus(); return absl::OkStatus();
} }
absl::Status GgpRsyncServer::Sync() { absl::Status CdcRsyncServer::Sync() {
// First, the client sends us options, e.g. the |destination_| directory. // First, the client sends us options, e.g. the |destination_| directory.
absl::Status status = HandleSetOptions(); absl::Status status = HandleSetOptions();
if (!status.ok()) { if (!status.ok()) {
@@ -281,7 +281,7 @@ absl::Status GgpRsyncServer::Sync() {
return absl::OkStatus(); return absl::OkStatus();
} }
absl::Status GgpRsyncServer::HandleSetOptions() { absl::Status CdcRsyncServer::HandleSetOptions() {
LOG_INFO("Receiving options"); LOG_INFO("Receiving options");
SetOptionsRequest request; SetOptionsRequest request;
@@ -324,7 +324,7 @@ absl::Status GgpRsyncServer::HandleSetOptions() {
return absl::OkStatus(); return absl::OkStatus();
} }
absl::Status GgpRsyncServer::FindFiles() { absl::Status CdcRsyncServer::FindFiles() {
Stopwatch stopwatch; Stopwatch stopwatch;
FileFinder finder; FileFinder finder;
@@ -350,7 +350,7 @@ absl::Status GgpRsyncServer::FindFiles() {
return absl::OkStatus(); return absl::OkStatus();
} }
absl::Status GgpRsyncServer::HandleSendAllFiles() { absl::Status CdcRsyncServer::HandleSendAllFiles() {
std::string current_directory; std::string current_directory;
for (;;) { for (;;) {
@@ -385,7 +385,7 @@ absl::Status GgpRsyncServer::HandleSendAllFiles() {
} }
} }
absl::Status GgpRsyncServer::DiffFiles() { absl::Status CdcRsyncServer::DiffFiles() {
LOG_INFO("Diffing files"); LOG_INFO("Diffing files");
// Be sure to move the data. It can grow quite large with millions of files. // Be sure to move the data. It can grow quite large with millions of files.
@@ -412,7 +412,7 @@ absl::Status GgpRsyncServer::DiffFiles() {
return absl::OkStatus(); return absl::OkStatus();
} }
absl::Status GgpRsyncServer::RemoveExtraneousFilesAndDirs() { absl::Status CdcRsyncServer::RemoveExtraneousFilesAndDirs() {
FileDeleterAndSender deleter(message_pump_.get()); FileDeleterAndSender deleter(message_pump_.get());
// To guarantee that the folders are empty before they are removed, files are // To guarantee that the folders are empty before they are removed, files are
@@ -451,7 +451,7 @@ absl::Status GgpRsyncServer::RemoveExtraneousFilesAndDirs() {
return absl::OkStatus(); return absl::OkStatus();
} }
absl::Status GgpRsyncServer::CreateMissingDirs() { absl::Status CdcRsyncServer::CreateMissingDirs() {
for (const DirInfo& dir : diff_.missing_dirs) { for (const DirInfo& dir : diff_.missing_dirs) {
// Make directory. // Make directory.
std::string path = path::Join(destination_, dir.filepath); std::string path = path::Join(destination_, dir.filepath);
@@ -475,7 +475,7 @@ absl::Status GgpRsyncServer::CreateMissingDirs() {
} }
template <typename T> template <typename T>
absl::Status GgpRsyncServer::SendFileIndices(const char* file_type, absl::Status CdcRsyncServer::SendFileIndices(const char* file_type,
const std::vector<T>& files) { const std::vector<T>& files) {
LOG_INFO("Sending indices of missing files to client"); LOG_INFO("Sending indices of missing files to client");
constexpr char error_fmt[] = "Failed to send indices of %s files."; constexpr char error_fmt[] = "Failed to send indices of %s files.";
@@ -516,7 +516,7 @@ absl::Status GgpRsyncServer::SendFileIndices(const char* file_type,
return absl::OkStatus(); return absl::OkStatus();
} }
absl::Status GgpRsyncServer::HandleSendMissingFileData() { absl::Status CdcRsyncServer::HandleSendMissingFileData() {
if (diff_.missing_files.empty()) { if (diff_.missing_files.empty()) {
return absl::OkStatus(); return absl::OkStatus();
} }
@@ -641,7 +641,7 @@ absl::Status GgpRsyncServer::HandleSendMissingFileData() {
return absl::OkStatus(); return absl::OkStatus();
} }
absl::Status GgpRsyncServer::SyncChangedFiles() { absl::Status CdcRsyncServer::SyncChangedFiles() {
if (diff_.changed_files.empty()) { if (diff_.changed_files.empty()) {
return absl::OkStatus(); return absl::OkStatus();
} }
@@ -729,7 +729,7 @@ absl::Status GgpRsyncServer::SyncChangedFiles() {
return absl::OkStatus(); return absl::OkStatus();
} }
absl::Status GgpRsyncServer::HandleShutdown() { absl::Status CdcRsyncServer::HandleShutdown() {
ShutdownRequest request; ShutdownRequest request;
absl::Status status = absl::Status status =
message_pump_->ReceiveMessage(PacketType::kShutdown, &request); message_pump_->ReceiveMessage(PacketType::kShutdown, &request);
@@ -746,7 +746,7 @@ absl::Status GgpRsyncServer::HandleShutdown() {
return absl::OkStatus(); return absl::OkStatus();
} }
void GgpRsyncServer::Thread_OnPackageReceived(PacketType type) { void CdcRsyncServer::Thread_OnPackageReceived(PacketType type) {
if (type != PacketType::kToggleCompression) { if (type != PacketType::kToggleCompression) {
return; return;
} }

View File

@@ -33,10 +33,10 @@ namespace cdc_ft {
class MessagePump; class MessagePump;
class ServerSocket; class ServerSocket;
class GgpRsyncServer { class CdcRsyncServer {
public: public:
GgpRsyncServer(); CdcRsyncServer();
~GgpRsyncServer(); ~CdcRsyncServer();
// Checks that the gamelet components (cdc_rsync_server binary etc.) are // Checks that the gamelet components (cdc_rsync_server binary etc.) are
// up-to-date by checking their sizes and timestamps. // up-to-date by checking their sizes and timestamps.

View File

@@ -48,7 +48,6 @@ ServerExitCode GetExitCode(const absl::Status& status) {
case Tag::kSocketEof: case Tag::kSocketEof:
// Usually means client disconnected and shut down already. // Usually means client disconnected and shut down already.
case Tag::kDeployServer: case Tag::kDeployServer:
case Tag::kInstancePickerNotAvailableInQuietMode:
case Tag::kConnectionTimeout: case Tag::kConnectionTimeout:
case Tag::kCount: case Tag::kCount:
// Should not happen in server. // Should not happen in server.
@@ -82,7 +81,7 @@ int main(int argc, const char** argv) {
cdc_ft::Log::Initialize( cdc_ft::Log::Initialize(
std::make_unique<cdc_ft::ConsoleLog>(cdc_ft::LogLevel::kWarning)); std::make_unique<cdc_ft::ConsoleLog>(cdc_ft::LogLevel::kWarning));
cdc_ft::GgpRsyncServer server; cdc_ft::CdcRsyncServer server;
if (!server.CheckComponents(components)) { if (!server.CheckComponents(components)) {
return cdc_ft::kServerExitCodeOutOfDate; return cdc_ft::kServerExitCodeOutOfDate;
} }

View File

@@ -51,10 +51,14 @@ class PortManager {
// Reserves a port in the range passed to the constructor. The port is // Reserves a port in the range passed to the constructor. The port is
// released automatically upon destruction if ReleasePort() is not called // released automatically upon destruction if ReleasePort() is not called
// explicitly. // explicitly.
// |timeout_sec| is the timeout for finding available ports on the gamelet // |check_remote| determines whether the remote port should be checked as
// instance. Returns a DeadlineExceeded error if the timeout is exceeded. // well. If false, the check is skipped and a port might be returned that is
// still in use remotely.
// |remote_timeout_sec| is the timeout for finding available ports on the
// remote instance. Not used if |check_remote| is false.
// Returns a DeadlineExceeded error if the timeout is exceeded.
// Returns a ResourceExhausted error if no ports are available. // Returns a ResourceExhausted error if no ports are available.
absl::StatusOr<int> ReservePort(int timeout_sec); absl::StatusOr<int> ReservePort(bool check_remote, int remote_timeout_sec);
// Releases a reserved port. // Releases a reserved port.
absl::Status ReleasePort(int port); absl::Status ReleasePort(int port);

View File

@@ -25,8 +25,8 @@
namespace cdc_ft { namespace cdc_ft {
namespace { namespace {
constexpr int kGameletPort = 12345; constexpr int kSshPort = 12345;
constexpr char kGameletIp[] = "1.2.3.4"; constexpr char kUserHost[] = "user@1.2.3.4";
constexpr char kGuid[] = "f77bcdfe-368c-4c45-9f01-230c5e7e2132"; constexpr char kGuid[] = "f77bcdfe-368c-4c45-9f01-230c5e7e2132";
constexpr int kFirstPort = 44450; constexpr int kFirstPort = 44450;
@@ -38,6 +38,9 @@ constexpr int kTimeoutSec = 1;
constexpr char kLocalNetstat[] = "netstat -a -n -p tcp"; constexpr char kLocalNetstat[] = "netstat -a -n -p tcp";
constexpr char kRemoteNetstat[] = "netstat --numeric --listening --tcp"; constexpr char kRemoteNetstat[] = "netstat --numeric --listening --tcp";
constexpr bool kCheckRemote = true;
constexpr bool kNoCheckRemote = false;
constexpr char kLocalNetstatOutFmt[] = constexpr char kLocalNetstatOutFmt[] =
"TCP 127.0.0.1:50000 127.0.0.1:%i ESTABLISHED"; "TCP 127.0.0.1:50000 127.0.0.1:%i ESTABLISHED";
constexpr char kRemoteNetstatOutFmt[] = constexpr char kRemoteNetstatOutFmt[] =
@@ -53,7 +56,7 @@ class PortManagerTest : public ::testing::Test {
void SetUp() override { void SetUp() override {
Log::Initialize(std::make_unique<ConsoleLog>(LogLevel::kInfo)); Log::Initialize(std::make_unique<ConsoleLog>(LogLevel::kInfo));
remote_util_.SetIpAndPort(kGameletIp, kGameletPort); remote_util_.SetUserHostAndPort(kUserHost, kSshPort);
} }
void TearDown() override { Log::Shutdown(); } void TearDown() override { Log::Shutdown(); }
@@ -70,7 +73,16 @@ TEST_F(PortManagerTest, ReservePortSuccess) {
process_factory_.SetProcessOutput(kLocalNetstat, "", "", 0); process_factory_.SetProcessOutput(kLocalNetstat, "", "", 0);
process_factory_.SetProcessOutput(kRemoteNetstat, "", "", 0); process_factory_.SetProcessOutput(kRemoteNetstat, "", "", 0);
absl::StatusOr<int> port = port_manager_.ReservePort(kTimeoutSec); absl::StatusOr<int> port =
port_manager_.ReservePort(kCheckRemote, kTimeoutSec);
ASSERT_OK(port);
EXPECT_EQ(*port, kFirstPort);
}
TEST_F(PortManagerTest, ReservePortNoRemoteSuccess) {
process_factory_.SetProcessOutput(kLocalNetstat, "", "", 0);
absl::StatusOr<int> port = port_manager_.ReservePort(kNoCheckRemote, 0);
ASSERT_OK(port); ASSERT_OK(port);
EXPECT_EQ(*port, kFirstPort); EXPECT_EQ(*port, kFirstPort);
} }
@@ -83,7 +95,8 @@ TEST_F(PortManagerTest, ReservePortAllLocalPortsTaken) {
process_factory_.SetProcessOutput(kLocalNetstat, local_netstat_out, "", 0); process_factory_.SetProcessOutput(kLocalNetstat, local_netstat_out, "", 0);
process_factory_.SetProcessOutput(kRemoteNetstat, "", "", 0); process_factory_.SetProcessOutput(kRemoteNetstat, "", "", 0);
absl::StatusOr<int> port = port_manager_.ReservePort(kTimeoutSec); absl::StatusOr<int> port =
port_manager_.ReservePort(kCheckRemote, kTimeoutSec);
EXPECT_TRUE(absl::IsResourceExhausted(port.status())); EXPECT_TRUE(absl::IsResourceExhausted(port.status()));
EXPECT_TRUE( EXPECT_TRUE(
absl::StrContains(port.status().message(), "No port available in range")); absl::StrContains(port.status().message(), "No port available in range"));
@@ -97,7 +110,8 @@ TEST_F(PortManagerTest, ReservePortAllRemotePortsTaken) {
process_factory_.SetProcessOutput(kLocalNetstat, "", "", 0); process_factory_.SetProcessOutput(kLocalNetstat, "", "", 0);
process_factory_.SetProcessOutput(kRemoteNetstat, remote_netstat_out, "", 0); process_factory_.SetProcessOutput(kRemoteNetstat, remote_netstat_out, "", 0);
absl::StatusOr<int> port = port_manager_.ReservePort(kTimeoutSec); absl::StatusOr<int> port =
port_manager_.ReservePort(kCheckRemote, kTimeoutSec);
EXPECT_TRUE(absl::IsResourceExhausted(port.status())); EXPECT_TRUE(absl::IsResourceExhausted(port.status()));
EXPECT_TRUE( EXPECT_TRUE(
absl::StrContains(port.status().message(), "No port available in range")); absl::StrContains(port.status().message(), "No port available in range"));
@@ -107,7 +121,8 @@ TEST_F(PortManagerTest, ReservePortLocalNetstatFails) {
process_factory_.SetProcessOutput(kLocalNetstat, "", "", 1); process_factory_.SetProcessOutput(kLocalNetstat, "", "", 1);
process_factory_.SetProcessOutput(kRemoteNetstat, "", "", 0); process_factory_.SetProcessOutput(kRemoteNetstat, "", "", 0);
absl::StatusOr<int> port = port_manager_.ReservePort(kTimeoutSec); absl::StatusOr<int> port =
port_manager_.ReservePort(kCheckRemote, kTimeoutSec);
EXPECT_NOT_OK(port); EXPECT_NOT_OK(port);
EXPECT_TRUE( EXPECT_TRUE(
absl::StrContains(port.status().message(), absl::StrContains(port.status().message(),
@@ -118,7 +133,8 @@ TEST_F(PortManagerTest, ReservePortRemoteNetstatFails) {
process_factory_.SetProcessOutput(kLocalNetstat, "", "", 0); process_factory_.SetProcessOutput(kLocalNetstat, "", "", 0);
process_factory_.SetProcessOutput(kRemoteNetstat, "", "", 1); process_factory_.SetProcessOutput(kRemoteNetstat, "", "", 1);
absl::StatusOr<int> port = port_manager_.ReservePort(kTimeoutSec); absl::StatusOr<int> port =
port_manager_.ReservePort(kCheckRemote, kTimeoutSec);
EXPECT_NOT_OK(port); EXPECT_NOT_OK(port);
EXPECT_TRUE(absl::StrContains(port.status().message(), EXPECT_TRUE(absl::StrContains(port.status().message(),
"Failed to find available ports on instance")); "Failed to find available ports on instance"));
@@ -129,7 +145,8 @@ TEST_F(PortManagerTest, ReservePortRemoteNetstatTimesOut) {
process_factory_.SetProcessNeverExits(kRemoteNetstat); process_factory_.SetProcessNeverExits(kRemoteNetstat);
steady_clock_.AutoAdvance(kTimeoutSec * 2 * 1000); steady_clock_.AutoAdvance(kTimeoutSec * 2 * 1000);
absl::StatusOr<int> port = port_manager_.ReservePort(kTimeoutSec); absl::StatusOr<int> port =
port_manager_.ReservePort(kCheckRemote, kTimeoutSec);
EXPECT_NOT_OK(port); EXPECT_NOT_OK(port);
EXPECT_TRUE(absl::IsDeadlineExceeded(port.status())); EXPECT_TRUE(absl::IsDeadlineExceeded(port.status()));
EXPECT_TRUE(absl::StrContains(port.status().message(), EXPECT_TRUE(absl::StrContains(port.status().message(),
@@ -146,10 +163,14 @@ TEST_F(PortManagerTest, ReservePortMultipleInstances) {
// Port managers use shared memory, so different instances know about each // Port managers use shared memory, so different instances know about each
// other. This would even work if |port_manager_| and |port_manager2| belonged // other. This would even work if |port_manager_| and |port_manager2| belonged
// to different processes, but we don't test that here. // to different processes, but we don't test that here.
EXPECT_EQ(*port_manager_.ReservePort(kTimeoutSec), kFirstPort + 0); EXPECT_EQ(*port_manager_.ReservePort(kCheckRemote, kTimeoutSec),
EXPECT_EQ(*port_manager2.ReservePort(kTimeoutSec), kFirstPort + 1); kFirstPort + 0);
EXPECT_EQ(*port_manager_.ReservePort(kTimeoutSec), kFirstPort + 2); EXPECT_EQ(*port_manager2.ReservePort(kCheckRemote, kTimeoutSec),
EXPECT_EQ(*port_manager2.ReservePort(kTimeoutSec), kFirstPort + 3); kFirstPort + 1);
EXPECT_EQ(*port_manager_.ReservePort(kCheckRemote, kTimeoutSec),
kFirstPort + 2);
EXPECT_EQ(*port_manager2.ReservePort(kCheckRemote, kTimeoutSec),
kFirstPort + 3);
} }
TEST_F(PortManagerTest, ReservePortReusesPortsInLRUOrder) { TEST_F(PortManagerTest, ReservePortReusesPortsInLRUOrder) {
@@ -157,7 +178,7 @@ TEST_F(PortManagerTest, ReservePortReusesPortsInLRUOrder) {
process_factory_.SetProcessOutput(kRemoteNetstat, "", "", 0); process_factory_.SetProcessOutput(kRemoteNetstat, "", "", 0);
for (int n = 0; n < kNumPorts * 2; ++n) { for (int n = 0; n < kNumPorts * 2; ++n) {
EXPECT_EQ(*port_manager_.ReservePort(kTimeoutSec), EXPECT_EQ(*port_manager_.ReservePort(kCheckRemote, kTimeoutSec),
kFirstPort + n % kNumPorts); kFirstPort + n % kNumPorts);
system_clock_.Advance(1000); system_clock_.Advance(1000);
} }
@@ -167,10 +188,11 @@ TEST_F(PortManagerTest, ReleasePort) {
process_factory_.SetProcessOutput(kLocalNetstat, "", "", 0); process_factory_.SetProcessOutput(kLocalNetstat, "", "", 0);
process_factory_.SetProcessOutput(kRemoteNetstat, "", "", 0); process_factory_.SetProcessOutput(kRemoteNetstat, "", "", 0);
absl::StatusOr<int> port = port_manager_.ReservePort(kTimeoutSec); absl::StatusOr<int> port =
port_manager_.ReservePort(kCheckRemote, kTimeoutSec);
EXPECT_EQ(*port, kFirstPort); EXPECT_EQ(*port, kFirstPort);
EXPECT_OK(port_manager_.ReleasePort(*port)); EXPECT_OK(port_manager_.ReleasePort(*port));
port = port_manager_.ReservePort(kTimeoutSec); port = port_manager_.ReservePort(kCheckRemote, kTimeoutSec);
EXPECT_EQ(*port, kFirstPort); EXPECT_EQ(*port, kFirstPort);
} }
@@ -180,10 +202,13 @@ TEST_F(PortManagerTest, ReleasePortOnDestruction) {
auto port_manager2 = std::make_unique<PortManager>( auto port_manager2 = std::make_unique<PortManager>(
kGuid, kFirstPort, kLastPort, &process_factory_, &remote_util_); kGuid, kFirstPort, kLastPort, &process_factory_, &remote_util_);
EXPECT_EQ(*port_manager2->ReservePort(kTimeoutSec), kFirstPort + 0); EXPECT_EQ(*port_manager2->ReservePort(kCheckRemote, kTimeoutSec),
EXPECT_EQ(*port_manager_.ReservePort(kTimeoutSec), kFirstPort + 1); kFirstPort + 0);
EXPECT_EQ(*port_manager_.ReservePort(kCheckRemote, kTimeoutSec),
kFirstPort + 1);
port_manager2.reset(); port_manager2.reset();
EXPECT_EQ(*port_manager_.ReservePort(kTimeoutSec), kFirstPort + 0); EXPECT_EQ(*port_manager_.ReservePort(kCheckRemote, kTimeoutSec),
kFirstPort + 0);
} }
TEST_F(PortManagerTest, FindAvailableLocalPortsSuccess) { TEST_F(PortManagerTest, FindAvailableLocalPortsSuccess) {

View File

@@ -121,7 +121,8 @@ PortManager::~PortManager() {
} }
} }
absl::StatusOr<int> PortManager::ReservePort(int timeout_sec) { absl::StatusOr<int> PortManager::ReservePort(bool check_remote,
int remote_timeout_sec) {
// Find available port on workstation. // Find available port on workstation.
std::unordered_set<int> local_ports; std::unordered_set<int> local_ports;
ASSIGN_OR_RETURN(local_ports, ASSIGN_OR_RETURN(local_ports,
@@ -129,13 +130,16 @@ absl::StatusOr<int> PortManager::ReservePort(int timeout_sec) {
process_factory_, false), process_factory_, false),
"Failed to find available ports on workstation"); "Failed to find available ports on workstation");
// Find available port on remote gamelet. // Find available port on remote instance.
std::unordered_set<int> remote_ports; std::unordered_set<int> remote_ports = local_ports;
ASSIGN_OR_RETURN(remote_ports, if (check_remote) {
ASSIGN_OR_RETURN(
remote_ports,
FindAvailableRemotePorts(first_port_, last_port_, "0.0.0.0", FindAvailableRemotePorts(first_port_, last_port_, "0.0.0.0",
process_factory_, remote_util_, process_factory_, remote_util_,
timeout_sec, false, steady_clock_), remote_timeout_sec, false, steady_clock_),
"Failed to find available ports on instance"); "Failed to find available ports on instance");
}
// Fetch shared memory. // Fetch shared memory.
void* mem; void* mem;

View File

@@ -58,8 +58,8 @@ absl::Status CreatePipeForOverlappedIo(ScopedHandle* pipe_read_end,
ScopedHandle* pipe_write_end) { ScopedHandle* pipe_write_end) {
// We need named pipes for overlapped IO, so create a unique name. // We need named pipes for overlapped IO, so create a unique name.
int id = g_pipe_serial_number++; int id = g_pipe_serial_number++;
std::string pipe_name = absl::StrFormat( std::string pipe_name = absl::StrFormat(R"(\\.\Pipe\CdcIoPipe.%08x.%08x)",
R"(\\.\Pipe\GgpRsyncIoPipe.%08x.%08x)", GetCurrentProcessId(), id); GetCurrentProcessId(), id);
// Set the bInheritHandle flag so pipe handles are inherited. // Set the bInheritHandle flag so pipe handles are inherited.
SECURITY_ATTRIBUTES security_attributes; SECURITY_ATTRIBUTES security_attributes;

View File

@@ -14,10 +14,9 @@
#include "common/remote_util.h" #include "common/remote_util.h"
#include <atomic>
#include <regex> #include <regex>
#include <sstream>
#include "absl/strings/str_cat.h"
#include "absl/strings/str_format.h" #include "absl/strings/str_format.h"
#include "common/path.h" #include "common/path.h"
#include "common/status.h" #include "common/status.h"
@@ -25,32 +24,6 @@
namespace cdc_ft { namespace cdc_ft {
namespace { namespace {
// Escapes command line argument for the Microsoft command line parser in
// preparation for quoting. Double quotes are backslash-escaped. Literal
// backslashes are backslash-escaped if they are followed by a double quote, or
// if they are part of a sequence of backslashes that are followed by a double
// quote.
std::string EscapeForWindows(const std::string& argument) {
std::string str =
std::regex_replace(argument, std::regex(R"(\\*(?=""|$))"), "$1$1");
return std::regex_replace(str, std::regex("\""), "\\\"");
}
// Quotes and escapes a command line argument following the convention
// understood by the Microsoft command line parser.
std::string QuoteArgument(const std::string& argument) {
return absl::StrFormat("\"%s\"", EscapeForWindows(argument));
}
// Quotes and escapes a command line arguments for use in ssh command. The
// argument is first escaped and quoted for Linux using single quotes and then
// it is escaped to be used by the Microsoft command line parser.
std::string QuoteAndEscapeArgumentForSsh(const std::string& argument) {
std::string quoted_argument = absl::StrFormat(
"'%s'", std::regex_replace(argument, std::regex("'"), "'\\''"));
return EscapeForWindows(quoted_argument);
}
// Gets the argument for SSH (reverse) port forwarding, e.g. -L23:localhost:45. // Gets the argument for SSH (reverse) port forwarding, e.g. -L23:localhost:45.
std::string GetPortForwardingArg(int local_port, int remote_port, std::string GetPortForwardingArg(int local_port, int remote_port,
bool reverse) { bool reverse) {
@@ -69,21 +42,29 @@ RemoteUtil::RemoteUtil(int verbosity, bool quiet,
process_factory_(process_factory), process_factory_(process_factory),
forward_output_to_log_(forward_output_to_log) {} forward_output_to_log_(forward_output_to_log) {}
void RemoteUtil::SetIpAndPort(const std::string& gamelet_ip, int ssh_port) { void RemoteUtil::SetUserHostAndPort(std::string user_host, int port) {
gamelet_ip_ = gamelet_ip; user_host_ = std::move(user_host);
ssh_port_ = ssh_port; ssh_port_ = port;
}
void RemoteUtil::SetScpCommand(std::string scp_command) {
scp_command_ = std::move(scp_command);
}
void RemoteUtil::SetSshCommand(std::string ssh_command) {
ssh_command_ = std::move(ssh_command);
} }
absl::Status RemoteUtil::Scp(std::vector<std::string> source_filepaths, absl::Status RemoteUtil::Scp(std::vector<std::string> source_filepaths,
const std::string& dest, bool compress) { const std::string& dest, bool compress) {
absl::Status status = CheckIpPort(); absl::Status status = CheckHostPort();
if (!status.ok()) { if (!status.ok()) {
return status; return status;
} }
std::string source_args; std::string source_args;
for (const std::string& sourceFilePath : source_filepaths) { for (const std::string& sourceFilePath : source_filepaths) {
source_args += QuoteArgument(sourceFilePath) + " "; // Workaround for scp thinking that C is a host in C:\path\to\foo.
source_args += QuoteArgument("//./" + sourceFilePath) + " ";
} }
// -p preserves timestamps. This enables timestamp-based up-to-date checks. // -p preserves timestamps. This enables timestamp-based up-to-date checks.
@@ -91,18 +72,10 @@ absl::Status RemoteUtil::Scp(std::vector<std::string> source_filepaths,
start_info.command = absl::StrFormat( start_info.command = absl::StrFormat(
"%s " "%s "
"%s %s -p -T " "%s %s -p -T "
"-F %s " "-P %i %s "
"-i %s -P %i "
"-oStrictHostKeyChecking=yes "
"-oUserKnownHostsFile=\"\"\"%s\"\"\" %s "
"cloudcast@%s:"
"%s", "%s",
QuoteArgument(sdk_util_.GetScpExePath()), scp_command_, quiet_ || verbosity_ < 2 ? "-q" : "", compress ? "-C" : "",
quiet_ || verbosity_ < 2 ? "-q" : "", compress ? "-C" : "", ssh_port_, source_args, QuoteArgument(user_host_ + ":" + dest));
QuoteArgument(sdk_util_.GetSshConfigPath()),
QuoteArgument(sdk_util_.GetSshKeyFilePath()), ssh_port_,
sdk_util_.GetSshKnownHostsFilePath(), source_args,
QuoteArgument(gamelet_ip_), QuoteAndEscapeArgumentForSsh(dest));
start_info.name = "scp"; start_info.name = "scp";
start_info.forward_output_to_log = forward_output_to_log_; start_info.forward_output_to_log = forward_output_to_log_;
@@ -111,7 +84,7 @@ absl::Status RemoteUtil::Scp(std::vector<std::string> source_filepaths,
absl::Status RemoteUtil::Sync(std::vector<std::string> source_filepaths, absl::Status RemoteUtil::Sync(std::vector<std::string> source_filepaths,
const std::string& dest) { const std::string& dest) {
absl::Status status = CheckIpPort(); absl::Status status = CheckHostPort();
if (!status.ok()) { if (!status.ok()) {
return status; return status;
} }
@@ -123,9 +96,9 @@ absl::Status RemoteUtil::Sync(std::vector<std::string> source_filepaths,
ProcessStartInfo start_info; ProcessStartInfo start_info;
start_info.command = absl::StrFormat( start_info.command = absl::StrFormat(
"%s --ip=%s --port=%i -z %s %s%s", "cdc_rsync --ip=%s --port=%i -z "
path::Join(sdk_util_.GetDevBinPath(), "cdc_rsync"), "%s %s%s",
QuoteArgument(gamelet_ip_), ssh_port_, QuoteArgument(user_host_), ssh_port_,
quiet_ || verbosity_ < 2 ? "-q " : " ", source_args, QuoteArgument(dest)); quiet_ || verbosity_ < 2 ? "-q " : " ", source_args, QuoteArgument(dest));
start_info.name = "cdc_rsync"; start_info.name = "cdc_rsync";
start_info.forward_output_to_log = forward_output_to_log_; start_info.forward_output_to_log = forward_output_to_log_;
@@ -135,16 +108,16 @@ absl::Status RemoteUtil::Sync(std::vector<std::string> source_filepaths,
absl::Status RemoteUtil::Chmod(const std::string& mode, absl::Status RemoteUtil::Chmod(const std::string& mode,
const std::string& remote_path, bool quiet) { const std::string& remote_path, bool quiet) {
std::string remote_command = absl::StrFormat( std::string remote_command =
"chmod %s %s %s", QuoteArgument(mode), absl::StrFormat("chmod %s %s %s", QuoteArgument(mode),
QuoteAndEscapeArgumentForSsh(remote_path), quiet ? "-f" : ""); EscapeForWindows(remote_path), quiet ? "-f" : "");
return Run(remote_command, "chmod"); return Run(remote_command, "chmod");
} }
absl::Status RemoteUtil::Rm(const std::string& remote_path, bool force) { absl::Status RemoteUtil::Rm(const std::string& remote_path, bool force) {
std::string remote_command = absl::StrFormat( std::string remote_command = absl::StrFormat("rm %s %s", force ? "-f" : "",
"rm %s %s", force ? "-f" : "", QuoteAndEscapeArgumentForSsh(remote_path)); EscapeForWindows(remote_path));
return Run(remote_command, "rm"); return Run(remote_command, "rm");
} }
@@ -152,14 +125,14 @@ absl::Status RemoteUtil::Rm(const std::string& remote_path, bool force) {
absl::Status RemoteUtil::Mv(const std::string& old_remote_path, absl::Status RemoteUtil::Mv(const std::string& old_remote_path,
const std::string& new_remote_path) { const std::string& new_remote_path) {
std::string remote_command = std::string remote_command =
absl::StrFormat("mv %s %s", QuoteAndEscapeArgumentForSsh(old_remote_path), absl::StrFormat("mv %s %s", EscapeForWindows(old_remote_path),
QuoteAndEscapeArgumentForSsh(new_remote_path)); EscapeForWindows(new_remote_path));
return Run(remote_command, "mv"); return Run(remote_command, "mv");
} }
absl::Status RemoteUtil::Run(std::string remote_command, std::string name) { absl::Status RemoteUtil::Run(std::string remote_command, std::string name) {
absl::Status status = CheckIpPort(); absl::Status status = CheckHostPort();
if (!status.ok()) { if (!status.ok()) {
return status; return status;
} }
@@ -201,25 +174,37 @@ ProcessStartInfo RemoteUtil::BuildProcessStartInfoForSshInternal(
start_info.command = absl::StrFormat( start_info.command = absl::StrFormat(
"%s " "%s "
"%s -tt " "%s -tt "
"-F %s "
"-i %s "
"-oServerAliveCountMax=6 " // Number of lost msgs before ssh terminates "-oServerAliveCountMax=6 " // Number of lost msgs before ssh terminates
"-oServerAliveInterval=5 " // Time interval between alive msgs "-oServerAliveInterval=5 " // Time interval between alive msgs
"-oStrictHostKeyChecking=yes " "%s %s -p %i %s",
"-oUserKnownHostsFile=\"\"\"%s\"\"\" %s" ssh_command_, quiet_ || verbosity_ < 2 ? "-q" : "", forward_arg,
"cloudcast@%s -p %i %s", QuoteArgument(user_host_), ssh_port_, remote_command_arg);
QuoteArgument(sdk_util_.GetSshExePath()),
quiet_ || verbosity_ < 2 ? "-q" : "",
QuoteArgument(sdk_util_.GetSshConfigPath()),
QuoteArgument(sdk_util_.GetSshKeyFilePath()),
sdk_util_.GetSshKnownHostsFilePath(), forward_arg,
QuoteArgument(gamelet_ip_), ssh_port_, remote_command_arg);
start_info.forward_output_to_log = forward_output_to_log_; start_info.forward_output_to_log = forward_output_to_log_;
return start_info; return start_info;
} }
absl::Status RemoteUtil::CheckIpPort() { std::string RemoteUtil::EscapeForWindows(const std::string& argument) {
if (gamelet_ip_.empty() || ssh_port_ == 0) { std::string str =
std::regex_replace(argument, std::regex(R"(\\*(?="|$))"), "$&$&");
return std::regex_replace(str, std::regex(R"(")"), R"(\")");
}
std::string RemoteUtil::QuoteArgument(const std::string& argument) {
return absl::StrCat("\"", EscapeForWindows(argument), "\"");
}
std::string RemoteUtil::QuoteArgumentForSsh(const std::string& argument) {
return absl::StrFormat(
"'%s'", std::regex_replace(argument, std::regex("'"), "'\\''"));
}
std::string RemoteUtil::QuoteAndEscapeArgumentForSsh(
const std::string& argument) {
return EscapeForWindows(QuoteArgumentForSsh(argument));
}
absl::Status RemoteUtil::CheckHostPort() {
if (user_host_.empty() || ssh_port_ == 0) {
return MakeStatus("IP or port not set"); return MakeStatus("IP or port not set");
} }

View File

@@ -22,7 +22,6 @@
#include "absl/status/status.h" #include "absl/status/status.h"
#include "common/process.h" #include "common/process.h"
#include "common/sdk_util.h"
namespace cdc_ft { namespace cdc_ft {
@@ -30,6 +29,8 @@ namespace cdc_ft {
// Windows-only. // Windows-only.
class RemoteUtil { class RemoteUtil {
public: public:
static constexpr int kDefaultSshPort = 22;
// If |verbosity| is > 0 and |quiet| is false, output from scp, ssh etc. // If |verbosity| is > 0 and |quiet| is false, output from scp, ssh etc.
// commands is shown. // commands is shown.
// If |quiet| is true, scp, ssh etc. commands use quiet mode. // If |quiet| is true, scp, ssh etc. commands use quiet mode.
@@ -38,61 +39,67 @@ class RemoteUtil {
RemoteUtil(int verbosity, bool quiet, ProcessFactory* process_factory, RemoteUtil(int verbosity, bool quiet, ProcessFactory* process_factory,
bool forward_output_to_log); bool forward_output_to_log);
// Returns the initialization status. Should be OK unless in case of some rare // Sets the SSH username and hostname of the remote instance, as well as the
// internal error. Should be checked before accessing any members. // SSH tunnel port. |user_host| must be of the form [user@]host.
const absl::Status& GetInitStatus() const { void SetUserHostAndPort(std::string user_host, int port);
return sdk_util_.GetInitStatus();
}
// Set IP of the remote instance and the ssh tunnel port. // Sets the SCP command binary path and additional arguments, e.g.
void SetIpAndPort(const std::string& gamelet_ip, int ssh_port); // C:\path\to\scp.exe -F <ssh_config> -i <key_file>
// -oStrictHostKeyChecking=yes -oUserKnownHostsFile="""file"""
// By default, searches scp.exe on the path environment variables.
void SetScpCommand(std::string scp_command);
// Sets the SSH command binary path and additional arguments, e.g.
// C:\path\to\ssh.exe -F <ssh_config> -i <key_file>
// -oStrictHostKeyChecking=yes -oUserKnownHostsFile="""file"""
// By default, searches ssh.exe on the path environment variables.
void SetSshCommand(std::string ssh_command);
// Copies |source_filepaths| to the remote folder |dest| on the gamelet using // Copies |source_filepaths| to the remote folder |dest| on the gamelet using
// scp. Must call either InitSsh or SetGameletIp before calling this method. // scp. If |compress| is true, compressed upload is used.
// If |compress| is true, compressed upload is used. // Must call SetUserHostAndPort before calling this method.
absl::Status Scp(std::vector<std::string> source_filepaths, absl::Status Scp(std::vector<std::string> source_filepaths,
const std::string& dest, bool compress); const std::string& dest, bool compress);
// Syncs |source_filepaths| to the remote folder |dest| on the gamelet using // Syncs |source_filepaths| to the remote folder |dest| on the gamelet using
// cdc_rsync. Must call either InitSsh or SetGameletIp before calling this // cdc_rsync. Must call SetUserHostAndPort before calling this method.
// method.
absl::Status Sync(std::vector<std::string> source_filepaths, absl::Status Sync(std::vector<std::string> source_filepaths,
const std::string& dest); const std::string& dest);
// Calls 'chmod |mode| |remote_path|' on the gamelet. // Calls 'chmod |mode| |remote_path|' on the gamelet.
// Must call either InitSsh or SetGameletIp before calling this method. // Must call SetUserHostAndPort before calling this method.
absl::Status Chmod(const std::string& mode, const std::string& remote_path, absl::Status Chmod(const std::string& mode, const std::string& remote_path,
bool quiet = false); bool quiet = false);
// Calls 'rm [-f] |remote_path|' on the gamelet. // Calls 'rm [-f] |remote_path|' on the gamelet.
// Must call either InitSsh or SetGameletIp before calling this method. // Must call SetUserHostAndPort before calling this method.
absl::Status Rm(const std::string& remote_path, bool force); absl::Status Rm(const std::string& remote_path, bool force);
// Calls `mv |old_remote_path| |new_remote_path| on the gamelet. // Calls `mv |old_remote_path| |new_remote_path| on the gamelet.
// Must call either InitSsh or SetGameletIp before calling this method. // Must call SetUserHostAndPort before calling this method.
absl::Status Mv(const std::string& old_remote_path, absl::Status Mv(const std::string& old_remote_path,
const std::string& new_remote_path); const std::string& new_remote_path);
// Runs |remote_command| on the gamelet. The command must be properly escaped. // Runs |remote_command| on the gamelet. The command must be properly escaped.
// |name| is the name of the command displayed in the logs. // |name| is the name of the command displayed in the logs.
// Must call either InitSsh or SetGameletIp before calling this method. // Must call SetUserHostAndPort before calling this method.
absl::Status Run(std::string remote_command, std::string name); absl::Status Run(std::string remote_command, std::string name);
// Builds an ssh command that executes |remote_command| on the gamelet. // Builds an SSH command that executes |remote_command| on the gamelet.
ProcessStartInfo BuildProcessStartInfoForSsh(std::string remote_command); ProcessStartInfo BuildProcessStartInfoForSsh(std::string remote_command);
// Builds an ssh command that runs SSH port forwarding to the gamelet, using // Builds an SSH command that runs SSH port forwarding to the gamelet, using
// the given |local_port| and |remote_port|. // the given |local_port| and |remote_port|.
// If |reverse| is true, sets up reverse port forwarding. // If |reverse| is true, sets up reverse port forwarding.
// Must call either InitSsh or SetGameletIp before calling this method. // Must call SetUserHostAndPort before calling this method.
ProcessStartInfo BuildProcessStartInfoForSshPortForward(int local_port, ProcessStartInfo BuildProcessStartInfoForSshPortForward(int local_port,
int remote_port, int remote_port,
bool reverse); bool reverse);
// Builds an ssh command that executes |remote_command| on the gamelet, using // Builds an SSH command that executes |remote_command| on the gamelet, using
// port forwarding with given |local_port| and |remote_port|. // port forwarding with given |local_port| and |remote_port|.
// If |reverse| is true, sets up reverse port forwarding. // If |reverse| is true, sets up reverse port forwarding.
// Must call either InitSsh or SetGameletIp before calling this method. // Must call SetUserHostAndPort before calling this method.
ProcessStartInfo BuildProcessStartInfoForSshPortForwardAndCommand( ProcessStartInfo BuildProcessStartInfoForSshPortForwardAndCommand(
int local_port, int remote_port, bool reverse, int local_port, int remote_port, bool reverse,
std::string remote_command); std::string remote_command);
@@ -100,9 +107,28 @@ class RemoteUtil {
// Returns whether output is suppressed. // Returns whether output is suppressed.
bool Quiet() const { return quiet_; } bool Quiet() const { return quiet_; }
// Escapes command line argument for the Microsoft command line parser in
// preparation for quoting. Double quotes are backslash-escaped. One or more
// backslashes are backslash-escaped if they are followed by a double quote,
// or if they occur at the end of the string, e.g.
// foo\bar -> foo\bar, foo\ -> foo\\, foo\\"bar -> foo\\\\\"bar.
static std::string EscapeForWindows(const std::string& argument);
// Quotes and escapes a command line argument following the convention
// understood by the Microsoft command line parser.
static std::string QuoteArgument(const std::string& argument);
// Quotes and escapes a command line argument for usage in SSH.
static std::string QuoteArgumentForSsh(const std::string& argument);
// Quotes and escapes a command line arguments for use in SSH command. The
// argument is first escaped and quoted for Linux using single quotes and then
// it is escaped to be used by the Microsoft command line parser.
static std::string QuoteAndEscapeArgumentForSsh(const std::string& argument);
private: private:
// Verifies that both |gamelet_ip_| and |ssh_port_| are set. // Verifies that both || and |ssh_port_| are set.
absl::Status CheckIpPort(); absl::Status CheckHostPort();
// Common code for BuildProcessStartInfoForSsh*. // Common code for BuildProcessStartInfoForSsh*.
ProcessStartInfo BuildProcessStartInfoForSshInternal( ProcessStartInfo BuildProcessStartInfoForSshInternal(
@@ -113,9 +139,10 @@ class RemoteUtil {
ProcessFactory* const process_factory_; ProcessFactory* const process_factory_;
const bool forward_output_to_log_; const bool forward_output_to_log_;
SdkUtil sdk_util_; std::string scp_command_ = "scp";
std::string gamelet_ip_; std::string ssh_command_ = "ssh";
int ssh_port_ = 0; std::string user_host_;
int ssh_port_ = kDefaultSshPort;
}; };
} // namespace cdc_ft } // namespace cdc_ft

View File

@@ -21,11 +21,11 @@
namespace cdc_ft { namespace cdc_ft {
namespace { namespace {
constexpr int kGameletPort = 12345; constexpr int kSshPort = 12345;
constexpr char kGameletPortArg[] = "-p 12345"; constexpr char kSshPortArg[] = "-p 12345";
constexpr char kGameletIp[] = "1.2.3.4"; constexpr char kUserHost[] = "user@example.com";
constexpr char kGameletIpArg[] = "cloudcast@\"1.2.3.4\""; constexpr char kUserHostArg[] = "\"user@example.com\"";
constexpr int kLocalPort = 23456; constexpr int kLocalPort = 23456;
constexpr int kRemotePort = 34567; constexpr int kRemotePort = 34567;
@@ -44,7 +44,7 @@ class RemoteUtilTest : public ::testing::Test {
void SetUp() override { void SetUp() override {
Log::Initialize(std::make_unique<ConsoleLog>(LogLevel::kInfo)); Log::Initialize(std::make_unique<ConsoleLog>(LogLevel::kInfo));
util_.SetIpAndPort(kGameletIp, kGameletPort); util_.SetUserHostAndPort(kUserHost, kSshPort);
} }
void TearDown() override { Log::Shutdown(); } void TearDown() override { Log::Shutdown(); }
@@ -64,42 +64,68 @@ class RemoteUtilTest : public ::testing::Test {
TEST_F(RemoteUtilTest, BuildProcessStartInfoForSsh) { TEST_F(RemoteUtilTest, BuildProcessStartInfoForSsh) {
ProcessStartInfo si = util_.BuildProcessStartInfoForSsh(kCommand); ProcessStartInfo si = util_.BuildProcessStartInfoForSsh(kCommand);
ExpectContains(si.command, ExpectContains(si.command, {"ssh", kSshPortArg, kUserHostArg, kCommand});
{"ssh.exe", "GGP\\ssh\\id", "oStrictHostKeyChecking=yes",
"oUserKnownHostsFile", "known_hosts", kGameletPortArg,
kGameletIpArg, kCommand});
} }
TEST_F(RemoteUtilTest, BuildProcessStartInfoForSshPortForward) { TEST_F(RemoteUtilTest, BuildProcessStartInfoForSshPortForward) {
ProcessStartInfo si = util_.BuildProcessStartInfoForSshPortForward( ProcessStartInfo si = util_.BuildProcessStartInfoForSshPortForward(
kLocalPort, kRemotePort, kRegular); kLocalPort, kRemotePort, kRegular);
ExpectContains(si.command, ExpectContains(si.command,
{"ssh.exe", "GGP\\ssh\\id", "oStrictHostKeyChecking=yes", {"ssh", kSshPortArg, kUserHostArg, kPortForwardingArg});
"oUserKnownHostsFile", "known_hosts", kGameletPortArg,
kGameletIpArg, kPortForwardingArg});
si = util_.BuildProcessStartInfoForSshPortForward(kLocalPort, kRemotePort, si = util_.BuildProcessStartInfoForSshPortForward(kLocalPort, kRemotePort,
kReverse); kReverse);
ExpectContains(si.command, ExpectContains(si.command,
{"ssh.exe", "GGP\\ssh\\id", "oStrictHostKeyChecking=yes", {"ssh", kSshPortArg, kUserHostArg, kReversePortForwardingArg});
"oUserKnownHostsFile", "known_hosts", kGameletPortArg,
kGameletIpArg, kReversePortForwardingArg});
} }
TEST_F(RemoteUtilTest, BuildProcessStartInfoForSshPortForwardAndCommand) { TEST_F(RemoteUtilTest, BuildProcessStartInfoForSshPortForwardAndCommand) {
ProcessStartInfo si = util_.BuildProcessStartInfoForSshPortForwardAndCommand( ProcessStartInfo si = util_.BuildProcessStartInfoForSshPortForwardAndCommand(
kLocalPort, kRemotePort, kRegular, kCommand); kLocalPort, kRemotePort, kRegular, kCommand);
ExpectContains(si.command, ExpectContains(si.command, {"ssh", kSshPortArg, kUserHostArg,
{"ssh.exe", "GGP\\ssh\\id", "oStrictHostKeyChecking=yes", kPortForwardingArg, kCommand});
"oUserKnownHostsFile", "known_hosts", kGameletPortArg,
kGameletIpArg, kPortForwardingArg, kCommand});
si = util_.BuildProcessStartInfoForSshPortForwardAndCommand( si = util_.BuildProcessStartInfoForSshPortForwardAndCommand(
kLocalPort, kRemotePort, kReverse, kCommand); kLocalPort, kRemotePort, kReverse, kCommand);
ExpectContains(si.command, ExpectContains(si.command, {"ssh", kSshPortArg, kUserHostArg,
{"ssh.exe", "GGP\\ssh\\id", "oStrictHostKeyChecking=yes", kReversePortForwardingArg, kCommand});
"oUserKnownHostsFile", "known_hosts", kGameletPortArg, }
kGameletIpArg, kReversePortForwardingArg, kCommand}); TEST_F(RemoteUtilTest, BuildProcessStartInfoForSshWithCustomCommand) {
constexpr char kCustomSshCmd[] = "C:\\path\\to\\ssh.exe --fooarg --bararg=42";
util_.SetSshCommand(kCustomSshCmd);
ProcessStartInfo si = util_.BuildProcessStartInfoForSsh(kCommand);
ExpectContains(si.command, {kCustomSshCmd});
}
TEST_F(RemoteUtilTest, EscapeForWindows) {
EXPECT_EQ("foo", RemoteUtil::EscapeForWindows("foo"));
EXPECT_EQ("foo bar", RemoteUtil::EscapeForWindows("foo bar"));
EXPECT_EQ("foo\\bar", RemoteUtil::EscapeForWindows("foo\\bar"));
EXPECT_EQ("\\\\foo", RemoteUtil::EscapeForWindows("\\\\foo"));
EXPECT_EQ("foo\\\\", RemoteUtil::EscapeForWindows("foo\\"));
EXPECT_EQ("foo\\\\\\\\", RemoteUtil::EscapeForWindows("foo\\\\"));
EXPECT_EQ("foo\\\"", RemoteUtil::EscapeForWindows("foo\""));
EXPECT_EQ("foo\\\"bar", RemoteUtil::EscapeForWindows("foo\"bar"));
EXPECT_EQ("foo\\\\\\\"bar", RemoteUtil::EscapeForWindows("foo\\\"bar"));
EXPECT_EQ("foo\\\\\\\\\\\"bar", RemoteUtil::EscapeForWindows("foo\\\\\"bar"));
EXPECT_EQ("\\\"foo\\\"", RemoteUtil::EscapeForWindows("\"foo\""));
EXPECT_EQ("\\\" \\file.txt", RemoteUtil::EscapeForWindows("\" \\file.txt"));
}
TEST_F(RemoteUtilTest, QuoteArgument) {
EXPECT_EQ("\"foo\"", RemoteUtil::QuoteArgument("foo"));
EXPECT_EQ("\"foo bar\"", RemoteUtil::QuoteArgument("foo bar"));
EXPECT_EQ("\"foo\\bar\"", RemoteUtil::QuoteArgument("foo\\bar"));
EXPECT_EQ("\"\\\\foo\"", RemoteUtil::QuoteArgument("\\\\foo"));
EXPECT_EQ("\"foo\\\\\"", RemoteUtil::QuoteArgument("foo\\"));
EXPECT_EQ("\"foo\\\\\\\\\"", RemoteUtil::QuoteArgument("foo\\\\"));
EXPECT_EQ("\"foo\\\"\"", RemoteUtil::QuoteArgument("foo\""));
EXPECT_EQ("\"foo\\\"bar\"", RemoteUtil::QuoteArgument("foo\"bar"));
EXPECT_EQ("\"foo\\\\\\\"bar\"", RemoteUtil::QuoteArgument("foo\\\"bar"));
EXPECT_EQ("\"foo\\\\\\\\\\\"bar\"",
RemoteUtil::QuoteArgument("foo\\\\\"bar"));
EXPECT_EQ("\"\\\"foo\\\"\"", RemoteUtil::QuoteArgument("\"foo\""));
EXPECT_EQ("\"\\\" \\file.txt\"", RemoteUtil::QuoteArgument("\" \\file.txt"));
} }
} // namespace } // namespace

View File

@@ -31,13 +31,6 @@ SdkUtil::SdkUtil() {
absl::Status status = path::GetEnv("GGP_SDK_PATH", &ggp_sdk_path_env_); absl::Status status = path::GetEnv("GGP_SDK_PATH", &ggp_sdk_path_env_);
if (absl::IsNotFound(status) || ggp_sdk_path_env_.empty()) if (absl::IsNotFound(status) || ggp_sdk_path_env_.empty())
ggp_sdk_path_env_ = path::Join(program_files_path_, "GGP SDK"); ggp_sdk_path_env_ = path::Join(program_files_path_, "GGP SDK");
// Create an empty config file if it does not exist yet.
const std::string ssh_config_path = GetSshConfigPath();
if (!path::Exists(ssh_config_path)) {
init_status_.Update(path::CreateDirRec(path::DirName(ssh_config_path)));
init_status_.Update(path::WriteFile(ssh_config_path, nullptr, 0));
}
} }
SdkUtil::~SdkUtil() = default; SdkUtil::~SdkUtil() = default;
@@ -57,37 +50,8 @@ std::string SdkUtil::GetLogPath(const char* log_base_name) const {
return path::Join(GetUserConfigPath(), "logs", log_base_name + timestamp_ext); return path::Join(GetUserConfigPath(), "logs", log_base_name + timestamp_ext);
} }
std::string SdkUtil::GetSshConfigPath() const {
return path::Join(GetUserConfigPath(), "ssh", "config");
}
std::string SdkUtil::GetSshKeyFilePath() const {
return path::Join(GetUserConfigPath(), "ssh", "id_rsa");
}
std::string SdkUtil::GetSshKnownHostsFilePath() const {
return path::Join(GetUserConfigPath(), "ssh", "known_hosts");
}
std::string SdkUtil::GetSDKPath() const {
assert(init_status_.ok());
return ggp_sdk_path_env_;
}
std::string SdkUtil::GetDevBinPath() const { std::string SdkUtil::GetDevBinPath() const {
return path::Join(GetSDKPath(), "dev", "bin"); return path::Join(ggp_sdk_path_env_, "dev", "bin");
}
std::string SdkUtil::GetSshPath() const {
return path::Join(GetSDKPath(), "tools", "OpenSSH-Win64");
}
std::string SdkUtil::GetSshExePath() const {
return path::Join(GetSshPath(), "ssh.exe");
}
std::string SdkUtil::GetScpExePath() const {
return path::Join(GetSshPath(), "scp.exe");
} }
} // namespace cdc_ft } // namespace cdc_ft

View File

@@ -51,38 +51,10 @@ class SdkUtil {
// %APPDATA%\GGP\logs\log_base_name.20210729-125930.log. // %APPDATA%\GGP\logs\log_base_name.20210729-125930.log.
std::string GetLogPath(const char* log_base_name) const; std::string GetLogPath(const char* log_base_name) const;
// Returns the path of the ssh configuration file, e.g.
// %APPDATA%\GGP\ssh\config.
std::string GetSshConfigPath() const;
// Returns the path of the ssh private key file in the SDK configuration, e.g.
// %APPDATA%\GGP\ssh\id_rsa.
std::string GetSshKeyFilePath() const;
// Returns the path of the ssh known_hosts file in the SDK configuration, e.g.
// %APPDATA%\GGP\ssh\known_hosts.
std::string GetSshKnownHostsFilePath() const;
// Returns the path of the installed SDK, e.g.
// C:\Program Files\GGP SDK.
std::string GetSDKPath() const;
// Returns the path of the dev tools that ship with the SDK, e.g. // Returns the path of the dev tools that ship with the SDK, e.g.
// C:\Program Files\GGP SDK\dev\bin. // C:\Program Files\GGP SDK\dev\bin.
std::string GetDevBinPath() const; std::string GetDevBinPath() const;
// Returns the path of the OpenSSH tools that ship with the SDK, e.g.
// C:\Program Files\GGP SDK\tools\OpenSSH-Win64.
std::string GetSshPath() const;
// Returns the path of ssh.exe that ships with the SDK, e.g.
// C:\Program Files\GGP SDK\tools\OpenSSH-Win64\ssh.exe.
std::string GetSshExePath() const;
// Returns the path of scp.exe that ships with the SDK, e.g.
// C:\Program Files\GGP SDK\tools\OpenSSH-Win64\scp.exe.
std::string GetScpExePath() const;
private: private:
std::string roaming_appdata_path_; std::string roaming_appdata_path_;
std::string program_files_path_; std::string program_files_path_;

View File

@@ -42,13 +42,6 @@ class SdkUtilTest : public ::testing::Test {
protected: protected:
void CheckSdkPaths(const SdkUtil& sdk_util, const std::string& sdk_dir) { void CheckSdkPaths(const SdkUtil& sdk_util, const std::string& sdk_dir) {
EXPECT_EQ(sdk_util.GetSDKPath(), sdk_dir);
EXPECT_EQ(sdk_util.GetSshPath(),
path::Join(sdk_dir, "tools\\OpenSSH-Win64"));
EXPECT_EQ(sdk_util.GetSshExePath(),
path::Join(sdk_dir, "tools\\OpenSSH-Win64\\ssh.exe"));
EXPECT_EQ(sdk_util.GetScpExePath(),
path::Join(sdk_dir, "tools\\OpenSSH-Win64\\scp.exe"));
EXPECT_EQ(sdk_util.GetDevBinPath(), path::Join(sdk_dir, "dev", "bin")); EXPECT_EQ(sdk_util.GetDevBinPath(), path::Join(sdk_dir, "dev", "bin"));
} }
@@ -81,11 +74,6 @@ TEST_F(SdkUtilTest, CheckRoamingAppDataPaths) {
const std::string ggp_path = path::Join(appdata_dir, "GGP"); const std::string ggp_path = path::Join(appdata_dir, "GGP");
EXPECT_EQ(sdk_util.GetUserConfigPath(), ggp_path); EXPECT_EQ(sdk_util.GetUserConfigPath(), ggp_path);
EXPECT_EQ(sdk_util.GetServicesConfigPath(), path::Join(ggp_path, "services")); EXPECT_EQ(sdk_util.GetServicesConfigPath(), path::Join(ggp_path, "services"));
EXPECT_EQ(sdk_util.GetSshConfigPath(), path::Join(ggp_path, "ssh", "config"));
EXPECT_EQ(sdk_util.GetSshKeyFilePath(),
path::Join(ggp_path, "ssh", "id_rsa"));
EXPECT_EQ(sdk_util.GetSshKnownHostsFilePath(),
path::Join(ggp_path, "ssh", "known_hosts"));
} }
TEST_F(SdkUtilTest, CheckSdkPathsWithoutGgpSdkPathEnv) { TEST_F(SdkUtilTest, CheckSdkPathsWithoutGgpSdkPathEnv) {

View File

@@ -77,14 +77,11 @@ enum class Tag : uint8_t {
// The gamelet components need to be re-deployed. // The gamelet components need to be re-deployed.
kDeployServer = 2, kDeployServer = 2,
// Something asks for user input, but we're in quiet mode.
kInstancePickerNotAvailableInQuietMode = 3,
// Timeout while trying to connect to the gamelet component. // Timeout while trying to connect to the gamelet component.
kConnectionTimeout = 4, kConnectionTimeout = 3,
// MUST BE LAST. // MUST BE LAST.
kCount = 5, kCount = 4,
}; };
// Tags a status. No-op if |status| is OK. Overwrites existing tags. // Tags a status. No-op if |status| is OK. Overwrites existing tags.

View File

@@ -4,7 +4,7 @@ VisualStudioVersion = 16.0.31702.278
MinimumVisualStudioVersion = 10.0.40219.1 MinimumVisualStudioVersion = 10.0.40219.1
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "CDC RSync", "CDC RSync", "{74FA49B8-56C3-4F9E-B9D5-35FA1C9A13C8}" Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "CDC RSync", "CDC RSync", "{74FA49B8-56C3-4F9E-B9D5-35FA1C9A13C8}"
EndProject EndProject
Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "cdc_rsync_cli", "cdc_rsync_cli\cdc_rsync_cli.vcxproj", "{3FAC852A-00A8-4CFB-9160-07EFF2B73562}" Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "cdc_rsync", "cdc_rsync\cdc_rsync.vcxproj", "{3FAC852A-00A8-4CFB-9160-07EFF2B73562}"
EndProject EndProject
Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "cdc_rsync_server", "cdc_rsync_server\cdc_rsync_server.vcxproj", "{4ECE65E0-D950-4B96-8AD5-0313261B8C8D}" Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "cdc_rsync_server", "cdc_rsync_server\cdc_rsync_server.vcxproj", "{4ECE65E0-D950-4B96-8AD5-0313261B8C8D}"
EndProject EndProject

View File

@@ -15,24 +15,22 @@ cc_binary(
srcs = [ srcs = [
"//cdc_rsync:all_test_sources", "//cdc_rsync:all_test_sources",
"//cdc_rsync/base:all_test_sources", "//cdc_rsync/base:all_test_sources",
"//cdc_rsync_cli:all_test_sources",
"//cdc_rsync_server:all_test_sources", "//cdc_rsync_server:all_test_sources",
], ],
data = [ data = [
"//cdc_rsync:all_test_data", "//cdc_rsync:all_test_data",
"//cdc_rsync/base:all_test_data", "//cdc_rsync/base:all_test_data",
"//cdc_rsync_cli:all_test_data",
"//cdc_rsync_server:all_test_data", "//cdc_rsync_server:all_test_data",
], ],
deps = [ deps = [
"//cdc_rsync:file_finder_and_sender", "//cdc_rsync:file_finder_and_sender",
"//cdc_rsync:parallel_file_opener", "//cdc_rsync:parallel_file_opener",
"//cdc_rsync:params",
"//cdc_rsync:progress_tracker", "//cdc_rsync:progress_tracker",
"//cdc_rsync:zstd_stream", "//cdc_rsync:zstd_stream",
"//cdc_rsync/base:cdc_interface", "//cdc_rsync/base:cdc_interface",
"//cdc_rsync/base:fake_socket", "//cdc_rsync/base:fake_socket",
"//cdc_rsync/base:message_pump", "//cdc_rsync/base:message_pump",
"//cdc_rsync_cli:params",
"//cdc_rsync_server:file_deleter_and_sender", "//cdc_rsync_server:file_deleter_and_sender",
"//cdc_rsync_server:file_diff_generator", "//cdc_rsync_server:file_diff_generator",
"//cdc_rsync_server:file_finder", "//cdc_rsync_server:file_finder",

View File

@@ -1,91 +0,0 @@
# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""
This is a simple cc_windows_shared_library rule for builing a DLL on Windows
that other cc rules can depend on.
Example useage:
cc_windows_shared_library(
name = "hellolib",
srcs = [
"hello-library.cpp",
],
hdrs = ["hello-library.h"],
# Use this to distinguish compiling vs. linking against the DLL.
copts = ["/DCOMPILING_DLL"],
)
Define COMPILING_DLL to export symbols in the header when compiling the DLL as
follows:
#ifdef COMPILING_DLL
#define DLLEXPORT __declspec(dllexport)
#else
#define DLLEXPORT __declspec(dllimport)
#endif
DLLEXPORT void foo();
For more information and sample usage, see:
https://github.com/bazelbuild/bazel/blob/master/examples/windows/dll/
"""
load("@rules_cc//cc:defs.bzl", "cc_binary", "cc_import", "cc_library")
def cc_windows_shared_library(
name,
srcs = [],
deps = [],
hdrs = [],
visibility = None,
**kwargs):
"""A simple cc_windows_shared_library rule for builing a Windows DLL."""
dll_name = name + ".dll"
import_lib_name = name + "_import_lib"
import_target_name = name + "_dll_import"
# Building a shared library requires a cc_binary with linkshared = 1 set.
cc_binary(
name = dll_name,
srcs = srcs + hdrs,
deps = deps,
linkshared = 1,
**kwargs
)
# Get the import library for the dll
native.filegroup(
name = import_lib_name,
srcs = [":" + dll_name],
output_group = "interface_library",
)
# Because we cannot directly depend on cc_binary from other cc rules in deps attribute,
# we use cc_import as a bridge to depend on the dll.
cc_import(
name = import_target_name,
interface_library = ":" + import_lib_name,
shared_library = ":" + dll_name,
)
# Create a new cc_library to also include the headers needed for the shared library
cc_library(
name = name,
hdrs = hdrs,
visibility = visibility,
deps = deps + [
":" + import_target_name,
],
)