Uses a bazel --disk_cache to cache build outputs between builds. Bazel
also has a local cache, e.g. in ~/.cache/bazel/_bazel_$USER/cache, but
that one can't be used as it won't reuse data across checkouts. A disk
cache is like a remote cache, except that it's on the local disk.
Github first looks for a cache with the given exact key in the current
branch, then in the main branch. If there's a cache hit, the cache
isn't updated (they're read-only!). To prevent that caches become
stale, they are timestamped using the current year and month, so that
the cache is force-renewed every month. Bazel disk caches also just
grow, so this technique prevents that the cache grows indefinitely,
eventually causing cache trashing.
Implements cdc_stream stop-service. Also fixes an issue in the
BackgroundService implementation where Exit() would deadlock since
server shutdown waits for all RPCs to exit.
Starts the streaming service if it's not up and running. This required
adding the ability to run a detached process. By default, all child
processes are killed when the parent process exits. Since detached
child processes don't run with a console, they need to create sub-
processes with CREATE_NO_WINDOW since otherwise a new console pops up,
e.g. for every ssh command.
Polls for 20 seconds while the service starts up. For this purpose,
a BackgroundServiceClient is added. This will be reused in a future CL
by a new stop-service command to exit the service.
Also adds --service-port as additional argument to start-service.
There is a race condition in RecreateWatchedDir where there was a
brief period between the second dir change event and when the file
watcher was actually watching again. If the file was written during
that bried period, it would be missed. The issue could be reproduced
easily by adding a sleep here:
// The watched directory exists and its handle is valid.
if (!first_run) {
++dir_recreate_count_;
if (dir_recreated_cb_) dir_recreated_cb_();
Util::Sleep(1);
}
This CL waits until the watcher is watching again.
Switch asset_stream_manager to use Lyra
Lyra has a nice simple interface, but a few quirks that we work
around, mainly in the BaseCommand class:
- It does not support return values from running a command.
- It does not support return values from a custom arg parser.
- Lyra interprets --bad_arg as positional argument.
Fixes#15
The issue was consistently reproducible by adding a sleep right after starting the process.
Use ping instead of timeout now, because ping doesn't read user input.
The test
FileWatcherTest/FileWatcherParameterizedTest.RecreateWatchedDir/ReadDirectoryChangesExW
is flaky. This CL doesn't fix the root cause, but it fixes the
indefinite spin in GetChangedFiles when there is no file change.
Expand path variables for sync destination
Running commands like cdc_rsync C:\assets\* host:~/assets -vr would create a directory called ~assets. This CL expands path variables properly.
Modifies the create_release workflow in 2 ways:
- It only runs now if something is pushed to main.
- It creates a tagged release if a tag is pushed.
To create a tagged release, run e.g.
git tag -a v0.1.0 -m "Release 0.1.0"
git push origin v0.1.0
Improve readme
This CL adds
- a history section with references to Stadia
- benchmarks
- animated gifs with demos
- a troubleshooting section
- and more info about cdc_stream
* Add a Github action for building and testing
On Windows, -- -//third_party/... doesn't seem to work, so add all test directories manually. Also run the tests_*. We run only fastbuild tests here, since the opt tests will be run in the release workflow.
Also fix a number of compilation and test issues found along the way.
So far, errors from the remote netstat process would only be logged in
the asset stream service, for instance when SSH auth failed. However,
the errors were not shown to the client, and that's the most important
thing.
Also adds some feedback to cdc_stream in case of success.
Implements the cdc_stream client and adjusts asset streaming in
various places to work better outside of a GGP environment.
This CL tries to get quoting for SSH commands right. It also brings
back the ability to start a streaming session from
asset_stream_manager.
Also cleans up Bazel targets setup. Since the sln file is now in root,
it is no longer necessary to prepend ../ to relative filenames to
make clicking on errors work.
Fixes a couple of issues with the FUSE:
- Creates the mount directory if it does not exist.
This assumes the mount dir to be the last arg. Ideally, we'd parse the
command line and then create the directory, but unfortunately
fuse_parse_cmdline already verifies that the dir exists.
- Expands the cache_dir (e.g. ~).
- Fixes a compile issue in manifest_iterator.
Improve cdc_fuse_fs and path
Improves the error handling in path so that std:error_codes are not
assumed to be of system category, and also that their messages are
displayed. Also improves debug messages in GameletComponent.
* Remove dependencies of cdc_sync from GGP
Allows overriding the SSH and SCP commands via command line flags.
Hence, strict host checking, SSH config etc. can be removed since it
is passed in by command line flags for GGP. Also deploys
cdc_rsync_server to ~/.cache/cdc_file_transfer/ and creates that dir
if it does not exist.
* Tweak RemoteUtil
Replaces localhost: by //./ in the workaround for scp since localhost:
had two disadvantages: 1) It required 2 gnubby touches for gLinux and
2) it didn't work for ggp. //./ works for both. Also tweaks quoting,
which didn't quite work for ggp.
* Don't check remote ports in cdc_rsync
Turns off checking remote ports in PortManager. In the future, the
server should return available ports after failing to connect to the
provided port.
Since now the first remote connection is running cdc_rsync_server,
the timeout check has to be done when running that process.
* Remove now-unused kInstancePickerNotAvailableInQuietMode enum
* Add more details to the readme
* [cdc_rsync] Accept [user@]host:destination
Removes the --ip command line argument and assumes user/host are
passed in along with the destination, so it works in the same way as
other popular tools.
* [ggp_rsync] Combine server deploy commands
Combines two chmod and one mv command into one ssh command. This makes
deploy a bit quicker, especially if each ssh command involves touching
your gnubby.
* Remove GGP specific stuff from VS build commands
* [cdc_rsync] Get rid of cdc_rsync.dll
Compile the CDC RSync client as a static library instead. This removes
quite a bit of boiler plate and makes string handling easier since
we can now pass std::strings instead of const chars.
Also fixes an issue where we were sometimes trying to assign nullptr
to std::strings, which is forbidden.
* Allow specifying ssh/scp commands with env vars
* Rename GgpRsync* to CdcRsync*
* Merge ggp_rsync_cli into ggp_rsync
* [cdc_rsync] Refactor cdc_rsync.cc/h
Merges cdc_rsync.cc/h with main.cc and CdcRsyncClient since code is
closer to where it's being used and should be more readable.
Lutz Justen