github-mirrors/netris-cdc-file-transfer
1
0
Fork 0
mirror of https://github.com/nestriness/cdc-file-transfer.git synced 2026-01-30 12:25:35 +02:00
Code Issues Packages Projects Releases Wiki Activity

Merge dynamic manifest updates to Github (#7)

Browse Source 
This change introduces dynamic manifest updates to asset streaming.

Asset streaming describes the directory to be streamed in a manifest, which is a proto definition of all content metadata. This information is sufficient to answer `stat` and `readdir` calls in the FUSE layer without additional round-trips to the workstation.

When a directory is streamed for the first time, the corresponding manifest is created in two steps:
1. The directory is traversed recursively and the inode information of all contained files and directories is written to the manifest.
2. The content of all identified files is processed to generate each file's chunk list. This list is part of the definition of a file in the manifest.
  * The chunk boundaries are identified using our implementation of the FastCDC algorithm.
  * The hash of each chunk is calculated using the BLAKE3 hash function.
  * The length and hash of each chunk is appended to the file's chunk list.

Prior to this change, when the user mounted a workstation directory on a client, the asset streaming server pushed an intermediate manifest to the gamelet as soon as step 1 was completed. At this point, the FUSE client started serving the virtual file system and was ready to answer `stat` and `readdir` calls. In case the FUSE client received any call that required file contents, such as `read`, it would block the caller until the server completed step 2 above and pushed the final manifest to the client. This works well for large directories (> 100GB) with a reasonable number of files (< 100k). But when dealing with millions of tiny files, creating the full manifest can take several minutes.

With this change, we introduce dynamic manifest updates. When the FUSE layer receives an `open` or `readdir` request for a file or directory that is incomplete, it sends an RPC to the workstation about what information is missing from the manifest. The workstation identifies the corresponding file chunker or directory scanner tasks and moves them to the front of the queue. As soon as the task is completed, the workstation pushes an updated intermediate manifest to the client which now includes the information to serve the FUSE request. The queued FUSE request is resumed and returns the result to the caller.

While this does not reduce the required time to build the final manifest, it splits up the work into smaller tasks. This allows us to interrupt the current work and prioritize those tasks which are required to handle an incoming request from the client. While this still takes a round-trip to the workstation plus the processing time for the task, an updated manifest is received within a few seconds, which is much better than blocking for several minutes. 

This latency is only visible when serving data while the manifest is still being created. The situation improves as the manifest creation on the workstation progresses. As soon as the final manifest is pushed, all metadata can be served directly without having to wait for pending tasks.
This commit is contained in:
chrschng
2022-11-16 11:20:32 +01:00
committed by GitHub
parent 23fcd5ef1d
commit 76bbdb01bb
32 changed files with 1286 additions and 411 deletions
Download Patch File Download Diff File Expand all files Collapse all files

43
common/file_watcher_win.cc
View File

@@ -14,7 +14,9 @@
#include "common/file_watcher_win.h"
#define WIN32_LEAN_AND_MEAN
#ifndef WIN32_LEAN_AND_MEAN
#define WIN32_LEAN_AND_MEAN 1
#endif
#include <windows.h>
#include <atomic>
@@ -98,7 +100,7 @@ class AsyncFileWatcher {
~AsyncFileWatcher() { Shutdown(); }
absl::Status GetStatus() ABSL_LOCKS_EXCLUDED(status_mutex_) const {
absl::Status GetStatus() const ABSL_LOCKS_EXCLUDED(status_mutex_) {
absl::MutexLock mutex(&status_mutex_);
return status_;
}
@@ -145,24 +147,24 @@ class AsyncFileWatcher {
modified_files_.clear();
}
uint32_t GetEventCount() ABSL_LOCKS_EXCLUDED(modified_files_mutex_) const {
uint32_t GetEventCount() const ABSL_LOCKS_EXCLUDED(modified_files_mutex_) {
absl::MutexLock mutex(&modified_files_mutex_);
return event_count_;
}
uint32_t GetDirRecreateEventCount()
ABSL_LOCKS_EXCLUDED(modified_files_mutex_) const {
uint32_t GetDirRecreateEventCount() const
ABSL_LOCKS_EXCLUDED(modified_files_mutex_) {
absl::MutexLock mutex(&modified_files_mutex_);
return dir_recreate_count_;
}
bool IsWatching() ABSL_LOCKS_EXCLUDED(state_mutex) const {
bool IsWatching() const ABSL_LOCKS_EXCLUDED(state_mutex_) {
absl::MutexLock mutex(&state_mutex_);
return state_ != FileWatcherState::kDefault &&
state_ != FileWatcherState::kShuttingDown;
}
bool IsShuttingDown() ABSL_LOCKS_EXCLUDED(state_mutex) const {
bool IsShuttingDown() const ABSL_LOCKS_EXCLUDED(state_mutex_) {
absl::MutexLock mutex(&state_mutex_);
return state_ == FileWatcherState::kShuttingDown;
}
@@ -186,7 +188,7 @@ class AsyncFileWatcher {
void WatchDirChanges() {
// TODO: Adjust also if there was no directory at the beginning. Currently,
// the directory exists; otherwise, ManifestUpdater would fail.
bool prev_dir_exists = true;
bool first_run = true, prev_run_was_success = false;
while (true) {
ScopedHandle read_event(CreateEvent(nullptr, /* no security attributes */
TRUE, /* manual-reset event */
@@ -200,27 +202,30 @@ class AsyncFileWatcher {
FILE_BASIC_INFO dir_info;
absl::StatusOr<ScopedHandle> status = GetValidDirHandle(&dir_info);
if (!status.ok()) {
SetStatus(status.status());
} else {
SetStatus(status.status());
if (status.ok()) {
// The watched directory exists and its handle is valid.
if (!prev_dir_exists) {
if (!first_run) {
++dir_recreate_count_;
prev_dir_exists = true;
SetStatus(absl::OkStatus());
if (dir_recreated_cb_) dir_recreated_cb_();
}
first_run = false;
prev_run_was_success = true;
// Keep reading directory changes. This function only returns once it
// gets the shutdown signal, the watched directory is removed, or an
// error occurs while reading file changes.
ReadDirChanges(*status, dir_info, read_event);
if (IsShuttingDown()) {
LOG_DEBUG("Shutting down watching '%s'.", dir_path_);
return;
}
LOG_WARNING("Watched directory '%s' was possibly removed.", dir_path_);
++dir_recreate_count_;
ClearModifiedFiles();
} else if (prev_run_was_success) {
prev_run_was_success = false;
++dir_recreate_count_;
if (dir_recreated_cb_) dir_recreated_cb_();
}
prev_dir_exists = false;
// The shutdown event should be caught on both levels: when the
// watched directory was not removed and when it was recreated. Here
// the shutdown event is considered when the watched directory itself was
@@ -530,7 +535,7 @@ class AsyncFileWatcher {
std::thread dir_reader_; // watching thread.
mutable absl::Mutex status_mutex_;
absl::Status status_ = absl::OkStatus() ABSL_GUARDED_BY(status_mutex_);
absl::Status status_ ABSL_GUARDED_BY(status_mutex_);
mutable absl::Mutex modified_files_mutex_;
FileMap modified_files_ ABSL_GUARDED_BY(modified_files_mutex_);
@@ -538,8 +543,8 @@ class AsyncFileWatcher {
uint32_t dir_recreate_count_ ABSL_GUARDED_BY(modified_files_mutex_) = 0;
mutable absl::Mutex state_mutex_;
FileWatcherState state_ = FileWatcherState::kDefault ABSL_GUARDED_BY(
state_mutex_); // the current watcher state.
FileWatcherState state_ ABSL_GUARDED_BY(state_mutex_) =
FileWatcherState::kDefault; // the current watcher state.
// Pointer to ReadDirectoryChangesExW function if available.
decltype(ReadDirectoryChangesExW)* read_directory_changes_ex_ = nullptr;

17
common/file_watcher_win_test.cc
View File

@@ -115,9 +115,8 @@ class FileWatcherParameterizedTest : public ::testing::TestWithParam<bool> {
bool changed = false;
do {
auto cond = [this]() { return files_changed_; };
files_changed_mutex_.AwaitWithTimeout(absl::Condition(&cond),
kWaitTimeout);
changed = files_changed_;
changed = files_changed_mutex_.AwaitWithTimeout(absl::Condition(&cond),
kWaitTimeout);
files_changed_ = false;
} while (changed && watcher_.GetEventCountForTesting() < min_event_count);
return changed;
@@ -128,9 +127,8 @@ class FileWatcherParameterizedTest : public ::testing::TestWithParam<bool> {
bool changed = false;
do {
auto cond = [this]() { return dir_recreated_; };
files_changed_mutex_.AwaitWithTimeout(absl::Condition(&cond),
kWaitTimeout);
changed = dir_recreated_;
changed = files_changed_mutex_.AwaitWithTimeout(absl::Condition(&cond),
kWaitTimeout);
dir_recreated_ = false;
} while (changed &&
watcher_.GetDirRecreateEventCountForTesting() < min_event_count);
@@ -511,7 +509,10 @@ TEST_P(FileWatcherParameterizedTest, ModifiedTime) {
TEST_P(FileWatcherParameterizedTest, DeleteWatchedDir) {
EXPECT_OK(watcher_.StartWatching([this]() { OnFilesChanged(); },
[this]() { OnDirRecreated(); }));
[this]() { OnDirRecreated(); }, kFWTimeout));
EXPECT_OK(path::WriteFile(first_file_path_, kFirstData, kFirstDataSize));
EXPECT_TRUE(WaitForChange(2u)); // 2x modify
EXPECT_OK(path::RemoveDirRec(watcher_dir_path_));
EXPECT_TRUE(WaitForDirRecreated(1u));
@@ -586,6 +587,8 @@ TEST_P(FileWatcherParameterizedTest, RecreateWatchedDirNoOldChanges) {
EXPECT_OK(path::WriteFile(first_file_path_, kFirstData, kFirstDataSize));
EXPECT_OK(path::RemoveDirRec(watcher_dir_path_));
EXPECT_TRUE(WaitForDirRecreated(1u));
EXPECT_OK(path::CreateDirRec(watcher_dir_path_));
EXPECT_TRUE(WaitForDirRecreated(2u));