[cdc_rsync] Improve throughput for local copies (#74)

On Windows, fclose() seems to be very expensive for large files, where
closing a 1 GB file takes up to 5 seconds. This CL calls fclose() in
background threads. This tremendously improves local syncs, e.g.
copying a 4.5 GB, 300 files data set takes only 7 seconds instead of
30 seconds.

Also increases the buffer size for copying from 16K to 128K (better
throughput for local copies), and adds a timestamp to debug and
verbose console logs (useful when comparing client and server logs).
This commit is contained in:
Lutz Justen
2023-01-31 16:33:03 +01:00
committed by GitHub
parent 1200b34316
commit 5a909bb443
9 changed files with 275 additions and 73 deletions

View File

@@ -623,7 +623,7 @@ absl::Status CdcRsyncClient::SendMissingFiles() {
ParallelFileOpener file_opener(&files_, missing_file_indices_);
constexpr size_t kBufferSize = 16000;
constexpr size_t kBufferSize = 128 * 1024;
for (uint32_t server_index = 0; server_index < missing_file_indices_.size();
++server_index) {
uint32_t client_index = missing_file_indices_[server_index];