mirror of
https://github.com/nestriness/cdc-file-transfer.git
synced 2026-01-30 14:35:37 +02:00
76bbdb01bb
This change introduces dynamic manifest updates to asset streaming. Asset streaming describes the directory to be streamed in a manifest, which is a proto definition of all content metadata. This information is sufficient to answer `stat` and `readdir` calls in the FUSE layer without additional round-trips to the workstation. When a directory is streamed for the first time, the corresponding manifest is created in two steps: 1. The directory is traversed recursively and the inode information of all contained files and directories is written to the manifest. 2. The content of all identified files is processed to generate each file's chunk list. This list is part of the definition of a file in the manifest. * The chunk boundaries are identified using our implementation of the FastCDC algorithm. * The hash of each chunk is calculated using the BLAKE3 hash function. * The length and hash of each chunk is appended to the file's chunk list. Prior to this change, when the user mounted a workstation directory on a client, the asset streaming server pushed an intermediate manifest to the gamelet as soon as step 1 was completed. At this point, the FUSE client started serving the virtual file system and was ready to answer `stat` and `readdir` calls. In case the FUSE client received any call that required file contents, such as `read`, it would block the caller until the server completed step 2 above and pushed the final manifest to the client. This works well for large directories (> 100GB) with a reasonable number of files (< 100k). But when dealing with millions of tiny files, creating the full manifest can take several minutes. With this change, we introduce dynamic manifest updates. When the FUSE layer receives an `open` or `readdir` request for a file or directory that is incomplete, it sends an RPC to the workstation about what information is missing from the manifest. The workstation identifies the corresponding file chunker or directory scanner tasks and moves them to the front of the queue. As soon as the task is completed, the workstation pushes an updated intermediate manifest to the client which now includes the information to serve the FUSE request. The queued FUSE request is resumed and returns the result to the caller. While this does not reduce the required time to build the final manifest, it splits up the work into smaller tasks. This allows us to interrupt the current work and prioritize those tasks which are required to handle an incoming request from the client. While this still takes a round-trip to the workstation plus the processing time for the task, an updated manifest is received within a few seconds, which is much better than blocking for several minutes. This latency is only visible when serving data while the manifest is still being created. The situation improves as the manifest creation on the workstation progresses. As soon as the final manifest is pushed, all metadata can be served directly without having to wait for pending tasks.
97 lines
3.7 KiB
C++
97 lines
3.7 KiB
C++
/*
|
|
* Copyright 2022 Google LLC
|
|
*
|
|
* Licensed under the Apache License, Version 2.0 (the "License");
|
|
* you may not use this file except in compliance with the License.
|
|
* You may obtain a copy of the License at
|
|
*
|
|
* http://www.apache.org/licenses/LICENSE-2.0
|
|
*
|
|
* Unless required by applicable law or agreed to in writing, software
|
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
* See the License for the specific language governing permissions and
|
|
* limitations under the License.
|
|
*/
|
|
|
|
#ifndef ASSET_STREAM_MANAGER_ASSET_STREAM_SERVER_H_
|
|
#define ASSET_STREAM_MANAGER_ASSET_STREAM_SERVER_H_
|
|
#include <memory>
|
|
#include <string>
|
|
|
|
#include "absl/status/status.h"
|
|
#include "absl/time/time.h"
|
|
#include "manifest/manifest_proto_defs.h"
|
|
|
|
namespace cdc_ft {
|
|
|
|
// Handles an event when content is transmitted from the workstation to a
|
|
// gamelet.
|
|
// |byte_count| number of bytes transferred during the session so far.
|
|
// |chunk_count| number of chunks transferred during the session so far.
|
|
// |instance_id| instance id, which identifies the session.
|
|
using ContentSentHandler = std::function<void(
|
|
size_t byte_count, size_t chunk_count, std::string instance_id)>;
|
|
|
|
// Handles requests to prioritize the given list of assets while updating the
|
|
// manifest. |rel_paths| relative Unix paths of assets to prioritize.
|
|
using PrioritizeAssetsHandler =
|
|
std::function<void(std::vector<std::string> rel_paths)>;
|
|
|
|
class DataStoreReader;
|
|
class FileChunkMap;
|
|
|
|
enum class AssetStreamServerType { kGrpc, kTest };
|
|
|
|
class AssetStreamServer {
|
|
public:
|
|
// Returns AssetStreamServer of |type|.
|
|
// |src_dir| is the directory on the workstation to mount.
|
|
// |data_store_reader| is responsible for loading content by ID.
|
|
// |file_chunks| is used for mapping data chunk ids to file locations.
|
|
// |content_sent| handles event when data is transferred from the workstation
|
|
// to a gamelet.
|
|
static std::unique_ptr<AssetStreamServer> Create(
|
|
AssetStreamServerType type, std::string src_dir,
|
|
DataStoreReader* data_store_reader, FileChunkMap* file_chunks,
|
|
ContentSentHandler content_sent, PrioritizeAssetsHandler prio_assets);
|
|
|
|
AssetStreamServer(const AssetStreamServer& other) = delete;
|
|
AssetStreamServer& operator=(const AssetStreamServer& other) = delete;
|
|
virtual ~AssetStreamServer() = default;
|
|
|
|
// Starts the asset stream server on the given |port|.
|
|
// Asserts that the server is not yet running.
|
|
virtual absl::Status Start(int port) = 0;
|
|
|
|
// Sets |manifest_id| to be distributed to gamelets.
|
|
// Thread-safe.
|
|
virtual void SetManifestId(const ContentIdProto& manifest_id) = 0;
|
|
|
|
// Waits until the FUSE for the given |instance| id has acknowledged the
|
|
// reception of the currently set manifest id. Returns a DeadlineExceeded
|
|
// error if the ack is not received within the given |timeout|.
|
|
// Thread-safe.
|
|
virtual absl::Status WaitForManifestAck(const std::string& instance,
|
|
absl::Duration timeout) = 0;
|
|
|
|
// Stops internal services and waits for the server to shut down.
|
|
virtual void Shutdown() = 0;
|
|
|
|
// Returns the used manifest id.
|
|
// Thread-safe.
|
|
virtual ContentIdProto GetManifestId() const = 0;
|
|
|
|
protected:
|
|
// Creates a new asset streaming server.
|
|
// |src_dir| is the directory on the workstation to mount.
|
|
// |data_store_reader| is responsible for loading content by ID.
|
|
// |file_chunks| is used for mapping data chunk ids to file locations.
|
|
AssetStreamServer(std::string src_dir, DataStoreReader* data_store_reader,
|
|
FileChunkMap* file_chunks);
|
|
};
|
|
|
|
} // namespace cdc_ft
|
|
|
|
#endif // ASSET_STREAM_MANAGER_ASSET_STREAM_SERVER_H_
|