Add FUSE Passthrough Support in Stargz-Snapshotter #1867 by wswsmao · Pull Request #1868 · containerd/stargz-snapshotter

wswsmao · 2024-11-19T04:59:15Z

Here’s a proposed implementation plan for FUSE passthrough:

During the Open phase, attempt to pre-read the entire file instead of reading it in chunks.
Utilize the existing cache's Get method to retrieve the fd of the cached file that has been written to local storage.
Implement the FilePassthroughFder interface in node.file, allowing the fd from step 2 to be registered with the kernel via go-fuse.

By following this approach, subsequent Read operations would not need to return to user space, and go-fuse would release the registered information when necessary.

for details,
#1867

Signed-off-by: abushwang <abushwangs@gmail.com>

ktock

Thanks. Could you fix linter errors? https://github.com/containerd/stargz-snapshotter/actions/runs/11906582478/job/33178914643?pr=1868

ktock · 2024-11-19T07:12:16Z

 	EntryTimeout int64 `toml:"entry_timeout"`
+
+	// PassThrough indicates whether to enable FUSE passthrough mode to improve local file read performance. Default is false.
+	PassThrough bool `toml:"passthrough"`


Can we test this in our CI?

ktock · 2024-11-19T07:13:18Z

+		b.Reset()
+		b.Grow(int(chunkSize))
+		ip := b.Bytes()[:chunkSize]
+		if _, err := sf.fr.ReadAt(ip, chunkOffset); err != nil && err != io.EOF {


Can we check if the data can be read from the cache?

ktock · 2024-11-19T07:15:08Z

 type Reader interface {
 	io.ReaderAt
 	Close() error
+	GetReaderAt() io.ReaderAt


Can we add a comment like If a blob is backed by a file, it should return *os.File so that it can be used for FUSE passthrough.?

ktock · 2024-11-19T07:16:03Z

+				n.fs.s.report(fmt.Errorf("node.Open: %v", err))
+				return nil, 0, syscall.EIO


Can we continue opening this as a non-passthrough file, instead of returning EIO?

ktock · 2024-11-19T07:19:23Z

 type Reader interface {
 	io.ReaderAt
 	Close() error
+	GetReaderAt() io.ReaderAt


When FUSE passthrough is enabled, we should always set directoryCache.direct to true so that we can ensure that *directoryCache.Get always return *os.File (not a buffer).

wswsmao · 2024-11-20T07:48:15Z

all done @ktock

wswsmao · 2024-11-20T11:14:12Z

This is my benchmark
https://github.com/wswsmao/fuse-performance

I create three sets of files with sizes of 50, 75, and 100MB respectively.
Using these, I've created a test image named abushwang/ocs9:fuseperf-orig.

Subsequently, I used the default value for the --estargz-chunk-size parameter to create an estargz image.

$ nerdctl image convert --estargz --oci abushwang/ocs9:fuseperf-orig abushwang/ocs9:fuseperf-esgz

To ensure a clean environment for each test, I removed the local image and cache, and then restarted the containerd-stargz-grpc service.

$ rm -rf /var/lib/containerd-stargz-grpc/stargz/fscache/*
$ nerdctl images -q | xargs nerdctl rmi -f"

This is the report:

no passthrough

$ nerdctl run --rm -t --snapshotter=stargz abushwang/ocs9:fuseperf-esgz
docker.io/abushwang/ocs9:fuseperf-esgz:                                           resolved       |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:13868ec859962d0ae07afe0db9f48208e54f38afbaa337e4237aa1e5831c24c0: done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:0abfbeeb608160b63cc9e9e3c351c639b1a53a0bf99a27109d96adc4a75e3c97:   done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 7.1 s                                                                    total:  5.1 Ki (740.0 B/s)                                       
Running file access tests...
Testing file: large_files/large_file_1
Testing file: large_files/large_file_2
Testing file: large_files/large_file_3
Performance Report
==================
File: large_file_1
  Sequential Read Time: 8329.96 ms
--------------------------
File: large_file_2
  Sequential Read Time: 11704.64 ms
--------------------------
File: large_file_3
  Sequential Read Time: 15271.63 ms
--------------------------
Tests completed. See performance_report.txt for details.

with passthrough

$ nerdctl run --rm -t --snapshotter=stargz abushwang/ocs9:fuseperf-esgz
docker.io/abushwang/ocs9:fuseperf-esgz:                                           resolved       |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:13868ec859962d0ae07afe0db9f48208e54f38afbaa337e4237aa1e5831c24c0: done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:0abfbeeb608160b63cc9e9e3c351c639b1a53a0bf99a27109d96adc4a75e3c97:   done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 7.1 s                                                                    total:  5.1 Ki (740.0 B/s)                                       
Running file access tests...
Testing file: large_files/large_file_1
Testing file: large_files/large_file_2
Testing file: large_files/large_file_3
Performance Report
==================
File: large_file_1
  Sequential Read Time: 344.62 ms
--------------------------
File: large_file_2
  Sequential Read Time: 518.38 ms
--------------------------
File: large_file_3
  Sequential Read Time: 667.25 ms
--------------------------
Tests completed. See performance_report.txt for details.

env:

$ uname -r
6.11.5-200.fc40.x86_64

ktock · 2024-11-20T14:18:29Z

+// isFusePthEnable prevents users from enabling passthrough mode on unsupported kernel versions
+func isFusePthEnable() (bool, error) {
+	cmd := exec.Command("sh", "-c", "grep 'CONFIG_FUSE_PASSTHROUGH=y' /boot/config-$(uname -r)")


nit: Let's try not to rely on the shell commands for now. Instead of having this check, let's put a document about how to check if passthrough is a supported on the node (maybe in the following PRs)

Signed-off-by: abushwang <abushwangs@gmail.com>

ktock

Thanks

ujjwal · 2024-12-02T07:23:44Z

Does the kernel version need to be updated to 6.9 to test out passthrough?

wswsmao · 2024-12-02T07:26:09Z

Does the kernel version need to be updated to 6.9 to test out passthrough?

yes，or rebase this feature
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=6ce8b2ce0d7e3a621cdc9eb66d74436ca7d0e66e

fuse: support passthrough mode

71930d7

Signed-off-by: abushwang <abushwangs@gmail.com>

wswsmao mentioned this pull request Nov 19, 2024

Add FUSE Passthrough Support in Stargz-Snapshotter #1867

Closed

fuse: add kernel fuse passthrough support check

39a2e55

Signed-off-by: abushwang <abushwangs@gmail.com>

wswsmao force-pushed the passthrough branch from 6424a04 to 39a2e55 Compare November 19, 2024 05:02

ktock reviewed Nov 19, 2024

View reviewed changes

wswsmao force-pushed the passthrough branch 9 times, most recently from 3fd3113 to 5dba826 Compare November 20, 2024 06:39

ktock reviewed Nov 20, 2024

View reviewed changes

fuse passthrough: fix some review comments

d16d065

Signed-off-by: abushwang <abushwangs@gmail.com>

wswsmao force-pushed the passthrough branch from 5dba826 to d16d065 Compare November 21, 2024 01:40

ktock approved these changes Nov 21, 2024

View reviewed changes

ktock merged commit 2a280d6 into containerd:main Nov 21, 2024

ktock mentioned this pull request Jul 9, 2025

CI: enable FUSE passthrough in CI #2074

Merged

Navigation Menu

Repository navigation

Add FUSE Passthrough Support in Stargz-Snapshotter #1867 - #1868

Add FUSE Passthrough Support in Stargz-Snapshotter #1867#1868
ktock merged 3 commits into
containerd:mainfrom
wswsmao:passthrough

wswsmao commented Nov 19, 2024 •

edited

Loading

Uh oh!

ktock left a comment

Uh oh!

ktock Nov 19, 2024

Uh oh!

ktock Nov 19, 2024

Uh oh!

ktock Nov 19, 2024

Uh oh!

ktock Nov 19, 2024

Uh oh!

ktock Nov 19, 2024

Uh oh!

wswsmao commented Nov 20, 2024

Uh oh!

wswsmao commented Nov 20, 2024 •

edited

Loading

Uh oh!

ktock Nov 20, 2024

Uh oh!

ktock left a comment

Uh oh!

ujjwal commented Dec 2, 2024

Uh oh!

wswsmao commented Dec 2, 2024

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		n.fs.s.report(fmt.Errorf("node.Open: %v", err))
		return nil, 0, syscall.EIO

Conversation

wswsmao commented Nov 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ktock left a comment

Choose a reason for hiding this comment

Uh oh!

ktock Nov 19, 2024

Choose a reason for hiding this comment

Uh oh!

ktock Nov 19, 2024

Choose a reason for hiding this comment

Uh oh!

ktock Nov 19, 2024

Choose a reason for hiding this comment

Uh oh!

ktock Nov 19, 2024

Choose a reason for hiding this comment

Uh oh!

ktock Nov 19, 2024

Choose a reason for hiding this comment

Uh oh!

wswsmao commented Nov 20, 2024

Uh oh!

wswsmao commented Nov 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ktock Nov 20, 2024

Choose a reason for hiding this comment

Uh oh!

ktock left a comment

Choose a reason for hiding this comment

Uh oh!

ujjwal commented Dec 2, 2024

Uh oh!

wswsmao commented Dec 2, 2024

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

wswsmao commented Nov 19, 2024 •

edited

Loading

wswsmao commented Nov 20, 2024 •

edited

Loading