Skip to content

Implement io.IOBase.seekable method - #50

Merged
shcheklein merged 1 commit into
fsspec:mainfrom
mxmlnkn:fix-seekable
Oct 3, 2024
Merged

Implement io.IOBase.seekable method#50
shcheklein merged 1 commit into
fsspec:mainfrom
mxmlnkn:fix-seekable

Conversation

@mxmlnkn

@mxmlnkn mxmlnkn commented Oct 2, 2024

Copy link
Copy Markdown
Contributor

SSHFile inherits from io.IOBase, but even though seek can be called on it, the seekable method is not overwritten and therefore returns False via the default implementation in io.IOBase.

Comment thread sshfs/file.py Outdated
return "r" in self.mode

def seekable(self):
return "r" in self.mode or "w" in self.mode

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to double check: can it be also '+' (update)?

Also, is it actually enforced? How do we actually open files, write to them? E.g do we always seek to the end in the append mode on each write?

@mxmlnkn mxmlnkn Oct 3, 2024

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to double check: can it be also '+' (update)?

Also, is it actually enforced? How do we actually open files, write to them? E.g do we always seek to the end in the append mode on each write?

Note that you have this code on line 21:

        # TODO: support r+ / w+ / a+
        if mode not in {"rb", "wb", "ab"}:

Which makes the questions for + currently moot and untestable, but I'd also like to see support for +-mode. I'd also like to see support for open via flags like in asnycssh, but this is not that important. Coming back to your question, yes + mode should also be seekable. I mean, in the end, it only means that it can read and write instead of only doing one of both.

Append mode is more difficult for me to decide on because I have never used it. It seems to be enforced inside asyncssh.sftp.SFTPFile and it seems to actually allow for seeking in general, but it only has an effect for read. I am not sure where it is enforced exactly, but there is this comment in the write method.

Some code for testing assumptions:

import os
import sshfs

path = "foo"

if os.path.exists(path):
    os.remove(path)
with open(path, "wb") as file:
    assert file.seekable()
    file.write(b"123456")
with open(path, "ab") as file:
    assert file.seekable()
    file.write(b"789")
    assert file.tell() == 9
    file.seek(3)
    assert file.tell() == 3
    # file.read(1)  # io.UnsupportedOperation: read
    file.write(b"xyz")
    # Note that tell does NOT return 12 here as one might be inclined to think!
    # Instead of doing implicit seeks before each write, append simply always appends.
    # https://docs.python.org/release/3.7.2/library/functions.html#open
    # > 'a' for appending (which on some Unix systems, means that all writes
    # > append to the end of the file regardless of the current seek position).
    assert file.tell() == 6
with open(path, "ab+") as file:
    assert file.seekable()
    file.seek(6)
    assert file.tell() == 6
    assert file.read() == b"789xyz"
with open(path, "rb") as file:
    assert file.read() == b"123456789xyz"

fs = sshfs.SSHFileSystem("127.0.0.1", port=2022)

if fs.exists(path):
    fs.delete(path)
with fs.open(path, "wb") as file:
    file.write(b"123456")
with fs.open(path, "ab") as file:
    file.write(b"789")
    assert file.tell() == 9
    file.seek(3)
    assert file.tell() == 3
    file.write(b"xyz")
    # This is different from the built-in open file.
    assert file.tell() == 12
with fs.open(path, "ab+") as file:
    file.seek(6)
    assert file.tell() == 6
    assert file.read() == b"789xyz"
with fs.open(path, "rb") as file:
    assert file.read() == b"123456789xyz"

For the test with a+ via sshfs, I simply commented out the mode check mentioned above.

Note that I would have delegated to asyncssh.sftp.SFTPFile.seekable if there were such a method in the first place ...

I think it might be more correct to return true for seekable in the case of a+ and all other modes except for simply a, for which I am conflicted. seek theoretically works in the way that it does not raise an exception and calling tell after it will return the seeked-to offset, but read is not allowed in pure a mode and writing will always be done to the end...

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But as a+ is not yet supported, the current commit should be sufficiently correct in my opinion.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have added asserts for seekable to the Python file object and it returns true even for a mode. Therefore, I have pushed the change to simply always return True from inside seekable.

@shcheklein
shcheklein merged commit 5b29464 into fsspec:main Oct 3, 2024
shcheklein pushed a commit to mxmlnkn/sshfs that referenced this pull request Aug 5, 2026
When the server implements limits@openssh.com, use its max_read_len /
max_write_len as the block size for opened files (OpenSSH reports
255 KiB, roughly 3x-5x the previous defaults, measured ~3x faster
reads in fsspec#49). When the extension is not supported, asyncssh
synthesizes 16 KiB floors with a zero max_packet_len -- keep the tuned
48 KiB / 240 KiB defaults there instead of degrading to the floors.
An explicitly passed block_size wins as before.

Also drop a shadowed duplicate seekable() definition left over from
merging fsspec#50 and fsspec#51.

Fixes fsspec#49

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
shcheklein added a commit that referenced this pull request Aug 5, 2026
* Use server-reported block sizes for SSHFile when none is specified

When the server implements limits@openssh.com, use its max_read_len /
max_write_len as the block size for opened files (OpenSSH reports
255 KiB, roughly 3x-5x the previous defaults, measured ~3x faster
reads in #49). When the extension is not supported, asyncssh
synthesizes 16 KiB floors with a zero max_packet_len -- keep the tuned
48 KiB / 240 KiB defaults there instead of degrading to the floors.
An explicitly passed block_size wins as before.

Also drop a shadowed duplicate seekable() definition left over from
merging #50 and #51.

Fixes #49

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* tests: cover SSHFile block size selection

test_determine_block_size pins all three cases (server-reported
limits, synthesized floors with zero max_packet_len, asyncssh without
the limits API); test_open_block_size checks end to end that a server
without limits@openssh.com keeps the tuned defaults and that an
explicit block_size wins.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Ivan Shcheklein <shcheklein@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants