Skip to content

Conversation

djeebus
Copy link
Contributor

@djeebus djeebus commented Oct 16, 2025

This allows us to cache, compress, etc.

  • Add a new OpenSeekableObject method to the storage provider that is used to read seekable files
  • Remove the ReadAt method from the ObjectProvider interface (only used by SeekableObjectProvider).
  • Remove the Delete method from the ObjectProvider interface (never used).
  • Add a SeekableObjectProvider interface.
  • Add new ObjectType and SeekableObjectType types that defines which file type is being passed in, in order to help the compiler prevent mistakes

Note

Split storage API into typed seekable and non-seekable object interfaces with OpenObject/OpenSeekableObject, updating providers, caching, mocks, tests, and orchestrator code to use explicit object types.

  • Storage (core API):
    • Introduce ObjectType and SeekableObjectType enums and new interfaces: ObjectProvider (non-seekable) and SeekableObjectProvider (seekable).
    • StorageProvider now exposes OpenObject(ctx, path, objectType) and OpenSeekableObject(ctx, path, seekableObjectType).
    • Add typed constants: MemfileObjectType, RootFSObjectType, MemfileHeaderObjectType, SnapfileObjectType, MetadataObjectType, LayerObjectType, LayerMetadataObjectType.
  • Providers:
    • AWS/GCP/FS providers implement both new interfaces; add Exists on non-seekable objects; helper ignoreNotExists (AWS) and typed open methods.
  • Caching:
    • Split cache into CachedObjectProvider (full-file) and CachedSeekableObjectProvider (chunked ReadAt/Size); reorganize cache code into storage_cache_object.go and storage_cache_seekable.go.
    • CachedProvider updated to wrap both OpenObject and OpenSeekableObject.
  • Orchestrator:
    • Switch reads of data files to OpenSeekableObject with storage.MemfileObjectType/storage.RootFSObjectType and headers/metadata to OpenObject with typed ObjectType.
    • Update storage diff/template build paths to carry and use object type; add helper mappers for diff/file types to object types.
    • Server layer upload now checks obj.Exists(ctx) instead of Size.
  • Mocks/Tests/Config:
    • Replace StorageObjectProvider mocks with ObjectProvider and add SeekableObjectProvider mocks; update usages.
    • Adjust tests to new interfaces and semantics (e.g., Exists and delete via provider).
    • Update .mockery.yaml to generate new mocks and filenames.

Written by Cursor Bugbot for commit 85a22ac. This will update automatically on new commits. Configure here.

this allows us to cache, compress, etc.
cursor[bot]

This comment was marked as outdated.

cursor[bot]

This comment was marked as outdated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants