Add README and docstrings for Julia API (#17)

nickrobinson251 · web-flow · commit fbaaebdbd692 · 2024-01-11T00:18:21.000Z
* Add docstrings to Julia API

* WIP README

* WIP README design notes

* Update test to more specific exception types

* More readme

* fixup! More readme

* Update README.md
diff --git a/Project.toml b/Project.toml
@@ -3,10 +3,12 @@ uuid = "1b5eed3d-1f46-4baa-87f3-a4a892b23610"
 version = "0.1.0"
 
 [deps]
+DocStringExtensions = "ffbed154-4ef7-542d-bbb7-c09d3a79fcae"
 object_store_ffi_jll = "0e112785-0821-598c-8835-9f07837e8d7b"
 
 [compat]
 CloudBase = "1"
+DocStringExtensions = "0.9"
 HTTP = "1"
 ReTestItems = "1"
 Sockets = "1"
diff --git a/README.md b/README.md
@@ -1 +1,121 @@
 # RustyObjectStore.jl
+
+[![CI](https://github.com/RelationalAI/RustyObjectStore.jl/actions/workflows/CI.yml/badge.svg)](https://github.com/RelationalAI/RustyObjectStore.jl/actions/workflows/CI.yml)
+
+RustyObjectStore.jl is a Julia package for getting and putting data in cloud object stores, such as Azure Blob Storage and AWS S3.
+It is built on top of the Rust [object_store crate](https://docs.rs/object_store/).
+It provides a minimal API and focusses on high throughput.
+
+_The package is under active development. Currently only Azure Blob Storage is supported._
+
+## Usage
+
+The object_store runtime must be started before any requests are sent.
+
+```julia
+using RustyObjectStore
+init_object_store()
+```
+
+Requests are sent via calling `blob_put` or `blob_get!`, providing the location of the object to put/get, either the data to send or a buffer that will receive data, and credentials.
+For `blob_put` the data must be a vector of bytes (`UInt8`).
+For `blob_get!` the buffer must be a vector into which bytes (`UInt8`) can be written.
+```julia
+credentials = AzureCredentials("my_account", "my_container", "my_key")
+input = "1,2,3,4,5,6,7,8,9,0\n" ^ 5  #  100 B
+
+nbytes_written = blob_put("path/to/example.csv", codeunits(input), credentials)
+@assert nbytes_written == 100
+
+buffer = Vector{UInt8}(undef, 1000)  # 1000 B
+@assert sizeof(buffer) > sizeof(input)
+
+nbytes_read = blob_get!("path/to/example.csv", buffer, credentials)
+@assert nbytes_read == 100
+@assert String(buffer[1:nbytes_read]) == input
+```
+
+## Design
+
+#### Packaging
+
+The Rust [object_store](https://github.com/apache/arrow-rs/tree/master/object_store) crate does not provide a C API, so we have defined a C API in [object_store_ffi](https://github.com/relationalAI/object_store_ffi).
+RustyObjectStore.jl depends on [object_store_ffi_jll.jl](https://github.com/JuliaBinaryWrappers/object_store_ffi_jll.jl) to provides a pre-built object_store_ffi library, and calls into the native library via `@ccall`.
+
+#### Rust/Julia Interaction
+
+Julia calls into the native library providing a libuv condition variable and then waits on that variable.
+In the native code, the request from Julia is passed into a queue that is processed by a Rust spawned task.
+Once the request to cloud storage is complete, Rust signals the condition variable.
+In this way, the requests are asynchronous all the way up to Julia and the network processing is handled in the context of native thread pool.
+
+For a GET request, Julia provides a buffer for the native library to write into.
+This requires Julia to know a suitable size before-hand and requires the native library to do an extra memory copy, but the upside is that Julia controls the lifetime of the memory.
+
+#### Threading Model
+
+Rust object_store uses the [tokio](https://docs.rs/tokio) async runtime.
+
+TODO
+
+#### Rust Configuration
+
+TODO
+
+## Developement
+
+When working on RustyObjectStore.jl you can either use [object_store_ffi_jll.jl](https://github.com/JuliaBinaryWrappers/object_store_ffi_jll.jl) or use a local build of [object_store_ffi](https://github.com/relationalAI/object_store_ffi).
+Using object_store_ffi_jll.jl is just like using any other Julia package.
+For example, you can change object_store_ffi_jll.jl version by updating the Project.toml `compat` entry and running `Pkg.update` to get the latest compatible release,
+or `Pkg.develop` to use an unreleased version.
+
+Alternatively, you can use a local build of object_store_ffi library by setting the `OBJECT_STORE_LIB` environment variable to the location of the build.
+For example, if you have the object_store_ffi repository at `~/repos/object_store_ffi` and build the library by running `cargo build --release` from the base of that repository,
+then you could use that local build by setting `OBJECT_STORE_LIB="~/repos/object_store_ffi/target/release"`.
+
+The `OBJECT_STORE_LIB` environment variable is intended to be used only for local development.
+The library path is set at package precompile time, so if the environment variable is changed RustyObjectStore.jl must recompile for the change to take effect.
+You can check the location of the library in use by inspecting `RustyObjectStore.rust_lib`.
+
+Since RustyObjectStore.jl is the primary user of object_store_ffi, the packages should usually be developed alongside one another.
+For example, updating object_store_ffi and then testing out the changes in RustyObjectStore.jl.
+A new release of object_store_ffi should usually be followed by a new release of object_store_ffi_jll.jl, and then a new release RustyObjectStore.jl.
+
+#### Testing
+
+Tests use the [ReTestItems.jl](https://github.com/JuliaTesting/ReTestItems.jl) test framework.
+
+Run tests using the package manager Pkg.jl like:
+```sh
+$ julia --project -e 'using Pkg; Pkg.test()'
+```
+or after starting in a Julia session started with `julia --project`:
+```julia
+julia> # press ] to enter the Pkg REPL mode
+
+(RustyObjectStore) pkg> test
+```
+Alternatively, tests can be run using ReTestItems.jl directly, which supports running individual tests.
+For example:
+```julia
+julia> using ReTestItems
+
+julia> runtests("test/azure_api_tests.jl"; name="AzureCredentials")
+```
+
+If `OBJECT_STORE_LIB` is set, then running tests locally will use the specified local build of the object_store_ffi library, rather than the version installed by object_store_ffi_jll.jl.
+This is useful for testing out changes to object_store_ffi.
+
+Adding new tests is done by writing test code in a `@testitem` in a file suffixed `*_tests.jl`.
+See the existing [tests](./test) or the [ReTestItems documentation](https://github.com/JuliaTesting/ReTestItems.jl/#writing-tests) for examples.
+
+#### Release Process
+
+New releases of RustyObjectStore.jl can be made by incrementing the version number in the Project.toml file following [Semantic Versioning](semver.org),
+and then commenting on the commit that should be released with `@JuliaRegistrator register`
+(see [example](https://github.com/RelationalAI/RustyObjectStore.jl/commit/1b1ba5a198e76afe37f75a1d07e701deb818869c#comments)).
+The [JuliaRegistrator](https://github.com/JuliaRegistries/Registrator.jl) bot will reply to the comment and automatically open a PR to the [General](https://github.com/JuliaRegistries/General/) package registry, that should then automatically be merged within a few minutes.
+Once that PR to General is merged the new version of RustyObjectStore.jl is available, and the TagBot Github Action will make add a Git tag and a GitHub release for the new version.
+
+RustyObjectStore.jl uses the object_store_ffi library via depending on object_store_ffi_jll.jl which installs pre-built binaries.
+So when a new release of object_store_ffi is made, we need there to be a new release of object_store_ffi_jll.jl before we can make a release of RustyObjectStore.jl that uses the latest object_store_ffi.
diff --git a/src/RustyObjectStore.jl b/src/RustyObjectStore.jl
@@ -2,8 +2,10 @@ module RustyObjectStore
 
 export init_object_store, blob_get!, blob_put, AzureCredentials, ObjectStoreConfig
 
-using object_store_ffi_jll
 using Base.Libc.Libdl: dlext
+using Base: @kwdef, @lock
+using DocStringExtensions
+using object_store_ffi_jll
 
 const rust_lib = if haskey(ENV, "OBJECT_STORE_LIB")
     # For development, e.g. run `cargo build --release` and point to `target/release/` dir.
@@ -20,38 +22,91 @@ else
     object_store_ffi_jll.libobject_store_ffi
 end
 
-struct ObjectStoreConfig
+"""
+    $TYPEDEF
+
+Global configuration for the object store requests.
+
+# Keywords
+$TYPEDFIELDS
+"""
+@kwdef struct ObjectStoreConfig
+    "The maximum number of times to retry a request."
     max_retries::Culonglong
+    "The number of seconds from the initial request after which no further retries will be attempted."
     retry_timeout_sec::Culonglong
 end
 
-const OBJECT_STORE_STARTED = Ref(false)
+function Base.show(io::IO, config::ObjectStoreConfig)
+    print(io, "ObjectStoreConfig("),
+    print(io, "max_retries=", Int(config.max_retries), ", ")
+    print(io, "retry_timeout_sec=", Int(config.retry_timeout_sec), ")")
+end
+
+const DEFAULT_CONFIG = ObjectStoreConfig(max_retries=15, retry_timeout_sec=150)
+
+const _OBJECT_STORE_STARTED = Ref(false)
 const _INIT_LOCK::ReentrantLock = ReentrantLock()
-function init_object_store(config::ObjectStoreConfig = ObjectStoreConfig(15, 150))
-    Base.@lock _INIT_LOCK begin
-        if OBJECT_STORE_STARTED[]
-            return
+
+struct InitException <: Exception
+    msg::String
+    return_code::Cint
+end
+
+"""
+    init_object_store()
+    init_object_store(config::ObjectStoreConfig)
+
+Initialise object store.
+
+This starts a `tokio` runtime for handling `object_store` requests.
+It must be called before sending a request e.g. with `blob_get!` or `blob_put`.
+The runtime is only started once and cannot be re-initialised with a different config,
+subsequent `init_object_store` calls have no effect.
+
+# Throws
+- `InitException`: if the runtime fails to start.
+"""
+function init_object_store(config::ObjectStoreConfig=DEFAULT_CONFIG)
+    @lock _INIT_LOCK begin
+        if _OBJECT_STORE_STARTED[]
+            return nothing
         end
         res = @ccall rust_lib.start(config::ObjectStoreConfig)::Cint
         if res != 0
-            error("Failed to init_object_store")
+            throw(InitException("Failed to initialise object store runtime.", res))
         end
-        OBJECT_STORE_STARTED[] = true
+        _OBJECT_STORE_STARTED[] = true
     end
+    return nothing
 end
 
+"""
+    $TYPEDEF
+
+# Arguments
+$TYPEDFIELDS
+"""
 struct AzureCredentials
+    "Azure account name"
     account::String
+    "Azure container name"
     container::String
+    "Azure access key"
     key::String
+    "(Optional) Alternative Azure host. For example, if using Azurite."
     host::String
+    function AzureCredentials(account::String, container::String, key::String, host::String="")
+        return new(account, container, key, host)
+    end
 end
 function Base.show(io::IO, credentials::AzureCredentials)
     print(io, "AzureCredentials("),
-    print(io, repr(credentials.account), ", ")
-    print(io, repr(credentials.container), ", ")
-    print(io, "\"*****\", ") # don't print the secret key
-    print(io, repr(credentials.host), ")")
+    print(io, repr(credentials.account), )
+    print(io, ", ", repr(credentials.container))
+    print(io, ", ", "\"*****\"") # don't print the secret key
+    !isempty(credentials.host) && print(io, ", ", repr(credentials.host))
+    print(io, ")")
 end
 
 const _AzureCredentialsFFI = NTuple{4,Cstring}
@@ -67,6 +122,7 @@ function Base.cconvert(::Type{Ref{AzureCredentials}}, credentials::AzureCredenti
     # safely in the unsafe_convert call.
     return credentials_ffi, Ref(credentials_ffi)
 end
+
 function Base.unsafe_convert(::Type{Ref{AzureCredentials}}, x::Tuple{T,Ref{T}}) where {T<:_AzureCredentialsFFI}
     return Base.unsafe_convert(Ptr{_AzureCredentialsFFI}, x[2])
 end
@@ -79,6 +135,37 @@ struct Response
     Response() = new(-1, 0, C_NULL)
 end
 
+abstract type RequestException <: Exception end
+struct GetException <: RequestException
+    msg::String
+end
+struct PutException <: RequestException
+    msg::String
+end
+
+# TODO: this should be `blob_get!(buffer, path, credentials)` i.e. mutated argument first.
+"""
+    blob_get!(path, buffer, credentials) -> Int
+
+Send a get request to Azure Blob Storage.
+
+Fetches the data bytes at `path` and writes them to the given `buffer`.
+
+# Arguments
+- `path::String`: The location of the data to fetch.
+- `buffer::AbstractVector{UInt8}`: The buffer to write the blob data to.
+  The contents of the buffer will be mutated.
+  The buffer must be at least as large as the data.
+  The buffer will not be resized.
+- `credentials::AzureCredentials`: The credentials to use for the request.
+
+# Returns
+- `nbytes::Int`: The number of bytes read from Blob Storage and written to the buffer.
+  That is, `buffer[1:nbytes]` will contain the blob data.
+
+# Throws
+- `GetException`: If the request fails for any reason, including if the `buffer` is too small.
+"""
 function blob_get!(path::String, buffer::AbstractVector{UInt8}, credentials::AzureCredentials)
     response = Ref(Response())
     size = length(buffer)
@@ -109,13 +196,35 @@ function blob_get!(path::String, buffer::AbstractVector{UInt8}, credentials::Azu
         if response.result == 1
             err = "failed to process get with error: $(unsafe_string(response.error_message))"
             @ccall rust_lib.destroy_cstring(response.error_message::Ptr{Cchar})::Cint
-            error(err)
+            throw(GetException(err))
         end
 
         return Int(response.length)
     end
 end
 
+# TODO: this should be `blob_put(buffer, path, credentials)` so match `blob_get!`
+# when that is changed to put its mutated argument first.
+"""
+    blob_put(path, buffer, credentials) -> Int
+
+Send a put request to Azure Blob Storage.
+
+Atomically writes the data bytes in `buffer` to `path`.
+
+# Arguments
+- `path::String`: The location to write data to.
+- `buffer::AbstractVector{UInt8}`: The data to write to Blob Storage.
+  This buffer will not be mutated.
+- `credentials::AzureCredentials`: The credentials to use for the request.
+
+# Returns
+- `nbytes::Int`: The number of bytes written to Blob Storage.
+  Is always equal to `length(buffer)`.
+
+# Throws
+- `PutException`: If the request fails for any reason.
+"""
 function blob_put(path::String, buffer::AbstractVector{UInt8}, credentials::AzureCredentials)
     response = Ref(Response())
     size = length(buffer)
@@ -146,7 +255,7 @@ function blob_put(path::String, buffer::AbstractVector{UInt8}, credentials::Azur
         if response.result == 1
             err = "failed to process put with error: $(unsafe_string(response.error_message))"
             @ccall rust_lib.destroy_cstring(response.error_message::Ptr{Cchar})::Cint
-            error(err)
+            throw(PutException(err))
         end
 
         return Int(response.length)
diff --git a/test/azure_api_tests.jl b/test/azure_api_tests.jl
@@ -0,0 +1,8 @@
+@testitem "AzureCredentials" begin
+    # access key is obscured when printing
+    @test repr(AzureCredentials("a", "b", "c", "d")) == "AzureCredentials(\"a\", \"b\", \"*****\", \"d\")"
+    # host is optional
+    @test AzureCredentials("a", "b", "c") == AzureCredentials("a", "b", "c", "")
+    # host is not shown if not set
+    @test repr(AzureCredentials("a", "b", "c")) == "AzureCredentials(\"a\", \"b\", \"*****\")"
+end
diff --git a/test/azure_blobs_exception_tests.jl b/test/azure_blobs_exception_tests.jl
