add an option to close the catalog

[djouallah]

Feature Request / Improvement

using the latest dev release, i can now write to onelake, and that's awesome, but i have a niche feature request :) , I use sqlite as a metastore, and it is located in a blobfuse path, the db get flushed to the persistent storage only when the connection to sqlite is closed, is there a way to have something like catalog.close() to automatically close the connection, sorry if the request does not make sense :)

from   pyiceberg.catalog.sql import SqlCatalog
import sqlite3
import duckdb
import os
ws     =  "iceberg"
lh     =  "data"
schema =  "first"  
tbl    =  'new'
warhouse_path = f'abfss://{ws}@onelake.dfs.fabric.microsoft.com/{lh}.Lakehouse/Tables'
catalog = SqlCatalog(
      "default",
      **{
          "uri"                       : "sqlite://///lakehouse/default/Files/pyiceberg.db",
          "adls.account-name"         : 'onelake' ,
          "adls.account-host"         : "onelake.blob.fabric.microsoft.com" ,
          "adls.token"                :  os.environ.get('AZURE_STORAGE_TOKEN') ,
          "warehouse"                 :  warhouse_path 
      },
                        )
catalog.create_namespace_if_not_exists(schema)
catalog.list_tables(schema)
#Write Data
df=duckdb.sql(""" SELECT cast(unnest(generate_series(cast ('2018-04-01' as date), cast('2024-12-31' as date), interval 1 day)) as date) as date,
            EXTRACT(year from date) as year,
            EXTRACT(month from date) as month
            """).arrow()
table = catalog.create_table_if_not_exists(f"{schema}.{tbl}",schema=df.schema )
table.overwrite(df)
### cose sqlite to make sure the db is stored in onelake 
sqlite3.connect("/lakehouse/default/Files/pyiceberg.db").close()

[kevinjqliu]

session.commit() is called on every commit_table call. that should be the same as calling .close()

iceberg-python/pyiceberg/catalog/sql.py

Line 477 in c72e286

session.commit()

[kevinjqliu]

what are you currently seeing?

[kevinjqliu]

When a file is the subject of a write operation, the data is first persisted to cache on a local disk. The data is written to Blob Storage only after the file handle is closed. If an issue attempting to persist the data to Blob Storage occurs, an error message appears.

https://learn.microsoft.com/en-us/azure/storage/blobs/blobfuse2-what-is#file-caching-on-disk

This seems like a quirk of blobfuse

[djouallah]

what will be really cool, if we can store sqlite on an abfss :)

[kevinjqliu]

maybe some of these options can help flush to blob storage right away
https://github.com/Azure/azure-storage-fuse/blob/main/README.md

Attribute cache options
--attr-cache-timeout=<TIMEOUT IN SECONDS>: The timeout for the attribute cache entries.

File cache options
--file-cache-timeout=<TIMEOUT IN SECONDS>: Timeout for which file is cached on local system.
--cache-size-mb=<SIZE IN MB>: Amount of disk cache that can be used by blobfuse. Default - 80% of free disk space.
...
--sync-to-flush=false : Sync call will force upload a file to storage container if this is set to true, otherwise it just evicts file from local cache.

[djouallah]

thanks, I am closing the issue as it is not really pyiceberg problem :)

[Fokko]

I know this one is closed, but I think there is room for a close operation, since that's also used in Java for closing connection pools: https://github.com/apache/iceberg/blob/88500ecb457299cd46c5e075d29556ffdd5eaad5/core/src/main/java/org/apache/iceberg/BaseMetastoreCatalog.java#L303-L307

We could add an __enter__ and __exit__ method to Catalog to introduce the ability to allocate and clean up resources.