You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/developer_guide/utilities/verifiable_datasets.rst
+3-2Lines changed: 3 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -11,7 +11,7 @@ To accommodate for the proliferation of data sources and the need for trusted da
11
11
This includes an extensible class hierarchy that enables the creation of datasets from various data sources, such as local file system, object storage and others.
12
12
13
13
The central abstraction is the :code:`VerifiableDatasetInfo` class that encapsulates the dataset's metadata and provides a method for verifying the integrity of the dataset.
14
-
A dataset can be built from multiple data sources (not necessarily from the same type):
14
+
A dataset can be built from multiple data sources (not necessarily of the same type):
:caption: Verifiable Dataset with Multiple Data Sources
@@ -21,6 +21,7 @@ The :code:`VerifiableDatasetInfo` class can then be used to create higher-order
21
21
The :code:`root_hash` is used as a reference for integrity when loading items from the the data sources in the :code:`VerifiableDatasetInfo` object.
22
22
23
23
OpenFL comes with a toolbox of dataset layout classes per ML framework. For PyTorch's :code:`torch.utils.data.Dataset` OpenFL curently provides:
24
+
24
25
- :code:`FolderDataset` - represents an iterable folder-layout dataset from a single data source, by implementing the :code:`__getitem__` method.
25
26
- :code:`ImageFolder` - a specialization of the :code:`FolderDataset` that is able to load binary images from a foler-like structure
26
27
- :code:`VerifiableMapStyleDataset` - a base class for map-style datasets that can be built from multiple data sources (as specified by a :code:`VerifiableDatasetInfo` object), including integrity checks.
@@ -33,4 +34,4 @@ A similar class hierarchy can be created for other ML frameworks that offer data
33
34
:caption: Dataset hierarchy
34
35
:align: center
35
36
36
-
A practical example for the :code:`VerifiableImageFolder` backed by a mix of :code:`LocalDataSource` and :code:`S3DataSource` objects is provided in the `s3_histology <https://github.com/securefederatedai/openfl/tree/develop/openfl-workspace/torch/histology_s3>`_ workspace template.
37
+
A practical example for the :code:`VerifiableImageFolder` backed by :code:`S3DataSource` is provided in the `s3_histology <https://github.com/securefederatedai/openfl/tree/develop/openfl-workspace/torch/histology_s3>`_ workspace template.
0 commit comments