Skip to content

Conversation

@SAHU-01
Copy link

@SAHU-01 SAHU-01 commented Oct 20, 2025

Closes #663

Summary

Adds a example demonstrating how to create UnixFS DAGs and export them as CAR files using MFS (Mutable File System)

What's Included

✅ Proper Chunking (Not io.ReadAll)

  • Size-based: --chunker size-[bytes]
  • Rabin: --chunker rabin-[min]-[avg]-[max] (content-defined)
  • Buzhash: --chunker buzhash
  • Streams data, handles files of any size

✅ Directory Support

  • Recursive directory traversal
  • Preserves directory structure with subdirectories
  • Full filesystem tree import

✅ HAMT Sharding

  • Automatic HAMT for directories >174 files
  • Configurable threshold: --max-directory-links

✅ All Major ipfs add Flags

  • Chunking: --chunker, --raw-leaves
  • CID options: --cid-version, --hash, --inline, --inline-limit
  • Layout: --trickle (trickle DAG)
  • Metadata: --preserve-mode, --preserve-mtime

✅ MFS Integration

  • Uses mfs.NewRoot() to create filesystem
  • Uses mfs.PutNode() for adding nodes
  • Uses root.Flush() to persist changes

Example Usage

Basic File

cd examples/unixfs-to-car
go build

echo "Hello, IPFS!" > test.txt
./unixfs-to-car test.txt output.car

Output:

Root CID: bafybei...

Directory

mkdir mydir
echo "File 1" > mydir/file1.txt
echo "File 2" > mydir/file2.txt
./unixfs-to-car mydir output.car

With All Flags

./unixfs-to-car \
  --chunker size-1048576 \
  --raw-leaves \
  --hash blake2b-256 \
  --preserve-mode \
  --preserve-mtime \
  myfile.txt output.car

Testing

All 13 tests pass:

$ go test -v
=== RUN   TestBasicFileConversion
--- PASS: TestBasicFileConversion (0.00s)
=== RUN   TestLargeFileWithChunking
--- PASS: TestLargeFileWithChunking (0.01s)
=== RUN   TestDirectoryConversion
--- PASS: TestDirectoryConversion (0.00s)
=== RUN   TestRawLeaves
--- PASS: TestRawLeaves (0.00s)
=== RUN   TestTrickleDAG
--- PASS: TestTrickleDAG (0.00s)
=== RUN   TestMetadataPreservation
--- PASS: TestMetadataPreservation (0.00s)
=== RUN   TestDifferentHashFunctions
--- PASS: TestDifferentHashFunctions (0.00s)
=== RUN   TestDifferentChunkers
--- PASS: TestDifferentChunkers (0.01s)
=== RUN   TestInvalidHashFunction
--- PASS: TestInvalidHashFunction (0.00s)
=== RUN   TestInvalidChunker
--- PASS: TestInvalidChunker (0.00s)
=== RUN   TestNonExistentFile
--- PASS: TestNonExistentFile (0.00s)
=== RUN   TestLargeDirectoryWithHAMT
--- PASS: TestLargeDirectoryWithHAMT (0.03s)
PASS
ok      github.com/ipfs/boxo/examples/unixfs-to-car     0.287s

Tests cover:

  • ✅ Basic file conversion
  • ✅ Large file chunking (1MB+)
  • ✅ Directory conversion with subdirectories
  • ✅ Raw leaves
  • ✅ Trickle DAG layout
  • ✅ Metadata preservation
  • ✅ Multiple hash functions (SHA2-256, SHA2-512, Blake2b-256)
  • ✅ Multiple chunkers (size, rabin, buzhash)
  • ✅ HAMT sharding (200 files)
  • ✅ Error cases

Implementation Details

MFS-Based Workflow

// 1. Create MFS root
root, _ := mfs.NewRoot(ctx, dagService, emptyDir, nil, nil)

// 2. Import file/directory using UnixFS importer
node, _ := importFile(ctx, path, dagService, config, hashCode)

// 3. Add to MFS
mfs.PutNode(root, "/filename", node)
root.Flush()

// 4. Get CID and export to CAR
rootNode, _ := root.GetDirectory().GetNode()
exportToCAR(ctx, blockstore, rootNode.Cid(), outputPath)

This follows the same pattern as Kubo's ipfs add --to-files (see kubo/core/commands/add.go).

Screenrecording

image
Screen.Recording.2025-10-20.at.11.39.03.AM.mp4

Documentation

README includes:

  • ✅ Usage examples for all features
  • ✅ All flags documented
  • ✅ Architecture explanation
  • ✅ Comparison to ipfs add
  • ✅ Performance characteristics
  • ✅ Code structure overview

@lidel this implementation satisfies all listed requirements and is ready for review!

@SAHU-01 SAHU-01 requested a review from a team as a code owner October 20, 2025 05:31
@welcome
Copy link

welcome bot commented Oct 20, 2025

Thank you for submitting this PR!
A maintainer will be here shortly to review it.
We are super grateful, but we are also overloaded! Help us by making sure that:

  • The context for this PR is clear, with relevant discussion, decisions
    and stakeholders linked/mentioned.

  • Your contribution itself is clear (code comments, self-review for the
    rest) and in its best form. Follow the code contribution
    guidelines

    if they apply.

Getting other community members to do a review would be great help too on complex PRs (you can ask in the chats/forums). If you are unsure about something, just leave us a comment.
Next steps:

  • A maintainer will triage and assign priority to this PR, commenting on
    any missing things and potentially assigning a reviewer for high
    priority items.

  • The PR gets reviews, discussed and approvals as needed.

  • The PR is merged by maintainers when it has been approved and comments addressed.

We currently aim to provide initial feedback/triaging within two business days. Please keep an eye on any labelling actions, as these will indicate priorities and status of your contribution.
We are very grateful for your contribution!

@hsanjuan hsanjuan added the kind/generated This issue might have been automatically generated label Oct 20, 2025
@github-actions
Copy link

It seems this issue might have been automatically generated. To help us address it effectively, please provide additional details.

We value the use of LLMs for code generation and welcome your contributions but please ensure your submission is of such quality that a maintainer will spend less time reviewing it than implementing it themselves. Verify the code functions correctly and meets our standards. If your change requires tests, kindly include them and ensure they pass.

If no further information is provided, the issue will be automatically closed in 7 days. Thank you for your understanding and for aiding us in maintaining quality contributions!

@github-actions github-actions bot added the need/author-input Needs input from the original author label Oct 21, 2025
@hsanjuan
Copy link
Contributor

Triage:

  • The example should be super concise and as short as possible. This is too verbose. We have helpers and wrappers that could have been used. Lots of flexibility used (cid versions etc. is out of the scope of an example).

@hsanjuan hsanjuan added the status/blocked Unable to be worked further until needs are met label Oct 28, 2025
@hsanjuan hsanjuan marked this pull request as draft October 28, 2025 15:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind/generated This issue might have been automatically generated need/author-input Needs input from the original author status/blocked Unable to be worked further until needs are met

Projects

None yet

Development

Successfully merging this pull request may close these issues.

example: turn directory into UnixFS DAG

2 participants