Skip to content

Conversation

PierreJeanjacquot
Copy link
Member

@PierreJeanjacquot PierreJeanjacquot commented Sep 10, 2025

bulk indexation from IPFS

image

Features:

  • index bulk content when a deal has a bulk_cid
  • Deal bulk content accessible via bulk field
  • Task bulk slice content accessible via bulkSlice field
  • Dataset usages in bulk accessible via bulkUsages field

Testing:

Once the test stack is running npm run start-test-stack, create-bulks can be used to test bulk indexation behaviours.

Deployed instance (Arbitrum Sepolia): https://api.studio.thegraph.com/query/11362/poco-bulk/version/latest

Limitations:

thegraph supports indexing IPFS files (via file handlers) with some limitations:

  • entity updated by file handler and on-chain event handler should not collide
  • file handlers cannot access the blockchain
  • thegraph indexers use a dedicated node at https://ipfs.thegraph.com. Documents should be uploaded to this node to ensure proper indexing (this can be done after the IPFS template is created; indexers will retry to download the document until they get it)

type DatasetOrder @entity {
id: ID!
dataset: Dataset!
dataset: Dataset # nullable: off-chain bulk datasetorder could reference non-existing dataset
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had to modify the nullability on this field because file handlers can't access on-chain handlers entities.
When indexing a datasetorder from a bulk (file handler), we cannot verify that the dataset exists on-chain or exists as an entity.

Comment on lines +135 to +140
if (dataset != Address.zero().toHex()) {
let datasetorder = fetchDatasetorder(event.params.datasetHash.toHex());
if (dataset) datasetorder.dataset = dataset;
datasetorder.datasetprice = deal.datasetPrice;
datasetorder.save();
}
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

many changes due to formatted file (NB: prettier ignore should be updated to format src)

@PierreJeanjacquot PierreJeanjacquot marked this pull request as ready for review October 1, 2025 12:07
let datasets = bulkSlice.datasets;
datasets.push(datasetAddress);
bulkSlice.datasets = datasets;
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we warn here in case the dataset order is invalid ?

Comment on lines +30 to +31
const hash = dataSource.stringParam();
const context = dataSource.context();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how this infos are loaded when a handleBulk occure ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the diff with the content is like having metadata plug to the actual content of the IPFS file ? and this metadata is set up in the matchOrders handler function ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants