Is there any way to automatically "scan" all files on S3 bucket for Upload

[t18n]

I am in a migration process with an S3 bucket full of images. I have a Media with Upload collection. Is there a way to programmatically add existing images from the S3 bucket to Payload Database without reuploading?

[matthijs166]

It's going to involve some complex custom code..

Maybe you can fork the S3 plugin and instead of uploading the file to S3 you can first try to match and move it to the correct location.

Or you can make the media not required on creation, add an background task to loop over each media item in the bucket, create an Media item without file, move the file in the right location and edit the database fields to point to the correct location.

Pro tip: ALWAYS MAKE BACKUPS hahaha

[takayumi]

I'm in the same boat.

It would be nice to "pull" images from S3 to sync up your PayloadCMS collection with the cloud.

[matthijs166]

@takayumi Its not that hard to pull off. This is our code to sync files

import { NodeHttpHandlerOptions } from "@smithy/node-http-handler"
import { HeadObjectCommand, S3Client } from "@aws-sdk/client-s3";
import { S3StorageOptions } from "@payloadcms/storage-s3";

let storageClient: S3Client | null = null
const defaultRequestHandlerOpts: NodeHttpHandlerOptions = {
	httpAgent: {
		keepAlive: true,
		maxSockets: 100,
	},
	httpsAgent: {
		keepAlive: true,
		maxSockets: 100,
	},
}

const privateAdapter: Omit<S3StorageOptions, 'collections'> = {
	acl: 'private',
	config: {
		forcePathStyle: true,
		region: process.env.S3_REGION,
		endpoint: process.env.S3_ENDPOINT,
		credentials: {
			accessKeyId: process.env.S3_ACCESS_KEY_ID || '',
			secretAccessKey: process.env.S3_SECRET_ACCESS_KEY || '',
		}
	},
	bucket: process.env.S3_BUCKET || '',
};

const getS3Client: () => S3Client = () => {
	if (storageClient) {
		return storageClient
	}

	storageClient = new S3Client({
		requestHandler: defaultRequestHandlerOpts,
		...privateAdapter.config,
	});
	return storageClient
}

// inside a hook or task
try {
	const client = getS3Client();
	const response = await client.send(new HeadObjectCommand({
		Bucket: privateAdapter.bucket,
		Key: `folder/${filename}.pdf`,
	}));
	await req.payload.db.updateOne({
		collection: 'collectionName',
		id: doc.id,
		data: {
			filename: `${filename}.pdf`,
			filesize: response.ContentLength ?? 0,
			mimeType: response.ContentType ?? 'application/pdf',
		},
	});
} catch {
	// file not found
}