Efficiently Storing and Retrieving Large Posts in Firebase Firestore

[github-actions]

This document addresses a common problem developers face when working with Firebase Firestore: efficiently storing and retrieving large posts, such as blog posts with extensive text and images. Storing large amounts of data in a single Firestore document can lead to performance issues and exceed document size limits.

Description of the Error:

When storing lengthy blog posts directly within a single Firestore document, you might encounter several problems:

• Document Size Limits: Firestore imposes limits on document size. Exceeding these limits results in errors when attempting to create or update the document.
• Slow Retrieval: Retrieving large documents can significantly impact application performance, leading to slow loading times and a poor user experience.
• Inefficient Queries: Querying on specific parts of a large document can be less efficient than querying smaller, more focused documents.

Fixing Step by Step (Code Example):

Instead of storing the entire post content in a single field, we'll break it down into smaller, manageable chunks. This approach leverages Firestore's scalability and improves performance.

We will use a strategy of storing the main post details in one document and referencing separate documents for rich text content (e.g., using a storage service for images and referencing them).

1. Data Structure:

We'll use two collections: posts and postContent.

• posts collection: This collection will store metadata about each post, including:

  • postId (String, unique ID)
  • title (String)
  • authorId (String)
  • createdAt (Timestamp)
  • contentReference (array of Strings representing references to the postContent documents)

• postContent collection: This collection will store chunks of the post content, with each document representing a section:

  • contentId (String, unique ID)
  • postId (String, referencing the corresponding post in posts collection)
  • content (String, a portion of the post's text)
  • contentType (String, e.g., "text", "image", "video")
  • contentUrl (String, URL for images or videos stored in Cloud Storage)

2. Code (using JavaScript):

// Add a new post
async function addPost(title, authorId, contentSections) {
  const postId = firestore.collection('posts').doc().id;
  const contentReferences = [];

  // Add content sections to postContent collection
  for (const section of contentSections) {
    const contentId = firestore.collection('postContent').doc().id;
    await firestore.collection('postContent').doc(contentId).set({
      contentId: contentId,
      postId: postId,
      content: section.content, //Text content
      contentType: section.contentType,
      contentUrl: section.contentUrl //For images stored in Cloud Storage.
    });
    contentReferences.push(contentId);
  }

  // Add post metadata to posts collection
  await firestore.collection('posts').doc(postId).set({
    postId: postId,
    title: title,
    authorId: authorId,
    createdAt: firebase.firestore.FieldValue.serverTimestamp(),
    contentReference: contentReferences,
  });
  return postId;
}

//Retrieve a post
async function getPost(postId){
  const postDoc = await firestore.collection('posts').doc(postId).get();
  if(!postDoc.exists){
    return null;
  }
  const postData = postDoc.data();
  const contentPromises = postData.contentReference.map(contentId => firestore.collection('postContent').doc(contentId).get());
  const contentDocs = await Promise.all(contentPromises);
  const content = contentDocs.map(doc => doc.data());
  return {...postData, content};

}

3. Cloud Storage Integration (for Images):

For images, use Firebase Cloud Storage to store them and retrieve their URLs to include in the postContent documents. Remember to handle upload and download properly.

Explanation:

This approach significantly improves scalability and performance by distributing the post content across multiple smaller documents. Retrieving a post now involves fetching the metadata and then retrieving only the necessary content sections, reducing the amount of data transferred and improving query efficiency.

External References:

• Firebase Firestore Documentation
• Firebase Cloud Storage Documentation
• Firebase JavaScript SDK

Copyrights (c) OpenRockets Open-source Network. Free to use, copy, share, edit or publish.