Google Cloud Storage: Node.js Client

Seamlessly integrate Node.js applications with Google Cloud Storage for robust, scalable, and secure object management.

---

📖 Table of Contents

• Overview
  • Purpose & Goals
  • Why it Matters
  • Target Audience
• 🚀 Feature Highlights
• 📐 Architecture & Design
  • High-Level Components
  • Technology Stack
• 🏁 Getting Started
  • Prerequisites
  • Installation
  • Configuration
  • Running the Application
• 💡 Usage & Workflows
  • Basic Operations
  • Advanced Use Cases
• ⚠️ Limitations, Known Issues & Future Roadmap
  • Current Limitations
  • Known Issues
  • Future Roadmap
• 🤝 Contributing & Development Guidelines
  • How to Contribute
  • Development Setup
  • Code Style & Quality
  • Testing
• © License, Credits & Contact
  • License
  • Acknowledgments
  • Maintainers & Contact
• 📚 Appendix
  • Changelog
  • FAQ
  • Troubleshooting Guide
  • API Reference

---

Overview

The nodejs-storage client library provides an idiomatic Node.js interface for interacting with Google Cloud Storage (GCS). It simplifies the complexities of object storage, allowing developers to easily store, retrieve, manage, and share data globally.

Purpose & Goals

This library aims to:

• Abstract GCS API complexity: Provide a clean, developer-friendly API for common GCS operations.
• Enable scalable data management: Facilitate handling small and large objects, from single files to vast datasets, with robust streaming and transfer capabilities.
• Ensure security and access control: Integrate seamlessly with GCP's IAM and ACLs for fine-grained permissions.
• Enhance developer productivity: Offer well-documented methods, clear examples, and support for modern Node.js features.

Why it Matters

In today's cloud-native landscape, efficient and reliable object storage is fundamental. This client library empowers Node.js applications to leverage Google Cloud Storage for:

• Backend for web applications: Storing user-generated content, media files, and static assets.
• Data archival and backup: Securely archiving critical data with various storage classes.
• Big data processing: Serving as a data lake for analytics, machine learning, and data warehousing.
• Disaster recovery: Ensuring business continuity with robust data replication and recovery options.
• Content delivery: Distributing large data objects globally with low latency.

Target Audience

This documentation is for:

• Node.js Developers: Building applications that require interaction with Google Cloud Storage.
• Cloud Architects: Designing solutions involving object storage on GCP.
• DevOps Engineers: Managing and automating cloud storage operations.
• Anyone new to Google Cloud Storage: Looking for practical examples and guidance on using GCS with Node.js.

⬆ Back to Top

---

🚀 Feature Highlights

The nodejs-storage client provides a rich set of features to manage your Cloud Storage resources effectively.

Core Storage Operations

• Bucket Management:
  • ✅ Create, list, delete, and get metadata for buckets.
  • ✅ Configure bucket policies: CORS, lifecycle management, versioning, retention.
  • 💡 Tip: Easily set up buckets with specific storage classes and locations.
• File (Object) Management:
  • ✅ Upload and download files of any size (including large files with resumable uploads and transfer manager).
  • ✅ Stream file contents directly to/from memory or disk.
  • ✅ Copy, move, delete, and rename files.
  • ✅ Get and set file metadata, including custom metadata.
  • 🔍 Explore: The transfer-manager.ts module provides advanced features for large-scale transfers.

Access Control & Security

• Identity and Access Management (IAM):
  • ✅ Add, remove, and list IAM members and their roles for buckets.
  • ✅ Set up conditional IAM bindings for more granular access.
• Access Control Lists (ACLs):
  • ✅ Manage object and bucket ACLs for specific users or groups.
  • ✅ Define default ACLs for new objects in a bucket.
• Data Encryption:
  • ✅ Support for Customer-Supplied Encryption Keys (CSEK).
  • ✅ Integration with Customer-Managed Encryption Keys (CMEK) via KMS.
  • ✅ Generate and rotate encryption keys.
• Signed URLs:
  • ✅ Generate time-limited URLs for granting temporary read/write access to objects without requiring full authentication.
  • ✅ Support for both V2 (legacy) and V4 signed URLs.

Advanced Features

• HMAC Keys:
  • ✅ Create, activate, deactivate, and delete HMAC keys for interoperable access with S3-compatible tools.
• Notifications:
  • ✅ Configure Pub/Sub notifications for changes to objects within a bucket.
• Object Lifecycle Management:
  • ✅ Automatically transition objects to colder storage classes or delete them after a specified period.
• Version Management:
  • ✅ Enable object versioning to keep multiple versions of an object, protecting against accidental deletions or overwrites.
• Resumable Uploads:
  • ✅ Automatically handle network interruptions for large file uploads.
• Transfer Manager:
  • ✅ Optimize bulk uploads and downloads of multiple files or entire directories, including parallel processing.

⬆ Back to Top

---

📐 Architecture & Design

The nodejs-storage client library is designed with modularity and idiomatic Node.js patterns in mind, providing a clear separation of concerns and robust error handling.

High-Level Components

The library acts as a thin wrapper around the Google Cloud Storage REST API, facilitating communication between your Node.js application and the GCS service.

graph TD
    A[Your Node.js Application] --> B(Node.js Storage Client Library)
    B --> C(Authentication Layer)
    C --> D{Google Cloud Storage REST API}
    D --> E[Google Cloud Storage Buckets & Objects]

    subgraph Client Library Modules
        B -- ACL, IAM --> F[Access Control Module]
        B -- Files, Buckets --> G[Core Storage Module]
        B -- Uploads, Downloads --> H[Transfer Manager Module]
        B -- Signed URLs --> I[Signer Module]
    end

    subgraph Authentication
        C -- Service Account Key --> C1[Service Account JSON]
        C -- Environment Variable --> C2[GOOGLE_APPLICATION_CREDENTIALS]
        C -- Default (GCP Env) --> C3[Application Default Credentials]
    end

    style A fill:#f9f,stroke:#333,stroke-width:2px
    style B fill:#bbf,stroke:#333,stroke-width:2px
    style D fill:#cff,stroke:#333,stroke-width:2px
    style E fill:#ffc,stroke:#333,stroke-width:2px

Loading

• Your Node.js Application: Your custom code that leverages the client library.
• Node.js Storage Client Library: The @google-cloud/storage package, encapsulating GCS API calls. It's structured into several key modules:
  • Core Storage Module (storage.ts, bucket.ts, file.ts): Handles the primary interactions with buckets and objects.
  • Access Control Module (acl.ts, iam.ts): Manages permissions and access policies.
  • Transfer Manager Module (transfer-manager.ts): Provides utilities for efficient bulk data transfers.
  • Signer Module (signer.ts): Responsible for generating signed URLs and policies.
• Authentication Layer: Manages how your application authenticates with GCP. It supports various methods, primarily Application Default Credentials (ADC), which automatically handles authentication in most GCP environments.
• Google Cloud Storage REST API: The underlying HTTP API that the client library interacts with.
• Google Cloud Storage Buckets & Objects: The actual storage infrastructure where your data resides.

Technology Stack

• Language: TypeScript (compiled to JavaScript)
• Runtime: Node.js (v14+ recommended)
• Dependencies: Primarily Google's common libraries for Node.js (google-gax, @google-cloud/common) and internal utilities.
• API Protocol: RESTful HTTP/JSON over SSL/TLS.

⬆ Back to Top

---

🏁 Getting Started

Follow these steps to set up and start using the Google Cloud Storage Node.js client in your project.

Prerequisites

Before you begin, ensure you have the following:

1. Node.js: Version 14 or higher.

   node -v
   npm -v

2. Google Cloud Platform Project:
   • Select or create a GCP project.
   • Enable billing for your project. GCS usage incurs costs.
   • Enable the Google Cloud Storage API: Go to APIs & Services > Library in the GCP Console and search for "Google Cloud Storage API" to enable it.
3. Authentication: You need to set up authentication for your application to interact with GCS.
   • Application Default Credentials (ADC): The recommended way. If running on GCP (e.g., Compute Engine, Cloud Run, App Engine), authentication is handled automatically.
   • Service Account Key (for local development):
     1. In the GCP Console, navigate to IAM & Admin > Service Accounts.
     2. Create a new service account or select an existing one.
     3. Ensure the service account has appropriate roles (e.g., Storage Admin, Storage Object Admin, Storage Object Viewer).
     4. Create a new JSON key for the service account and download it.
     5. Set the GOOGLE_APPLICATION_CREDENTIALS environment variable to the path of this JSON key file:

        export GOOGLE_APPLICATION_CREDENTIALS="/path/to/your/keyfile.json"

        💡 Tip: For production, avoid storing key files directly in your application. Use environment variables, secret managers (like Secret Manager), or rely on ADC.

Installation

Install the client library using npm or yarn:

npm install @google-cloud/storage
# OR
yarn add @google-cloud/storage

Configuration

The Storage client constructor can be initialized in several ways:

Detailed Configuration Options

const {Storage} = require('@google-cloud/storage');

// 1. Using Application Default Credentials (Recommended for most cases)
// Automatically discovers credentials from the environment (e.g., GOOGLE_APPLICATION_CREDENTIALS, GCE metadata)
const storage = new Storage();
console.log('Storage client initialized using Application Default Credentials.');

// 2. Explicitly providing a service account key file path
// Useful for local development or CI/CD where ADC is not readily available.
// Ensure the path is correct and the service account has necessary permissions.
// const storageWithKeyFile = new Storage({
//   keyFilename: '/path/to/your/service-account-key.json',
// });
// console.log('Storage client initialized using a service account key file.');

// 3. Explicitly providing service account credentials as an object
// Useful for environments where credentials are provided directly, e.g., from a secret manager.
// const storageWithCredentials = new Storage({
//   credentials: {
//     client_email: '[email protected]',
//     private_key: '-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n',
//   },
// });
// console.log('Storage client initialized using explicit credentials object.');

// 4. Specifying a project ID (optional, often inferred from credentials)
// const storageWithProjectId = new Storage({
//   projectId: 'your-gcp-project-id',
// });
// console.log('Storage client initialized with a specific project ID.');

// 5. Configuring an alternative API endpoint (for custom backends or testing)
// const storageWithEndpoint = new Storage({
//   apiEndpoint: 'https://storage.googleapis.com', // Default
//   // apiEndpoint: 'https://custom.storage.endpoint.com',
// });
// console.log('Storage client initialized with a custom API endpoint.');

// Example of using the initialized client
// async function listBucketsExample() {
//   const [buckets] = await storage.getBuckets();
//   console.log('Buckets:');
//   buckets.forEach(bucket => console.log(`- ${bucket.name}`));
// }
// listBucketsExample().catch(console.error);

Running the Application

This library typically runs as part of a larger Node.js application. You can test individual functionalities using the provided samples/.

To run a sample:

1. Clone the repository:

   git clone https://github.com/GDSC-FSC/nodejs-storage.git
   cd nodejs-storage
   npm install

2. Navigate to the samples directory:

   cd samples
   npm install # Install sample-specific dependencies if any

3. Set up environment variables (e.g., GOOGLE_APPLICATION_CREDENTIALS, GCS_BUCKET_NAME if required by the sample).
4. Execute a sample file:

   node createNewBucket.js your-unique-bucket-name

⬆ Back to Top

---

💡 Usage & Workflows

The client library offers an intuitive API for common and advanced GCS operations. Here are some fundamental workflows.

Basic Operations

1. Creating a New Bucket

// Imports the Google Cloud client library
const {Storage} = require('@google-cloud/storage');

// Creates a client
const storage = new Storage();

/**
 * TODO(developer): Uncomment these variables before running the sample.
 */
const bucketName = 'your-unique-bucket-name'; // The ID of your GCS bucket

async function createNewBucket() {
  // Creates the new bucket
  await storage.createBucket(bucketName);
  console.log(`Bucket ${bucketName} created.`);
}

createNewBucket().catch(console.error);

⬆ Back to Top

2. Uploading a File

const {Storage} = require('@google-cloud/storage');
const storage = new Storage();

/**
 * TODO(developer): Uncomment these variables before running the sample.
 */
const bucketName = 'your-unique-bucket-name'; // The ID of your GCS bucket
const filename = 'your-file-name.txt'; // The path to your file to upload
const destination = 'path/in/bucket/new-file-name.txt'; // The destination file name in your GCS bucket

async function uploadFile() {
  await storage.bucket(bucketName).upload(filename, {
    destination: destination,
  });
  console.log(`${filename} uploaded to ${bucketName}/${destination}.`);
}

uploadFile().catch(console.error);

⬆ Back to Top

3. Downloading a File

const {Storage} = require('@google-cloud/storage');
const storage = new Storage();

/**
 * TODO(developer): Uncomment these variables before running the sample.
 */
const bucketName = 'your-unique-bucket-name'; // The ID of your GCS bucket
const srcFilename = 'path/in/bucket/new-file-name.txt'; // The name of the GCS file to download
const destFilename = 'local-downloaded-file.txt'; // The path to which the file should be downloaded

async function downloadFile() {
  const options = {
    destination: destFilename,
  };

  // Downloads the file
  await storage.bucket(bucketName).file(srcFilename).download(options);
  console.log(`gs://${bucketName}/${srcFilename} downloaded to ${destFilename}.`);
}

downloadFile().catch(console.error);

⬆ Back to Top

4. Listing Files in a Bucket

const {Storage} = require('@google-cloud/storage');
const storage = new Storage();

/**
 * TODO(developer): Uncomment these variables before running the sample.
 */
const bucketName = 'your-unique-bucket-name'; // The ID of your GCS bucket

async function listFiles() {
  // Lists files in the bucket
  const [files] = await storage.bucket(bucketName).getFiles();

  console.log('Files:');
  files.forEach(file => {
    console.log(file.name);
  });
}

listFiles().catch(console.error);

⬆ Back to Top

5. Generating a Signed URL (V4 for Read Access)

Signed URLs provide temporary, limited-privilege access to an object.

const {Storage} = require('@google-cloud/storage');
const storage = new Storage();

/**
 * TODO(developer): Uncomment these variables before running the sample.
 */
const bucketName = 'your-unique-bucket-name'; // The ID of your GCS bucket
const filename = 'path/in/bucket/new-file-name.txt'; // The name of the GCS file
const expiration = Date.now() + 15 * 60 * 1000; // Signed URL will be valid for 15 minutes

async function generateV4ReadSignedUrl() {
  // These options will allow temporary read access to the file
  const options = {
    version: 'v4',
    action: 'read',
    expires: expiration,
  };

  // Get a v4 signed URL for the file
  const [url] = await storage.bucket(bucketName).file(filename).getSignedUrl(options);

  console.log(`The GET signed URL for ${filename} is:\n${url}`);
  console.log('You can use this URL to access the file for the next 15 minutes.');
}

generateV4ReadSignedUrl().catch(console.error);

⬆ Back to Top

Advanced Use Cases

Bulk Uploads with Transfer Manager

The Transfer Manager (transfer-manager.ts) is ideal for efficiently uploading or downloading many files, especially large ones, by managing concurrency and resumability.

const {Storage} = require('@google-cloud/storage');
const storage = new Storage();

/**
 * TODO(developer): Uncomment these variables before running the sample.
 */
const bucketName = 'your-unique-bucket-name'; // The ID of your GCS bucket
const directoryPath = './my-local-directory'; // The path to the directory to upload
const destinationPrefix = 'uploaded-files/'; // Optional: prefix for objects in the bucket

async function uploadDirectoryWithTransferManager() {
  const [files] = await storage.bucket(bucketName).upload(directoryPath, {
    resumable: true, // Enable resumable uploads for better reliability
    preconditionOpts: {
      ifGenerationMatch: 0 // Only upload if object does not exist
    },
    // The TransferManager will handle concurrent uploads
    // You can configure `concurrency` here if needed
    // concurrency: 10,
    destination: destinationPrefix, // All files will be prefixed with this
  });

  console.log(`Uploaded ${files.length} files from ${directoryPath} to gs://${bucketName}/${destinationPrefix}`);
}

uploadDirectoryWithTransferManager().catch(console.error);

Configuring Object Lifecycle Management

Automate storage class transitions or deletions to optimize costs and data retention.

const {Storage} = require('@google-cloud/storage');
const storage = new Storage();

/**
 * TODO(developer): Uncomment these variables before running the sample.
 */
const bucketName = 'your-unique-bucket-name';

async function enableBucketLifecycleManagement() {
  const [metadata] = await storage.bucket(bucketName).setMetadata({
    lifecycle: {
      rule: [
        {
          action: {type: 'Delete'},
          condition: {age: 365}, // Delete objects older than 365 days
        },
        {
          action: {type: 'SetStorageClass', storageClass: 'NEARLINE'},
          condition: {age: 30}, // Move to NEARLINE after 30 days
        },
      ],
    },
  });

  console.log(`Bucket ${bucketName} lifecycle management enabled:`);
  console.log(JSON.stringify(metadata.lifecycle.rule, null, 2));
}

enableBucketLifecycleManagement().catch(console.error);

⬆ Back to Top

---

⚠️ Limitations, Known Issues & Future Roadmap

This section outlines current constraints, identified problems, and the planned evolution of the client library.

Current Limitations

• Rate Limits: Google Cloud Storage has API rate limits. While the client library has built-in retries and exponential backoff, very high-volume operations might still hit these limits, requiring application-level throttling.
• Dependency on GCS Service: Features and capabilities are inherently tied to the Google Cloud Storage service itself. New GCS features require updates to the client library.
• Client-side Encryption: While CSEK and CMEK are supported, client-side encryption of data before it leaves your machine is not directly provided by this library and must be implemented separately.

Known Issues

• Refer to the GitHub Issues page for the most up-to-date list of known bugs and issues.
• If you encounter an issue not listed, please open a new issue following the contributing guidelines.

Future Roadmap

The development team is committed to continuously improving the nodejs-storage client. Planned enhancements include:

• Support for new GCS features: Integrating new capabilities as they are released by the GCS service.
• Performance optimizations: Ongoing work to improve transfer speeds and resource utilization, especially for very large datasets.
• Expanded Transfer Manager capabilities: Adding more fine-grained control and reporting for bulk operations.
• Enhanced samples and documentation: Providing more real-world use cases and detailed guides.
• TypeScript strictness: Continuing to improve type safety and developer experience with TypeScript.

💡 Have a Feature Request?

We welcome your ideas! Please open a feature request on our GitHub repository.

⬆ Back to Top

---

🤝 Contributing & Development Guidelines

We welcome contributions from the community! By participating, you agree to abide by our Code of Conduct.

How to Contribute

1. Read the CONTRIBUTING.md: This document contains detailed instructions on how to contribute, including setup, testing, and PR submission guidelines.
2. Report Bugs: If you find a bug, please open an issue on GitHub.
3. Suggest Enhancements: Have an idea for a new feature or improvement? Open a feature request.
4. Submit Pull Requests (PRs):
   • Fork the repository and create your branch from main.
   • Ensure your code adheres to the project's code style.
   • Write clear commit messages.
   • Include relevant tests for your changes.
   • Open a pull request, referencing any related issues.

Development Setup

1. Clone the repository:

   git clone https://github.com/GDSC-FSC/nodejs-storage.git
   cd nodejs-storage

2. Install dependencies:

   npm install

3. Build the project: The project uses TypeScript. You'll need to compile the TypeScript source files.

   npm run compile # or npm run build

   This will generate JavaScript files in the build/ directory.

Code Style & Quality

• Linting: We use ESLint to enforce code style and best practices.

  npm run lint

• Formatting: Prettier is used for consistent code formatting.

  npm run format

  Please run npm run lint and npm run format before submitting a PR.

Testing

• Unit Tests:

  npm test

• System Tests: These tests interact with actual Google Cloud Storage resources. Ensure your GOOGLE_APPLICATION_CREDENTIALS are set and you have a dedicated bucket for testing.

  npm run system-test

  ⚠️ Caution: System tests may incur costs and modify resources in your GCP project. It's recommended to use a dedicated test project.

⬆ Back to Top

---

© License, Credits & Contact

License

This project is licensed under the Apache 2.0 License. See the LICENSE file for details.

Copyright [yyyy] [name of copyright owner]

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

Acknowledgments

• This project is based on the official Google Cloud Storage Node.js Client.
• Special thanks to all contributors and the broader open-source community for their invaluable feedback and contributions.

Maintainers & Contact

This repository is maintained by GDSC-FSC. For questions, support, or general inquiries, please:

• Open an issue on GitHub.
• Refer to the Google Cloud Community Forums.

⬆ Back to Top

---

📚 Appendix

Changelog

A comprehensive list of changes in each version can be found in the CHANGELOG.md file. This includes new features, bug fixes, and performance improvements.

FAQ

Q: How do I handle large file uploads efficiently?

A: The client library automatically handles resumable uploads for files. For bulk uploads or directories, use the Transfer Manager for optimized, concurrent operations. You can stream files directly using `createWriteStream`.

Q: What are the best practices for authentication in production?

A: In production environments on GCP (e.g., Cloud Run, GKE, Compute Engine), use Application Default Credentials (ADC). This means you don't need to manage service account key files yourself; the environment provides the necessary credentials. Avoid embedding service account key files directly in your codebase.

Q: How can I debug issues with GCS operations?

A: 1. **Enable detailed logging**: Many client libraries offer verbose logging options. Check the library's `debug` or `logging` configurations. 2. **Check GCP Cloud Logging**: GCS access logs and audit logs in Cloud Logging (`Logs Explorer`) provide detailed information on requests made to GCS, including errors and access denials. 3. **Verify permissions**: Most `401 Unauthorized` or `403 Forbidden` errors are due to incorrect IAM permissions for the service account or user making the request. 4. **Network connectivity**: Ensure your application has outbound network access to `storage.googleapis.com`.

Q: Can I use this client with other S3-compatible storage?

A: No, this client is specifically designed for Google Cloud Storage. While GCS offers S3 Interoperability, which allows S3 tools to interact with GCS, this client library uses the native GCS API. For S3-compatible storage, you'd typically use an S3 SDK.

Troubleshooting Guide

Common Errors & Solutions

Error Message	Possible Cause	Solution
401 Unauthorized / 403 Forbidden	Incorrect IAM permissions or expired credentials.	Ensure your service account has the necessary roles/storage.objectAdmin, roles/storage.admin, etc. Verify GOOGLE_APPLICATION_CREDENTIALS is set correctly and points to a valid, unexpired key.
Bucket not found	The specified bucket name is incorrect or does not exist in the project linked to your credentials.	Double-check the bucket name. Ensure the bucket is in the correct GCP project.
The object <filename> does not exist	Attempting to download/access a non-existent file.	Verify the file path and name are correct in the bucket. List files to confirm its presence.
EADDRINUSE / Port already in use	Running multiple instances of your app trying to use the same port.	Stop other instances or configure your application to use a different port.
Slow upload/download speeds	Network congestion, large file sizes without optimization, or insufficient concurrency.	Use Transfer Manager for large/many files. Ensure good network connectivity. Consider setting concurrency in upload/download options.
Memory Exhaustion (JavaScript heap out of memory)	Processing very large files entirely in memory.	For large files, use streams (createReadStream, createWriteStream) to process data in chunks without loading the entire file into memory.

API Reference

For detailed information on all classes, methods, and their parameters, please refer to the official API documentation:

• Google Cloud Storage Node.js Client API Reference
• Google Cloud Storage Documentation

⬆ Back to Top