Skip to content

Conversation

@xxhZs
Copy link
Contributor

@xxhZs xxhZs commented Oct 11, 2025

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

Checklist

  • I have written necessary rustdoc comments.
  • I have added necessary unit tests and integration tests.
  • I have added test labels as necessary.
  • I have added fuzzing tests or opened an issue to track them.
  • My PR contains breaking changes.
  • My PR changes performance-critical code, so I will run (micro) benchmarks and present the results.
  • I have checked the Release Timeline and Currently Supported Versions to determine which release branches I need to cherry-pick this PR into.

Documentation

  • My PR needs documentation updates.
Release note

We can use pubsub like

create table t1(v1 float, v2 float,v3 varchar, v4 varchar);
CREATE SINK s1
FROM
    t1 WITH (
    primary_key = 'v3,v4',
    connector = 'redis',
    redis.url= 'redis://127.0.0.1:6379/',
)FORMAT plain ENCODE TEMPLATE
(redis_value_type ='stream', 
stream = 'test123',
key_format = abcd3:{v3}',
value_format = 'abcd3:{v4}'
);

or

CREATE SINK s1
FROM
    t1 WITH (
    primary_key = 'v2,v3,v4',
    connector = 'redis',
    redis.url= 'redis://127.0.0.1:6379/',
)FORMAT plain ENCODE TEMPLATE
(redis_value_type ='stream', 
stream_column = 'v2',
key_format = abcd3:{v3}',
value_format = 'abcd3:{v4}'
);
insert into t1 values(1.1, 1.1, 'test','test');

value_format Required: Format of the value passed to stream
key_format Required: Format of the key passed to stream
stream stream_column both must set at least one.
stream The name of the message stream to send to
stream_column Use the values in this column as the name of the message stream to send to (type must be varchar)

@github-actions github-actions bot added the type/feature Type: New feature. label Oct 11, 2025
@xxhZs xxhZs requested review from Copilot and hzxa21 October 13, 2025 03:27
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds support for Redis Stream data type to the Redis sink connector in RisingWave, expanding beyond the existing support for strings, geospatial data, and pub/sub channels.

  • Added REDIS_VALUE_TYPE_STREAM support for Redis Streams
  • Implemented stream configuration with stream and stream_column parameters
  • Added template encoders for Redis Stream key-value pairs

Reviewed Changes

Copilot reviewed 4 out of 5 changed files in this pull request and generated 1 comment.

File Description
src/connector/src/sink/redis.rs Added stream constants and pipeline operations for Redis streams
src/connector/src/sink/formatter/mod.rs Added stream value type handling in template encoder build logic
src/connector/src/sink/encoder/template.rs Implemented stream encoders and renamed PubSub encoders for consistency
src/connector/Cargo.toml Added "streams" feature to redis dependency
Comments suppressed due to low confidence (1)

src/connector/src/sink/formatter/mod.rs:1

  • Missing space after comma in error messages. Should be 'Cannot find '{VALUE_FORMAT}', please set it.' and 'Cannot find '{KEY_FORMAT}', please set it.'
// Copyright 2025 RisingWave Labs

|| (stream.is_some() && stream_column.is_some())
{
return Err(SinkError::Config(anyhow!(
"`{STREAM}` and `{STREAM_COLUMN}` only one can be set"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nits:

Please specific either `{STREAM}` or `{STREAM_COLUMN}`. They are mutually exclusive options.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this expected? Should we add .vscode in gitignore?

test added 2 commits October 22, 2025 16:33
@xxhZs xxhZs force-pushed the xxh/support-redis-stream branch from d7b4ad4 to 0d91bea Compare October 22, 2025 08:35
@xxhZs xxhZs requested a review from hzxa21 October 22, 2025 08:38
Copy link
Collaborator

@hzxa21 hzxa21 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@hzxa21 hzxa21 added the user-facing-changes Contains changes that are visible to users label Oct 27, 2025
@hzxa21
Copy link
Collaborator

hzxa21 commented Oct 27, 2025

Please update the PR description. Also, please mention that this is a license feature in the PR description and the release note.

@tabVersion
Copy link
Contributor

also request review from redis expert @shanicky :doge

@tabVersion tabVersion requested a review from shanicky October 28, 2025 08:03
Copy link
Contributor

@tabVersion tabVersion left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No functionality tests on CI?

@hzxa21
Copy link
Collaborator

hzxa21 commented Oct 28, 2025

No functionality tests on CI?

@xxhZs can we add redis stream e2e test? It seems that this feature is available in open source edition.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci/run-e2e-redis-sink-tests type/feature Type: New feature. user-facing-changes Contains changes that are visible to users

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants