Add produce batch api to producer

[k-raina]

This pull request adds a new batch message production API to the Kafka Python client, enabling efficient sending of multiple messages in a single call.

Changes

• Added Producer_produce_batch C function to support producing a batch of messages, including per-message and global delivery callbacks, input validation, and error reporting on failed messages. Message headers are parsed but ignored due to underlying library limitations, and timestamps are currently not supported in batch mode. (src/confluent_kafka/src/Producer.c,
• Registered the new produce_batch method in the Python API, with detailed documentation describing usage, parameters, limitations, and examples.

Testing

Test Name	Total Scenarios	Detailed Scenarios Tested
test_produce_batch_core_functionality()	36	• Mixed data types (bytes/string/unicode/None/empty) • Zero-length vs None distinction • Mixed encoding strings (UTF-8, emoji, Latin, Cyrillic, CJK) • Binary data edge cases (null bytes, all byte values) • Partition handling, empty batch, single message • Large batch (100 messages) • Advanced Unicode and encoding • Large messages and scalability • Batch size scalability (1, 10, 100, 500) • Timestamp not supported (NotImplementedError) • Headers parsed but ignored
test_produce_batch_validation_and_errors()	27	• Invalid input types (not list, not dict, wrong types) • Message field validation (value/key/partition types) • Partition edge values (negative, max int32, large values) • Unexpected message fields (ignored gracefully) • Special topic names validation • Missing/wrong argument types • Invalid topic names (empty, very long, None) • Non-callable callbacks • Queue buffer limits and partial failures • Error annotations on failed messages • Restrictive producer limits • All messages failing scenario
test_produce_batch_callbacks_and_exceptions()	18+	• Mixed callback scenarios (per-message vs global) • Callback distribution verification • No callbacks scenario • Callback parameter aliases (callback vs on_delivery) • Circular reference in callback • Slow/blocking callbacks (performance impact) • Exception propagation during flush • Multiple callback exceptions • Critical callback errors • RuntimeError and ValueError handling
test_produce_batch_concurrency_and_threading()	19+	• Different producer configurations (acks, compression, batch.size, linger.ms) • Thread safety with shared producer • Multiple threads with shared data races • Rapid successive batch calls • Resource contention scenarios • Racing threads with shared data structures • Message list modification during callback • Recursive produce_batch calls from callback • Corruption attempt handling
test_produce_batch_memory_and_resources()	11	• Maximum message count stress test (5000 messages) • Deep nested message structures (500 messages) • Memory cleanup verification with GC • Producer destruction during active callbacks • Memory pressure scenarios • Many producers with batch operations (10 producers) • Rapid batch creation and destruction (20 iterations) • Handle exhaustion scenarios (50 handles) • Resource cleanup verification
test_produce_batch_limits_and_performance()	8+	• BufferError with small queue limits • Very large batch size (1000 messages) • Single very large message (1MB) • Mixed success/failure with queue limits • Error annotation verification • Performance with different batch sizes • Per-message time performance verification • Concurrent performance testing (5 workers) • Performance scaling validation
test_produce_batch_integrations()	6	• Transactional producer with batch operations • Transaction initialization and commit • Transaction error handling • SerializingProducer compatibility • StringSerializer integration • AvroProducer compatibility • Schema-serialized data handling • Type error handling for serialization

Librdkafka Produce batch

https://github.com/confluentinc/librdkafka/blob/13a2bbae88d8be073986f974db1b853cfc91ae8f/src/rdkafka_msg.c#L710-L837

[confluent-cla-assistant]

🎉 All Contributor License Agreements have been signed. Ready to merge.
_{Please push an empty commit if you would like to re-run the checks to verify CLA status for all contributors.}

[sonarqube-confluent]

• 67.50% Coverage on New Code (is less than 80.00%)

Analysis Details

24 Issues

• 2 Bugs
• 0 Vulnerabilities
• 22 Code Smells

Coverage and Duplications

• 67.50% Coverage (66.10% Estimated after merge)
• No duplication information (5.10% Estimated after merge)

Project ID: confluent-kafka-python

View in SonarQube