Skip to content

Commit 865a1e1

Browse files
kanakbkanakb
committed
Initial commit
0 parents  commit 865a1e1

File tree

11 files changed

+1638
-0
lines changed

11 files changed

+1638
-0
lines changed

‎.gitignore‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
cscope.*
2+
.classpath
3+
.project
4+
.svn
5+
target/
6+
.idea
7+
*.iml
8+
*.ipr
9+
*.iws
10+
.settings/
11+
out/
12+
.DS_Store

‎LICENSE‎

Lines changed: 307 additions & 0 deletions
Large diffs are not rendered by default.

‎NOTICE‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
kafka-assigner
2+
Copyright 2016 Sift Science.
3+
4+
I. Included Software
5+
6+
This product includes software developed at
7+
Sift Science (http://www.siftscience.com/).
8+
Licensed under the Apache License 2.0.
9+
10+
This product includes software developed at
11+
The Apache Software Foundation (http://www.apache.org/).
12+
Licensed under the Apache License 2.0.
13+
14+
This product includes software developed at
15+
args4j (http://args4j.kohsuke.org/).
16+
Licensed under the MIT License.
17+
18+
This product includes software developed at
19+
junit (http://junit.org/).
20+
Licensed under the Eclipse Public License.
21+
22+
This product includes software developed at
23+
Google (http://www.google.com/).
24+
Licensed under the Apache License 2.0.
25+
26+
This product includes software developed at
27+
json.org (http://www.json.org/).
28+
Licensed under the JSON License.
29+
30+
This product includes software developed at
31+
zkclient (https://github.com/sgroschupf/zkclient).
32+
Licensed under the Apache License 2.0.
33+
34+
This product includes software developed at
35+
scala (http://www.scala-lang.org/).
36+
Licensed under the Scala License.
37+
38+
II. License Summary
39+
- Apache License 2.0
40+
- MIT License
41+
- Eclipse Public License
42+
- JSON License
43+
- Scala License

‎README.md‎

Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,85 @@
1+
kafka-assigner
2+
==============
3+
This is a rack-aware tool for assigning Kafka partitions to brokers that minimizes data movement. It also includes the ability to inspect the current live brokers in the cluster and the current partition assignment.
4+
5+
**Using this tool will greatly simplify operations like decommissioning a broker, adding a new broker, or replacing a broker.**
6+
7+
# Why is this necessary?
8+
Kafka's built-in algorithm is easy to use and monitor, but it does not take into account existing assignments of partitions to nodes. Instead, the burden is on the operator to either move entire topics across brokers, or come up with a sane way of moving some number of partitions of existing topics. This is extremely disruptive.
9+
10+
This tool _minimizes_ the number of partitions already assigned that need to leave a given node, while ensuring that each broker is responsible for a similar number of partitions. This enables use cases like node replacement, in which we would like to bring up a broker that is responsible for the same data as a misbehaving broker that it is replacing.
11+
12+
# How does this work?
13+
This tool uses a strategy that behaves similarly to [Apache Helix](http://helix.apache.org)'s auto-rebalancing algorithm. It first assigns as many already-assigned partitions back to nodes as it can (while ensuring that no node is overloaded), and then evenly assigns all other partitions such that every node eventually ends up responsible for roughly the same number of partitions.
14+
15+
# How is this tool used?
16+
17+
## Get the tool
18+
1. Download from the "Releases" page
19+
2. `tar xf kafka-assigner-1.0-pkg.tar`
20+
3. `cd kafka-assigner-1.0/bin`
21+
22+
## Run the tool
23+
Requires Java 1.7+
24+
25+
```
26+
./kafka-assignment-generator.sh [options...] arguments...
27+
--broker_hosts VAL : comma-separated list of broker
28+
hostnames (instead of broker IDs)
29+
--broker_hosts_to_remove VAL : comma-separated list of broker
30+
hostnames to exclude (instead of
31+
broker IDs)
32+
--disable_rack_awareness : set to true to ignore rack
33+
configurations
34+
--integer_broker_ids VAL : comma-separated list of Kafka broker
35+
IDs (integers)
36+
--mode [PRINT_CURRENT_ASSIGNMENT | : the mode to run (PRINT_CURRENT_ASSIGNM
37+
PRINT_CURRENT_BROKERS | ENT, PRINT_CURRENT_BROKERS,
38+
PRINT_REASSIGNMENT] PRINT_REASSIGNMENT)
39+
--topics VAL : comma-separated list of topics
40+
--zk_string VAL : ZK quorum as comma-separated
41+
host:port pairs
42+
```
43+
44+
### Example: reassign partitions to all live hosts
45+
```
46+
./kafka-assignment-generator.sh --zk_string my-zk-host:2181 --mode PRINT_REASSIGNMENT
47+
```
48+
49+
The output JSON can then be fed into Kafka's reassign partitions command. See [here](http://kafka.apache.org/0100/ops.html#basic_ops_partitionassignment) for instructions.
50+
51+
### Example: reassign partitions to all but a few live hosts
52+
This mode is useful for decommissioning or replacing a node. The partitions will be assigned to all live hosts, excluding the hosts that are specified.
53+
```
54+
./kafka-assignment-generator.sh --zk_string my-zk-host:2181 --mode PRINT_REASSIGNMENT --broker_hosts_to_remove misbehaving-host1,misbehaving-host2
55+
```
56+
57+
The output JSON can then be fed into Kafka's reassign partitions command. See [here](http://kafka.apache.org/0100/ops.html#basic_ops_partitionassignment) for instructions.
58+
59+
### Example: reassign partitions to specific hosts
60+
Note that in this mode, it is expected that every host that should own partitions should be specified, including existing ones.
61+
```
62+
./kafka-assignment-generator.sh --zk_string my-zk-host:2181 --mode PRINT_REASSIGNMENT --broker_hosts host1,host2,host3
63+
```
64+
65+
The output JSON can then be fed into Kafka's reassign partitions command. See [here](http://kafka.apache.org/0100/ops.html#basic_ops_partitionassignment) for instructions.
66+
67+
### Example: print current brokers
68+
```
69+
./kafka-assignment-generator.sh --zk_string my-zk-host:2181 --mode PRINT_CURRENT_BROKERS
70+
```
71+
72+
### Example: print current assignment
73+
```
74+
./kafka-assignment-generator.sh --zk_string my-zk-host:2181 --mode PRINT_CURRENT_ASSIGNMENT
75+
```
76+
77+
# Building
78+
Requires Java 1.7+ and Maven 3.2+
79+
80+
1. Clone this repository
81+
2. `mvn install package`
82+
3. Artifacts are in `target/kafka-assigner-pkg`
83+
84+
# License
85+
Licensed under the Apache License 2.0.

‎pom.xml‎

Lines changed: 133 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,133 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
3+
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
4+
<modelVersion>4.0.0</modelVersion>
5+
<groupId>siftscience</groupId>
6+
<artifactId>kafka-assigner</artifactId>
7+
<packaging>jar</packaging>
8+
<version>1.0</version>
9+
10+
<name>kafka-assigner</name>
11+
<description>Tools for reassigning Kafka partitions with minimal movement</description>
12+
<url>http://maven.apache.org</url>
13+
14+
<licenses>
15+
<license>
16+
<name>Apache License, Version 2.0</name>
17+
<url>http://www.apache.org/licenses/LICENSE-2.0.txt</url>
18+
<distribution>repo</distribution>
19+
</license>
20+
</licenses>
21+
22+
<dependencies>
23+
<dependency>
24+
<groupId>junit</groupId>
25+
<artifactId>junit</artifactId>
26+
<version>4.8.1</version>
27+
<scope>test</scope>
28+
</dependency>
29+
<dependency>
30+
<groupId>args4j</groupId>
31+
<artifactId>args4j</artifactId>
32+
<version>2.0.29</version>
33+
</dependency>
34+
<dependency>
35+
<groupId>com.google.guava</groupId>
36+
<artifactId>guava</artifactId>
37+
<version>13.0.1</version>
38+
</dependency>
39+
<dependency>
40+
<groupId>org.json</groupId>
41+
<artifactId>json</artifactId>
42+
<version>20131018</version>
43+
</dependency>
44+
<dependency>
45+
<groupId>org.apache.commons</groupId>
46+
<artifactId>commons-lang3</artifactId>
47+
<version>3.1</version>
48+
</dependency>
49+
<dependency>
50+
<groupId>org.apache.kafka</groupId>
51+
<artifactId>kafka_2.11</artifactId>
52+
<version>0.10.0.0</version>
53+
</dependency>
54+
<dependency>
55+
<groupId>org.apache.kafka</groupId>
56+
<artifactId>kafka-clients</artifactId>
57+
<version>0.10.0.0</version>
58+
</dependency>
59+
</dependencies>
60+
61+
<build>
62+
<defaultGoal>clean install</defaultGoal>
63+
<plugins>
64+
65+
<plugin>
66+
<groupId>org.codehaus.mojo</groupId>
67+
<artifactId>appassembler-maven-plugin</artifactId>
68+
<version>1.1.1</version>
69+
<configuration>
70+
<binFileExtensions>
71+
<unix>.sh</unix>
72+
</binFileExtensions>
73+
<!-- Set the target configuration directory to be used in the bin scripts -->
74+
<configurationDirectory>conf</configurationDirectory>
75+
<!-- Copy the contents from "/src/main/config" to the target configuration directory in the assembled application -->
76+
<copyConfigurationDirectory>true</copyConfigurationDirectory>
77+
<!-- Include the target configuration directory in the beginning of the classpath declaration in the bin scripts -->
78+
<includeConfigurationDirectoryInClasspath>true</includeConfigurationDirectoryInClasspath>
79+
<assembleDirectory>${project.build.directory}/${project.artifactId}-pkg</assembleDirectory>
80+
<!-- Extra JVM arguments that will be included in the bin scripts -->
81+
<extraJvmArguments>-Xms512m -Xmx512m</extraJvmArguments>
82+
<!-- Generate bin scripts for windows and unix pr default -->
83+
<platforms>
84+
<platform>windows</platform>
85+
<platform>unix</platform>
86+
</platforms>
87+
<programs>
88+
<program>
89+
<mainClass>siftscience.kafka.tools.KafkaAssignmentGenerator</mainClass>
90+
<name>kafka-assignment-generator</name>
91+
</program>
92+
</programs>
93+
</configuration>
94+
<executions>
95+
<execution>
96+
<phase>package</phase>
97+
<goals>
98+
<goal>assemble</goal>
99+
</goals>
100+
</execution>
101+
</executions>
102+
</plugin>
103+
<plugin>
104+
<groupId>org.apache.maven.plugins</groupId>
105+
<artifactId>maven-assembly-plugin</artifactId>
106+
<version>2.3</version>
107+
<configuration>
108+
<descriptors>
109+
<descriptor>${project.basedir}/src/assemble/assembly.xml</descriptor>
110+
</descriptors>
111+
</configuration>
112+
<executions>
113+
<execution>
114+
<phase>package</phase>
115+
<goals>
116+
<goal>single</goal>
117+
</goals>
118+
</execution>
119+
</executions>
120+
</plugin>
121+
<plugin>
122+
<groupId>org.apache.maven.plugins</groupId>
123+
<artifactId>maven-compiler-plugin</artifactId>
124+
<version>3.0</version>
125+
<configuration>
126+
<source>1.5</source>
127+
<target>1.5</target>
128+
</configuration>
129+
</plugin>
130+
</plugins>
131+
</build>
132+
133+
</project>

‎src/assemble/assembly.xml‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<!--
3+
Licensed to the Apache Software Foundation (ASF) under one
4+
or more contributor license agreements. See the NOTICE file
5+
distributed with this work for additional information
6+
regarding copyright ownership. The ASF licenses this file
7+
to you under the Apache License, Version 2.0 (the
8+
"License"); you may not use this file except in compliance
9+
with the License. You may obtain a copy of the License at
10+
11+
http://www.apache.org/licenses/LICENSE-2.0
12+
13+
Unless required by applicable law or agreed to in writing,
14+
software distributed under the License is distributed on an
15+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
16+
KIND, either express or implied. See the License for the
17+
specific language governing permissions and limitations
18+
under the License.
19+
-->
20+
<assembly>
21+
<id>pkg</id>
22+
<formats>
23+
<format>tar</format>
24+
</formats>
25+
<fileSets>
26+
<fileSet>
27+
<directory>${project.build.directory}/${project.artifactId}-pkg/bin</directory>
28+
<outputDirectory>bin</outputDirectory>
29+
<lineEnding>unix</lineEnding>
30+
<fileMode>0755</fileMode>
31+
<directoryMode>0755</directoryMode>
32+
</fileSet>
33+
<fileSet>
34+
<directory>${project.build.directory}/${project.artifactId}-pkg/repo/</directory>
35+
<outputDirectory>repo</outputDirectory>
36+
<fileMode>0755</fileMode>
37+
<directoryMode>0755</directoryMode>
38+
<excludes>
39+
<exclude>**/*.xml</exclude>
40+
</excludes>
41+
</fileSet>
42+
<fileSet>
43+
<directory>${project.build.directory}/${project.artifactId}-pkg/conf</directory>
44+
<outputDirectory>conf</outputDirectory>
45+
<lineEnding>unix</lineEnding>
46+
<fileMode>0755</fileMode>
47+
<directoryMode>0755</directoryMode>
48+
</fileSet>
49+
<fileSet>
50+
<directory>${project.basedir}</directory>
51+
<outputDirectory>/</outputDirectory>
52+
<includes>
53+
<include>LICENSE</include>
54+
<include>NOTICE</include>
55+
</includes>
56+
<fileMode>0755</fileMode>
57+
</fileSet>
58+
</fileSets>
59+
</assembly>

‎src/main/config/log4j.properties‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
#
2+
# Licensed to the Apache Software Foundation (ASF) under one
3+
# or more contributor license agreements. See the NOTICE file
4+
# distributed with this work for additional information
5+
# regarding copyright ownership. The ASF licenses this file
6+
# to you under the Apache License, Version 2.0 (the
7+
# "License"); you may not use this file except in compliance
8+
# with the License. You may obtain a copy of the License at
9+
#
10+
# http://www.apache.org/licenses/LICENSE-2.0
11+
#
12+
# Unless required by applicable law or agreed to in writing,
13+
# software distributed under the License is distributed on an
14+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
# KIND, either express or implied. See the License for the
16+
# specific language governing permissions and limitations
17+
# under the License.
18+
#
19+
20+
# Set root logger level to DEBUG and its only appender to A1.
21+
log4j.rootLogger=ERROR,A1
22+
23+
# A1 is set to be a ConsoleAppender.
24+
log4j.appender.A1=org.apache.log4j.ConsoleAppender
25+
26+
# A1 uses PatternLayout.
27+
log4j.appender.A1.layout=org.apache.log4j.PatternLayout
28+
log4j.appender.A1.layout.ConversionPattern=%-4r [%t] %-5p %c %x - %m%n
29+
30+
log4j.logger.org.I0Itec=ERROR
31+
log4j.logger.org.apache=ERROR

0 commit comments

Comments
 (0)