fix bug: urldecode difference between python & java

.github/workflows/docker-images-build-and-push.yml

@@ -0,0 +1,43 @@
+name: Docker build and push
+
+on:
+  push:
+    # Publish `master` as Docker `latest` image.
+    branches:
+      - master
+
+    # Publish `v1.2.3` tags as releases.
+    tags:
+      - v*
+
+jobs:
+  # no test is required
+  push:
+    runs-on: ubuntu-18.04
+    if: github.event_name == 'push'
+
+    steps:
+      - uses: actions/checkout@v2
+
+      - name: Prepare the TAG
+        id: prepare-the-tag
+        run: |
+          # strip git ref prefix from version
+          TAG=""
+          VERSION=$(echo "${{ github.ref }}" | sed -e 's,.*/\(.*\),\1,')
+          if [ $VERSION = "master" ]; then
+            TAG=latest
+          else
+            TAG=${VERSION##*v}-release
+          fi
+          echo "::notice col=5 title=print tag::TAG=$TAG"
+          echo "::set-output name=tag::$TAG"
+
+      - name: Build image
+        run: TAG=${{steps.prepare-the-tag.outputs.tag}} bash docker-build/build_cluster_docker.sh all
+
+      - name: Log into DockerHub
+        run: docker login -u ${{ secrets.DOCKERHUB_USERNAME }} -p ${{ secrets.DOCKERHUB_TOKEN }}
+
+      - name: Push image
+        run: TAG=${{steps.prepare-the-tag.outputs.tag}} bash docker-build/build_cluster_docker.sh push

.readthedocs.yml

@@ -0,0 +1,12 @@
+version: 2
+
+mkdocs:
+  configuration: document/mkdocs.yml
+  fail_on_warning: false
+
+formats: all
+
+python:
+  version: 3.7
+  install:
+    - requirements: document/docs/requirements.txt

README.md

@@ -18,6 +18,6 @@ FATE-Serving is a high-performance, industrialized serving system for federated
 
 
 ###  document
-[document](https://webank-ai-1251170195.cos.ap-guangzhou.myqcloud.com/FATE-SERVING-2.0.html    ) 
+[document](https://fate-serving.readthedocs.io/en/develop/) 
 
 

RELEASE.md

@@ -1,3 +1,36 @@
+# Release 2.1.5
+## Major Features and Improvements
+* Upgrade log4j dependent version.
+
+# Release 2.1.4
+## Major Features and Improvements
+* Upgrade SpringBoot dependent version.
+
+# Release 2.1.3
+## Major Features and Improvements
+* Fix the HttpAdapter thread leakage problem.
+* Upgrade log4j dependent version.
+
+# Release 2.1.2
+## Major Features and Improvements
+* Fix serious security vulnerabilities in log4j.
+
+# Release 2.1.1
+## Major Features and Improvements
+* Fix the bug that the serving-server service fails to start when zookeeper is not used as the service registration and coordination center.
+
+# Release 2.1.0
+## Major Features and Improvements
+* Support multi-host algorithm model online prediction,currently, the supported multi-host algorithms include HeteroLR and HeteroSecureBoost.
+* Support to display model information in pipeline on admin page.
+* Increase the model synchronization function, support the synchronous replication of models between different serving-server nodes in the cluster, which is convenient for machine expansion and contraction.
+* Admin page model operation/display function optimization.
+* Add a cluster one-click self-check tool on the admin page, you can view the cluster service self-check status, and the cluster will be self-checked regularly from the background, and the user can also take the initiative to perform one-click self-check.
+* Add routing table editing function, you can directly edit the routing table information of the current serving-proxy configuration through the page.
+* Service topology map optimization, showing the true mapping relationship between each machine, the specified machine can be selected on the topology map and its details can be viewed.
+* Add support for http protocol between guest and host.
+* Add a general httpAdaptor for host to obtain features.
+
 # Release 2.0.0
 ## Major Features and Improvements
 *  For single inference, the guest side and host side of version 2.0. * will be calculated in parallel, thus reducing the time consumption.

docker-build/.env

@@ -0,0 +1,3 @@
+#PREFIX=federatedai
+#TAG=2.1.5-release
+#version=2.1.5

docker-build/build_cluster_docker.sh

@@ -0,0 +1,78 @@
+########################################################
+# Copyright 2019-2020 program was created VMware, Inc. #
+# SPDX-License-Identifier: Apache-2.0                  #
+########################################################
+
+#!/usr/bin/env bash
+set -e
+
+BASEDIR=$(dirname "$0")
+cd $BASEDIR
+WORKINGDIR=`pwd`
+
+source_code_dir=$(cd `dirname ${WORKINGDIR}`; pwd)
+
+
+if [ -z "$TAG" ]; then
+        TAG=latest
+fi
+if [ -z "$PREFIX" ]; then
+        PREFIX=federatedai
+fi
+
+version=$(grep "<fate.version>" ${source_code_dir}/pom.xml  | sed "s/[<|>]/ /g" | awk '{print $2}')
+
+source .env
+
+echo "Docker build"
+echo "Info:"
+echo "  version: ${version}"
+echo "  PREFIX: ${PREFIX}"
+echo "  Tag: ${TAG}"
+echo "  BASEDIR: ${BASEDIR}"
+echo "  WORKINGDIR: ${WORKINGDIR}"
+echo "  source_code_dir: ${source_code_dir}"
+
+package() {
+  docker run --rm -u $(id -u):$(id -g) -v ${source_code_dir}:/data/projects/fate/FATE-Serving --entrypoint="" maven:3.6-jdk-8 /bin/bash -c "cd /data/projects/fate/FATE-Serving && mvn clean package -DskipTests"
+}
+
+buildModule() {
+  for module in "serving-proxy" "serving-server" "serving-admin"
+  do
+      echo "### START BUILDING ${module} ###"
+      docker build --build-arg version=${version} -t ${PREFIX}/${module}:${TAG} -f ${source_code_dir}/docker-build/${module}/Dockerfile ${source_code_dir}
+      echo "### FINISH BUILDING ${module} ###"
+      echo ""
+  done;
+}
+
+pushImage() {
+  ## push image
+  for module in "serving-proxy" "serving-server" "serving-admin"
+  do
+      echo "### START PUSH ${module} ###"
+      docker push ${PREFIX}/${module}:${TAG}
+      echo "### FINISH PUSH ${module} ###"
+      echo ""
+  done;
+}
+
+while [ "$1" != "" ]; do
+    case $1 in
+         modules)
+                 buildModule
+                 ;;
+         package)
+                 package
+                 ;;
+         all)
+                 package
+                 buildModule
+                 ;;
+         push)
+                pushImage
+                ;;
+    esac
+    shift
+done

docker-build/serving-admin/Dockerfile

@@ -0,0 +1,25 @@
+FROM mcr.microsoft.com/java/jre:8u192-zulu-alpine as builder
+
+
+WORKDIR /data/projects/fate-serving/serving-admin/
+USER root
+
+ARG version
+
+COPY fate-serving-admin/target/fate-serving-admin-${version}-release.zip ./
+
+RUN unzip fate-serving-admin-${version}-release.zip && \
+    rm fate-serving-admin-${version}-release.zip && \
+    ln -s fate-serving-admin-*.jar fate-serving-admin.jar
+
+FROM mcr.microsoft.com/java/jre:8u192-zulu-centos
+
+WORKDIR /data/projects/fate-serving/serving-admin/
+
+COPY --from=builder /data/projects/fate-serving/serving-admin  /data/projects/fate-serving/serving-admin
+
+USER root
+
+EXPOSE 8350
+
+CMD java -Dspring.config.location=conf/application.properties -cp "conf/:lib/*:fate-serving-admin.jar" com.webank.ai.fate.serving.admin.Bootstrap

docker-build/serving-proxy/Dockerfile

@@ -0,0 +1,25 @@
+FROM mcr.microsoft.com/java/jre:8u192-zulu-alpine as builder
+
+WORKDIR /data/projects/fate-serving/serving-proxy/
+
+USER root
+
+ARG version
+
+COPY fate-serving-proxy/target/fate-serving-proxy-${version}-release.zip ./
+
+RUN unzip fate-serving-proxy-${version}-release.zip && \
+    rm fate-serving-proxy-${version}-release.zip && \
+    ln -s fate-serving-proxy-*.jar fate-serving-proxy.jar
+
+FROM mcr.microsoft.com/java/jre:8u192-zulu-centos
+
+WORKDIR /data/projects/fate-serving/serving-proxy/
+
+COPY --from=builder /data/projects/fate-serving/serving-proxy  /data/projects/fate-serving/serving-proxy
+
+USER root
+
+EXPOSE 8059 8869 8879
+
+CMD java -Dspring.config.location=conf/application.properties -cp conf/:lib/*:fate-serving-proxy.jar com.webank.ai.fate.serving.proxy.bootstrap.Bootstrap -c conf/application.properties

docker-build/serving-server/Dockerfile

@@ -0,0 +1,25 @@
+FROM mcr.microsoft.com/java/jre:8u192-zulu-alpine as builder
+
+WORKDIR /data/projects/fate-serving/serving-server/
+
+USER root
+
+ARG version
+
+COPY fate-serving-server/target/fate-serving-server-${version}-release.zip ./
+
+RUN unzip fate-serving-server-${version}-release.zip && \
+    rm fate-serving-server-${version}-release.zip && \
+    ln -s fate-serving-server-*.jar fate-serving-server.jar
+
+FROM mcr.microsoft.com/java/jre:8u192-zulu-centos
+
+WORKDIR /data/projects/fate-serving/serving-server/
+
+COPY --from=builder /data/projects/fate-serving/serving-server  /data/projects/fate-serving/serving-server
+
+USER root
+
+EXPOSE 8000
+
+CMD java -cp conf/:lib/*:extension/*:fate-serving-server.jar com.webank.ai.fate.serving.Bootstrap -c conf/serving-server.properties

document/docs/algo/base.md

@@ -0,0 +1,80 @@
+### 基础类
+PipelineModelProcessor类是模型处理逻辑的实现类，serving-server在收到推送模型的请求后，会在内存中初始化一个PipelineModelProcessor实例，该实例中包含了由模型中各算法组件组成的pipeline。pipeline处理逻辑分为两个阶段：  
+1) 本地模型处理 （这一步在Guest方，以及Host方都会存在）  
+2) 合并远端数据 （这一步只在Guest方存在）  
+
+BaseComponent 类是所有算法组件的基类，它实现了LocalInferenceAware接口  
+```java
+public interface LocalInferenceAware {
+​
+    public Map<String, Object> localInference(Context context, List<Map<String, Object>> input);
+​
+}
+```
+PipelineModelProcessor中的pipeline在检测到组件为LocalInferenceAware时，会调用localInference方法来进行本地预测。
+另外一个重要的接口MergeInferenceAware，实现该接口的组件可以执行合并远端返回数据的逻辑。  
+```java
+public interface MergeInferenceAware {
+​
+    public Map<String, Object> mergeRemoteInference(Context context, List<Map<String, Object>> localData, Map<String, Object> remoteData);
+}
+```
+
+### HeteroSecureBoost 在线推理
+HeteroSecureBoost 为纵向联邦SecureBoost在线推理的实现过程,  
+它主要包含3个文件"HeteroSecureBoost", "HeteroSecureBoostingTreeGuest", "HeteroSecureBoostingTreeHost"，下面对这三个文件展开说明
+
+#### HeteroSecureBoost
+HeteroSecureBoost是HeteroSecureBoostingTreeGuest和HeteroSecureBoostingTreeHost的模型基类，该基类提供了模型的初始化、层次遍历等功能。
+
+1. HeteroSecureBoost继承自BaseComponent类，BaseComponent类为所有模型的基类，所有算法模型必须继承该类和实现相关接口，用于统一的调度。
+2. 模型初始化：initModel函数，功能是对输入的Meta和Param两个序列化的模型文件进行反序列话，同时，初始化相关的类属性，初始化的内容包括：  
+treeNum: 树的数量
+initScore: boost的初始化得分，具体可参考FATE离线建模文档
+trees: 具体的树信息列表，可参考对应的DecisionTreeModelParam
+numClasses: 多少类，二分类问题为2，多分类问题则为具体分类数，回归问题为0，通过该字段可以判断具体建模任务类型
+classes: 类别标签，对于分类问题，用预测的结果下表去索引真正的分类标签
+treeDim: boost的每轮树的数量，对于回归和二分类等于1，对于多分类，是类别数量，每轮每个分类都有一个对应的树
+learningRate: 学习率和权重放缩因子，推理时每个树得到的权重都会乘以learning_rate。
+3. 功能函数说明：  
+getSite: 离线的时候，每个树节点的域信息是role:partyid，如host:10000，通过该函数获取role
+generateTag: 用来存储和读取每轮使用的数据，用法在下面辉介绍
+gotoNextLevel: 输入当前的树、节点编号，特征值，输出树的下一层节点编号  
+
+#### HeteroSecureBoostingTreeGuest and HeteroSecureBoostingTreeHost
+HeteroSecureBoostingTreeGuest和HeteroSecureBoostingTreeHost是party guest和host对应的实现代码，其中party guest收到请求后，会执行推理流程，与此同时，需要与host一起决策每个树的预测流程，在2.0.x版本中，serving对预测流程做了专门优化，现预测流程如下:
+
+1. HeteroSecureBoostingTreeGuest 首先执行本地推导，对于所有的树进行遍历，如果某棵树遇到host节点，则将host节点记录；如果所有树都没遇到host节点，则得到叶子节点权重，跳至步骤5。
+2. HeteroSecureBoostingTreeGuest 遇到的host节点，通过通信通知Host。
+3. Host 接收到推理指令后，执行完整的推理调用逻辑（数据预处理 -> 特征工程 -> HeteroSecureBoost)，完成数据处理后，Host本地提取节点路由，对于该预测的样本，对于所有自身拥有的Host节点，判断该样本的遍历方向（向左还是向右），并把所有的路由方向记录下来。
+4. Host将提取出的路由方向整理发送至Guest
+5. HeteroSecureBoostingTreeGuest继续推理逻辑，由于已经获得Host方发送的路由表，则在后续的推理过程中，遇到任何的Host的节点，都可以知道遍历方向。 Guest从记录的host节点继续往下推导，直至到达叶子节点。
+6. Guest得到每个树的节点编号，利用节点编号索引出叶子权重，经过处理后得到预测结果，并将结果输出，推理流程完成。  
+
+当前的HeteroSecureBoost交互次数最坏情况下也仅需要1次数据交换。
+另外，HeteroSecureBoost在2.0.x版本开始支持批量预测。
+
+HeteroSecureBoost在线推理流程图如下：
+![inference_flow](../img/inference_flow.jpg)
+
+### HeteroLR 在线推理
+HeteroLR为纵向联邦逻辑回归在线推理的实现过程。
+它主要包含3个文件"HeteroLRBoost", "HeteroLRGuest", "HeteroLRHost"，下面对这三个文件展开说明
+#### HeteroLR
+HeteroLR是HeteroLRGuest和HeteroLRHost的模型基类，该基类提供了模型的初始化、模型评分等功能。
+
+1. HeteroLR继承自BaseComponent类，BaseComponent类为所有模型的基类，所有算法模型必须继承该类和实现相关接口，用于统一的调度。
+2. 模型初始化: initModel函数，功能是对输入的Meta和Param两个序列化的模型文件进行反序列话，同时，初始化相关的类属性，初始化的内容包括:  
+    >weight: LR模型的权值  
+    >intercept: LR模型的偏置
+
+3. 功能函数说明:  
+    forward: 计算score = weight * value + intercept, 若是host方，则intercept为0
+
+#### HeteroLRGuest and HeteroLRHost
+HeteroLRGuest和HeteroLRHost是party guest和host对应的实现代码，其中party guest收到请求后，会执行推理过程，与此同时，需要与host也一起进行推理，下面给出具体说明：
+```text
+1. 系统针对请求的id，同时给guest和host发起推理请求
+2. Host接收到推理指令后，执行forward函数的前向计算流程，并将结果返回，由系统调度给guest
+3. Guest接收到推理指令后，执行forward函数的前向计算流程，并且获取host的计算结果，组合一起，并计算sigmod得到最终评分，完成完整的推理流程  
+```