Add automated security policy test scripts #1194
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| Copyright (c) 2018 The Go Authors. All rights reserved. | ||
|
|
||
| Redistribution and use in source and binary forms, with or without | ||
| modification, are permitted provided that the following conditions are | ||
| met: | ||
|
|
||
| * Redistributions of source code must retain the above copyright | ||
| notice, this list of conditions and the following disclaimer. | ||
| * Redistributions in binary form must reproduce the above | ||
| copyright notice, this list of conditions and the following disclaimer | ||
| in the documentation and/or other materials provided with the | ||
| distribution. | ||
| * Neither the name of Google Inc. nor the names of its | ||
| contributors may be used to endorse or promote products derived from | ||
| this software without specific prior written permission. | ||
|
|
||
| THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS | ||
| "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT | ||
| LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR | ||
| A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT | ||
| OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, | ||
| SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT | ||
| LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, | ||
| DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY | ||
| THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT | ||
| (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE | ||
| OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| [[source]] | ||
| name = "pypi" | ||
| url = "https://pypi.org/simple" | ||
| verify_ssl = true | ||
|
|
||
| [dev-packages] | ||
|
|
||
| [packages] | ||
| requests = "*" | ||
| pytz = "*" | ||
| bokeh = "*" | ||
| pandas = "==0.24.2" | ||
| numpy = "*" | ||
| pyyaml = "*" | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,71 @@ | ||
| #!/bin/bash | ||
|
|
||
| # Copyright Istio Authors | ||
|
|
||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
|
|
||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
|
|
||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| echo "Creating tests for authorizationPolicy with a variable number of path rules" | ||
| echo "Running each test in small load (conn=8, qps=100), medium load (conn=32, qps=500), and large load (conn=64, qps=1000)" | ||
|
|
||
| echo ' | ||
| { | ||
| "authZ":{ | ||
| "numPolicies":1, | ||
| "numPaths":10 | ||
| } | ||
| }' > authZPath10.json | ||
| go run ../generate_policies.go ../generate.go ../jwt.go -configFile="authZPath10.json" > authZPath10.yaml | ||
| echo "Generated a single authZ policy with 10 path rules" | ||
| kubectl apply -f authZPath10.yaml | ||
| echo "Running variable number of path rules" | ||
| echo "Running perf test with conn=8 and qps=100" | ||
| pipenv run python3 ../../../runner/runner.py --conn 8 --qps 100 --baseline --duration 240 --load_gen_type=nighthawk --telemetry_mode=none | ||
|
There was a problem hiding this comment. please make this |
||
| echo "Running perf test with conn=32 and qps=500" | ||
| pipenv run python3 ../../../runner/runner.py --conn 32 --qps 500 --baseline --duration 240 --load_gen_type=nighthawk --telemetry_mode=none | ||
| echo "Running perf test with conn=64 and qps=1000" | ||
| pipenv run python3 ../../../runner/runner.py --conn 64 --qps 1000 --baseline --duration 240 --load_gen_type=nighthawk --telemetry_mode=none | ||
| echo "Removing policies" | ||
| kubectl delete -f authZPath10.yaml | ||
| rm authZPath10.json | ||
| rm authZPath10.yaml | ||
|
|
||
| echo ' | ||
| { | ||
| "authZ":{ | ||
| "numPolicies":1, | ||
| "numPaths":1000 | ||
| } | ||
| }' > authZPath1000.json | ||
| go run ../generate_policies.go ../generate.go ../jwt.go -configFile="authZPath1000.json" > authZPath1000.yaml | ||
| echo "Generated a single authZ policy with 1000 paths" | ||
|
There was a problem hiding this comment. there are a lot of duplicate here, can we put the common part into a function? |
||
| kubectl apply -f authZSourceIP1000.yaml | ||
| echo "Running perf test with conn=8 and qps=100" | ||
| pipenv run python3 ../../../runner/runner.py --conn 8 --qps 100 --baseline --duration 240 --load_gen_type=nighthawk --telemetry_mode=none | ||
| echo "Running perf test with conn=32 and qps=500" | ||
| pipenv run python3 ../../../runner/runner.py --conn 32 --qps 500 --baseline --duration 240 --load_gen_type=nighthawk --telemetry_mode=none | ||
| echo "Running perf test with conn=64 and qps=1000" | ||
| pipenv run python3 ../../../runner/runner.py --conn 64 --qps 1000 --baseline --duration 240 --load_gen_type=nighthawk --telemetry_mode=none | ||
| echo "Removing policies" | ||
| kubectl delete -f authZPath1000.yaml | ||
| rm authZPath1000.json | ||
| rm authZPath1000.yaml | ||
|
|
||
| echo "Fetching data" | ||
| FORTIO_CLIENT_URL=http://$(kubectl get services -n twopods-istio fortioclient -o jsonpath="{.status.loadBalancer.ingress[0].ip}"):9076 | ||
| kubectl -n istio-prometheus port-forward svc/istio-prometheus 9090:9090 & | ||
| PROMETHEUS_URL=http://localhost:9090 | ||
| python3 ./../../../runner/fortio.py "$FORTIO_CLIENT_URL" --prometheus=$PROMETHEUS_URL --csv StartTime,ActualDuration,Labels,NumThreads,ActualQPS,p50,p90,p99,cpu_mili_avg_istio_proxy_fortioclient,cpu_mili_avg_istio_proxy_fortioserver,cpu_mili_avg_istio_proxy_istio-ingressgateway,mem_Mi_avg_istio_proxy_fortioclient,mem_Mi_avg_istio_proxy_fortioserver,mem_Mi_avg_istio_proxy_istio-ingressgateway | ||
|
|
||
| echo "Cleanup started" | ||
| kubectl delete --all pods --namespace=twopods-istio | ||
| echo "Cleanup finished, data collected" | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,72 @@ | ||
| #!/bin/bash | ||
|
|
||
| # Copyright Istio Authors | ||
|
|
||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
|
|
||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
|
|
||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| echo "Creating tests for authorizationPolicy with a variable number of paths" | ||
| echo "Running each test in small load (conn=8, qps=100), medium load (conn=32, qps=500), and large load (conn=64, qps=1000)" | ||
|
|
||
| echo ' | ||
| { | ||
| "authZ":{ | ||
| "numPolicies":10, | ||
| "numPaths":1 | ||
| } | ||
| }' > authZ10Path1.json | ||
| go run ../generate_policies.go ../generate.go ../jwt.go -configFile="authZ10Path1.json" > authZ10Path.yaml | ||
| echo "Generated 10 authZ policy with 1 path rule each" | ||
| kubectl apply -f authZ10Path.yaml | ||
| echo "Running variable number of policies" | ||
| echo "Running perf test with conn=8 and qps=100" | ||
| pipenv run python3 ../../../runner/runner.py --conn 8 --qps 100 --baseline --duration 240 --load_gen_type=nighthawk --telemetry_mode=none | ||
| echo "Running perf test with conn=32 and qps=500" | ||
| pipenv run python3 ../../../runner/runner.py --conn 32 --qps 500 --baseline --duration 240 --load_gen_type=nighthawk --telemetry_mode=none | ||
| echo "Running perf test with conn=64 and qps=1000" | ||
| pipenv run python3 ../../../runner/runner.py --conn 64 --qps 1000 --baseline --duration 240 --load_gen_type=nighthawk --telemetry_mode=none | ||
| echo "Removing policies" | ||
| kubectl delete -f authZ10Path.yaml | ||
| rm authZ10Path1.json | ||
| rm authZ10Path.yaml | ||
|
|
||
| echo ' | ||
| { | ||
| "authZ": | ||
| { | ||
| "numPolicies":1000, | ||
| "numPaths":1 | ||
| } | ||
| }' > authZ1000Path1.json | ||
| go run ../generate_policies.go ../generate.go ../jwt.go -configFile="authZ1000Path1.json" > authZ1000Path.yaml | ||
| echo "Generated 1000 authZ policy with 1 path each" | ||
| kubectl apply -f authZ1000Path.yaml | ||
| echo "Running perf test with conn=8 and qps=100" | ||
| pipenv run python3 ../../../runner/runner.py --conn 8 --qps 100 --baseline --duration 240 --load_gen_type=nighthawk --telemetry_mode=none | ||
| echo "Running perf test with conn=32 and qps=500" | ||
| pipenv run python3 ../../../runner/runner.py --conn 32 --qps 500 --baseline --duration 240 --load_gen_type=nighthawk --telemetry_mode=none | ||
| echo "Running perf test with conn=64 and qps=1000" | ||
| pipenv run python3 ../../../runner/runner.py --conn 64 --qps 1000 --baseline --duration 240 --load_gen_type=nighthawk --telemetry_mode=none | ||
| echo "Removing policies" | ||
| kubectl delete -f authZ1000Path.yaml | ||
| rm authZ1000Path.yaml | ||
| rm authZ1000Path1.json | ||
|
|
||
| echo "Fetching data" | ||
| FORTIO_CLIENT_URL=http://$(kubectl get services -n twopods-istio fortioclient -o jsonpath="{.status.loadBalancer.ingress[0].ip}"):9076 | ||
| kubectl -n istio-prometheus port-forward svc/istio-prometheus 9090:9090 & | ||
| PROMETHEUS_URL=http://localhost:9090 | ||
| python3 ./../../../runner/fortio.py "$FORTIO_CLIENT_URL" --prometheus=$PROMETHEUS_URL --csv StartTime,ActualDuration,Labels,NumThreads,ActualQPS,p50,p90,p99,cpu_mili_avg_istio_proxy_fortioclient,cpu_mili_avg_istio_proxy_fortioserver,cpu_mili_avg_istio_proxy_istio-ingressgateway,mem_Mi_avg_istio_proxy_fortioclient,mem_Mi_avg_istio_proxy_fortioserver,mem_Mi_avg_istio_proxy_istio-ingressgateway | ||
|
|
||
| echo "Cleanup started" | ||
| kubectl delete --all pods --namespace=twopods-istio | ||
| echo "Cleanup finished, data collected" |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why do we need put a Pipfile in this repo? since all test was trigger under runner/ folder. Just add the dependencies on the existing Pipfile there?