Skip to content

Change ECS Agent log rollover defaults to size-based rotation #4776

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: dev
Choose a base branch
from
+98 −34
Open

Change ECS Agent log rollover defaults to size-based rotation #4776

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -235,9 +235,9 @@ additional details about how to configure the agent.
| `ECS_NVIDIA_RUNTIME` | nvidia | The Nvidia Runtime to be used to pass Nvidia GPU devices to containers. | nvidia | Not Applicable |
| `ECS_ALTERNATE_CREDENTIAL_PROFILE` | default | An alternate credential role/profile name. | default | default |
| `ECS_ENABLE_SPOT_INSTANCE_DRAINING` | `true` | Whether to enable Spot Instance draining for the container instance. If true, if the container instance receives a [spot interruption notice](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-interruptions.html), agent will set the instance's status to [DRAINING](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/container-instance-draining.html), which gracefully shuts down and replaces all tasks running on the instance that are part of a service. It is recommended that this be set to `true` when using spot instances. | `false` | `false` |
| `ECS_LOG_ROLLOVER_TYPE` | `size` | `hourly` | Determines whether the container agent logfile will be rotated based on size or hourly. By default, the agent logfile is rotated each hour. | `hourly` | `hourly` |
| `ECS_LOG_ROLLOVER_TYPE` | `size` | `hourly` | Determines whether the container agent logfile will be rotated based on size or hourly. By default, the agent logfile is rotated based on size. | `size` | `size` |
| `ECS_LOG_OUTPUT_FORMAT` | `logfmt` | `json` | Determines the log output format. When the json format is used, each line in the log would be a structured JSON map. | `logfmt` | `logfmt` |
| `ECS_LOG_MAX_FILE_SIZE_MB` | `10` | When the ECS_LOG_ROLLOVER_TYPE variable is set to size, this variable determines the maximum size (in MB) the log file before it is rotated. If the rollover type is set to hourly then this variable is ignored. | `10` | `10` |
| `ECS_LOG_MAX_FILE_SIZE_MB` | `1` | When the ECS_LOG_ROLLOVER_TYPE variable is set to size, this variable determines the maximum size (in MB) the log file before it is rotated. If the rollover type is set to hourly then this variable is ignored. | `1` | `1` |
| `ECS_LOG_MAX_ROLL_COUNT` | `24` | Determines the number of rotated log files to keep. Older log files are deleted once this limit is reached. | `24` | `24` |
| `ECS_LOG_DRIVER` | `awslogs` | `fluentd` | `gelf` | `json-file` | `journald` | `logentries` | `syslog` | `splunk` | The logging driver to be used by the Agent container. | `json-file` | Not applicable |
| `ECS_LOG_OPTS` | `{"option":"value"}` | The options for configuring the logging driver set in `ECS_LOG_DRIVER`. | `{}` | Not applicable |
Expand Down
4 changes: 2 additions & 2 deletions agent/vendor/github.com/aws/amazon-ecs-agent/ecs-agent/logger/log.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions ecs-agent/logger/log.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,10 +39,10 @@ const (

DEFAULT_LOGLEVEL = "info"
DEFAULT_LOGLEVEL_WHEN_DRIVER_SET = "off"
DEFAULT_ROLLOVER_TYPE = "date"
DEFAULT_ROLLOVER_TYPE = "size"
DEFAULT_OUTPUT_FORMAT = logFmt
DEFAULT_TIMESTAMP_FORMAT = time.RFC3339
DEFAULT_MAX_FILE_SIZE float64 = 10
DEFAULT_MAX_FILE_SIZE float64 = 1
DEFAULT_MAX_ROLL_COUNT int = 24
DEFAULT_LOGTO_STDOUT = true
)
Expand Down
65 changes: 48 additions & 17 deletions ecs-agent/logger/log_unix_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,8 +48,8 @@ func TestSeelogConfig_Default(t *testing.T) {
<console />
</filter>
<filter levels="info,warn,error,critical">
<rollingfile filename="foo.log" type="date"
datepattern="2006-01-02-15" archivetype="none" maxrolls="24" />
<rollingfile filename="foo.log" type="size"
maxsize="1000000" archivetype="none" maxrolls="24" />
</filter>
</outputs>
<formats>
Expand Down Expand Up @@ -125,8 +125,8 @@ func TestSeelogConfig_DebugLevel(t *testing.T) {
<console />
</filter>
<filter levels="info,warn,error,critical">
<rollingfile filename="foo.log" type="date"
datepattern="2006-01-02-15" archivetype="none" maxrolls="24" />
<rollingfile filename="foo.log" type="size"
maxsize="1000000" archivetype="none" maxrolls="24" />
</filter>
</outputs>
<formats>
Expand Down Expand Up @@ -157,7 +157,7 @@ func TestSeelogConfig_SizeRollover(t *testing.T) {
</filter>
<filter levels="info,warn,error,critical">
<rollingfile filename="foo.log" type="size"
maxsize="10000000" archivetype="none" maxrolls="24" />
maxsize="1000000" archivetype="none" maxrolls="24" />
</filter>
</outputs>
<formats>
Expand Down Expand Up @@ -249,8 +249,8 @@ func TestSeelogConfig_JSONOutput(t *testing.T) {
<console />
</filter>
<filter levels="info,warn,error,critical">
<rollingfile filename="foo.log" type="date"
datepattern="2006-01-02-15" archivetype="none" maxrolls="10" />
<rollingfile filename="foo.log" type="size"
maxsize="1000000" archivetype="none" maxrolls="10" />
</filter>
</outputs>
<formats>
Expand All @@ -277,8 +277,8 @@ func TestSeelogConfig_JSONNoStdout(t *testing.T) {
<seelog type="asyncloop">
<outputs formatid="json">
<filter levels="info,warn,error,critical">
<rollingfile filename="foo.log" type="date"
datepattern="2006-01-02-15" archivetype="none" maxrolls="10" />
<rollingfile filename="foo.log" type="size"
maxsize="1000000" archivetype="none" maxrolls="10" />
</filter>
</outputs>
<formats>
Expand Down Expand Up @@ -338,8 +338,8 @@ func TestSeelogConfig_NoOnInstanceLog(t *testing.T) {
<console />
</filter>
<filter levels="off">
<rollingfile filename="foo.log" type="date"
datepattern="2006-01-02-15" archivetype="none" maxrolls="24" />
<rollingfile filename="foo.log" type="size"
maxsize="1000000" archivetype="none" maxrolls="24" />
</filter>
</outputs>
<formats>
Expand All @@ -366,8 +366,8 @@ func TestSeelogConfig_DifferentLevels(t *testing.T) {
<console />
</filter>
<filter levels="info,warn,error,critical">
<rollingfile filename="foo.log" type="date"
datepattern="2006-01-02-15" archivetype="none" maxrolls="24" />
<rollingfile filename="foo.log" type="size"
maxsize="1000000" archivetype="none" maxrolls="24" />
</filter>
</outputs>
<formats>
Expand Down Expand Up @@ -400,8 +400,8 @@ func TestSeelogConfig_FileLevelDefault(t *testing.T) {
<console />
</filter>
<filter levels="off">
<rollingfile filename="foo.log" type="date"
datepattern="2006-01-02-15" archivetype="none" maxrolls="24" />
<rollingfile filename="foo.log" type="size"
maxsize="1000000" archivetype="none" maxrolls="24" />
</filter>
</outputs>
<formats>
Expand Down Expand Up @@ -450,8 +450,8 @@ func TestSetOutputFormat(t *testing.T) {
<console />
</filter>
<filter levels="info,warn,error,critical">
<rollingfile filename="foo.log" type="date"
datepattern="2006-01-02-15" archivetype="none" maxrolls="24" />
<rollingfile filename="foo.log" type="size"
maxsize="1000000" archivetype="none" maxrolls="24" />
</filter>
</outputs>
<formats>
Expand Down Expand Up @@ -522,6 +522,37 @@ func TestSetLogToStdout(t *testing.T) {
require.Equal(t, `
<seelog type="asyncloop">
<outputs formatid="logfmt">
<filter levels="info,warn,error,critical">
<rollingfile filename="foo.log" type="size"
maxsize="1000000" archivetype="none" maxrolls="24" />
</filter>
</outputs>
<formats>
<format id="logfmt" format="%EcsAgentLogfmt" />
<format id="json" format="%EcsAgentJson" />
<format id="windows" format="%EcsMsg" />
</formats>
</seelog>`, c)
}

func TestSeelogConfig_HourlyRollover(t *testing.T) {
Config = &logConfig{
logfile: "foo.log",
driverLevel: DEFAULT_LOGLEVEL,
instanceLevel: DEFAULT_LOGLEVEL,
RolloverType: "hourly",
outputFormat: DEFAULT_OUTPUT_FORMAT,
MaxFileSizeMB: DEFAULT_MAX_FILE_SIZE,
MaxRollCount: DEFAULT_MAX_ROLL_COUNT,
logToStdout: DEFAULT_LOGTO_STDOUT,
}
c := seelogConfig()
require.Equal(t, `
<seelog type="asyncloop">
<outputs formatid="logfmt">
<filter levels="info,warn,error,critical">
<console />
</filter>
<filter levels="info,warn,error,critical">
<rollingfile filename="foo.log" type="date"
datepattern="2006-01-02-15" archivetype="none" maxrolls="24" />
Expand Down
55 changes: 44 additions & 11 deletions ecs-agent/logger/log_windows_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,8 +44,8 @@ func TestSeelogConfigWindows_Default(t *testing.T) {
<custom name="wineventlog" formatid="windows" />
</filter>
<filter levels="info,warn,error,critical">
<rollingfile filename="foo.log" type="date"
datepattern="2006-01-02-15" archivetype="none" maxrolls="24" />
<rollingfile filename="foo.log" type="size"
maxsize="1000000" archivetype="none" maxrolls="24" />
</filter>
</outputs>
<formats>
Expand Down Expand Up @@ -105,8 +105,8 @@ func TestSeelogConfigWindows_DebugLevel(t *testing.T) {
<custom name="wineventlog" formatid="windows" />
</filter>
<filter levels="info,warn,error,critical">
<rollingfile filename="foo.log" type="date"
datepattern="2006-01-02-15" archivetype="none" maxrolls="24" />
<rollingfile filename="foo.log" type="size"
maxsize="1000000" archivetype="none" maxrolls="24" />
</filter>
</outputs>
<formats>
Expand Down Expand Up @@ -139,7 +139,7 @@ func TestSeelogConfigWindows_SizeRollover(t *testing.T) {
</filter>
<filter levels="info,warn,error,critical">
<rollingfile filename="foo.log" type="size"
maxsize="10000000" archivetype="none" maxrolls="24" />
maxsize="1000000" archivetype="none" maxrolls="24" />
</filter>
</outputs>
<formats>
Expand Down Expand Up @@ -237,8 +237,8 @@ func TestSeelogConfigWindows_JSONOutput(t *testing.T) {
<custom name="wineventlog" formatid="windows" />
</filter>
<filter levels="info,warn,error,critical">
<rollingfile filename="foo.log" type="date"
datepattern="2006-01-02-15" archivetype="none" maxrolls="10" />
<rollingfile filename="foo.log" type="size"
maxsize="1000000" archivetype="none" maxrolls="10" />
</filter>
</outputs>
<formats>
Expand Down Expand Up @@ -270,8 +270,8 @@ func TestSeelogConfigWindows_NoOnInstanceLog(t *testing.T) {
<custom name="wineventlog" formatid="windows" />
</filter>
<filter levels="off">
<rollingfile filename="foo.log" type="date"
datepattern="2006-01-02-15" archivetype="none" maxrolls="24" />
<rollingfile filename="foo.log" type="size"
maxsize="1000000" archivetype="none" maxrolls="24" />
</filter>
</outputs>
<formats>
Expand Down Expand Up @@ -303,6 +303,39 @@ func TestSeelogConfigWindows_DifferentLevels(t *testing.T) {
<custom name="wineventlog" formatid="windows" />
</filter>
<filter levels="critical">
<rollingfile filename="foo.log" type="size"
maxsize="1000000" archivetype="none" maxrolls="24" />
</filter>
</outputs>
<formats>
<format id="logfmt" format="%EcsAgentLogfmt" />
<format id="json" format="%EcsAgentJson" />
<format id="windows" format="%EcsMsg" />
</formats>
</seelog>`, c)
}

func TestSeelogConfigWindows_HourlyRollover(t *testing.T) {
Config = &logConfig{
logfile: "foo.log",
driverLevel: DEFAULT_LOGLEVEL,
instanceLevel: DEFAULT_LOGLEVEL,
RolloverType: "hourly",
outputFormat: DEFAULT_OUTPUT_FORMAT,
MaxFileSizeMB: DEFAULT_MAX_FILE_SIZE,
MaxRollCount: DEFAULT_MAX_ROLL_COUNT,
logToStdout: DEFAULT_LOGTO_STDOUT,
}
c := seelogConfig()
require.Equal(t, `
<seelog type="asyncloop">
<outputs formatid="logfmt">
<filter levels="info,warn,error,critical">
<console />

<custom name="wineventlog" formatid="windows" />
</filter>
<filter levels="info,warn,error,critical">
<rollingfile filename="foo.log" type="date"
datepattern="2006-01-02-15" archivetype="none" maxrolls="24" />
</filter>
Expand Down Expand Up @@ -339,8 +372,8 @@ func TestSeelogConfigWindows_FileLevelDefault(t *testing.T) {
<custom name="wineventlog" formatid="windows" />
</filter>
<filter levels="off">
<rollingfile filename="foo.log" type="date"
datepattern="2006-01-02-15" archivetype="none" maxrolls="24" />
<rollingfile filename="foo.log" type="size"
maxsize="1000000" archivetype="none" maxrolls="24" />
</filter>
</outputs>
<formats>
Expand Down
Loading