Skip to content

fix(aws): Fix AWS lambda JSON logging ingestion pipeline #15376

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
+22 −19
Open

fix(aws): Fix AWS lambda JSON logging ingestion pipeline #15376

Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
4077122
Added missing parsed data format (required by nodejs lambda https://d…
adam-brzozowski-covantis Sep 17, 2025
9ef320c
Removed duplicated rename processors for timestamp fields
adam-brzozowski-covantis Sep 17, 2025
a236e6f
Added proper usage of parsed.message field
adam-brzozowski-covantis Sep 17, 2025
31deb5e
Merge branch 'elastic:main' into main
adam-prohack Sep 17, 2025
8c71ed8
Added ignore_failure parameters to some processors in aws-lambda-json
adam-brzozowski-covantis Sep 17, 2025
0a9ecb7
Merge branch 'main' into main
adam-prohack Sep 18, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 22 additions & 19 deletions packages/aws/data_stream/lambda_logs/elasticsearch/ingest_pipeline/aws-lambda-json.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ processors:
if: "ctx.parsed?.timestamp != null"
field: parsed.timestamp
target_field: "@timestamp"
formats: ["yyyy-MM-dd HH:mm:ss,SSSZ"]
formats: ["yyyy-MM-dd HH:mm:ss,SSSZ", "ISO8601"]
ignore_failure: true

# 2. Platform report logs (time field)
Expand Down Expand Up @@ -43,6 +43,18 @@ processors:
copy_from: parsed._aws
ignore_failure: true

- set:
field: aws.lambda.message.log
if: "ctx.parsed?.message != null && ctx.parsed?.message instanceof String"
copy_from: parsed.message
ignore_failure: true

- set:
field: aws.lambda.message
if: "ctx.parsed?.message != null && ctx.parsed?.message instanceof Map"
copy_from: parsed.message
ignore_failure: true

- rename:
field: parsed.service
target_field: service.name
Expand Down Expand Up @@ -231,6 +243,13 @@ processors:
ignore_failure: true
if: "ctx['@timestamp'] == null"

- rename:
field: parsed.timestamp
target_field: "@timestamp"
ignore_missing: true
ignore_failure: true
if: "ctx['@timestamp'] == null"

- rename:
field: parsed.errorMessage
target_field: aws.lambda.error.message
Expand Down Expand Up @@ -273,18 +292,6 @@ processors:
ignore_missing: true
ignore_failure: true

- rename:
field: parsed.time
target_field: "@timestamp"
ignore_missing: true
ignore_failure: true

- rename:
field: parsed.timestamp
target_field: "@timestamp"
ignore_missing: true
ignore_failure: true

- rename:
field: parsed.type
target_field: aws.lambda.event_type
Expand Down Expand Up @@ -346,12 +353,6 @@ processors:
ignore_missing: true
ignore_failure: true

- rename:
field: parsed.timestamp
target_field: "@timestamp"
ignore_missing: true
ignore_failure: true

- rename:
field: parsed.tracing.xray_trace_id
target_field: aws.lambda.xray_trace_id
Expand Down Expand Up @@ -417,6 +418,7 @@ processors:
field: aws.lambda.message
copy_from: event.original
if: "ctx.parsed == null"
ignore_failure: true

- remove:
field: parsed
Expand All @@ -431,6 +433,7 @@ processors:
- script:
description: Drops null/empty values recursively
lang: painless
ignore_failure: true
source: |
boolean drop(Object o) {
if (o == null || o == "") {
Expand Down