Purpose of the log service

[moltar]

Could you please explain what is the purpose of the log service?

Thank you.

[alexbouchardd]

The log service is responsible for pulling results of event deliveries (logs) from the logs queue, formatting the data for the log adapter (ie, Postgres) and batch inserting into the log storage. We will also be adding more storage adapters, such as Clickhouse, in the near future.

See architecture diagram: https://outpost.hookdeck.com/docs/concepts

[moltar]

Ok, perfect, thanks. So, technically it's not required, if we don't need to put the delivery logs into the the storage, or if we want to use alternative storages, e.g. Kinesis -> S3?

[alexbouchardd]

It's "required" since longer-term retries rely on the data stored in the log storage to hydrate the data for delivery. It would be possible to use blob storage to store the event data and hydrate from there instead, which would make both the log service and log storage optional, BUT that would lead to losing the ability to retrieve the events via the /events API. What might make more sense is an S3 adapter, which would still allow for some functionalities (but not all)?

Can you share your use case and why you wouldn't want/need it?

[moltar]

No, it's fine! I just didn't understand the importance. I just thought it was literally logs. But if these are important events then I'll keep the service.

Maybe the name is just throwing me off.

[alexbouchardd]

Fair point, we did struggle with the naming here in the sense that it's not service logs (stdout) it's the event delivery results & data. Would you happen to have an idea for better terminology?

[leggetter]

Event storage