Add ability to exclude containers similar to filter includes

[rocketraman]

We can use filters to include specific containers by id, name, etc. It would be useful to exclude containers in the same way.

Sometimes the environment / label approach does not work since the container is managed by a third party e.g. on a cloud environment, the hosting provider may install and manage containers that provide services specific to that environment. We don't have control over these containers, so it may not be convenient or possible to set their environment or labels.

[dcassiero]

This would be useful. We use Rancher and recently upgraded to 1.2.x which introduced new infrastructure containers. One of which is extremely chatty. There is no way to turn off logging for these containers or add LOGSPOUT=ignore to their run command. The routesapi is not useful here. We do Rancher+logspout->Loggly.

[josegonzalez]

Would this pr be useful for that? https://github.com/gliderlabs/logspout/pull/222/files

[rocketraman]

@josegonzalez Well, it does get us part of the way there -- it lets us explicitly label all the containers that we wan to include. But unfortunately, now we have the opposite problem. If there are infrastructure containers that we do want to include, we have no way of including them.

[rocketraman]

I think this is the situation:

• Label-based exclude (in master now)
• Label-based include (pending pull added the ability to include only container having label using INCLUDE_LABEL param #222 )
• Environment-based exclude (in master now)
• Environment-based include (missing?)
• Filter-based exclude (missing?)
• Filter-based include (in master now)

[josegonzalez]

Mind making a pull request?

[rocketraman]

Mind making a pull request?

I wouldn't, though I've never written any Go code before and for some reason I find the syntax somewhat opaque and unreadable (:=? what was wrong with just =?). Everyone else seems to love it, so it must be just me :-)

But this is an easy enough task to do. I might have time to work on this in about a month or so. If no one has tackled it by then, I'll likely do so. If I implement it, I'd set it up so all the following rules all must be true for a container to be logged:

1. Container must match all include filters. If no include filter(s) are provided, all containers are included (i.e. equivalent of filter.name="*"). If filters are not currently multi-valued parameters, make them so.

2. Container must not be a part of any exclude filters. If no exclude filter(s) are provided, no containers are excluded. Exclude filters will have the same syntax as include filters, except they are prefixed with !.

3. If a logspout INCLUDE_LABEL label was provided, then container must be labeled with the configured include label set to true or 1.

4. If a logspout EXCLUDE_LABEL label was provided, then container must not be labeled with the configured exclude label set to true or 1.

5. If an environment variable LOGSPOUT is present, it must not be set to ignore or false or 0.

6. If an environment variable LOGSPOUT is present, it must be set to true or 1.

Now, you would be able to default to exclude via a filter like filter.name=!* and then explicitly include via environment variable or label. Or you can default to include with no filter, and then explicitly exclude via environment variable or label. Or you can just specify everything in a set of filters. Not sure if the way the current code uses filters would easily support this logic.

[dcassiero]

An exclude filter would be desirable since I know for a fact how many, and what names the exclude containers will be. Is that something that is easier to implement? The dozens of other arbitrary containers that we know need to be logged and I will never know how many of them there are or what their name might be (--name).

[garceri]

@dcassiero : You can add the LOGSPOUT=ignore env variable to the infra stacks by "Upgrading" them, Select the Service within the stack you want to apply the change, click on the 3 dots, then select Upgrade, you will be presented with a screen where you can add the environmental variable, this is how i did it.
Unfortunately this solved only part of the issue as we are still experiencing very high cpu usage on dockerd after starting logspout with the logstash adapter as described here

[michaelshobbs]

@dcassiero LOGSPOUT=ignore seems like it would work for this case. Closing for now. Please comment if this is not sufficient for your use case.

[rocketraman]

@michaelshobbs @dcassiero was not the original reporter. Not sure why your comment addresses his use case. Unless there is a solution for my use case, not sure why this was closed.

[michaelshobbs]

Apologies. Re-opening