support filtering jobs by queue in job-list and flux-jobs(1)

[chu11]

splitting issue #4441 into two issues, this is the second half. #4441 is now for just getting a job's queue.

Possibly we could add a flux jobs --queue=NAME option to filter the job list by queue?
I'm unsure how this affects the job-list module or the python api. Separate bugs could be opened for those.

[chu11]

should we allow the user to specify a filter for jobs that are not in a queue? hypothetically via a special "none" keyword? I can't imagine a real use case, just something we'd want to do for testing.

[grondo]

Jobs can only 'not be in a queue' (or in the anonymous queue) if no queues are configured (and in that case all jobs are not in a queue), so this doesn't seem useful.

For now, could we just filter queues in the flux-jobs.py command itself to avoid adding another parameter to the job list RPC? I'm now thinking it was probably a misstep to separately add name and since attributes to the job list RPC and instead we should have taken a step back an evaluate how filtering works.

I was actually wondering if json-logic like format that we used in RFC 31 would be a more powerful and extensible way to filter jobs. And when we eventually get active jobs into a DB like job-archive that same format could be translated to a sql query.

[chu11]

For now, could we just filter queues in the flux-jobs.py command itself to avoid adding another parameter to the job list RPC? I'm now thinking it was probably a misstep to separately add name and since attributes to the job list RPC and instead we should have taken a step back an evaluate how filtering works.

Oops, already did it :P The work was minor at the end of the day, I may just append to #4579.

I don't know if we'd want to do the filtering in flux-jobs.py though, since that means a ton of data could be sent from job-list and (ultimately) very few are presented to the user? Although the common case would likely be "display all jobs", so maybe it doesn't matter.

I was actually wondering if json-logic like format that we used in RFC 31 would be a more powerful and extensible way to filter jobs. And when we eventually get active jobs into a DB like job-archive that same format could be translated to a sql query.

I was just thinking about that with #4336. As we move to a system in which the majority of inactive jobs are stored in a database (mysql in this case) we can begin to "punt" filtering upon that database.

[grondo]

I don't know if we'd want to do the filtering in flux-jobs.py though, since that means a ton of data could be sent from job-list and (ultimately) very few are presented to the user? Although the common case would likely be "display all jobs", so maybe it doesn't matter.

Yeah, if it was trivial it doesn't matter for now I suppose.

As we move to a system in which the majority of inactive jobs are stored in a database (mysql in this case) we can begin to "punt" filtering upon that database.

We should try to get active jobs in that database too if possible. Having a different interface for active and inactive job listing will likely not be easily maintained.

[chu11]

We should try to get active jobs in that database too if possible. Having a different interface for active and inactive job listing will likely not be easily maintained.

perhaps we should start an issue for this discussion, b/c I'm not sure sqlite would be right if we went with that approach. I can't remember all the details, but when users are regularly reading and writing to sqlite, performance is worse. And we'd be punting a lot of sorting onto the database too (i.e. if we're adding jobs to the db as they get submitted, then the natural sorting of "running" vs "inactive" is thrown out, and we'll be sorting all results all of the time). Probably some other issues too that I can't think of at the moment.

[grondo]

Yeah, you're right, we've already discussed these things at length in #4273 and associated issues and that's inappropriate for this issue. (This is more of an immediate need for queue support) Apologies.