[query] Various Benchmark Suite Improvements

[ehigham]

Changes include:

• Remove custom benchmark() decorator - a test being in 'benchmark/' is sufficient to indicate that it's a benchmark.
• Add xtimeout marker to gracefully handle benchmarks that exceed time limits
• Set default data directory from environment variable HAIL_BENCHMARK_DIR
• Add configurable maximum duration for benchmark trials
• Implement a failure counter to stop benchmarking after multiple consecutive failures
• Set a temporary directory handling for each benchmark so that consecutive runs don’t run out of disk space
• Enhance error reporting and output formatting
• Hack to clean up Spark temporary files between benchmark runs. I didn’t find a nicer way of doing this from a quick glance at the spark docs.

This change has no impact on the GCP hail batch deployment as it is limited to refactoring of query benchmark infrastructure. Changes only affect how benchmarks are run and reported, not the actual functionality being tested.

[ehigham]

• [query] Benchmark configurations #14762
• [benchmark] notebook for interactively computing benchmark config #14757
• [query] Various Benchmark Suite Improvements #14761 👈 (View in Graphite)
• [query] make local and remote tmp settable on backend #14748
• [query] Lowering + Optimisation with implict timing context #14731 : 2 other dependent PRs (#14732 , #14952 )
• [query] Lift backend state into (Batch|Py4J)QueryDriver #14698
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[chrisvittal]

So, you're deleting this comment. Have we every diagnosed why we can't use python to download these files?

[ehigham]

No clue. urlretrieve seems to be deprecated now so I'm not going to look into it.
This is tested and seems to work - not sure if looking into a python only implementation is worth it.

[chrisvittal]

Use a set comprehension.

Suggested change

	diff = set(include) - set([nodeid for nodeid, *_ in items])
	diff = set(include) - {nodeid for nodeid, *_ in items}

[chrisvittal]

There's an analyzer warning here. Is there a reason that we aren't re-raising?

[ehigham]

Yes - __Timeout is an internal implementation detail. I'll supress the warning.