Overhaul Qiskit Experiments (RFC0014)

[nkanazawa1989]

Epic issue to manage implementation of RFC0014

Migration plan:

• Rework of composite analysis (assigned to @Musa-Sina-Ertugrul )
• Implementation of circuit executor (not active)
• Implementation of analysis executor (not active)
• Implementation of executor and cleanup of experiment data (not active)
• JAX-based curve fitter (assigned to @to24toro)
• Lazy figure generation (Add option to control figure generation in composite experiments #1240)

Optional:

• Better syntax to restart experiment workflow
• Circuit metadata validation with analysis class

[Musa-Sina-Ertugrul]

can I use inliner module to prevent a huge thread stack or should I use processes

[Musa-Sina-Ertugrul]

Also, Can you assign the first job to me

[nkanazawa1989]

I'm not sure if inliner module helps performance improvement in our case. Indeed it optimizes bytecode and reduces the function call overhead, but we are targeting 1000 parallel post-processing tasks for efficient experiment execution on 1000 qubit device. I think sub-processing is necessary technology to avoid Python GIL, but still open to another suggestion (e.g. implementing everything with Rust).

Also I'm bit hesitate to add inliner to dependency because likely the software doesn't have any public release.

[Musa-Sina-Ertugrul]

Can we use pointers in Python to prevent object copying for different processes if this works we don't need ray or shared memory

[Musa-Sina-Ertugrul]

I am trying to use tox but I got bunch of errors

File "C:\Users\msert\AppData\Local\Temp\pip-build-env-25wrzqy8\overlay\lib\python3.10\site-packages\setuptools\_distutils\msvc9compiler.py", line 295, in <module>
          raise DistutilsPlatformError("VC %0.1f is not supported by this module" % VERSION)
      distutils.errors.DistutilsPlatformError: VC 6.0 is not supported by this module

[nkanazawa1989]

Pointers could be an option. However I don't think tight coupling between analysis and experiment data is the way to go. Namely, current BaseAnalysis.run takes the data from experiment data, perform analysis, create analysis results, and mutates (populates) the original experiment data. I think the last operation should be a part of executor. If we delegate mutation part to executor, analysis doesn't need to mutate experiment data, and it just needs to return analysis result objects to the caller. This means we don't need shared memory in the first place. Analysis just need to return result objects to the main process, and the executor running on the main process must update experiment data. But reducing object copying overhead could lead to performance gain. In this sense, use of pointer is worth considering as long as it reliably works on all platforms that Qiskit must support.

Regarding the tox issue, I think this is a problem of your IDE (VS code?). I'm only using MacOS and not eligible to support Windows issue. Probably @mtreinish can help?