add multicurve bootstrap

[pcaspers]

add global bootstrap over multiple curves

unit test to be added, this should also illustrate the process

[coveralls]

coverage: 74.31% (+0.4%) from 73.914%
when pulling 1e8c557 on pcaspers:multicurve_bootstrap
into eea8696 on lballabio:master.

[pcaspers]

The canonical example would be Euribor3m vs 6m where the two forward curves depend on each other via tenor basis swaps. We don't have tenor basis swap helpers in QuantLib though, so try to come up with another example. If someone has suggestions? Otherwise I might migrate the tenor basis helper to QuantLib.

[lballabio]

We do have tenor basis swaps — see IborIborBasisSwapRateHelper in ql/experimental/termstructures/basisswapratehelpers.hpp.

[pcaspers]

Ah fantastic :-) thank you

[eltoder]

@pcaspers I only briefly skimmed through the changes, but I have 2 general comments:

1. Naively creating circular dependencies between curves will cause cycles in shared_ptrs and in QL notification graph. Both are bad. I haven't noticed any code to deal with this.
2. It seems easier to add costFunctionsSet, costFunctionsEval and makeGuess as methods to MultiCurveBootstrapContributor instead of having 4 parallel vectors. There's only one of the these per contributor if I'm not mistaken. This will also be more efficient for single curve bootstapping if we mark GlobalBootstrap class as final.

[pcaspers]

Thanks @eltoder. I'll have a look at 2. As for 1.: 1a) Do we think that a cycle in the QL notification graph which contains at least one LazyObject is handled ok, at least if QL_THROW_IN_CYCLES is not defined because of the way LazyObject::update() is implemented? 1b) any thoughts on how to resolve the issue with shared_ptr cycles - it looks like we need to touch Handle?

[eltoder]

@pcaspers

1a) I think that this used to work but causes infinite recursion since #1566. In fact, some tests and examples used to contain (unnecessary) cycles, and this started failing in QL 1.31 (#1688).

1b) I think we don't need to touch Handle, because shared_ptr has a lot of features. I looked into this previously. My idea was to have a MultiCurve (temporary name) object that owns all curves in the cycle. MultiCurve can return Handles/shared_ptrs of two kinds to the contained curves:
a) non-owning: these use null_deleter and should only be used by ratehelpers/etc of other curves in the cycle
b) owning: these use shared_ptrs constructed with the aliasing constructor: shared_ptr(shared_from_this(), curve) -- they all share the ownership of the whole MultiCurve object, but return pointers to the individual curves from get().

If we do that, I think we can solve the problem 1a) as well by creating non-owning Handles with registerAsObserver=false.

[pcaspers]

@eltoder how would you implement a non-owning shared_ptr? I guess this is exactly what we would need to avoid memory leaks from shared_ptr cycles, but at the moment I do not see how we would implement this?

[eltoder]

@pcaspers using null_deleter. It's already used in QL for similar purposes, for example, this is a common pattern.

[pcaspers]

Doesn't that just mean that the shared_ptr won't free the underlying pointer when the reference count goes to zero - but it does not solve the problem that shared_ptr in a cycle will never have a zero reference count?

[pcaspers]

I was thinking the standard solution would be weak_ptr and a corresponding WeakHandle, but this is of course not straightforward to integrate with QuantLib.

[eltoder]

weak_ptr is not different from using null_deleter in this context -- you still need to know which pointers should be weak and change your code accordingly. Let me write a code snippet. Hopefully that will be more clear.

[pcaspers]

Sounds good, thank you.

[eltoder]

@pcaspers something like this

// this class must be used with ext::shared_ptr
class MultiCurve : public ext::enable_shared_from_this<MultiCurve> {
  public:
    void addCurve(const std::string& name, ext::shared_ptr<YieldTermStructure> curve) {
        QL_REQUIRE(curve != nullptr, "got null curve for " << name);
        auto& entry = curves_[name];
        QL_REQUIRE(entry.ptr == nullptr, "curve " << name << " was already added");
        // ideally we set up bootstrapping here as well,
        // but this needs some changes to your code
        auto contrib = dynamic_cast<MultiCurveBootstrapContributor*>(curve.get());
        QL_REQUIRE(contrib != nullptr, "curve " << name << " is not compatible with MultiCurve");
        contrib->setParentBootstrapper(&bootstrap_);
        // TODO: setup notifications so that when any curve is updated
        // we update all other curves as well.

        // this handle should be used within the cycle
        bool observer = false;
        entry.internal.linkTo(
            ext::shared_ptr<YieldTermStructure>(curve.get(), null_deleter()), observer);
        // this handle should be used outside of the cycle
        entry.external.linkTo(
            ext::shared_ptr<YieldTermStructure>(shared_from_this(), curve.get()));
        entry.ptr = std::move(curve);
    }

    // this handle should be used within the cycle
    const Handle<YieldTermStructure>& getInternalHandle(const std::string& name) {
        return curves_[name].internal;
    }

    // this handle should be used outside of the cycle
    const Handle<YieldTermStructure>& getExternalHandle(const std::string& name) {
        return curves_[name].external;
    }
  private:
    struct Entry {
        RelinkableHandle<YieldTermStructure> internal;
        RelinkableHandle<YieldTermStructure> external;
        ext::shared_ptr<YieldTermStructure> ptr;
    };
    std::unordered_map<std::string, Entry> curves_;
    MultiCurveBootstrap bootstrap_;
};

void test() {
    auto mc = ext::make_shared<MultiCurve>();

    // build euribor3m
    auto intEuribor6m = mc->getInternalHandle("euribor6m");
    // use intEuribor6m in rate helpers that need euribor6m
    ...
    mc->addCurve("euribor3m", ext::make_shared<PiecewiseYieldCurve>(...));

    // build euribor6m
    auto intEuribor3m = mc->getInternalHandle("euribor3m");
    // use intEuribor3m in rate helpers that need euribor3m
    ...
    mc->addCurve("euribor6m", ext::make_shared<PiecewiseYieldCurve>(...));

    // done building the curves
    auto euribor3m = mc->getExternalHandle("euribor3m");
    auto euribor6m = mc->getExternalHandle("euribor6m");
    // any uses external to the cycle should use these external handles
}

[pcaspers]

Thanks a lot @eltoder - I will set up a euribor3m / 6m bootstrap using this approach as a unit test on the branch, this makes it easier to discuss.

[eltoder]

@pcaspers btw, to create a cycle I think you'll always have to create an empty RelinkableHandle, use it in rate helpers of the other curve, and then link it to the curve to complete the cycle. This should work with QL's basis swap helper, but IIUC this doesn't work with the one in ORE, because it checks which handle is empty to determine which curve is being bootstrapped.

[pcaspers]

@eltoder yes, thanks, we changed the rate helpers with regards to that, see e.g. here:

https://github.com/OpenSourceRisk/Engine/blob/master/QuantExt/qle/termstructures/tenorbasisswaphelper.cpp#L75

[pcaspers]

So speaking of ORE we don't care about notification cycles, since the curves are built once and then copied to interpolated curves anyhow. But of course we do care about possible memory leaks, so I am particularly interested in this. And when migrating the bootstrap to QL we have to have correct notifications, too, of course. I'll set up that unit test as soon as I have a chance.

[eltoder]

Sounds good. Thanks a lot for working on this.

[pcaspers]

Yes that seems to work well. And now I understand how you break the shared_ptr cycles :-) Let me run a couple of additional tests on our side and then update this branch

[eltoder]

@pcaspers nice! I realized that requiring to name curves and keeping them in a hash table is not very QL-like. This is easy to change and the result seems a bit simpler and more like other QL APIs:

// this class must be used with ext::shared_ptr
class MultiCurve : public ext::enable_shared_from_this<MultiCurve> {
  public:
    // addCurve() takes an internal handle and returns an external handle.
    // Internal handle, which must be an empty RelinkableHandle, should be
    // used within the cycle. External handle should be used outside of the
    // cycle.
    Handle<YieldTermStructure> addCurve(
            RelinkableHandle<YieldTermStructure>& internalHandle,
            ext::shared_ptr<YieldTermStructure> curve) {
        QL_REQUIRE(internalHandle.empty(),
                   "internal handle must be empty; was the curve added already?");
        QL_REQUIRE(curve != nullptr, "curve must not be null");
        // ideally we set up bootstrapping here as well,
        // but this needs some changes to your code
        auto contrib = dynamic_cast<MultiCurveBootstrapContributor*>(curve.get());
        QL_REQUIRE(contrib != nullptr, "curve is not compatible with MultiCurve");
        contrib->setParentBootstrapper(&bootstrap_);
        // TODO: setup notifications so that when any curve is updated
        // we update all other curves as well.

        bool observer = false;
        internalHandle.linkTo(
            ext::shared_ptr<YieldTermStructure>(curve.get(), null_deleter()), observer);
        Handle<YieldTermStructure> externalHandle(
            ext::shared_ptr<YieldTermStructure>(shared_from_this(), curve.get()));
        curves_.push_back(std::move(curve));
        return externalHandle;
    }
  private:
    std::vector<ext::shared_ptr<YieldTermStructure>> curves_;
    MultiCurveBootstrap bootstrap_;
};

void test() {
    auto mc = ext::make_shared<MultiCurve>();

    // internal handles that should be used by rate helpers
    // of the curves in the cycle to refer to each other
    RelinkableHandle<YieldTermStructure> intEuribor3m, intEuribor6m;

    // build euribor3m
    // use intEuribor6m in rate helpers
    ...
    auto euribor3m = mc->addCurve(
        intEuribor3m, ext::make_shared<PiecewiseYieldCurve>(...));

    // build euribor6m
    // use intEuribor3m in rate helpers
    ...
    auto euribor6m = mc->addCurve(
        intEuribor6m, ext::make_shared<PiecewiseYieldCurve>(...));

    // any uses external to the cycle should use the external
    // handles: euribor3m and euribor6m
}

[pcaspers]

@eltoder indeed, looks simpler, I'll change this

[pcaspers]

Hm. Somehow deriving MultiCurve from Observer, Observable introduces compile issues and bad::weak_ptr exceptions in the multicurve unit test in the "cmake-linux-with-options" build.

I suppose QL_USE_STD_SHARED_PTR is the relevant setting.

Still have to get to the bottom of this. I can fix the compile issue, although I don't fully understand why I have to do this, but not the weak_ptr issue in the unit test. I am also not able to reproduce this locally with QL_USE_STD_SHARED_PTR = ON.

[eltoder]

Does it need to be Observable?

[pcaspers]

Not sure. I thought it would be natural? Do you think this causes the issues?

[eltoder]

I think MultiCurve is an internal helper object, so no one would observe it. Observers will be on the curve handles.

I'm guessing the issues are caused by QL_ENABLE_THREAD_SAFE_OBSERVER_PATTERN. Specifically, MultiCurve now inherits from both Observer and ext::enable_shared_from_this, but Observer also inherits from ext::enable_shared_from_this, so probably only one of those two is populated by make_shared. To fix this we either have to separate the two, or maybe put our own ext::enable_shared_from_this under a #ifndef QL_ENABLE_THREAD_SAFE_OBSERVER_PATTERN and add a ext::static_pointer_cast around shared_from_this().

[eltoder]

A common trick to work this around is this: https://stackoverflow.com/a/16083526/5190601

[pcaspers]

agreed, MultiCurve does not need to be observable, I removed that

[pcaspers]

many thanks for the hint with the thread-safe-observer build, I think this fixes it

[pcaspers]

This is an improvement in any case, so I cherry-picked that. I have a slight preference for option 1 because it lets us easily integrate this with ore. But 3 will also work, I guess. Maybe Luigi has additional thoughts.

As for ZeroSpreadedTermStructure: I haven't tested this, but I think this should not be added to MultiCurve and use an external handle as an input (if the input curve is a MultiCurve component)? Not sure about this.

[eltoder]

Imagine a hypothetical case like: euribor3m -> add spread ->euribor6m -> euribor3m. The spread is added using ZeroSpreadedTermStructure. I think ZeroSpreadedTermStructure has to use an internal handle, and euribor3m use an internal handle for it, because otherwise we won't be able to break the cycle (euribor3m no longer accesses euribor6m directly). So ZeroSpreadedTermStructure needs to be added to MultiCurve.

[eltoder]

I guess you can revert adding virtual destructors to other bootstrap classes now?

[pcaspers]

yes!

[pcaspers]

If euribor6m uses and internal handle of euribor3m as the input, how is euribor6m updated during the bootstrap?

[pcaspers]

oh maybe it does not have to receive an update(), because it computes the zero rate on the fly anyway

[pcaspers]

and you are right about the circle, but that's kind of a limitation to keep in mind, i.e. that curves participating in a multicurve bootstrap can not rely on notifications from other curves that they depend on?

[eltoder]

Both euribor3m and euribor6m need to be registered with MultiCurveBootstrap, the same way as if ZeroSpreadedTermStructure was not present. Yes, rate helpers force recalculation on the underlying swaps.

[eltoder]

I agree that it's definitely something to keep in mind, but I think it will work out. Rate helpers already disable notifications from the curve being bootstrapped, and so they have to force recalculation on the instruments. Unless there's some extra caching that only applies to the second curve and not cleared by calling deepUpdate(), this should be enough. If there is such caching (I don't know any examples on top of my head), it has to be disabled for multi-curve bootstrapping.

My point was just that in this example we have a curve that participates in the cycle, but does not need to be bootstrapped, because it's just a spread on another curve. But actually, something needs to happen, because ZeroSpreadedTermStructure currently relies on update() to update interpolation. So we either need it to keep receiving updates or become a MultiCurveBootstrapContributor.

EDIT: more precisely, ZeroSpreadedTermStructure will work out, because it's a constant spread, but if we had InterpolatedPiecewiseZeroSpreadedTermStructure, it needs to update interpolation when the original curve changes.

We can leave this out for now. We can add this later on with either of the options.

[pcaspers]

Could we add a method

MultiCurveBootstrap::add(Observable* o)

and these observables are updated in MultiCurveBootstrap::runMultiCurveBootstrap() after the block

// call the contributors' cost functions' set part

and before the block

// collect the contributoes result

We could then add curves like the spreaded ones as observables that are updated at the correct point during the multicurve bootstrap.

For convenience, we can add MultiCurve::add(Observable* o) which passes o on to MultiCurveBootstrap.

[eltoder]

Yes, this probably works. I thought we could also create MultiCurveBootstrapContributors for them that do nothing in setupCostFunction and in evaluateCostFunction and call update() in setCostFunctionArgument. This will have a similar effect, but a separate list of Observables sounds cleaner.

EDIT: I think the main thing to be careful about is separating curves that only need to be added as observables from curves that use incompatible bootstrapping. The former can be done like you suggested, but the latter should be an error.

[pcaspers]

We check for an incompatible bootstrapper, i.e., whether it implements MultiCurveBootstrapContributor. Or do you mean something else?

[eltoder]

I just meant that there should be separate APIs to add a bootstrapped curve and an observer curve.

[lballabio]

Apologies, I haven't been able to look at this discussion yet. I will.