The implementation of WithMeta() is flawed. Not only is it not concurrency-safe, every nested call will be modifying the parent map.

The way to do this in a safe and performant manner is to structure the metadata as a tree, with a parent pointing to the previous metadata. You'd probably want to do some pooling and other optimizations to avoid allocating a map every time. Then all the maps can be immutable and therefore not require any locks. To construct the final map at error time, you simply traverse the map depth-first, building a merged map.

I'm not sure I agree with the approach, however. This system will incur a performance and memory penalty every time you descend into a new metadata context, even when no errors are occurring. Building up this contextual data (which presumably already exists on the call stack in the form of local variables) will be constantly going on and causing trouble in hot paths.

A better approach is to return a structured error describing the failed action that includes data known to the returner, which should have enough data to be meaningful. Then, every time you pass an error up the stack, you augment it with additional data so that everything can be gleaned from it. Rather than:

    val, err := GetStuff()
    if err != nil {
      return err
    }

You do:

    val, err := GetStuff()
    if err != nil {
      return fmt.Errorf("getting stuff: %w")
    }

Or maybe:

    val, err := GetStuff()
    if err != nil {
      return wrapWithMetadata(err, meta.KV("database", db.Name))
    }

Here, wrapWithMetadata() can construct an efficient error value that implements Unwrap().

This pays the performance cost only at error time, and the contextual information travels up the stack with a tree of error causes that can be gotten with `errors.Unwrap()`. The point is that Go errors already are a tree of causes.

Sometimes tracking contextual information in a context is useful, of course. But I think the benefit of my approach is that a function returning an error only needs to provide what it knows about the failing error. Any "ambient" contextual information can be added by the caller at no extra cost when following the happy path.