It would be really interesting to see if this could be applied to other analogous scenarios that are sometimes called Pythagorean theorems, in particular I’m thinking of the Pythagorean Theorem of Information Geometry-
p* = argmin p of P (a set of possible distributions) of D_KL (p||q) (where q is eg your model's distribution)
You're missing the actual theorem in your comment, but the one you are referring to is essentially the pythagorean theorem for Bregman divergences, which I think may be a bit too far removed from geometry to allow this proof to generalize.
Ah, was just a flicker in my mind. Thanks - though I will ask, is there anything like the law of sines/cosines (in high dimensional statistics) that could get you to at least the first step of the proof?
The wikipedia lists a law of cosines [1] for the Bregman divergence and someone in this thread posted a proof that basically showed the law of sines follows directly from the definition of the sine. So neither form an obstruction per se, but the generalized pythagorean theorem kind of just follows directly from the definitions you need to make to even be able to say what it means for a triangle to be rectangular.
