The easiest way to approach this problem is to consider nine optical waves propagating in the fibres: the incoming wave, main waves in the two semi-arcs and in the two outgoing fibres, and the double-frequency waves in the the semi-arcs and in the outgoing fibres. For more details why this works and how to proceed, cf. https://www.ioc.ee/~kalda/ipho/waveopt.pdf, idea 1 and problem 7 (which is solved therein on Pg 6).
You might still run into mathematical difficulties which can be avoided or resolved with appropriate parametrization; this will be the topic of the next hint.
Please submit the solution of this problem via e-mail to physcs.cup@gmail.com.
For full regulations, see the “Participate” tab.