Bifurcation and three-wave-mixing instabilities in nonlinear propagation in birefringent dispersive media

Polarization modulationalal instability in a nonlinear dispersive birefringent medium refers to the exponential growth of polarized sidebands due to mixing with an orthogonally polarized pump. We show that this instability extends to the nonlinear regime of strong coupling between the waves. By reducing the coupled nonlinear Schrodinger equations that govern the interaction to a finite-dimensional integrable Hamiltonian system, we show that nonlinear modulational instability originates from bifurcations of spatial eigensolutions of the three-wave interaction. Spatial recurrence of the field evolutions and period doubling related to the existence of spatially unstable eigensolutions are relevant for applications which make use of polarization modulational instability (or birefringence-matched third-order threewave mixing) in the strong-depletion regime. In contrast to previous results obtained by means of linearized equations, we find that efficient frequency conversion, when strong coupling is accounted for, occurs for initially wave-vector-mismatched waves.