We present results from a search for neutrinoless double-beta (0 nu beta beta) decay using 36.6 g of the isotope Nd-150 with data corresponding to a live time of 5.25 y recorded with the NEMO-3 detector. We construct a complete background model for this isotope, including a measurement of the two-neutrino double-beta decay half-life of T-1/2(2 nu) = [9.34 +/- 0.22(stat)(-0.60)(+0.62)(syst)] x 10(18) y for the ground state transition, which represents the most precise result to date for this isotope. We perform a multivariate analysis to search for 0 nu beta beta decays in order to improve the sensitivity and, in the case of observation, disentangle the possible underlying decay mechanisms. As no evidence for 0 nu beta beta decay is observed, we derive lower limits on half-lives for several mechanisms involving physics beyond the standard model. The observed lower limit, assuming light Majorana neutrino exchange mediates the decay, is T-1/2(0 nu) > 2 > 2.0 x 10(22) y at the 90% C.L., corresponding to an upper limit on the effective neutrino mass of < m(nu)> < 1.6-5.3 eV.