I think you'll need more than that. Googling shows a single Li ion cell, nominal 3.7V, and the CR123A's I have and a charger for same work by putting current in during a second or so, then sampling the voltage for a few milliseconds, repeating till threshold is passed. As the discharge curve is rapid to fall to around nominal voltage, you might get by with a zener circuit to keep the input voltage strictly at 3.8V max. That way it should never exceed safe limits, and still give you 90% of full charge at best. An LM317 regulator would be better, too, but a zener circuit might be adequate if you test carefully to see what voltage results for the current you find there. Try it on a spare cell, and do it in a place that won't mind if one explodes.
Edit: I'm assuming it is wise to provide your own protection. Best not to rely on existing stuff, for these batteries.