In the cosmetic industry, active ingredients often consist of organic compounds that are hydrophobic, making oil-based formulations impractical. Oil-in-water (O/W) nanoemulsions are used to deliver such ingredients in an aqueous medium. However, creating stable nanoemulsions with uniform size, especially from natural products, poses significant challenges. This study presents the development of micellar O/W nanoemulsions to deliver meroterpenoid-rich fractions from the ethanolic extract of Sargassum serratifolium (MES). We systematically investigated the effects of mixing order and chemical compositions on the size and stability of nanoemulsions, enhancing stability with a tri-block copolymer for micellar features. Under optimised conditions, nanoemulsions with a size around 20 nm were prepared. The MES-loaded nanoemulsion effectively inhibited melanin production over multiple cycles, demonstrating increased stability against oxidation. Molecular docking revealed strong binding affinities of MES compounds to tyrosinase-related protein 1 (TYRP1), indicating potential as effective skin-whitening agents. © 2024 Informa UK Limited, trading as Taylor & Francis Group.