Download the mod in the zip format. If you have an .utmod file, unpack it using Santaduck toolpack.
Place the Mod folder inside UT2004.app package (ctrl+click>show package content). Now UTModLauncher shows the mod in its panel. Note that you can manually launch mods by typing in Terminal: /Path/To/UT2004.app/System/ut2004-bin -mod=ModName.