I don’t think you can. I was looking for this the other day too. I didn’t find any way to do this in the options. I think it just has a standard rotation of colors that it applies based on the number of categories.
I too need this feature. My own recourse at the moment is to save the SVG and modify it directly. Not a great workflow.
It’s really frustrating because those colors are not accessible. White text on the yellow, for example, has a contrast ratio of only 1.55:1.