I've heard it argued that this is one of the definitional and most draining aspects of micro-aggressions: it's always theoretically possible that it might in any particular instance have been an innocent mistake or coincidence (or or or), so it forces you into that position of suspecting but not ever knowing for sure, wondering if you're just being paranoid or over-sensitive, not being able to raise it because it's deniable (or choosing to raise it knowing it will be denied).

Please enjoy this excellent article on microaggressions, which agrees with you. "I suggest that ambiguity is not a bug, but is in fact a key feature."