Heath Fogg Davis is a trans man and associate professor in political science at Temple University, and his book, Beyond Trans: Does Gender Matter? suggests that there are many situations in which clinging to gender categories is not necessary and even counterproductive. The context appears to largely be USAn, although I only got a little way into the book so that might not be true of later chapters.
The book opens with the case of a public transit system in Philadelphia that used to issue passes in both male and female variants. It begins with the dilemma of a trans woman who bought a female pass, only to be bounced off the bus because the bus driver judged her not to be a "real" woman, so she bought a male pass, and was bounced off the bus for not being male. At that point, she's screwed--what does she do? But trans people weren't the only one hit by this--a lot of cis people who didn't match certain bus drivers' preconceptions of gender presentation/appearance were also sometimes denied passage.
Davis then goes on to examine the reason why bus passes even had this designation to begin with. Apparently the stated intent was to reduce fraud--basically, each person was supposed to buy their own pass, and they were trying to prevent husbands and wives from sharing a single pass. Except, of course, if you look at the problem and the "solution," it makes no sense--you could easily still have fraud with two people of the same "sex" (whatever that means, a topic Davis takes up later) sharing a pass. So basically the "solution" screwed a lot of people, was intrusive and humiliating, and didn't even solve the problem.
The chapters in this book are:
Introduction: Sex Stickers
1. The Sex Markers We Carry: Sex-Marked Identity Documents
2. Bathroom Bouncers: Sex-Segregated Restrooms 
3. Checking a Sex Box to Get into College: Single-Sex Admissions
4. Seeing Sex in the Body: Sex-Segregated Sports
Conclusion: Silence on the Bus
Appendix: The Gender Audit: A How-to Guide for Organizations
 I lived for two years in a dorm in undergrad that had co-ed restrooms. Nothing bad happened. My dad would have blown a gasket if he had found out, though. :p
I only got through the intro and the very beginning of chapter 1 and what I saw looked encouraging and thought-provoking, but please don't ask me what's in the rest of the book because I genuinely don't know. I'm going to return this and hope to check it out later when I have more brain so I can think about the issues properly; it's good knowing the book exists so I can return to it at some later point.