Am I overreacting to my husband touching me when I’m asleep?

My ‘43F’ husband ‘43M’ and I have been married 17 years, together 24. He has developed a habit that really irritates me. While I’m sleeping he will fondle my private parts (breast, crotch, bottom). It ranges from a light touch to vaginal penetration with fingers. This sometimes happens even if I’m sleeping in a different room. We have three kids, and this started 9 years ago when my middle was a baby, so I truly don’t appreciate being woken up for a non-emergency when a full nights sleep is far from guaranteed. I’ve been telling him for years how I don’t like this. This situation is a Catch 22 for us because it makes me want to have sex less, which makes him long for sex more. Last night I woke up to him touching my crotch and had to drag an apology out of him. He doesn’t see the problem as much since our sex life isn’t very active. Am I overreacting and this isn’t unusual for couples, or is it a violation of boundaries? I told him I’d ask. TL;DR: My husband sometimes fondles me or fingers me when I’m asleep. He blames our subpar sex life, I say it’s a boundary violation.