Exactly. My first assistant was Siri, but after several months of just playing with it, I eventually just turned it off. That is one of the more irritating features, if you can call it that, in iOS 11, Apple removed the ability to disable Siri, now you have to go in and block Siri's access to everything, one app at a time.
My next assistant was an Amazon Echo, I gave up on it because of the many non ordering related bugs that Amazon was just ignoring.
So now I have a Google Home, I like it better than Siri or Alexa, but it too has problems, like the timers, alarms, and reminders, that don't work, mistaking even silence for the attention phrase, and very skippy voice recognition, that despite plenty of feedback, just don't seem to be getting addressed.
Other than Siri, which I don't count because it's on my phone and I disabled it. I started exploring these digital assistants less than a year ago, and all those years before that I got along fine without them.
The one thing I'll give them is their ability to stream radio stations, my favorite local radio station is a public radio station, so it doesn't have a lot of power and is hard to pick up with a standard radio. It would be nice if I could find a blind friendly internet radio, I switch out the Google Home for it in a heartbeat.
Maybe I'll just do what you are doing and keep the mikes turned off. I wish Google would make a voice remote for their Home devices like Amazon has for the Echos. It's about the size of a large bic lighter so it's easy to put in your pocket. You could turn off the Echo's mikes and it would still respond to the remote, and since the remote is battery powered, it's on and listening only while you are pushing the talk button, and you don't have to get its attention by saying its wake up phrase before giving it a command. It was the ideal way to use the Echo and not have it listening all the time.