Yes, I am guilty of the same 🙂
I honestly never had an issue not siphoning the substrate, either with fish or with algae. It's safe if you have shrimp and Malaysian trumpet snails because organics are reduced to minimum size that bacteria can colonize and produce nutrients and co2, providing there's plenty of oxygen in the tank and the substrate is not extremely deep, plus well planted of course.