If you think about it, Sarutobi Hiruzen wasn't probably self-centered/power-hungry/malicious in how he treated Naruto, unlike Danzo. His wife literally died to deliver the baby. His eldest son, Konohamaru's dad, is dead. He's probably just really depressed a lot of the time... just let him peep on girls bathing in peace, damn it.