In old days Japan, there was a certain type of religion called Koshin which originally meant a certain day under the Chinese zodiac calender. On this day, people stayed up to make poems, play board games, have parties, etc. Koshin under the zodiac calender is a monkey. Therefore, later on during Edo Era between 16th – 19th century, it turned into a custom to worship monkeys. In some shrines, there are Koshin-to which was built by the worshipers of Koshin.
The above is a photo of Toyosakae Inari Shrine in Shibuya. This shrine has 13 Koshin-to altogether. Many of them have carvings of three monkeys.
This is the main building of the Shrine.
Address: 4-7 Shibuya 3-chome, Shibuya-ku, Tokyo
For more information abouth the religion of Koshin, I found an interesting website. It does not have much explanation but you will find a wide variety of Koshin-to.