According to legend, Toko-ji Temple was built in the late Heian Period (794–1185). It was originally a prayer hall called Kokoku-in Temple built for Minamoto Yoshimitsu (Shinra Saburo Yoshimitsu), founder of the Kai Genji Clan.
In the mid-Kamakura Period (1222–1287), it was restored as a temple of the Rinzai sect of Japanese Zen Buddhism by Rankei Doryu, and was newly named Toko-ji at this time. Doryu was a priest who came to Japan from China during the reign of the Southern Song Dynasty (1127–1279). After being ousted from Kamakura by the reigning political regime, he exiled to Kai Province where he was welcomed by the local Kai Genji Clan. The Zen garden gracing the temple grounds is said to be designed by Doryu, and is a prefecturally designated Cultural Property. Letters written by Doryu have also been discovered at Toko-ji.
During the Warring States Period (1467–1603), Tokoji flourished as one of the Kofu Gozan (Five Temples of Kofu) designated for protection by Takeda Shingen. As a temple closely related to the Takeda Clan, it houses the graves of Shingen’s eldest son Yoshinobu and of Suwa Yorishige, the father of Shingen’s concubine Suwahime.
The Buddhist sanctum (Yakushi-do Hall), believed to have been built in the mid-16th century during the Muromachi Period (1336-1573), is designated a National Important Cultural Property. It escaped being destroyed when Oda Nobunaga set fire to Toko-ji after defeating the Takeda Clan and moving his headquarters to Kai Zenko-ji Temple. It also suffered the Kofu Air Raid in World War II, but miraculously escaped being burnt down, again. Sword scars from Nobunaga’s invasion remain on the pillars of the sanctum.