The temple complex is large and much of it is open to the public. Many tours and tourists visit this area but, remarkably, the garden itself is relatively quiet. It also has the advantage of having several spots where a would-be meditator can tuck him or herself away for a few minutes of repose. The "pano" shot below illustrates two such places (the tree stump and the wall beside the shrine which is at a sort of dead end, both on the right side of the photo).
The garden is surprisingly empty so finding some solitude shouldn't be too difficult, especially right at opening time or lunch time on a weekday.