Here's a simple question: Why did William James take the approach that he did in Varieties of Religious Experience, an approach to that field marked by the individual experiences of believers in some higher power?
There is on one level the 'official' answer: the one that James gives. He told his lectures' audience that he has been invited to give lectures about religion, a wide-open mandate, and that in order to proceed he would have to select "out of the many meanings of the word [religion] ... the one meaning in which I wish to interest you particularly." The selection is presented as an arbitrary one, defining religion only "for the purpose of these lectures" as "the feelings, acts, and experiences of individual men in their solitude, so far as they apprehend themselves to stand in relation to whatever they may consider the divine."
But no careful student of William James thinks that this is the whole answer. There are at least three not-at-all arbitrary reasons why James proceeded as he did.
First, because he was his father's son. He learned religion from Henry James Sr., the man who "wrote The Secret of Swedenborg and kept it." To a James, depth of religious experience would naturally have seemed coextensive with personal idiosyncrasies. More broadly, he was a product of the America that was also producing or had recently produced Joseph Smith, Dwight Moody, Mary Baker Eddy ... it was a time and place where highly individualized religious experience, pressing against the envelope of Protestant Christianity, was itself collectively valued.
Second, James wrote this way because James saw himself as a student of history, and the notions that (a) individuals matter in history, and (b) that individual action was in his day in danger of being lost in scholarly talk of aggregates and impersonal forces, was at or very near the heart of what all his scholarly endeavors were about. So when he had an opportunity to lecture about religion, it naturally became a exhibit in that broader case. Religion in its ecclesiastical form can of course be treated historically too, and when it does it can lend itself to interpretations that don't really need human names. Clericalism and Sectarianism and 'The Protestant Ethic' are all the sort of aggregating concepts of which James firmly disapproved. So he made the point that if you go beyond the forms to the human heart, forces and aggregates drop away in favor of "him who had it" where "it" is first-hand experience of the divine. Those who had "it" were accordingly often driven "into the wilderness ... where the Buddha, Jesus, Mohammed, St. Francis, George Fox, and so many others had to go."
Third, James looks to individual experiences for the essence of religion because he is a pragmatist. His pragmatism is of a sort that makes of truth itself a very individual product, a report on some one person's wrestling with the world, and of what has so far worked in that struggle, as the turn to a higher power works for those who have found sobriety through AA (a movement in part inspired by James).
For James, at base, everything in the present and future as well as in the past, is about individuals "in their solitude."