Freedom of religion is based on the right to choose your faith community, that is, to associate with those who share your spiritual journey.
Freedom of religion is not something that began with the U.S. Constitution. The Roman Empire generally accepted the right of its subject populations to practice whatever religious beliefs they chose, so long as they did not appear to threaten the empire.
The Jewish Jehovah God was intimately bound to his people, caring for them and demanding a certain standard from them. Prophets tied the worship of God to justice and special concern for the vulnerable. God required his people to worship him and him alone, but the rest of the world could go its own way, as long as no other nation interfered with the Jewish worship of their God.
With Christianity and Islam, a different outlook emerged. At first, Christians saw the gospel as good news to be proclaimed, but they sought no political power. Only later did political leaders try to fuse Christianity with governing authority. This joining led to a perverted view of Christianity as simply one more lever of power.
Islam, the other evangelistic religion, conquered lands for their religion but generally allowed Christians and Jews to live in their own religious communities so long as they paid a tax for the privilege. Unlike Christianity, Islam was from the beginning a state religion.
Lands influenced by Christianity began to see the value of individuals choosing their own religious communities or no community. Christianity was a religion of the heart, not of an external state. Christianity shook off the shackles of Christendom.
When a religion—any religion—begins to force itself on those who wish to believe otherwise, that religion begins to lose its moral authority. When religion must force itself onto a society, it has failed.