Joe Wallam was born in Bunbury but as an infant moved with his family to Wagin reserve after his father died. At the age of eight, Wallam was removed from his mother's care and placed in Marribank Mission. His mother fought to get him back and he was returned nine months later.
Wallam began a trade apprenticeship but wanted to use his mind more. He moved to Perth to study and became involved with Aboriginal organisations. Wallam worked as a Research Assistant for an anthropologist, and then joined the Sites Department of the Western Australian Museum as Heritage Officer.