Peter Prior's father was a descendant of Aboriginal and Malayan parents and his mother was the daughter of Aboriginal and South Sea Islander parents. Prior grew up traditionally with his family and close relations until they were forced off their land by government officials and transported to Yarrabah Mission. Prior's uncle Tommy King, who was also known as King Billy, was among his relatives that were captured and forced to relocate to the mission. After his family escaped from Yarrabah Mission they were captured on their way to Cairns and taken to Hull River, then to Palm Island by the authorities.
Prior began working as a teenager on Magnetic Island building a holiday resort with his brother. On the 3rd of February in 1930, Prior shot and killed the Palm Island Aboriginal reserve superintendant Robert Curryin self defence. After going to trial and being held for three months in police protection, he was acquitted of murder. Peter Prior is the father of Renarta Prior (q.v.) who recorded his life story in Straight from the Yuduman's Mouth. Prior's nickname, 'Gullumbah', translates to being left-handed.