Ray Kelly grew up in Bellbrook Reserve. The son of rabbit trappers Margaret Adelaide Campbell and Raymond Chevrot Kelly, Ray was cared for by a woman named Ivy Long who belonged to the Gambaynggar people and and her husband Donald Thompson who was a Danggadi man.
Early in life, Ray Kelly learned to box to defend himself and later used his skills as a boxer to support his family until retiring due to an eye injury. In 1958, Kelly was a new father working for Ken Jones, the Mayor of Armidale. In 1966 he became involved with the Association for Assimilation of Aborigines (AAA) and was chosen to represent the Association on a tour of New Zealand. He advocated for the revival and maintenance of Aboriginal culture in New South Wales often with friend John Delaney whom he met in 1956.
Ray Kelly was the first Aboriginal person to be employed by the New South Wales Parks and Wild Life Service in 1973. He was instrumental in beginning the process of recording Aboriginal Heritage sites across New South Wales.