Ali was born in the village of Kaleki in the Tehsil Daska Sialkot District of Punjab, in 1940 in British India now Pakistan. He belongs to a musical family, his father was a vocalist and a sarangi player who initiated Ghulam Ali to music from his childhood. Ali's father named him after Bade Ghulam Ali. At 15, he became a student of Bade Ghulam Ali Khan, a master of the Patiala Gharana of Hindustani music. Due to the busy schedule of Bade Ghulam Ali, he was trained mainly by Bade Ghulam Ali Khan's three brothers, Barqat Ali Khan, Mubarak Ali Khan, and Amanat Ali Khan, in Lahore.
All these distinguished classical musicians taught him the finer nuances of classical music. His solid foundation of classical music included studying Thumri and learning to sing raga.
His name 'Ghulam' was given by his father, a great fan of Bade Ghulam Ali Khansaab. He used to live in Lahore. He had always been listening to Khansaab since childhood.
Ghulam Ali encountered Ustad Bade Ghulam Ali Khansaab, an unparalled legend of classical music for the first time when he was in early teens. When Ustad Bade Ghulam Ali Khansaab had come to Kabul, Ghulam Ali's father requested the Ustad to take his son as a disciple. But the maestro insisted that since he was hardly in tow, regular training wouldn't be possible. But after repeated requests from Ghulam Ali's father, Utsad Bade Ghulam Ali Khansaab asked the young Ghulam Ali to sing something. It wasn't easy to sing before him. He mastered the courage to sing the Thumri "Saiyyan Bolo Tanik Mose Rahiyo Na Jaye..". After he finished, Ustad hugged him and made him his disciple.
Ghulam Ali started singing for Radio Lahore in 1960. Along with singing ghazals, Ghulam Ali composes music for his bhajan. His compositions are raga-based and sometimes include a scientific mixture of ragas. He is known for blending gharana-gaayaki into ghazal and this gives his singing the capability to touch hearts. He beautifully sings Punjabi songs too. Most of his Punjabi songs have been extremely popular. Though from Pakistan, Ghulam Ali remains as popular in India as in Pakistan.
He entered Hindi cinema with a Hindi film song Chupke Chupke Raat Din in B. R. Chopra's film, Nikaah (1982). Other popular ghazals include Hungama Hai Kyon Barpa and Awaargi. He always selected the ghazals of famous poets. One of his ghazal 'Hum ko kis ke gham ne mara' written by Masroor Anwar was a top hit.
On being questioned about Pakistani pop groups, Ghulam Ali replied, "Frankly, I am really bewildered at their style of singing. How can you sing a song by running and jumping around the stage? The stage is meant for performing not for acrobatics."
Ali has also sung some beautiful ghazals like Kina kina timro tasveer, Gajalu tee thula thula aankha, Lolaaeka tee thula and Ke chha ra diun in Nepali language with Narayan Gopal, a well known Nepali singer, and composer Deepak Jangam.Those songs were written by King Mahendra of Nepal These songs were compiled in an album entitled Narayan Gopal, Ghulam Ali Ra Ma, and are popular among Nepali music lovers to this day. One of his memorable concerts was at the Taj Mahal.