Born in New York, Ruby failed in his early ambition to become a professional baseball player. Touring the vaudeville circuit as a pianist with the Bootblack Trio and the Messenger Boys Trio he met his long-time partner Bert Kalmar. Together, Ruby and Kalmar formed a successful songwriting team until the latter's death in 1947.