Billy Packer developed his love of college basketball while he was growing up as the son of the coach at Lehigh University. Packer attended Wake Forest University, where he was an All-Atlantic Coast Conference (ACC) guard (1960-62) and led his team to the Final Four in 1962. After graduation that year with an economics degree, he spent more time in the gym as a Wake Forest assistant coach (1965-69). In 1972, Packer began his career in broadcasting in Raleigh, North Carolina, when he was asked to fill in as an analyst for a regionally televised ACC game. Packer became a regular the next season.

After covering ACC games, Billy Packer joined NBC Sports for the opening round of the 1974 NCAA tournament. In 1975, he teamed with Curt Gowdy for the entire tournament, including the UCLA-Kentucky title game. The following year, Packer worked with Dick Enberg for regular-season games as well as the tournament. Al McGuire joined the pair in 1978, and they became one of the most popular broadcasting triumvirates ever.