Instead a group of eager Latinos persuaded him to come inside and speak to a packed room of Mexican-American civil rights activists. And then he persuaded his wife, first lady Jacqueline Kennedy, to address the crowd in Spanish.
It was Nov. 21, 1963. Hours later, the president was dead, his assassination overshadowing the significance of a speech that can be seen as the birth of the Latino vote, so instrumental in 2012 in helping re-elect the first black president, Barack Obama.
To historians, Kennedy's appearance at the Rice Ballroom in Houston was likely the first time that a president officially acknowledged Latinos as an important voting bloc.
Though there are no plaques marking the historic occasion, the event is a touchstone for activists even if the spot where Kennedy sat and heard a band play Mexican ballads and where the crowd yelled "Viva Kennedy!" is now a refurbished ballroom in a loft apartment complex that often plays host to weddings.
"That evening ... that's where it began," said Ignacio Garcia, author of "Viva Kennedy: Mexican Americans in Search of Camelot" and a history professor at Brigham Young University. "But because very few people know about the meeting, it's like it never happened."
The surprise visit came after Mexican-Americans in Texas, New Mexico, California, Arizona, Illinois and Indiana helped Kennedy win critical swing states in 1960, thanks to an unprecedented voter registration drive in Latino communities. Independent "Viva Kennedy!" clubs sprang up. Sen. Dennis Chavez, D-N.M., and Texas legislator Henry B. Gonzalez of San Antonio, a future congressman, began speaking in Hispanic neighborhoods across the country and positioned themselves as the first recognizable national Latino political figures.
Just as in 2012, Republicans in 1960 did little to woo Latinos to support their presidential candidate, Richard Nixon. Latinos also identified with Kennedy, who was Catholic and Irish-American, a member of an ethnic group that had battled discrimination similar to what Latinos faced in the segregated Southwest.
On Election Day in 1960, Kennedy won 85 percent of the Mexican-American vote.
But during Kennedy's first months in office, Latino leaders expressed dismay that the president had failed to appoint Hispanics in his administration. Chavez even openly criticized Kennedy for his lack of appointments; other leaders embarked on a letter-writing campaign over the slow movement on civil rights.
Sensing another close election in 1964 and hoping to ease tensions, Kennedy visited Texas in November 1963. Advisers suggested that he at least pay a quick visit to Mexican-American activists at a Houston gala sponsored by the League of United Latin American Citizens, then the largest Latino civil rights group in the country.
"The Secret Service told us that he may stop by, but not to advertise it because it wasn't part of his official schedule," said Alexander Arroyos, 76, who was an officer in LULAC at the time. "We could spread it through word of mouth. No one believed us."
Then Kennedy showed up.
The president was greeted at the door by Macario Garcia, who was awarded the Medal of Honor for his service during World War II. Inside the ballroom, Kennedy and the first lady found an enthusiastic crowd of World War II veterans, civil rights advocates and future elected officials.
Kennedy spoke briefly about foreign policy in Latin America and the importance of LULAC. The first lady told the crowd in Spanish that Texas had a deep history with Latinos. The crowd responded with chants of "Viva Kennedy!" A band played a ballad in Spanish as photographers took photos of the Kennedys and Vice President Lyndon B. Johnson and his wife, Lady Bird Johnson.
Before that moment, historians believe that no president had ever acknowledged Latinos as a voting bloc, said Emilio Zamora, a historian at the University of Texas at Austin.
President William Taft, who served from 1909-1913, may have met with a tiny group of Latino activists in El Paso, Texas, Zamora said. President Dwight D. Eisenhower likely shock hands with some Mexican-American voters in a campaign visit to South Texas in 1952. "But I think no president had ever publically thanked Mexican-Americans in that manner," said Zamora.
Fifteen hours after the historic meeting, Kennedy was dead.
Band members who had played for the president the night before wept as the news unfolded. When Arroyos heard about the assassination, he told his boss at an import company he was too upset to work. Arroyos rushed to collect from friends as many photos as possible of Kennedy's visit at the Rice Hotel as he could for a future edition of a LULAC newspaper.
On Election Day 2012, analysts routinely spoke of Latinos finally awakening as a "sleeping giant" by giving Obama around 70 percent of their vote. But Ignacio Garcia said that assessment ignores how Latinos have influenced presidential elections for more than 50 years.
In 1960, for example, their overwhelming backing helped put Texas and New Mexico in Kennedy's column during the tight race against Nixon. The Republican's campaign did not have a presence in Mexican-American neighborhoods and did not have a Spanish language TV ad, unlike Kennedy, who tapped the first lady for it. Kennedy also made promises to appoint Mexican-Americans to his administration.
Johnson enjoyed support from Hispanics who campaigned for him during his landslide victory in 1964, and Mexican-Americans came out strongly for Sen. Robert F. Kennedy, D-N.Y., during the 1968 Democratic primary in California.
In 2000, then-Texas. Gov. George W. Bush, a Republican, was able to edge Democrat Al Gore, thanks in party to receiving about 40 percent of the Latino vote, according to various estimates.
"The Latino vote did not come of age the night Obama was re-elected," said Garcia. "It came of age Nov. 21, 1963."
The reason the Latino vote is attracting attention in 2012 is that Latinos are now the largest minority group in the U.S. and voter participating rates are up, Garcia said.
Voter participation for eligible Latino voters has gone from 3.7 million in 1988 to an estimated 12.5 million in 2012, according to the Pew Hispanic Center. That number could to double within two decades, the center said.
Arroyos said most of the older activists shrug off the pronouncements that Hispanics are finally influencing national elections even though his generation helped give birth to the Latino vote. Still, he said even those who are still alive and remember that Kennedy speech probably don't even know what role they played that eventually led to the voting numbers in 2012.
"I didn't know that evening was so historic," said Arroyos. "I was just happy that he dropped by and just didn't say hi."
Follow Russell Contreras at http://twitter.com/russcontreras