President-elect Donald Trump is upending the modern convention that the U.S. speaks with one voice on foreign affairs, plunging into some of the most sensitive national-security matters before he takes office.

Mr. Trump has launched a series of challenges to President Barack Obama’s policies on nuclear weapons, China and the Israeli-Palestinian peace process, setting up a rare and increasingly public confrontation between outgoing and incoming administrations.

Mr. Obama on Friday brushed back pressure from Mr. Trump to block a United Nations Security Council resolution harshly criticizing the expansion of Israeli settlements. Mr. Trump on Thursday called on the administration to veto the resolution. But Mr. Obama instead chose to break from longstanding U.S. policy and allow it to pass.

“There’s one president at a time,” said Ben Rhodes, a deputy national-security adviser. He said the president believes “it’s important that the world understands who is speaking on behalf of the United States until Jan. 20.”

Mr. Trump took to Twitter after the vote, saying: “As to the U.N., things will be different after Jan. 20th.”

While Mr. Obama’s move suggests it may be difficult to eclipse a sitting president who has said he intends to “run through the tape,” Mr. Trump’s policy pronouncements as president-elect could send mixed signals to America’s allies and partners overseas about who is in charge, experts and analysts said.

Mr. Trump signaled soon after the election that he planned to take a different posture during the transition, when he took a protocol-breaking phone call from the president of Taiwan. The White House was caught off guard and fielded angry protests from Beijing.

Jon Alterman, a national-security expert at the Center for Strategic and International Studies, said Mr. Trump appears to be using his transition to test the waters on some issues. “It feels to me partly like he’s just thinking out loud trying to imagine what a Trump foreign policy will be,” Mr. Alterman said.