Washington (CNN) Lawyers for two Iraqis with ties to the US military who had been granted visas to enter the United States have filed a lawsuit against President Donald Trump and the US government after they were detained when they arrived in New York Friday.

The lawsuit could represent the first legal challenge to Trump's controversial executive order, which indefinitely suspends admissions for Syrian refugees and limits the flow of other refugees into the United States by instituting what the President has called "extreme vetting" of immigrants.

Trump's order also bars Iraqi citizens, as well as people from six other Muslim-majority nations, from entering the US for 90 days, and suspends the US Refugee Admissions Program for 120 days until it is reinstated "only for nationals of countries for whom" members of Trump's Cabinet deem can be properly vetted.

According to court papers, both men legally were allowed to come into the US but were detained in accordance with Trump's move to ban travel from several Muslim-majority nations.

The lawyers for the two men called for a hearing because they maintain the detention of people with valid visas is illegal. They were still at John F. Kennedy International Airport as of late Saturday morning, one of the lawyers told CNN.

"Because the executive order is unlawful as applied to petitioners, their continued detention based solely on the executive order violates their Fifth Amendment procedural and substantive due process rights," the lawyers argue in court papers.

RELATED: Trump's executive orders, memorandums and proclamations

The two Iraqi men named as plaintiffs in the suit are Hameed Khalid Darweesh, who worked as an interpreter for the US during the Iraq War, and Haider Sameer Abdulkaleq Alshawi. The suit said Darweesh held a special immigrant visa, which he was granted the day of Trump's inauguration on January 20, due to his work for the US government from 2003 to 2013.

The lawsuit said the US granted Alshawi a visa earlier this month to meet with his wife and son, whom the US already granted refugee status for their association with the US military.

Court papers said Customs and Border Protection authorities did not allow the lawyers to meet with the men and told them to try reaching Trump. Democratic Reps. Jerrold Nadler and Nydia Velázquez, both of New York, attempted to speak to Darweesh and Alshawi at JFK's Terminal 4 but were denied.

"When Mr. Darweesh's attorneys approached CBP requesting to speak with Mr. Darweesh, CBP indicated that they were not the ones to talk to about seeing their client. When the attorneys asked, 'Who is the person to talk to?' the CBP agents responded, 'Mr. President. Call Mr. Trump,'" the court papers read.

Lawyers for Iraqi refugees filed the suit in federal court arguing Darweesh and Alshawi were being unlawfully held "solely pursuant" to an executive order issued on January 27, 2017.

Arguing that their clients have "valid entry documents," they say the were blocked from exiting JFK and detained.

One of the attorneys, Mark Doss of the International Refugee Assistance Project, told CNN his clients knew they had to get to the US as soon as possible so they boarded the first flight they could.

The two men, one of whom has now been held for 14 hours, have been allowed to make phone calls. The two men do not know each other, and it is unclear if they are being held together or separately, or if they are being kept in a holding cell, according to Doss. One man traveled from Sweden and one from Iraq.

One man is married and his wife lives in the US. She had prior refugee status and she had secured the same for her husband. The other man had a US-affiliated special immigrant visa, which allows Iraqis and Afghans who had served on behalf of US troops to enter, Doss said.

Some of the International Refugee Assistance Project's clients entered Friday with no issue, Doss said, adding that a handful of families entered JFK and Los Angeles International Airport without any problems.

The lawsuit was earlier reported by The New York Times.

Immigration-rights groups, including the National Immigration Law Center and the ACLU, are representing the Iraqi men. Lawyers for the Iraqis are aiming to file a class-action lawsuit on behalf of other refugees.

"Our courageous plaintiff and countless others risked their lives helping US service members in Iraq. Trump's order puts those who have helped us in harm's way by denying them the safe harbor they have been promised in the United States," said Karen Tumlin, the legal director of the NILC.

An administration official told CNN if a person has a valid visa to enter the US but is a citizen of one of the seven countries under the temporary travel ban, then the person cannot come into the US. If the person landed after the order was signed Friday afternoon, then the person would be detained and put back on a flight to their country of citizenship.

Separately, Department of Homeland Security officials acknowledged people who were in the air would be detained upon arrival and put back on a plane to their home country. An official was not able to provide numbers of how many have already been detained.

Legality questioned

Trump's executive order, titled "Protection Of The Nation From Foreign Terrorist Entry Into The United States," makes good on his longtime campaign promise to tighten borders and halt certain refugees from entering the United States.

The countries impacted are Iran, Iraq, Syria, Sudan, Libya, Yemen and Somalia, according to a White House official. It also caps the total number of refugees admitted into the United States during the 2017 fiscal year at 50,000, down more than half from the current level of 110,000.

"I am establishing new vetting measures to keep radical Islamic terrorists out of the United States of America," Trump said during the signing at the Pentagon. "We don't want them here."

But the order immediately raised questions about its legal standing.

"What this order does is take people who played by the rules who have been rigorously vetted and slams the door of the country in their face because of their religion," said David Leopold, an immigration attorney and the former president of the American Immigration Lawyers Association. "What this lawsuit does is asks the court to declare that Trump's illegal executive order cannot be used as a pretext to ban Muslims and lock up legal immigrants without due process."

But Stephen Yale-Loehr, a professor of immigration law practice at Cornell Law School, said Trump's order may have merit.

"The plaintiffs are alleging a variety of reasons why they should not be detained under the executive order. Their arguments include the equal protection clause of the Constitution and a provision in the immigration law that prohibits national origin discrimination," Yale-Loehr said. "However, presidents have wide discretion on immigration, because immigration touches national sovereignty and foreign relations. Courts tend to defer to whatever a president declares on immigration. I think the administration could win."

Progressives outraged

Democrats and liberal-leaning groups continued to denounce Trump's order Saturday as racist.

"The first two refugees kept out of America by President Trump's order are Iraqis who helped US troops survive during the war," Rep. Jim Cooper, D-Tennessee, said on Twitter. "These men risked their lives for years to keep our troops safe. One was with the 101st Airborne. This is no way to treat our allies."

Vermont Sen. Bernie Sanders called the action an "anti-Muslim order" that "plays into the hands of fanatics wishing to harm America."

And the ACLU repeatedly referred to the executive order as Trump's "Muslim ban."

Iran says it will ban US citizens in response

Iran's government described Trump's immigration order as an "insult," a "gift to extremists," and announced that its government would instill reciprocal measures banning US citizens, according to a Foreign Ministry statement published on state media.

"The US decision to restrict travel for Muslims to the US, even if for a temporary period of three months, is an obvious insult to the Islamic world and in particular to the great nation of Iran," the statement said.

Also Saturday, some US-bound travelers who were either refugees or from countries specified in Trump's executive ban have been turned back from Cairo International Airport, according to a Cairo airport official who is not authorized to speak to the media.

The official said they were required to follow the instructions of the United State in the matter. He said "(The airport personnel) have no choice but to follow orders. I don't want to blame anyone here."

"This is a new era we are witnessing," the official said.

When pressed about the exact protocol on who to turn back, including whether or not US green card holders from the seven Muslim-majority countries were banned entry under the order, the official said, "I don't know, we are waiting for more clarifications." He also would not specify how many were being turned back.

CNN's Kevin Bohn, Sheena Jones, Mohammed Tawfeeq, Rachel Crane, Chris Welch and Sarah Jorgensen contributed to this report.