President Barack Obama said prosecuting pot users in states that have legalized the drug won't be a top priority for his administration.

"We've got bigger fish to fry," Obama told ABC News' Barbara Walters . "It would not make sense for us to see a top priority as going after recreational users in states that have determined that it's legal."

Last month, voters in Colorado and Washington legalized recreational pot use for adults, though marijuana remains illegal under federal law.

The Obama administration suggested last week that it was considering plans to undermine the voter initiatives . In his interview with Walters, Obama did not say whether his administration would go after producers and suppliers of marijuana in those states. The administration has cracked down extensively on the medical marijuana industry in California, despite its legality under state law there.

A slim majority of Americans want the Department of Justice to leave pot smokers alone in the states where the drug has been legalized, according to a HuffPost/YouGov poll .

Obama himself smoked pot as a youngster in Hawaii, where he and his high school pals called themselves the Choom Gang.

"There are a bunch of things I did that I regret when I was a kid," he told Walters . "My attitude is, substance abuse generally is not good for our kids, not good for our society."