BRUSSELS — The number of refugees arriving in Greece has plummeted thanks in part to the European Union’s pact with Turkey, the EU border agency said Friday, amid signs that the widely criticized agreement to stop people heading to Europe could unravel.