I cancel my application for abroad but the agency took my passport and not giving it back what should i do?

By international law, passports are the property of the government that issued them. It is illegal for anyone else to confiscate them. Tell the agency that you will file a complaint with the police if they do not give yours back. Hopefully, you live in a country where the police will actually respond to your complaint.