Absolutely. You have full control over which agencies have access to your roles. To remove an agency from a JobCast that has already been sent, first, click on the name of the JobCast. This will bring you to the JobCast-specific page. Next, click the 'Invited Agencies' tab on the left side navigation. This is in the 'JobCast Info' section.
On this page, you will see each individual agency, if they have accepted your job, if they have submitted candidates, and some additional statistics. If you would like to remove this agency from your role, click the three vertical dots on the right side of their row. You'll see the option 'Remove From JobCast.' By selecting this option, you are turning off the agency's ability to submit new candidates for the role. Just so you know, any previously submitted candidates will still be visible, and the agency may chat with you on those candidates.