To Negotiate with an enemy, you first have to get them to Hesitate. They'll only do this if they are down to their last heart of health and are showing Bleedout animations, where every now and again they'll just kinda spurt blood effects. They'll then have a chance to Hesitate instead of using an action, but be careful! If they get another action, such as if they do so at the end of the turn order, their hesitation will stop.
Some enemies will never hesitate, so you will never get the option to Ransom or Recruit. This includes prison guards, bandits and highwaymen, and enemies who are spawned via an event, or who play the boss theme music during combat.
There are two methods you can use for a "False Swipe".
Down to Earth
The highest Labor skill pip, Down to Earth, will make that character's attacks never be a one-hit KO. At most they will drop them down to one health heart remaining. (But they can kill enemies who are also bleeding out.) This skill also makes Recruit attempts more successful when you make the attempt with that character.
Poison, aka the Pox Curse. Poison effects cannot kill enemies or allies outright. It can reduce them to 1 HP and no more. If you don't have anyone with Down to Earth, poisoning them to whittle their health down can be a decent alternative.
When they do hesitate, you can right-click to Negotiate with them, as long as you're not using a reckless action. (Using a reckless action will attack a hesitating monster.) When Negotiating, a dialogue will pop up. If the enemy can understand the character you're negotiating with, you'll have the option to either Release them or press negotiations. (If the monster can't understand you due to not having the right skill to speak with them, they will automatically flee.)
You'll then have the option for either Ransom or Recruit.
Recruit and Ransom
You do not need inventory space to Ransom an enemy, but you must have room in your Stronghold to Recruit successfully. Remember that service NPCs (like the Quartermaster) take up a vacancy as well. This is not indicated on the world map, so refer to the "Base" menu option in your Stronghold or refer to the Tent Counter instead.
Recruit and Ransom attempts are not guaranteed to work. An enemy can refuse both requests, though there are Utility skill pips that can make either more successful. If an enemy refuses your attempt, you have to wait for them to Hesitate again to make another try, assuming you don't kill them in the interim.
Recruited enemies will behave the same as any other Bastard, up to and including being able to Bond. Some monsters will have unique or powerful skills, so be sure you read theirs before taking them out on Location.
Ransomed items will always be the same for the same enemy types. Ifreets, (explicitly named "Ifreet") will always ransom Rubies, no matter where you encounter that Ifreet.