Always think about the structure of a unit in cases like these.
The smallest element is a squad of typically 6-9 men. A squad does not have an officer, and conseqently never strays far from home.
A Platoon normally has three squads plus a small HQ element and has 20-30, perhaps up to 40 with attachments like AA or AT. A village that was garrisoned on an OFP island would probably have at least a platoon there. A squad hardly counts as a garrison. So a village that has troops in it is likely to have 20-40.
A platoon is part of a company, which normally has three platoons and a small HQ element. It's normally 80-120 men and is the largest size unit you can really have in OFP. You might use a company to guard a town.
A company is part of a battalion, which usually has 600-700 men. About half the battalion are infantry (which is what I have been talking about up till now), about a quarter are fire support soldiers (mortars, AT, snipers, heavy MG, assault pioneers, recce) and the remaining quarter are supporting elements (medical, transport, mechanics, signals, clerks, cooks etc). These are all organised in companies and platoons like the front line troops, but the companies are purely administrative - they never fight in companies - and the "platoons" vary enormously in size from about 10 (sniper section) to perhaps 50 (mortar platoon). However you would never find the mortar platoon alone in a village.
You could imagine that the town in question is battalion HQ. In which case there might be 150 soldiers there, but poorly armed. There would be lots of tents and trucks.
50 is a bad number - in realism terms - to put in a place. Of course its possible, but it would be made up of various groups - its not a coherent unit. You would be better to have about 25-30 (a platoon) at the start, reinforced by another 30 later. If you are going to have three lots of 30 then remember to included company HQ (about 10) as well.
Btw this is all based on my sketchy knowledge of British light role infantry. Other armies have slightly different systems, but the basic idea is very widespread. The Roman army used it for example.
In OFP of course, it's not the number of loons that's important. It's how well placed they are and what waypoints they have that really matters.
You would never approach a village that you thought was occupied by the enemy with less than a platoon. This is where OFP is unrealistic, you frequently approach with just a squad. And of course special ops are an exception.