There could be a variety of reasons. Peppers do not like extremes of weather so if too hot or too cold they will suffer blossom drop. Peppers like moist well drained soill but become unhappy if the soil is either water logged or too dry. Watering every 2 or 3 days and avoid watering in the evenings can help.Your flowers might not be pollunating properly which means the immature fruits will fail to develop and the stems will yellow and fall. Although peppers are self fertile, giving them a little shake will help the pollen move around. Peppers have a tedency to cross so If you are looking to save your own seeds and have more than one variety you might want to cover a few flowers to isolate them (I use old thights cut and sewn into little bags) then pop a piece of twine or wire on resultant fruit so you know those seeds will be true to type.
It could also be a deficiency, I use Epsom salts as a forthnightly foliage feeder and a high potash feed on alternate weeks. Make sure you do not feed with nitrogen as peppers do not like this.