High wind

Wind is a balloons biggest enemy.  Wind of more than about 10mph will make the balloon almost uncontrollable.  To put a balloon up you must first fill it with air.  This is done by laying the balloon on the ground and starting a large fan.  You must hold the balloon still long enough to fill it with air before you turn on the burner to stand up the balloon.  If you have any wind the balloon will begin to catch the wind like a giant sail and begin to drag itself across the field.  A balloon can generate as much as 6,000lbs of pulling force in as little as a 12 mph wind.  At this time we don't even want to talk about landing in windy conditions.

Balloon flying is a VFR type of aircraft.  This means Visual Flight Rules.  These are rules written by the Federal Aviation Administration.  Not only is it illegal to operate a balloon in poor visibility but imagine driving your car with your eyes closed.  Your bound to hit something and it wont be pretty.  To land a balloon the pilot must slow the decent of the balloon to get it to level off right at the ground level.  Hard to do when you can not see the ground.

Generally when you have rain you have weather conditions that can change dramatically.  Here in Florida rain is either driven by an advancing cold front or by the convective heating of the surface of the earth by our intense sunshine.  In either case the rain will add dramatic weight to the skin of the balloon forcing the pilot to add extra amounts of heat to keep the balloon aloft.  It is possible for this weight to become so heavy that it literally forces the balloon into the ground.  If the pilot can successfully land the balloon the damage to the fabric of the balloon will completely ruin the aircraft.