A pot of soup left out on the counter overnight isn't safe to eat and you need to throw it out. Soup that has sat at room temperature for more than two hours has been in the "Danger Zone" for too long and has a high risk of making you sick.
Food Safety First
Disease-causing bacteria are the biggest culprits when it comes to food safety and they multiply quickly in the "Danger Zone" between 40 and 140 degrees Fahrenheit. Once food, including soup, is cooked, if you cannot keep it at 140 F or higher, you must cool it quickly to room temperature and refrigerate or freeze it within two hours. The U.S. Department of Agriculture Food Safety and Inspection Service recommends throwing out any foods that were left at room temperature for more than two hours, and after one hour if the room temperature is above 90 F.
Special Considerations for Soups
Because of their volume, divide soups into smaller quantities for quicker cooling and refrigerate them as soon as possible. The refrigerator should be at a temperature of 40 F or below. Store soups no longer than four days in the refrigerator or three months in the freezer.