If you are scared, you should not go to the restaurant. Just stay home. That is your right. If you ignore it, however stupid it may be, it is your right as well.
Definition of 'add fuel to the fire'
The phrase 'add fuel to the fire' means when you do or say something that makes a miserable situation even worse. 'Adding fuel to the fire' means to make a situation or conflict intensify, especially via provocative comments.