Is it possible to be allergic to olives and still be able to use olive oil in cooking or in dressings and dips?

For the longest time I have been unable to eat whole olives without getting sick later, however, I use olive oil in cooking and with meals and I don't get sick.