It’s important to not harm the carpenter bees, since they pollinate plants. Also, be certain they are carpenter bees, so you don’t chase away other helpful insects.
Oils like citrus and almond oil can also repel carpenter bees. Mix them in a spray bottle and spray it on the affected wood to deter further bee activity.
A more permanent solution is to paint, as the insects prefer to drill into unfinished wood. You can also attach a mesh screen on top of the wood or use a carpenter bee trap.
If you can't seem to put a stop to the problem, you may want to call a pest exterminator. However, it is not recommended to use a pesticide yourself since the bees are helpful.