The translation given by the WordReference dictionary seems to be correct. Baptistère is a noun which is defined as ‘une chapelle où l’on baptise’, in other words ‘la chapelle où se trouvent les fonts baptismaux’.
The English word ‘baptismal’, on the other hand, is an adjective. It has its own entry in the WordReference dictionary, and is defined as ‘relating to baptism’. The French translation of the adjective given by the WordReference dictionary is ‘baptismal’.
Thank you for your message. As Upswing has already pointed out, baptistery is the correct translation. If you are looking for advice on translations, you should post in our French-English Vocabulary forum. This forum is for reporting errors or omissions in the dictionary or suggesting improvements to it. That might lead to changes being made to the dictionary, but those changes will take some time to appear online. You will get much more immediate help in the vocabulary forums.
I'm more accustomed to the spelling baptistry, which appears in the entry in the opposite direction, if you wouldn't mind adding that while we're at it?