Netfilter can by default not track replies to multicast(/broadcast for IPv4) messages. Thus they are, unless they are explicitely allowed through a stateless rule, not matched by a rule using the conntrack match module. To work around this limitation, I developed a equivalent workaround using ipset and iptables. Theoretically the same can be done with nftables easily. But it has to be done manually. Unless that is done, multicast and broadcast communication is severely limited.
ipset definitions for ipset.conf:create track-multicast4 hash:net,port family inet timeout 15create track-multicast6 hash:net,port family inet6 timeout 15
iptables rules:-A INPUT -m set --match-set track-multicast4 dst,dst -j ACCEPT
-A OUTPUT -m addrtype --dst-type BROADCAST -j SET --add-set track-multicast4 src,src
-A OUTPUT -m addrtype --dst-type MULTICAST -j SET --add-set track-multicast4 src,src
ip6tables rules:-A INPUT -m set --match-set track-multicast6 dst,dst -j ACCEPT-A OUTPUT -m addrtype --dst-type MULTICAST -j SET --add-set track-multicast6 src,src