Why do custom QoS rules only match bypassed traffic?

Non-bypassed traffic is handled differently because it goes through the Untangle network stack, not the normal Linux packet flow. You can use the Bandwidth Control application to assign priorities to non-bypassed traffic.