Is Distinct From Incorrectly Handles Masked Nulls

[tustvold]

Describe the bug

is_distinct_from is computed as

left_isnull != right_isnull || left_value != right_value

This is incorrect as regardless of if left_value != right_value if both left_isnull & right_isnull the result should be false.

IsNotDistinct has the same issue

This appears to have been introduced in #4560

To Reproduce

 #[test]
    fn is_distinct_from_nulls() -> Result<()> {
        // [0, 0, NULL, 0, 0, 0]
        let left_int_array = Int32Array::new(
            vec![0, 0, 1, 3, 0, 0].into(),
            Some(NullBuffer::from(vec![true, true, false, true, true, true])),
        );
        // [0, NULL, NULL, NULL, 0, NULL]
        let right_int_array = Int32Array::new(
            vec![0; 6].into(),
            Some(NullBuffer::from(vec![
                true, false, false, false, true, false,
            ])),
        );

        assert_eq!(
            BooleanArray::from(vec![
                Some(false),
                Some(true),
                Some(false),
                Some(true),
                Some(false),
                Some(true),
            ]),
            is_distinct_from(&left_int_array, &right_int_array)?
        );
    }

Expected behavior

No response

Additional context

apache/arrow-rs#960 tracks implementing these upstream

[comphead]

@tustvold I'll take it as it was my PR which brought up the bug

[tustvold]

There is already a fix upstream - apache/arrow-rs#4716

[comphead]

I see, so that means current is_distinct_* are supposed to be removed from DF?

[tustvold]

Correct, the upstream implementation should also be significantly faster

[comphead]

I would like to try to migrate DF to new implementation once arrow-rs released

[tustvold]

#7282 already has this completed - I like to verify things before cutting the release just in case 😄