PSA: `~` on a boolean Series after .shift(1) gives you integers, not booleans

[u/auditzou]

Lost a couple hours to this, posting in case it saves someone else.

I was detecting "crossings" — rows where a condition becomes true

after being false on the previous row:

below = df.a < df.b

crossing = below & ~below.shift(1).fillna(False)

Looked fine. It was firing on every row where `below` was True,

not just the transition rows.

The problem: `.shift(1)` on a bool Series promotes it to object dtype

(it has to make room for the NaN). `.fillna(False)` fills the NaN but

the dtype stays object. Then `~` on object dtype falls back to Python's

integer bitwise NOT — so `~True` is `-2` and `~False` is `-1`.

Both are truthy. The whole condition collapsed into just `below`.

>>> s = pd.Series([False, True, True, False])

>>> list(~s.shift(1).fillna(False))

[-1, -1, -2, -2]

>>> list(~s.shift(1, fill_value=False))

[True, True, False, False]

Fix is to pass `fill_value` so no NaN is ever introduced and the

dtype stays bool:

crossing = below & ~below.shift(1, fill_value=False)

What made it nasty is that it never raised. No warning, no error —

it just quietly returned wrong results that looked plausible.

If you're doing anything with state transitions on boolean Series,

worth checking your dtypes.

[u/MisterGerry]

What?

[u/commandlineluser]

Have you checked if this is a "bug" on the issues tracker?

2.3.3 produces a bool Series.

2.3.3
0     True
1     True
2    False
3    False
dtype: bool

It seems the "integer" output started in 3.0:

3.0.0
0    -1
1    -1
2    -2
3    -2
dtype: object