Missing data¶
A missing value is a null, in a column of any type. It is not NaN, a float that is not a number, and ursus keeps the two apart everywhere: see Types and nulls. The examples read six hours of temperatures with gaps, and a backup reading for some of them:
func readings() *ursus.LazyFrame {
ok := []bool{true, false, false, true, true, false}
return ursus.Frame(
ursus.Values("hour", []int64{0, 1, 2, 3, 4, 5}),
ursus.ValuesNullable("temp", []float64{10, 0, 0, 13, 12, 0}, ok),
ursus.ValuesNullable("backup", []float64{9, 11, 0, 0, 0, 14},
[]bool{true, true, false, false, false, true}),
)
}
Counting¶
NullCount counts a column's nulls, and over All() every column's:
shape: (1, 3)
┌────────┬────────┬────────┐
│ hour │ temp │ backup │
│ Uint64 │ Uint64 │ Uint64 │
├────────┼────────┼────────┤
│ 0 │ 3 │ 3 │
└────────┴────────┴────────┘
IsNull and IsNotNull test each value, for a Filter or a When.
Filling with a value¶
FillNullWith(v) fills with a value, FillNull(strategy) with one the column gives:
FillMean, FillMin, FillMax, FillZero or FillOne. Coalesce(exprs...) takes
each row's first value that is not null, here the temperature, else the backup, else
−1:
temp := ursus.Col("temp")
show(readings().Select(
ursus.Col("hour"),
temp,
temp.FillNullWith(0.0).Alias("zero"),
temp.FillNull(ursus.FillMean).Alias("mean"),
ursus.Coalesce(temp, ursus.Col("backup"), ursus.Lit(-1.0)).Alias("coalesced"),
))
shape: (6, 5)
┌───────┬─────────┬─────────┬────────────────────┬───────────┐
│ hour │ temp │ zero │ mean │ coalesced │
│ Int64 │ Float64 │ Float64 │ Float64 │ Float64 │
├───────┼─────────┼─────────┼────────────────────┼───────────┤
│ 0 │ 10 │ 10 │ 10 │ 10 │
│ 1 │ null │ 0 │ 11.666666666666666 │ 11 │
│ 2 │ null │ 0 │ 11.666666666666666 │ -1 │
│ 3 │ 13 │ 13 │ 13 │ 13 │
│ 4 │ 12 │ 12 │ 12 │ 12 │
│ 5 │ null │ 0 │ 11.666666666666666 │ 14 │
└───────┴─────────┴─────────┴────────────────────┴───────────┘
A fill meets the column's type: FillNullWith(0) of an Int32 column stays Int32,
and so does FillNullWith(2.0), a whole number. A fill with a fraction, 0.5, widens
an integer column to Float64. Polars widens it for any float.
Filling from the neighbours¶
FillForward carries the last value forward and FillBackward the next one back;
ForwardFill(limit) and BackwardFill(limit) fill at most that many rows of a gap.
Interpolate draws a straight line across a gap, and leaves a gap at either end as it
is:
temp := ursus.Col("temp")
show(readings().Select(
ursus.Col("hour"),
temp,
temp.FillNull(ursus.FillForward).Alias("forward"),
temp.ForwardFill(1).Alias("forward_1"),
temp.FillNull(ursus.FillBackward).Alias("backward"),
temp.Interpolate().Alias("interpolated"),
))
shape: (6, 6)
┌───────┬─────────┬─────────┬───────────┬──────────┬──────────────┐
│ hour │ temp │ forward │ forward_1 │ backward │ interpolated │
│ Int64 │ Float64 │ Float64 │ Float64 │ Float64 │ Float64 │
├───────┼─────────┼─────────┼───────────┼──────────┼──────────────┤
│ 0 │ 10 │ 10 │ 10 │ 10 │ 10 │
│ 1 │ null │ 10 │ 10 │ 13 │ 11 │
│ 2 │ null │ 10 │ null │ 13 │ 12 │
│ 3 │ 13 │ 13 │ 13 │ 13 │ 13 │
│ 4 │ 12 │ 12 │ 12 │ 12 │ 12 │
│ 5 │ null │ 12 │ 12 │ null │ null │
└───────┴─────────┴─────────┴───────────┴──────────┴──────────────┘
They run in the rows' order; under Over(keys...), within each partition, so a
sensor's gap is never filled from another sensor's readings.
New in 0.6.7. InterpolateBy(by) draws the line at each row's by, a time or a
number, where Interpolate steps by rows, as Polars' interpolate_by: readings at
hours 0, 1 and 4, the middle one missing, fill it a quarter of the way along. by
need not be sorted, and a null in it is refused, as Polars refuses it.
New in 0.6.7. A strategy takes a limit too, FillForward.Limit(n) and
FillBackward.Limit(n), as Polars' fill_null(strategy=..., limit=n), for a column or
for every column of a frame at once:
shape: (6, 3)
┌───────┬─────────┬─────────┐
│ hour │ temp │ backup │
│ Int64 │ Float64 │ Float64 │
├───────┼─────────┼─────────┤
│ 0 │ 10 │ 9 │
│ 1 │ 10 │ 11 │
│ 2 │ null │ 11 │
│ 3 │ 13 │ null │
│ 4 │ 12 │ null │
│ 5 │ 12 │ 14 │
└───────┴─────────┴─────────┘
A limit is at least 1; leave it off for none. A limit on another strategy is refused, as Polars refuses it.
Dropping¶
DropNulls(names...) drops the rows with a null in any of those columns, or in any
column at all:
shape: (3, 3)
┌───────┬─────────┬─────────┐
│ hour │ temp │ backup │
│ Int64 │ Float64 │ Float64 │
├───────┼─────────┼─────────┤
│ 0 │ 10 │ 9 │
│ 3 │ 13 │ null │
│ 4 │ 12 │ null │
└───────┴─────────┴─────────┘
shape: (1, 3)
┌───────┬─────────┬─────────┐
│ hour │ temp │ backup │
│ Int64 │ Float64 │ Float64 │
├───────┼─────────┼─────────┤
│ 0 │ 10 │ 9 │
└───────┴─────────┴─────────┘
New in 0.6.7. DropNans(names...) drops the rows with a NaN in any of those
columns, or in any float column. A null is not a NaN, and its row stays:
nan := math.NaN()
present := []bool{true, true, false, true} // the third is null
ratios := ursus.Frame(
ursus.Values("id", []int64{1, 2, 3, 4}),
ursus.ValuesNullable("ratio", []float64{0.5, nan, 0, 2}, present),
)
show(ratios.DropNans())
shape: (3, 2)
┌───────┬─────────┐
│ id │ ratio │
│ Int64 │ Float64 │
├───────┼─────────┤
│ 1 │ 0.5 │
│ 3 │ null │
│ 4 │ 2 │
└───────┴─────────┘
The whole frame¶
New in 0.6.7. A frame's Interpolate() interpolates every column it can: numbers,
an integer becoming a Float64, and dates and times, which keep their type, as
Polars' interpolate does. Text and Booleans are left as they are:
day := func(d int) time.Time { return time.Date(2026, 1, d, 0, 0, 0, 0, time.UTC) }
ends := []bool{true, false, false, true} // two rows missing between
gappy := ursus.Frame(
ursus.ValuesNullable("day", []time.Time{day(1), {}, {}, day(7)}, ends),
ursus.ValuesNullable("level", []int64{10, 0, 0, 40}, ends),
ursus.ValuesNullable("note", []string{"start", "", "", "end"}, ends),
).WithColumns(ursus.Col("day").Cast(ursus.Date))
show(gappy.Interpolate())
shape: (4, 3)
┌────────────┬─────────┬─────────┐
│ day │ level │ note │
│ Date │ Float64 │ String │
├────────────┼─────────┼─────────┤
│ 2026-01-01 │ 10 │ "start" │
│ 2026-01-03 │ 20 │ null │
│ 2026-01-05 │ 30 │ null │
│ 2026-01-07 │ 40 │ "end" │
└────────────┴─────────┴─────────┘
A frame's FillNull(v) fills every column that can hold a null and that v can
fill, chosen from the schema before any data is read. hour, built by Values, cannot
hold a null, and is left as it is.
shape: (6, 3)
┌───────┬─────────┬─────────┐
│ hour │ temp │ backup │
│ Int64 │ Float64 │ Float64 │
├───────┼─────────┼─────────┤
│ 0 │ 10 │ 9 │
│ 1 │ 0 │ 11 │
│ 2 │ 0 │ 0 │
│ 3 │ 13 │ 0 │
│ 4 │ 12 │ 0 │
│ 5 │ 0 │ 14 │
└───────┴─────────┴─────────┘
How nulls behave¶
- Arithmetic with a null is null, and so is a comparison.
- Logic is three-valued, as SQL's:
null.Or(true)is true,null.And(false)false. - A filter keeps the rows where its predicate is true, so a null drops a row.
- An aggregate skips nulls; a group of nulls alone sums to null, not 0.
- A group-by makes the nulls of a key one group, and a join matches no null key,
unless
JoinNullsEqual(true). - A sort puts nulls first, unless a key asks for
NullsLast().