Reshaping¶
Wide: pivot¶
Pivot spreads a column's values into columns of their own: a row for each distinct
Index, and a column for each distinct On value, holding Values. Agg reduces the
values that fall in one cell; without it, a cell of two values is refused.
ctx := context.Background()
df, err := sales().Collect(ctx)
if err != nil {
log.Fatal(err)
}
wide, err := df.Pivot(ctx, ursus.PivotOptions{
On: []string{"product"},
Index: []string{"region"},
Values: []string{"qty"},
Agg: ursus.Expr.Sum,
SortColumns: true,
})
if err != nil {
log.Fatal(err)
}
fmt.Println(wide)
shape: (3, 4)
┌─────────┬────────┬────────┬────────┐
│ region │ apple │ fig │ pear │
│ String │ Int128 │ Int128 │ Int128 │
├─────────┼────────┼────────┼────────┤
│ "north" │ 10 │ 3 │ 7 │
│ "south" │ 4 │ 9 │ 6 │
│ "east" │ 12 │ null │ 2 │
└─────────┴────────┴────────┴────────┘
A cell is Agg of the values that fall in it. East sold no figs, so its cell is the
sum of nothing: Polars' sum gives 0 there; here, as for every sum of nothing, it is
null (see Coming from Polars).
A count of nothing is 0, as Polars' is (fixed in 0.6.7; before, every such cell was
null). New columns come in the order their values first appear, or sorted with
SortColumns.
New in 0.6.7. Pivot is a LazyFrame's too, as Polars' LazyFrame.pivot is, its
new columns given in OnColumns, since a lazy plan knows its columns before it reads
the data. A value given that the data never holds is a column of empty cells, plum
here; one the data holds and OnColumns leaves out is left out, fig here. A
DataFrame's pivot finds them itself. Either way a group-by of the index and the
On values reduces the rows first, and spills past a memory limit as any group-by
does.
products := ursus.Values("product", []string{"apple", "pear", "plum"})
wide := sales().Pivot(ursus.PivotOptions{
On: []string{"product"},
OnColumns: []*ursus.Column{products},
Index: []string{"region"},
Values: []string{"qty"},
Agg: ursus.Expr.Sum,
})
show(wide)
shape: (3, 4)
┌─────────┬────────┬────────┬────────┐
│ region │ apple │ pear │ plum │
│ String │ Int128 │ Int128 │ Int128 │
├─────────┼────────┼────────┼────────┤
│ "north" │ 10 │ 7 │ null │
│ "south" │ 4 │ 6 │ null │
│ "east" │ 12 │ 2 │ null │
└─────────┴────────┴────────┴────────┘
New in 0.6.7. Several On columns name a new column by their values together, as
Polars writes them, text quoted:
ctx := context.Background()
df, err := ursus.Frame(
ursus.Values("shop", []string{"a", "a", "b", "b"}),
ursus.Values("quarter", []string{"q1", "q2", "q1", "q1"}),
ursus.Values("channel", []string{"web", "web", "store", "web"}),
ursus.Values("sales", []int64{10, 12, 7, 3}),
).Collect(ctx)
if err != nil {
log.Fatal(err)
}
wide, err := df.Pivot(ctx, ursus.PivotOptions{
On: []string{"quarter", "channel"},
Index: []string{"shop"},
Values: []string{"sales"},
})
if err != nil {
log.Fatal(err)
}
fmt.Println(wide)
shape: (2, 4)
┌────────┬──────────────┬──────────────┬────────────────┐
│ shop │ {"q1","web"} │ {"q2","web"} │ {"q1","store"} │
│ String │ Int64 │ Int64 │ Int64 │
├────────┼──────────────┼──────────────┼────────────────┤
│ "a" │ 10 │ 12 │ null │
│ "b" │ 3 │ null │ 7 │
└────────┴──────────────┴──────────────┴────────────────┘
Long: unpivot¶
Unpivot is the opposite: several columns become two, which column and what it held.
Index names the columns carried along; empty, it is every column not in On.
quarters := ursus.Frame(
ursus.Values("region", []string{"north", "south"}),
ursus.Values("q1", []int64{120, 80}),
ursus.Values("q2", []int64{135, 95}),
)
show(quarters.Unpivot(ursus.UnpivotOptions{
On: []string{"q1", "q2"},
Index: []string{"region"},
VariableName: "quarter",
ValueName: "sales",
}))
shape: (4, 3)
┌─────────┬─────────┬───────┐
│ region │ quarter │ sales │
│ String │ String │ Int64 │
├─────────┼─────────┼───────┤
│ "north" │ "q1" │ 120 │
│ "north" │ "q2" │ 135 │
│ "south" │ "q1" │ 80 │
│ "south" │ "q2" │ 95 │
└─────────┴─────────┴───────┘
New in 0.6.7. On may be left out instead: it is then every column not in Index,
as Polars' on=None, and every column when Index is left out too. The melted
columns share one value column, so their types must meet in one that holds each of
them exactly: an Int32 beside an Int64 melts into an Int64, but text beside a number
is refused, where Polars writes the numbers as text. Cast first.
quarters := ursus.Frame(
ursus.Values("region", []string{"north", "south"}),
ursus.Values("q1", []int64{120, 80}),
ursus.Values("q2", []int64{135, 95}),
ursus.Values("q3", []int32{150, 90}),
)
show(quarters.Unpivot(ursus.UnpivotOptions{Index: []string{"region"}}))
shape: (6, 3)
┌─────────┬──────────┬───────┐
│ region │ variable │ value │
│ String │ String │ Int64 │
├─────────┼──────────┼───────┤
│ "north" │ "q1" │ 120 │
│ "north" │ "q2" │ 135 │
│ "north" │ "q3" │ 150 │
│ "south" │ "q1" │ 80 │
│ "south" │ "q2" │ 95 │
│ "south" │ "q3" │ 90 │
└─────────┴──────────┴───────┘
The row order is not Polars'
Polars gives every row's first column, then every row's second. ursus gives each row's columns together, then the next row's, because it unpivots a batch at a time and its order must not depend on the batches. Sort afterwards when the order matters.
Lists into rows: explode¶
Explode gives each element of a list its own row, the other columns repeated:
posts := ursus.Frame(
ursus.Values("post", []int64{1, 2, 3}),
ursus.Values("tags", []string{"go,data", "go", ""}),
).WithColumns(ursus.Col("tags").Str().Split(","))
show(posts)
show(posts.Explode("tags"))
shape: (3, 2)
┌───────┬────────────────┐
│ post │ tags │
│ Int64 │ List(String) │
├───────┼────────────────┤
│ 1 │ ["go", "data"] │
│ 2 │ ["go"] │
│ 3 │ [""] │
└───────┴────────────────┘
shape: (4, 2)
┌───────┬────────┐
│ post │ tags │
│ Int64 │ String │
├───────┼────────┤
│ 1 │ "go" │
│ 1 │ "data" │
│ 2 │ "go" │
│ 3 │ "" │
└───────┴────────┘
Several columns explode together, zipped, so their lists must be the same length in each row. An empty list and a null list each become one row holding a null.
Structs into columns: unnest¶
Unnest turns a struct column into a column for each of its fields:
points := ursus.Frame(
ursus.Values("id", []int64{1, 2}),
ursus.Values("x", []float64{0.5, 2}),
ursus.Values("y", []float64{1, -1}),
).Select(
ursus.Col("id"),
ursus.Struct(ursus.Col("x"), ursus.Col("y")).Alias("point"),
)
show(points)
show(points.Unnest("point"))
shape: (2, 2)
┌───────┬────────────────────────────────┐
│ id │ point │
│ Int64 │ Struct(x: Float64, y: Float64) │
├───────┼────────────────────────────────┤
│ 1 │ {x: 0.5, y: 1} │
│ 2 │ {x: 2, y: -1} │
└───────┴────────────────────────────────┘
shape: (2, 3)
┌───────┬─────────┬─────────┐
│ id │ x │ y │
│ Int64 │ Float64 │ Float64 │
├───────┼─────────┼─────────┤
│ 1 │ 0.5 │ 1 │
│ 2 │ 2 │ -1 │
└───────┴─────────┴─────────┘
Concatenating¶
Concat stacks frames' rows. By default each frame must have the same columns in
the same order; an Int32 and an Int64 of one name meet at Int64.
january := ursus.Frame(
ursus.Values("day", []int64{30, 31}),
ursus.Values("qty", []int64{4, 6}),
)
february := ursus.Frame(
ursus.Values("day", []int64{1, 2}),
ursus.Values("qty", []int64{3, 8}),
)
show(ursus.Concat([]*ursus.LazyFrame{january, february}))
shape: (4, 2)
┌───────┬───────┐
│ day │ qty │
│ Int64 │ Int64 │
├───────┼───────┤
│ 30 │ 4 │
│ 31 │ 6 │
│ 1 │ 3 │
│ 2 │ 8 │
└───────┴───────┘
WithConcatMode(ConcatDiagonal) takes the union of the columns instead, each frame's
missing ones null:
old := ursus.Frame(
ursus.Values("id", []int64{1, 2}),
ursus.Values("name", []string{"Ada", "Ben"}),
)
recent := ursus.Frame(
ursus.Values("id", []int64{3}),
ursus.Values("email", []string{"cai@example.com"}),
)
show(ursus.Concat([]*ursus.LazyFrame{old, recent},
ursus.WithConcatMode(ursus.ConcatDiagonal)))
shape: (3, 3)
┌───────┬────────┬───────────────────┐
│ id │ name │ email │
│ Int64 │ String │ String │
├───────┼────────┼───────────────────┤
│ 1 │ "Ada" │ null │
│ 2 │ "Ben" │ null │
│ 3 │ null │ "cai@example.com" │
└───────┴────────┴───────────────────┘
New in 0.6.7. ConcatRelaxed and ConcatDiagonalRelaxed meet a column's types where
the default refuses them, at Polars' supertype: an Int64 beside a Float64 at
Float64, which may round; a number beside text at text. The default meets types only
where every value comes out as it went in.
counted := ursus.Frame(ursus.Values("qty", []int64{4, 6}))
weighed := ursus.Frame(ursus.Values("qty", []float64{2.5}))
show(ursus.Concat([]*ursus.LazyFrame{counted, weighed},
ursus.WithConcatMode(ursus.ConcatRelaxed)))
lf.Concat(others...) and VStack are the same for one frame. Concatenation streams:
the frames are read one after another.
Aligned by key¶
New in 0.6.7. ConcatAlign puts frames side by side by key, as Polars'
concat(how="align"): it joins them on the columns every frame has, a full join, and
sorts by those columns. ConcatAlignInner, ConcatAlignLeft and ConcatAlignRight
join so instead. A column in some frames but not every one would be two columns of
one name, and is refused; Polars keeps the first frame's.
prices := ursus.Frame(
ursus.Values("day", []int64{3, 1, 2}),
ursus.Values("close", []float64{10.5, 10.0, 11.0}),
)
volumes := ursus.Frame(
ursus.Values("day", []int64{2, 4, 1}),
ursus.Values("volume", []int64{700, 900, 500}),
)
show(ursus.Concat([]*ursus.LazyFrame{prices, volumes},
ursus.WithConcatMode(ursus.ConcatAlign)))
shape: (4, 3)
┌───────┬─────────┬────────┐
│ day │ close │ volume │
│ Int64 │ Float64 │ Int64 │
├───────┼─────────┼────────┤
│ 1 │ 10 │ 500 │
│ 2 │ 11 │ 700 │
│ 3 │ 10.5 │ null │
│ 4 │ null │ 900 │
└───────┴─────────┴────────┘
The joins spill past a memory limit, and the rows come out in the same order with one or without, ties in each frame's order.
Side by side¶
HStack places frames side by side, row i of each together, as
Concat with ConcatHorizontal does. The frames must be the same height and their
column names distinct. New in 0.6.7: ConcatHorizontalExtend pads a shorter frame's
columns with nulls instead, as Polars' horizontal_extend. Either reads its frames in
step, holding a batch of each.
names := ursus.Frame(ursus.Values("name", []string{"Ada", "Ben"}))
scores := ursus.Frame(ursus.Values("score", []int64{90, 75}))
show(names.HStack(scores))
shape: (2, 2)
┌────────┬───────┐
│ name │ score │
│ String │ Int64 │
├────────┼───────┤
│ "Ada" │ 90 │
│ "Ben" │ 75 │
└────────┴───────┘
Transpose¶
Transpose turns rows into columns. IncludeHeader keeps the old column names as a
column, and ColumnNames names the new ones. New in 0.6.7: ColumnNamesFrom takes
their names from a text column instead, which is left out, as Polars' column_names
given a column's name:
ctx := context.Background()
df, err := ursus.Frame(
ursus.Values("metric", []string{"orders", "revenue"}),
ursus.Values("north", []float64{3, 39.75}),
ursus.Values("south", []float64{3, 46.5}),
).Collect(ctx)
if err != nil {
log.Fatal(err)
}
t, err := df.Transpose(ctx, ursus.TransposeOptions{
IncludeHeader: true,
HeaderName: "region",
ColumnNamesFrom: "metric",
})
if err != nil {
log.Fatal(err)
}
fmt.Println(t)
shape: (2, 3)
┌─────────┬─────────┬─────────┐
│ region │ orders │ revenue │
│ String │ Float64 │ Float64 │
├─────────┼─────────┼─────────┤
│ "north" │ 3 │ 39.75 │
│ "south" │ 3 │ 46.5 │
└─────────┴─────────┴─────────┘
Every column of the answer has the type its values meet at, Polars' supertype, so
integers beside floats become floats and, new in 0.6.7, text beside numbers text,
where it was refused. It is a DataFrame's only: its columns are the frame's rows, so
its schema waits for the data.
Dummies¶
New in 0.6.7. ToDummies replaces a column with a Uint8 column for each of its
distinct values, 1 where a row holds the value and 0 where it does not, as Polars'
to_dummies. A null is a value of its own. Each dummy is named the column, a
separator, _ by default, and the value as Polars displays it, so 1.0 is 1.0 and
an instant 2024-01-02 03:04:05. A column's dummies stand in its place, ordered by
name as Polars orders them, so 10 comes before 2:
ctx := context.Background()
df, err := ursus.Frame(
ursus.Values("id", []int64{1, 2, 3, 4}),
ursus.ValuesNullable("size", []int64{10, 2, 0, 10},
[]bool{true, true, false, true}),
).Collect(ctx)
if err != nil {
log.Fatal(err)
}
d, err := df.ToDummies(ctx, ursus.DummiesOptions{Columns: []string{"size"}})
if err != nil {
log.Fatal(err)
}
fmt.Println(d)
shape: (4, 4)
┌───────┬─────────┬────────┬───────────┐
│ id │ size_10 │ size_2 │ size_null │
│ Int64 │ Uint8 │ Uint8 │ Uint8 │
├───────┼─────────┼────────┼───────────┤
│ 1 │ 1 │ 0 │ 0 │
│ 2 │ 0 │ 1 │ 0 │
│ 3 │ 0 │ 0 │ 1 │
│ 4 │ 1 │ 0 │ 0 │
└───────┴─────────┴────────┴───────────┘
DropFirst leaves out the first value to appear that is not null, which the others
imply, and DropNulls the null. Columns names the columns to encode, every column
when empty, and a Series has its own ToDummies. The values are numbered in one
pass, as a group-by numbers its groups, so a column of many distinct values costs no
more than one of few, beside the dummies it makes.
Unstack¶
New in 0.6.7. Unstack lays each column out as a grid without keys, as Polars'
unstack. Vertically, a column x's values go Step at a time down new columns
x_0, x_1 and on, Step rows high; with UnstackHorizontal, Step at a time
across Step new columns:
ctx := context.Background()
df, err := ursus.Frame(
ursus.Values("reading", []float64{1.5, 2, 2.5, 3, 3.5, 4, 4.5}),
).Collect(ctx)
if err != nil {
log.Fatal(err)
}
g, err := df.Unstack(ctx, ursus.UnstackOptions{Step: 3})
if err != nil {
log.Fatal(err)
}
fmt.Println(g)
shape: (3, 3)
┌───────────┬───────────┬───────────┐
│ reading_0 │ reading_1 │ reading_2 │
│ Float64 │ Float64 │ Float64 │
├───────────┼───────────┼───────────┤
│ 1.5 │ 3 │ 4.5 │
│ 2 │ 3.5 │ null │
│ 2.5 │ 4 │ null │
└───────────┴───────────┴───────────┘
The slots after the last value are null, or Fill, one value for every column or
one for each, cast to the column's type as ExtendConstant casts it. Each suffix is
zero-padded to the digits of the count of new columns, x_00 to x_12 for 13, as
Polars pads it. A vertical column shares the old one's values, a slice of them.
Split by key: partition¶
New in 0.6.7. PartitionBy splits a frame into a part for each distinct key, in
the order the keys first appear, as Polars' partition_by. Each part has its Key, a
one-row frame of the key's values, typed as the columns are, beside its Frame, the
rows holding that key. Go has no map keyed by a tuple, so the parts are a slice where
Polars' as_dict gives a dict. DropKey leaves the key columns out of each frame:
ctx := context.Background()
df, err := sales().Collect(ctx)
if err != nil {
log.Fatal(err)
}
parts, err := df.PartitionBy(ctx, ursus.PartitionOptions{
By: []string{"region"},
DropKey: true,
})
if err != nil {
log.Fatal(err)
}
for _, p := range parts {
region, _, _ := p.Key.At[string](0, "region")
fmt.Println(region, p.Frame.Height())
}
fmt.Println(parts[0].Frame)
north 3
south 3
east 2
shape: (3, 5)
┌──────────┬────────────┬─────────┬───────┬─────────┐
│ order_id │ date │ product │ qty │ price │
│ Int64 │ Date │ String │ Int64 │ Float64 │
├──────────┼────────────┼─────────┼───────┼─────────┤
│ 1 │ 2026-01-03 │ "apple" │ 10 │ 1.5 │
│ 3 │ 2026-01-04 │ "pear" │ 7 │ 2.25 │
│ 7 │ 2026-01-07 │ "fig" │ 3 │ 3 │
└──────────┴────────────┴─────────┴───────┴─────────┘
Keys group as a group-by groups them: a null key is a part of its own, and -0 and 0
are one key. It is a DataFrame's, and its parts together hold what the frame did.