Expressions¶

An expression says how to compute a column: Col("qty").Mul(Col("price")). It holds no data, and it is computed only where a query puts it. The same expression gives a column per row in Select and WithColumns, one value per group in Agg, and keeps rows in Filter. These are Polars' contexts, and ursus follows them.

The same expression, three ways¶

ctx := context.Background()
revenue := ursus.Col("qty").Mul(ursus.Col("price"))

perRow, err := sales().
    Select(ursus.Col("order_id"), revenue.Alias("revenue")).
    Head(3).
    Collect(ctx)
if err != nil {
    log.Fatal(err)
}
total, err := sales().Select(revenue.Sum().Alias("revenue")).Collect(ctx)
if err != nil {
    log.Fatal(err)
}
byRegion, err := sales().GroupBy(ursus.Col("region")).
    Agg(revenue.Sum().Alias("revenue")).
    Sort(ursus.Asc(ursus.Col("region"))).Collect(ctx)
if err != nil {
    log.Fatal(err)
}
fmt.Println(perRow)
fmt.Println(total)
fmt.Println(byRegion)

Output

shape: (3, 2)
┌──────────┬─────────┐
│ order_id │ revenue │
│ Int64    │ Float64 │
├──────────┼─────────┤
│ 1        │ 15      │
│ 2        │ 6       │
│ 3        │ 15.75   │
└──────────┴─────────┘

shape: (1, 1)
┌─────────┐
│ revenue │
│ Float64 │
├─────────┤
│ 108.75  │
└─────────┘

shape: (3, 2)
┌─────────┬─────────┐
│ region  │ revenue │
│ String  │ Float64 │
├─────────┼─────────┤
│ "east"  │ 22.5    │
│ "north" │ 39.75   │
│ "south" │ 46.5    │
└─────────┴─────────┘

in a Select of row-wise expressions, revenue is a value per row;
its Sum() is an aggregate: in a Select it is one value for the frame;
in GroupBy(...).Agg, the same aggregate is one value per group.

GroupBy() with no keys also aggregates the whole frame, as one group.

Windows: an aggregate beside the rows¶

Over computes an expression within each group of rows and gives the answer back on every row of the group, so the frame keeps its height. A sum over a region beside each order, and each order's rank within its region:

df, err := sales().
    Select(
        ursus.Col("order_id"),
        ursus.Col("region"),
        ursus.Col("qty"),
        ursus.Col("qty").Sum().Over(ursus.Col("region")).Alias("region_qty"),
        ursus.Col("qty").Rank(ursus.RankDense, true).
            Over(ursus.Col("region")).Alias("rank"),
    ).
    Sort(ursus.Asc(ursus.Col("region")), ursus.Asc(ursus.Col("rank"))).
    Collect(context.Background())
if err != nil {
    log.Fatal(err)
}
fmt.Println(df)

Output

shape: (8, 5)
┌──────────┬─────────┬───────┬────────────┬────────┐
│ order_id │ region  │ qty   │ region_qty │ rank   │
│ Int64    │ String  │ Int64 │ Int128     │ Uint32 │
├──────────┼─────────┼───────┼────────────┼────────┤
│ 6        │ "east"  │ 12    │ 14         │ 1      │
│ 4        │ "east"  │ 2     │ 14         │ 2      │
│ 1        │ "north" │ 10    │ 20         │ 1      │
│ 3        │ "north" │ 7     │ 20         │ 2      │
│ 7        │ "north" │ 3     │ 20         │ 3      │
│ 5        │ "south" │ 9     │ 19         │ 1      │
│ 8        │ "south" │ 6     │ 19         │ 2      │
│ 2        │ "south" │ 4     │ 19         │ 3      │
└──────────┴─────────┴───────┴────────────┴────────┘

Window functions include the cumulative ones (CumSum, CumMax, ...), Shift, Diff, Rank, RollingMean and the other rolling windows by rows or by time, and the exponentially weighted EwmMean and its kin.

Many columns at once¶

Col takes several names, and All() and Exclude(...) select by what to leave out. An expression over several columns is one expression per column, and Suffix or Prefix names them:

df, err := sales().
    Select(
        ursus.Col("product"),
        ursus.Col("qty", "price").Mul(int64(2)).Suffix("_x2"),
    ).
    Head(3).
    Collect(context.Background())
if err != nil {
    log.Fatal(err)
}
fmt.Println(df)

Output

shape: (3, 3)
┌─────────┬────────┬──────────┐
│ product │ qty_x2 │ price_x2 │
│ String  │ Int64  │ Float64  │
├─────────┼────────┼──────────┤
│ "apple" │ 20     │ 3        │
│ "apple" │ 8      │ 3        │
│ "pear"  │ 14     │ 4.5      │
└─────────┴────────┴──────────┘

A selection inside a function across columns is that function's operands, as in Polars: SumHorizontal, MaxHorizontal and their kin, ConcatStr, Struct, ConcatList, Coalesce and HashRows answer one column over every column selected (since 0.6.7; before, the whole function was repeated a column at a time):

scores := ursus.Frame(
    ursus.Values("name", []string{"ada", "ben"}),
    ursus.Values("maths", []int64{90, 60}),
    ursus.Values("art", []int64{70, 85}),
    ursus.Values("music", []int64{80, 75}),
)
show(scores.Select(
    ursus.Col("name"),
    ursus.SumHorizontal(ursus.ColDType(ursus.Int64)).Alias("total"),
    ursus.MaxHorizontal(ursus.ColDType(ursus.Int64)).Alias("best"),
))

Output

shape: (2, 3)
┌────────┬────────┬───────┐
│ name   │ total  │ best  │
│ String │ Int128 │ Int64 │
├────────┼────────┼───────┤
│ "ada"  │ 240    │ 90    │
│ "ben"  │ 220    │ 85    │
└────────┴────────┴───────┘

Namespaces¶

Methods for one kind of column sit behind a namespace:

Namespace	For	For example
`Str()`	text	`Contains`, `Replace`, `Split`, `ToUpper`, `Strptime`, `Slice`
`Dt()`	dates and times	`Year`, `Weekday`, `Truncate`, `OffsetBy`, `ConvertTimeZone`
`List()`	lists	`Len`, `Get`, `Contains`, `Sum`, `Eval`, `Sort`, `Unique`
`Struct()`	structs	`Field`, `WithFields`, `RenameFields`, `JsonEncode`
`Bin()`	binary	`Contains`, `StartsWith`, `Encode`, `Size`
`Cat()`	enums	`GetCategories`, `LenBytes`, `Slice`

df, err := sales().
    Select(
        ursus.Col("product").Str().ToUpper().Alias("product"),
        ursus.Col("date").Dt().Weekday().Alias("weekday"),
        ursus.Col("product").Str().Contains("p", true).Alias("has_p"),
    ).
    Head(3).
    Collect(context.Background())
if err != nil {
    log.Fatal(err)
}
fmt.Println(df)

Output

shape: (3, 3)
┌─────────┬─────────┬───────┐
│ product │ weekday │ has_p │
│ String  │ Int32   │ Bool  │
├─────────┼─────────┼───────┤
│ "APPLE" │ 6       │ true  │
│ "APPLE" │ 6       │ true  │
│ "PEAR"  │ 7       │ true  │
└─────────┴─────────┴───────┘

Conditions¶

When(cond).Then(a).Otherwise(b) is a conditional column, and When may follow Then for further conditions, the first true one winning:

ursus.When(ursus.Col("qty").Ge(10)).Then("large").
    When(ursus.Col("qty").Ge(5)).Then("medium").
    Otherwise("small")

Comparisons give a Bool column, combined with And, Or and Not under SQL's three-valued logic: null.And(false) is false, and null.And(true) is null. A condition that is null counts as false, so its row falls to the next When, or to Otherwise, as in Polars and SQL's CASE WHEN (since 0.6.7; before, the row was null). Write Otherwise(ursus.Null(...)) where a null is wanted.

Literals¶

A Go value is a literal wherever an expression goes: Col("qty").Gt(3), Then("bulk"), FillNullWith(0.0). Lit(v) makes a literal expression of its own, and a Go slice is a List literal.

A literal has its Go value's type. A Go integer constant is an int, an Int64, and a float constant a Float64, so Col("a").Add(3) of an Int32 column is an Int64 column, and Col("f").Mul(2.5) of a Float32 one a Float64. Write Add(int32(3)) to stay at Int32. Polars' literals take the column's type instead. The fills are the exception: FillNullWith, FillNan and their kin meet the column, so FillNullWith(0) of an Int32 column stays Int32, and only a fill with a fraction widens an integer column.

Names¶

An expression is named after the leftmost column it reads: Col("qty").Mul(Col("price")) is named qty. Alias names it, and goes last, since an alias inside an expression does not name the whole.

New in 0.6.7: Meta() asks an expression about itself, as Polars' meta does: RootNames the columns it reads by name, OutputName its name, HasMultipleOutputs, IsColumn, IsColumnSelection, IsLiteral and IsRegexProjection; UndoAliases the expression without its names, Eq whether two are one computation, Pop the outermost operation's inputs, and TreeFormat a drawing of it.