Lists and structs

A List column holds a list of values a row, all of one type; a Struct column holds named fields a row. Both come from Parquet and Arrow as they are, and are built from other columns as below.

Building lists

Str().Split makes a list a row, as does any aggregation that gives several values, Implode among them:

func baskets() *ursus.LazyFrame {
    return ursus.Frame(
        ursus.Values("order", []int64{1, 2, 3}),
        ursus.Values("items", []string{"pear,apple,fig", "fig", "apple,pear"}),
        ursus.Values("prices", []string{"2.25,1.5,3", "3", "1.5,2.25"}),
    ).WithColumns(
        ursus.Col("items").Str().Split(","),
        ursus.Col("prices").Str().Split(",").
            List().Eval(ursus.Element().Cast(ursus.Float64)),
    )
}

show(baskets())
show(sales().
    GroupBy(ursus.Col("region")).
    Agg(ursus.Col("qty").Implode().Alias("qtys")).
    Sort(ursus.Asc(ursus.Col("region"))))

Output

shape: (3, 3)
┌───────┬──────────────────────────┬────────────────┐
│ order │ items                    │ prices         │
│ Int64 │ List(String)             │ List(Float64)  │
├───────┼──────────────────────────┼────────────────┤
│ 1     │ ["pear", "apple", "fig"] │ [2.25, 1.5, 3] │
│ 2     │ ["fig"]                  │ [3]            │
│ 3     │ ["apple", "pear"]        │ [1.5, 2.25]    │
└───────┴──────────────────────────┴────────────────┘

shape: (3, 2)
┌─────────┬─────────────┐
│ region  │ qtys        │
│ String  │ List(Int64) │
├─────────┼─────────────┤
│ "east"  │ [2, 12]     │
│ "north" │ [10, 7, 3]  │
│ "south" │ [4, 9, 6]   │
└─────────┴─────────────┘

ConcatList(exprs...) joins columns and lists into one list a row, and a Go slice is a List literal: ursus.Lit([]string{"apple", "pear"}).

Questions of each list

Methods for lists sit behind List(), and each answers once per row:

items, prices := ursus.Col("items").List(), ursus.Col("prices").List()
show(baskets().Select(
    ursus.Col("order"),
    items.Len().Alias("n"),
    items.Get(0).Alias("first"),
    items.Contains("fig").Alias("has_fig"),
    prices.Sum().Alias("total"),
    prices.Max().Alias("dearest"),
))

Output

shape: (3, 6)
┌───────┬────────┬─────────┬─────────┬─────────┬─────────┐
│ order │ n      │ first   │ has_fig │ total   │ dearest │
│ Int64 │ Uint32 │ String  │ Bool    │ Float64 │ Float64 │
├───────┼────────┼─────────┼─────────┼─────────┼─────────┤
│ 1     │ 3      │ "pear"  │ true    │ 6.75    │ 3       │
│ 2     │ 1      │ "fig"   │ true    │ 3       │ 3       │
│ 3     │ 2      │ "apple" │ false   │ 3.75    │ 2.25    │
└───────┴────────┴─────────┴─────────┴─────────┴─────────┘

• Len, Get(i), First and Last; Get is null past the end;
• Contains(v) and CountMatches(v);
• Sum, Mean, Min, Max, Median, Std, Var, Quantile and NUnique;
• Any and All for lists of Bool.

Transforming lists

Sort, Reverse, Unique, Head, Tail, Slice, Shift and Diff give a list back; Join(sep) joins a list of text into one string. Eval(expr) runs any expression over each list's elements, Element() naming the element:

items := ursus.Col("items").List()
show(baskets().Select(
    ursus.Col("order"),
    items.Sort().Alias("sorted"),
    items.Join(" + ").Alias("joined"),
    ursus.Col("prices").List().Eval(ursus.Element().Mul(2)).Alias("doubled"),
))

Output

shape: (3, 4)
┌───────┬──────────────────────────┬──────────────────────┬───────────────┐
│ order │ sorted                   │ joined               │ doubled       │
│ Int64 │ List(String)             │ String               │ List(Float64) │
├───────┼──────────────────────────┼──────────────────────┼───────────────┤
│ 1     │ ["apple", "fig", "pear"] │ "pear + apple + fig" │ [4.5, 3, 6]   │
│ 2     │ ["fig"]                  │ "fig"                │ [6]           │
│ 3     │ ["apple", "pear"]        │ "apple + pear"       │ [3, 4.5]      │
└───────┴──────────────────────────┴──────────────────────┴───────────────┘

Filter(pred) keeps the elements a predicate holds for, and in Eval a window or a rank is over the list's elements:

prices := ursus.Col("prices").List()
show(baskets().Select(
    ursus.Col("order"),
    prices.Filter(ursus.Element().Ge(2.0)).Alias("from_2"),
    prices.Eval(ursus.Element().Rank(ursus.RankDense, true)).Alias("rank"),
))

Output

shape: (3, 3)
┌───────┬───────────────┬──────────────┐
│ order │ from_2        │ rank         │
│ Int64 │ List(Float64) │ List(Uint32) │
├───────┼───────────────┼──────────────┤
│ 1     │ [2.25, 3]     │ [2, 3, 1]    │
│ 2     │ [3]           │ [1]          │
│ 3     │ [2.25]        │ [2, 1]       │
└───────┴───────────────┴──────────────┘

Set operations

SetIntersection, SetUnion, SetDifference and SetSymmetricDifference combine two lists a row, in the order elements first appear:

items := ursus.Col("items")
fruit := ursus.Lit([]string{"apple", "pear"})
show(baskets().Select(
    ursus.Col("order"),
    items.List().SetIntersection(fruit).Alias("in_both"),
    items.List().SetDifference(fruit).Alias("only_here"),
))

Output

shape: (3, 3)
┌───────┬───────────────────┬──────────────┐
│ order │ in_both           │ only_here    │
│ Int64 │ List(String)      │ List(String) │
├───────┼───────────────────┼──────────────┤
│ 1     │ ["pear", "apple"] │ ["fig"]      │
│ 2     │ []                │ ["fig"]      │
│ 3     │ ["apple", "pear"] │ []           │
└───────┴───────────────────┴──────────────┘

Back into rows

Explode gives every element its own row, several list columns zipped together; see Reshaping:

show(baskets().Explode("items", "prices"))

Output

shape: (6, 3)
┌───────┬─────────┬─────────┐
│ order │ items   │ prices  │
│ Int64 │ String  │ Float64 │
├───────┼─────────┼─────────┤
│ 1     │ "pear"  │ 2.25    │
│ 1     │ "apple" │ 1.5     │
│ 1     │ "fig"   │ 3       │
│ 2     │ "fig"   │ 3       │
│ 3     │ "apple" │ 1.5     │
│ 3     │ "pear"  │ 2.25    │
└───────┴─────────┴─────────┘

Arrays

New in 0.6.7. An Array(T, n) is a fixed-size list: each row holds n values of T, or is null. A Go array is one, [3]float64 an Array(Float64, 3), in FromRows and back in Rows; List().ToArray(n) makes one of a list whose every row holds n, as a cast to the type does, and refuses one that does not, as Polars does; Arr().ToList() is the list again:

points := ursus.FromRows([]point{
    {"a", [3]float64{1, 2, 2}},
    {"b", [3]float64{0, 3, 4}},
})
show(points.WithColumns(
    ursus.Col("XYZ").Arr().ToList().List().Sum().Alias("sum"),
    ursus.Col("XYZ").Arr().ToList().List().Head(2).List().ToArray(2).Alias("xy"),
))

Output

shape: (2, 4)
┌────────┬───────────────────┬─────────┬───────────────────┐
│ Name   │ XYZ               │ sum     │ xy                │
│ String │ Array(Float64, 3) │ Float64 │ Array(Float64, 2) │
├────────┼───────────────────┼─────────┼───────────────────┤
│ "a"    │ [1, 2, 2]         │ 5       │ [1, 2]            │
│ "b"    │ [0, 3, 4]         │ 7       │ [0, 3]            │
└────────┴───────────────────┴─────────┴───────────────────┘

Arr() is Polars' arr namespace, a list's functions of the same elements: Sum, Mean, Max, ArgMin, Get, Contains, Join, Unique, Head and the rest answer as List()'s do, a value or a list a row; Reverse, Sort, Shift and Eval of an element's value keep each array's length and so its Array; ToStruct makes a field of each element, field_0 on unless named, and Dot sums two arrays' products.

Reshape(-1, 3) makes Arrays of three of a column's values, as Polars' reshape, -1 the rows inferred, more dimensions an Array of Arrays and Reshape(-1) the values flattened; ConcatArr makes one Array of each row's Arrays and values.

An Array is read and written as Polars writes it: an Arrow FixedSizeList in IPC, and in Parquet a list whose size the file's Arrow schema keeps; as JSON, a list.

Structs

Struct(exprs...) packs columns into one struct column, a field for each, named as its expression:

show(sales().Head(3).Select(
    ursus.Col("order_id"),
    ursus.Struct(ursus.Col("product"), ursus.Col("qty")).Alias("line"),
))

Output

shape: (3, 2)
┌──────────┬─────────────────────────────────────┐
│ order_id │ line                                │
│ Int64    │ Struct(product: String, qty: Int64) │
├──────────┼─────────────────────────────────────┤
│ 1        │ {product: "apple", qty: 10}         │
│ 2        │ {product: "apple", qty: 4}          │
│ 3        │ {product: "pear", qty: 7}           │
└──────────┴─────────────────────────────────────┘

Methods for structs sit behind Struct():

line := ursus.Struct(ursus.Col("product"), ursus.Col("qty")).Alias("line")
l := ursus.Col("line").Struct()
lines := sales().Head(2).Select(line)
show(lines.Select(
    l.Field("product").Alias("product"),
    l.WithFields(l.Field("qty").Mul(10).Alias("qty")).Alias("scaled"),
))
show(lines.Select(l.JsonEncode().Alias("json")))

Output

shape: (2, 2)
┌─────────┬─────────────────────────────────────┐
│ product │ scaled                              │
│ String  │ Struct(product: String, qty: Int64) │
├─────────┼─────────────────────────────────────┤
│ "apple" │ {product: "apple", qty: 100}        │
│ "apple" │ {product: "apple", qty: 40}         │
└─────────┴─────────────────────────────────────┘

shape: (2, 1)
┌──────────────────────────────────────┐
│ json                                 │
│ String                               │
├──────────────────────────────────────┤
│ "{\"product\":\"apple\",\"qty\":10}" │
│ "{\"product\":\"apple\",\"qty\":4}"  │
└──────────────────────────────────────┘

• Field(name) reads one field, and Field of several names reads each as a column;
• WithFields(exprs...) replaces fields of the same names and adds new ones; read a field inside it through Field, as anywhere;
• RenameFields(names...) renames them in order, and new in 0.6.7, Name().PrefixFields, SuffixFields and MapFields rename each from its name;
• new in 0.6.7, Drop(names...) leaves fields out, as Polars' struct.drop, and refuses a name that is no field;
• JsonEncode writes each struct as JSON text, and Str().JsonDecode(type) reads it back;
• Unnest turns the fields into columns, as the frame's Unnest does.

A struct is also how ursus gives several values a row: SplitExact, ExtractGroups, ValueCounts of an expression, and Rle.

Lists and structs as keys

New in 0.6.7. A List, an Array or a Struct is a key, of GroupBy, Unique, Join, Over, PartitionBy, Mode, Rle and HashRows, as Polars keys one: element by element and field by field, an empty list apart from a null one, a null element equal to a null, NaN to NaN and -0.0 to 0.0. A list still has no order, so it is no sort key.

What is not here yet

Lists of lists or of structs are refused by name. The scope records each in feature-gaps.md.