KVStore interface. The problem with working with KVStore is that it forces you to think of state as a bytes KV pairings when in reality the majority of state comes from complex concrete golang objects (strings, ints, structs, etc.).
Collections allows you to work with state as if they were normal golang objects and removes the need for you to think of your state as raw bytes in your code.
It also allows you to migrate your existing state without causing any state breakage that forces you into tedious and complex chain state migrations.
Installation
To install collections in your cosmos-sdk chain project, run the following command:Core types
Collections offers 5 different APIs to work with state, which will be explored in the next sections, these APIs are:Map: to work with typed arbitrary KV pairings.KeySet: to work with just typed keysItem: to work with just one typed valueSequence: which is a monotonically increasing number.IndexedMap: which combinesMapandKeySetto provide aMapwith indexing capabilities.
Preliminary components
Before exploring the different collections types and their capability it is necessary to introduce the three components that every collection shares. In fact when instantiating a collection type by doing, for example,collections.NewMap/collections.NewItem/... you will find yourself having to pass them some common arguments.
For example, in code:
SchemaBuilder
The first argument passed is theSchemaBuilder
SchemaBuilder is a structure that keeps track of all the state of a module, it is not required by the collections to deal with state but it offers a dynamic and reflective way for clients to explore a module’s state.
We instantiate a SchemaBuilder by passing it a function that given the modules store key returns the module’s specific store.
We then need to pass the schema builder to every collection type we instantiate in our keeper, in our case the AllowList.
Prefix
The second argument passed to ourKeySet is a collections.Prefix, a prefix represents a partition of the module’s KVStore where all the state of a specific collection will be saved.
Since a module can have multiple collections, the following is expected:
- module params will become a
collections.Item - the
AllowListis acollections.KeySet
types/keys.go file, example: https://github.com/cosmos/cosmos-sdk/blob/main/x/feegrant/key.go#L27
your old:
Rules
collections.NewPrefix accepts either uint8, string or []bytes it’s good practice to use an always increasing uint8for disk space efficiency.
A collection MUST NOT share the same prefix as another collection in the same module, and a collection prefix MUST NEVER start with the same prefix as another, examples:
Human-Readable Name
The third parameter we pass to a collection is a string, which is a human-readable name. It is needed to make the role of a collection understandable by clients who have no clue about what a module is storing in state.Rules
Each collection in a module MUST have a unique humanised name.Key and Value Codecs
A collection is generic over the type you can use as keys or values. This makes collections dumb, but also means that hypothetically we can store everything that can be a go type into a collection. We are not bounded to any type of encoding (be it proto, json or whatever) So a collection needs to be given a way to understand how to convert your keys and values to bytes. This is achieved throughKeyCodec and ValueCodec, which are arguments that you pass to your collections when you’re instantiating them using the collections.NewMap/collections.NewItem/... instantiation functions.
NOTE: Generally speaking you will never be required to implement your own Key/ValueCodec as the SDK and collections libraries already come with default, safe and fast implementation of those. You might need to implement them only if you’re migrating to collections and there are state layout incompatibilities.
Let’s explore an example:
uint64. We already know the first three arguments of the NewMap function.
The fourth parameter is our KeyCodec, we know that the Map has string as key so we pass it a KeyCodec that handles strings as keys.
The fifth parameter is our ValueCodec, we know that the Map as a uint64 as value so we pass it a ValueCodec that handles uint64.
Collections already comes with all the required implementations for golang primitive types.
Let’s make another example, this falls closer to what we build using cosmos SDK, let’s say we want to create a collections.Map that maps account addresses to their base account. So we want to map an sdk.AccAddress to an auth.BaseAccount (which is a proto):
collections.Map maps sdk.AccAddress to authtypes.BaseAccount, we use the sdk.AccAddressKey which is the KeyCodec implementation for AccAddress and we use codec.CollValue to encode our proto type BaseAccount.
Generally speaking you will always find the respective key and value codecs for types in the go.mod path you’re using to import that type. If you want to encode proto values refer to the codec codec.CollValue function, which allows you to encode any type implement the proto.Message interface.
Map
We analyse the first and most important collection type, thecollections.Map. This is the type that everything else builds on top of.
Use case
Acollections.Map is used to map arbitrary keys with arbitrary values.
Example
It’s easier to explain acollections.Map capabilities through an example:
Set method
Set maps with the providedAccAddress (the key) to the auth.BaseAccount (the value).
Under the hood the collections.Map will convert the key and value to bytes using the key and value codec. It will prepend to our bytes key the prefix and store it in the KVStore of the module.
Has method
The has method reports if the provided key exists in the store.Get method
The get method accepts theAccAddress and returns the associated auth.BaseAccount if it exists, otherwise it errors.
Remove method
The remove method accepts theAccAddress and removes it from the store. It won’t report errors if it does not exist, to check for existence before removal use the Has method.
Iteration
Iteration has a separate section.KeySet
The second type of collection iscollections.KeySet, as the word suggests it maintains only a set of keys without values.
Implementation curiosity
Acollections.KeySet is just a collections.Map with a key but no value. The value internally is always the same and is represented as an empty byte slice []byte{}.
Example
As always we explore the collection type through an example:KeySet needs use to specify only one type parameter: the key (sdk.ValAddress in this case). The second difference we notice is that KeySet in its NewKeySet function does not require us to specify a ValueCodec but only a KeyCodec. This is because a KeySet only saves keys and not values.
Let’s explore the methods.
Has method
Has allows us to understand if a key is present in thecollections.KeySet or not, functions in the same way as collections.Map.Has
Set method
Set inserts the provided key in theKeySet.
Remove method
Remove removes the provided key from theKeySet, it does not error if the key does not exist, if existence check before removal is required it needs to be coupled with the Has method.
Item
The third type of collection is thecollections.Item. It stores only one single item, it’s useful for example for parameters, there’s only one instance of parameters in state always.
implementation curiosity
Acollections.Item is just a collections.Map with no key but just a value. The key is the prefix of the collection!
Example
KeyCodec, since we store only one item we already know the key and the fact that it is constant.
Iteration
One of the key features of theKVStore is iterating over keys.
Collections which deal with keys (so Map, KeySet and IndexedMap) allow you to iterate over keys in a safe and typed way. They all share the same API, the only difference being that KeySet returns a different type of Iterator because KeySet only deals with keys.
Every collection shares the same
Iterator semantics.Map.Iterate method:
collections.Ranger[K], which is an API that instructs map on how to iterate over keys. As always we don’t need to implement anything here as collections already provides some generic Ranger implementers that expose all you need to work with ranges.
Example
We have acollections.Map that maps accounts using uint64 IDs.
Iterate and the returned Iterator API.
GetAllAccounts
InGetAllAccounts we pass to our Iterate a nil Ranger. This means that the returned Iterator will include all the existing keys within the collection.
Then we use some the Values method from the returned Iterator API to collect all the values into a slice.
Iterator offers other methods such as Keys() to collect only the keys and not the values and KeyValues to collect all the keys and values.
IterateAccountsBetween
Here we make use of thecollections.Range helper to specialise our range. We make it start in a point through StartInclusive and end in the other with EndExclusive, then we instruct it to report us results in reverse order through Descending
Then we pass the range instruction to Iterate and get an Iterator, which will contain only the results we specified in the range.
Then we use again th Values method of the Iterator to collect all the results.
collections.Range also offers a Prefix API which is not appliable to all keys types, for example uint64 cannot be prefix because it is of constant size, but a string key can be prefixed.
IterateAccounts
Here we showcase how to lazily collect values from an Iterator.Keys/Values/KeyValues fully consume and close the Iterator, here we need to explicitly do a defer iterator.Close() call.Iterator also exposes a Value and Key method to collect only the current value or key, if collecting both is not needed.
For this
callback pattern, collections expose a Walk API.Composite keys
So far we’ve worked only with simple keys, likeuint64, the account address, etc. There are some more complex cases in, which we need to deal with composite keys.
A key is composite when it is composed of multiple keys, for example bank balances as stored as the composite key (AccAddress, string) where the first part is the address holding the coins and the second part is the denom.
Example, let’s say address BOB holds 10atom,15osmo, this is how it is stored in state:
getting (address, denom), or getting all the balances of an address by prefixing over (address).
Let’s see now how we can work with composite keys using collections.
Example
In our example we will show-case how we can use collections when we are dealing with balances, similar to bank, a balance is a mapping between(address, denom) => math.Int the composite key in our case is (address, denom).
Instantiation of a composite key collection
The Map Key definition
First of all we can see that in order to define a composite key of two elements we use thecollections.Pair type:
collections.Pair defines a key composed of two other keys, in our case the first part is sdk.AccAddress, the second part is string.
The Key Codec instantiation
The arguments to instantiate are always the same, the only thing that changes is how we instantiate theKeyCodec, since this key is composed of two keys we use collections.PairKeyCodec, which generates a KeyCodec composed of two key codecs. The first one will encode the first part of the key, the second one will encode the second part of the key.
Working with composite key collections
Let’s expand on the example we used before:SetBalance
As we can see here we’re setting the balance of an address for a specific denom. We use thecollections.Join function to generate the composite key. collections.Join returns a collections.Pair (which is the key of our collections.Map)
collections.Pair contains the two keys we have joined, it also exposes two methods: K1 to fetch the 1st part of the key and K2 to fetch the second part.
As always, we use the collections.Map.Set method to map the composite key to our value (math.Intin this case)
GetBalance
To get a value in composite key collection, we simply usecollections.Join to compose the key.
GetAllAddressBalances
We usecollections.PrefixedPairRange to iterate over all the keys starting with the provided address. Concretely the iteration will report all the balances belonging to the provided address.
The first part is that we instantiate a PrefixedPairRange, which is a Ranger implementer aimed to help in Pair keys iterations.
collections.Pair because golang type inference with respect to generics is not as permissive as other languages, so we need to explitly say what are the types of the pair key.
GetAllAddressesBalancesBetween
This showcases how we can further specialise our range to limit the results further, by specifying the range between the second part of the key (in our case the denoms, which are strings).IndexedMap
collections.IndexedMap is a collection that uses under the hood a collections.Map, and has a struct, which contains the indexes that we need to define.
Example
Let’s say we have anauth.BaseAccount struct which looks like the following:
sdk.AccAddress. If it were to be a collections.Map it would be collections.Map[sdk.AccAddres, authtypes.BaseAccount].
Then we also want to be able to get an account not only by its sdk.AccAddress, but also by its AccountNumber.
So we can say we want to create an Index that maps our BaseAccount to its AccountNumber.
We also know that this Index is unique. Unique means that there can only be one BaseAccount that maps to a specific AccountNumber.
First of all, we start by defining the object that contains our index:
AccountIndexes struct which contains a field: Number. This field represents our AccountNumber index. AccountNumber is a field of authtypes.BaseAccount and it’s a uint64.
Then we can see in our AccountIndexes struct the Number field is defined as:
uint64, which is the field type of our index. The second type parameter is the primary key sdk.AccAddress And the third type parameter is the actual object we’re storing authtypes.BaseAccount.
Then we implement a function called IndexesList on our AccountIndexes struct, this will be used by the IndexedMap to keep the underlying map in sync with the indexes, in our case Number. This function just needs to return the slice of indexes contained in the struct.
Then we create a NewAccountIndexes function that instantiates and returns the AccountsIndexes struct.
The function takes a SchemaBuilder. Then we instantiate our indexes.Unique, let’s analyse the arguments we pass to indexes.NewUnique.
Instantiating a indexes.Unique
The first three arguments, we already know them, they are: SchemaBuilder, Prefix which is our index prefix (the partition where index keys relationship for the Number index will be maintained), and the human name for the Number index.
The second argument is a collections.Uint64Key which is a key codec to deal with uint64 keys, we pass that because the key we’re trying to index is a uint64 key (the account number), and then we pass as fifth argument the primary key codec, which in our case is sdk.AccAddress (remember: we’re mapping sdk.AccAddress => BaseAccount).
Then as last parameter we pass a function that: given the BaseAccount returns its AccountNumber.
After this we can proceed instantiating our IndexedMap.
collections.Map. We pass it the SchemaBuilder, the Prefix where we plan to store the mapping between sdk.AccAddress and authtypes.BaseAccount, the human name and the respective sdk.AccAddress key codec and authtypes.BaseAccount value codec.
Then we pass the instantiation of our AccountIndexes through NewAccountIndexes.
Full example:
Working with IndexedMaps
Whilst instantiatingcollections.IndexedMap is tedious, working with them is extremely smooth.
Let’s take the full example, and expand it with some use-cases.
Collections with interfaces as values
Although cosmos-sdk is shifting away from the usage of interface registry, there are still some places where it is used. In order to support old code, we have to support collections with interface values. The genericcodec.CollValue is not able to handle interface values, so we need to use a special type codec.CollValueInterface. codec.CollValueInterface takes a codec.BinaryCodec as an argument, and uses it to marshal and unmarshal values as interfaces. The codec.CollValueInterface lives in the codec package, whose import path is github.com/cosmos/cosmos-sdk/codec.
Instantiating Collections with interface values
In order to instantiate a collection with interface values, we need to usecodec.CollValueInterface instead of codec.CollValue.