Building a Limit Order Ingestion on Solana
TL; DR
In this post, we go through our journey to determine the best way to receive transaction intents to be sent to our protocol: a combination of durable nonces, PDAs, and a crafted transaction with the order data.
Challenge and constraints
What are our requirements for the first iteration of how Urani ingests limit orders for intents?
First, we knew that:
1️⃣ the implementation needs to give users the best experience, and
2️⃣ the protocol expects certain information to be submitted by these intents to the protocol and the agents.
Plus:
➡️ The user submits an order intent that can be fulfilled at a later point in time.
➡️ The protocol must store data related to the user's order intent in a specific format on a dedicated account for later use.
➡️ The program needs to call instructions on other programs to complete the order flow.
No single technique allows us to achieve all that; rather, it requires combining techniques, starting with a durable nonce.
In these examples, we'll use Typescript and Solana's web3.js. We won't dive into Rust programs today.
We note that, in many instances, much of the online documentation (including official documents) was outdated or incomplete. This post includes contributions that, to the best of our knowledge, are the most up-to-date.
The durable nonce
Suppose you sign a transaction that transfers 2 SOL from your account to your friend's account. What stops the program from repeatedly using this same signature to withdraw 2 SOL?
According to the Solana's official documentation:
The network could check all past transactions to see if any other transactions were already submitted with the same signature.However, as the number of transactions grows, that becomes increasingly costly (computation-wise) and more time-consuming to do.
This is when the concept of "latest blockhash" comes in:
Solution: Crosscheck signatures within only a set period of recent time, and discard the transaction if it gets "too" old.
Recent Blockhashes are used to achieve this. A blockhash contains a 32-bytes SHA-256 hash. It is used to indicate when a client last observed the ledger. Using recent blockhashes, transactions are checked in the last 150 blocks. If they are found, they are rejected. They are also rejected if they get older than 150 blocks. The only case they are accepted is if they are unique and the blockhash is more recent than 150 blocks (~80-90 seconds).
What if you need to submit your transaction later, for example, in a scheduled order?
Urani's agents can pick up limit orders and fulfill them immediately. However, depending on several factors (e.g., price, network congestion, agent concurrency), the order may only be fulfilled at a much later time.
Solana's solution for this use case has a name and a surname: durable nonces. They have a few key ingredients:
➡️ A nonce account
➡️ A nonce authority account to take control of its funds
➡️ A different way of assigning nonces to a transaction
➡️ The gotcha: the first instruction of the transaction
Let's break those down one by one.
Nonce account
Nonce accounts according to the Solana documentation:
The account that stores the value of the nonce. This account is owned by the SystemProgram and is rent-exempt; thus needs to maintain the minimum balance for rent exemption (around 0.0015 SOL).
Nonce authority account
And the piece on nonce authority account Solana documentation:
Nonce authority is the account that controls the Nonce Account. It has the authority to generate a new nonce, advance the nonce or withdraw SOL from the Nonce Account. By default, the account that creates the Nonce Account is delegated as the Nonce Authority, but it's possible to transfer the authority onto a keypair account or a PDA.
Creating the nonce account pair
This is how they translate to code:
A few things worth noticing in this transaction:
1️⃣ Both the nonce account and the nonce authority account are only regular keypairs created on the spot.
2️⃣ The nonce account has to be created with a certain amount of lamports to be rent-free and stay in the network. In this case, the user is paying for this fee.
3️⃣ createNonceAccount both create and initialize the nonce account. This process can be done in a few different ways, but this is the most straightforward of them.
4️⃣ The nonceAccount we retrieved at the end has the nonce authority account's public key and the nonce itself.
5️⃣ We had to get the latest blockhash and assign it to the transaction. We won't have to do that anymore from now on.
Assigning the nonce to a transaction
To be able to utilize the durable nonces in a transaction, we need a different format than when specifying the blockhash.
For the transaction to have them to be executed, their very first instruction has to be one that advances the nonce to the latest blockhash.
After that, we can line up other instructions to do whatever else we need. This can be done in the following way:
We're no longer using the recentBlockhash and lastValidBlockHeight fields in this transaction. Rather, we pass a nonceInfo object with the nonce taken from our nonceAccunt and this first instruction we just created. This will advance the nonce to the latest block.
When using the recent blockhash, a common issue is that the transaction takes so long to be confirmed, that the network goes beyond the 150 blocks threshold and the blockhash included in the transaction is no longer valid. But by using a durable nonce, we don't suffer from that issue.
PDAs
Program Derived Addresses are an essential block to building Solana applications. There are many interesting use cases and particularities of PDAs but, for this article, we'll focus on our own use case.
PDAs enable our program to do three important things:
➡️ Consistently find accounts tied to our program for each of our users.
➡️ Store program-relate state in the form of hashmaps.
➡️ Allow our program to call other programs in the Solana network.
Since the agents will be picking up orders and working on them, our program needs to be able to call other programs when needed. That is, it needs to perform a Cross-Program Invocation, aka CPI. PDAs are important to CPIs, as they allow the program to sign for an instruction to be executed on a second program on behalf of the PDA.
By leveraging this method of deterministically finding the PDA's address based on the same inputs, we don't need to keep track of specific account addresses. That's why PDAs aren't created; they are found based on the same inputs every time we need them.
Here we're using the method findProgramAddressSync to find a PDA with the user's publicKey and the string "orders" as seeds.
Note that we get two pieces of information here: the address of the PDA and a bump.
This bump is there because PDAs are not a regular keypair: there's only a public key for it, but not a corresponding private key.
The bump then is a number between 255 and 0 that is used to "force" the public key derived from the seeds (the first argument to findProgramAddressSync) and the program ID to fall off the Ed25519 cryptographic curve and not have a corresponding private key. To be more specific findProgramAddressSync tries to find the first number in that range (starting from 255) that forces the fall off the curve, and once it does, it returns this value.
Since the account derivation is deterministic, the bump will always be the same for the same inputs to the function. We also need to keep track of this bump for security reasons. Without the bump, the address is incomplete, and we don't know how to reproduce this PDA. By sending both to our program, it can independently verify the correctness of this address.
Crafting the final transaction
With this foundation in place, we can combine the resilience of durable nonces and the power of PDAs to craft a transaction that calls programs in the protocol in the exact way we need:
The Solana program will receive our JSON-stringified payload, with all the relevant order information, and we're passing all the relevant accounts (like the user's public key and the PDA) to the program in the keys field, which represent the AccountsMeta of a transaction.
And we're done! Now it's off to the Solana program to fulfill those orders.
Conclusion
The Solana ecosystem is vibrant, with many undiscovered pathways. Building on blockchains is a marathon, not a sprint, and we are ready for it. 🫡