Introducing Challenge #1: Decentralized Credit Scoring in Web3

While traditional finance relies on credit scores to gauge the risk of default, decentralized finance (DeFi) has been largely dependent on over-collateralization (i.e. the equivalent of a secured credit card). An effective on-chain credit score would improve capital efficiency and create a more inclusive, efficient DeFi ecosystem that could one day surpass traditional financial institutions.

In August 2022, we released our first Multi-Asset Credit Risk Oracle (MACRO) Score. The most recent version of the model used over 150 features, and was created from analyzing eight years of Ethereum transactions and over 2.5 million borrowing events on Ethereum, Polygon and Avalanche. In May 2023, we backtested the MACRO Score against a year’s worth of borrowing events on the on-chain lending protocol Aave and showed the Score was capable of improving capital efficiency. 

We’re asking you to develop an entirely new credit scoring model, and providing the dataset that we developed for the MACRO Score as your starting point. Building an on-chain credit score means wielding complex on-chain data and preparing a model that’s ready to provide inferences from day one. In exchange, we’re offering the top ten modelers who hit our performance benchmarks (listed below) a bounty of $100,000 and an 85% share of the revenue collected from the model after the competition ends. 

Challenge Details

Your goal is to predict liquidation (as per binary classification) of an active borrower on Aave v2 Ethereum and Compound v2 Ethereum. Liquidation here includes both:

• Actual liquidation, when a borrower’s health factor drops below the liquidation threshold, triggering a liquidation event; AND
• Technical liquidation, when a borrower’s health factor drops below a specific threshold.

In essence, we’re asking for a prediction that can adequately account for a borrower’s risk when their health factor deteriorates below healthy thresholds or their borrowing behavior leaves them vulnerable to volatility, even when they haven’t yet been liquidated.

The target label therefore is a hybrid target and is equal to 1 where:

• The borrower was liquidated within 30 days from the time of the borrow event, OR
• The borrower’s health factor dropped below a specific threshold (1.2) within 30 days from the time of the borrow event.

Output for Submission: Your model’s predictions must include both the predicted probability of liquidation and the predicted label (liquidated or not liquidated—as per the above definition of liquidation).  Your model will be benchmarked against the 7 evaluation metrics referenced below. 

Spectral’s Unique Dataset

Spectral spent three years developing a sophisticated on-chain creditworthiness assessment model, which required crunching an enormous number of Ethereum, Polygon and Avalanche transactions, separating out DeFi transactions, unraveling which ones were borrowing events and then extrapolating over 150 features from the data. We’re proud of the model we created, we’ve validated it against real data in a capital efficiency simulation, and consider it—and the dataset we created—second-to-none. We’re giving a substantial portion of it to you for this competition.

Modelers will be able to fetch this data using our Spectral SDK once the platform goes live. The basic dataset will consist of borrowing events, arrayed as rows in a table, and several dozen features as columns. These will range from obvious features such as a borrower’s balance in ETH to more advanced features such as health metrics.

DeFi Features

One broad category of features available will be DeFi features. These provide insights into a user's financial behavior and risk profile. We can extract valuable information about a user's creditworthiness by analyzing DeFi events, such as borrowing, lending, and liquidations. This information can then be used to improve the accuracy and reliability of credit scoring models.

Wallet Features and Data Engineering

There are also wallet features, which include details pertaining to the contents and history of a borrower’s wallet. If you’re not familiar with Web3, Ethereum users execute transactions on the Ethereum network using pseudonymous accounts called wallets. Wallet features important to credit include: historical balances, the current ETH balance, the number of transactions made, a wallet’s age, and the presence of various highly volatile tokens.

To engineer these features, we rely on sources such as Messari subgraphs on The Graph’s decentralized network, Erigon, Reth, Cryo, and Transpose.

Specifications 

To provide zero knowledge machine learning (zkML) proofs, which you can read about here, we support the following types of machine learning (ML) models for modeling during this initial product release:

• Neural networks using PyTorch (other than recurrent neural networks, e.g., LSTM)
• Logistic regression using PyTorch

Once the competition formally opens, we will provide a Starter Kit that includes a dataset for model training, scripts for data fetching, and a step-by-step tutorial to help illustrate how these procedures can be done. In addition to the datasets and scripts provided in the Starter Kit, feel free to source any other data for modeling purposes so long as a real-time data fetching script is provided (together with any potential subsequent data wrangling and feature engineering). 

Additional submission details will be posted once the challenge officially opens later this year.

Evaluation

All models submitted will be evaluated against the weighted average of the following seven model validation metrics:

• Area Under the Receiver Operating Characteristic Curve (AUC/AUROC)
• Area Under the Precision-Recall Curve (PR-AUC)
• Recall Score
• F1 Score
• Brier Score
• Kolmogorov-Smirnov Statistic (KS Statistic)
• Predicted Probability Densities (difference between the median predicted probability of the two labels)

These metrics will be calculated for the predictions (probabilities + labels) returned by the modeler on the testing dataset that will be made available by the validator(s).

More Information About the MACRO Score

Last year Spectral released an on-chain creditworthiness assessment score called the MACRO Score. Read about the reasoning behind an on-chain creditworthiness assessment score:

• How Web3 Creditworthiness can create Financial Inclusion 

Read about our MACRO Score, and how we created and validated the MACRO Score here:

• Introduction to the MACRO Score
• A Deeper Dive into the MACRO Score (Part One)
• A Deeper Dive into the MACRO Score (Part Two)

The MACRO Score has been incorporated into several protocols including Teller, and we’re happy to welcome yet another new partner!

We’re proud to announce BullaNetwork will be integrating the MACRO Score into its p2p lending application, using our Scores to determine creditworthiness, interest rates, and collateral. BullaNetwork is a revolutionary new Web3 accounting protocol that, through its app, BullaBanker, allows users to quickly and easily conduct personal and commercial transactions on-chain.