Introduction to MACRO Score
Think of it as an on-chain credit risk assessment score, similar to a traditional credit score except it is generated from on-chain transactions and under a user’s control.
Decentralized finance (DeFi) offers an unprecedented opportunity to rebuild an archaic and often unfair aspect of traditional finance. For retail consumers in North America, the idea of risk assessment and creditworthiness often boils down to a single three-digit score. While a single score is easy to understand, maintaining one can be taxing. When scores slip, there’s often little recourse for consumers and expensive financial consequences. Moreover, opting out isn’t possible, and mistakes are stressful and time-consuming to correct.
Credit, in the form of written obligations, has existed for millennia. Our current system was developed in the 1980s and 1990s, and, while generally accurate and successful at facilitating the growth of consumer credit and mortgages, at times excluded certain communities and demographics. Web3 has created an entirely new category of finance and financing. Surely it's time for a new approach to assessing creditworthiness.
The Multi-Asset Credit Risk Oracle Score
The Multi-Asset Credit-Risk Oracle (MACRO) Score is a three-digit numerical representation of on-chain creditworthiness, computed by a machine-learning-based credit risk model, capable of near-instantly processing the complete transaction history of wallet(s) provided. Think of it as an on-chain credit risk assessment score, similar to a traditional credit score except generated from on-chain transactions and under a user’s control.
The MACRO Score is created by connecting either a single wallet or a bundle of several wallets to Spectral’s App. The App retrieves all DeFi and non-DeFi-related transactions associated with this wallet (or wallets), engineers a number of features from these transactions (see below for details), turns them into a vector, and uses this as an input into a number of machine learning (ML) models to generate a Score ranging from 300—indicating very low creditworthiness—to a maximum Score of 850, representing very high creditworthiness.
The Spectral App delivers a Non-Fungible Credit (NFC) token, an ERC-721 token representing wallet addresses. This token can be bundled with other wallets to create an even more comprehensive Score, burned, or used as a data object in a smart contract, allowing it to become a reputation primitive.
Use the NFC for an objective look at how risky your on-chain behavior is, to begin building and optimizing your on-chain credit history for the next generation of DeFi, and, soon, to receive personalized suggestions for DeFi opportunities.
A closed beta version of the App launched in November 2021. Our Open Beta launched in August 2022.
Spectral
How your MACRO Score is generated
When an address or multiple addresses in the form of a bundle, are connected to Spectral’s App, a bundle ID is provided to Spectral’s scoring system, which returns a MACRO Score and a generation timestamp that together quantify the risk associated with those addresses. It doesn’t matter whether a single wallet or dozens are attached to a bundle; they’re treated the same. All transactions associated with those addresses are tallied into a single Score.
There are seven broad categories of information used to determine the Score, which cover nearly 100 features. Let's dive into the specifics of each of the 7 Score Ingredients.
DeFi Transaction History
Spectral examines who and what you’ve transacted with, the nature and volume of each transaction in terms of borrowings, repayments, and more. This effectively defines an on-chain identity and can substantially affect your MACRO Score.
Liquidation History
The MACRO Score uses liquidation as a proxy for loan default. A higher appetite for risk translates into less creditworthiness or a flag for speculative behavior. Past liquidations generally lower your MACRO Score. For more information, please visit Liquidations as a Proxy for Default.
Loan Safety Margin
Similar to the concept of loan-to-value (LTV) ratio in traditional finance, Spectral evaluates the amount of headroom maintained as part of your borrowing activities. The higher the historical level of DeFi collateralization maintained, the more risk-averse a user tends to be, improving the MACRO Score.
Age or Time-Based Factors
The MACRO Score also evaluates whether a user has an established and reasonably long history of DeFi and on-chain activities. Older wallets can usually be expected to have higher MACRO Scores, as long as they have maintained a reasonably clean credit history.
General Wallet History
Factors such as wallet balance, wallet composition, and general ERC-20 and NFT transactions can represent changing balances, and risk tolerance, and give insight into a user's on-chain behavior. Together, these factors help Spectral better understand a user’s behavior and familiarity with certain protocols.
Market Conditions
Spectral evaluates the general market volatility prevalent at the time of borrowing activities and considers how they could have impacted liquidations. Liquidations during times of extreme market volatility may not impact MACRO Scores as heavily as during normal conditions. Undertaking risky borrowing transactions during volatile or uncertain market conditions is not advised.
Credit Mix
The MACRO Score also considers the mix of DeFi lending protocols a user has interacted with. Although a diversified credit mix (in terms of the number of DeFi protocols interacted with) is not required for a high MACRO Score, it's generally advisable to interact with various DeFi lending protocols to spread one's risk.
Given that the MACRO Score represents a bundle’s creditworthiness, a wallet’s borrowing and repayment history generally have the most impact on a Score. Fidelity increases with the number of transactions recorded. Wallets with no transactions or few transactions yield correspondingly less accurate MACRO Scores.
Measuring the MACRO Score’s Fidelity
A common question asked is whether Spectral has enough data to apply machine learning concepts, given how young of an industry decentralized finance is. The open-source nature of the Ethereum blockchain allows Spectral to extract a substantially broad and deep dataset. Currently, this includes all on-chain borrowing transactions and behavior recorded on several leading DeFi protocols (including Compound and Aave) and several data points related to a user's wallet history and transactions that aren't necessarily executed on DeFi platforms. Additional sources, including transactions on Uniswap and MakerDAO, are being added to production soon.
At the time of publication, Spectral's training data included four years of transactional data from leading DeFi protocols, representing approximately 50,000 unique borrowers who made 3 million DeFi transactions encompassing borrowing, repayment, and liquidation transactions. The model also uses 30 million non-DeFi-related wallet transactions (e.g. an ERC-20 token transfer) to provide supplemental credit risk insights.
Spectral is constantly searching for and adding new signals and historical data, which includes records stretching back to Ethereum's genesis block. The team uses more than 90 starting sets of features representing various aggregations and trends in addition to point-in-time features.
Spectral strives to understand the data being processed. Most of the data science and machine learning team are long-time DeFi users, who understand the nuances and unique transactions involved with Web3. To ensure Spectral doesn't need to reinvent the wheel when scoring risk, Spectral also has years of experience in traditional finance and credit scoring.
From this data, Spectral has created more than 20 candidate models, which are evaluated using five criteria.
Model Evaluation and Validation
The various hyperparameters of Spectral's 20 candidate models, tuned via Bayesian search, were evaluated through time-based k-fold cross validation. Each candidate model was evaluated in terms of its reasonableness and feature importance before being compared through a weighted average evaluation of 5 model validation metrics on out-of-time and out-of-sample data.
Consider the following validation metrics provided by Spectral’s data science team. Backtesting Spectral’s best models so far yield the following validation metrics and other relevant distributions:
A Receiver Operating Characteristic (ROC) curve plots the True Positive Rate against the False Positive Rate at all the possible probability thresholds given a binary classifier. The Area Under the Curve (AUC) of the ROC measures the ability of a classifier to distinguish between classes (e.g., good or bad) and is used as a summary of the ROC curve. Spectral’s AUC of 87 is considered very good and is higher than the AUC of traditional credit scores, where anything above 70 is generally deemed to be good.
A Precision-Recall (PR) curve plots a classifier’s Precision against its Recall at all the possible probability thresholds. A high area under the PR curve (PR AUC) represents both high recall and high precision, where high precision relates to a low false positive rate, and high recall relates to a low false negative rate. Spectral's best models have a PR AUC of 89, which is substantially higher than traditional credit scores.
Kolmogorov–Smirnov statistic (KS) is a standard validation metric in credit scoring that measures the discriminatory power of a model, i.e., its ability to differentiate between the good and bad observations. KS of greater than 45 is generally considered to be very good in the credit scoring domain.
The above-predicted probability density function plots show that Spectral’s model performs reasonably well in identifying bad observations from the good observations – with the probability density function of the bad observations (label = 1) being concentrated on the right (i.e., high probability of liquidation) and vice versa.
The above plot shows the distribution of MACRO Scores across various Score buckets – with each bucket of 50 points. As expected, very few addresses either had a close-to-perfect or worst-possible Score – with most observations falling somewhere in the middle.
The above plot shows how the probability of liquidation is related to the MACRO Scores – with higher probabilities resulting in a low MACRO Score and vice versa.
For more information about our Scoring Methodology, please see the Scoring documentation.
What comes next?
The above is just a starting point. Spectral is in the process of integrating additional DeFi lending protocols and other relevant on-chain data sources (e.g., decentralized exchanges, NFTs, etc.), as well as expanding to other chains as they experiment with other L1/L2s. Other research areas include verifiable computation and zero-knowledge proof implementation to provide provably fair and even more privacy-preserving results to users and developers.
The highest priority is decentralization. Spectral firmly believes in the creation of a "Scoracle Network" where decentralized model creators are incentivized to co-create accurate machine learning models, allowing them to evolve credit risk infrastructure together as a community. This will help safeguard MACRO Scores from bias, help ensure neutrality, and continuously improve accuracy.
If you are a data scientist and are interested in learning more about this topic, you can reach out to us at contact@spectral.finance.
Learning from our Community
There is an active community surrounding Spectral Finance. Join the conversation on Discord to ask questions, give suggestions, and leave feedback or subscribe to our monthly newsletter to keep abreast with the latest about Spectral.
Website | Docs | Github | Discord | Twitter | LinkedIn | Newsletter
