With the introduction of the New Capital Accord (Basel II), the banking industry has had to comply with requirements for more robust risk management practices. 

Banks adopting the so called Advanced Approaches have to develop models to assess probability of default (PD), loss given default (LGD) and exposure at default (EAD) for various classes of asset.

The system that produces these estimates is called a rating system and Basel II imposes a requirement to demonstrate that the ratings produced by these systems are robust and reliable. 

Experian has been working closely with a number of banks to help them demonstrate that their rating systems are objective, accurate and stable. Our experience has been that this is not a straightforward exercise and that it requires a considerable amount of effort. 

Our approach to validation is to match quantitative tools and statistical measures to the type of model being assessed. This has been an instructive process for us as well as our clients. The key lessons we have learned can be summarised as follows:

	Looking at the stability of individual data items, the predictive power of multiple scorecards or the stability of PD, LGD and EAD estimates at pool level can result in the production of hundreds of reports. Even with automated report production, the interpretation of the reports and statistical measures is time consuming. It is vitally important to be able to generate a concise summary of this information which focuses attention where it is most needed. We have developed a ‘traffic light indicator ‘, defined on each report based on the differences between observed and expected results, to help summarise the validation status of each model. This can then be used to prioritise any corrective actions that are deemed necessary.
	Many banks do not have sufficient data to construct a reliable, long running data archive. Without such an archive, it is not possible to estimate the average values of PD, LGD and EAD over the long run. In this scenario, banks have to rely on external data sources or their own ‘business experts’ to form a view of the long run behaviour of these parameters and the level of any tolerances that should be applied to these estimates.
	It is important to compare ‘like with like’ when comparing validation and development samples to ensure that the differences observed are true differences in the data. Our experience has thrown up many instances of data differences between development and validation. An accurate audit of the changes, since the development and during the validation window, has helped interpret observed differences in the reported behaviour of the rating system. These changes can then be made retrospectively to the data, to produce a more reliable view of the rating system's performance.

Validation is an area where banks and regulators are still developing their understanding of what constitutes a reliable and robust rating system and Experian is ideally positioned to assist both groups.

Victoria Carr
Senior Business Consultant
Decision Analytics
Experian

“Source: Experian Decision Analytics worldwide e-news”