TRANSMITTAL MEMORANDUM

     June 2000

TO: Members of Actuarial Organizations Governed by the Standards of Practice of the Actuarial Standards Board and Other Persons In­terested in the Use of Models Outside the Actuary’s Area of Expertise in Property and Casualty Insurance

FROM: Actuarial Standards Board (ASB)

SUBJ: Actuarial Standard of Practice (ASOP) No. 38

This booklet contains the final version of ASOP No. 38, Using Models Outside the Actuary’s Area of Expertise (Property and Casualty).

Background

The Casualty Practice Council of the American Academy of Actuaries requested that the ASB consider drafting an actuarial standard of practice concerning the use of complex models. In submitting to the ASB its proposal for a new ASOP, the council expressed concern over the use of catastrophe models when estimating catastrophe costs. Catastrophe models are developed by groups of scientists, engineers, and actuaries working together to simulate catastrophic events. While most actuaries conceptually agree that catastrophe models may provide more realistic measures of catastrophic risk than those provided by analyzing the latest twenty to fifty years of catastrophe losses, most actuaries are not experts in many of the underpinnings of these models.

Of course, catastrophe models are not the only models with which actuaries work. Actuaries also may utilize interest rate models, investment return models, credit scoring models, asbestos and pollution models, and dynamic financial analysis models, to name a few. The standard would not apply to models that incorporate specialized knowledge within the actuary’s own area of expertise, since working with these components is part of the normal actuarial effort and is covered by other ASOPs.

In order to feel comfortable with relying on models that incorporate specialized knowledge outside the actuary’s area of expertise, actuaries seek guidance in defining their duty of care in understanding and relying upon these models. This was another reason for the development of the standard, and why the ASB created the Task Force on Complex Models, under its Casualty Committee, to initiate the project.

The task force intended that the standard should define the guidelines that an actuary should follow when working with models outside of the actuary’s own area of expertise. In providing such guidance, the standard makes it clear that an actuary may rely upon a model evaluation by another actuary who has performed his or her evaluation in accordance with this standard, and that the standard is not intended to discourage the use of new methodologies in advancement of the profession.

First Exposure Draft

The first draft of a proposed standard, titled The Use of Models with Nonactuarial Components, was exposed for review in a document dated May 1998. As originally proposed in this first exposure draft, the standard would have applied to models in all areas of actuarial practice. In response to the fifty-two comment letters and forty-two comment postcards received, the scope of the standard was narrowed to apply only to property and casualty practice. In addition, the standard was refocused to apply to models that incorporate specialized knowledge outside the actuary’s own area of expertise. Each actuary must determine what this boundary means to him or her. The title of the standard was changed accordingly. The significant issues and questions contained in the comment letters on the first exposure draft as well as the task force’s responses to them are summarized in Appendix 2 of the second exposure draft titled Using Models Outside the Actuary’s Area of Expertise (Property and Casualty) dated September 1999.

Second Exposure Draft

The second draft of the standard was exposed for review in a document dated September 1999, with a comment deadline of March 1, 2000. Ten comment letters were received. The task force considered the issues and questions raised in these letters and made some editorial changes to the text, but no substantive changes were necessary. For a summary of the issues contained in these ten comment letters and the task force’s responses, please see Appendix 2.

The Task Force on Complex Models and the Casualty Committee thank everyone who took the time to contribute comments and suggestions on both exposure drafts.

The Casualty Committee would like to thank Godfrey Perrott and Kurt Reichle for their assistance in the initial drafting of this standard.

The ASB voted in June 2000 to adopt this standard.

Task Force on Complex Models of the Casualty Committee

Karen F. Terry, Chairperson

                      Kay A. Cleary                                                     Jeffrey F. McCarty

                      Alice H. Gannon                                                 Daniel M. Scheinbreif

                      Paul E. Kinson                                                   A. Eric Thorlacius

                      Ronald T. Kozlowski                                           Joan M. Weiss

                      David A. Lalonde

Casualty Committee of the ASB

Michael A. LaMonica, Chairperson

                      Christopher S. Carlson                                       Karen F. Terry

                      Anne Kelley                                                        William J. VonSeggern

                      Ronald T. Kozlowski                                           Alfred O. Weller

                      Robert J. Lundquist                                            Patrick B. Woods

                      Robert S. Miccolis

Actuarial Standards Board

David G. Hartman, Chairperson

                     Phillip N. Ben-Zvi                                                    Roland E. King

                     Heidi R. Dexter                                                       William C. Koenig

                     David G. Hartman                                                  James R. Swenson

                     Ken W. Hartwell                                                     Robert E. Wilcox

Section 1. Purpose, Scope, Cross References, and Effective Date

1.1 Purpose

The purpose of this standard is to provide guidance to the actuary in using models that incorporate specialized knowledge outside of the actuary’s own area of expertise when developing an actuarial work product. This guidance addresses the actuary’s obligation to review the model and make appropriate disclosures.

1.2 Scope

This standard applies to actuaries who use models that incorporate specialized knowledge outside of the actuary’s own area of expertise when performing professional services in connection with property and casualty insurance coverages (including risk financing systems, such as self-insurance and securitization products, that provide similar coverages). This standard applies to the use of all models whether or not they are proprietary in nature. If the actuary departs from the guidance set forth in this standard in order to comply with applicable law (statutes, regulations, and other legally binding authority), or for any other reason the actuary deems appropriate, the actuary should refer to Section 4.

1.3 Cross References

When this standard refers to the provisions of other documents, the reference includes the referenced documents as they may be amended or restated in the future, and any successor to them, by whatever name called. If any amended or restated document differs materially from the originally referenced document, the actuary should consider the guidance in this standard to the extent it is applicable and appropriate.

1.4 Effective Date

This standard will be effective for work performed on or after December 15, 2000.

The terms below are defined for use in this actuarial standard of practice.

2.1 Expert

One who is qualified by knowledge, skill, experience, training, or education to render an opinion concerning the matter at hand.

2.2 Model

An information structure, such as a set of mathematical equations, logic, or algorithms, that is used to represent the behavior of specified phenomena.

Section 3. Analysis of Issues and Recommended Practices

3.1 Introduction

In performing actuarial work, an actuary may find it appropriate to use models that incorporate specialized knowledge outside of the actuary’s own area of expertise. When using such a model, the actuary should do all of the following:

a. determine appropriate reliance on experts;

b. have a basic understanding of the model;

c. evaluate whether the model is appropriate for the intended application;

d. determine that appropriate validation has occurred; and

e. determine the appropriate use of the model.

The actuary’s level of effort in understanding and evaluating a model should be consistent with the intended use of the model and its materiality to the results of the actuarial analysis.

3.2 Appropriate Reliance on Experts

An actuary may rely on experts con­cerning those aspects of a model that are outside of the actuary’s own area of expertise. The experts relied upon may either be the experts who provided the model or other experts. In determining the appropriate level of reliance, the actuary should consider the following:

a. whether the individual or individuals upon whom the actuary is relying are experts in the applicable field;

b. the extent to which the model has been reviewed or opined on by experts in the applicable field, including any known significant differences of opinion among experts concerning aspects of the model that could be material to the actuary’s use of the model; and

c. whether there are standards that apply to the model or to the testing or validation of the model, and whether the model has been certified as having met such standards.

3.3 Understanding of the Model

The actuary should be reasonably familiar with the basic components of the model and understand both the user input and the model output, as discussed below.

3.3.1 Model Components

The actuary should be reasonably familiar with the basic components of the model and have a basic understanding of how such components interrelate within the model. In addition, the actuary should identify which fields of expertise were used in developing or updating the model and should make a reasonable effort to determine if the model is based on generally accepted practices within the applicable fields of expertise. The actuary should also be reasonably familiar with how the model was tested or validated and the level of independ­ent expert review and testing.

3.3.2 User Input

Certain user input may be required to produce model output for the specific application. The actuary should understand the user input that is required to produce the model output. This understanding includes the level of detail required in the user input to produce results that are consistent with the intended use of the model.

3.3.3 Model Output

The actuary should determine that the model output is consistent with the actuary’s intended use of the model.

3.4 Appropriateness of the Model for the Intended Application

The actuary should evaluate whether the model is appropriate for the particular actuarial analysis, and consider limitations of the model, modifications to the model, and the assumptions needed in order to apply the model output. Some additional considerations include the following:

a. Applicability of Historical Data: To the extent historical data are used in the development of the model or the establishment of model parameters, the actuary should consider the adequacy of the historical data in representing the range of reasonably expected outcomes consistent with current knowledge about the phenomena being analyzed.

b. Developments in Relevant Fields: The actuary should make a reasonable effort to be aware of significant developments in relevant fields of expertise. The actuary should evaluate whether such developments are likely to materially affect the current actuarial analysis.

3.5 Appropriate Validation

The actuary should evaluate the user input and the reasonableness of the model output, as discussed below.

3.5.1 User Input

With respect to the quality and availability of the user input data to be used in the model, the actuary should refer to ASOP No. 23, Data Quality.

3.5.2 Model Output

In view of the intended use of the model, the actuary should examine the model output for reasonableness, considering factors such as the following:

a. the results derived from alternate models or methods, where available and appropriate;

b. how historical observations, if applicable, compare to results produced by the model;

c. the consistency and reasonableness of relationships among various output results; and

d. the sensitivity of the model output to variations in the user input and model assumptions.

3.6 Appropriate Use of the Model

Having completed the analysis described in sections 3.2–3.5 above, the actuary should use his or her professional judgment to determine whether it is appropriate to use the model results, subject to any appropriate adjustments. The actuary should disclose any such adjustments in accordance with section 4.3.

3.7 Reliance on Model Evaluation by Another Actuary

The actuary may rely on another actuary who has, for a particular model, conducted some or all of the evaluations and processes described in this standard. However, the relying actuary should be satisfied that the other actuary’s evaluation was performed in accordance with this standard and is appropriate for the intended application. The actuary should document the extent of such reliance in accordance with section 4.1.

Section 4. Communications and Disclosures

4.1 Documentation

This standard requires documentation whether or not a legal or regulatory requirement exists. The actuary should maintain appropriate documentation on the evaluation of the model and the use of the model output in the analysis. Documentation should demonstrate how the actuary has met the requirements of sections 3.2–3.7 above.

4.2 Proprietary Information

If the model has proprietary aspects or contains proprietary information, the actuary should document the steps taken to comply with this standard in light of the proprietary aspects or information.

4.3 Disclosures

In communicating the results of actuarial work using a model that incorporates specialized knowledge outside of the actuary’s own area of expertise, the actuary should disclose the model(s) used and any adjustments made to the model results as described in section 3.6. In addition, the actuary should include the following, as applicable, in an actuarial communication:

a. the disclosure in ASOP No. 41, Actuarial Communications, section 4.2, if any material assumption or method was prescribed by applicable law (statutes, regulations, and other legally binding authority);

b. the disclosure in ASOP No. 41, section 4.3, if the actuary states reliance on other sources and thereby disclaims responsibility for any material assumption or method selected by a party other than the actuary; and

c. the disclosure in ASOP No. 41, section 4.4, if, in the actuary’s professional judgment, the actuary has otherwise deviated materially from the guidance of this ASOP.

Appendix 1 – Background and Current Practices

Note:  This appendix is provided for informational purposes, but is not part of the standard of practice.

Background

Actuaries have always used models. Most of the models used by actuaries are developed using expertise that is common to actuaries, and their use by actuaries is addressed by existing standards of practice and statements of principles.

However, actuaries have also used models that contain compo­nents that are outside the actuary’s own area of expertise. For example, certain catastrophe models, interest rate models, dynamic financial analysis models, credit scoring models, and pollution models contain components that are outside the expertise of many of the actuaries who use them. Although in retrospect the use of models may have posed the need for a specific standard of practice, it was not until recently, as actuaries grappled with the financial issues surrounding various natural catastrophes, that the need for such a standard was recognized and acted on by the Actuarial Standards Board.

Specifically, Hurricane Andrew in 1992 and the Northridge Earthquake in 1994 led actuaries involved in evaluating hurricane and earthquake exposures to recognize the severe inadequacy of the traditional, empirical actuarial methods used for ratemaking for these exposures. In recognition of the need to replace these methods, many actuaries began using stochastic computer simulation models for their actuarial analysis of hurricane and earthquake exposure.

Comput­er simulation models had been commonly used for some time by actuaries and others for the purpose of evaluating probable maximum loss but had not been widely used for ratemaking. Computer simulation models are now widely used by actuaries for calculating expected losses due to hurricane and earthquake perils. The accuracy of these models is heavily dependent on the accuracy of meteorological, seismological, or engineer­ing assumptions, areas clearly outside the expertise of most actuaries.

Because models sometimes contain components that incorporate specialized knowledge outside the actuary’s own area of expertise, this raises the question as to what is required of an actuary before he or she makes use of model output in his or her actuarial analysis. This standard addresses such requirements. Although the development of this standard originated with the problem of pro­viding accurate actuarial analysis of hurricane and earthquake exposure, the standard applies to any model that incorporates specialized knowledge outside the actuary’s own area of expertise used in connection with property and casualty insurance coverages.

Current Practices

The use of output from models is an evolving area of actuarial theory and practice. To date, current practices have been governed by the former Guides and Inter­pretative Opinions as to Professional Conduct, and their successor documents, the Code of Professional Conduct and the Qualification Standards for Prescribed State­ments of Actuarial Opinion. Practices have varied according to individual inter­pretations of the Guides and the Code.

Appendix 2 – Comments on the Second Exposure Draft and Task Force Responses

The second exposure draft of this actuarial standard of practice (ASOP) was exposed for review in September 1999, with a comment deadline of March 1, 2000. Ten letters of comment were received on the second exposure draft.

Click here to view Appendix 2 in its entirety.