Actuarial Standard Board

Modeling

TRANSMITTAL MEMORANDUM

June 2013

TO: Members of Actuarial Organizations Governed by the Standards of Practice of the Actuarial Standards Board and Other Persons Interested in Modeling

FROM: Actuarial Standards Board (ASB)

SUBJ: Proposed Actuarial Standard of Practice (ASOP)

This document contains the exposure draft of a proposed actuarial standard of practice, Modeling. Please review this exposure draft and give the ASB the benefit of your comments and suggestions. Each written response and each response sent by e-mail to the address below will be acknowledged, and all responses will receive appropriate consideration by the drafting committee in preparing the final document for approval by the ASB.

The ASB accepts comments by either electronic or conventional mail. The preferred form is e-mail, as it eases the task of grouping comments by section. However, please feel free to use either form. If you wish to use e-mail, please send a message to comments@actuary.org. You may include your comments either in the body of the message or as an attachment prepared in any commonly used word processing format. Please do not password protect any attachments. Include the phrase “ASB COMMENTS” in the subject line of your message. Please note: Any message not containing this exact phrase in the subject line will be deleted by our system’s spam filter. Comments will be posted in the order that they are received. Comments received after the deadline will not be posted.

If you wish to use conventional mail, please send comments to the following address:

Modeling
Actuarial Standards Board
1850 M Street, NW, Third Floor
Washington, DC 20036

The ASB posts all signed comments received to its website to encourage transparency and dialogue. Unsigned or anonymous comments will not be considered by the ASB nor posted to the website. The comments will not be edited, amended, or truncated in any way. Comments will be posted in the order that they are received. Comments will be removed when final action on a proposed standard is taken. The ASB website is a public website and all comments will be available to the general public. The ASB disclaims any responsibility for the content of the comments, which are solely the responsibility of those who submit them.

Deadline for receipt of responses in the ASB office: September 30, 2013

Background

The ASB first began work on a standard for modeling in the late 1990s. Motivated primarily to address the role catastrophe modeling of earthquakes and hurricanes played in property ratemaking, this work was focused on the use of specialized models where the actuaries would have to rely on a model that was developed by professionals other than actuaries. As a result of this work, ASOP No. 38, Using Models Outside the Actuary’s Area of Expertise, was approved by the ASB in June of 2000 with the scope of the standard limited to the Property/Casualty area of practice. Currently, this is the only ASOP that specifically addresses modeling.

Recently, the number and importance of modeling applications in actuarial science has increased, with the results of actuarial models often entering financial statements directly. Recognizing this trend, the ASB asked the Life Committee in 2010 to begin work on an ASOP focused on modeling. The Life Committee formed a task force to address this issue and, in February of 2012, a discussion draft titled Modeling in Life Insurance and Annuities was released and 19 comment letters were received.

Based upon this feedback and numerous other discussions on the topic of modeling, in December of 2012 the ASB created two multi-disciplinary task forces under the direction of the General Committee:

i) a general Modeling Task Force, charged with developing an ASOP to address modeling applications in all practice areas, and

ii) a Task Force to consider expanding ASOP No. 38 to all practice areas while still maintaining the significant influence that ASOP No. 38 currently commands among property/casualty actuaries and regulators.

This exposure draft is the work of that general Modeling Task Force, whose membership has experience in life insurance, health insurance, property/casualty insurance, enterprise risk management, and pension/benefits. They would especially like to point out that much of this document is drawn from the work of the Life Committee’s task force that produced the discussion draft Modeling in Life Insurance and Annuities and thank its members—Dale S. Hagstrom, David A. Brentlinger, Timothy C. Cardinal, Julie H. Fried, Jack L. Gibson, Ronald J. Harasym, and John O. Nigh—for their work.

Actuaries generally agree that almost all actuarial work involves modeling of some type and, at the direction of the ASB, this standard was developed to apply to all practice areas and all forms of models. However, in light of this very broad scope, the proposed ASOP recognizes the fact that situations occur where some of its guidance is not appropriate to the intended application of the model or the project’s objective, perhaps because the guidance is not practical or feasible for the actuary to follow. In this case, the actuary is permitted to use professional judgment in determining where it is appropriate to deviate from the guidance included in the proposed standard and is required to disclose those deviations only if they are material.

Work is currently being done by another task force on the revision of ASOP No. 38. Any potential changes in ASOP No. 38 are expected to be in conformance with this proposed ASOP but will provide more detail with regard to a narrower scope while maintaining the vast majority of guidance that now applies only to property/casualty work. The ASB tentatively plans to release an exposure draft of that revision later in 2013.

Request for Comments

The task force would appreciate comments on all areas of this proposed ASOP and would like to draw the readers’ attention to the following questions in particular:

1. Does the proposed standard provide sufficient guidance to actuaries working with models?

2. Is the proposed standard sufficiently flexible to allow for new developments?

3. The draft ASOP starts with a wide scope, but allows the actuary to use professional judgment to identify those instances (such as those involving minimal reliance by the user, or resulting in a non-material financial effect) where some guidance described in this ASOP is not appropriate or practical. Is this clear and appropriate?

4. In those instances where some guidance described in this ASOP is not appropriate or practical and the deviations from guidance are “not material,” the actuary does not need to disclose these deviations. Is this clear and appropriate?

5. Appropriate documentation simplifies later use and development of current models as well as allowing easier review by principals and other actuaries. Section 3 contains guidance with regard to documentation. Is this guidance clear and appropriate?

6. Does the use of bold font to identify defined terms improve the readability and clarity of the standard? If not, what suggestions do you have to improve the recognition of defined terms in the standard?

The ASB voted in June 2013 to approve this exposure draft.

 

Modeling Task Force

Dale S. Hagstrom, Chairperson

                      Maryellen J. Coggins                                          Kenneth R. Kasner

                      Julie H. Fried                                                     Aaron R. Weindling

 

General Committee of the ASB

Michael S. Abroe, Chairperson

Maria M. Sarli, Vice-Chairperson

                      Albert J. Beer                                                     Martin M. Simons

                      Paul Braithwaite                                                 Barbara Snyder

                      Raymond Brouillette                                           Thomas D. Snook

                      Charles Cook                                                     James E. Turpin

                      Mary Simmons

Actuarial Standards Board

Robert G. Meilander, Chairperson

                      Beth E. Fitzgerald                                               Thomas D. Levy

                      Alan D. Ford                                                       Patricia E. Matson

                      Patrick J. Grannan                                              James J. Murphy

                      Stephen G. Kellison                                            James F. Verlautz

The ASB establishes and improves standards of actuarial practice. These ASOPs identify what the actuary should consider, document, and disclose when performing an actuarial assignment. The ASB’s goal is to set standards for appropriate practice for the U.S.

Section 1. Purpose, Scope, Cross References, and Effective Date

1.1 Purpose

This actuarial standard of practice (ASOP) provides guidance to actuaries selecting, designing, building, modifying, developing, or using models when performing professional services.

1.2 Scope

This ASOP applies to actuaries selecting, designing, building, modifying, developing, or using models when performing professional services. This ASOP applies to all forms of models in all practice areas.

Given the wide use of models in actuarial practice, there may be less significant instances, such as those involving minimal reliance by the user, or resulting in a minimal financial effect, where some guidance described in this ASOP is not appropriate or practical, as discussed in section 3.1. For example, the specifications, development, documentation, and controls for models used in less critical situations may not need to be as rigorous as stated in this ASOP because the ASOP’s guidance might not be practical or appropriate for the intended application of the model or the project’s objective.

If the actuary departs from the guidance set forth in this ASOP in order to comply with applicable law (statutes, regulations, and other legally binding authority), or for any other reason, the actuary should refer to section 4.

1.3 Cross References

When this ASOP refers to the provisions of other documents, the reference includes the referenced documents as they may be amended or restated in the future, and any successor to them, by whatever name called. If any amended or restated document differs materially from the originally referenced document, the actuary should consider the guidance in this ASOP to the extent it is applicable and appropriate.

1.4 Effective Date

This ASOP is effective for work performed on or after four months after adoption by the Actuarial Standards Board.

Section 2. Definitions

The terms below are defined for use in this actuarial standard of practice.

2.1 Assumptions

A type of input to a model that represents expectations or possibilities based on professional judgment.

2.2 Data

A type of input to a model that represents facts or information usually collected from records, experience, or observation.

2.3 Granularity

The extent to which a model contains separate components such as cells, or assumptions that vary by cell or time intervals. Models with a higher degree of granularity (more cells or assumption variations) may provide more model precision or flexibility, but may also require greater effort and expense to design, maintain, assemble, and run.

2.4 Implementation

An executable form of the model. Examples of implementation may include, but not be limited to, a computer program, database, spreadsheet, or any combination thereof.

2.5 Input

Assumptions, data, or parameters used in a model.

2.6 Intended Application

The designer’s planned uses for the model.

2.7 Intended Purpose

The intended application or the project’s objective or both, depending on the actuary’s role. The intended application applies if the actuary’s role includes designing, building, or developing the model. The project’s objective applies if the actuary’s role includes selecting or using the model in an actual project.

2.8 Margin

An adjustment for uncertainty, such as that caused by a lack of full credibility of the data.

2.9 Model

A representation of relationships among entities or events using statistical, financial, economic, or mathematical concepts and equations. Models are used to help explain a system, to study the effects of different components, and to derive estimates and guide decisions. A model consists of (1) a specification that describes the input and the relationships among them, (2) an implementation that is achieved through a set of mathematical formulas and algorithms, and (3) a realization that produces a set of outputs.

2.10 Modeling

Selecting, designing, building, modifying, developing, or using models.

2.11 Model Risk

The risk of adverse consequences to output and decisions as a result of a flawed model, inappropriate inputs, or misapplication of the model.

2.12 Neutral

A description of model inputs or methodologies that are intended to anticipate expected future experience without any adjustment for uncertainty or for asymmetric alternative outcomes.

2.13 Organization

The entity that is being modeled in whole or in part. Examples include public or private companies, benefit plans, government entities, and associations, whether for profit or not for profit.

2.14 Parameter

Mathematical, financial, economic, or statistical input to models that, when varied, results in different realizations. Examples include expected values, and the coefficients of variables in mathematical distributions or regression formula. As input to a model, parameters are sometimes considered assumptions and are sometimes considered data, but are named separately in this standard.

2.15 Principal

A client or employer of the actuary.

2.16 Project’s Objective

The specific goal or question the actuary is addressing when selecting or using a model to meet the needs of the principal.

2.17 Realization

Model results that are derived from a given set of inputs. This concept is also sometimes referred to as a “scenario” or a “run.”

2.18 Reproducible

A property of a model that implies that each time the model is run with the same inputs, the realization will be identical.

2.19 Specification

A description of a model that identifies the inputs and their interactions with each other, the formulas and algorithms to be used, and the outputs to be produced.

Section 3. Analysis of Issues and Recommended Practices

3.1 Application of ASOP Guidance

The guidance in this ASOP applies to actuarial practice regarding all models in all practice areas.

3.1.1 Model Reliance and Financial Importance

Full application of this guidance is appropriate when intended model users rely heavily on the results and the model has a material financial effect.

In modeling situations where the results are either not heavily relied upon or do not have material financial effect, full application of the guidance in this ASOP may not be necessary. For example, efforts such as those concerning specifications, development, documentation, and controls may not need to be as rigorous as stated in this ASOP. The resources committed and controls the actuary applies to a model should relate to the degree of reliance on model results and the financial importance of decisions based upon these model results.

In deciding the extent to which the guidance in this ASOP applies, the actuary should use professional judgment, considering the extent of reliance by the intended user and the materiality of the financial effect. This consideration should be made within the context of the use of the model results and the requirements of the principal.

3.1.2 Models Developed by Others

If the actuary uses a model designed or built by someone else, such as a vendor or colleague, there may be limited ability to understand the underlying workings of the model and, therefore, full application of the guidance in this ASOP may not be necessary. Nonetheless, the actuary should make a reasonable and appropriate attempt, given the project’s objective, to understand the following:

a. the basic workings of the model;

b. major sensitivities and dependencies within the model; and

c. key strengths and limitations of the model.

3.1.3 Responsibility of the Actuary

If, in the actuary’s professional judgment, circumstances are such that applying some or all of the guidance in this ASOP is not appropriate, the actuary should be prepared to identify such circumstances and justify limiting the full application of the guidance in this ASOP. In those instances where the deviation from guidance is material, the actuary should disclose that deviation from guidance as addressed in section 4.2.

3.2 Model Meeting the Intended Purpose

The actuary should select, design, build, modify, develop, or use a model that meets the intended purpose.

3.2.1 Designing, Building, or Developing the Model for the Intended Application

The actuary should confirm that the capability of the model is consistent with the intended application when the actuary designs, builds, or develops the model. In this evaluation, the actuary should consider items such as the granularity of inputs, the causal relationships recognized, the model’s ability to perform stochastic analyses or stress testing, and the model’s ability to identify possible volatility around expected values.

3.2.2 Selecting or Using the Model for the Project’s Objective

The actuary should select or use the model to meet the project’s objective. The selection or use of the model, including the judgments, efforts to improve the model inputs and formulas, documentation, controls, validation, checking, and presentation of results, should be consistent with the project’s objective.

3.2.3 Modifying the Model

When modifying a model to change the intended application or to improve the model’s ability to meet its intended application, the actuary should be guided by section 3.2.1. When modifying a model to improve the model inputs, formulas, and outputs to meet the project’s objective, the actuary should be guided by section 3.2.2.

3.2.4 Understanding the Model

The actuary’s responsibilities may include expressing an opinion, using or communicating results, or preparing documentation. In these instances, the actuary should do the following:

a. understand important aspects of the model being used, including but not limited to, basic operations, important relationships, major sensitivities, strengths and potential weaknesses;

b. understand whether, and the extent to which, the model can fulfill its intended purpose, given limited information, time constraints, and other practical considerations;

c. consider documenting how the model meets the intended purpose; and

d. consider documenting potential limitations.

3.2.5 Model Structure

The actuary should consider how the structure of the model meets its intended purpose. For example, where applicable and where appropriate for the model’s intended purpose, the actuary should consider the following:

a. which provisions and risks specific to a contract or plan are material and appropriate to reflect in the model;

b. whether grouping model inputs will produce reasonable results;

c. whether the use of the model requires a particular level of granularity;

d. whether documenting the rationale for grouping data would be appropriate;

e. whether deterministic or stochastic results, or both, are needed; and

f. whether the projection of future results might be materially influenced by the existence of choices and options available to the organization and its members (that is, company management and policyholders, or plan sponsors and plan participants) and counter parties (such as debtors whose bonds are assets of the organization).

3.2.6 Inputs to the Model

The actuary should refer to ASOP No. 23, Data Quality, in determining the sources for deriving assumptions, data, and parameters for the model.

3.2.7 Assumptions and Parameters

The actuary should use assumptions and parameters that are appropriate in light of the intended purpose.

a. Experience Used—The actuary should consider experience that is based on appropriate available data, given time or budget constraints, in light of the model’s intended purpose. The actuary should consider the following:

1. using assumptions based on actual experience, to the extent it is available, relevant, and credible;

2. if relevant and credible experience is not available, using professional judgment in modifying available sources of information; and

3. whether it would be appropriate to include a margin for an assumption or parameter where experience data are not fully credible and where the assumption or parameter is significant.

b. Range of Assumptions and Parameters—The actuary should consider whether the range of assumptions and parameters used and the number of realizations analyzed reflect a range of conditions consistent with the intended purpose.

c. Consistency—The actuary should use assumptions for the model that are consistent with one another. For example, where appropriate, the actuary should consider using assumptions consistent with the underlying economic scenario assumed in the model. However, if inconsistency among assumptions is required by legal constraints, by the principal, or as the result of a deliberate redundancy such as added conservatism, the actuary should disclose the inconsistency and the reasons for it in accordance with section 4.1.

d. Monitoring of Assumptions—Where practical, the actuary should consider monitoring that the assumptions are still appropriate for use in the current realization of the model. For example, models used in financial reporting offer frequent opportunities to compare assumptions to emerging experience in the aggregate.

e. Documentation—The actuary should document the assumptions, data, and parameters used in the model.

3.3 Mitigation of Model Risk

The actuary should attempt to mitigate model risk using validation, checking, analysis, governance, and controls as appropriate to the intended purpose.

3.3.1 Validation, Checking, and Analysis

The nature and degree of validation, checking, and analysis selected by the actuary should be consistent with the complexity of the model and the intended purpose.

a. Model Integrity—For each realization (or a set of realizations) that is to be relied upon by the intended user, the actuary should validate that the model properly represents the situation under study. Validation of the model could include, but is not limited to, the following:

1. a reconciliation of relevant output values to actual data, addressing and documenting the differences appearing in the reconciliation, if material.

2. checking formulas, logic, and table references. The degree of checking that is appropriate will depend on the intended application; the project’s objective for which the model is being used; the context and nature of the model; the operating environment and controls; and whether there have been any changes to the model or the model environment.

3. where applicable, testing the model projection results against historical data to verify that modeled results bear a reasonable relationship to actual results over a given time period; and

4. examining the potential for model risk and then undertaking reasonable and appropriate steps to mitigate or eliminate it.

b. Analyzing the Output—The actuary should take appropriate steps to evaluate whether the model results are reasonable. Depending on the project’s objective, the actuary should consider the following:

1. performing analytical tests on model results to assess the reasonableness of the projection (for example, testing for the appropriate application of assumptions);

2. reconciling the results of a realization to prior realizations, given any changes in assumptions, parameters, data, formulas, or other aspects of the model since the prior realizations. If such reconciliation can be developed and would be appropriate to the project’s objective, the reconciliation should be documented in the actuary’s files; and

3. running sensitivity tests on key assumptions to test that the model has been used correctly and that changes in the results are consistent with the changes in those assumptions.

c. Peer Review—The actuary should consider a peer review, where practical, of both model construction and the reasonableness of model results, given the intended purpose.

3.3.2 Appropriate Governance and Controls

The actuary should use appropriate model governance and controls to minimize model risk, to maintain the integrity of the model, and to avoid the introduction or use of unintentional or untested changes. For example, if the model is deterministic, implementations and realizations used in reports should be reproducible. For stochastic simulations in models that are not deterministic, the actuary should consider if similar inputs will produce similar outputs. The actuary may want to confirm that different simulations or random number generator seeds produce similar distributions of results.

3.4 Presentation of Results

As indicated in section 3.7.1, the actuary should communicate the results in compliance with ASOP No. 41, Actuarial Communications. The actuary should present results of a realization of the model, explaining methodology, key assumptions, possible limitations, and any changes made subsequent to a prior realization.

3.4.1 Explanation of Model in Actuarial Report

If an actuarial report includes information derived from models, the actuary should consider including explanations of the following:

a. the intended application of the models and how the users’ needs are addressed by those models;

b. the extent to which the models fulfill their intended purpose, given limited information, time constraints, and other practical considerations;

c. any material limitations of the models that have been used and the implications of those limitations; and

d. uncertainty in model results.

3.4.2 Reconciliation

The actuary should consider including in the actuarial report reconciliation to a prior actuarial report. Such reconciliation, if any, should include an explanation of assumptions or methods that have changed from the prior realization.

3.4.3 Description of Judgment

The actuary should consider including a description of the judgment applied in the selection of model inputs and methodology in relation to a neutral position. Terminology may include language such as “conservative,” “most likely,” or “optimistic,” along with a description of the relationship to the neutral position by appropriate quantitative, qualitative, or directional language.

When using the term “neutral” or other terms made meaningful by reference to “neutral,” the actuary should consider whether an additional description is appropriate to avoid ambiguity. For example, several sets of inputs or methodologies may align with different characteristics of expected future experience. Inputs that align with the mean, median, or mode of a random variable could each be described as neutral. In other cases, an input or methodology could be neutral with respect to one aspect of future experience but not with respect to another.

3.4.4 Terms from Applicable Law

If applicable law (statutes, regulations,and other legally binding authority) specifies that an estimate described as a “best estimate” or other similar term should be derived using methods, assumptions, or judgments that are not neutral, section 3.4.3 does not apply. In this instance, the actuary should refer to section 4.2(a) and should explain in the report the basis used for the derivation.

3.5 Reliance on Data or Other Information Supplied by Others

When relying on data or other information supplied by others, the actuary should refer to ASOP Nos. 23 and 41, for guidance. When relying on projections or supporting analysis supplied by others, the actuary should refer to ASOP No. 23, deeming such projections or supporting analysis as data covered by that standard. Similarly, the actuary should refer to ASOP No. 41 (including sections 4.2 and 4.3) with respect to the disclosure of responsibility for data, assumptions, and methods.

3.6 Documentation

Where appropriate to the intended purpose, the actuary should retain documentation or other file material. The actuary should also prepare and retain documentation to demonstrate compliance with the disclosure requirements of section 4 of this ASOP.

All documentation required by this ASOP should include a statement of the purpose of the documentation and sufficient detail to enable another actuary qualified in the same practice area to understand the matters involved and assess the judgments made.

3.7 Relation to Other ASOPs

The actuary should refer to other relevant ASOPs, including the following.

3.7.1 ASOP Nos. 23 and 41

Important guidance appropriate to various aspects of modeling, such as inputs and disclosures, is included in ASOP Nos. 23 and 41.

3.7.2 Other ASOPs

Other ASOPs provide specific modeling requirements, including guidance on setting assumptions and parameters. The actuary selecting, designing, building, modifying, developing, or using models should satisfy not only the requirements of this ASOP, but also any specific modeling requirements from an applicable ASOP. If such specific modeling guidance from an applicable ASOP is inconsistent with the guidance of this ASOP, the guidance of such other ASOP supersedes the guidance of this ASOP.

Section 4. Communications and Disclosures

4.1 Actuarial Communications

When issuing actuarial communications incorporating modeling within the scope of this ASOP, the actuary should disclose the following, as appropriate:

4.1.1 The Intended Purpose of the Model

What the actuary understands to be the intended application of the model and the project’s objective, as discussed in section 3.2.4.

4.1.2 Failure to Meet Intended Purpose

Any reasons that prevent the model from meeting its intended purpose, as discussed in section 3.2.4.

4.1.3 Inconsistent Assumptions

Any inconsistency in assumptions and the reasons therefore, whether in situations covered by section 4.2 or as the result of a deliberate redundancy such as added conservatism, as discussed in section 3.2.7(c).

4.2 Deviation from Guidance in the Standard

When issuing actuarial communications incorporating modeling, the actuary should refer to ASOP No. 41 and should include the following where applicable:

a. the disclosure in ASOP No. 41, section 4.2, if any material assumption or method was prescribed by applicable law (statutes, regulations, and other legally binding authority);

b. the disclosure in ASOP No. 41, section 4.3, if the actuary states reliance on other sources and thereby disclaims responsibility for any material assumption or method selected by a party other than the actuary; and

c. the disclosure in ASOP No. 41, section 4.4, if, in the actuary’s professional judgment, the actuary has otherwise deviated materially from the guidance of this ASOP.

Appendix

Note: This appendix is provided for informational purposes but is not part of the standard of practice.

Background

Models have always played a fundamental role in actuarial work with every discipline relying on a very broad range of modeling applications, ranging from simple spreadsheets to complex capital models. Recently, the number and importance of modeling applications in actuarial science have continued to increase, with the results of actuarial models often entering financial statements directly.

The ASB first began work on a standard for modeling in the late 1990s. Motivated primarily to address the role catastrophe modeling of earthquakes and hurricanes played in property ratemaking, this work was focused on the use of specialized models where the actuaries would have to rely on a model that was developed by professionals other than actuaries. As a result of this work, ASOP No. 38, Using Models Outside the Actuary’s Area of Expertise, was approved by the ASB in June of 2000 with the scope of the standard limited to the property/casualty area of practice. Currently, this is the only ASOP that specifically addresses modeling.

In light of these developments, and the fact that many ASOPs currently reference “models” or “modeling” in their guidance, the ASB felt it was appropriate to develop a general Modeling ASOP, which addresses all areas of practice.

Current Practices

Actuaries often develop and use models when analyzing uncertain outcomes. In these instances, even a model that is prudently developed and carefully used does not eliminate inherent uncertainty and variability and actual experience may differ, sometimes significantly, from the estimates derived from the model results. These differences, by themselves, do not indicate a flawed model or noncompliance with standards.

Comments

Click here to view comments in their entirety.