Proposed Actuarial Standard of Practice

Modeling

 SECOND EXPOSURE DRAFT

TRANSMITTAL MEMORANDUM

November 2014

TO: Members of Actuarial Organizations Governed by the Standards of Practice of the Actuarial Standards Board and Other Persons Interested in Modeling

FROM: Actuarial Standards Board (ASB)

SUBJ: Proposed Actuarial Standard of Practice (ASOP) on Modeling

This document contains a second exposure draft of a proposed ASOP titled Modeling. Please review this exposure draft and give the ASB the benefit of your comments and suggestions. Each written response and each response sent by e-mail to the address below will be acknowledged, and all responses will receive appropriate consideration by the drafting committee in preparing the final document for approval by the ASB.

The ASB accepts comments by either electronic or conventional mail. The preferred form is e- mail, as it eases the task of grouping comments by section. However, please feel free to use either form. If you wish to use e-mail, please send a message to comments@actuary.org. You may include your comments either in the body of the message or as an attachment prepared in any commonly used word processing format. Please do not password protect any attachments. If the attachment is in the form of a PDF, please do not “copy protect” the PDF. Include the phrase “ASB COMMENTS” in the subject line of your message. Please note: Any message not containing this exact phrase in the subject line will be deleted by our system’s spam filter.

If you wish to use conventional mail, please send comments to the following address:

Modeling (Second Exposure)
Actuarial Standards Board
1850 M Street, Suite 300
Washington, DC 20036

The ASB posts all signed comments received to its website to encourage transparency and dialogue. Unsigned or anonymous comments will not be considered by the ASB nor posted to the website. The comments will not be edited, amended, or truncated in any way. Comments will be posted in the order that they are received. Comments will be removed when final action on a proposed standard is taken. The ASB website is a public website, and all comments will be available to the general public. The ASB disclaims any responsibility for the content of the comments, which are solely the responsibility ofthose who submit them.

Deadline for receipt of responses in the ASB office: March 1, 2015

Background

The ASB first began work on a standard for modeling in the late 1990s. Motivated primarily to address the role catastrophe modeling of earthquakes and hurricanes played in casualty ratemaking, this work was focused on the use of specialized models where actuaries would have to rely on a model that was developed by professionals other than actuaries. As a result of this work, ASOP No. 38, Using Models Outside the Actuary’s Area of Expertise, was approved by the ASB in June of 2000 with the scope of the standard limited to the Property/Casualty area of practice. Historically, ASOP No.38 had been the only ASOP that specifically addressed modeling.

Recently, the number and importance of modeling applications in actuarial science has increased, with the results of actuarial models often entering financial statements directly. Recognizing this trend, the ASB asked the Life Committee in 2010 to begin work on an ASOP focused on modeling. The Life Committee formed a task force to address this issue and, in February of 2012, a discussion draft titled Modeling in Life Insurance and Annuities was released and nineteen comment letters were received. The transmittal letter also mentioned that the scope might be expanded to all practice areas and asked for comments on this idea.

Based upon the feedback received, and numerous other discussions on the topic of modeling, in December of 2012 the ASB created two multi-disciplinary task forces under the direction of the General Committee: i) a general Modeling Task Force, charged with developing an ASOP to address modeling applications in all practice areas, and ii) a Catastrophe Modeling Task Force to consider expanding ASOP No.38 to all practice areas while focusing exclusively on using catastrophe models. The membership of these task forces has experience in all actuarial practice areas, including enterprise risk management.

A new exposure draft titled Modeling was released in June 2013 and was the work of that general Modeling Task Force. At that time, the task force pointed out that much of the exposure draft was drawn from the work of the Life Committee’s task force that produced the discussion draft Modeling in Life Insurance and Annuities and recognized its members—Dale S. Hagstrom, David A. Brentlinger, Timothy C. Cardinal, Julie H. Fried, Jack L. Gibson, Ronald J. Harasym, and John O. Nigh—for their work.

Actuaries generally agree that almost all actuarial work involves modeling of some type and, at the direction of the ASB, both the first and second exposure drafts of this proposed modeling ASOP apply to all practice areas and all forms of models. However, in light of this very broad scope, both the first and second exposure drafts recognize that situations occur where use of the results of the model does not have a material financial effect, or no intended user is relying upon the results heavily, and full application of the guidance in this ASOP may not be necessary or practical. In these cases, the second exposure draft clarifies the actuary’s use of professional judgment to determine whether full application of the guidance included in the standard is warranted.

As the guidance in this proposed modeling ASOP and ASOP No.38 currently titled Catastrophe Modeling (for All Practice Areas)is intended to be coordinated, the ASB will issue final versions of both ASOPs to be effective concurrently. To facilitate review of this proposed modeling ASOP, a link to the current working draft of ASOP No.38 is provided here for your information. The working draft of ASOP No. 38 is not being exposed for comment but does reflect guidance that the ASB and General Committee believe works inconcert with the guidance in the second exposure draft of this proposed modeling ASOP.

First Exposure Draft

In June 2013, the ASB approved a first exposure draft with a comment deadline of September 30, 2013. Forty-eight comment letters were received and considered in making changes that are reflected in this second exposure draft. For asummary of issues contained in these comment letters, please see appendix 2.

Changes made to the second exposure draft in response to the comment letters include clarifying the following:

1. the ASOP’s guidance;

2. guidance regarding the applicability of the modeling guidance;

3. the responsibility of the actuary when the actuary is part of a team;

4. guidance when the actuary is reviewing the modeling work of others; and

5. documentation versus disclosure guidance.

Given the extensive clarifications, the ASB believes it would be appropriate to obtain additional feedback on the proposed Modeling ASOP through the issuance of this second exposure draft. The ASB thanks everyone who took the time to contribute comments and suggestions on the first exposure draft.

Key Issues

In redrafting the standard, the reviewers focused on the following key issues:

1. making the standard clearer that actuarial judgment is needed to determine the extent to which full application of the standard is warranted, or alternatively whether following some of the guidance may not be necessary or practical given the intended application of the model and the project’s objective;

2. enhancing the guidance for situations where the actuary may be relying on other team members, or other colleagues or vendors, who may or may not be actuaries, with respect to the development or use of some parts or all of a model, or may be relying on others to confirmthat the requirements of this standard have been followed;

3. enhancing the guidance that applies when a model has material limitations or otherwise may not fulfill its intended purpose; and

4. providing a more complete discussion about the reasons why adding margins to assumptions or parameters might be appropriate.

Request for Comments

The ASB would appreciate comments on all areas of this proposed standard and would like to draw the reader’s attention in particular to the following questions:

1. Section 3.1.1 discusses situations when the actuary judges whether full guidance is or is not warranted. Is this section clear and appropriate? If not what changes would you suggest?

2. Section 3.1.3 discusses the actuary’s responsibility when the actuary is part of a modeling team. Is this section clear and appropriate? If not what changes would you suggest?

3. Section 3.3.1(a)(2) describes the degree of checking as being dependent on a list of possible factors, and this list includes both the “intended application” and the “project objective,” which apply in different stages of modeling, rather than just referring to the “intended purpose,” which encompasses either. Is this separate mention of the two possible stages of purpose helpful? Would the guidance be clearer if only the term “intended purpose” was used?

4. Does the proposed standard provide sufficient guidance to actuaries working with models?

The ASB voted in November 2014 to approve for exposure this draft standard.

Modeling Task Force

Dale S. Hagstrom, Chairperson

                      Maryellen J. Coggins                                              Kenneth R. Kasner

                      Julie H. Fried                                                            Aaron R. Weindling

General Committee

Maria M. Sarli, Chairperson

                      Shawna S. Ackerman                                           John C. Lloyd

                      Paul Braithwaite                                                    Mary Simmons

                      Raymond Brouilette                                              Thomas D. Snook

                      Charles Cook                                                         Barbara Snyder

                      Dale S. Hagstrom                                                 James E. Turpin

Actuarial Standards Board

Patricia E. Matson, Chairperson

                      Michael S. Abroe                                                     Thomas D. Levy

                      Christopher S. Carlson                                          Robert G. Meilander

                      Maryellen J. Coggins                                              James J. Murphy

                      Beth E. Fitzgerald                                                    James F. Verlautz

The Actuarial Standards Board (ASB) sets standards for appropriate actuarial practice in the United States through the development and promulgation of Actuarial Standards of Practice (ASOPs). These ASOPs describe the procedures an actuary should follow when performing actuarial services and identify what the actuary should disclose when communicating the results of those services.

Section 1. Purpose, Scope, Cross References, and Effective Date

1.1 Purpose

This actuarial standard of practice (ASOP) provides guidance to actuaries selecting, designing, building, modifying, developing, using, reviewing, or evaluating models when performing actuarial services.

1.2 Scope

This ASOP applies to actuaries selecting, designing, building, modifying, developing, using, reviewing, or evaluating models when performing actuarial services. Using a model includes using the results of a model. This ASOP applies to all forms of models in all practice areas.

Given the wide use of models in actuarial practice, there may be situations where the model results either are not heavily relied upon or do not have material financial effects. In such situations, some of the guidance described in this ASOP may not be necessary or practical, as discussed in section 3.1. For example, efforts related to tasks such as data validation and sensitivity testing for models used in less critical situations may not need to be as rigorous as stated in this ASOP because the guidance might not be necessary or practical for the intended application of the model or for the project objective.

If the actuary departs from the guidance set forth in this ASOP in order to comply with applicable law (statutes, regulations, and other legally binding authority), or for any other reason, the actuary should refer to section 4.

1.3 Cross References

When this ASOP refers to the provisions of other documents, the reference includes the referenced documents as they may be amended or restated in the future, and any success or to them, by whatever name called. If any amended or restated document differs materially from the originally referenced document, the actuary should consider the guidance in this ASOP to the extent it is applicable and appropriate.

1.4 Effective Date

This ASOP is effective for work performed on or after nine months after adoption by the Actuarial Standards Board.

Section 2. Definitions

The terms below are defined for use in this actuarial standard of practice.

2.1 Assumptions

Input to a model that represent expectations or possibilities based on professional judgment, or that may be prescribed by law or by others.

2.2 Data

Input to a model that represent facts or information collected from sources such as records, experience, experiments, surveys, or observations.

2.3 Granularity

The level of detail built into a model. Models with a higher degree of granularity may provide more model precision or flexibility but may also require greater effort and expense to design, maintain, assemble, and run.

2.4 Implementation

An executable form of a model .

2.5 Input

Information such as data , assumptions, or parameters used in a model to produce output.

2.6 Intended Application

The designer’s planned uses for the model .

2.7 Intended Purpose

The intended application or the project objective or both, depending on the actuary’s role at the time actuarial services are performed. The intended application applies if the actuary’s role includes designing, building, or developing the model, or if the actuary’s role includes modifying, reviewing or evaluating the model before being selected or used in a specific project. The project objective applies if the actuary’s role includes selecting or using the model in a specific project or if the actuary’s role includes modifying, reviewing or evaluating the model when it is being selected or used in a specific project.

2.8 Model

A representation of relationships among variables, entities, or events using statistical, financial, economic, mathematical, or scientific concepts and equations. Models are used to help explain a system, to study the effects of different components, and to derive estimates and guide decisions. A model consists of: (1) a specification, (2) an implementation , and (3) one or more model runs.

2.9 Modeling

Selecting, designing, building, modifying, developing, using, reviewing, or evaluating models .

2.10 Model Risk

The risk of adverse consequences from decisions made as a result of a model that does not adequately represent that which is being modeled.

2.11 Model Run

The output of a model derived from a given set of input.

2.12 Parameters

Mathematical, financial, economic, scientific, or statistical input to models that, when varied, result in different model output. Examples include expected values and coefficients of variables in mathematical distributions or regression formulas.

2.13 Principal

A client or employer of the actuary.

2.14 Project Objective

The specific goal or question the actuary is addressing when selecting or using a model to meet the needs of the principal or the actuary.

2.15 Specification

A description of a model that identifies the inputs and their interactions with each other to produce output through logic, algorithms, or a set of mathematical formulas.

Section 3. Analysis of Issues and Recommended Practices

3.1 Application of ASOP Guidance

The guidance in this ASOP applies to actuarial practice regarding models in all practice areas, subject to the following.

3.1.1 Applicability of Guidance

Full application of the guidance in this ASOP is appropriate when, in the actuary’s professional judgment, intended users of the model rely heavily on the results and the use of the results of the model has a material financial effect. For example, corporate financial planning, ratemaking, and reserving models would typically require full application of the guidance. In assessing materiality, the actuary should be guided by ASOP No.1 , Introductory Actuarial Standard of Practice, section 2.6.

In modeling situations where the results are either not heavily relied upon or do not have material financial effect, full application of the guidance in this ASOP may not be necessary or practical. For example, efforts related to tasks such as data validation or sensitivity testing may not need to be as rigorous as stated in this ASOP.

In deciding the extent to which the guidance in this ASOP applies, the actuary should use professional judgment, considering the extent of reliance by the intended user and the materiality of the financial effect. This consideration should be made within the context of the use of the model results and the requirements of the principal, based on facts reasonably known by the actuary at the time the actuarial services are performed.

If, in the actuary’s professional judgment, circumstances are such that applying some or all of the guidance in this ASOP is not warranted for the specific intended purpose as described above, this is not considered a deviation. The actuary should be able to identify these circumstances, if asked.

If, in the actuary’s professional judgment, circumstances are such that applying some or all of the guidance in this ASOP is warranted but such guidance is not followed, this is considered a deviation. For example, even if following warranted guidance is not practical, failure to follow such guidance is considered a deviation.

In instances where a deviation from guidance is material, the actuary should disclose that deviation from guidance as addressed in section 4.2.

3.1.2 Models Developed by Others

If the actuary uses a model designed or built by someone else, such as a vendor or colleague, there may be limited ability to understand the underlying workings of the model. However, the actuary should make a reasonable attempt, given the project objective, to have a basic understanding of the model including the following:

a. the intended application of the model;

b. the general operation of the model;

c. major sensitivities and dependencies within the model; and

d. key strengths and limitations of the model.

3.1.3 Role of the Actuary on a Modeling Team

When the actuary is part of a modeling team, the actuary should confirm or may reasonably rely on others who have confirmed that the applicable guidance from this ASOP has been followed.

3.2 Model Meeting the Intended Purpose

The actuary should select, design, build, modify, develop, or use a model that meets the intended purpose. An actuary who is reviewing or evaluating a model should confirm that the model meets the intended purpose.

3.2.1 Designing, Building, Developing, Reviewing, or Evaluating the Model for the Intended Application

The actuary should confirm that the capability of the model is consistent with the intended application when the actuary designs, builds, develops, reviews or evaluates the model. In this confirmation, examples of items that the actuary should consider, if applicable, include but are not limited to the granularity of inputs, the relationships recognized, and the model’s ability to identify possible volatility around expected values.

3.2.2 Selecting, Reviewing, Evaluating, or Using the Model for the Project Objective

The actuary should select or use the model to meet the project objective, or review or evaluate the model and its use within this context. In the actuary’s use of the model, efforts to improve the model inputs and formulas, documentation, controls, validation, and presentation of results should be consistent with the project objective.

3.2.3 Modifying the Model

When modifying a model to change the intended application or to improve the model’s ability to meet its intended application, the actuary should be guided by section 3.2.1. When modifying a model to improve the model inputs, formulas, and outputs to meet the project objective, the actuary should be guided by section 3.2.2.

3.2.4 Understanding the Model

The actuary’s responsibilities may include expressing an opinion, using or communicating results, or preparing documentation based on or in relation to a model. In these instances, the actuary should understand:

a. important aspects of the model being used, including but not limited to, basic operations, important relationships, major sensitivities, strengths and potential weaknesses; and

b. whether, and the extent to which, the model can fulfill its intended purpose, given limited information, time constraints, and other practical considerations.

3.2.5 Model Structure

The actuary should evaluate whether the structure of the model is appropriate for the intended purpose. Where applicable and where appropriate for the model’s intended purpose, the actuary’s considerations should include the following:

a. which provisions and risks specific to a business segment, contract, or plan are material and appropriate to reflect in the model;

b. whether grouping model inputs will produce reasonable results;

c. whether the use of the model requires a particular level of granularity;

d. whether deterministic or stochastic results, or both, are needed; and

e. whether the projection of future results might be materially influenced by the existence of choices and options available to the entity that is being modeled in whole or in part, its members, or its counterparties.

3.2.6 Inputs to the Model

The actuary should refer to ASOP No. 23, Data Quality, when selecting, reviewing, or evaluating data to be used in the model ,either directly or as the basis for deriving assumptions and parameters.

3.2.7 Assumptions and Parameters

The actuary should use assumptions and parameters that are appropriate in light of the model’s intended purpose.

a. Experience Reflected in Setting Assumptions and Parameters—When setting assumptions and parameters, the actuary should consider using the following:

1. assumptions and parameters based on actual experience, to the extent it is available, relevant, and sufficiently reliable;

2. other relevant and sufficiently reliable experience, such as industry experience that is properly modified to reflect the circumstances being modeled, if actual experience is not available or relevant, or is not sufficiently reliable; and

3. professional judgment to modify other available sources of information.

b. Margins—The   actuary   should   determine   whether   adjusting the assumption or parameter to include a margin having a material effect would be appropriate. A margin might be included for reasons such as a) experience data that are not fully credible, b) conservatism, c) an adjustment for the cost of bearing risk, or d) future unpredictability.

c. Range of Assumptions and Parameters—The actuary should consider whether the range of assumptions and parameters used and the number of model runs analyzed reflect a range of conditions consistent with the intended purpose.

d. Consistency—The actuary should use assumptions and parameters for the model that are consistent with one another. For example, where appropriate, the actuary should use assumptions consistent with the underlying economic scenario(s) assumed in the model .

If material inconsistency among assumptions and parameters used by the actuary exists, whether (i) required by legal constraints or by the principal, (ii) the result of intentional redundancy such as added conservatism, or (iii) for any other reason, the actuary should disclose the inconsistency and the reasons for it in accordance with section 4.1.2. However, in the case of assumptions prescribed by applicable law (statutes, regulation, or other legally binding authority), the actuary’s disclosure may be limited to identifying the possibility of an inconsistency with other assumptions.

e. Appropriateness of Input in Current Model Run—Where practical and appropriate, the actuary reusing an existing model should evaluate whether the input is still appropriate for use in the current model run. For example, models used in financial reporting offer frequent opportunities to compare assumptions and parameters to emerging experience in the aggregate.

3.3 Mitigation of Model Risk

The actuary should examine the potential for model risk and undertake reasonable and appropriate steps to mitigate such risk, using validation, governance, and controls,as appropriate to the intended purpose.

3.3.1 Validation

The nature and degree of validation (including checking and analysis) selected by the actuary should be consistent with the complexity of the model and the intended purpose.

a. Model Integrity—For each model run (or set of model runs ) that is to be relied upon by the intended user, the actuary should validate that the model properly represents that which is being modeled. Validation of the model could include, but is not limited to, the following:

1. are conciliation of relevant model input values to actual data , addressing and documenting the differences appearing in the reconciliation, if material;

2. checking formulas, logic, and table references. The degree of checking that is appropriate will depend on the intended application; the project objective for which the model is being used; the context and nature of the model ; the operating environment and controls; and whether there have been any changes to the model or the model environment; and

3. where applicable, testing the model projection results against historical actual results to verify that modeled results bear a reasonable relationship to actual results over a given time period.

b. Analyzing the Output—The actuary should take appropriate steps to evaluate whether the model results are reasonable. Depending on the project objective, the actuary should consider the following:

1. performing analytical tests on model results to assess their reasonableness;

2. reconciling the results of a model run to prior model runs , given any changes in assumptions and parameters, data, formulas, or other aspects of the model since the prior model run. If such reconciliation is developed and appropriate to the project objective, the actuary should consider retaining the reconciliation;

3. running tests of variations on key assumptions and parameters to test that the model has been used correctly and that changes in the results are consistent with the changes in those assumptions and parameters; and

4. comparing model results to those of alternative model(s).

c. Peer Review—The actuary should consider obtaining a peer review, where practical and appropriate, depending on the intended purpose and the actuary’s role. Such peer review, if obtained, may include items such as review of the reasonableness of the input to the model, the model construction, and the model results.

3.3.2 Appropriate Governance and Controls

The actuary should use or, if appropriate, rely on others to use appropriate governance and controls to minimize model risk, to maintain the integrity of the model, and to avoid the introduction or use of unintentional or untested changes.

3.4 Presentation of Results

When the actuary presents results of the model, the actuary should explain methodology, key assumptions and parameters, possible limitations, and any material changes in any of these that were made since the most recent comparable model results were communicated.

3.4.1 Explanation of Limitations of Models

In actuarial reports that include information derived from models, the actuary should include explanations of the following, if applicable:

a. the extent to which a model fails to fulfill its intended purpose, due to limited information, time constraints, or other practical considerations; and

b. any other material limitations of the models that have been used and the implications of those limitations.

If there is anything to explain pursuant to (a) or (b), then the actuary should refer to section 4.1.

3.4.2 Discussion of Models

In actuarial reports that include information derived from models, the actuary should consider including explanations of the following:

a. the intended purpose of the models and how the users’ needs are addressed by those models;and

b. any uncertainty in model results.

3.4.3 Reconciliation

The actuary should consider including in the actuarial report a reconciliation to comparable items in a prior actuarial report. Such reconciliation, if any and where reasonably possible, should include an explanation of assumptions and parameters or methods that have changed materially from that prior actuarial report.

3.4.4 Description of Conservatism or Optimism

The actuary should consider including a description of the conservatism or optimism inherent in the model inputs and methodology selected in relation to anticipated future experience. Terminology may include language such as“conservative,”“most likely,” “reflecting asymmetric outcomes,” or “optimistic,” along with a description of the relationship to the anticipated future experience by appropriate quantitative, qualitative, or directional language.

If applicable law (statutes, regulations, and other legally binding authority) specifies the model inputs or methodology, then this section 3.4.4 does not apply.

3.5 Reliance on Data or Other Information Supplied by Others

When relying on data or other information supplied by others, the actuary should refer to ASOP Nos. 23 and 41 for guidance. When relying on projections or supporting analysis supplied by others, the actuary should refer to ASOP No. 23, deeming such projections or supporting analysis as data covered by that standard. Similarly, the actuary should refer to ASOP No. 41 with respect to the disclosure of responsibility for data, assumptions, parameters , and methods.

3.6 Documentation

For model results used in actuarial communications,the actuary should document the nature of the data used, and material assumptions and parameters used in the model and should follow the guidance of ASOP No. 41, including its section 3.2 in the case of an actuarial report.

The actuary should consider documenting the items mentioned in sections 3.4.1 and 3.4.2 of this standard, even if no actuarial report is created.

3.7 Relation to Other ASOPs

Other ASOPs provide specific modeling requirements, including guidance on selecting assumptions, parameters, and data (see ASOP No. 23) and providing disclosures (see ASOP No. 41). The actuary selecting, designing, building, modifying, developing, reviewing, evaluating, or using models should satisfy not only the requirements of this ASOP, but also any specific modeling requirements from an applicable ASOP. If such specific modeling guidance from an applicable ASOP is inconsistent with the guidance of this ASOP, the guidance of such other ASOP governs.

Section 4. Communications and Disclosures

4.1 Actuarial Communications

In any actuarial communication that uses the results of work subject to this ASOP, the actuary should disclose the following, as applicable:

4.1.1 Failure to Meet Intended Purpose

Any reasons that prevent the model from meeting its intended purpose, as discussed in sections 3.2.4 and 3.4.1. In this situation, the actuary should disclose the intended purpose of the model.

4.1.2 Inconsistent Assumptions and Parameters

Any material inconsistencies among assumptions and parameters and the reasons for such inconsistencies, as discussed in section 3.2.7(d).

4.2 Deviation from Guidance in the Standard

In any actuarial communication that uses the results of work subject to this ASOP,  the actuary should refer to ASOP No. 41 and should include the following where applicable:

a. the disclosure in ASOP No. 41, section 4.2, if any material assumption, parameter, or method was prescribed by applicable law (statutes,regulations, and other legally binding authority);

b. the disclosure in ASOP No.41, section4.3, if the actuary states reliance on other sources and thereby disclaims responsibility for any material assumption, parameter, or method selected by a party other than the actuary; and

c. the disclosure in ASOP No. 41,section 4.4, if, in the actuary’s professional judgment, the actuary has otherwise deviated materially from the guidance of this ASOP.

Appendix 1

Note: This appendix is provided for informational purposes but is not part of the standard of practice.

Current Practices

Models are used to help explain a system; to study the effects of different components; and to derive estimates and guide decisions. Models have always played a fundamental role in actuarial work with every discipline relying on a broad range of modeling applications, ranging from simple spreadsheets to complex capital models. The number and importance of modeling applications in actuarial science have continued to increase, with the results of actuarial models often entering financial statements directly.

Actuaries often develop and use models when analyzing uncertain outcomes. In these instances, even a model that is prudently developed and carefully used does not eliminate inherent uncertainty and variability and actual experience may differ, sometimes significantly, from the estimates derived from the model results. A model is only an approximation of reality, not the reality itself, and the differences between the model and actual experience, by themselves, do not indicate a flawed model or noncompliance with standards.

When a model will be used repeatedly, it is common that the model will be subject to appropriate governance and controls. Examples of model governance and controls include the following:

• limitations on access to use and modify the model (that is, restricting access to model inputs, model code and calculations, and model outputs);

• confirmation that model results are reproducible upon rerun (if the model allows for such reproducibility);

• implementing a model change management process;

• specification, documentation, and programming standards for the implementation;

• procedures for secure back-up of the media storing the implementation and data;

• appropriate staff training or cross-training for continuity of use;

• plans for periodic consideration of the organization’s continued ability to access and maintain the model, including data, software, staff, hardware, and vendor relationships;

• plans for periodic updating of model input; and

• plans for periodic review of the assumptions, parameters, functionality, and methodology.

Appendix 2

Comments and Exposure Draft Responses

The first exposure draft of this proposed ASOP, Modeling, was issued in June 2013 with a comment deadline of September 30, 2013. Forty-eight comment letters were received, some of which were submitted on behalf of multiple commentators, such as by firms or committees. Some commentators submitted multiple letters. For purposes of this appendix, the term “commentator” may refer to more than one person associated with a particular comment letter. The Modeling Task Force carefully considered all comments received, reviewed the exposure draft, and proposed changes. The General Committee and the ASB reviewed the proposed changes and made modifications where appropriate.

Click here to view Appendix 2 in its entirety.

COMMENTS

Click here to view Comments in their entirety.