Risk-based and compliance-focused model validation

Map the usage boundaries of your models
Maximize your stakeholders' confidence

Map the low-confidence regions and the decision boundary margins of your models for safe, ethical, and high-performing decisions

Are your models truly reliable in all circumstances?

Machine learning is redefining decision-making, but a model used outside its training context can generate biases, costly errors, or legal risks. Without a clear understanding of its usage limitations, you can leave the door open to uncertainty and risky predictions.

Questions to ask yourself

  • Is your model doing what it's supposed to do?

  • Is it doing it for the right reasons?

  • Are usage limits set every time your models are retrained?

  • Are low-confidence regions and decision boundary margins well communicated to users?

  • Do users have a good understanding of the information given?

Sources of modeling risk

  • Inadequate specifications

  • Insufficient volume of data

  • Inaccurate, inadequate, or incomplete data

  • Overfitting / Underfitting

  • Biased or unrealistic assumptions

  • Poor hyperparameter estimates

  • Inaccurate mathematical calculations

  • Lack of model performance monitoring through production

Preventing modeling risk

  • Erroneous decisions

  • Bias and discrimination

  • Loss of stakeholder trust

  • Non-compliance with regulations

  • Inability to defend decisions made by the model

  • Costs related to errors

  • Unforeseen expenses to correct problems

Growing requirements for model risk management

  • Identification of risks associated with models

  • Documentation of usage limits

  • Independent and continuous validation of model performance

A solution to assist in model validation for risk management, compliance, and audit teams

  • Even the best machine learning models have areas of uncertainty that limit their use.

  • Insightbounds assists all stakeholders involved in model risk management (MRM) by providing them with crucial information on the contextual uncertainty of predictions.

  • This information allows them to validate their domains of use from a business, risk, compliance, and audit perspective, thus facilitating more enlightened and informed decisions.

  • For data science teams, Insightbounds enables the extraction of useful knowledge to refine original models and improve their performance.

Where do your models perform well?

  • Insightbounds maps the low-confidence regions and the decision boundary margins of your models to detect where their predictions lose reliability compared to where they perform effectively.

  • When you evaluate multiple models, Insightbounds precisely identifies which one excels in each data context.

Understanding why it works… or not

  • Why does a model work in one context and fail in another?

  • Insightbounds provides a granular analysis of the root causes behind your model’s underperforming regions.

  • Discover the combinations of variables and intervals in which your models’ accuracy is most sensitive to changes.

Comparing models

  • When comparing models, Insightbounds allows you to systematically analyze prediction discrepancies and perform tests to verify if the differences are statistically significant.

  • By comparing results obtained on the same validation set, you identify data domains where models perform differently and scenarios where one candidate model outperforms another.

Evaluating the economic performance of the model

  • Based on the distribution of errors and the individual cost of prediction errors, the solution identifies regions of maximum loss and quantifies the associated losses.

  • With each retraining or modification of the model, its economic performance is reassessed by incorporating the new data distribution and/or the changes made.

Overview of the solution

Prepare the validation dataset needed

  • The dataset must contain the same features used to train the model, with the same transformations or normalizations applied.

  • The first line of the file must correspond to the header listing the variable names.

  • The Predict and Actual columns must not contain any missing values.

  • The dataset can be provided in CSV or XLSX format (without calculation formulas).

  • The dataset package must include a description of the features.

  • The validation data must be distinct from the training datasets to avoid bias and ensure an independent evaluation.

  • The dataset should reflect the actual distribution of data that the model will encounter in production, in terms of variety, target classes, and feature distributions. If the real data is imbalanced, the validation set should also reflect these proportions.

  • The dataset must contain a sufficiently large number of observations to provide a statistically significant evaluation.

  • It must include examples from all relevant subpopulations to avoid algorithmic biases related to underrepresented groups.

  • If possible, the dataset should include edge cases, disruptions, or typical anomalies to test the model’s robustness.

Book a demo and discover how Insightbounds can accelerate your validation processes

Pricing of the solution

Rely on our solution to accelerate and enhance the reliability of your model validation processes

Core

Features :

  • Analysis of statistical dependencies within your validation dataset

  • Audit of the signal contained in explanatory variables

  • Identification and description of regions where the model generates errors repeatably

  • Calculation of statistical metrics evaluating risky regions

499 $ / month

Standard

Features :

  • All Core features, plus:

  • Automated documentation of model usage limits (LLM)

  • Visualization of regions where the model is not reliable

  • Comparison of risky regions for different versions of the model

  • Evaluation of the economic performance of your model

999 $ / month

Pro

Features :

  • All Standard features, plus:

  • Connection to the model’s API

  • Calibration of uncertainty levels related to predictions (conformal prediction)

  • Identification of the model’s risky regions according to the level of uncertainty (calculation of region boundaries by alpha level)

1,999 $ / month

For organizations wishing to make a group subscription :

  • We offer customized pricing options and support for group subscriptions.

  • A 30% discount is offered for a subscription of 10 licenses.

Managed Services

Do you want to free your in-house teams from complex model validation tasks?
Our experts can produce regulatory deliverables on a regular basis for you.

Mapping Regions at Risk

  • Identifying input data ranges where the model performs well and regions at risk

  • Re-evaluating confidence zones and regions at risk through regular validations

  • Notifying in case of detected risk (decision located in low-confidence regions, out-of-range data, …)

Regulatory Checks

  • Defining recommended and prohibited usage contexts

  • Verifying alignments with regulations and internal standards

  • Verifying compliance with fairness principles

  • Documenting model underlying assumptions

Producing Recommendations

  • Drafting a contingency plan and corrective actions to implement if unexpected results or insufficient model performance

  • Proposing model improvement or replacement

  • Defining max. loss regions and calculating losses caused by model errors or limitations

Team

Michael Albo

MSc Computer Science, MBA

Having built his experience on both sides of the Atlantic, Michael has been working for 30 years at the intersection of computer science, decision-making mathematics, and finance.
For over 10 years, he has been designing and deploying machine learning models in production to optimize his clients' critical processes.
Michael is a Canadian expert in auditing and validating machine learning models.

David Streliski

MSc, CFA, Qualified Risk Director

With 28 years of experience, David leads business and technology projects for major financial institutions, investment firms, and audit firms in Canada and Switzerland.
As an HEC Montréal former lecturer and governance, risk, compliance, and quantitative finance expert, David is a privileged interlocutor for addressing challenges related to the design, validation, monitoring, and auditing of models.

Noé Aubin-Cadot

MSc Mathematics

Data scientist and mathematical modeling expert,
Noé spent six years during his doctorate working on a range of cutting-edge mathematical theories and numerical methods for simulating physical phenomena. For the past 5 years, he has been developing machine learning models and applying his expertise to address business problems for Canadian companies across various sectors.

Privacy Policy

Last updated: December 2, 2024

Insightsolver, designer of the Insightbounds solution, is committed to protecting the privacy of its prospects and users and to comply with applicable personal information protection laws. This policy explains how we collect, use, and protect your personal information in the context of our services.

1. Collection of personal information

We collect your personal information only when necessary to provide our services, improve your user experience, or comply with our legal obligations. The types of information we collect include, but are not limited to:
• Identification information: Name, first name, email address, phone number, organization.
• Data usage: IP address, browser type, preferences, interactions with our services.
• Financial information (if applicable): Credit card number or billing details.We collect this information directly from you when you:
• Create an account on our platform.
• Use our services or features.
• Contact our support service.

2. Purposes of collection

Personal information is used for the following purposes:
• Provide and personalize our services.
• Manage payments and billing.
• Communicate with you regarding our services, updates, or technical support.
• Comply with our legal and regulatory obligations.
• Improve our products and analyze the use of our platform.

3. Consent

We obtain your explicit consent before collecting, using, or disclosing your personal information, except in cases provided for by law. You can withdraw your consent at any time by contacting us at the address mentioned below. However, this may limit your access to certain services.

4. Sharing of personal information

We share your personal information with third parties in the following circumstances:
• Service providers: Technology partners or third-party providers necessary to provide our services (cloud hosting, payment operator, technical support platform).
• Legal obligations: When required by law or in the context of legal proceedings.

5. Data security

We apply physical, administrative, and technological security measures to protect your personal information against unauthorized access, use, or disclosure.
In the event of a confidentiality incident presenting a serious risk of harm, we will inform the Commission d'accès à l'information du Québec and the persons concerned in accordance with the law.

6. Your rights

You have the following rights regarding your personal information:
• Access: Obtain a copy of the information we hold about you.
• Rectification: Correct any inaccurate or incomplete information.
• Withdrawal of consent: Withdraw your consent at any time.
• Deletion: Request the deletion of your personal information, subject to legal obligations.
To exercise these rights, please contact us at the address mentioned in section 8.

7. Information retention

We retain your personal information only for the duration necessary to achieve the purposes stated in this policy, unless a longer retention period is required or permitted by law.

8. Contact us

For any questions or requests related to this privacy policy or your rights, you can contact us:Postal address:
Insightsolver
4 Pl. Ville-Marie Suite 300,
Montreal, Quebec H3B 2E7Email address: [email protected]

9. Changes to this policy

We reserve the right to modify this policy at any time. Any changes will be posted on our platform. Please check this page regularly to stay informed of updates.

Terms of Use

Last updated: December 2, 2024

Welcome to the Insightbounds website ("the Site"). By accessing or using the Site, you agree to comply with these Terms of Use. Please read them carefully. If you do not accept these terms, please do not use the Site.

1. Acceptance of terms

By using this Site, you confirm that you have reached the age of majority in your province of residence and are legally capable of entering into a contract.

2. Use of the Site

You agree to use the Site only for legal purposes and in compliance with the laws in force in Quebec and Canada. It is prohibited to:
• Download, transmit or disseminate any harmful, illegal, or offensive content.
• Attempt to compromise the security of the Site or disrupt its operation.

3. Intellectual property

All content on the Site (texts, images, logos, videos, etc.) is protected by copyright and belongs to InsightSolver or is used with authorization.
It is prohibited to reproduce, distribute or modify any content without prior written authorization.

4. Confidentiality

The use of this Site is also subject to our Privacy Policy. We encourage you to read it to understand how we collect, use, and protect your personal information.

5. External links

Our Site may contain links to third-party sites. InsightSolver does not control these sites and disclaims any responsibility for their content or practices.

6. Disclaimer of warranties

The Site and its contents are provided "as is", without any express or implied warranty, particularly regarding their accuracy or availability at all times.

7. Limitation of liability

In no event shall InsightSolver be held liable for direct or indirect damages resulting from the use or inability to use the Site.

8. Modification of terms

InsightSolver reserves the right to modify these Terms of Use at any time. The modifications will take effect upon their publication on the Site. We encourage you to consult them regularly.

9. Applicable law and jurisdiction

These Terms of Use are governed by the laws of the province of Quebec and applicable federal laws of Canada. Any dispute will be subject to the exclusive jurisdiction of the courts of the province of Quebec.

10. Contact

For any questions regarding these Terms of Use, please contact us at:
Insightsolver
4 Pl. Ville-Marie Suite 300,
Montreal, Quebec H3B 2E7
Email: [email protected]