SAS - Certified ModelOps Specialist

Curriculum

Module 1: Define the ModelOps Framework (18 – 22%)

• Emphasis on governance and stakeholders
• Determine relevant stakeholders based on desired business outcome
• How will collaboration be achieved
• What are standard channels of communication
• Determine the business process workflow
• Outline feedback loops and decision validation
• Articulate handoffs and individual roles/responsibilities
• Establish an approval process
• Create a peer review process
• Validate assumptions
• Review for potential bias
• Test against hold out data
• Adherence to compliance and standards
• Confirm the process is documented
• Create a business review process
• Adherence to compliance and standards
• Verify consumption method
• Make a Go/No-Go decision
• Define audit and tracking needs
• Automated vs. Manual tasks
• Batch vs. OnDemand vs. Streaming
• Identify third party integration points
• CI/CD
• Model development tools
• Git
• Mobile Apps Develop workflow template(s) to execute the process
• Model Type
• Supervised
• Unsupervised
• Scoring Environments
• Cloud
• On-prem o Containers o In-database o Edge

Module 2: Develop Model Proposal (23 – 28%)

• Define the business objective
• Define the business objective
• Explicitly define the target variable
• Assess business impact
• Identify potential consumption methods
• Verify alignment with corporate strategy and policies Associated assets to include
• Cost/benefit analysis
• Optimal business value
• Time to return on investment
• Level of risk
• Caveats/Assumptions
• Finalize consumption method Evaluation against alternative resource allocation Establish preferred deployment method Establish monitoring assessment metrics
• Business outcome metrics
• Optimal model accuracy fit statistics
• Model monitoring frequency and evaluation criteria
• Operational performance metrics

Module 3: Model Development and Management (23 – 28%)

New or Re-build Model Development

• Pre-processing – data extracts and data preparation
• Model training
• Post-processing
• Integration with third party software Model tournament
• Evaluate and compare multiple candidate models
• Utilize fit statistics presented in Business Proposal
• Consider balance of fit vs. model interpretability
• Alignment to corporate ethics and data privacy standards
• Select a champion model Execute peer review Execute business review
• Exporting score code
• Registering a model
• Publish a model Model registration
• Metadata o Input/output variables
• Properties
• Artifacts
• Score code
• Pre/post processing code
• Associated external files and documentation Establish the monitoring process
• Build data extracts for input data
• Create metrics to monitor
• Model metrics
• Business metrics
• System metrics
• Setup thresholds and alerts
• Develop monitoring reports
• Document the process and parameters
• Prepare a model for use in production

Module 4: Model Deployment (18 – 22%)

• Develop the data ingestion pipeline
• Confirm naming conventions/standards used
• Review adherence to architecture standards
• Process request for additional data source (if needed)
• Build data extract for monitoring
• Define production jobs
• Create consumption application(s) (on-demand/streaming)
• Integrate with decisioning process
• Setup scoring, includes pre-/post-processing
• Setup performance data creation
• Setup performance monitoring reports
• Model accuracy
• Business outcome
• Embed alerts
• Develop follow-up actions
• Model updates based on performance monitoring
• Business decisions IT compliance check
• Validate regulatory compliance
• Cross-check against standards/policy Pre-production Testing
• Unit testing
• Regression testing
• Performance testing

Module 5: Production (9 – 12%)

• Execute prep-processing
• Execute scoring/decision
• Execute post-processing
• Execute business actions associated with score/decision
• Execute performance monitoring
• Evaluation
• Diagnose data drift
• Assess model accuracy
• Business outcomes
• Operational impact
• Alerts based on thresholds
• Model update (Recalibrate/Retrain/Rebuild/Retire)