Topics in Verification: Reuse, Coverage, Regression Engineering, Planning, Qualification

Topics in Verification: Reuse, Coverage, Regression Engineering, Planning, Qualification

• 1.
  Welcome to RTPWelcometo RTP DVClubDVClub!! Topics in VerificationTopics in Verification Pete LaFauciPete LaFauci October 18, 2006October 18, 2006
• 2.
  AgendaAgenda Brief Discussion/Overview aboutBriefDiscussion/Overview about ““the Clubthe Club”” Historical Perspective of Languages & SimulatorsHistorical Perspective of Languages & Simulators Topics in VerificationTopics in Verification ReuseReuse CoverageCoverage Regression EngineeringRegression Engineering PlanningPlanning QualificationQualification SummarySummary
• 3.
  DVClubDVClub OverviewOverview Voluntary, CommunityBased Verification GroupVoluntary, Community Based Verification Group Discuss Present Day ChallengesDiscuss Present Day Challenges EDA neutralEDA neutral Presentation ofPresentation of ““sharablesharable”” ideas, experiences, and resultsideas, experiences, and results Advisory BoardAdvisory Board Currently about 10 members, spanning 6 companies (looking forCurrently about 10 members, spanning 6 companies (looking for additional participants)additional participants) Planning, Logistics, Speakers, TopicsPlanning, Logistics, Speakers, Topics Topics should be Chip Verification related, but can spanTopics should be Chip Verification related, but can span adjacent areas:adjacent areas: Design for Verification, IT, Project ManagementDesign for Verification, IT, Project Management Please Contact Pete LaFauci or Justin Sprague regarding interestPlease Contact Pete LaFauci or Justin Sprague regarding interest in the Advisory Board or Presenting atin the Advisory Board or Presenting at DVClubDVClub!!
• 4.
  Historical Perspective: WhatHistoricalPerspective: What’’s dones done Produce some test vector stimulus, and simulateProduce some test vector stimulus, and simulate Look at the output and waveforms on workstation, debug the restLook at the output and waveforms on workstation, debug the rest in the labin the lab Produce a test plan and lots of directed, procedural test casesProduce a test plan and lots of directed, procedural test cases Run each test case in the test bucketRun each test case in the test bucket Completion: each individual test case passesCompletion: each individual test case passes Produce a verification plan, which includes modelProduce a verification plan, which includes model requirements, functions & coverage goalsrequirements, functions & coverage goals Combine generation capabilities, checkers, randomness, and coverCombine generation capabilities, checkers, randomness, and coverage monitors toage monitors to gauge the simulationsgauge the simulations Run tests with multipleRun tests with multiple ““seedsseeds”” Completion: all tests pass, and coverage output isCompletion: all tests pass, and coverage output is ““analyzedanalyzed”” Produce a verification methodology,Produce a verification methodology, ““architectarchitect”” aa verification environment & plan, and leverage powerful toolsverification environment & plan, and leverage powerful tools Automate both efficiency and thoroughness, through prediction, sAutomate both efficiency and thoroughness, through prediction, steering, and proofsteering, and proofs VIP, Environment, Library, and TestVIP, Environment, Library, and Test ““ReuseReuse”” ““RankRank”” the test suite efficiency, directed tests for coverage closurethe test suite efficiency, directed tests for coverage closure Completion: all tests pass, coverage goals are met using measuraCompletion: all tests pass, coverage goals are met using measurable metricsble metrics Then Now
• 5.
  Historical Prevalence:Historical Prevalence: VerificationLanguagesVerification Languages VHDL Records / Dynamic Variable Indices & Slicing 2-d ports HVL – ‘e’ & Vera Constraint Solvers Easy RTL Access Functional Coverage Built-in libraries C, C++. Perl Complex Data Structs Superior String Functions Verilog Concurrency Tasks & Functions PLI Early ’90s Late ’90s – Early 00’s Mid ’90s Today / Tomorrow / Future? Modeling – SystemC Higher Abstraction, TLM, HW/SW Algorithmic Modeling & Prototype HVDL – SystemVerilog RTL/HVL Integration/Unification SystemC & TLM Interfaces Assertion Based Verification DPI
• 6.
  ReuseReuse
• 7.
  Reuse: Going BeyondTraditional VIPReuse: Going Beyond Traditional VIP ““ComponentsComponents””: traditional VIP: traditional VIP Common bus and data protocolsCommon bus and data protocols Monitors, Predictors, DriversMonitors, Predictors, Drivers Environments:Environments: new(ernew(er) with OOP/HVL) with OOP/HVL TestbenchesTestbenches && ““HarnessesHarnesses”” Constraints & Test CasesConstraints & Test Cases Configuration ComponentsConfiguration Components Addresses ScalabilityAddresses Scalability “instance & program” “inherit, derive, build, configure”
• 8.
  Reuse: Inheritance andAbstractionReuse: Inheritance and Abstraction Base Class Packages DUT templateTest Harness Globals MemoriesRegistersData Objects Interface & Protocol templates Harness Drivers Scoreboard User Test Case & Constraints Environment Test Methods and User Interfaces Test Benches/Harnesses DUT VIP Tool Packages: any_unit, any_env, any_sequence Application Specific Verification Environment Company Library Inheritance And Instantiations Base Library Protocol VIP Interface VIP Note : “Horizontal (Interface/Protocol) and Vertical (Base Classes and Methods) Reuse Strategies are Deployable using OO Design Techniques”
• 9.
  CoverageCoverage
• 10.
  Coverage: TypesCoverage: Types CodeCoverageCode Coverage Assertion & Formal CoverageAssertion & Formal Coverage Functional CoverageFunctional Coverage Integration CoverageIntegration Coverage System & Validation CoverageSystem & Validation Coverage
• 11.
  Coverage: AchievingCoverage: Achieving ThoroughnessThoroughness Identifywhich Design Functions have not been exercisedIdentify which Design Functions have not been exercised Identify What Test Code has not been simulatedIdentify What Test Code has not been simulated Complements the Checking Code (traffic types, latency,Complements the Checking Code (traffic types, latency, performance, etc)performance, etc) Structural Coverage Assertions Code Coverage Functional Coverage
• 12.
  Coverage: Collection, Merging,Coverage:Collection, Merging, and Reportingand Reporting Project Level Coverage Progress Reports Regression 1 Test Cases, Procedural Code Test Cases, Procedural Code Test Cases, Procedural Code Test Cases, Procedural Code Simulation Coverage Output (Cover DB) Coverage Data Merging “Process” Test Cases, Procedural Code Test Cases, Procedural Code Test Cases, Procedural Code Test Cases, Procedural Code Simulation Coverage Output Regression n “Total Coverage” Reports & HTML views Test Cases, Procedural Code Test Cases, Procedural Code Test Cases, Procedural Code Test Cases, Procedural Code Simulation Coverage Output (Cover DB)
• 13.
  Coverage: Challenges inCoverageCoverage: Challenges in Coverage Driven MethodologyDriven Methodology Goals are manually created from the engineeringGoals are manually created from the engineering SpecsSpecs --> Labor Intensive> Labor Intensive Random generation can be wasteful if overallRandom generation can be wasteful if overall coverage is not increasingcoverage is not increasing Simulation cycles can be slow and/or expensive,Simulation cycles can be slow and/or expensive, especially if they are being wastedespecially if they are being wasted Execution Predictability with Schedules &Execution Predictability with Schedules & ResourcesResources
• 14.
  Coverage: Execution PredictabilityCoverage:Execution Predictability Tigr is 2.0Test Cover ageCommit 1500 1700 1900 2100 2300 2500 2700 2900 Total Planned Wr ittenActual Wr ittenCommit Cover edActual Cover edCommit Coverage Progress Chart: “S” Curve Phenomena
• 15.
  RegressionRegression ““EngineeringEngineering””
• 16.
  RegressionRegression ““EngineeringEngineering””:: ChallengesChallenges Manage thousandsof simulation jobs & resultsManage thousands of simulation jobs & results Multiple environmentsMultiple environments Many Servers, Lots of output!Many Servers, Lots of output! Team environment, Individual OwnersTeam environment, Individual Owners AnalysisAnalysis Reproducing test fails easily & accuratelyReproducing test fails easily & accurately --> test/bench, machine, seed, tools,> test/bench, machine, seed, tools, versionsversions More debug output necessary?More debug output necessary? Correctly assigned or delegated to appropriateCorrectly assigned or delegated to appropriate owner(sowner(s)) EfficiencyEfficiency Dynamically Allocating Right Machines for the Right Job TypesDynamically Allocating Right Machines for the Right Job Types Maximize Utilization of both Hardware and SoftwareMaximize Utilization of both Hardware and Software Coverage Closure: Hole Analysis, RankingCoverage Closure: Hole Analysis, Ranking 24/7 rebalancing, including interactive session priority managem24/7 rebalancing, including interactive session priority managementent
• 17.
  PlanningPlanning
• 18.
  Planning: Improvements inPlanPlanning: Improvements in Plan Creation and Change ManagementCreation and Change Management More Tightly Couple the Design Specification with theMore Tightly Couple the Design Specification with the Verification PlanVerification Plan Change Management SystemChange Management System More Automatic Synchronization and NotificationMore Automatic Synchronization and Notification Improve the Efficiency between Coverage Plan andImprove the Efficiency between Coverage Plan and ImplementationImplementation Reduce or Remove the Disconnect between the FunctionalReduce or Remove the Disconnect between the Functional Coverage Code and the Coverage PlanCoverage Code and the Coverage Plan Automate Functional Coverage CodeAutomate Functional Coverage Code ““GenerationGeneration””
• 19.
  QualificationQualification
• 20.
  Qualification: WhatQualification: What’’sthis?s this? What is Qualification?What is Qualification? WhoWho’’s checking the verification, and when?s checking the verification, and when? Error insertion to prove the accuracy and effectiveness of yourError insertion to prove the accuracy and effectiveness of your verification systemsverification systems Error insertion is placed in the DUT (not verification code)Error insertion is placed in the DUT (not verification code) Challenges in adoption:Challenges in adoption: ““how, what, whenhow, what, when””, effort levels, effort levels Benefits for SimulationBenefits for Simulation Qualify the Test BenchQualify the Test Bench Qualify the Models & VIPQualify the Models & VIP Qualify the Test Cases & ConstraintsQualify the Test Cases & Constraints Benefits for Formal ProofsBenefits for Formal Proofs Qualify the AssertionsQualify the Assertions Qualify the ConstraintsQualify the Constraints *Qualify the Answer the Tool is Giving you**Qualify the Answer the Tool is Giving you*
• 21.
  SummarySummary Combining innovative techniques,tools, processes canCombining innovative techniques, tools, processes can yield significant results!yield significant results! NotNot ““uncommonuncommon”” to achieve 10x productivity andto achieve 10x productivity and thoroughness improvements through methodology andthoroughness improvements through methodology and tuningtuning Verification processes will continue to changeVerification processes will continue to change considerably to keep up with managing the exponentialconsiderably to keep up with managing the exponential growth in design complexitygrowth in design complexity
• 22.
  Interesting Materials, Good Food, Catchup with other Engineers in the Industry Invite your Co-Workers! www.dvclub.org