CincyPy Presentation for Requirements Tracer Server

Introduction

In this presentation I will be showing:

The concept of requirement traceability and the value I see in it.
A prototype web application to automate some of the process of achieving traceability.

Questions

These are some common questions I faced pretty often as a developer:

"Is this product behavior correct?"
"What is this code supposed to do?"
"Are we testing this feature?"
"Do we still need this functionality?"
"What data can I expect to receive? What should I produce? Is this field necessary?"

How does this happen?

Large, legacy codebase.
Distributed systems, distributed teams.
Bitrot, people coming and going, silos and braindrain.

These questions are productivity killers.

Lower confidence of making and deploying changes.
Generate meetings and inter-team conflict.
Encourages rewriting systems without solving the core problem.

Requirement Traceability

Idea: maintain a database of requirements and tests, and link them together.

Answers

Building a robust database of relationships between requirements and tests allows you to answer:

What requirements are untested or undertested? How often is this requirement tested?
What defects have been verified as fixed? What part of a system did a defect affect?
What tests/suites cover which requirements? What tests would comprise a good regression test?
What requirements are the cause of the most defects? (Pareto Analysis)
What code affects which requirements?
What manual tests are due to run and possibly update?
... And all of the questions listed above.

So how do we do it?

Requirement traceability can be tough.

Maintaining this database is a time consuming, tedius, error prone task.
Like documentation: as soon as it is written, it is stale.
There's often a disconnect between developers and project managers.

But it is possible, with:

Spreadsheets, wikis, and other shared documentation.
Project management tools like Jira.
Homegrown/legacy/specialized requirements analysis software.

requirements-tracer-server

A prototype tool for assisting in the automation of requirements traceability.

Some details:

Written with FastAPI, uv, SQLModel, pytest, and Pydantic.
Reads a git-backed config file of requirements to build the requirements database.
A REST API to collect test results and link them to the requirements.
Currently done via a pytest plugin, but is flexible and could be even used with manual tests.

Some Code

The schema file, made of SQLModel (Pydantic) models.


class Requirement(SQLModel, table=True):
    id: int | None = Field(primary_key=True, default=None)
    title: str
    description: str = ""
    state: RequirementStates = RequirementStates.closed
    product: str = ""
    stakeholder: str = ""
    note: str = ""
    software_release: str = ""
    priority: RequirementPriority = RequirementPriority.low
    severity: RequirementSeverity = RequirementSeverity.normal
    decline_note: str = ""
    verification_method: RequirementVerificationMethod = RequirementVerificationMethod.testing
    verification_status: RequirementVerificationStatus = RequirementVerificationStatus.incomplete

    #todo: functional_spec fields?

    # Many-to-many testcase relationship for testcases
    testcases: list["TestCase"] = Relationship(back_populates="requirements", link_model=RequirementTestCaseLink)
    
class TestCase(SQLModel, table=True):
    #metadata
    id: int | None = Field(primary_key=True, default=None)
    title: str
    status: TestCaseStatus = TestCaseStatus.released
    objective: str = ""
    author: str = ""
    category: TestCategory = TestCategory.other
    #test procedure
    setup: str = ""
    steps: str = ""
    cleanup: str = ""
    pass_fail_criteria: str = ""
    #traceability
    requirements: list["Requirement"] = Relationship(back_populates="testcases", link_model=RequirementTestCaseLink)
    testsuites: list["TestSuite"] = Relationship(back_populates="testcases", link_model=TestCaseTestSuiteLink)
    testresults: list["TestResult"] = Relationship(back_populates="testcase")
    defects: list["Defect"] = Relationship(back_populates="testcases", link_model=TestCaseDefectLink)
    #automation
    automated: bool = True
    candidate_for_automation: bool = True
    automation_priority: TestCaseAutomationPriority = TestCaseAutomationPriority.low

class TestSuite(SQLModel, table=True):
    #metadata
    id: int = Field(primary_key=True)
    title: str
    objective: str = ""
    category: TestCategory = TestCategory.other
    regression: bool = False
    test_cycle: int = 1
    #traceability
    testcases: list["TestCase"] = Relationship(back_populates="testsuites", link_model=TestCaseTestSuiteLink)

class TestResult(SQLModel, table=True):
    #metadata
    id: int = Field(primary_key=True)
    status: TestResultStatus = TestResultStatus.untested
    run_date: datetime
    time: timedelta
    release: str = ""
    
    #traceability
    testcase_id: int | None = Field(default=None, foreign_key="testcase.id")
    testcase: TestCase = Relationship(back_populates="testresults")

class Defect(SQLModel, table=True):
    id: int = Field(primary_key=True)
    state: DefectState = DefectState.new
    headline: str
    severity: DefectSeverity = DefectSeverity.low
    priority: DefectPriority = DefectPriority.low
    reproducible: bool = False
    crash: bool = False
    keywords: str = ""
    product: str = ""
    category: TestCategory = TestCategory.other
    release: str = ""
    submit_date: datetime
    description: str = ""
    notes: str = ""
    fix_description: str = ""
    fix_date: str = ""
    duplicate_defect_id: int | None = Field(default=None, foreign_key="defect.id")
    new_requirement_title: str = ""
    #orthogonal defect classification
    activity: str = ""
    trigger: str = ""
    impact: str = ""
    target: DefectODCTarget = DefectODCTarget.code
    defect_type: str = ""
    qualifier: DefectODCQualifier = DefectODCQualifier.incorrect
    source: DefectODCSource = DefectODCSource.source
    age: DefectODCAge = DefectODCAge.new
    #traceability
    testcases: list["TestCase"] = Relationship(back_populates="defects", link_model=TestCaseDefectLink)

A test, using a decorator to enable communication with server. Docstring is for TestCase metadata.


@requirements_tracer_server_enabled
def test_read_requirements(base_db, db_session, db_testdata, test_client):
    """
    {
    "title": "test_read_requirements",
    "requirements": ["requirements_read_all"],
    "test_suites": ["unit_tests"]
    }
    """
    response = test_client.get("/requirements/")
    assert response.status_code == 200
    assert response.json() == [json.loads(Requirement(title="Test Requirement", id=1).model_dump_json())]

The pytest plugin that sends results to the API.


@pytest.hookimpl(tryfirst=True, hookwrapper=True)
def pytest_runtest_makereport(item, call):
    #Pre hook section
        
    outcome = yield

    #Post hook section
    result = outcome.get_result()
    if result.when == 'call': #for call, not setup or teardown
        if REQUIREMENTS_TRACER_SERVER_ENABLED == "true" and hasattr(item.function, "requirements_tracer_server_enabled"):
            #update testcase
            testcase_metadata = json.loads(item.function.__doc__)
            url = "{}/testcases".format(REQUIREMENTS_TRACER_SERVER_HOST)
            r = httpx.post(url, json=testcase_metadata, follow_redirects=True)
            if r.status_code != 201:
                pass #TODO: error handling
            #update result
            test_result = {
                "status": result.outcome,
                "run_date": datetime.now().isoformat(),
                "time": str(result.duration),
                "testcase": testcase_metadata["title"]
            }
            url = "{}/testresults".format(REQUIREMENTS_TRACER_SERVER_HOST)
            r = httpx.post(url, json=test_result, follow_redirects=True)
            if r.status_code != 201:
                pass #TODO: error handling

Demo

Conclusion

Here's what I want to work on next:

Better code quality, tests, cleanup. (dogfooding)
Finish schema for Defects and TestSuites.
UI Dashboard with HTMX and Jinja2.
Different data format, like TOML, for requirements and test case metadata.
Keep experimenting, refining, learning this concept.

Any thoughts? I'd love to hear what you think.

Do you see value in requirement traceability?
What other ways could you accomplish automated traceability?
What do you do for traceability?