NORFAB Developing and Testing Framework

Set up the development environment

Install uv, then run these commands from the repository root:

uv sync --locked --python 3.12 --extra full --extra docs
uv run inv --list

uv creates a local .venv; shell activation is optional when using uv run. Python 3.12 supports all service extras, including FakeNOS (Python 3.11-3.12). For core development only, use uv sync --locked; add --extra <name> for individual services or --extra docs for documentation tools. Development tools are installed by default.

When migrating an existing checkout, stop using its Poetry environment and run uv sync to create the uv environment. Run commands with uv run and select extras with uv sync --extra <name>.

Dependencies and extras are declared in pyproject.toml; commit uv.lock after changing them. Use uv add <package>, uv add --dev <tool>, or uv add --optional <extra> <package>. Run uv lock after editing dependencies manually. uv sync --locked checks that the lockfile matches the project. To update a pinned dependency, change its version in pyproject.toml and run uv lock; uv lock --upgrade refreshes transitive dependencies within the existing constraints.

Docker images install dependencies with pip. The test image installs the checkout with the full extra and pytest; runtime images install from PyPI. Docker installs do not use uv.lock. Rebuild test images after changing project dependencies.

Develop features with Spec Kit

Use Spec Kit to agree on feature behavior, plan small changes, and implement them with public-client integration tests.

Install Spec Kit

Install the Specify CLI with uv, then check available integrations:

uv tool install specify-cli
specify check

NorFab already includes .specify/ configuration and Codex skills under .agents/skills/. Open the repository in your coding agent to use them. Initialization is only needed when setting up Spec Kit in a repository without this configuration. For Codex with PowerShell scripts:

specify init --here --integration codex --integration-options="--skills" --script ps

See the upstream installation guide for other integrations and installation options.

Follow the feature workflow

Run each prompt separately in the agent chat. Review its result before proceeding. Replace the feature description with the behavior you want to build.

Maintain the constitution when project rules change. It does not need to be regenerated for every feature:

$speckit-constitution
Derive project principles from AGENTS.md and its linked guides.
Preserve existing rules. Keep temporary working notes under private/.
Store durable feature artifacts under docs/development/specs/<NNN-feature-name>/.

Define the feature and resolve missing requirements:

$speckit-specify
Describe the feature's user-visible behavior, scope, and acceptance criteria:
<describe the feature here>
Store its artifacts under docs/development/specs/<NNN-feature-name>/.

$speckit-clarify
Resolve missing requirements before planning.

Plan the implementation and break it into small changes:

$speckit-plan
Design the implementation using NorFab's existing worker and client patterns.
Prefer simple, direct changes and reuse existing methods and utilities.

$speckit-tasks
Break the plan into small changes with simple public-client integration tests.

Review consistency, resolve reported issues, then implement:

$speckit-analyze
Check consistency across the specification, plan, and tasks.

$speckit-implement
Implement and validate the agreed tasks.

Keep feature artifacts together

Use the next sequential number and a descriptive feature name. Keep spec.md, plan.md, research.md, data-model.md, quickstart.md, contracts/, checklists/requirements.md, and generated tasks.md together under docs/development/specs/<NNN-feature-name>/. Keep the specification and checklist separate. Set feature_directory in .specify/feature.json to that folder and link its spec.md from mkdocs.yml.

Record acceptance and implementation status separately. Acceptance does not mean the feature is implemented. Use private/ for temporary notes and review reports. ADRs record separate architecture decisions.

TL;DR

For local debugging, run pytest from tests/ with the required services available:

# Enter the test root so relative pytest paths resolve.
cd tests
# Run the Nornir worker tests locally.
uv run pytest services/nornir/test_worker.py

For isolated Docker runs, return to the repository root and use Invoke:

# Build the Docker images used by the test suites.
uv run inv docker-tests-build
# Run only the Nornir suite in Docker.
uv run inv docker-tests-nornir
# Run all regular suites and write one consolidated report.
uv run inv docker-tests-all

Choose a suite run or the all-suites run. Invoke prints Docker test report: <path>; open that file under docker/norfab-docker-tests/reports/ for the final result.

Tests and local pytest

Tests follow the code layout: tests/core/, tests/services/<service>/, tests/clients/, and tests/nfcli/. Shared fixtures are in tests/conftest.py; integration test inventory is in tests/nf_tests_inventory/. NetBox test helpers are in tests/services/netbox/common.py, with seed data in tests/netbox_data.py.

Run pytest through uv from tests/:

# Enter the test root so relative pytest paths resolve.
cd tests
# Run the full Nornir test directory.
uv run pytest services/nornir
# Run one NetBox test class.
uv run pytest services/netbox/test_interfaces.py::TestGetInterfaces
# Run only NetBox get_interfaces tests.
uv run pytest services/netbox -m "netbox and netbox_get_interfaces"
# Run NFCLI tests with printed output and individual test names.
uv run pytest nfcli -s -v

Use a directory, file, class, or method path to narrow collection; use -m for registered service or task markers. Local integration tests start NorFab through fixtures and need their configured external services. Local pytest shows its results in the terminal and does not create an Invoke Markdown report.

Available pytest markers

Service	Marker	Description
Core	core	NORFAB core component tests
Nornir	nornir	Nornir service tests
NetBox	netbox	NetBox service tests
Containerlab	containerlab	Containerlab service tests
FakeNOS	fakenos	FakeNOS service tests
FastAPI	fastapi	FastAPI service tests
FastMCP	fastmcp	FastMCP service tests
FileSharing	filesharing	FileSharing service tests
FileSharing	filesharing_git	File Sharing authenticated Git remote tests
FileSharing	filesharing_get_remotes	FileSharing get_remotes task tests
FileSharing	filesharing_create_remote_git	FileSharing create_remote_git task tests
FileSharing	filesharing_delete_remote_git	FileSharing delete_remote_git task tests
FileSharing	filesharing_git_clone	FileSharing git_clone task tests
FileSharing	filesharing_resolve_git_url	FileSharing resolve_git_url task tests
Workflow	workflow	Workflow service tests
Dummy	dummy	dummy plugin service tests
Client agent	clientagent	Client agent tests
NFCLI	nfcli	NFCLI shell client tests
Nornir	nornir_cfg	Nornir cfg task tests
Nornir	nornir_cli	Nornir CLI task tests
Nornir	nornir_fakenos	Tests that use FakeNOS-backed devices through Nornir
Nornir	nornir_file_copy	Nornir file_copy task tests
Nornir	nornir_network	Nornir network task tests
Nornir	nornir_parse	Nornir parse task tests
Nornir	nornir_runtime_inventory	Nornir runtime_inventory task tests
Nornir	nornir_gnmi	Nornir gNMI get task tests
Nornir	nornir_snmp	Nornir snmp task tests
Nornir	nornir_task	Nornir task task tests
Nornir	nornir_test	Nornir test task tests
NetBox	netbox_cache_clear	NetBox cache_clear task tests
NetBox	netbox_cache_get	NetBox cache_get task tests
NetBox	netbox_cache_list	NetBox cache_list task tests
NetBox	netbox_check_device_sync	NetBox check_device_sync task tests
NetBox	netbox_create_bgp_peering	NetBox create_bgp_peering task tests
NetBox	netbox_create_asn	NetBox create_asn task tests
NetBox	netbox_design_deploy	NetBox design_deploy task tests
NetBox	netbox_create_device_interfaces	NetBox create_device_interfaces task tests
NetBox	netbox_move_ip	NetBox move_ip task tests
NetBox	netbox_create_ip	NetBox create_ip task tests
NetBox	netbox_create_ip_bulk	NetBox create_ip_bulk task tests
NetBox	netbox_create_prefix	NetBox create_prefix task tests
NetBox	netbox_create_vlan	NetBox create_vlan task tests
NetBox	netbox_crud	NetBox CRUD task tests
NetBox	netbox_crud_create	NetBox crud_create task tests
NetBox	netbox_crud_delete	NetBox crud_delete task tests
NetBox	netbox_crud_get_changelogs	NetBox crud_get_changelogs task tests
NetBox	netbox_crud_list_objects	NetBox crud_list_objects task tests
NetBox	netbox_crud_read	NetBox crud_read task tests
NetBox	netbox_crud_search	NetBox crud_search task tests
NetBox	netbox_crud_update	NetBox crud_update task tests
NetBox	netbox_get_bgp_peerings	NetBox get_bgp_peerings task tests
NetBox	netbox_get_circuits	NetBox get_circuits task tests
NetBox	netbox_get_connections	NetBox get_connections task tests
NetBox	netbox_get_containerlab_inventory	NetBox get_containerlab_inventory task tests
NetBox	netbox_get_devices	NetBox get_devices task tests
NetBox	netbox_get_interfaces	NetBox get_interfaces task tests
NetBox	netbox_get_nornir_inventory	NetBox get_nornir_inventory task tests
NetBox	netbox_get_topology	NetBox get_topology task tests
NetBox	netbox_graphql	NetBox graphql task tests
NetBox	netbox_sync_all	NetBox sync_all task tests
NetBox	netbox_sync_bgp_asn	NetBox sync_bgp_asn task tests
NetBox	netbox_sync_bgp_community	NetBox sync_bgp_community task tests
NetBox	netbox_sync_bgp_peerings	NetBox sync_bgp_peerings task tests
NetBox	netbox_sync_device_interfaces	NetBox sync_device_interfaces task tests
NetBox	netbox_sync_vlans	NetBox sync_vlans task tests
NetBox	netbox_sync_vrrp	NetBox sync_vrrp task tests
NetBox	netbox_sync_vrfs	NetBox sync_vrfs task tests
NetBox	netbox_sync_device_inventory	NetBox sync_device_inventory task tests
NetBox	netbox_sync_device_ip	NetBox sync_device_ip task tests
NetBox	netbox_sync_device_prefixes	NetBox sync_device_prefixes task tests
NetBox	netbox_sync_mac_addresses	NetBox sync_mac_addresses task tests
NetBox	netbox_update_bgp_peering	NetBox update_bgp_peering task tests
NetBox	netbox_update_interfaces_description	NetBox update_interfaces_description task tests
Containerlab	containerlab_deploy	Containerlab deploy task tests
Containerlab	containerlab_deploy_netbox	Containerlab deploy_netbox task tests
Containerlab	containerlab_get_nornir_inventory	Containerlab get_nornir_inventory task tests
Containerlab	containerlab_inspect	Containerlab inspect task tests
Containerlab	containerlab_restart_lab	Containerlab restart_lab task tests
Containerlab	containerlab_save	Containerlab save task tests
FakeNOS	fakenos_get_nornir_inventory	FakeNOS get_nornir_inventory task tests
FakeNOS	fakenos_auto_start	FakeNOS inventory auto_start tests
FakeNOS	fakenos_inspect_networks	FakeNOS inspect_networks task tests
FakeNOS	fakenos_restart	FakeNOS restart task tests
FakeNOS	fakenos_start	FakeNOS start task tests
FakeNOS	fakenos_stop	FakeNOS stop task tests
FastMCP	fastmcp_get_prompts	FastMCP get_prompts task tests
FastMCP	fastmcp_get_tools	FastMCP get_tools task tests
FileSharing	filesharing_fetch_file	FileSharing fetch_file task tests
FileSharing	filesharing_file_details	FileSharing file_details task tests
FileSharing	filesharing_list_files	FileSharing list_files task tests
FileSharing	filesharing_walk	FileSharing walk task tests
Workflow	workflow_run	Workflow run task tests

Writing tests

• Use integration tests through the public client interface and the nfclient fixture whenever possible. Exercise real services and assert observable results.
• Keep each test small and focused on one behavior. Use only the setup, data, and assertions needed to verify it.
• Prefer extending related tests for new behavior when they remain focused. Create a dedicated test when existing tests cover unrelated behavior.
• Group tests in Test... classes. Put service markers on test files and register new task markers in pytest_configure in tests/conftest.py.
• Keep test data explicit, clean up only data created by the test, and make cleanup safe after partial failures. Reuse shared seed records for read-only references; create owned records for mutations.

NetBox test data ownership

tests/netbox_data_manifest.json records which NetBox objects each test area reads or changes. It is a reference for test authors and agents; pytest does not load it. Read and update it whenever adding or changing tests that use NetBox data. Mutating tests must use records and address space distinct from other test areas and clean up only their own objects. Put persistent shared data in tests/netbox_data.py; read-only tests may use that data.

Docker test environment

Invoke runs pytest in isolated Docker Compose containers using the shared test inventory. Run these commands from the repository root:

# List all available Invoke commands.
uv run inv --list

# Check the Docker Compose test configuration without starting containers.
uv run inv docker-tests-config
# Build images for every test suite.
uv run inv docker-tests-build

# Build only the Nornir test image.
uv run inv docker-tests-build --suite=nornir
# Rebuild the Nornir image before running its suite.
uv run inv docker-tests-nornir --build

# Run all tests in the Nornir suite.
uv run inv docker-tests-nornir
# Run one Nornir test file in the suite container.
uv run inv docker-tests-nornir --selector=tests/services/nornir/test_worker.py
# Run only NetBox get_devices tests.
uv run inv docker-tests-netbox --marker="netbox and netbox_get_devices"

# Run the NetBox CRUD file in a dedicated container.
uv run inv docker-tests-netbox-crud
# Run Nornir files in separate containers, at most two concurrently.
uv run inv docker-tests-nornir --parallel-runs=2

Suite names are core, nornir, netbox, fakenos, containerlab, workflow, agent, fastmcp, fastapi, filesharing, dummy, and nfcli. docker-tests-all runs the regular suites concurrently; run docker-tests-containerlab separately. The distributed topology uses docker-tests-distributed. The default suite run collects its test directory and applies its service marker. --selector accepts a directory, file, or pytest node ID; --marker, --keyword, and --pytest-args narrow the run. The default NetBox suite runs each test_*.py file in a dedicated container. See uv run inv --help docker-tests-nornir for all options.

Run Compose directly

From docker/norfab-docker-tests/, you can build and run a Compose service without Invoke:

# Enter the Docker test environment directory.
cd docker/norfab-docker-tests
# Build the Nornir test image with Compose.
docker compose build nornir-service-tests
# Run the Nornir service's default pytest command.
docker compose run --rm nornir-service-tests
# Override that command to run one file with the Nornir marker.
docker compose run --rm nornir-service-tests -m nornir tests/services/nornir/test_worker.py

Arguments after the service name replace its default Compose command, so include -m nornir when selecting tests this way. Compose writes JUnit XML under nornir-service-tests/__norfab__/artifacts/, but does not create the Invoke Markdown summary. The container exit status indicates whether the direct run passed.

Read Invoke reports

Each Invoke test run prints the path of its timestamped Markdown report in docker/norfab-docker-tests/reports/. Reports are named docker-tests-<suite>-<timestamp>.md or docker-tests-all-<timestamp>.md.

• The summary table shows passed, failed, errors, skipped, duration, and suite status.
• Failed tests by service provides expandable lists of failed/error test names below the summary. Expand a service to quickly find its failing tests.
• Failures and report problems shows failing tests and error details.
• JUnit artifacts lists XML files produced by that run. JUnit files and worker logs are under the suite's ignored docker/norfab-docker-tests/<compose-service>/__norfab__/ tree; NetBox group and file-parallel runs use groups/ and parallel/ subdirectories.
• NO REPORT means no JUnit file was produced; INVALID REPORT means the XML could not be parsed. Check the container output and runtime logs.

Individual suite and NetBox file tasks report failures but do not return a failing Invoke status. Always check their report. docker-tests-all returns a nonzero status if a suite fails. Reports use only JUnit files created or updated during that run.

See docker/norfab-docker-tests/README.md for service prerequisites and detailed container setup.