Netbox Sync Device Interfaces Task¤

task api name: sync_device_interfaces

The Netbox Sync Device Interfaces Task is a feature of the NorFab Netbox Service that allows you to synchronize and update the interface data of your network devices in Netbox. This task ensures that the interface configurations in Netbox are accurate and up-to-date, reflecting the current state of your network infrastructure.

Keeping interface data accurate and up-to-date is crucial for effective network management. The Netbox Update Device Interfaces Task automates the process of updating interface information, such as interface names, statuses, mac addresses, and other relevant details.

Branching Support¤

Update device interfaces task is branch aware and can push updates to the branch. Netbox Branching Plugin need to be installed on Netbox instance.

How it works - Netbox worker on a call to update interfaces task fetches live data from network devices using nominated datasource, by default it is Nornir service parse task using NAPALM get_interfaces getter. Once data retrieved from network, Netbox worker updates records in Netbox database for device interfaces.

Netbox Update Device Interfaces

Client submits and on-demand request to NorFab Netbox worker to update device interfaces
Netbox worker sends job request to nominated datasource service to fetch live data from network devices
Datasource service fetches data from the network
Datasource returns devices interfaces data back to Netbox Service worker
Netbox worker processes device interfaces data and updates records in Netbox for requested devices

Limitations¤

Datasource nornir uses NAPALM get_interfaces getter and as such only supports these device platforms:

Arista EOS
Cisco IOS
Cisco IOSXR
Cisco NXOS
Juniper JUNOS

Update Device Interfaces Sample Usage¤

NORFAB Netbox Update Device Interfaces Command Shell Reference¤

NorFab shell supports these command options for Netbox sync_device_interfaces task:

nf# man tree netbox.update.device.interfaces
root
└── netbox:    Netbox service
    └── sync:    Update Netbox data
        └── device:    Update device data
            └── interfaces:    Update device interfaces
                ├── timeout:    Job timeout
                ├── workers:    Filter worker to target, default 'any'
                ├── verbose-result:    Control output details, default 'False'
                ├── progress:    Display progress events, default 'True'
                ├── instance:    Netbox instance name to target
                ├── dry-run:    Return information that would be pushed to Netbox but do not push it
                ├── devices:    List of Netbox devices to update
                ├── datasource:    Service to use to retrieve device data, default 'nornir'
                │   └── nornir:    Use Nornir service to retrieve data from devices
                │       ├── FO:    Filter hosts using Filter Object
                │       ├── FB:    Filter hosts by name using Glob Patterns
                │       ├── FH:    Filter hosts by hostname
                │       ├── FC:    Filter hosts containment of pattern in name
                │       ├── FR:    Filter hosts by name using Regular Expressions
                │       ├── FG:    Filter hosts by group
                │       ├── FP:    Filter hosts by hostname using IP Prefix
                │       ├── FL:    Filter hosts by names list
                │       ├── FM:    Filter hosts by platform
                │       ├── FN:    Negate the match
                │       ├── add-details:    Add task details to results, default 'False'
                │       ├── num-workers:    RetryRunner number of threads for tasks execution
                │       ├── num-connectors:    RetryRunner number of threads for device connections
                │       ├── connect-retry:    RetryRunner number of connection attempts
                │       ├── task-retry:    RetryRunner number of attempts to run task
                │       ├── reconnect-on-fail:    RetryRunner perform reconnect to host on task failure
                │       ├── connect-check:    RetryRunner test TCP connection before opening actual connection
                │       ├── connect-timeout:    RetryRunner timeout in seconds to wait for test TCP connection to establish
                │       ├── creds-retry:    RetryRunner list of connection credentials and parameters to retry
                │       ├── tf:    File group name to save task results to on worker file system
                │       ├── tf-skip-failed:    Save results to file for failed tasks
                │       ├── diff:    File group name to run the diff for
                │       ├── diff-last:    File version number to diff, default is 1 (last)
                │       └── progress:    Display progress events, default 'True'
                ├── batch-size:    Number of devices to process at a time, default '10'
                └── branch:    Branching plugin branch name to use
nf#

Python API Reference¤

Update or create device interfaces in Netbox using devices interfaces data sourced via Nornir service parse task using NAPALM getter.

Interface parameters updated:

interface name
interface description
mtu
mac address
admin status
speed

Parameters:

Name	Type	Description	Default
`job`	`Job`	NorFab Job object containing relevant metadata.	required
`instance`	`str`	The Netbox instance name to use.	`None`
`dry_run`	`bool`	If True, no changes will be made to Netbox.	`False`
`datasource`	`str`	The data source to use. Supported datasources: nornir - uses Nornir Service parse task to retrieve devices' data using NAPALM get_interfaces getter	`'nornir'`
`timeout`	`int`	The timeout for the job.	`60`
`devices`	`list`	List of devices to update.	`None`
`create`	`bool`	If True, new interfaces will be created if they do not exist.	`True`
`batch_size`	`int`	The number of devices to process in each batch.	`10`
`branch`	`str`	Branch name to use, need to have branching plugin installed, automatically creates branch if it does not exist in Netbox.	`None`
`**kwargs`	`Any`	Additional keyword arguments to pass to the datasource job.	`{}`

Returns:

Name	Type	Description
`dict`	`Result`	A dictionary containing the results of the update operation.

Raises:

Type	Description
`Exception`	If a device does not exist in Netbox.
`UnsupportedServiceError`	If the specified datasource is not supported.

Source code in norfab\workers\netbox_worker\interfaces_tasks.py

@Task(
    fastapi={"methods": ["PATCH"], "schema": NetboxFastApiArgs.model_json_schema()}
)
def sync_device_interfaces(
    self,
    job: Job,
    instance: Union[None, str] = None,
    dry_run: bool = False,
    datasource: str = "nornir",
    timeout: int = 60,
    devices: Union[None, list] = None,
    create: bool = True,
    batch_size: int = 10,
    branch: str = None,
    **kwargs: Any,
) -> Result:
    """
    Update or create device interfaces in Netbox using devices interfaces
    data sourced via Nornir service `parse` task using NAPALM getter.

    Interface parameters updated:

    - interface name
    - interface description
    - mtu
    - mac address
    - admin status
    - speed

    Args:
        job: NorFab Job object containing relevant metadata.
        instance (str, optional): The Netbox instance name to use.
        dry_run (bool, optional): If True, no changes will be made to Netbox.
        datasource (str, optional): The data source to use. Supported datasources:

            - **nornir** - uses Nornir Service parse task to retrieve devices' data
                using NAPALM get_interfaces getter

        timeout (int, optional): The timeout for the job.
        devices (list, optional): List of devices to update.
        create (bool, optional): If True, new interfaces will be created if they do not exist.
        batch_size (int, optional): The number of devices to process in each batch.
        branch (str, optional): Branch name to use, need to have branching plugin installed,
            automatically creates branch if it does not exist in Netbox.
        **kwargs: Additional keyword arguments to pass to the datasource job.

    Returns:
        dict: A dictionary containing the results of the update operation.

    Raises:
        Exception: If a device does not exist in Netbox.
        UnsupportedServiceError: If the specified datasource is not supported.
    """
    devices = devices or []
    instance = instance or self.default_instance
    ret = Result(
        task=f"{self.name}:sync_device_interfaces",
        result={},
        resources=[instance],
        dry_run=dry_run,
        diff={},
    )
    nb = self._get_pynetbox(instance, branch=branch)
    log.info(
        f"{self.name} - Sync device interfaces: Syncing interfaces for {len(devices)} device(s) in '{instance}'"
    )
    kwargs["add_details"] = True

    if datasource == "nornir":
        # source hosts list from Nornir
        if kwargs:
            devices.extend(self.get_nornir_hosts(kwargs, timeout))
            devices = list(set(devices))
            job.event(f"syncing {len(devices)} devices")

        # fetch devices interfaces data from Netbox
        nb_interfaces_data = self.get_interfaces(
            job=job,
            instance=instance,
            devices=copy.copy(devices),
            cache="refresh",
        ).result

        # fetch devices data from Netbox
        nb_devices_data = self.get_devices(
            job=job,
            instance=instance,
            devices=copy.copy(devices),
        ).result

        # iterate over devices in batches
        for i in range(0, len(devices), batch_size):
            kwargs["FL"] = devices[i : i + batch_size]
            kwargs["getters"] = "get_interfaces"
            job.event(
                f"retrieving interfaces for devices {', '.join(kwargs['FL'])}"
            )
            data = self.client.run_job(
                "nornir",
                "parse_napalm",
                kwargs=kwargs,
                workers="all",
                timeout=timeout,
            )

            # Collect interfaces to update and create in bulk
            interfaces_to_update = []
            interfaces_to_create = []
            mac_addresses_to_create = []

            for worker, results in data.items():
                if results["failed"]:
                    msg = f"{worker} Get interfaces failed, errors: {'; '.join(results['errors'])}"
                    ret.errors.append(msg)
                    log.error(msg)
                    continue

                for host, host_data in results["result"].items():
                    if host_data["napalm_get"]["failed"]:
                        msg = f"{host} interfaces update failed: '{host_data['napalm_get']['exception']}'"
                        ret.errors.append(msg)
                        log.error(msg)
                        continue

                    nb_interfaces = nb_interfaces_data.get(host, {})
                    if not nb_interfaces:
                        msg = f"'{host}' has no interfaces in Netbox, skipping"
                        ret.errors.append(msg)
                        log.warning(msg)
                        continue

                    # Get device ID for creating new interfaces
                    nb_device = nb_devices_data.get(host)
                    if not nb_device:
                        msg = f"'{host}' does not exist in Netbox"
                        ret.errors.append(msg)
                        log.error(msg)
                        continue

                    interfaces = host_data["napalm_get"]["result"]["get_interfaces"]

                    sync_key = "sync_device_interfaces"
                    create_key = "created_device_interfaces"
                    if dry_run:
                        sync_key = "sync_device_interfaces_dry_run"
                        create_key = "created_device_interfaces_dry_run"
                    ret.result[host] = {
                        sync_key: {},
                        create_key: {},
                    }
                    if branch is not None:
                        ret.result[host]["branch"] = branch

                    # Process network device interfaces
                    for intf_name, interface_data in interfaces.items():
                        if intf_name in nb_interfaces:
                            # Interface exists - prepare update
                            nb_intf = nb_interfaces[intf_name]

                            # Build desired state
                            desired_state = {
                                "description": interface_data.get(
                                    "description", ""
                                ),
                                "enabled": interface_data.get("is_enabled", True),
                            }
                            if 10000 > interface_data.get("mtu", 0) > 0:
                                desired_state["mtu"] = interface_data["mtu"]
                            if interface_data.get("speed", 0) > 0:
                                desired_state["speed"] = (
                                    interface_data["speed"] * 1000
                                )

                            # Build current state
                            current_state = {
                                "description": nb_intf.get("description", ""),
                                "enabled": nb_intf.get("enabled", True),
                            }
                            if nb_intf.get("mtu"):
                                current_state["mtu"] = nb_intf["mtu"]
                            if nb_intf.get("speed"):
                                current_state["speed"] = nb_intf["speed"]

                            # Compare and get fields that need updating
                            updates, diff = self.compare_netbox_object_state(
                                desired_state=desired_state,
                                current_state=current_state,
                            )

                            # Only update if there are changes
                            if updates:
                                updates["id"] = int(nb_intf["id"])
                                interfaces_to_update.append(updates)
                                ret.diff.setdefault(host, {})[intf_name] = diff

                            ret.result[host][sync_key][intf_name] = (
                                updates if updates else "Interface in sync"
                            )

                            mac_address = (
                                interface_data.get("mac_address", "")
                                .strip()
                                .lower()
                            )
                            if mac_address and mac_address not in ["none", ""]:
                                # Check if MAC already exists
                                for nb_mac in nb_intf.get("mac_addresses") or []:
                                    if nb_mac["mac_address"].lower() == mac_address:
                                        break
                                else:
                                    # Prepare MAC address for creation
                                    mac_addresses_to_create.append(
                                        {
                                            "mac_address": mac_address,
                                            "assigned_object_type": "dcim.interface",
                                            "assigned_object_id": int(
                                                nb_intf["id"]
                                            ),
                                        }
                                    )
                        elif create:
                            # Interface doesn't exist - prepare creation
                            new_intf = {
                                "name": intf_name,
                                "device": int(nb_device["id"]),
                                "type": "other",
                                "description": interface_data.get(
                                    "description", ""
                                ),
                                "enabled": interface_data.get("is_enabled", True),
                            }
                            if 10000 > interface_data.get("mtu", 0) > 0:
                                new_intf["mtu"] = interface_data["mtu"]
                            if interface_data.get("speed", 0) > 0:
                                new_intf["speed"] = interface_data["speed"] * 1000

                            mac_address = (
                                interface_data.get("mac_address", "")
                                .strip()
                                .lower()
                            )
                            if mac_address and mac_address not in ["none", ""]:
                                mac_addresses_to_create.append(
                                    {
                                        "mac_address": mac_address,
                                        "assigned_object_type": "dcim.interface",
                                        "assigned_object_id": int(nb_intf["id"]),
                                    }
                                )

                            interfaces_to_create.append(new_intf)
                            ret.result[host][create_key][intf_name] = new_intf

            # Perform bulk updates and creations
            if interfaces_to_update and not dry_run:
                try:
                    nb.dcim.interfaces.update(interfaces_to_update)
                    job.event(
                        f"bulk updated {len(interfaces_to_update)} interfaces"
                    )
                except Exception as e:
                    msg = f"Bulk interface update failed: {e}"
                    ret.errors.append(msg)
                    log.error(msg)

            if interfaces_to_create and not dry_run:
                try:
                    _ = nb.dcim.interfaces.create(interfaces_to_create)
                    job.event(
                        f"bulk created {len(interfaces_to_create)} interfaces"
                    )
                except Exception as e:
                    msg = f"Bulk interface creation failed: {e}"
                    ret.errors.append(msg)
                    log.error(msg)

            # Bulk create MAC addresses
            if mac_addresses_to_create and not dry_run:
                try:
                    nb.dcim.mac_addresses.create(mac_addresses_to_create)
                    job.event(
                        f"bulk created {len(mac_addresses_to_create)} MAC addresses"
                    )
                except Exception as e:
                    msg = f"Bulk MAC address creation failed: {e}"
                    ret.errors.append(msg)
                    log.error(msg)

    else:
        raise UnsupportedServiceError(
            f"'{datasource}' datasource service not supported"
        )

    return ret