ADR - NFAPI Client-First Startup and Client Connection Overrides¤
Status¤
Proposed.
Date¤
2026-06-20.
Decision¤
Improve NorFab startup so default NFCLI interactive mode can return the prompt as soon as the client is constructed, without waiting for all local worker processes to finish initialization.
NorFab.start() must no longer expose or use a public wait argument. Startup
uses one simple path:
nf = NorFab(inventory="inventory.yaml", log_level=log_level)
nf.start()
client = nf.make_client()
NorFab.start() schedules requested broker and worker startup through a
short-lived background startup thread and returns using the existing return
behavior. Broker and workers continue starting in the background.
NorFab.__enter__() must support the same behavior as direct lifecycle usage.
Context manager usage should remain valid and should not require a separate
NFCLI flow:
with NorFab(inventory="inventory.yaml", log_level=log_level) as nf:
client = nf.client
NFCLI default shell mode should keep using with NorFab(...) as nf, with all
CLI overrides passed through NorFab.__init__().
When broker authentication is enabled, client certificate setup should wait for the local broker-generated key only when all of these conditions are true:
- broker authentication is enabled;
- no broker token/shared key is available from API arguments, CLI arguments, environment variables, or inventory;
- a local broker is requested in the same NorFab instance.
If a token is available from broker_token, --broker-token,
NORFAB_BROKER_TOKEN, or inventory, the client should use it immediately and
must not wait for local broker startup or local __norfab__ state.
Jobs submitted before broker or target workers are ready should be accepted
into the local client job database. If the broker reports no matching workers
with the existing 400 / workers=None response, the client should mark the
job as WAITING_WORKERS and retry every 5 seconds until the job deadline.
Management requests that are not stored in the job database keep their current
synchronous timeout behavior.
Add client connection overrides:
nfcli --broker-url tcp://host:portnfcli --broker-token <broker-public-key>
Add .env loading in NFAPI before inventory rendering. Existing process
environment variables must win over .env values.
Restart support should be designed in this ADR but implemented as a follow-up change after client-first startup.
Context¤
Current nfcli default shell mode calls
start_picle_shell(..., run_workers=True, run_broker=True) from
norfab/utils/nfcli.py. start_picle_shell() in
norfab/clients/nfcli_shell/nfcli_shell_client.py enters with NorFab(...) as
nf, and NorFab.__enter__() calls self.start() followed by
self.make_client().
NorFab.start() in norfab/core/nfapi.py currently blocks in three places:
start_broker()starts the broker process and waits up to 30 seconds for brokerinit_done_event.- Worker processes are started according to topology and dependency rules.
- The final worker readiness loop waits until every worker
init_done_eventis set, or untilworkers_init_timeoutexpires.
As a result, the interactive client is created only after broker and all selected workers have initialized. Slow workers, plugin imports, remote API checks, large inventories, or worker dependency chains delay the NFCLI prompt.
Existing useful properties:
NFPClientconnects with ZeroMQ DEALER sockets, so connecting to an endpoint is non-blocking at the socket layer.- The client already has receiver and dispatcher threads and future-based job handling.
submit_job()already writes jobs to the local SQLite job database before the dispatcher sends anything to the broker.- Worker readiness is already represented by per-process
init_done_event. - Workers and clients already read broker endpoint and shared key from
inventory.broker.
The current client job path needs a small adjustment for client-first startup.
dispatch_new_jobs() sends NEW jobs to the broker and marks them
SUBMITTING. The common early-startup failure is
broker-ready-but-workers-not-ready: NFPBroker.dispatch() returns a 400
response with workers=None when the service has no matching active workers,
and handle_response() currently treats all 4xx and 5xx responses as
terminal failures.
The authentication constraint is local key bootstrap. NFPClient calls
generate_certificates() during construction. That helper creates the client
keypair and then either copies the broker public key from
__norfab__/files/broker/public_keys/broker.key or writes
inventory.broker["shared_key"] into the client's local broker.key. After
that, reconnect_to_broker() loads the client's local
public_keys/broker.key.
If broker authentication is enabled and no shared key is provided, client construction fails when the local broker key has not been generated yet. Certificate setup should support a bounded wait for the local broker key when a local broker is being started in the same NorFab instance.
Goals¤
- Return the NFCLI prompt quickly in default shell mode.
- Keep startup behavior simple: no public wait flags and no separate broker-wait or worker-wait modes.
- Keep context manager usage and direct lifecycle usage equivalent.
- Keep immediate job submissions durable during background startup by retrying worker-unavailable startup conditions until the job deadline.
- Allow client-only NFCLI connections to remote brokers without editing
inventory.yaml. - Load
.envbefore inventory Jinja2 rendering so inventory can use{{ env.VAR_NAME }}values. - Keep startup hooks and worker dependency ordering intact unless the new background startup model requires a narrow adjustment.
- Keep code changes minimal and close to existing NFAPI/client patterns.
Non-Goals¤
- Do not rewrite broker, client, or worker protocol flows.
- Do not make job execution complete before the broker or target workers are actually reachable.
- Do not make broker state persistent.
- Do not change worker task semantics.
- Do not add public wait flags.
- Do not implement restart support in the first client-first startup change.
- Do not implement broker watchdog support in the first restart change.
- Do not add a
show startupshell command in the first startup change.
Proposed API¤
NorFab Constructor¤
Add optional arguments:
NorFab(
inventory="./inventory.yaml",
inventory_data=None,
base_dir=None,
log_level=None,
run_broker=True,
run_workers=True,
broker_url=None,
broker_token=None,
load_env=True,
)
Behavior:
broker_urloverridesinventory.broker["endpoint"].broker_tokenoverridesinventory.broker["shared_key"].- If
broker_urlis not supplied,NORFAB_BROKER_URLfrom the environment is used when present. - If
broker_tokenis not supplied,NORFAB_BROKER_TOKENfrom the environment is used when present. - If neither explicit arguments nor environment variables are present, keep the broker endpoint and shared key values provided by the inventory.
- Precedence is API/CLI argument first, environment variable second, inventory value last.
- Broker overrides mutate only the in-memory inventory and are not persisted to disk.
load_env=Trueloads the local.envbeforeNorFabInventoryreads and renders inventory.- For path-based inventory,
load_env=Trueonly checks the directory that containsinventory.yaml. load_env=<filepath>loads the explicit env file. Relative paths are resolved from the inventory file directory.- Missing explicit
load_env=<filepath>raises an error. load_env=Falseorload_env=Noneskips env loading..envloading should be supported for bothinventoryandinventory_datamodes.- For
inventory_datamode, the resolvedNorFabInventory.base_diranchorsload_env=Trueand relativeload_env=<filepath>values. After the inventory base directory fix, omittedbase_dirdefaults to the current working directory.
NorFab Startup¤
Keep start() close to the existing public shape, but remove any public wait
argument:
nf.start(
run_broker=None,
run_workers=None,
)
Behavior:
start()schedules requested broker/workers using a short-lived startup thread.start()does not wait for all workers to initialize.- If both
run_broker=Falseandrun_workers=False,start()returns immediately without creating a startup thread, as long as this keeps the code smaller and clearer. start()keeps the existing return behavior unless the new threading logic makes a narrow return change unavoidable.- The startup thread handles broker startup, worker dependency ordering, worker readiness tracking, startup hooks, and minimal startup status recording.
- Use explicit
is Nonechecks instead ofrun_broker = run_broker or self.run_brokerandrun_workers = run_workers or self.run_workers, so explicitFalsevalues remain meaningful. - If another
start()call happens while startup is already running, keep the closest existing behavior and avoid adding new public state unless required.
Minimal startup status should be limited to what is needed now:
startingrunningerror
Do not add future-oriented status fields such as completed_at,
per-worker states, or a public startup status object unless implementation
requires them.
Startup hooks should keep existing behavior unless the new background startup flow would break current semantics.
Context Manager¤
NorFab.__enter__() should use the same lifecycle as direct startup:
- call
self.start()using constructor-resolvedrun_brokerandrun_workers; - call
self.make_client(); - return
self.
This allows NFCLI to keep:
with NorFab(
inventory=inventory,
run_broker=run_broker,
run_workers=run_workers,
broker_url=broker_url,
broker_token=broker_token,
) as nf:
builtins.NFCLIENT = nf.client
shell.start()
Client Creation¤
Extend make_client():
nf.make_client(
broker_endpoint=None,
broker_token=None,
name="NFPClient",
)
Behavior:
broker_endpointoverrides the endpoint for that client.broker_tokenupdates the broker shared key before certificate generation.nameallows future clients to avoid sharing the same local client database and certificate directory.- When the client must read the broker token from a local broker-generated key
file, client certificate setup waits up to the broker startup timeout for
that file before raising a clear
RuntimeError. - The key wait only happens when auth is enabled, no broker token exists, and a local broker is requested.
- Client-only remote mode with broker authentication enabled and no token should fail quickly with a clear error instead of waiting for a local key that will never be generated.
NFCLI Arguments¤
Add arguments in norfab/utils/nfcli.py:
--broker-url Broker ZMQ endpoint for the client.
--broker-token Broker public key/shared key for client authentication.
Rules:
- Use only
--broker-url; do not add--broker-endpoint. --broker-tokenaccepts the raw public key/shared key value, same as inventory, not a file path.- If
--broker-urlis omitted andNORFAB_BROKER_URLexists, useNORFAB_BROKER_URL. - If
--broker-tokenis omitted andNORFAB_BROKER_TOKENexists, useNORFAB_BROKER_TOKEN. - NFCLI help text should mention the environment-variable fallbacks.
- Apply these arguments to all modes that create a NorFab client:
--client, default shell mode,--tui, and--web-uiif supported by that path.
When --broker is also used, --broker-url is the local broker bind endpoint.
When --workers is used, both overrides are passed through the in-memory
inventory so workers connect to the same broker.
Implementation Plan¤
Keep the first implementation intentionally small. The first client-first startup change should include:
- NFAPI constructor argument handling for
.env, broker URL, and broker token. - NFAPI startup orchestration through one startup thread.
- Context manager behavior matching direct startup behavior.
- Client certificate setup wait for locally generated broker keys.
- Client job dispatch retry for broker no-workers responses.
- NFCLI argument parsing and passing broker overrides into
NorFab. - Moving mutable
NorFabclass attributes such asworkers_processesandworker_pluginsto instance attributes if needed for correct background startup ownership. - Documentation updates for changed startup semantics, environment loading, broker overrides, and job retry behavior.
The restart capability is a follow-up change and should not be included in the first client-first startup implementation.
Avoid new broker protocol commands, remote administrative APIs, database schema changes, and broad worker refactors in the client-first startup change.
Code and Style Guidelines¤
Follow the style used by the other ADRs in this folder:
- Keep the common path first and keep sections scannable.
- Keep examples short and use fenced code blocks with a language tag.
- Use the same terms consistently: broker, worker, client, startup thread, lifecycle supervisor, lifecycle signal.
- Put compatibility, risks, and open questions in their own sections.
- Prefer staged implementation notes over broad refactors.
Keep code changes small and close to the current code shape:
- Reuse existing
start_broker_process(),start_worker_process(),multiprocessing.Process,multiprocessing.Event, andthreading.Threadpatterns. - Do not introduce asyncio, a new process manager, a new broker protocol command, or a remote admin service.
- Put new NFAPI state on the
NorFabinstance, not class attributes. - Use explicit
is Nonechecks for optional booleans soFalseremains a meaningful caller value. - Do not log
broker_tokenor broker shared keys. - Give the startup thread a clear name such as
norfab-startup. - Prefer small helper methods over new classes unless the helper starts carrying independent state.
- Keep NFCLI changes in argument parsing and shell startup wiring.
1. Load .env Before Inventory Rendering¤
Use python-dotenv for env file parsing:
from dotenv import load_dotenv
load_dotenv(dotenv_path=path, override=False)
Add python-dotenv as a core dependency because .env loading is part of
NFAPI startup behavior.
Call load_dotenv() at the top of NorFab.__init__() before constructing
NorFabInventory, because render_jinja2_template() reads os.environ during
inventory load.
For path-based inventory:
load_env=Trueloads only<inventory-directory>/.env.load_env=<relative path>resolves from<inventory-directory>.load_env=<absolute path>uses that exact path.- Missing explicit env file paths raise an error.
For inventory_data mode:
load_env=Trueloads<resolved-base-dir>/.env.load_env=<relative path>resolves from the resolved base directory.- If
base_diris omitted, the resolved base directory is the current working directory becauseNorFabInventorynow usesos.path.abspath(base_dir or os.getcwd())for dictionary inventory. - Compute this resolved base directory before constructing
NorFabInventoryso.envvalues are loaded before any inventory or worker file template rendering readsos.environ.
2. Apply Broker Overrides Centrally¤
After inventory is loaded, apply:
broker_url = broker_url or os.environ.get("NORFAB_BROKER_URL")
broker_token = broker_token or os.environ.get("NORFAB_BROKER_TOKEN")
if broker_url:
self.inventory.broker["endpoint"] = broker_url
if broker_token:
self.inventory.broker["shared_key"] = broker_token
If neither explicit arguments nor environment variables are present, do not
modify these inventory fields. Then set self.broker_endpoint from the final
inventory value.
3. Handle Local Broker Key Wait In Client Setup¤
Do not add NFAPI key-readiness checks or startup branches. NFPClient already
knows the exact certificate paths during construction, so the bounded wait
belongs close to generate_certificates() and reconnect_to_broker().
The common flow stays the same for all local shell startups:
nf.start()schedules broker and worker startup in the background.nf.make_client()constructs the client.- Client certificate setup uses the configured broker token immediately, or
waits for the broker-generated local
broker.keyfile if auth is enabled, no token is configured, and a local broker is requested. - NFCLI starts the shell once the client exists.
If certificate setup cannot obtain a required broker key before the broker
startup timeout, raise RuntimeError and exit NFCLI startup.
If __norfab__ is missing but a broker token is available from CLI/API,
environment, or inventory, the client should proceed using the provided token.
Do not pre-generate or rotate broker keys outside broker startup. The broker remains responsible for creating its own key files.
4. Make Job Dispatch Retry No-Worker Startup Responses¤
Keep submit_job() as the durable local enqueue point. A submitted job should
be written to the client database and represented by an NFPJobFuture before
any broker or worker availability is required.
Update dispatcher/response handling:
- Add
JobStatus.WAITING_WORKERS = "WAITING_WORKERS". - When
handle_response()receivesstatus == "400"withpayload["workers"] is None, update the job toWAITING_WORKERS, store a useful warning, setlast_poll_timestamp, and do not callfuture.mark_done(). - Apply this retry behavior to all
workers=Noneresponses, including requests that target a specific worker name that does not currently exist. Most calls useworkers="all"orworkers="any", and the first implementation should keep the behavior simple by retrying until the job deadline. - Have
dispatch_new_jobs()pick upNEWandWAITING_WORKERSjobs. - Retry
WAITING_WORKERSjobs every 5 seconds. - Keep other client errors and server errors terminal.
- Keep the job deadline timer starting when the job is stored locally.
- Keep existing
future.result(timeout=...)behavior; it should not mutate job state on caller timeout. - Keep MMI behavior unchanged.
- Log a
log.infomessage when a job entersWAITING_WORKERS.
If the job deadline is reached, the job should be marked STALE according to
the existing stale-job behavior. Avoid broad deadline refactors unless tests
show the current behavior does not handle the new WAITING_WORKERS status.
5. Use a Startup Thread For Local Components¤
Move the existing body of NorFab.start() into an internal method such as
_start_components(run_broker, run_workers).
Make add_built_in_workers_inventory() idempotent before background startup or
future restart paths can call it more than once. It should not insert duplicate
filesharing-worker-1 entries into inventory.topology["workers"].
start() creates a startup thread for requested local broker/workers:
self.startup_thread = threading.Thread(
target=self._start_components,
args=(run_broker, run_workers),
name="norfab-startup",
daemon=True,
)
self.startup_thread.start()
The startup thread is not a supervisor loop. It should start broker/workers,
track minimal startup status, run startup hooks when their existing
prerequisites are met, and exit. Broker and workers continue running in their
existing multiprocessing.Process instances.
If no local broker or workers are requested, start() should return
immediately without creating a startup thread when that keeps the code smaller.
Broker startup failure is fatal for local all-in-one startup. NFAPI/NFCLI
should call destroy() for cleanup and terminate with SystemExit carrying a
useful error message and traceback details when local broker startup failure is
detected. If the failure is detected inside the background startup thread, that
thread must record the fatal error and propagate it to the NFCLI/main control
path; raising SystemExit only inside the background thread is not sufficient
to stop the interactive shell. Worker startup failure should be logged as an
error without preventing the prompt from being returned, unless existing
behavior already treats that specific failure as fatal.
NFCLI should display a small startup line before the prompt so users know local broker/workers are starting in the background.
6. Preserve Worker Dependency Ordering And Hooks¤
Keep current depends_on behavior:
- a worker with dependencies is only started after dependency processes are
alive and their
init_done_eventvalues are set; - workers without dependencies can start immediately.
Startup hooks currently run only after all workers initialize. Keep that semantic unless broker-only mode or background startup exposes a current bug.
If worker initialization times out, log an error. Do not add new public status commands for this first change.
7. Avoid Worker Readiness CPU Spin¤
The current final worker readiness loop has no sleep in the polling loop. Add a
small sleep, for example time.sleep(0.05), to avoid CPU spin while waiting for
slow workers.
8. Cleanup¤
destroy() must set exit events and join the startup thread briefly if it is
still running before joining processes. This avoids races where startup creates
processes while shutdown is trying to stop them.
Restart Follow-Up Design¤
Restart support should be implemented after the client-first startup change. It should be a normal supported feature, not experimental.
Required public Python API methods:
nf.restart_worker("nornir-worker-1", reason="client-request")
nf.restart_broker(reason="client-request")
Do not add restart_all in the first restart implementation.
Lifecycle commands should be serialized through one NFAPI-owned
multiprocessing.Queue with many producers and one supervisor consumer. This
is the smallest portable option for structured requests from both shell methods
and child processes. The lifecycle supervisor is the only place that mutates
broker/worker process state.
Use plain dictionary lifecycle signals initially:
{
"action": "restart_worker",
"target": "nornir-worker-1",
"reason": "worker-request",
"source": "nornir-worker-1",
"timestamp": time.time(),
}
Worker Restart¤
Worker restart requirements:
- expose public
restart_worker(name, reason=None); - allow worker watchdog code to request its own restart;
- watchdog restart is enabled by default and should not require an inventory knob to turn it on;
- if watchdog settings are needed later, place them under individual worker inventory;
- do not add restart limits or rate limits in the first restart implementation;
- the worker process must not stop or replace itself directly; NFAPI parent process handles the restart.
The supervisor should restart a worker with this sequence:
- Mark the worker as restarting internally.
- Set only that worker's stop signal.
- Join the worker process with a bounded timeout.
- If the process is still alive, terminate it as a last resort and record the forced stop.
- Remove the old process and init event from
self.workers_processes. - Create a new
init_done_event. - Call
start_worker(worker_name, worker_data)using existing dependency checks. - Wait for the new worker
init_done_eventup toworkers_init_timeout. - Mark internal status as running or error.
The current shared self.workers_exit_event is sufficient for global shutdown
but not for individual worker restart. Add a per-worker exit event stored in
self.workers_processes[worker_name]["exit_event"]. Global shutdown should set
all per-worker exit events.
Broker Restart¤
Broker restart requirements:
- expose public
restart_broker(reason=None); - restart only the broker by default;
- keep local workers intact and rely on reconnect behavior;
- do not rotate broker keys during broker restart;
- leave broker watchdog support for a future ADR.
Broker restart flow:
- Stop the broker process and join it with a bounded timeout.
- Create a fresh broker exit event.
- Start a new broker process on the same endpoint.
- Wait for broker readiness.
- Keep workers running unless a later explicit feature adds worker restart.
During worker or broker restart, worker-local in-flight jobs that should be
replayed must be returned to PENDING. worker.py inserts received jobs as
PENDING, get_next_pending_job() only selects PENDING jobs, and the worker
loop atomically marks them STARTED before submitting work to the thread pool.
The restart implementation should reset interrupted STARTED or
WAITING_CLIENT_INPUT worker DB rows to PENDING and clear fields such as
started_timestamp or completed_timestamp where needed so the worker thread
can pick them up normally after restart.
Startup Speed Review¤
The largest user-visible delay is not process creation itself. It is waiting for all worker initialization before constructing the client and starting the shell.
Other improvement opportunities:
- Avoid the busy wait in
NorFab.start()worker readiness loop. - Keep plugin entry point registration cheap by storing entry points and only loading worker classes in child processes or immediately before starting the relevant worker.
- Consider lazy importing service-specific NFCLI shell modules so client-only startup does not import every service command tree up front.
- Avoid repeatedly rendering and merging worker inventory where a worker can be given already-resolved data safely, but treat this as a later optimization because it changes process payload and inventory semantics.
- Add timestamped startup logs for each phase: inventory loaded, logging ready, broker process started, broker ready, worker process started, worker ready, startup hooks complete.
Documentation Updates¤
Update user and developer documentation as part of the implementation.
Required documentation updates:
- Update NFAPI docs and NFCLI docs for client-first startup behavior.
- NFAPI usage examples should show context manager usage and explicit lifecycle usage side by side, ideally in tabbed views.
- NFAPI constructor documentation should describe
load_env,broker_url, andbroker_token, including precedence: explicit API/CLI argument, environment variable, then inventory value. .envdocumentation should describeload_env=True,load_env=<filepath>, andload_env=False/None, with accurate workable examples.- Environment variable documentation should include
NORFAB_BROKER_URLandNORFAB_BROKER_TOKEN. - NFCLI documentation and help text should include
--broker-urland--broker-token, with examples for client-only remote broker connections and local all-in-one startup. - Startup behavior documentation should state that
NorFab.start()schedules broker and worker startup in the background and returns before all components are necessarily ready. - Job documentation should describe early submission as local queueing before
broker/worker readiness, and explain
WAITING_WORKERS. - MMI and other non-job command documentation should remain clear that these calls are synchronous and may time out if broker/workers are not ready.
- Startup hooks documentation should state that hooks run after background worker readiness, not before client construction or prompt display.
- Restart documentation should describe public
restart_worker()andrestart_broker()when the follow-up restart change is implemented. - Troubleshooting documentation should include local broker key wait timeout
failures, missing broker token/key cases,
.envprecedence confusion, and jobs waiting for workers.
Compatibility¤
- This is accepted as a non-backward-compatible startup behavior change.
- Existing inventory files keep working.
- Existing client-only mode keeps working and gains explicit broker endpoint and token overrides.
- Existing worker dependency configuration should keep working.
- Existing context manager usage remains supported.
- Restart support is additive and implemented later.
Testing And Validation¤
- Run the full test suite and refactor tests for the new startup behavior as needed.
- Add focused tests for first-run encrypted local startup and existing-runtime client-first startup.
- Create a small test fixture/folder with a minimal environment for these startup tests.
- Add tests for
.envloading, env variable precedence, and explicit broker overrides. - Add tests for
WAITING_WORKERSretry behavior using a 5 second retry interval where practical. - Test restart follow-up behavior with the existing dummy worker.
- No separate Windows-specific validation is required.
Risks¤
Commands Before Readiness¤
The shell prompt may appear before the broker or workers are reachable. Jobs stored in the local client database should retry worker-unavailable responses until their deadline. MMI and other synchronous non-job requests keep their existing timeout behavior and should return clear errors.
This changes early job submission semantics from "broker accepted the job" to "client queued the job locally for dispatch." Documentation and logging should make this visible.
Broker Failure In Background¤
Broker startup failure is fatal for local broker mode, but startup now happens
in a background thread. The implementation must make that failure visible and
exit NFCLI when detected, without reintroducing full worker startup blocking.
On fatal local broker failure, NFAPI should call destroy() and make the
NFCLI/main control path raise SystemExit with a useful message and traceback
details.
Local Broker Key Wait¤
NFCLI cannot show the prompt until client construction completes. When a local broker is requested, broker authentication is enabled, and no broker token is available from API/CLI arguments, environment variables, or inventory, client certificate setup waits up to the broker startup timeout for the broker-generated local key file.
If the key does not appear, client construction raises RuntimeError and NFCLI
exits.
Shared Inventory Mutation¤
Broker override values mutate the in-memory inventory. This is intentional for
one NorFab process, but the implementation must not write these overrides back
to inventory.yaml.
Background Thread Cleanup¤
destroy() must set exit events and join the startup thread briefly if it is
still running before joining processes, otherwise startup and shutdown can
race.
Broker Restart Disruption¤
Broker restart drops the broker's in-memory worker registry and can interrupt in-flight jobs. The follow-up restart implementation must verify worker reconnect and job requeue behavior carefully.
Open Questions¤
No open questions remain for the first client-first startup implementation.