Cisco IOS XE ZTP

Cisco IOS-XE Zero Touch Provisioning (ZTP)¶

Executive Summary¶

A production-ready ZTP script for Day 0 provisioning of Cisco Catalyst switches running IOS-XE. Automatically downloads and applies device-specific configurations when switches boot without a startup-config, enabling hands-free deployment at scale.

Key Differentiators:

• Serial-based configuration lookup — Each device fetches its own config (<SERIAL>.cfg) from HTTP server
• Retry logic with exponential backoff — Handles transient network issues gracefully (3 attempts)
• Structured JSON logging to Graylog/Syslog — Real-time ZTP monitoring with searchable device context
• Automatic SSH key generation — Enables secure remote access immediately post-provisioning
• Rotating file logs on flash — Persistent debugging (5MB max, 2 backups)
• Secure file cleanup — Removes sensitive config files after merging to running-config
• JSON device reports — Optional inventory data export for automation pipelines

Why Zero Touch Provisioning Matters¶

The Day 0 Challenge¶

Traditional Cisco switch deployment requires manual intervention:

1. Console Access — Physical connection to configure initial management IP
2. Manual Configuration — Type (or paste) base config via console
3. Error-Prone — Human typing errors, inconsistent configurations
4. Time-Consuming — 15-30 minutes per switch for experienced engineers
5. Scalability Bottleneck — 100 new switches = 25-50 hours of manual labor

ZTP Solution: Switch powers on → DHCP provides IP + script URL → Script downloads device-specific config → Device is production-ready in ~90 seconds.

Business Impact¶

Architecture Overview¶

High-Level Workflow¶

    graph TD
        A[Switch Powers On
No Startup-Config] --> B[DHCP Request]
        B --> C[DHCP Server]
        C --> D[IP Address + Option 67
ZTP Script URL]
        D --> E[Native ZTP Agent]
        E --> F[Download Python Script
from HTTP Server]
        F --> G[Execute in Guestshell]
        G --> H[Extract Serial Number]
        H --> I[Download SERIAL.cfg
from HTTP Server]
        I --> J{File Transfer
Successful?}
        J -->|No| K[Retry with Backoff
Up to 3 Attempts]
        K --> J
        J -->|Yes| L[Merge Config to
Running-Config]
        L --> M[Generate SSH Keys]
        M --> N[Save to Startup-Config]
        N --> O[Production Ready]
        J -->|Failure After 3 Retries| P[Log Error + Exit]

Component Architecture¶

    flowchart LR
        subgraph Infrastructure["Infrastructure Components"]
            DHCP[DHCP Server
Options 66/67]
            HTTP[HTTP Server
Apache/Nginx]
            SYSLOG[Syslog/Graylog
Optional]
        end

        subgraph Switch["Catalyst Switch"]
            ZTP[Native ZTP Agent]
            GS[Guestshell Environment]
            SCRIPT[day_0_provisioning.py]
            FLASH[Flash Storage]
        end

        subgraph HTTPServer["HTTP Server Structure"]
            SCRIPTDIR[scripts directory
day_0_provisioning.py]
            FILESDIR[files directory
SERIAL.cfg files]
        end

        DHCP -->|IP + Script URL| ZTP
        ZTP -->|Download Script| HTTP
        HTTP --> SCRIPTDIR
        SCRIPTDIR --> GS
        GS --> SCRIPT
        SCRIPT -->|Download Config| FILESDIR
        FILESDIR --> FLASH
        SCRIPT -->|JSON Logs| SYSLOG

        style Infrastructure fill:#fff4e6
        style Switch fill:#e1f5ff
        style HTTPServer fill:#f3e5f5

Detailed Workflow Stages¶

Stage 0: Pre-Execution (Automatic - Cisco Native ZTP)¶

Cisco IOS-XE devices include a built-in ZTP agent that activates automatically when:

1. Switch boots without a startup-config file
2. No manual console interaction occurs during bootup timer (default 5 minutes)

ZTP Agent Behaviour:

• Sends DHCP request with vendor-specific options
• Looks for DHCP Option 67 (Boot File Name) containing HTTP URL
• Downloads script from URL (typically .py or .cfg file)
• For Python scripts: Activates Guestshell and executes script
• For config files: Applies directly to running-config

Stage 1: Device Identification¶

    # Script queries device facts using IOS-XE Python 'cli' module
    model = get_model()              # Extract model (C9300-48U, etc.)
    serial = get_serial()            # Extract serial number
    version = get_version()          # Extract IOS-XE version

Key Operations:

• Execute show version command via Guestshell cli() function
• Parse output using regex to extract:
• Model Number: C9300-48U, C9200-24P, C9400, etc.
• Serial Number: FCW2144L045 (used as configuration filename)
• IOS-XE Version: 17.3.1, 17.6.4, etc.

Logging Initialization:

• Create rotating log file on flash: /flash/guest-share/ztp.log
• Configure JSON formatter for syslog (if enabled)
• Add device context to every log message (serial, model, session ID)

Switch# show version | include Serial
System Serial Number : FCW2144L045

Stage 2: Configuration Download with Retry Logic¶

    config_file = f"{serial_number}.cfg"  # Build filename
    file_transfer_with_retry(HTTP_SERVER, config_file, max_attempts=3)

Retry Schedule (Exponential Backoff):

Why Retry Logic Matters:

• Network Transients: Spanning tree convergence, switch uplink negotiation
• HTTP Server Load: Concurrent requests from multiple switches during mass deployment
• DHCP Timing: IP address acquisition may still be stabilizing

Verification After Each Attempt:

    # Don't just trust the copy command - verify file exists
    if check_file_exists(config_file):
        log_and_print(f"File '{config_file}' successfully downloaded and verified.")
        return True

Stage 3: Configuration Application¶

    configure_merge(config_file)     # Merge config to running-config
    secure_delete(config_file)       # Remove temporary file for security
    hostname = get_hostname()        # Extract configured hostname for logging

Configuration Merge Process:

1. Execute copy flash:SERIAL.cfg running-config
2. IOS-XE merges configuration (additive operation)
3. Validate merge success by querying hostname from running-config
4. Delete configuration file from flash (security best practice)

Why Delete Config Files?

Configuration files contain sensitive data:

• Local user passwords (even if hashed, still sensitive)
• SNMP community strings
• TACACS+ / RADIUS shared secrets
• VTY passwords

Leaving these files on flash creates a security risk. The secure_delete() function removes them immediately after applying.

Stage 4: SSH Key Generation¶

    generate_crypto_keys()  # Create 2048-bit RSA keys

Operation:

    crypto key generate rsa label ZTP_Key_General modulus 2048

Why This Matters:

• SSH Server Activation: IOS-XE doesn't enable SSH until crypto keys exist
• Immediate Remote Access: Network engineers can SSH to device as soon as ZTP completes
• Time Savings: Manual key generation takes 10-30 seconds per device (automated here)

Performance Impact:

• Key Generation Time: 15-30 seconds (varies by platform)
• CPU Usage: Brief spike during key creation (normal, non-disruptive)

Stage 5: Save Configuration¶

    save_configuration()  # Write memory

Critical Importance:

• Running-config is volatile — lost on reload
• Startup-config is persistent (NVRAM) — survives reboots
• If not saved, device re-enters ZTP mode on next boot

Verification:

    Switch# show startup-config | include hostname
    hostname SW-ACCESS-01

If hostname appears in startup-config, configuration save succeeded.

Stage 6: Optional Reporting and Telemetry¶

    if ENABLE_REPORT_UPLOAD:
        facts = collect_device_facts()
        push_report_file(facts)

JSON Report Structure:

    {
    "model": "C9300-48U",
    "serial_number": "FCW2144L045",
    "version": "17.3.1",
    "hostname": "SW-ACCESS-01",
    "timestamp": "2026-02-06 14:21:45",
    "ztp_status": "SUCCESS"
    }

Use Cases:

• Inventory Automation: NetBox, CMDB auto-population
• Compliance Validation: Verify device matches expected model/version
• Audit Trails: Maintain deployment records for compliance

Infrastructure Requirements¶

Component Checklist¶

DHCP Server Configuration¶

ISC DHCP (Linux)¶

File: /etc/dhcp/dhcpd.conf

    subnet 192.168.1.0 netmask 255.255.255.0 {
        range 192.168.1.100 192.168.1.200;
        option routers 192.168.1.1;
        option domain-name-servers 8.8.8.8, 8.8.4.4;

        # ZTP Configuration (Option 67 - Boot File Name)
        option bootfile-name "http://192.168.1.235/scripts/day_0_provisioning.py";

        # Optional: Option 66 (TFTP Server - not used but some devices check it)
        # option tftp-server-name "192.168.1.235";
    }

Restart DHCP:

    sudo systemctl restart isc-dhcp-server
    sudo systemctl status isc-dhcp-server  # Verify running

Windows DHCP Server¶

PowerShell Method:

    # Set DHCP Option 67 (Boot File Name)
    Set-DhcpServerv4OptionValue -OptionId 67 `
        -Value "http://192.168.1.235/scripts/day_0_provisioning.py" `
        -ScopeId 192.168.1.0

    # Verify configuration
    Get-DhcpServerv4OptionValue -OptionId 67 -ScopeId 192.168.1.0

GUI Method:

1. Open DHCP Management Console
2. Navigate to Scope → Scope Options
3. Configure Option 67 (Bootfile Name):
   • String Value: http://192.168.1.235/scripts/day_0_provisioning.py
4. Click Apply and OK

Cisco IOS/IOS-XE DHCP Server¶

    ip dhcp pool ZTP_POOL
    network 192.168.1.0 255.255.255.0
    default-router 192.168.1.1
    dns-server 8.8.8.8 8.8.4.4
    option 67 ascii http://192.168.1.235/scripts/day_0_provisioning.py
    !
    ip dhcp excluded-address 192.168.1.1 192.168.1.99

HTTP Server Setup¶

Option A: Apache (Ubuntu/Debian)¶

    # Install Apache
    sudo apt update && sudo apt install apache2

    # Create directory structure
    sudo mkdir -p /var/www/html/scripts
    sudo mkdir -p /var/www/html/files

    # Copy ZTP script
    sudo cp day_0_provisioning.py /var/www/html/scripts/

    # Copy device configuration files
    sudo cp *.cfg /var/www/html/files/

    # Set permissions
    sudo chown -R www-data:www-data /var/www/html
    sudo chmod 644 /var/www/html/scripts/day_0_provisioning.py
    sudo chmod 644 /var/www/html/files/*.cfg

    # Restart Apache
    sudo systemctl restart apache2

    # Verify (should return Python script content)
    curl http://localhost/scripts/day_0_provisioning.py

Expected Directory Structure:

    /var/www/html/
    ├── scripts/
    │   └── day_0_provisioning.py
    └── files/
        ├── FCW2144L045.cfg
        ├── FDO2129Y06B.cfg
        └── FOC2201X0QY.cfg

Option B: Nginx (Ubuntu/Debian)¶

    # Install Nginx
    sudo apt update && sudo apt install nginx

    # Create directory structure
    sudo mkdir -p /usr/share/nginx/html/scripts
    sudo mkdir -p /usr/share/nginx/html/files

    # Copy files
    sudo cp day_0_provisioning.py /usr/share/nginx/html/scripts/
    sudo cp *.cfg /usr/share/nginx/html/files/

    # Set permissions
    sudo chown -R www-data:www-data /usr/share/nginx/html
    sudo chmod 644 /usr/share/nginx/html/scripts/*.py
    sudo chmod 644 /usr/share/nginx/html/files/*.cfg

    # Restart Nginx
    sudo systemctl restart nginx

    # Test
    curl http://localhost/scripts/day_0_provisioning.py

Option C: Python SimpleHTTPServer (Testing Only)¶

    # Create directory structure
    mkdir -p ztp-server/{scripts,files}
    cd ztp-server

    # Copy files
    cp /path/to/day_0_provisioning.py scripts/
    cp /path/to/*.cfg files/

    # Start server (Python 3)
    python3 -m http.server 80

    # Verify from another terminal
    curl http://localhost/scripts/day_0_provisioning.py

Syslog/Graylog Configuration (Optional)¶

Why Centralized Logging?

• Search by Serial Number — Find specific device logs without knowing IP address
• Real-Time Monitoring — Watch ZTP progress across entire deployment
• Alerting — Trigger notifications on ZTP failures
• Compliance — Maintain audit trail of all provisioning activities

Graylog Input Configuration:

1. Navigate to System → Inputs
2. Select Syslog UDP input type
3. Bind address: 0.0.0.0
4. Port: 514
5. Click Launch Input

Graylog Search Examples:

    # Find specific device by serial number
    serial_number:FCW2144L045

    # Find all C9300 provisioning events
    device_model:C9300*

    # Find successful completions
    message:"ZTP PROCESS COMPLETE"

    # Find failures
    message:"FAILED" OR level:CRITICAL

    # Find events from specific ZTP session
    ztp_session:FCW2144L045_1675891234

Alerting Example:

Create alert for ZTP failures:

• Condition: message:"FAILED" OR level:CRITICAL
• Action: Send email to network-ops@company.com
• Threshold: 1 message in 5 minutes

Configuration File Management¶

Filename Convention (Critical)¶

Rule: Configuration files MUST be named <SERIAL_NUMBER>.cfg exactly.

Finding Serial Numbers:

    Switch# show version | include Serial
    System Serial Number : FCW2144L045

Configuration Filename: FCW2144L045.cfg

Configuration File Template¶

Minimal Production Config:

    !
    ! Minimal ZTP Configuration Template
    ! Device: Cisco Catalyst 9300
    ! Purpose: Day 0 provisioning via ZTP
    !
    hostname SW-ACCESS-01
    !
    ! Enable secret (Cisco Type 5 hash)
    enable secret YourEnablePassword
    !
    ! Management user (privilege 15 = full admin)
    username admin privilege 15 secret YourAdminPassword
    !
    ! DNS domain (required for SSH key generation)
    ip domain-name company.local
    !
    ! Management VLAN
    vlan 10
    name MGMT
    !
    interface Vlan10
    description Management VLAN
    ip address 10.1.10.50 255.255.255.0
    no shutdown
    !
    ip default-gateway 10.1.10.1
    !
    ! NTP Configuration
    ntp server 10.1.1.1
    !
    ! Syslog Configuration
    logging host 10.1.1.31
    logging trap informational
    !
    ! VTY Access (SSH only, local authentication)
    line vty 0 15
    transport input ssh
    login local
    exec-timeout 15 0
    !
    ! Console settings
    line con 0
    exec-timeout 15 0
    logging synchronous
    !
    ! Disable ZTP after first provision (prevents re-ZTP after 'write erase')
    ! Uncomment if you want ZTP permanently disabled:
    ! no service dhcp
    !
    end

Advanced Configuration (AAA + TACACS+):

    !
    ! Advanced ZTP Configuration with AAA
    !
    hostname SW-CORE-01
    !
    ! AAA Configuration
    aaa new-model
    aaa authentication login default group tacacs+ local
    aaa authorisation exec default group tacacs+ local
    aaa authorisation commands 15 default group tacacs+ local
    aaa accounting exec default start-stop group tacacs+
    aaa accounting commands 15 default start-stop group tacacs+
    !
    ! TACACS+ Server
    tacacs server ISE-TACACS
    address ipv4 10.1.1.20
    key 7 YourEncryptedTACACSKey
    !
    ! Fallback local account (emergency access)
    username admin privilege 15 secret YourStrongPassword
    !
    ip domain-name company.local
    !
    ! Management VLAN
    vlan 10
    name MGMT
    !
    interface Vlan10
    description Management Interface
    ip address 10.1.10.100 255.255.255.0
    no shutdown
    !
    ip default-gateway 10.1.10.1
    !
    ! NTP with authentication
    ntp authenticate
    ntp authentication-key 1 md5 YourNTPKey
    ntp trusted-key 1
    ntp server 10.1.1.1 key 1
    !
    ! SNMP (read-only)
    snmp-server community YourROCommunity RO
    snmp-server location Building-A Floor-3 IDF-1
    snmp-server contact network-ops@company.com
    !
    ! Syslog
    logging host 10.1.1.31 transport udp port 514
    logging trap notifications
    logging source-interface Vlan10
    !
    ! VTY hardening
    ip access-list standard VTY-ACCESS
    permit 10.1.0.0 0.0.255.255
    deny any log
    !
    line vty 0 15
    access-class VTY-ACCESS in
    transport input ssh
    exec-timeout 15 0
    !
    ! Disable unused services
    no ip http server
    no ip http secure-server
    !
    end

Generating Config Files at Scale¶

For Small Deployments (1-20 devices): Manual creation or Excel-based templating

For Medium Deployments (20-100 devices): Python/Jinja2 templating

For Large Deployments (100+ devices): NetBox + CI/CD pipeline

Example: Python + Jinja2 Templating¶

    from jinja2 import Template
    import openpyxl

    # Load Excel inventory (columns: Serial, Hostname, IP, Gateway, VLAN)
    wb = openpyxl.load_workbook('device_inventory.xlsx')
    ws = wb.active

    # Jinja2 template
    template_str = """
    hostname {{ hostname }}
    !
    enable secret {{ enable_password }}
    username admin privilege 15 secret {{ admin_password }}
    !
    ip domain-name company.local
    !
    vlan {{ mgmt_vlan }}
    name MGMT
    !
    interface Vlan{{ mgmt_vlan }}
    description Management VLAN
    ip address {{ mgmt_ip }} {{ mgmt_mask }}
    no shutdown
    !
    ip default-gateway {{ default_gateway }}
    !
    line vty 0 15
    transport input ssh
    login local
    !
    end
    """

    template = Template(template_str)

    ## Generate config for each device

    for row in ws.iter_rows(min_row=2, values_only=True):
        serial, hostname, mgmt_ip, gateway, vlan = row

        config = template.render(
            hostname=hostname,
            enable_password='YourEnableSecret',
            admin_password='YourAdminSecret',
            mgmt_vlan=vlan,
            mgmt_ip=mgmt_ip,
            mgmt_mask='255.255.255.0',
            default_gateway=gateway
        )

        # Write to file named by serial number
        with open(f'configs/{serial}.cfg', 'w') as f:
            f.write(config)

        print(f'Generated: {serial}.cfg ({hostname})')

Script Configuration Options¶

All configurable parameters are located at the top of day_0_provisioning.py:

HTTP Server Settings¶

    HTTP_SERVER = "192.0.2.235"  # IP or hostname of HTTP server

Change this to: Your HTTP server's IP address or DNS hostname

Note: DNS hostname requires functioning DNS resolution on management VLAN

Logging Settings¶

    LOG_TO_FILE = True                       # Enable persistent flash logging
    LOG_PATH_PRIMARY = "/flash/guest-share/ztp.log"  # Preferred log location
    LOG_PATH_FALLBACK = "/flash/ztp.log"     # Fallback if guest-share unavailable

Log Rotation:

• Max Size: 5MB per log file
• Backups: 2 (ztp.log.1, ztp.log.2)
• Format: 2026-02-06 14:20:03 :: INFO :: Message

Retry Policy¶

    MAX_COPY_ATTEMPTS = 3      # Number of download attempts
    BASE_BACKOFF_SECONDS = 2   # Base delay for exponential backoff

Retry Schedule:

• Attempt 1: Immediate
• Attempt 2: Wait 2 seconds
• Attempt 3: Wait 4 seconds

Increasing Retries (High-Latency Networks):

    MAX_COPY_ATTEMPTS = 5
    BASE_BACKOFF_SECONDS = 5  # 5s, 10s, 20s, 40s

Configuration Persistence¶

    WRITE_MEMORY = True  # MUST BE TRUE for production

Syslog/Graylog Integration¶

    ENABLE_SYSLOG = False              # Set to True to enable
    SYSLOG_SERVER = "192.0.2.50"       # Graylog/Syslog server IP
    SYSLOG_PORT = 514                  # UDP port (514 = standard syslog)

JSON Log Structure:

    {
    "timestamp": "2026-02-06 14:21:45",
    "level": "INFO",
    "message": "Configuration merge completed",
    "script": "cisco_ztp",
    "version": "2.0",
    "serial_number": "FCW2144L045",
    "device_model": "C9300-48U",
    "ztp_session": "FCW2144L045_1675891234"
    }

Device Reporting¶

    ENABLE_REPORT_UPLOAD = False  # Set to True to enable local JSON reports

When enabled, creates /flash/ztp_report_<SERIAL>.json:

    {
    "model": "C9300-48U",
    "serial_number": "FCW2144L045",
    "version": "17.3.1",
    "hostname": "SW-ACCESS-01",
    "timestamp": "2026-02-06 14:21:45",
    "ztp_status": "SUCCESS"
    }

Use Case: Post-ZTP automation (Ansible/AWX) can collect these JSON files and populate CMDB/NetBox.

Deployment Workflow¶

Pre-Deployment (One-Time Setup)¶

1. Prepare Infrastructure

   • Deploy HTTP server (Apache/Nginx)
   • Configure DHCP server with Option 67
   • (Optional) Set up Graylog/Syslog server

2. Customize Script

   • Edit HTTP_SERVER variable in day_0_provisioning.py
   • Enable syslog if desired
   • Upload script to HTTP server: /var/www/html/scripts/day_0_provisioning.py

3. Create Configuration Files

   • Generate device-specific .cfg files (manual or scripted)
   • Upload to HTTP server: /var/www/html/files/<SERIAL>.cfg

4. Validate Infrastructure

           # Test HTTP server accessibility
           curl http://192.168.1.235/scripts/day_0_provisioning.py
           curl http://192.168.1.235/files/FCW2144L045.cfg
   
           # Verify DHCP Option 67
           # (From DHCP client test or switch console)
           show dhcp lease  # Should show Option 67 with script URL

Per-Device Deployment¶

1. Verify Serial Number

           Switch# show version | include Serial
           System Serial Number : FCW2144L045

2. Create Matching Config File

   • Filename: FCW2144L045.cfg
   • Upload to HTTP server: /var/www/html/files/FCW2144L045.cfg

3. Erase Existing Config (If Re-Provisioning)

           Switch# write erase
           Erasing the nvram filesystem will remove all configuration files! Continue? [confirm]
           [OK]
           Erase of nvram: complete
   
           Switch# reload
           Proceed with reload? [confirm]

4. Connect to Network and Power On

   • Connect management port or uplink to network with DHCP access
   • Power on switch
   • Do not touch console — ZTP needs uninterrupted boot

5. Monitor Progress

Option A: Console Monitoring¶

    ###### STARTING ZTP SCRIPT ######

    Device Model: C9300-48U
    Serial Number: FCW2144L045
    IOS-XE Version: 17.3.1

    --- Stage 1: Downloading Device Configuration ---
    Downloading 'FCW2144L045.cfg' from server 192.168.1.235...
    File 'FCW2144L045.cfg' successfully downloaded and verified.

    --- Stage 2: Applying Configuration ---
    Merging 'FCW2144L045.cfg' with running-config...
    Configuration merge completed.
    Device configured with hostname: SW-ACCESS-01

    --- Stage 3: Generating SSH Keys ---
    Generating new 2048-bit RSA keys...
    New RSA keys generated successfully.

    --- Stage 4: Saving Configuration ---
    Writing config to Memory...
    Configuration saved successfully.

    ###### ZTP PROCESS COMPLETE ######

Option B: Graylog Monitoring¶

    # Real-time search (auto-refresh every 5 seconds)
    serial_number:FCW2144L045

    # Successful completion indicator
    message:"ZTP PROCESS COMPLETE" AND serial_number:FCW2144L045

Option C: Flash Log Review (Post-ZTP)¶

    Switch# more flash:guest-share/ztp.log
    2026-02-06 14:20:03 :: INFO :: ###### STARTING ZTP SCRIPT ######
    2026-02-06 14:20:04 :: INFO :: Device Model: C9300-48U
    2026-02-06 14:20:04 :: INFO :: Serial Number: FCW2144L045
    ...
    2026-02-06 14:21:45 :: INFO :: ###### ZTP PROCESS COMPLETE ######

1. Post-Provisioning Verification

       ! Verify SSH keys exist
       Switch# show crypto key mypubkey rsa
       % Key pair was generated at: 14:21:30 UTC Feb 6 2026
       Key name: ZTP_Key_General
       Storage Device: not specified
       Usage: General Purpose Key
       Key is not exportable.
       Key Data:
       [...key data...]
   
       ! Verify startup-config saved
       Switch# show startup-config | include hostname
       hostname SW-ACCESS-01
   
       ! Test SSH access (from management workstation)
       ssh admin@10.1.10.50
       Password:
       SW-ACCESS-01#

Troubleshooting Guide¶

Common Issues and Solutions¶

Issue 1: ZTP Doesn't Start¶

Symptoms:

• Switch boots normally, prompts for initial configuration dialog
• No ZTP activity visible in logs

Causes:

Verification Steps:

    ! Check if ZTP is enabled (should be enabled by default)
    Switch# show boot | include ZTP
    ZTP is enabled

    ! Verify DHCP lease and Option 67
    Switch# show dhcp lease
    Temp IP addr: 192.168.1.105 for peer on Interface: GigabitEthernet1/0/1
    Temp sub net mask: 255.255.255.0
    DHCP Lease server: 192.168.1.1, state: 5 Bound
    ...
    Option 67 = "http://192.168.1.235/scripts/day_0_provisioning.py"

Issue 2: HTTP 404 Error (Config File Not Found)¶

Symptoms:

    File 'FCW2144L045.cfg' not found.
    Failed to transfer 'FCW2144L045.cfg' after 3 attempts.

Causes:

Verification:

    # From HTTP server
    ls -la /var/www/html/files/ | grep FCW2144L045
    -rw-r--r-- 1 www-data www-data 1234 Feb  6 09:30 FCW2144L045.cfg

    # Test HTTP access from another machine
    curl http://192.168.1.235/files/FCW2144L045.cfg

Issue 3: Network Timeout / Retry Failures¶

Symptoms:

    Attempt 1 failed: Network timeout
    Attempt 2 failed: Network timeout
    Attempt 3 failed: Network timeout
    Failed to transfer 'FCW2144L045.cfg' after 3 attempts.

Causes:

Mitigation:

    # In day_0_provisioning.py, increase retry attempts
    MAX_COPY_ATTEMPTS = 5        # Up from 3
    BASE_BACKOFF_SECONDS = 5     # Up from 2 (gives STP more time)

Issue 4: Configuration Syntax Errors¶

Symptoms:

    % Invalid input detected at '^' marker.
    Configuration merge completed.  # But not all commands applied

Causes:

• Invalid IOS-XE commands in .cfg file
• Commands not supported on device platform
• Syntax errors (typos, missing keywords)

Solution:

1. Pre-validate config files:

           ! Test config on lab device first
           Switch# configure terminal
           Switch(config)# [paste entire config]
           Switch(config)# end
           Switch# show running-config

2. Common syntax errors:

           ! WRONG: Missing 'no shutdown' on SVI
           interface Vlan10
           ip address 10.1.10.50 255.255.255.0
           !
           ! CORRECT: Always include 'no shutdown' on SVIs
           interface Vlan10
           ip address 10.1.10.50 255.255.255.0
           no shutdown

Issue 5: SSH Connection Refused Post-ZTP¶

Symptoms:

• ZTP completes successfully
• Cannot SSH to device: "Connection refused"

Causes:

Manual Fix:

    ! Configure IP domain (prerequisite for SSH)
    Switch(config)# ip domain-name company.local

    ! Generate RSA keys
    Switch(config)# crypto key generate rsa modulus 2048

    ! Configure VTY for SSH
    Switch(config)# line vty 0 15
    Switch(config-line)# transport input ssh
    Switch(config-line)# login local
    Switch(config-line)# end

    ! Verify SSH enabled
    Switch# show ip ssh
    SSH Enabled - version 2.0

Issue 6: Config Not Saved (Re-ZTP on Reboot)¶

Symptoms:

• ZTP completes successfully
• Device reboots and re-runs ZTP

Cause:

• WRITE_MEMORY = False in script (wrong setting)
• write memory command failed silently

Verification:

    ! Check if startup-config matches running-config
    Switch# show startup-config | include hostname
    hostname SW-ACCESS-01  # Should match running-config

    ! If missing, manually save
    Switch# write memory
    Building configuration...
    [OK]

Fix:

Ensure WRITE_MEMORY = True in day_0_provisioning.py (default setting)

Security Considerations¶

Threat Model¶

Best Practices¶

1. Use Isolated Management VLAN¶

    ! Trunk only management VLAN to ZTP devices
    interface GigabitEthernet1/0/48
    description ZTP Uplink
    switchport mode trunk
    switchport trunk allowed vlan 10
    spanning-tree portfast trunk

Why: Prevents ZTP traffic from reaching production VLANs

2. Enable DHCP Snooping¶

    ! On distribution/access switches
    ip dhcp snooping
    ip dhcp snooping vlan 10

    ! Trust port connected to legitimate DHCP server
    interface GigabitEthernet1/0/1
    description Trunk to DHCP Server
    ip dhcp snooping trust

Why: Prevents rogue DHCP servers from hijacking ZTP

3. Restrict HTTP Server Access¶

Firewall Rule (Linux iptables):

    # Allow HTTP only from management VLAN
    sudo iptables -A INPUT -p tcp --dport 80 -s 192.168.1.0/24 -j ACCEPT
    sudo iptables -A INPUT -p tcp --dport 80 -j DROP

Apache Virtual Host (IP-based restriction):

    <VirtualHost *:80>
        DocumentRoot /var/www/html

        <Directory /var/www/html>
            Require ip 192.168.1.0/24
        </Directory>
    </VirtualHost>

4. Disable ZTP After Provisioning¶

Option A: Explicitly Disable DHCP-Based Provisioning¶

    ! Prevents device from re-attempting ZTP
    Switch(config)# no service dhcp

Option B: Remove Startup-Config Detection¶

Add to ZTP config template:

    ! This command is processed only after startup-config exists
    ! Prevents future ZTP even if startup-config is erased
    event manager applet DISABLE_ZTP
    event none
    action 1.0 cli command "enable"
    action 2.0 cli command "config t"
    action 3.0 cli command "no boot network"
    action 4.0 cli command "end"
    action 5.0 cli command "write memory"

5. Secure Credential Storage¶

username admin privilege 15 password MyPassword123

! Use Type 8 (PBKDF2) or Type 9 (scrypt) secrets
username admin privilege 15 secret MyStrongPassword

! Result in running-config (hashed):
username admin privilege 15 secret 8 $8$xyz...

Generating Hashed Passwords:

    ! On any Cisco device
    Switch# show run | include username
    username admin privilege 15 secret YourPassword

    ! Output (Type 5 hash - MD5):
    username admin privilege 15 secret 5 $1$abc123$xyz...

    ! Copy hash to config template

6. Audit Trail and Compliance¶

Logging Requirements:

• Who: Operator/automation system initiated ZTP
• What: Device serial, model, config applied
• When: Timestamp (use NTP for accuracy)
• Where: Device location (building, floor, IDF)
• Result: Success or failure with error details

Implementation:

Enable Syslog to Graylog with retention policies:

• Retention: 90 days minimum (compliance requirement)
• Backup: Daily Graylog index backups
• Access Control: Role-based access to logs (NetOps, SecOps, Audit)

Advanced Topics¶

Multi-Platform Support¶

Supported Platforms (Tested):

• Catalyst 9200 Series (all models)
• Catalyst 9300 Series (all models)
• Catalyst 9400 Series (all models)
• Catalyst 9500 Series (all models)
• Catalyst 9600 Series (all models)

Untested (Should Work):

• Catalyst 3850/3650 (IOS-XE 16.x with Guestshell)
• ASR 1000 Series (IOS-XE routers)
• ISR 4000 Series (IOS-XE routers)

Platform-Specific Considerations:

StackWise Considerations¶

Challenge: StackWise stack has multiple members, each with unique serial number. Which serial do you use for config filename?

Solution Options:

Option 1: Stack Master Serial (Recommended)¶

    # Modify get_serial() function to detect stack and return master serial
    def get_serial():
        show_version = cli("show version")

        # Check if device is in a stack
        if "Switch Ports Model" in show_version:  # Stack detected
            # Extract master switch serial
            switch_lines = re.findall(r"\*?\s+\d+\s+\d+\s+(\S+)\s+(\S+)", show_version)
            for line in switch_lines:
                if line.startswith("*"):  # Master switch
                    return line.split()[3]  # Return master serial

        # Non-stack device or fallback
        serial_match = re.search(r"System Serial Number\s+:\s+(\S+)", show_version)
        if serial_match:
            return serial_match.group(1)

Option 2: Separate Configs Per Member¶

Create configs for each stack member:

• FCW2144L045.cfg (Member 1 serial)
• FDO2129Y06B.cfg (Member 2 serial)
• FOC2201X0QY.cfg (Member 3 serial)

Challenge: Only master runs ZTP, so only master's config gets applied. Members inherit config automatically.

Best Practice: Use Option 1 (master serial) with stack-wide configuration.

HTTPS Support (Custom Implementation)¶

Native ZTP Limitation: Cisco ZTP only supports HTTP, not HTTPS.

Workaround: Use HTTP for ZTP script, implement HTTPS within script for config download.

Modified Script:

    import ssl
    import urllib.request

    def file_transfer_https(server, file):
        """Download file via HTTPS (bypassing native ZTP limitation)"""
        url = f"https://{server}/files/{file}"
        dest = f"/flash/{file}"

        # Create SSL context (disable cert verification for self-signed)
        ssl_context = ssl._create_unverified_context()

        try:
            with urllib.request.urlopen(url, context=ssl_context) as response:
                with open(dest, 'wb') as f:
                    f.write(response.read())
            return True
        except Exception as e:
            log_and_print(f"HTTPS download failed: {e}", "error")
            return False

Trade-off: Requires Python ssl module (available in Guestshell) but adds complexity.

Integration with NetBox/IPAM¶

Use Case: Dynamic config generation based on NetBox device records.

Workflow:

1. Pre-populate NetBox with device records (serial, hostname, IP, site)
2. ZTP script queries NetBox API using device serial number
3. NetBox returns device attributes (hostname, IP, VLAN, etc.)
4. Script generates config dynamically using Jinja2 template
5. Apply config (no pre-created .cfg files needed)

Modified Script:

    import requests
    import json
    from jinja2 import Template

    def get_device_from_netbox(serial):
        """Query NetBox API for device by serial number"""
        netbox_url = "https://netbox.company.local/api/dcim/devices/"
        headers = {"Authorisation": "Token YOUR_NETBOX_API_TOKEN"}
        params = {"serial": serial}

        response = requests.get(netbox_url, headers=headers, params=params, verify=False)
        devices = response.json()['results']

        if devices:
            return devices[0]  # Return first match
        return None

    def generate_config_from_template(device_data):
        """Generate config using Jinja2 template and NetBox data"""
        template_str = """
    hostname {{ hostname }}
    !
    interface Vlan{{ mgmt_vlan }}
    ip address {{ mgmt_ip }} {{ mgmt_mask }}
    no shutdown
    !
    ip default-gateway {{ default_gateway }}
    !
        """

        template = Template(template_str)
        config = template.render(
            hostname=device_data['name'],
            mgmt_vlan=device_data['primary_ip']['vlan'],
            mgmt_ip=device_data['primary_ip']['address'].split('/')[0],
            mgmt_mask='255.255.255.0',
            default_gateway=device_data['primary_ip']['gateway']
        )

        return config

    # In main() function, replace file download with:
    device_data = get_device_from_netbox(serial_number)
    if device_data:
        config_content = generate_config_from_template(device_data)
        with open(f"/flash/{serial_number}.cfg", 'w') as f:
            f.write(config_content)

Benefits:

• No manual config file creation
• Single source of truth (NetBox)
• Dynamic updates (change NetBox, re-provision device)

Comparison: Native ZTP vs. Manual Provisioning vs. PnP¶

Feature Comparison Matrix¶

When to Use Each Approach¶

Best Practices and Lessons Learned¶

Configuration Management¶

# Initialize Git repository for config files
cd /var/www/html/files/
git init
git add *.cfg
git commit -m "Initial ZTP config files"

# Track changes over time
git log --oneline
git diff FCW2144L045.cfg

Testing Strategy¶

Lab Validation Checklist:

• Test ZTP with single device (happy path)
• Test with wrong serial number (404 error handling)
• Test with network disconnection during download (retry logic)
• Test with config syntax errors (graceful failure)
• Test with multiple devices simultaneously (HTTP server load)
• Test StackWise configuration (if applicable)
• Test Graylog integration (verify logs appear)
• Test SSH access post-ZTP (key generation success)

Operational Considerations¶

Maintenance Window Planning:

• Small deployment (1-20 devices): 2-hour window
• Medium deployment (20-100 devices): 4-hour window
• Large deployment (100+ devices): Stagger over multiple nights

Staggered Deployment:

    # Power on devices in batches to avoid overwhelming HTTP/DHCP
    # Batch 1: Devices 1-50 (wait for completion)
    # Batch 2: Devices 51-100 (wait for completion)
    # Batch 3: Devices 101-150

Rollback Plan:

If ZTP fails catastrophically:

1. Console access — Connect to failed devices manually
2. Manual config — Apply minimal config for SSH access
3. Post-mortem — Analyze logs to identify root cause
4. Fix and retry — Correct issue, write erase, reload

Performance Benchmarks¶

Typical Execution Times (Catalyst 9300)¶

Scalability Testing Results¶

Key Takeaway: Modern HTTP servers (Apache/Nginx) handle 50-100 concurrent ZTP sessions without tuning. For >100 devices, stagger power-on or increase server resources.

Future Enhancements (Roadmap)¶

Troubleshooting Decision Tree¶

    graph TD
        A[ZTP Issue Detected] --> B{Did ZTP Start?}
        B -->|No| C{Startup-config exists?}
        C -->|Yes| D[write erase + reload]
        C -->|No| E{DHCP Option 67 configured?}
        E -->|No| F[Configure DHCP Option 67]
        E -->|Yes| G[Check network connectivity]

        B -->|Yes| H{Config file downloaded?}
        H -->|No| I{HTTP 404 error?}
        I -->|Yes| J[Verify filename matches serial]
        I -->|No| K{Network timeout?}
        K -->|Yes| L[Increase retry attempts
Check STP convergence]
        K -->|No| M[Check HTTP server accessibility]

        H -->|Yes| N{Config applied successfully?}
        N -->|No| O[Check config syntax errors]
        N -->|Yes| P{SSH keys generated?}
        P -->|No| Q[Verify ip domain-name configured]
        P -->|Yes| R{Config saved to startup?}
        R -->|No| S[Manually: write memory]
        R -->|Yes| T[ZTP Success!]

Support and Contributions¶

Author: Christopher Davies
Email: enquiries@nautomationprime.io
License: GNU General Public License v3.0

Repository: (Add GitHub/GitLab URL when published)

Reporting Issues:

• Provide device model and IOS-XE version
• Include flash log: more flash:guest-share/ztp.log
• Attach sanitized config file (redact passwords)
• Describe expected vs. actual behaviour

Conclusion¶

This Cisco IOS-XE Zero Touch Provisioning script transforms Day 0 deployment from a manual, error-prone process into a reliable, scalable automation workflow. With structured logging, retry logic, and enterprise-grade error handling, it's production-ready for deployments ranging from 10 devices to 1000+.

Key Takeaways:

✅ Hands-free provisioning — 60-90 seconds from power-on to production-ready
✅ Serial-based config lookup — Each device gets unique configuration
✅ Centralized monitoring — Graylog integration for real-time visibility
✅ Production-grade reliability — Exponential backoff, retry logic, secure cleanup
✅ Zero licensing cost — Open-source alternative to Cisco PnP

Next Steps:

1. Review your DHCP/HTTP infrastructure compatibility
2. Create device-specific configuration files
3. Deploy in lab environment (1-5 test devices)
4. Validate with Graylog monitoring
5. Scale to production (staged rollout recommended)

Happy automating! 🚀