CLI Parsing — TextFSM, TTP & Genie for Structured Network Data

The Problem — Unstructured CLI Output and Why Parsing Matters

Network engineers frequently interact with network devices through command-line interfaces (CLI), executing commands such as show ip route, show interfaces, or show version. While these commands provide vital operational data, their output is often unstructured, formatted as human-readable text rather than machine-readable data. This unstructured nature poses significant challenges when automating network management tasks, troubleshooting, or integrating device outputs into network automation workflows.

For instance, a typical show ip route command might output multiple routes with varying formats, making it difficult for scripts to reliably extract specific information like network prefixes, next-hop addresses, or route types. Manually parsing such output is error-prone, tedious, and not scalable across large networks or diverse device types.

To address these challenges, network engineers need efficient methods to convert raw CLI output into structured data formats such as dictionaries or JSON objects. Structured data simplifies automation, enabling tasks like generating network topology maps, verifying configurations, or integrating device states into orchestration tools. This is where CLI parsing libraries and techniques such as TextFSM TTP CLI parsing become essential, providing a systematic way to interpret and extract meaningful information from unstructured CLI outputs.

In the context of Python-based network automation, tools like TextFSM, TTP, and Cisco Genie offer powerful solutions for structured CLI parsing. By leveraging these tools, network engineers can build robust automation scripts that reliably interpret device outputs, reduce manual intervention, and accelerate network operations. This chapter explores these parsing techniques in detail, equipping you with the knowledge to implement effective CLI parsing in your network automation projects.

TextFSM — Template Syntax, Value Definitions & Rules

TextFSM is a Python library designed to parse semi-structured text output, particularly CLI commands from network devices, into structured data. It achieves this through the use of templates that define how to interpret specific command outputs. These templates specify patterns, value extraction rules, and data structures, allowing for consistent parsing across similar command outputs.

Understanding TextFSM Templates

At the core of TextFSM are templates written in a simple syntax that combines regular expressions with a structured format. A typical TextFSM template contains:

• Header Section: Declares the variables to be extracted.
• States: Define the different parsing stages, each associated with regex rules.
• Rules: Regular expressions that match lines and extract data into variables.

For example, consider parsing the output of show ip route. A simplified TextFSM template might look like:

Value ROUTE_TYPE (\S+)
Value PREFIX (\d+\.\d+\.\d+\.\d+\/\d+)
Value NEXTHOP (\d+\.\d+\.\d+\.\d+)
Value INTERFACE (\S+)

Start
  ^${ROUTE_TYPE} +${PREFIX} +via +${NEXTHOP} +dev +${INTERFACE} -> RecordRoute
  ^${ROUTE_TYPE} +${PREFIX} -> RecordRoute

Values and Rules Explained

The Value lines define regex patterns for each piece of information to extract, such as route type, prefix, next-hop IP, and interface. The Start state processes each line, matching regex patterns. When a match is found, the captured groups populate the variables, and the parser moves to the corresponding state.

This approach allows precise extraction even from complex or multiline outputs. TextFSM templates are reusable and can be shared across different scripts or projects, making them a cornerstone of effective network automation.

Creating Effective TextFSM Templates

Designing templates requires understanding the CLI command output thoroughly. Key steps include:

1. Run the command manually and observe the output structure.
2. Identify consistent patterns or delimiters that can be matched with regex.
3. Define variables for dynamic data points.
4. Construct regex rules for each pattern, testing iteratively for accuracy.

Tools like TextFSM GitHub repository provide example templates and documentation to assist in template development. Mastery of template syntax significantly improves the reliability and robustness of CLI parsing tasks.

NTC Templates — Pre-Built TextFSM Templates for 100+ Commands

The NetworkToCode (NTC) Templates project offers a comprehensive collection of pre-built TextFSM templates tailored for common network device show commands across multiple vendors, including Cisco, Juniper, Arista, and more. These templates simplify the process of parsing CLI output without creating new templates from scratch, accelerating automation projects and reducing errors.

Benefits of Using NTC Templates

• Time-Saving: Ready-to-use templates for over 100 commands, saving hours of development.
• Vendor Support: Templates cover multiple device types and OS versions.
• Community-Driven: Maintained by a community of network engineers, continuously updated with new templates.
• Standardization: Ensures consistent parsing across different environments.

Examples of NTC Templates

Some commonly used templates include:

• show ip route
• show interfaces
• show version
• show vlan brief
• show mac address-table

For example, the show ip route template from NTC captures routes with regex patterns matching the typical output, enabling scripts to parse and manipulate routing information effectively.

Integrating NTC Templates with Python

Using these templates involves downloading the relevant .textfsm files and applying them with the TextFSM Python library. A typical approach is:

1. Download the template file from the NTC Templates repository.
2. Load the template into a Python script with textfsm.TextFSM(open('show_ip_route.template')).
3. Pass the CLI output to the parser to obtain structured data.

This integration streamlines automation workflows, allowing network engineers at Networkers Home to efficiently parse device outputs and build network inventories, verify configurations, and generate reports.

TTP — Template Text Parser with Python-Native Syntax

While TextFSM relies on a specialized template syntax, TTP (Template Text Parser) offers a more Pythonic approach to CLI parsing, making template creation more intuitive for Python developers. TTP enables writing parsing rules directly in Python, improving readability and flexibility, especially for complex or multiline outputs.

Advantages of TTP

• Pythonic Syntax: Parsing rules are written using native Python idioms, such as regex in functions or dictionaries.
• Flexibility: Supports complex parsing scenarios, including multiline and nested structures.
• Extensibility: Easily integrates with other Python libraries and tools.

Basic TTP Example

Suppose we want to parse show ip route output. A simple TTP script might look like:

import ttp

model = {
    'routes': [
        {
            'route_type': r'(?P\S+)',
            'prefix': r'(?P\d+\.\d+\.\d+\.\d+\/\d+)',
            'nexthop': r'(via\s+(?P\d+\.\d+\.\d+\.\d+))?',
            'interface': r'(dev\s+(?P\S+))?'
        }
    ]
}

parser = ttp.TTP(model=model)
cli_output = ''''''  # Replace with actual CLI output
parser.parse(input=cli_output)
for route in parser.work['routes']:
    print(route)

In this example, the model defines parsing rules using regular expressions mapped to Python dictionary keys. TTP processes the CLI output and returns structured data, which can be used directly within Python scripts for automation tasks.

Comparison: TextFSM vs TTP

Choosing between TextFSM and TTP depends on your familiarity with regex, complexity of CLI outputs, and project requirements. For straightforward templates, TextFSM is efficient; for advanced parsing, TTP offers greater flexibility. Both tools are supported within the Python ecosystem and can be integrated seamlessly with network automation frameworks.

Cisco Genie — pyATS Parsers for Structured Show Command Data

Cisco Genie, part of the Cisco pyATS framework, provides an extensive set of pre-built parsers for Cisco IOS, IOS-XE, NX-OS, and other platforms. Unlike TextFSM or TTP, Genie outputs structured data directly from CLI commands, eliminating the need for manual template creation. It is designed for large-scale automation, testing, and validation tasks.

How Cisco Genie Works

Genie employs a combination of pre-defined parsers and models that interpret CLI outputs into Python dictionaries. When executing commands via pyATS, Genie automatically parses the output and returns structured data, like interface states, routing tables, VLANs, and more. This capability simplifies automation workflows and reduces errors associated with manual parsing.

Advantages of Cisco Genie

• Pre-Built Parsers: Over 200 commands covered, ready to use out-of-the-box.
• Vendor Support: Primarily Cisco devices, with some support for other vendors.
• Integration: Seamless with pyATS testbeds and automation scripts.
• Robustness: Handles variations across device OS versions and models.

Example Usage

from genie.conf import Genie
from pyats.topology import loader

testbed = loader.load('testbed.yaml')
device = testbed.devices['CSR1kv']
device.connect()
output = device.parse('show ip interface brief')
print(output)

The parse method returns a structured dictionary containing interface statuses, IP addresses, and other details, which can be directly used for automation or validation.

Comparison with TextFSM and TTP

For organizations heavily invested in Cisco infrastructure, Genie provides a streamlined, reliable method for extracting structured data, complementing or replacing manual parsing with TextFSM or TTP. Networkers Home offers specialized courses that cover pyATS and Genie for advanced network automation, enhancing your skill set in structured CLI parsing.

Choosing the Right Parser — TextFSM vs TTP vs Genie

Selecting the appropriate CLI parsing technique depends on several factors, including project complexity, device environment, and team expertise. Here is a detailed comparison to guide your decision:

In summary, if you require quick, reliable parsing with minimal setup for Cisco devices, Genie is optimal. For custom parsing needs or multi-vendor environments, TextFSM and TTP offer greater flexibility. Networkers Home provides specialized training on all these tools, enabling you to choose and implement the best solution for your network automation goals.

Integrating Parsers with Netmiko — use_textfsm & use_genie

To streamline CLI parsing in your network automation scripts, integrating with libraries like Netmiko is essential. Netmiko provides use_textfsm and use_genie parameters, which enable seamless parsing of command outputs using TextFSM templates or Genie parsers, respectively.

Using use_textfsm with Netmiko

When executing commands with Netmiko, set use_textfsm=True to automatically parse the output using matching TextFSM templates from the NTC Templates repository. Example:

from netmiko import ConnectHandler

device = {
    'device_type': 'cisco_ios',
    'host': '192.168.1.1',
    'username': 'admin',
    'password': 'password',
}
net_connect = ConnectHandler(**device)
output = net_connect.send_command('show ip route', use_textfsm=True)
print(output)  # Structured list of dictionaries

This approach simplifies extracting routing information, interface details, or other command outputs directly into Python data structures, facilitating further automation or analysis.

Using use_genie with Netmiko

Similarly, setting use_genie=True enables parsing with Cisco Genie parsers:

from netmiko import ConnectHandler

device = {
    'device_type': 'cisco_ios',
    'host': '192.168.1.1',
    'username': 'admin',
    'password': 'password',
}
net_connect = ConnectHandler(**device)
output = net_connect.send_command('show ip interface brief', use_genie=True)
print(output)  # Structured dictionary from Genie parser

Benefits of Integration

• Efficiency: Eliminates manual parsing code.
• Reliability: Uses tested templates and parsers.
• Scalability: Supports automation across large networks with minimal code changes.

Incorporating these features into your workflow enhances productivity and accuracy, especially when combined with training at Networkers Home, which offers courses on network automation best practices and tools integration.

Practice: Parse show ip route and Build a Route Table Dict

Let's apply what we've learned by parsing the show ip route command output and constructing a structured routing table dictionary.

Sample CLI Output

Codes: C - connected, S - static, R - RIP, O - OSPF, B - BGP

Gateway of last resort is 0.0.0.0 to network 0.0.0.0

C    192.168.1.0/24 is directly connected, GigabitEthernet0/1
S    10.0.0.0/8 [1/0] via 192.168.1.254, GigabitEthernet0/1
O    172.16.0.0/12 [110/20] via 192.168.2.1, GigabitEthernet0/2
R    192.168.2.0/24 [120/1] via 192.168.1.2, GigabitEthernet0/3

Parsing with TextFSM

Assuming you have a TextFSM template for show ip route, here's how to parse and build the route table:

import textfsm

# Load the TextFSM template
with open('ntc_templates/ios/show_ip_route.textfsm') as template_file:
    re_table = textfsm.TextFSM(template_file)

# Sample CLI output
cli_output = ''''''

# Parse output
parsed_rows = re_table.ParseText(cli_output)

# Build route table dictionary
route_table = []
for row in parsed_rows:
    route_entry = {
        'route_type': row[0],
        'prefix': row[1],
        'administrative_distance': row[2],
        'metric': row[3],
        'next_hop': row[4],
        'interface': row[5],
    }
    route_table.append(route_entry)

print(route_table)

Resulting Data Structure

[
  {
    'route_type': 'C',
    'prefix': '192.168.1.0/24',
    'administrative_distance': '',
    'metric': '',
    'next_hop': '',
    'interface': 'GigabitEthernet0/1'
  },
  {
    'route_type': 'S',
    'prefix': '10.0.0.0/8',
    'administrative_distance': '1/0',
    'metric': '0',
    'next_hop': '192.168.1.254',
    'interface': 'GigabitEthernet0/1'
  },
  ...
]

This structured approach facilitates further automation tasks such as route validation, network mapping, or dynamic routing adjustments. Mastering such techniques at Networkers Home enhances your ability to develop scalable network automation solutions.

Key Takeaways

• Unstructured CLI output hampers automation; parsing converts it into actionable data.
• TextFSM templates utilize regex-based syntax for flexible, reusable parsing rules.
• NTC Templates provide a library of pre-built patterns for common network commands, accelerating automation projects.
• TTP offers a Python-native approach to CLI parsing, suitable for complex or nested outputs.
• Cisco Genie supplies pre-built parsers for Cisco devices, integrating seamlessly with pyATS for large-scale automation.
• Choosing between TextFSM, TTP, and Genie depends on project scope, device environment, and team expertise.
• Integration with Netmiko simplifies applying parsers to live device commands, enabling efficient data extraction.

Frequently Asked Questions