High Memory Utilization due to hulc_flash

Facebooktwittergoogle_plusredditpinterestlinkedin

Two of my Cisco switches’ are having high memory utilization. They are all Cisco 2960 Catalyst switches using different IOS version.

Model: WS-C2960G-24TC-L
IOS: c2960-lanbasek9-mz.122-55.SE5.bin

Model: WS-C2960G-24TC-L
IOS: c2960-lanbasek9-mz.150-1.SE3.bin

Upon investigation these switches have Dead processes holding more than 1Mb of memory which considered to be high and indicates a possible memory leak as depicted by the output of show processes memory sorted.

cisco2960switch#sh processes memory sorted
Processor Pool Total:   23481200 Used:   17466136 Free:    6015064
      I/O Pool Total:    4194304 Used:    2370532 Free:    1823772
Driver te Pool Total:    1048576 Used:         40 Free:    1048536
 
PID TTY  Allocated      Freed    Holding    Getbufs    Retbufs Process
   0   0   23068504    6363484   13717016          0          0 *Init*
   0   0 2817000660  148397120    1442632 2321578851    1973474 *Dead*
   0   0          0          0     394484          0          0 *MallocLite*
 148   0    7020688    4151724     387324          0          0 Auth Manager
  69   0     476940     139404     247344          0          0 Stack Mgr Notification

To determine what were these dead processes, show memory dead totals output was analyzed. There were multiple instances of hulc_flash init.

cisco2960switch#show memory dead totals
                Head    Total(b)     Used(b)     Free(b)   Lowest(b)  Largest(b)
Processor    25D93F0    23481200    17473588     6007612     5223684     5887432
      I/O    2C00000     4194304     2375432     1818872     1700204     1817552
Driver te    1800000     1048576          44     1048532     1048532     1048532
 
 
Dead Proc Summary for: Processor
 
    PC          Total   Count  Name
0x00527388     733212      73  Flashfs Sector
0x013C5D40     327900       5  DSensor 128
0x00D38D78     186204      12  Virtual Exec
0x00D3DE1C      47416     200  Virtual Exec
0x00B86CD0      32812       1  AAA Acct Req ch
0x00525C8C      18432      32  Virtual Exec
0x005271DC      18264       6  file_map_entry_
0x0052B9F4      10044       1  Flashfs Sector Pointers
0x00B53FC4       9756       1  EEM Shell Director
0x0052C9C0       5044       1  Flashfs Sector buffers
0x0052C888       4092       1  hulc_flash init
0x012C01E8       3600       8  Spanning Tree Control Block
0x00C370C8       3256      22  CDP sw subblock
0x0068E638       3180      24  LLDP-MED local annex
0x0052233C       3160       2  hulc_flash init
0x005271BC       3044       1  file_map_entry_chunk
0x0001FF28       2736      27  Smartport macro autoexec subblock
0x008ABC74       2624      28  DSensor Cache CDP
0x005320A0       1972      19  hulc_flash init
0x01059744       1880       1  Virtual Exec
0x00C371BC       1876      22  CDP hw subblock
0x00533F24       1580       1  hulc_flash init
0x00312524       1544       1  INTF_RANGE_SWIDB
0x0017331C       1536      16  State Machine debug meta data
0x0017336C       1520      16  State Machine debug data
0x00D07B20       1428       1  Connection
0x005F6084       1404       9  HULC Spanning Tree instance coordination
0x00532058       1308      19  hulc_flash init
0x00532048       1292      19  hulc_flash init
0x006F52DC       1100       8  PM vp_data
0x0052CA14       1068       1  hulc_flash init
0x012CBE5C       1024       6  STP CONFIG sw subblock
0x003DD724        948       3  DTP SWSB
0x00BD90F4        916       1  Virtual Exec
0x013C59B8        800       5  DSensor 32
0x00726320        696       8  PM app data

One last important show command is show memory debug leaks. This command is a Memory Leak Detector tool that can be used to detect memory leaks on a switch/router that is running Cisco IOS software.

cisco2960switch#sh mem debug leaks
Adding blocks for GD...
 
                 Driver text memory
 
Address    Size   Alloc_pc  PID  Alloc-Proc       Name
 
                 I/O memory
 
Address    Size   Alloc_pc  PID  Alloc-Proc       Name
 
                 Processor memory
 
Address    Size   Alloc_pc  PID  Alloc-Proc       Name
26BB4D4       80   533ED8  0    *Dead*          hulc_flash init

Digging onto Cisco bug database, I’m able to find related bugs to this issue. These are CSCth60511 and CSCua52463. I haven’t reloaded or upgraded its IOS to higher version yet, I’ll update this post once I done so.

CSCth60511

CSCua52463

Rejohn Ronald Cuares

He is Network Engineer working in one of the Managed Service Providers in Brisbane, Australia. His role as an engineer spans across multiple technologies like Unified Communications, Security, Wireless and Routing/Switching. He loves learning new things that makes his job fun and interesting.

Latest posts by Rejohn Ronald Cuares (see all)

11 comments to High Memory Utilization due to hulc_flash

Leave a Reply

You can use these HTML tags

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

  

  

  

*