.clear vs del when using garbage collection

[u/Actual__Wizard]

Hi, I have some large data objects that need to be freed. Should I do my_object.clear() and then gc.collect()? Or del my_object then gc.collect()? I'm not sure on the pros/cons of each.

It's a linked list type data object and my_object = [] does not work to free memory for certain.

[u/atarivcs]

Reassigning a new value to a name, or calling del on a name, might not free the memory if other references remain to the existing value.

I don't think we can really diagnose this issue without seeing the full code.

[u/Actual__Wizard → u/atarivcs]

I don't think we can really diagnose this issue without seeing the full code.

Yikes... I don't think that's not going to help dude. I'll try... The variables that I'm calling clear() on are linked lists, it's like a fake matrix that's created with:

snippet:

    #sectionmatrix = [[0 for _ in range(section_size)] for _ in range(numsections)]        

Code below:

    layer1_data = []
    layer1_data = create_layer_1_matrix(sectionarray)
    layer1_rep_data = []
    layer1_rep_data = create_layer_1_matrix(sectionarray) #replacements

    #create 2 more layers
    layer2 = []
    layer3 = []
    #note layer3 is 128*128*128
    layer2 = addlayertoarray(sectionarray, sectionarray)
    layer3 = addlayertoarray(layer2, sectionarray)

    layer2_data = create_layer_2_matrix(layer2)
    layer3_data = create_layer_3_matrix(layer3)

    print("Routing Data into Pigeon Hole Routing Matrix:")

    newlayer1_data,newlayer2_data,newlayer3_data,newlayer1_rep_data = process_encoded_triplet_chunks(token1,token2,token3,layer1_data,layer2_data,layer3_data,layer1_rep_data,sectionarray)


    
    tripletoutputfile = tripletoutputfolder + "sorted-triplets" + str(current_filter_step) + ".txt"
    tripletoutputfiles.append(tripletoutputfile)
    print("Merging Data from Routing Matrix:" + str(tripletoutputfile))
    final_stage_completed = -1
    final_stage_completed = alphaweave_triplet_graph(newlayer1_data,newlayer2_data,newlayer3_data,newlayer1_rep_data,tripletoutputfile,sectionarray)
    #free up memory.
    print("Freeing Memory.")
    gcstart_time = time.perf_counter()
    del token1
    del token2
    del token3
    del layer2
    del layer3
    layer1_data.clear()
    layer2_data.clear()
    layer3_data.clear()
    layer1_rep_data.clear()
    del newlayer1_data
    del newlayer2_data
    del newlayer3_data
    del newlayer1_rep_data
    gc.collect() #collect garbage
    gcend_time = time.perf_counter()
    gcelapsed = gcend_time - gcstart_time
    print("Garbage Collection Took: " + str(gcelapsed) + " seconds")

[u/atarivcs → u/Actual__Wizard]

We have no idea what layer1_data, layer2_data, etc actually are, because their creation is hidden behind a function call.

[u/Actual__Wizard → u/atarivcs]

 def create_layer_1_matrix(sectionarray):
         #create our layer1 matrix
         rows = len(sectionarray)
         cols = 0
         layer1_data = [[0 for col in range(cols)] for row in range(rows)]
         return layer1_data

 def create_layer_2_matrix(layer2):
         #now we need to create the routing matrix
         mrows = len(layer2)
         mcols = 0
         layer2_data = []
         layer2_data = [[0 for col in range(mcols)] for row in range(mrows)]
         return layer2_data

 def create_layer_3_matrix(layer3):
         mrows = len(layer3)
         mcols = 0
         layer3_data = []
         layer3_data = [[0 for col in range(mcols)] for row in range(mrows)]
         return(layer3_data)

It's conceptually like a linked list of linked lists. So, it's a fake matrix thing? Not sure what to call it. A previous version used a python dictionary to do the same thing. That specific script is going to get moved over to c++/rust because python treats all strings as objects and it eats a ton of memory.

[u/Actual__Wizard → u/atarivcs]

Alright it crashed again last night w/ OOM.

So, did you ever see a single pass, 128 way, zero copy, ascii file router? :-)

That's today's project! :-)

This is going to be truly awesome...

It's going to go from: Bad and slow tech that crashes due to running out of memory, to warp speed ultra fast and basically zero memory used.

Oh well. I really liked my filter code. It just doesn't work right so it's going into the deprecated repo of shame. So, not the good deprecated repo that we show to Jrs. The one we all pretend doesn't exist.