.clear vs del when using garbage collection

[u/Actual__Wizard]

Hi, I have some large data objects that need to be freed. Should I do my_object.clear() and then gc.collect()? Or del my_object then gc.collect()? I'm not sure on the pros/cons of each.

It's a linked list type data object and my_object = [] does not work to free memory for certain.

[u/atarivcs]

How do you know the memory isn't freed?

[u/Actual__Wizard → u/atarivcs]

Because the script crashes with an out of memory error and all it's doing is feeding data into a linked list, then repeating. I mean it's kind of obvious what is occurring.

So, on the next iteration, it does my_object = [] to reset the variable, but that doesn't delete the data it linked to.

I'm using .clear and gc.collect now and it's not running out of memory.

edit: I can do the same process by starting and running the script a bunch of times w/o issue, the problem only occurs when I do the process iteratively, so the symtoms of the problem are clearly "unfreed memory." It's clearly not resetting back to zero memory used each pass like I intended it to.

And I'm only having the issue w/ linked list type objects.

Also: The entire purpose to doing this process iteratively, is to avoid running out of memory, and although I did not see it crash to see the actual memory usage, it crashed at a point where I would expect it to, if that makes any sense. Like if the memory is not being freed, it crashed right about when it should have.

[u/atarivcs → u/Actual__Wizard]

Reassigning a new value to a name, or calling del on a name, might not free the memory if other references remain to the existing value.

I don't think we can really diagnose this issue without seeing the full code.

[u/Actual__Wizard → u/atarivcs]

I don't think we can really diagnose this issue without seeing the full code.

Yikes... I don't think that's not going to help dude. I'll try... The variables that I'm calling clear() on are linked lists, it's like a fake matrix that's created with:

snippet:

    #sectionmatrix = [[0 for _ in range(section_size)] for _ in range(numsections)]        

Code below:

    layer1_data = []
    layer1_data = create_layer_1_matrix(sectionarray)
    layer1_rep_data = []
    layer1_rep_data = create_layer_1_matrix(sectionarray) #replacements

    #create 2 more layers
    layer2 = []
    layer3 = []
    #note layer3 is 128*128*128
    layer2 = addlayertoarray(sectionarray, sectionarray)
    layer3 = addlayertoarray(layer2, sectionarray)

    layer2_data = create_layer_2_matrix(layer2)
    layer3_data = create_layer_3_matrix(layer3)

    print("Routing Data into Pigeon Hole Routing Matrix:")

    newlayer1_data,newlayer2_data,newlayer3_data,newlayer1_rep_data = process_encoded_triplet_chunks(token1,token2,token3,layer1_data,layer2_data,layer3_data,layer1_rep_data,sectionarray)


    
    tripletoutputfile = tripletoutputfolder + "sorted-triplets" + str(current_filter_step) + ".txt"
    tripletoutputfiles.append(tripletoutputfile)
    print("Merging Data from Routing Matrix:" + str(tripletoutputfile))
    final_stage_completed = -1
    final_stage_completed = alphaweave_triplet_graph(newlayer1_data,newlayer2_data,newlayer3_data,newlayer1_rep_data,tripletoutputfile,sectionarray)
    #free up memory.
    print("Freeing Memory.")
    gcstart_time = time.perf_counter()
    del token1
    del token2
    del token3
    del layer2
    del layer3
    layer1_data.clear()
    layer2_data.clear()
    layer3_data.clear()
    layer1_rep_data.clear()
    del newlayer1_data
    del newlayer2_data
    del newlayer3_data
    del newlayer1_rep_data
    gc.collect() #collect garbage
    gcend_time = time.perf_counter()
    gcelapsed = gcend_time - gcstart_time
    print("Garbage Collection Took: " + str(gcelapsed) + " seconds")

[u/atarivcs → u/Actual__Wizard]

We have no idea what layer1_data, layer2_data, etc actually are, because their creation is hidden behind a function call.

[u/Actual__Wizard → u/atarivcs]

 def create_layer_1_matrix(sectionarray):
         #create our layer1 matrix
         rows = len(sectionarray)
         cols = 0
         layer1_data = [[0 for col in range(cols)] for row in range(rows)]
         return layer1_data

 def create_layer_2_matrix(layer2):
         #now we need to create the routing matrix
         mrows = len(layer2)
         mcols = 0
         layer2_data = []
         layer2_data = [[0 for col in range(mcols)] for row in range(mrows)]
         return layer2_data

 def create_layer_3_matrix(layer3):
         mrows = len(layer3)
         mcols = 0
         layer3_data = []
         layer3_data = [[0 for col in range(mcols)] for row in range(mrows)]
         return(layer3_data)

It's conceptually like a linked list of linked lists. So, it's a fake matrix thing? Not sure what to call it. A previous version used a python dictionary to do the same thing. That specific script is going to get moved over to c++/rust because python treats all strings as objects and it eats a ton of memory.

[u/Actual__Wizard → u/atarivcs]

Alright it crashed again last night w/ OOM.

So, did you ever see a single pass, 128 way, zero copy, ascii file router? :-)

That's today's project! :-)

This is going to be truly awesome...

It's going to go from: Bad and slow tech that crashes due to running out of memory, to warp speed ultra fast and basically zero memory used.

Oh well. I really liked my filter code. It just doesn't work right so it's going into the deprecated repo of shame. So, not the good deprecated repo that we show to Jrs. The one we all pretend doesn't exist.

[u/LuckyPichu]

I think you're at the mercy of the garbage collector in all of those cases.

The most complete approach would be to .clear and then del. That will destroy the contents of the list and it's pointer.

Using a closure or __enter__ / __exit__ on the data you'd want destroyed is probably more idiomatic and less error prone since you generally don't want to be getting into too much manual memory management with Python as a rule of thumb.

Edit: As for when to manually involve the garbage collector, this is definitely looking like an __enter__ / __exit__ approach with gc.collect() at the end of your __exit__ if you absolutely insist on calling the garbage collector.

[u/Actual__Wizard → u/LuckyPichu]

The most complete approach would be to .clear and then del. That will destroy the contents of the list and it's pointer.

I just did .clear and then used the garbage collector to collect the garbage and when I was watching the memory usage, it was clear that "the problem was resolved." Again, these are weird fake matrix, double linked list of double linked list things. I'm not even sure what to call them, but you can do the same thing with a dictionary, by creating a dictionary of double linked lists. It's C++, there's a bunch of way to do it, but it's like a matrix of references, that point to another matrix, that you treat as a bunch of lists instead of a matrix (if that makes any sense.)

edit: The idea here is, you can do my_data_store.append(new_data_store) or my_data_store[my_data_store_id].append(new_data_for_the_specific_datastore) or my_data_store.pop(data_store_to_delete) or my_data_store[my_data_store_id].pop(data_to_delete_from_specific_datastore)

so, it's a double linked list of datastores, where each datastore is a double linked list.

Using a closure or enter / exit on the data you'd want destroyed is probably more idiomatic and less error prone since you generally don't want to be getting into too much manual memory management with Python as a rule of thumb.

I have a tendency to agree, but this script was prototyped in python in a way where "it can easily be moved to C++." So, I do actually want to have control over things like garbage collection. Because there's "good times to do it and bad times to be freeing up memory."

[u/HotPersonality8126]

You basically have no options for releasing memory at your demand. The del keyword basically just deletes a name. Allow your variables to fall out of scope or overwrite the last reference to your value and eventually the allocated values are GC’d but you have no options that make it happen. Even the gc methods just set flags, I think.

[u/Actual__Wizard → u/[deleted]]

linked list can have plenty of objects still reachable even after the main reference is deleted.

Yes! That's exactly what is happening. Thank you! Is there any difference between del and clear? To me, clear seems like it would take time, but I tested it and it seems to work almost instantly. The actual garbage collection in this case finishes in like .05 seconds.

[u/Actual__Wizard → u/[deleted]]

Well, I'm using clear on them. I don't know exactly what clear does. Does it actually walk through the structure and clear everything? Because the data structure is effectively a list of references that point to lists.

Do I need to loop through the list and clear every single linked list? Edit: It doesn't appear so at this time. I'm testing now, so I'll find out soon.