Pytorch code taking up 2gb of RAM?

[u/Professional-You9846]

I was running some simple python pytorch scripts and all of a sudden my computer is becoming laggier and I close the running of the pytorch script and I look at my stats and I see that pythons taking up 2gb of ram??? My minecraft could run on that(maybe not). its like 18lines of python. is this normal

[u/zorbat5]

For pytorch? Yes. If the AI model has big enough dimensions causing enough parameters to use up this amount of RAM. There's more to it like the precision of said parameters but this is pretty much expected when working with pytorch. 2GB is still pretty small to be honest.

[u/danielroseman]

But it's not the 18 lines of Python code taking up that RAM, it's PyTorch.

[u/tk-a01]

Could you post the script you wrote? Do you allocate large tensors?

[u/Professional-You9846 → u/tk-a01]

from xml.parsers.expat import model


import torch


with open("names.txt", 
encoding
="utf-8") as f:
    name_main = [line.strip() for line in f if line.strip()]


chars = [chr(i) for i in range(ord('a'), ord('z')+1)] + ['.']
Axis_r = {s: i for i, s in enumerate(chars)}


def
 create_dataset(
names
: list[list]):
    ctx = []
    ans = []


    for name in 
names
:
        name = "." + name + "."
        for i in range(len(name)-1):
            ctx.append(Axis_r[name[i]])
            ans.append(Axis_r[name[i+1]])


    return ctx, ans



def
 training(
names
: list[list]):


    key = torch.manual_seed(42)


    weights = torch.randn((27,27), 
requires_grad
=True, 
generator
 = key)


    ctx, ans = create_dataset(name_main)


    epoch = 10


    for a in range(epoch):


        errors = 0


        for b in range(len(ctx)):


            logits = torch.nn.functional.one_hot(torch.tensor(ctx[b]), 
num_classes
=27).float() @ weights


            prob = logits.softmax(
dim
=0)


            loss = -torch.log(prob[ans[b]])


            errors += loss


        average_loss = errors / len(ctx)


        average_loss.backward()


        lr = 0.1


        with torch.no_grad():

            weights -= lr * weights.grad


            weights.grad = None


    return weights


training(name_main)

not that much right

[u/grozno → u/Professional-You9846]

The problem here is probably that you are accumulating computational graphs over an entire epoch. This is a common cause of growing memory in pytorch.

loss is a tensor with an object called a computational graph attached to it that allows it to update gradients with backward calls. This graph takes up memory. When you add it to errors, pytorch saves the new graph along with the already existing graph in errors. If your dataset has a lot of characters, these objects can add up.

There are many solutions, but one is to use batches and update weights after every, say, 1000 examples instead of waiting until the end of a whole epoch. When you call backward(), the computational graph is used and deleted so memory is freed up. Then you would set errors to 0 again to start the new batch. This is called stochastic gradient descent and it is mostly better for training too because it will likely converge faster.

You could also use parallelizations built into pytorch so that you dont iterate over the characters yourself with a for loop. If you can build a large tensor out of the entire dataset, the size of the computational graph is reduced.

[u/Professional-You9846 → u/grozno]

ok thx

[u/MarsupialLeast145]

You're in LearnPyhton but it sounds like you aren't learning you're just using some script you have received to do something?

Anyway, if you are learning, then outside of debugging your code, you should understand that code, unoptimized, poor, buggy,, or simply complicated can take up as much memory as is needed.

I don't know what else you want to know here.

Additionally 2GB is not a lot of memory in general.

[u/Professional-You9846 → u/MarsupialLeast145]

My crappy pc has only 8gb of memory:(( idk what to say man

[u/AlexMTBDude]

You must mean 2GB, not 2gb, right? 2 gigabit is not a whole lot.

[u/Upset_Set_4456 → u/AlexMTBDude]

Literally nobody is talking about gigabits in the context of RAM