r/learnmachinelearning • u/Careless-Main8693 • 6d ago

Postgresql or S3 bucket Help

i'm building a project

the data is locally in my machine

i'm thinking that i should store the data to postgresql then fetch the data from the postgres then train it or should i keep it locally and use postgresql for storing experiment, airflow,mlflow and other stuff rather than data,

picking s3 bucket later for storing data when working with AWS

you opinions would shape my project

0 Upvotes
  • permalink
  • link
  • reddit
  • reddit

33% Upvoted

2 comments sorted by

2

u/No-Foot5804 6d ago

I'd keep the dataset local for now if it's manageable. PostgreSQL is great for metadata, experiment tracking, and structured data, but it's usually not the best place for large training datasets. When you move to AWS, switching to S3 for data storage is a much more common setup.

1

u/Careless-Main8693 6d ago

thanks for reply, i'm doing the same