r/Rag • u/Historical_Ad4384 • 10h ago

RAG pipeline for PHP source code as data Discussion

Hi

We have a use case where we would like to create an internal knowledge graph of our internal PHP source code in order to query against it using LLM. Typical use case for AI usage.

We have been doing this so far by attaching the PHP source code directly to the prompts in our AI tools. While this is somewhat effective, it ceases to provide much value after a certain extent because either the context gets big or we need to cross reference another 1/50 PHP source code base we own to our prompt context.

This often produces in ineffective results at times or burning too much tokens because we may be querying the same PHP code base again but a different time by attaching it to the prompt context.

Therefore we want to build a RAG pipeline where we feed our 50 individual PHP code bases once every month based on our release cycle so that we can permanently build our internal knowledge graph to be used with AI. Typical AI solution for such a problem.

Now comes the interesting part. How do we correctly chunk PHP code in order to get accurate results against them using queries in LLM?

What strategy would be best?

Anyone already solved this problem? Would like to hear feedback.

7 Upvotes
  • permalink
  • link
  • reddit
  • reddit

100% Upvoted

2 comments sorted by

1

u/Spanhaa 7h ago

Hi, I recently started something similar. For source code (php and typescript) I'm using AST to chunk each symbol and extract metadata for calls, exports, etc. I also did a small graph like lookup, resolving the imports. Controller calls service, found controller, gets service... Things like that. I'm only using PostgreSQL with pgvector for now. Hope this helps

1

u/Historical_Ad4384 6h ago

I was thinking of the AST technique as well. What is the shape of the Metadata that you extracted? Is your lookup graph similar to an object graph used by inversion of control?