The LLM JSON Problem: From Recipe Apps to Automated Graphs

Picture this: It's early 2023, and I'm staring at my monitor at 2 AM like a zombie, trying to get the “latest and greatest” GPT-3.5 model to spit out some simple JSON. My coffee cup’s been empty for hours, my eyes are bloodshot, and I'm about one failed prompt away from throwing my keyboard through the window.

Why might I be doing this to myself, you ask? Because earlier that evening, I had this “brilliant” idea of building a recipe recommendation app that dynamically generates its interface using LLMs.

I know, I know—as a product, it probably had zero market potential (my friends still make fun of me for it), but the technical challenge was enough to keep me hooked.

The Decision Tree Disaster

The concept was simple: build a dynamic decision tree that would guide users through a quick series of questions to find their perfect recipe. I had envisioned the app to look something like this:

Each choice would branch into further questions—like a choose-your-own-adventure, but with tasty recipes waiting at the end. Selecting "Vegetarian," for example, would open up a range of cuisine options: Asian, French, Italian… the possibilities were endless.

However, instead of a straightforward development spree that would bring my idea to life, what instead ensued was a two-month descent into prompt-induced madness. I tried everything to get ChatGPT to cooperate: polite prompts, angry prompts, ALL CAPS PROMPTS, even emojis. You name it, I’ve tried it.

And what did I get in return? A whole lot of nothing. The damn thing would simply not output the JSON I needed. It loved free-flowing text and adding its “helpful” commentary (which is often a nightmare for JSON), but brackets, commas, and quotes were a different story. It wasn't even a complex task—just mapping out a simple decision tree structure. But the LLM had other plans.

Two Years Later…

Fast forward to today, I’m building an entire semantic memory layer for LLMs—a product that actually matters, and I’m doing it with a great team behind me.

But guess what? We're still fighting the same battle with LLMs and their JSON output—only now, the stakes are higher.

The problem hasn't gone away; it's very much still there, now complicating matters at a higher level of complexity.

Decoding the Knowledge Graph Mess

Here at cognee, we're not playing around with recipes anymore. We're grappling with far bigger, messier data structures, building a system that can:

1. Dynamically generate robust knowledge graphs
2. Keep data consistent and meaningful (because chaos is so 2023)
3. Manage diverse data types and rapidly evolving models
4. Ensure developers can actually understand and interact with the data they're handling.

If you’ve read some of our previous blog posts, such as the one on The Building Blocks of Knowledge Graphs (where we explain how our graphs extract entities and their relationships from ingested data), you've probably seen this code snippet before:

from cognee.low_level import DataPoint
from typing import List

class Person(DataPoint):
    name: str
    metadata: dict = {"index_fields": ["name"], "type": "Person"}

class Book(DataPoint):
    title: str
    metadata: dict = {"index_fields": ["title"], "type": "Object"}

class PurchaseEvent(DataPoint):
    buyer: Person
    item: Book
    date: str
    location: str
    metadata: dict = {"index_fields": ["date", "location"], "type": "Event"}

We send these neatly structured Pydantic models to LLMs and let them generate graphs for us.

We use the ability of the LLMs to produce JSON outputs, but instead of receiving the JSON, we ask the LLM to produce graphs for us based on the data we provided.

But here's the kicker—it's not just about getting the LLM to spit out data structures. We need to make sure that:

1. The relationships make logical sense (no more "person married to a book" nonsense)
2. The generated data is accurate and trustworthy
3. The data is correctly stored in the appropriate databases
4. The system efficiently chunks, organizes, and merges data—without human babysitting.

What We've Built (And Why It Actually Works)

After months of head-scratching, frustration, and caffeine-induced existential crises, we’ve finally built something that does the job: Dreamify.

Rather than continuing to pray to the fickle LLM gods, we’ve created that secret sauce that ensures cognee runs smoothly in production, with only minimal tinkering required.

🚀 Dreamify: Converting Chaos to Cognee

Dreamify is our internal graph optimization engine that transforms our graph-memory pipeline into something ready for real-world use, and quickly. Think of it as a personal trainer for your system—it makes everything leaner, faster, stronger, and more efficient.

We were inspired by the tools like Dspy and others, but we found that the approaches were a bit too opinionated and closely connected to prompting the LLMs

We thought that there are many more knobs to turn in systems like ours, and we spent a lot of time fixing these.

Here's what makes Dreamify special:

1. Each optimization step is benchmarked, tested rigorously, and fine-tuned for maximum efficiency. Time to do away with those "it works on my machine" excuses.
2. Snaps seamlessly into our existing cognee architecture—no duct tape or late-night hacks needed.
3. Modular, flexible, and ready to evolve with us. Because the only constant is change, right?

JSON Nightmares→Real-World Solutions

From optimizing frustrating, evolving recipe app to building genuinely impactful tools, we've learned one thing: We do not have the language to instruct the LLMs—our English is not their English. By building the right tools and systems around them (like Dreamify), we can finally make them do what we actually need.

Curious to see what we're building? Check out our GitHub and join our Discord community. I can't promise it won’t drive you a little crazy at times, but at least it'll actually work.