Case Study 01: Seven Iterations to Production

Building a Notification System Through Iterative Refinement

Context

Elena, a backend developer at a SaaS company called WorkflowPro, was tasked with building a notification system for the platform. The system needed to notify users about events such as task assignments, deadline reminders, comments on their work, and system announcements. Users could receive notifications through multiple channels: in-app, email, and push notifications to mobile devices.

Elena had three days to build the initial version. She decided to use vibe coding with an AI assistant, applying the iterative refinement techniques she had learned. What follows is a detailed account of each iteration, what changed, and why.

Iteration 1: The Data Model Foundation

Elena's Prompt:

I need to build a notification system for a SaaS project management tool.
Start with the data model only. I need:
- A Notification model that stores the notification content, type, recipient,
  read/unread status, and timestamps
- A NotificationPreference model that lets users configure which notification
  types they want to receive and through which channels
- Use SQLAlchemy with type hints

Keep it simple --- just the models, no business logic yet.
Note: We'll later add support for multiple delivery channels (email, push,
in-app) and batching/digest capabilities, so keep the design extensible.

What the AI produced: Two SQLAlchemy models with appropriate fields, enums for notification types and channels, and relationship definitions. The code was clean and well-structured.

Elena's evaluation: The models were solid but she noticed three issues. First, the notification_type enum had only three types; she needed more. Second, there was no concept of a "notification template" --- each notification would need its content hardcoded. Third, the preference model did not support per-channel configuration (a user might want email for task assignments but only in-app for comments).

Key decision: Elena did not ask the AI to fix all three issues at once. She followed the critique-modify-improve cycle, addressing the structural issue (templates) first because it would affect everything else.

Iteration 2: Templates and Flexible Preferences

Elena's Prompt:

Good start. Two changes to the data model:

1. Add a NotificationTemplate model. Each notification type should have a
   template with a title pattern, body pattern, and action URL pattern.
   Templates should support variable substitution like {user_name} and
   {task_title}. The Notification model should reference a template.

2. Redesign NotificationPreference so preferences are per-type-per-channel.
   For example, a user should be able to enable email for "task_assigned"
   but disable email for "comment_added" while keeping in-app for both.

Keep all existing fields on Notification; just add the template relationship.

What the AI produced: A NotificationTemplate model with pattern fields and a junction-style NotificationPreference model keyed on (user_id, notification_type, channel). The template system used Python string formatting syntax for variable substitution.

Elena's evaluation: The template approach was solid but used simple Python str.format(), which was unsafe with user-controlled data. The preference model was correct structurally. She noted that the AI had also proactively added a priority field to the template, which she had not requested but actually needed.

Key decision: Elena accepted the proactive addition (priority) and made a targeted fix for the template rendering safety issue. She also decided the models were now "good enough" and it was time to move to business logic.

Iteration 3: Core Service Layer

Elena's Prompt:

The models look good. Now build the NotificationService class with these methods:
- create_notification(recipient_id, notification_type, context_data) -> Notification
  This should look up the template, render it with context_data, check user
  preferences, and create the notification record.
- mark_as_read(notification_id, user_id) -> None
- get_unread_notifications(user_id) -> List[Notification]
- get_notification_count(user_id) -> int

For the template rendering, use jinja2 instead of str.format() for safety.
Do not implement the actual delivery (email, push) yet --- just create the
database records and return which channels should be used.

What the AI produced: A NotificationService class with all four methods. The create_notification method looked up templates, rendered them with Jinja2, checked preferences, and returned the notification with a list of channels to deliver through. The service used dependency injection for the database session.

Elena's evaluation: The code was well-structured but had a significant issue: the create_notification method was doing too much work in a single database transaction. If the template lookup succeeded but the preference check failed, the behavior was unclear. She also noticed there was no error handling for missing templates or invalid notification types.

Key decision: Elena applied the critique-modify-improve cycle. She critiqued the transaction handling and error cases, then asked for modifications.

Iteration 4: Robustness and Error Handling

Elena's Prompt:

Three changes to NotificationService:

Critique: create_notification does too much in one method. If the template is
missing, it should fail fast before doing any database work.

Modify:
1. Split create_notification into three internal methods:
   - _resolve_template(notification_type) -> NotificationTemplate (raises
     TemplateNotFoundError if missing)
   - _check_preferences(user_id, notification_type) -> List[Channel]
     (returns empty list if user has opted out of everything)
   - _create_record(recipient_id, template, rendered_content, channels)
     -> Notification

2. Add custom exceptions: TemplateNotFoundError, InvalidRecipientError

3. Add logging: log when a notification is created, when a user has opted
   out of all channels, and when a template is not found.

Improve: Add a bulk_create method that efficiently creates notifications for
multiple recipients (e.g., notifying all team members about a new announcement).

What the AI produced: A refactored service with clean method decomposition, custom exceptions, structured logging, and a bulk_create method that batched database operations.

Elena's evaluation: Significant improvement. The code was clean, well-decomposed, and properly handled errors. The bulk_create method correctly used session.bulk_save_objects() for efficiency. One issue: the logging used print statements instead of the logging module, and the bulk_create did not respect per-user preferences --- it created the same channels for all recipients.

Key decision: These were relatively minor fixes, so Elena combined them into a single follow-up.

Iteration 5: Delivery Channel Implementation

Elena's Prompt:

Two quick fixes:
1. Replace all print statements with proper Python logging module usage.
2. In bulk_create, check each recipient's preferences individually.

Now let's add the delivery system. Create a ChannelDispatcher that:
- Takes a notification and its target channels
- Dispatches to the appropriate channel handler
- Each channel handler is a separate class implementing a DeliveryChannel protocol
- Implement InAppChannel (just marks as delivered in the DB)
- Implement EmailChannel (constructs the email payload but uses an
  abstract send method --- we'll inject the real email service later)
- Implement PushChannel (similar to email --- constructs payload, abstract send)
- If delivery fails on one channel, continue with others and log the failure
- Return a DeliveryReport with success/failure status per channel

What the AI produced: A ChannelDispatcher with a clean protocol-based design, three channel implementations, and a DeliveryReport dataclass. The error handling was solid, with each channel failure isolated from others.

Elena's evaluation: This was excellent work. The protocol-based design made it easy to add new channels. The DeliveryReport was a nice touch that made it easy to track delivery status. She noticed one improvement opportunity: there was no retry mechanism for failed deliveries.

Key decision: Elena decided that retry logic was important for production but would add complexity. She chose to add it as a separate concern rather than complicating the dispatcher.

Iteration 6: Retry Logic and API Endpoints

Elena's Prompt:

Add a RetryManager class that:
- Wraps the ChannelDispatcher
- For failed deliveries, schedules retries with exponential backoff
- Maximum 3 retries per channel
- Uses a simple in-memory queue for now (we'll replace with a proper
  message queue later, so keep the interface abstract)
- Tracks retry attempts in the notification record

Also, create FastAPI endpoints for:
- POST /notifications - Create a notification (admin/system use)
- GET /notifications - List current user's notifications (paginated)
- PATCH /notifications/{id}/read - Mark as read
- GET /notifications/unread-count - Get unread count
- PUT /notifications/preferences - Update notification preferences

Use proper request/response Pydantic models.

What the AI produced: A RetryManager with exponential backoff and an abstract queue interface, plus five FastAPI endpoints with Pydantic models, pagination support, and proper HTTP status codes.

Elena's evaluation: The retry logic was sound. The API endpoints were well-designed with proper validation. She noticed two issues: the endpoints did not have authentication (no way to know the "current user"), and the pagination did not include a total count for the frontend to render page numbers.

Key decision: Elena was at iteration 6 and nearing her deadline. She assessed the code against the "good enough" checklist: correctness (pass), requirements (mostly pass --- missing auth), no known bugs (pass), readable (pass), error handling (pass), security (fail --- no auth). Authentication was critical, so one more iteration was needed.

Iteration 7: Authentication, Pagination Fix, and Final Polish

Elena's Prompt:

Final iteration. Three things:

1. Add JWT authentication to all endpoints. Create a get_current_user
   dependency that extracts the user from the Authorization header.
   The preferences and notification list endpoints should automatically
   scope to the authenticated user.

2. Fix pagination to return total count:
   {"items": [...], "total": 150, "page": 1, "per_page": 20}

3. Final polish:
   - Add OpenAPI tags and descriptions to all endpoints
   - Add rate limiting: 100 requests/minute per user on GET endpoints,
     20 requests/minute on POST/PATCH/PUT
   - Add a health check endpoint at GET /notifications/health
   - Add __repr__ to all models

What the AI produced: Complete implementation with JWT auth, fixed pagination, OpenAPI documentation, rate limiting using a simple in-memory counter (with a note that it should be replaced with Redis in production), and a health check endpoint.

Elena's evaluation: The code passed all six points on the "good enough" checklist. The in-memory rate limiter was not production-grade, but the AI had noted this, and Elena planned to replace it with Redis rate limiting during the next sprint. The system was ready for initial deployment.

Post-Mortem Analysis

What Went Right

1. Incremental building paid off. By starting with data models and building layer by layer, Elena caught the template design issue in iteration 2 before it could propagate through the business logic.

2. Progressive disclosure worked naturally. Elena did not dump all requirements upfront. Each iteration revealed the next set of needs based on what she saw in the current implementation.

3. The CMI cycle kept iterations focused. Each prompt clearly identified what to critique, what to modify, and what to improve, giving the AI unambiguous direction.

4. Strategic foreshadowing saved rework. In iteration 1, Elena mentioned future delivery channels and batching. This caused the AI to design the data model with extensibility, which paid off when adding the dispatcher in iteration 5.

5. The "good enough" checklist prevented premature shipping. After iteration 6, Elena could have shipped, but the checklist identified the missing authentication as a critical gap.

What Could Have Been Better

1. Iteration 4 was dense. Elena combined critique, modify, and improve in one prompt. This worked, but one of the requested changes (per-user preferences in bulk_create) was implemented incorrectly, requiring a fix in iteration 5.

2. The retry logic could have been foreshadowed earlier. If Elena had mentioned retry requirements in iteration 5 when designing the dispatcher, the dispatcher's interface might have been better designed for retry integration.

3. Testing was deferred entirely. None of the seven iterations included test creation. Elena had to create tests separately, which meant finding integration issues late.

Iteration Map

Key Takeaway

Seven iterations might seem like a lot, but each one was focused and productive. The total development time was approximately six hours --- well within Elena's three-day deadline. The incremental approach produced a system that was well-structured, properly decomposed, and production-ready. Attempting to build the same system in a single prompt would likely have produced a monolithic implementation requiring far more debugging time.

The seven-iteration journey also produced valuable artifacts beyond the code: Elena had a clear record of design decisions, the rationale behind architectural choices, and a trail of validated intermediate states she could return to if needed.

Lessons for Your Practice

1. Layer your iterations. Models first, then business logic, then integration, then hardening. Each layer validates the one below it.

2. Use the CMI cycle deliberately. When you find yourself writing a long, unstructured follow-up, pause and organize it into critique, modify, and improve phases.

3. Foreshadow future requirements strategically. You do not need to implement everything now, but giving the AI a heads-up about what is coming leads to better architectural decisions.

4. Apply the "good enough" checklist before shipping. It takes 30 seconds and can prevent a critical oversight from reaching production.

5. Keep iterations focused. Dense prompts with many requests increase the chance that the AI drops or incorrectly implements one of your changes. When in doubt, split into two prompts.

6. Include testing earlier. Elena's biggest regret was deferring tests. Consider making testing part of your iteration cycle, not an afterthought.