ProLearning — what shipping a learning platform taught me about scope

ProLearning was a personalised learning platform — an attempt to replace passive course content with adaptive, exercise-first paths. We took it from blank repo to production in five months. This is the unvarnished retro: what we shipped, what I’d do differently, and the one decision that made everything else possible.

What shipped#

• Adaptive path engine. Each learner sees a sequence of exercises tuned to their current skill estimate, not a fixed syllabus.
• Authoring UI. Subject-matter experts wrote and reviewed exercises without engineering involvement after week 8.
• Spaced-repetition layer. Items learners struggled with re-surfaced at increasing intervals across paths.
• Cohort mode. Optional small-group threads tied to a path checkpoint.

By the time we opened to the first 1.2k learners, week-2 retention was 78% — about 3× our baseline reference for the same content delivered as a static course.

Timeline#

What I’d redo#

1. Built the adaptive engine before the authoring tool.

This was backwards. We spent four weeks tuning the scoring model on synthetic data, then discovered the SMEs were writing exercises in a shape the model couldn’t score well. We re-cut the content schema in week 14 and lost a week of authoring throughput. The right order is: ship authoring → get real content → calibrate the model on it. Models are cheap to swap; content is not.

2. Underweighted the support load.

I budgeted oncall as a “couple of pings a week.” Soft launch hit 30 support touches per day at peak. Not because the product was broken — because learning generates support questions in ways software products don’t. (“Why did the path skip module 4?”) We needed an in-app explainer for every adaptive decision, and we shipped that in week 23 instead of week 19.

3. Treated cohort mode as a feature.

Cohort mode was the third-most-requested thing on the wishlist, so I scheduled it for week 16. It turned out to be the load-bearing retention mechanic. Every learner who joined a cohort retained at week 4. Every learner who didn’t join one retained at well below baseline. Cohort mode wasn’t a feature; it was the product. I should have moved it to week 4.

The one decision that mattered most#

Spending three weeks on prototypes instead of one.

Most of the team wanted to lock the design after the first prototype — it worked, the demo was compelling, and we were burning time. I held the line on three because the authoring UX felt off in the first two and I couldn’t articulate why.

Prototype 1 had a nicer learner experience but a hostile authoring flow.
Prototype 2 had elegant authoring but a confusing learner state machine.
Prototype 3 was the worst-looking of the three and the best-feeling to both roles.

If we’d shipped Prototype 1, we’d have spent the next four months wrestling SMEs into a flow they hated, and the model would have been calibrated against content nobody wanted to write. Instead, by week 8, the SMEs were producing 12 exercises a day with no engineering involvement, and the rest of the project was downhill.

Three weeks of throwaway code bought eighteen weeks of compounding output. It’s the trade I’d make every time, and the one I’d have to argue for every time.

What I took from it#

• The authoring UX is the product when humans-in-the-loop are part of the system. Pick it before you pick the model.
• Bimodal metrics describe the product, not features. Look for them early.
• Throwaway prototypes are a tax on speed and a discount on the next eighteen weeks. Be willing to pay it.