Home
/ Product Management Projects Topic 6 lessons ~7h 15m
Lessons, in order 01
Eval Harness Build the thing that replaces your test suite. A directory of cases, three kinds of grader, several runs per case, and a pass rate reported with a confidence interval, so that a change to a … 1h 30m
02
Cost Per Task Model Compute what one unit of useful work actually costs, including the parts nobody counts: retries, the judge calls, the escalations to a human, and the context you are sending every single … 1h
03
Trust Budget Calculator Turn "is the model good enough" into arithmetic. Given a product shape, the volume, and what one mistake costs the person who absorbs it, compute the accuracy the feature actually requires, … 45 min
04
Cohort Retention Table Build the one chart that tells the truth about a product: cohorts down the side, months across the top, and the share of each cohort still present in each month. Then answer the only … 1h
05
Agent Reliability Simulator Turn per step accuracy into the number you can actually promise a customer. A loop of ten steps at 95% each succeeds 60% of the time, and that single fact decides more agent designs than any … 1h 30m
06
Pricing and Margin Model Find the usage level at which your pricing stops working. Put a seat price, a credit price and a per task price against a cost model that scales with use, and print gross margin per customer … 1h 30m
About this topic Six things to build. Each one is a small Go program, standard library only, that produces a number
you can take into a real argument at work. They are ordered so that each builds on the last, and
every one of them exists because the corresponding lesson is easier to believe once you have run
the arithmetic on your own product.
None of them need a GPU, an API key, a network connection or an account. Expect an hour or two
each.
# Project What you end up with Topic it belongs to 01 Eval Harness A pass rate with a confidence interval over a real case set Evaluation and Quality 02 Cost Per Task Model What one unit of useful work costs, including retries and judging AI Cost and Margin 03 Trust Budget Calculator The accuracy your product shape actually requires What a Model Changes 04 Cohort Retention Table A retention curve, and whether it flattens Funnel, Retention and Experiments 05 Agent Reliability Simulator Per step accuracy turned into a number you can promise Agent and Task Metrics 06 Pricing and Margin Model The usage level at which your pricing stops working AI Pricing and Packaging
Build them with your own numbers, not the ones in the briefs. The defaults are there so the program
runs on the first try; they are not benchmarks.