Welcome to issue #79 of the iOS Coffee Break Newsletter 📬 and to a new edition of the "Building a Newsletter App" series!

In issue #77, I made long issues fit the on-device model by trimming them. In issue #78, I added a Private Cloud Compute path that preserves the full text. That solved the context-window problem, but it did not prove that keeping every paragraph produces a better summary.

I care about more than whether a summary sounds convincing. I want it to preserve the part of an edition I would fight to keep, especially when that idea appears near the end.

This week, I am using Apple's Evaluations framework to compare both paths on real Coffee Break News issues: a trimmed on-device summary against a full-text summary from Private Cloud Compute.

For this draft, I am using Xcode 27. The Evaluations framework is part of that toolchain, along with the dynamic profiles and Private Cloud Compute path from the previous issue.

A Small Evaluation

I am starting with three archived issues, not a large synthetic dataset. One already fits the on-device context window. The other two trigger trimming, and both place an important conclusion near the end.

That gives me one control and two cases where the missing text could matter. For each issue, I record the summary from the current trimmed path as the baseline. The evaluation then generates a second summary from the complete stripped issue through Private Cloud Compute.

I want to answer three questions:

• Did the summary stay within the existing two-or-three-sentence contract?
• Is it faithful to the original issue?
• Did it preserve the issue's central idea, including a conclusion that appears late?

The first question is mechanical, so I can check it in code. The other two require comparing each result with the source, so they belong in a model-as-judge comparison.

Comparing Both Paths

The dataset uses real Issue fixtures from the app. makePrompt receives the full stripped body, while expected holds the summary I recorded after forcing the existing on-device fallback. In a pairwise evaluation, that expected value becomes the baseline automatically. strippedContent(from:) throws, so the loader is built in a throwing helper instead of a file-level property:

func makeSummaryDataset() throws -> ArrayLoader<ModelSample<String>> {
    let archivedIssues = [
        IssueFixtures.shortIssue,
        IssueFixtures.longIssue,
        IssueFixtures.longIssueWithLateConclusion
    ]
 
    return ArrayLoader(samples: try archivedIssues.map { issue in
        guard let baseline = trimmedSummaries[issue.id] else {
            throw EvaluationFixtureError.missingBaseline(issue.id)
        }
 
        return ModelSample(
            prompt: makePrompt(
                issue: issue,
                content: try strippedContent(from: issue)
            ),
            expected: baseline
        )
    })
}

I keep those trimmed summaries as fixtures instead of regenerating them inside every run. Otherwise a new baseline could move at the same time as the full-text result, making the comparison harder to interpret.

The evaluation stays close to IssueLiveSummarizer. Its subject sends the same prompt and instructions through the full-text Private Cloud Compute strategy. One evaluator counts sentences. The other asks a Private Cloud Compute model to compare that result with the recorded trimmed baseline on two separate scales:

import NaturalLanguage
 
struct IssueSummaryEvaluation: Evaluation {
    let dataset: ArrayLoader<ModelSample<String>>
 
    private let shortSummary = Metric("ShortSummary")
    private let faithfulness = ScoreDimension(
        "Faithfulness",
        description: "Which summary stays closer to facts in the original issue?",
        scale: .numeric([
            4: "Full-text summary is much more faithful to the source.",
            3: "Full-text summary is more faithful to the source.",
            2: "Baseline is more faithful to the source.",
            1: "Baseline is much more faithful to the source."
        ])
    )
    private let coverage = ScoreDimension(
        "Coverage",
        description: "Which summary keeps the issue's central idea, including a conclusion that appears late?",
        scale: .numeric([
            4: "Full-text summary preserves a central idea the baseline drops.",
            3: "Full-text summary preserves a meaningful idea the baseline misses.",
            2: "Baseline preserves a meaningful idea the full-text summary misses.",
            1: "Baseline preserves a central idea the full-text summary drops."
        ])
    )
 
    init() throws {
        dataset = try makeSummaryDataset()
    }
 
    func subject(
        from sample: ModelSample<String>
    ) async throws -> ModelSubject<String> {
        let session = LanguageModelSession(
            profile: IssueSummaryProfile(
                instructions: instructions,
                usePrivateCloud: true
            )
        )
        let response = try await session.respond(
            to: sample.prompt,
            options: GenerationOptions(samplingMode: .greedy)
        )
        return ModelSubject(value: response.content)
    }
 
    var evaluators: Evaluators {
        Evaluator { _, subject in
            let tokenizer = NLTokenizer(unit: .sentence)
            tokenizer.string = subject.value
            let count = tokenizer.tokens(
                for: subject.value.startIndex..<subject.value.endIndex
            ).count
            return (2...3).contains(count)
                ? shortSummary.passing(rationale: "\(count) sentences")
                : shortSummary.failing(rationale: "\(count) sentences")
        }
 
        ModelJudgeEvaluator.pairwise(
            judge: PrivateCloudComputeLanguageModel(),
            dimensions: [faithfulness, coverage]
        )
    }
 
    func aggregateMetrics(using aggregator: inout MetricsAggregator) {
        aggregator.computeMean(of: shortSummary)
        aggregator.group("Judge") { group in
            group.computeMean(of: faithfulness.metric)
            group.computeMean(of: coverage.metric)
        }
    }
}

The subject reuses the same IssueSummaryProfile, instructions and greedy generation as IssueLiveSummarizer; the only fixed choice is the full-text Private Cloud Compute strategy. The app still exposes that work through IssueSummarizing, so the evaluation does not need a second production service just to select a branch. This run does not repeat the quota fallback from issue #78. If Private Cloud Compute is unavailable, the evaluation fails, which tells me the comparison never produced a summary.

The sample prompt contains the original issue, so the pairwise judge sees that source together with the candidate and baseline. Pairwise evaluation builds its own prompt, so the scale labels are where these questions live. Faithfulness asks which summary stays closer to the source. Coverage asks which one keeps the central idea. A score of 4 is the opposite of 1, and a score of 3 is the opposite of 2, so a win means the same size of gap either way. The judge is Private Cloud Compute because a long issue plus both summaries can exceed the on-device window.

What the Comparison Can Tell Me

The short issue is a useful control. The recorded on-device summary and the Private Cloud Compute summary were both written from that complete issue, and the judge belongs to the same Private Cloud Compute family that writes the full-text summary. If that control prefers the full-text result, I treat it as a preference for the model and stop drawing conclusions about context.

The two long issues are the actual test, and only on coverage. A coverage score of 3 or 4 means the full-text summary kept an idea the baseline missed. A coverage score of 1 or 2 means the extra context did not preserve that idea. Faithfulness stays separate: a fluent summary can still invent a detail, and a faithful one can still drop the ending.

🤝 Wrapping Up

The loop I started in issue #77 now has its measurement: make long content fit, preserve it when possible, then check whether that extra complexity earns its place. I care about the result because the ending is often the part of an issue I would fight to keep.

For Coffee Break News, I will keep the Private Cloud Compute path only if coverage prefers the full-text summary on both issues where the ending was trimmed, the short issue does not show that same preference, and both presentation orders agree. Otherwise, the simpler trimmed path wins.

Have any feedback, suggestions, or ideas to share? Feel free to reach out to me on Twitter.

Have a great week ahead 🤎

References

• Meet the Evaluations framework
• Scoring with model-as-judge evaluators
• Designing effective evaluations
• Designing effective model-as-judge evaluators