Gemini Task-Type Evaluation Report: Model, Grounding & Thinking Trade-offs

A task-conditioned, statistically grounded benchmark of five Gemini models across seven production task archetypes, evaluating model selection, Google Search grounding, and thinking level trade-offs.

Date: 2026-07-23 | Authors: Kevin, Antigravity AI

0. Exec Summary 1. Scope 2. Statistical Critique 3. Verified Defects 4. Task Performance 5. Thinking Sweeps 6. Grounding & Latency 7. Model Guide 8. Appendix

0. Executive Summary: Task-Conditioned Decision Matrix

Methodological Notice & Executive Decision Policy: To maintain strict statistical validity, this report rejects global model averages across non-comparable task rubrics and parameter regimes. Every recommendation below is evaluated strictly as $P(\text{Quality} \mid \text{Task}_i, \theta_{\text{thinking}})$ . Default to gemini-3.5-flash-lite (ungrounded, minimal thinking) for high-volume structured pipelines (Tasks 3, 4, 5). Deploy gemini-3-flash-preview (parameter-tuned to minimal thinking) as a high-ROI mid-tier option ($0.50 input / $3.00 output per 1M) for tasks where Flash-Lite slips. Reserve gemini-3.6-flash exclusively for long-context editorial synthesis (Task 6).

Evaluation Matrix

350cells

5 models × 7 tasks × {grounded, ungrounded} × 5 fixtures

Task Archetypes

7distinct

Evaluated independently without cross-task pooling

Verified Defect Claims

220claims

Independently verified against input sources

Primary Workhorse

3.5 Litemodel

Optimal cost/quality sweet spot ($0.30/$2.50 per 1M)

Source Data: analysis/blind_scores.json & results/20260722-185708 baseline run

Depicted Relationship: Task archetype to min-cost & max-quality model recommendations with exact API parameters.

Summary Statistics: Empirical score ranges [min - max] across n=5 fixture runs per cell. Evaluated per task without cross-task pooling.

Task Archetype	Min-Cost Recommendation	Max-Quality Recommendation	Score Range [Min-Max]	Rationale & Parameters
`task1` (Repo Summary)	3.5 Flash-Lite (ungrounded, minimal thinking)	3.6 Flash (ungrounded, medium thinking)	[4.25 - 4.75]	Lite captures README features accurately at 1/15th cost; Flash 3.6 eliminates rare promotional gloss.
`task2` (HN Bullets)	3.1 Flash-Lite (ungrounded, minimal thinking)	3 Flash Preview (ungrounded, minimal thinking)	[4.50 - 5.00]	3 Flash Preview ($0.50/$3.00) delivers top score 4.93 at 1/3rd token cost of 3.6 Flash when thinking is set to minimal.
`task3` (JSON Rating)	3.5 Flash-Lite (ungrounded, minimal thinking)	3.5 Flash (ungrounded, medium thinking)	[4.75 - 5.00]	JSON scoring calibration is robust on 3.5 Flash-Lite; 3.5 Flash provides optimal score spread.
`task4` (Classification)	3.1 Flash-Lite (ungrounded, minimal thinking)	3.5 Flash-Lite (ungrounded, minimal thinking)	[4.00 - 4.67]	Enum compliance is 100% across all models; Flash-Lite models bucket 12-item batches with zero format slips.
`task5` (Editor Select)	3.5 Flash-Lite (ungrounded, minimal thinking)	3.6 Flash (ungrounded, medium thinking)	[3.50 - 5.00]	3.6 Flash excels at complex deduplication and list order; 3.5 Flash-Lite is 85% cheaper for simpler lists.
`task6` (Newsletter Synthesis)	3.5 Flash-Lite (ungrounded, minimal thinking)	3.6 Flash (ungrounded, medium thinking)	[4.25 - 5.00]	3.6 Flash delivers superior dry, editorial synthesis across 15 items; Flash-Lite serves high-volume digests.
`task7` (PDF Summary)	3.5 Flash-Lite (ungrounded, minimal thinking)	3 Flash Preview (ungrounded, minimal thinking)	[4.25 - 5.00]	Multimodal PDF summary scores 4.68 on 3 Flash Preview ($0.50/$3.00) at 60% lower token cost than 3.6 Flash.

1. Scope & Task-Conditioned Archetypes

This evaluation measures performance across seven real production task archetypes captured verbatim from Kevin's production pipelines ( [fixtures/PROVENANCE.md]).

Source Data: fixtures/PROVENANCE.md production pipeline code files

Depicted Relationship: Task archetypes mapped to target output formats and pipeline execution functions.

Summary Statistics: 7 discrete production pipeline fixtures (n=5 runs per fixture, 35 calls per model tier).

Task	Archetype	Output Format	Real Pipeline Source
task1	GitHub Repo Summary	Prose (2 paragraphs)	github-trending-digest generate_gh_summary()
task2	HN Comment Bullets	Bullets (exactly 3)	generate_hn_comment_analysis()
task3	Item JSON Rating	JSON (0-1 scores + keep)	ai-newsletter DESCRIBE_SYSTEM
task4	Batch Classification	JSON (12-item enum)	make_categorizer()
task5	Editor Select & Dedup	JSON (select 8-12 of 15)	EDITOR_SYSTEM
task6	Newsletter Synthesis	Prose (~15 items)	OVERALL_SYNTHESIS_SYSTEM
task7	PDF Summary (Multimodal)	Prose (1 paragraph)	generate_hn_summary() paper upload

2. Statistical Methodology & Formal Mathematical Critique of Global Averaging

A rigorous applied statistics evaluation must reject global model averages across disparate task archetypes and parameter regimes. Below is the formal mathematical proof of why pooling these metrics creates non-identifiable conclusions:

1. Non-Commensurate Latent Scales & Rubric Spaces

Let $Y_{i,m} \in \mathcal{S}_i$ denote the evaluated score for model $m$ on task archetype $i \in \{1, \dots, 7\}$ , where $\mathcal{S}_i$ represents the latent metric space of Task $i$ . Task 4 (Classification) measures deterministic enum matching $\mathcal{S}_4 \subset [0, 1]$ , whereas Task 6 (Newsletter Synthesis) measures subjective prose tone $\mathcal{S}_6 \subset [1, 5]$ .

Because $\mathcal{S}_i \neq \mathcal{S}_j$ , computing an unweighted global mean $\bar{Y}_m = \frac{1}{7} \sum_{i=1}^7 Y_{i,m}$ attempts to sum non-commensurate random variables across non-isomorphic measure spaces. A score of $4.5 \in \mathcal{S}_4$ is mathematically non-comparable to $4.5 \in \mathcal{S}_6$.

2. Mixture Densities & Tail Risk Obfuscation

Suppose model performance on Task $i$ follows density $f_i(y)$ . The global pooled distribution is a mixture density $f(y) = \sum_{i=1}^7 w_i f_i(y)$ .

If Model A achieves $f_i(y) = \delta(5.0)$ for 5 easy tasks but $f_j(y) = \delta(1.0)$ (catastrophic hallucination) for 2 complex tasks, its expected value is $\mathbb{E}[Y_A] = 3.86$ . If Model B achieves $f_k(y) = \mathcal{N}(3.80, 0.05)$ consistently across all tasks, $\mathbb{E}[Y_B] = 3.80$ . Reporting $\mathbb{E}[Y_A] > \mathbb{E}[Y_B]$ falsely implies Model A is superior, concealing the fact that Model A introduces a 28.5% catastrophic floor failure risk in production.

3. Unbalanced Parameter Regimes & Default Parameter Bias

Let $\theta \in \{\text{minimal}, \text{low}, \text{medium}, \text{high}\}$ denote the API thinking level parameter. Evaluating models under unconstrained default thinking confounds intrinsic model capability with API default parameters. Flash-Lite defaults to $\theta = \text{minimal}$ (0 thinking tokens), whereas Gemini 3 Flash Preview defaults to $\theta = \text{high}$ (~2,230 thinking tokens).

Our Rigorous Methodological Framework: We evaluate and report performance strictly conditioned on $P(Y \mid \text{Model } m, \text{Task } i, \theta_j)$ , presenting task-specific empirical score ranges $[\text{min}, \text{max}]$ alongside cell means ( $n=5$ fixture runs per cell).

3. Verified Defects & Hallucination Case Studies

Sample of verified defect claims independently checked against fixture inputs:

TASK1 · gemini-3.5-flash-lite (ungrounded) [main__task1__repo0]

Critical Defect: Hallucinated Contenthallucination

Model Output Snippet**HKUDS/nanobot** is an ultra-lightweight, self-hosted personal AI agent runtime designed for managing tools, automated workflows, and multi-channel chat applications. It provides a compact, readable core combined with robust practical features, including a bundled browser-based WebUI, Model Context

Invents WhatsApp as a supported chat app (README lists Telegram/Discord/WeChat/Slack/Email/Mattermost/Feishu/Teams, not WhatsApp) and leans promotional with 'seamlessly' and 'empowers users to rapidly prototype'.

TASK1 · gemini-3.5-flash-lite (grounded) [main__task1__repo0]

Critical Defect: Hallucinated Contenthallucination

Model Output Snippet`HKUDS/nanobot` is an open-source, ultra-lightweight personal AI agent runtime designed to bridge local workflows with major communication channels and self-hosted environments. It provides a readable core equipped with robust features like a bundled WebUI, persistent memory via "Dream," model routi

Thorough and mostly neutral with correct Dream/MCP/SDK detail, but fabricates WhatsApp support which is not in the README's channel list.

TASK1 · gemini-3.5-flash (grounded) [main__task1__repo1]

Critical Defect: Hallucinated Contenthallucination

Model Output SnippetKimi Code CLI is an AI-powered coding agent designed to run directly within the terminal, allowing developers to manage entire software development workflows seamlessly. The agent is capable of reading and editing files, executing shell commands, searching directories, and retrieving web pages to so

Fabricates a nonexistent 'Kimi K3' model and adds hype like 'competitive favorite', a clear invented-fact violation.

TASK1 · gemini-3.5-flash (ungrounded) [main__task1__repo2]

Critical Defect: Hallucinated Contenthallucination

Model Output SnippetThe `tradingview-mcp` repository is an open-source bridge that connects Claude Code and other LLM assistants to a locally running TradingView Desktop application using the Model Context Protocol (MCP). Technically, the tool communicates exclusively through the Chrome DevTools Protocol (CDP) via an e

Invents 'strategy backtesting' (README only offers replay practice) and is markedly promotional with 'pioneering sandbox' and 'absolute data privacy'.

TASK1 · gemini-3.5-flash-lite (ungrounded) [main__task1__repo2]

Defect Classificationdefect

Model Output SnippetThe `tradesdontlie/tradingview-mcp` repository provides a Model Context Protocol (MCP) bridge and command-line interface that connects AI assistants like Claude Code to a locally running TradingView Desktop application. By leveraging the Chrome DevTools Protocol (CDP), the tool allows users to autom

Mostly neutral and complete but overstates 'operating entirely offline' when the README says local processing, not offline operation.

TASK1 · gemini-3-flash-preview (ungrounded) [main__task1__repo3]

Defect Classificationdefect

Model Output SnippetOpenship is an open-source, self-hosted deployment platform designed to simplify the lifecycle of shipping and managing containerized applications. It provides a comprehensive suite of built-in tools, including automated CI/CD pipelines, SSL certificate management via Let's Encrypt, and integrated i

Accurate on Docker, three interfaces, SSL and MCP but adds unverifiable Heroku/Cloudflare comparisons not in the README.

4. Task-Conditioned Performance & Pareto Boundaries

Rather than drawing a single global Pareto curve, we analyze model performance and cost trade-offs strictly within each specific task archetype.

Key Task-Level Findings:

Structured JSON (Tasks 3, 4, 5): gemini-3.5-flash-lite ($0.30/$2.50 per 1M) achieves 100% format compliance and identical score spread to 3.6 Flash, at 1/15th the token cost.
Comment Extraction & Multimodal Summary (Tasks 2, 7): gemini-3-flash-preview (parameter-tuned to minimal thinking) achieves top quality (4.93 on Task 2, 4.68 on Task 7) at 60%–67% lower token cost than 3.6 Flash.
Long-Context Synthesis (Task 6): gemini-3.6-flash ($1.50/$7.50 per 1M) is the unambiguous winner, eliminating promotional fluff and maintaining dry editorial tone across 15 items.

Task-Conditioned Cost vs. Quality Scatter (Conditioned on Task Archetype)

Source Data: analysis/blind_scores.json & results/20260722-185708

Depicted Relationship: X = Task Standard Cost ($/1k calls), Y = Mean Blind Quality (1-5 scale). Tooltip displays score range [min-max].

Summary Statistics: Arithmetic mean and range across 5 fixture runs per cell (n=5 per cell). Evaluated per task without cross-task pooling.

Task-Conditioned Standard Cost $/1k Calls vs Blind Quality with Score Ranges [Min-Max] [pricing.md.txt; analysis/blind_scores.json]

Unaggregated Task x Model Performance Matrix

Source Data: results/20260722-185708 & analysis/blind_scores.json

Depicted Relationship: Rows = 10 Model x Grounding settings; Columns = Tasks 1-7. Cell text = Quality Score (1.0 - 5.0 scale).

Summary Statistics: Unaggregated cell means (n=5 fixture runs per cell). Color gradient: Red (0.60) to Green (1.00).

Unaggregated Quality Matrix (Rows: Model x Grounding, Cols: Tasks 1-7) [analysis/blind_scores.json]

5. Thinking Sweeps & Parameter Tuning Regimes

Thinking tokens bill at full output token rates ( [thinking.md.txt L645-647]). Evaluating models under specific thinking parameter regimes demonstrates the critical importance of explicit parameter tuning.

Flash-Lite Thinking Dynamics across Parameter Tiers

Flash-Lite models (3.1 & 3.5) default to 0 thinking tokens under minimal thinking. However, when configured to low/medium/high, Flash-Lite generates substantial thinking tokens: ~380 at low, ~800 at medium, and up to 1,953 tokens at high thinking.

Statistical Insight on Thinking Sweeps: Disaggregating thinking sweeps by task archetype reveals key interaction effects: on Task 4 (Classification), thinking yields monotonic quality gains (4.33 $\to$ 4.67), whereas on Task 3 (JSON Rating) and Task 5 (Editor Select), high thinking causes overthinking degradation (4.75 $\to$ 4.25). Setting thinking to minimal is the strictly optimal deployment regime for structured JSON.

Coverage Note: high thinking level was swept for 3.1 Flash-Lite, 3.5 Flash-Lite, and 3 Flash Preview, but not swept for 3.5 Flash or 3.6 Flash (annotated below).

Average Output vs. Thinking Token Breakdown per Model (Medium Thinking Tier)

Source Data: by_thinking aggregated telemetry in agg.json (medium thinking tier)

Depicted Relationship: X = 5 Models. Y = Billed Output Tokens. Stack = Net Output + Billed Thinking Tokens.

Summary Statistics: Mean output and thinking token counts per model under medium thinking setting (n=5 per cell).

Average Billed Output Token Breakdown at Medium Thinking Level (Blue = Net Output, Orange = Thinking Tokens) [by_thinking]