On-device fixes: optimized native build, 16 KB align, disable thinking
Validated on a Pixel 7 (adb). Key fixes: - Force Release/-O3 for native even in the debug variant (AGP defaulted to -O0, making llama.cpp ~10x too slow / effectively unusable). - 16 KB-align all native LOAD segments (Play requirement; 16 KB-page devices). - Disable Qwen3 thinking (empty <think></think> prefill + /no_think) so replies are fast and punchy instead of spending the whole budget on hidden reasoning. - Cap replies at 220 tokens. Verified: model downloads, loads (n_ctx=4096, 6 threads), streams an on-persona reply with emoji intact, and "new tsjet" clears the conversation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
12
README.md
12
README.md
@@ -34,7 +34,9 @@ conversation and start fresh.
|
||||
**downloaded on first launch** from Hugging Face into the app's private storage.
|
||||
It is *not* bundled in the APK. (Swap the quant in `ModelDownloader.kt`.)
|
||||
- **Persona:** set via `SYSTEM_PROMPT` in `MainActivity.kt`; sampling is a little
|
||||
hot (temp 0.9) for playful answers.
|
||||
hot (temp 0.9) for playful answers. The model is a Qwen3 "thinking" model, so the
|
||||
prompt disables reasoning (empty `<think></think>` prefill + `/no_think`) to keep
|
||||
replies fast and punchy instead of burning the token budget on hidden thoughts.
|
||||
- **Conversation:** the web layer holds the full history and sends it each turn;
|
||||
native rebuilds the ChatML prompt and clears the KV cache before every reply, so
|
||||
"new tsjet" is just: clear JS state + reset cache.
|
||||
@@ -85,5 +87,9 @@ manager, and tap it. First launch downloads the ~1.9 GB model over Wi-Fi.
|
||||
display and from history (`stripThink` in `app.js`).
|
||||
- Bumping the llama.cpp tag? Re-check the C API calls in `llama-jni.cpp` against
|
||||
that tag's `include/llama.h` — it uses the raw C API directly.
|
||||
- **Build verified:** `./gradlew assembleDebug` produces a ~13 MB `arm64-v8a` APK
|
||||
(native libs stripped; the ~1.9 GB model downloads on first launch).
|
||||
- **Native is always built optimized.** AGP compiles the debug variant's C/C++ at
|
||||
`-O0` by default, which makes llama.cpp ~10x too slow; `CMakeLists.txt` forces
|
||||
`Release`/`-O3` regardless of variant. Native libs are also 16 KB page-aligned.
|
||||
- **Verified on-device** (Pixel 7, Android, 4 KB pages): downloads the model, loads
|
||||
it (`n_ctx=4096`, 6 threads), streams a reply, and "new tsjet" clears the chat.
|
||||
`./gradlew assembleDebug` → ~13 MB `arm64-v8a` APK (model downloads on first launch).
|
||||
|
||||
Reference in New Issue
Block a user