Init

2026-09-13 12:15:36 -07:00
commit e473d00f4f
104 changed files with 16080 additions and 0 deletions
@@ -0,0 +1,415 @@
+# Portal capability test app
+
+Source for the diagnostic APK used on the Portal TV. It checks camera enumeration,
+displays frames from camera 0, samples the microphone, and exercises the Portal
+Smart Camera services in `com.facebook.portal.aiservice`:
+
+- the **external** control service (`com.facebook.portal.SMART_CAMERA_EXTERNAL_CONTROL_SERVICE`)
+  with the four mode requests `DefaultAuto`, `Desk`, `Meeting`, `Fixed`
+- the **external** metadata service (`com.facebook.portal.SMART_CAMERA_EXTERNAL_METADATA_SERVICE`)
+  for reading the current mode back and watching mode changes
+
+## Files
+
+- `src/com/portaltv/capability/MainActivity.java` — app source
+- `smartcamera/src/com/portaltv/smartcamera/` — Kotlin client library (see below)
+- `AndroidManifest.xml` — permissions and app declaration
+- `build-apk.sh` — reproducible build (javac -> d8 -> aapt2 -> zipalign -> apksigner)
+- `portal-capability-test.apk` — already-built debug APK
+
+## Webcam streaming
+
+## Authentication and Security Architecture
+
+Communication between the macOS client (`PortalCam.app` / `CamExtension`) and the Portal TV service is secured using an authenticated, channel-bound transport protocol:
+
+1. **Transport Layer**: HTTPS via `SSLServerSocket` on port 5654 with forward-secret cipher suites (`ECDHE-ECDSA-AES128-GCM-SHA256`).
+2. **Device Identity**: Self-signed ECDSA NIST P-256 (`secp256r1`) certificate generated inside hardware-backed `AndroidKeyStore`.
+3. **Pairing & Mutual Authentication**: RFC 5054 SRP-6a (2048-bit MODP group) with **cryptographic TLS channel binding**.
+4. **Post-Pairing Enforcement**: Strict TLS certificate SHA-256 fingerprint pinning in macOS Keychain, combined with 256-bit Bearer token authentication.
+
+```
+Mac (PortalCam.app)                                  Portal TV
+      |                                                  |
+      |   1. POST /auth/srp/init (HTTPS)                 |
+      |------------------------------------------------->|  Generates 6-digit PIN, salt s,
+      |                                                  |  derives v = g^x mod N, B = (kv + g^b) mod N
+      |   2. { pairingId, salt, B }                      |
+      |<-------------------------------------------------|  Displays PIN on TV UI
+Extracts server TLS cert DER,                            |
+computes tls_hash = SHA256(cert_der)                     |
+User enters 6-digit PIN                                  |
+Computes x, A = g^a mod N, u, S, K                       |
+M1 = SHA256(A || B || K || s || tls_hash)                |
+      |                                                  |
+      |   3. POST /auth/srp/verify { pairingId, A, M1 }  |
+      |------------------------------------------------->|  Computes u, S, K
+      |                                                  |  Verifies M1 using Portal's tls_hash
+      |                                                  |  (Rate limit: max 3 attempts)
+      |   4. { M2, token }                               |  Computes M2 = SHA256(A || M1 || K || tls_hash)
+      |<-------------------------------------------------|
+Verifies M2 using tls_hash                               |
+Pins tls_hash & saves token in Keychain                  |
+      |                                                  |
+      |==================================================|
+      |   Subsequent Media & Control (HTTPS)             |
+      |   Strict Pinning: cert_sha256 == pinned_hash     |
+      |   Authorization: Bearer <token>                  |
+      |==================================================|
+```
+
+### Cryptographic & Protocol Choices
+
+#### 1. Server Certificate: ECDSA P-256 (`secp256r1`)
+* **Why Elliptic Curve over RSA**:
+  * **Handshake Speed & CPU**: EC scalar point multiplication is dramatically lighter on the Portal's Snapdragon 835 ARM cores than 2048-bit modular exponentiation.
+  * **Certificate Size**: The ECDSA certificate is only **352 bytes** DER (vs ~1.5 KB for RSA-2048), keeping the TLS handshake packet well within a single TCP MTU.
+  * **Key Generation Time**: Key generation in `AndroidKeyStore` takes ~100 ms compared to 1.5–3 seconds for RSA.
+  * **Clean Platform Integration**: Android Keymaster implements `SHA256withECDSA` natively, avoiding raw digest (`NONEwithRSA`) driver inconsistencies present on Android 9.
+* **Storage**: Stored under alias `portalcam_tls_ec_p256` in `AndroidKeyStore` with private key non-exportable and protected for signing.
+
+#### 2. TLS Protocol Version & Cipher Suite
+* **Protocol**: TLS 1.2 with opportunistic negotiation up to TLS 1.3 (`enabledProtocols = arrayOf("TLSv1.3", "TLSv1.2")`).
+  * *Android 9 Platform Context*: Conscrypt on Android 9 (API 28) enables TLS 1.3 for client sockets, while server-mode `SSLServerSocket` natively negotiates `TLSv1.2`.
+* **Cipher Suite**: `TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256` providing authenticated encryption (AEAD) and Ephemeral Diffie-Hellman forward secrecy.
+
+#### 3. Password-Authenticated Key Exchange: RFC 5054 SRP-6a
+* **Group Parameters**: 2048-bit MODP prime group ($N$, generator $g = 2$, multiplier $k = \text{SHA256}(N \parallel g)$).
+* **PIN Entropy**: 6-digit random decimal number ($10^6$ combinations).
+* **Key Derivation**: $x = \text{SHA256}(s \parallel \text{PIN})$, with a 16-byte cryptographically secure random salt $s$.
+* **Zero-Knowledge Property**: The PIN is never transmitted over the wire in cleartext, hashed, or encrypted form. Mutual authentication proves knowledge of the PIN without exposing it to eavesdroppers.
+
+---
+
+### Cryptographic Channel Binding & MITM Prevention
+
+Standard self-signed TLS pairing typically suffers from a **Trust-on-First-Use (TOFU)** vulnerability: a client connecting to an untrusted self-signed certificate has no initial root of trust, making it susceptible to an active Man-in-the-Middle (MITM) proxy that injects its own TLS certificate.
+
+Our implementation eliminates this vulnerability by cryptographically binding the transport layer into the SRP evidence proofs:
+
+$$\mathbf{tls\_hash} = \text{SHA256}(\text{LeafCertificate}_{\text{DER}})$$
+
+$$M_1 = \text{SHA256}(\text{pad256}(A) \parallel \text{pad256}(B) \parallel K \parallel s \parallel \mathbf{tls\_hash})$$
+
+$$M_2 = \text{SHA256}(\text{pad256}(A) \parallel M_1 \parallel K \parallel \mathbf{tls\_hash})$$
+
+#### Why an Active MITM Cannot Succeed:
+1. Suppose an attacker places a rogue proxy between the Mac and the Portal TV.
+2. The proxy terminates TLS to the Mac using a rogue certificate $C_{\text{rogue}}$, and opens a second TLS connection to the Portal TV using $C_{\text{portal}}$.
+3. The Mac computes $\text{tls\_hash}_{\text{mac}} = \text{SHA256}(C_{\text{rogue}})$.
+4. The Mac computes evidence $M_1$ incorporating $\text{tls\_hash}_{\text{mac}}$ and sends it to the server.
+5. The Portal TV verifies $M_1$ using its own certificate hash: $\text{tls\_hash}_{\text{portal}} = \text{SHA256}(C_{\text{portal}})$.
+6. Because $C_{\text{rogue}} \neq C_{\text{portal}}$, the hashes differ: $\text{tls\_hash}_{\text{mac}} \neq \text{tls\_hash}_{\text{portal}}$.
+7. **Verification fails immediately** on the Portal TV. The server aborts the handshake, increments the failed attempt counter, and refuses to issue a bearer token.
+8. The attacker cannot forge a valid $M_1$ or $M_2$ because computing either proof requires the shared session key $K$, which requires knowledge of the PIN displayed exclusively on the physical TV screen.
+
+---
+
+### Certificate Pinning & Key Storage
+
+Once the client successfully verifies $M_2$:
+
+1. **Keychain Storage**:
+   * The 256-bit bearer token is stored under `portalAuthToken`.
+   * The server certificate's SHA-256 fingerprint is stored under `portalPinnedCertSha256`.
+   * Stored in the macOS Data Protection Keychain with shared access group `ENT9X9U544.com.kovtash.portalcam` and synchronized to the App Group `UserDefaults` (`ENT9X9U544.com.kovtash.portalcam`) for sandboxed access by the CMIO camera extension.
+2. **Strict Pinning Enforcement**:
+   * `PortalPinnedSessionDelegate` intercepts `URLAuthenticationChallenge` on all subsequent HTTPS sessions (`/video.h264`, `/audio.aac`, `/control/*`).
+   * It extracts the server leaf certificate via `SecTrustCopyCertificateChain`, computes its SHA-256 digest, and performs a constant-time comparison against `PortalAuth.pinnedCertSha256`.
+   * Any certificate change (e.g. Rogue CA injection, LAN redirect, proxy) causes `completionHandler(.cancelAuthenticationChallenge, nil)`, severing the connection immediately.
+
+---
+
+### Brute-Force Rate Limiting
+
+* The Portal TV strictly limits failed pairing attempts to **a maximum of 3**.
+* If 3 failed `verify` attempts occur, or if the 120-second expiration window elapses:
+  * The active pairing session, PIN, salt, and verifier are completely erased from memory.
+  * Future attempts are rejected with `400 / 401`.
+* **Security Margin**: With $10^6$ possible 6-digit PINs and 3 maximum attempts, the probability of an attacker guessing the PIN within the lifetime of a pairing session is:
+  $$P(\text{guess}) = \frac{3}{10^6} = 0.0003\%$$
+
+### Verified milestone
+
+The foreground streaming service is installed and verified on Portal TV. It
+survives a device reboot, starts its listener automatically, starts camera or
+microphone capture on the first client, and tears capture down when the last
+client disconnects. Returning the Portal app to the foreground restores the
+service path for new clients.
+
+The app runs an HTTPS server on port 5654 (starts on launch), encoding with the
+Snapdragon 835 hardware codecs (H.264 via `OMX.qcom.video.encoder.avc`, AAC-LC
+via MediaCodec from the mic tee). All streaming and control endpoints require an `Authorization: Bearer <token>` header obtained via the SRP pairing handshake:
+
+- `https://<portal-ip>:5654/` — info page
+- `https://<portal-ip>:5654/stream.ts` — **MPEG-TS: H.264 + AAC, synced (recommended)**
+- `https://<portal-ip>:5654/video.h264` — raw H.264 Annex B, 720p30 @ ~2.5 Mbps
+- `https://<portal-ip>:5654/audio.aac` — AAC ADTS, 48 kHz mono @ 64 kbps
+
+Camera control (Bearer auth). Success bodies are always current state
+`{"mode":"<Name>","config":{…}}`; failures are
+`{"error":"<code>","message":"<text>"}` with a non-2xx status:
+
+| Path | Purpose |
+|---|---|
+| `GET /control/state` | Read current mode + config |
+| `GET /control/mode?mode=DefaultAuto\|Desk\|Meeting\|Fixed` | Switch mode; returns new state |
+| `GET /control/fixed?x=&y=&scale=` | Apply Fixed crop; returns new state |
+| `GET /control/desk?tightness=` | Apply Desk framing tightness; returns new state |
+
+`config` is mode-specific: empty for DefaultAuto/Meeting; Fixed includes
+`centerX`/`centerY`/`scale`; Desk includes optional tuning fields such as
+`framingTightness`.
+
+The TS stream is muxed on-device (hand-rolled PAT/PMT/PES/TS with PCR and
+encoder PTS; MediaMuxer has no TS support on API 28). SPS/PPS are re-sent
+before every IDR, so players can join mid-stream. This is the endpoint that
+behaves in players — raw elementary streams make them flaky:
+
+```sh
+curl -k -H "Authorization: Bearer <token>" https://<portal-ip>:5654/stream.ts | ffplay -
+```
+
+Watch directly with ffplay or mpv (accepting pinned cert or with bearer token):
+
+```sh
+curl -k -H "Authorization: Bearer <token>" https://<portal-ip>:5654/video.h264 | \
+  ffplay -framerate 30 -probesize 500k -analyzeduration 500ms -fflags nobuffer -flags low_delay -f h264 -
+```
+  -f h264 http://<portal-ip>:5654/video.h264
+ffplay -nodisp -probesize 50k -analyzeduration 200ms -f aac http://<portal-ip>:5654/audio.aac
+```
+
+Both at once, one window (live remux to MPEG-TS; wallclock timestamps keep the
+muxer happy since raw H.264 carries no timestamps):
+
+```sh
+ffmpeg -use_wallclock_as_timestamps 1 -f h264 -i http://<portal-ip>:5654/video.h264 \
+       -use_wallclock_as_timestamps 1 -f aac  -i http://<portal-ip>:5654/audio.aac \
+       -c copy -f mpegts - | ffplay -
+```
+
+Use as a virtual webcam/mic on the computer:
+
+```sh
+# Linux (v4l2loopback):
+ffmpeg -f h264 -i http://<portal-ip>:5654/video.h264 -pix_fmt yuv420p -f v4l2 /dev/video2
+ffmpeg -f aac  -i http://<portal-ip>:5654/audio.aac -f pulse portal-mic
+# macOS: add a Media Source in OBS pointing at /video.h264 (uncheck "local file"),
+# then start OBS Virtual Camera.
+```
+
+Multiple clients are supported; a slow video client is dropped rather than
+allowed to corrupt its H.264 stream. New video clients are sent cached SPS/PPS
+and a fresh IDR is requested from the encoder.
+
+(USB gadget / UVC webcam mode is not possible on this device: Android 9 has no
+UVC function in UsbDeviceManager, the kernel exposes no configfs gadget, and
+configuring it would require root on a locked bootloader.)
+
+## Smart Camera Kotlin library (`smartcamera/`)
+
+Drop-in client wrapping the raw binder protocol below. Requires only
+`kotlinx-coroutines` and the two manifest permissions.
+
+```kotlin
+val camera = SmartCameraController(context, lifecycleScope)
+camera.start()                       // binds both services, subscribes mode + crop
+
+camera.state.collect { s ->          // connection, current mode, live crop window
+    Log.i("cam", "${s.connection} mode=${s.mode} crop=${s.cropWindow}")
+}
+
+camera.desk.activate()               // per-mode sub-controllers
+camera.desk.activate(DeskModeController.Tuning(framingTightness = 0.8f))
+camera.fixed.setCrop(CropConfig(centerX = 0.3f, scale = 0.5f))  // ~2x zoom, left third
+camera.fixed.pan(dx = 0.05f, dy = 0f)
+camera.fixed.zoomBy(1.25f)
+camera.auto.activate()
+camera.meeting.isActive.collect { active -> ... }
+
+camera.stop()
+```
+
+- `SmartCameraController` owns one control session (the service only honors the
+  top priority-queue connection), keeps death tokens alive, and rebinds on
+  service death. State comes from `subscribeModeChanges` / `subscribeFrameMetadata("crop")`.
+- `ModeController` subclasses (`auto`, `desk`, `meeting`, `fixed`) expose
+  `isActive` (from the mode subscription) plus mode-specific params; `fixed`
+  also exposes `appliedCrop` from the crop subscription.
+- Only DefaultAuto/Desk/Meeting/Fixed are whitelisted on the external API.
+
+Build and install:
+
+```sh
+./build-apk.sh
+adb install -r -g portal-capability-test.apk
+```
+
+## Reaching the private APIs
+
+Reverse engineering `aiservice.apk` (see `jadx-ai/`) shows the external services
+are gated by nothing more than a normal-level manifest permission:
+
+### External services: normal-level manifest permission (SOLVED)
+
+`SmartCameraControlService` / `SmartCameraMetadataService` are protected only by
+`android:permission="com.facebook.portal.permission.SMART_CAMERA_CONTROL"` /
+`..._METADATA`, both defined by `aiservice` with `protectionLevel="0x0"` (normal).
+Any app installed after `aiservice` that declares the uses-permission is granted
+it. This app declares them, and all four mode requests are accepted — verified on
+device. No exploit needed. No runtime FbPermission check exists on the external
+path (`SmartCameraControlService` never calls `SmartCameraIPCPermissionManager`).
+
+Practical notes for the external control service (all learned on device):
+
+- **One connection only.** Each `connect()` creates a `Connection` in a priority
+  queue; only the top connection is *enabled* and `setMode` on any other is
+  silently dropped (`Enabled: false; returning...` in logcat). Priority: the
+  camera-editor package always wins, otherwise the client with the foreground
+  process wins. Reuse a single session — the app caches its `IControlSession`.
+- The crop/zoom applies to Portal's AIService pipeline (calls, photobooth),
+  not to the app's own raw Camera2 preview.
+
+## Smart Camera external API reference
+
+All discovered by decompiling `aiservice.apk` and verified on device.
+Binder wire format notes: every call writes its interface token first;
+`ModeSetting` parcels as `writeInt(1) + writeString(name) + writeBundle(params)`;
+`MetadataBundle` parcels as a bare `Bundle`; replies are
+`readException()` then `readInt()` flag (1 = payload follows).
+
+### Services
+
+| Service class | Bind action | Permission (normal level) |
+|---|---|---|
+| `SmartCameraControlService` | `com.facebook.portal.SMART_CAMERA_EXTERNAL_CONTROL_SERVICE` | `com.facebook.portal.permission.SMART_CAMERA_CONTROL` |
+| `SmartCameraMetadataService` | `com.facebook.portal.SMART_CAMERA_EXTERNAL_METADATA_SERVICE` | `com.facebook.portal.permission.SMART_CAMERA_METADATA` |
+
+Both live in package `com.facebook.portal.aiservice` (set it explicitly on the
+bind intent).
+
+### Control: `ISmartCameraControlService`
+
+| # | Method | Args | Returns |
+|---|---|---|---|
+| 1 | `getVersion()` | — | int (1) |
+| 2 | `connect(IBinder deathToken)` | any Binder (strong-ref it client-side!) | `ISmartCameraControlConnection` |
+
+### Control: `ISmartCameraControlConnection`
+
+| # | Method | Notes |
+|---|---|---|
+| 1 | `close()` | |
+| 2 | `requestControls(IControlStateCallback)` | returns `IControlSession`; pass any Binder as the callback (only used to signal "control revoked") |
+| 3/4 | `subscribeAvailabilityChanges` / `unsubscribe…` | deprecated; immediately reports available |
+| 5 | `setMode(ModeSetting)` | deprecated direct path, same whitelist as sessions |
+| 6 | `isSessionPropertySupported(String)` | always false |
+| 7 | `setSessionProperties(...)` | throws (unimplemented) |
+
+### Control: `IControlSession` (from `requestControls`)
+
+| # | Method | Notes |
+|---|---|---|
+| 1 | `close()` | closes the whole connection |
+| 3 | `setMode(ModeSetting)` | the call this app uses; whitelist enforced |
+| 4 | `isSessionPropertySupported(String)` | always false |
+| 5 | `setSessionProperties(...)` | delegates, then throws |
+
+### Modes (`ModeSetting_<name>` + Bundle params)
+
+Whitelist (all an external caller may set): **DefaultAuto, Desk, Meeting, Fixed**.
+
+| Mode | Bundle params | Effect |
+|---|---|---|
+| `DefaultAuto` | — | normal auto tracking |
+| `Meeting` | — | group framing; forces `PERFORMANCE_NO_TRACKER` resource profile |
+| `Desk` | all optional floats: `additional_stable_framing_tightness` (zoom-like framing tightness), `tracking_response_delay_percentage`, `tracking_sensitivity_percentage`, `transition_speed_percentage` | desk framing; mobileconfig defaults when absent; same resource profile as Meeting |
+| `Fixed` | **required**: `camera.relative_crop_center_x` / `_y` (float, normalized), `camera.relative_crop_scale` (float, window size vs full frame: 1.0 = full, 0.5 ≈ 2x zoom), `camera.relative_exposure_region` (RectF, nullable), `camera.relative_face_metering_regions` (ArrayList<RectF>, nullable) | static crop, no tracking; missing keys throw `Missing key` server-side; keep center within `[scale/2, 1-scale/2]` |
+
+Internal-only modes (whitelist-blocked externally, see "gated" section):
+`Spotlight` (`person_id` int — follow one tracked person; reads back as
+`BasicSpotlight`), `SpotlightStoryTime` (`person_id` + optional
+`target_rect_landscape/portrait` RectF + `target_regions` = `head`/`shoulders`
+list), `FullWide`, `Storytime`, `NoOp`.
+
+### Metadata: `ISmartCameraMetadataService`
+
+| # | Method | Args | Returns |
+|---|---|---|---|
+| 1 | `getVersion()` | — | int |
+| 2 | `connect(IBinder deathToken)` | any Binder | `ISmartCameraMetadataConnection` |
+
+### Metadata: `ISmartCameraMetadataConnection`
+
+| # | Method | Args | Returns |
+|---|---|---|---|
+| 1 | `close()` | | |
+| 2 | `getMode()` | — | current `ModeSetting` or null |
+| 3 | `subscribeModeChanges(IModeListener)` | listener binder | current mode + push on every change |
+| 4 | `unsubscribeModeChanges(...)` | | |
+| 5 | `getFrameMetadata(List<String> topics)` | topic names | `MetadataBundle` snapshot |
+| 6 | `subscribeFrameMetadata(IStreamingMetadataReceiver, List<String> topics, float rateHz)` | receiver + topics + rate | initial snapshot + push at rate |
+| 7 | `unsubscribeFrameMetadata(...)` | | |
+
+Callback binder stubs (implement `onTransact` code 1; skip token via
+`readInt(); readString()`, then `readInt()` flag, then payload):
+
+- `IModeListener` (`com.facebook.portal.smartcamera.external.metadata.IModeListener`):
+  payload = `ModeSetting`
+- `IStreamingMetadataReceiver` (`com.facebook.portal.smartcamera.metadata.IStreamingMetadataReceiver`):
+  payload = `MetadataBundle`
+
+External frame-metadata topics: `crop` (RectF — the live crop window; watch it
+to *see* tracking happen), `frame_orientation`, `frame_rotation`,
+`effective_device_rotation`, `full_fov_aspect_ratio`.
+Unknown topics throw `IllegalArgumentException: Unknown frame metadata topic`.
+
+### Person-selection modes are gated to Meta-signed apps
+
+The internal-only modes listed in the mode table above are gated two ways:
+
+1. The **external** control service hardcodes a whitelist in
+     `AidlConnection.setMode` (`ModeSetting.A02` = DefaultAuto/Desk/Fixed/
+   Meeting) applied to *every* caller; bytecode inspection confirms the
+   session path (`IControlSession.setMode`) delegates to that same checked
+   method, so there is no bypass. Other modes fail with
+   `IllegalArgumentException: Mode not supported`.
+2. The **internal** control service (`SMART_CAMERA_INTERNAL_CONTROL_SERVICE`)
+   has *no* whitelist — its `setMode` accepts every mode — but every method
+   first calls `SmartCameraIPCPermissionManager
+   .enforceAccessSmartCameraControlPermission()`, which requires the
+   caller's package to declare the
+   `com.facebook.portal.smartcamera.fbpermission.ACCESS_SMART_CAMERA_*`
+   meta-data **and be signed with Meta's key**. A sideloaded app cannot
+   satisfy the signature check, so Spotlight/person-selection is
+   unreachable. (Valid `person_id`s would also require the internal
+   world-metadata topic `person_ids`.)
+
+(Two apparent Spotlight successes observed during testing were not
+reproducible under controlled conditions; `BasicSpotlight` is also what the
+Portal's own auto-tracking reports as the current mode when it follows
+someone, which likely explains the readings.)
+
+**Tracked-people data is likewise internal-only.** The world-model topics —
+`person_count`, `person_ids`, `person_biometric_ids`, `person_distance_ft`,
+`person_alignment_ratio`, `is_hand_waving`, `is_arm_raised`,
+`is_voice_activity_detected`, `update_time_ms` — are served by
+`getWorldMetadata` / `subscribeWorldMetadata` (transacts 8/9), which exist only
+on the internal metadata connection (signature-gated). The external binder stub
+only dispatches transactions 1-7, so there is no transaction-number trick to
+reach them.
+
+### Internal services: out of scope
+
+`SmartCameraInternalControlService`, `SmartCameraInternalMetadataService` and
+`AiInternalControlService` enforce a runtime FbPermission check requiring Meta's
+signing key, so they are not reachable from a sideloaded app and are not used
+here.
+
+## Notes
+
+- The `camera_denied_package=com.portaltv.capability` flag seen in
+  `platform_state_service` output is set by Portal's privacy layer the first
+  time the app opens the camera without the privacy system expecting it; it does
+  not affect the Smart Camera service path.