Portal-Camera/portal-capability-test

Portal capability test app

Source for the diagnostic APK used on the Portal TV. It checks camera enumeration, displays frames from camera 0, samples the microphone, and exercises the Portal Smart Camera services in com.facebook.portal.aiservice:

• the external control service (com.facebook.portal.SMART_CAMERA_EXTERNAL_CONTROL_SERVICE) with the four mode requests DefaultAuto, Desk, Meeting, Fixed
• the external metadata service (com.facebook.portal.SMART_CAMERA_EXTERNAL_METADATA_SERVICE) for reading the current mode back and watching mode changes

Files

• src/com/portaltv/capability/MainActivity.java — app source
• smartcamera/src/com/portaltv/smartcamera/ — Kotlin client library (see below)
• AndroidManifest.xml — permissions and app declaration
• build-apk.sh — reproducible build (javac -> d8 -> aapt2 -> zipalign -> apksigner)
• portal-capability-test.apk — already-built debug APK

Webcam streaming

Authentication and Security Architecture

Communication between the macOS client (PortalCam.app / CamExtension) and the Portal TV service is secured using an authenticated, channel-bound transport protocol:

1. Transport Layer: HTTPS via SSLServerSocket on port 5654 with forward-secret cipher suites (ECDHE-ECDSA-AES128-GCM-SHA256).
2. Device Identity: Self-signed ECDSA NIST P-256 (secp256r1) certificate generated inside hardware-backed AndroidKeyStore.
3. Pairing & Mutual Authentication: RFC 5054 SRP-6a (2048-bit MODP group) with cryptographic TLS channel binding.
4. Post-Pairing Enforcement: Strict TLS certificate SHA-256 fingerprint pinning in macOS Keychain, combined with 256-bit Bearer token authentication.

Mac (PortalCam.app)                                  Portal TV
      |                                                  |
      |   1. POST /auth/srp/init (HTTPS)                 |
      |------------------------------------------------->|  Generates 6-digit PIN, salt s,
      |                                                  |  derives v = g^x mod N, B = (kv + g^b) mod N
      |   2. { pairingId, salt, B }                      |
      |<-------------------------------------------------|  Displays PIN on TV UI
Extracts server TLS cert DER,                            |
computes tls_hash = SHA256(cert_der)                     |
User enters 6-digit PIN                                  |
Computes x, A = g^a mod N, u, S, K                       |
M1 = SHA256(A || B || K || s || tls_hash)                |
      |                                                  |
      |   3. POST /auth/srp/verify { pairingId, A, M1 }  |
      |------------------------------------------------->|  Computes u, S, K
      |                                                  |  Verifies M1 using Portal's tls_hash
      |                                                  |  (Rate limit: max 3 attempts)
      |   4. { M2, token }                               |  Computes M2 = SHA256(A || M1 || K || tls_hash)
      |<-------------------------------------------------|
Verifies M2 using tls_hash                               |
Pins tls_hash & saves token in Keychain                  |
      |                                                  |
      |==================================================|
      |   Subsequent Media & Control (HTTPS)             |
      |   Strict Pinning: cert_sha256 == pinned_hash     |
      |   Authorization: Bearer <token>                  |
      |==================================================|

Cryptographic & Protocol Choices

1. Server Certificate: ECDSA P-256 (secp256r1)

• Why Elliptic Curve over RSA:
  • Handshake Speed & CPU: EC scalar point multiplication is dramatically lighter on the Portal's Snapdragon 835 ARM cores than 2048-bit modular exponentiation.
  • Certificate Size: The ECDSA certificate is only 352 bytes DER (vs ~1.5 KB for RSA-2048), keeping the TLS handshake packet well within a single TCP MTU.
  • Key Generation Time: Key generation in AndroidKeyStore takes ~100 ms compared to 1.5–3 seconds for RSA.
  • Clean Platform Integration: Android Keymaster implements SHA256withECDSA natively, avoiding raw digest (NONEwithRSA) driver inconsistencies present on Android 9.
• Storage: Stored under alias portalcam_tls_ec_p256 in AndroidKeyStore with private key non-exportable and protected for signing.

2. TLS Protocol Version & Cipher Suite

• Protocol: TLS 1.2 with opportunistic negotiation up to TLS 1.3 (enabledProtocols = arrayOf("TLSv1.3", "TLSv1.2")).
  • Android 9 Platform Context: Conscrypt on Android 9 (API 28) enables TLS 1.3 for client sockets, while server-mode SSLServerSocket natively negotiates TLSv1.2.
• Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 providing authenticated encryption (AEAD) and Ephemeral Diffie-Hellman forward secrecy.

3. Password-Authenticated Key Exchange: RFC 5054 SRP-6a

• Group Parameters: 2048-bit MODP prime group (N, generator g = 2, multiplier k = \text{SHA256}(N \parallel g)).
• PIN Entropy: 6-digit random decimal number (10^6 combinations).
• Key Derivation: x = \text{SHA256}(s \parallel \text{PIN}), with a 16-byte cryptographically secure random salt s.
• Zero-Knowledge Property: The PIN is never transmitted over the wire in cleartext, hashed, or encrypted form. Mutual authentication proves knowledge of the PIN without exposing it to eavesdroppers.

---

Cryptographic Channel Binding & MITM Prevention

Standard self-signed TLS pairing typically suffers from a Trust-on-First-Use (TOFU) vulnerability: a client connecting to an untrusted self-signed certificate has no initial root of trust, making it susceptible to an active Man-in-the-Middle (MITM) proxy that injects its own TLS certificate.

Our implementation eliminates this vulnerability by cryptographically binding the transport layer into the SRP evidence proofs:

\mathbf{tls\_hash} = \text{SHA256}(\text{LeafCertificate}_{\text{DER}}) M_1 = \text{SHA256}(\text{pad256}(A) \parallel \text{pad256}(B) \parallel K \parallel s \parallel \mathbf{tls\_hash}) M_2 = \text{SHA256}(\text{pad256}(A) \parallel M_1 \parallel K \parallel \mathbf{tls\_hash})

Why an Active MITM Cannot Succeed:

1. Suppose an attacker places a rogue proxy between the Mac and the Portal TV.
2. The proxy terminates TLS to the Mac using a rogue certificate C_{\text{rogue}}, and opens a second TLS connection to the Portal TV using C_{\text{portal}}.
3. The Mac computes \text{tls\_hash}_{\text{mac}} = \text{SHA256}(C_{\text{rogue}}).
4. The Mac computes evidence M_1 incorporating \text{tls\_hash}_{\text{mac}} and sends it to the server.
5. The Portal TV verifies M_1 using its own certificate hash: \text{tls\_hash}_{\text{portal}} = \text{SHA256}(C_{\text{portal}}).
6. Because C_{\text{rogue}} \neq C_{\text{portal}}, the hashes differ: \text{tls\_hash}_{\text{mac}} \neq \text{tls\_hash}_{\text{portal}}.
7. Verification fails immediately on the Portal TV. The server aborts the handshake, increments the failed attempt counter, and refuses to issue a bearer token.
8. The attacker cannot forge a valid M_1 or M_2 because computing either proof requires the shared session key K, which requires knowledge of the PIN displayed exclusively on the physical TV screen.

---

Certificate Pinning & Key Storage

Once the client successfully verifies M_2:

1. Keychain Storage:
   • The 256-bit bearer token is stored under portalAuthToken.
   • The server certificate's SHA-256 fingerprint is stored under portalPinnedCertSha256.
   • Stored in the macOS Data Protection Keychain with shared access group ENT9X9U544.com.kovtash.portalcam and synchronized to the App Group UserDefaults (ENT9X9U544.com.kovtash.portalcam) for sandboxed access by the CMIO camera extension.
2. Strict Pinning Enforcement:
   • PortalPinnedSessionDelegate intercepts URLAuthenticationChallenge on all subsequent HTTPS sessions (/video.h264, /audio.aac, /control/*).
   • It extracts the server leaf certificate via SecTrustCopyCertificateChain, computes its SHA-256 digest, and performs a constant-time comparison against PortalAuth.pinnedCertSha256.
   • Any certificate change (e.g. Rogue CA injection, LAN redirect, proxy) causes completionHandler(.cancelAuthenticationChallenge, nil), severing the connection immediately.

---

Brute-Force Rate Limiting

• The Portal TV strictly limits failed pairing attempts to a maximum of 3.
• If 3 failed verify attempts occur, or if the 120-second expiration window elapses:
  • The active pairing session, PIN, salt, and verifier are completely erased from memory.
  • Future attempts are rejected with 400 / 401.
• Security Margin: With 10^6 possible 6-digit PINs and 3 maximum attempts, the probability of an attacker guessing the PIN within the lifetime of a pairing session is: P(\text{guess}) = \frac{3}{10^6} = 0.0003\%

Verified milestone

The foreground streaming service is installed and verified on Portal TV. It survives a device reboot, starts its listener automatically, starts camera or microphone capture on the first client, and tears capture down when the last client disconnects. Returning the Portal app to the foreground restores the service path for new clients.

The app runs an HTTPS server on port 5654 (starts on launch), encoding with the Snapdragon 835 hardware codecs (H.264 via OMX.qcom.video.encoder.avc, AAC-LC via MediaCodec from the mic tee). All streaming and control endpoints require an Authorization: Bearer <token> header obtained via the SRP pairing handshake:

• https://<portal-ip>:5654/ — info page
• https://<portal-ip>:5654/stream.ts — MPEG-TS: H.264 + AAC, synced (recommended)
• https://<portal-ip>:5654/video.h264 — raw H.264 Annex B, 720p30 @ ~2.5 Mbps
• https://<portal-ip>:5654/audio.aac — AAC ADTS, 48 kHz mono @ 64 kbps

Camera control (Bearer auth). Success bodies are always current state {"mode":"<Name>","config":{…}}; failures are {"error":"<code>","message":"<text>"} with a non-2xx status:

Path	Purpose
GET /control/state	Read current mode + config
GET /control/mode?mode=DefaultAuto|Desk|Meeting|Fixed	Switch mode; returns new state
GET /control/fixed?x=&y=&scale=	Apply Fixed crop; returns new state
GET /control/desk?tightness=	Apply Desk framing tightness; returns new state

config is mode-specific: empty for DefaultAuto/Meeting; Fixed includes centerX/centerY/scale; Desk includes optional tuning fields such as framingTightness.

The TS stream is muxed on-device (hand-rolled PAT/PMT/PES/TS with PCR and encoder PTS; MediaMuxer has no TS support on API 28). SPS/PPS are re-sent before every IDR, so players can join mid-stream. This is the endpoint that behaves in players — raw elementary streams make them flaky:

curl -k -H "Authorization: Bearer <token>" https://<portal-ip>:5654/stream.ts | ffplay -

Watch directly with ffplay or mpv (accepting pinned cert or with bearer token):

curl -k -H "Authorization: Bearer <token>" https://<portal-ip>:5654/video.h264 | \
  ffplay -framerate 30 -probesize 500k -analyzeduration 500ms -fflags nobuffer -flags low_delay -f h264 -

-f h264 http://:5654/video.h264 ffplay -nodisp -probesize 50k -analyzeduration 200ms -f aac http://:5654/audio.aac

Both at once, one window (live remux to MPEG-TS; wallclock timestamps keep the
muxer happy since raw H.264 carries no timestamps):

```sh
ffmpeg -use_wallclock_as_timestamps 1 -f h264 -i http://<portal-ip>:5654/video.h264 \
       -use_wallclock_as_timestamps 1 -f aac  -i http://<portal-ip>:5654/audio.aac \
       -c copy -f mpegts - | ffplay -

Use as a virtual webcam/mic on the computer:

# Linux (v4l2loopback):
ffmpeg -f h264 -i http://<portal-ip>:5654/video.h264 -pix_fmt yuv420p -f v4l2 /dev/video2
ffmpeg -f aac  -i http://<portal-ip>:5654/audio.aac -f pulse portal-mic
# macOS: add a Media Source in OBS pointing at /video.h264 (uncheck "local file"),
# then start OBS Virtual Camera.

Multiple clients are supported; a slow video client is dropped rather than allowed to corrupt its H.264 stream. New video clients are sent cached SPS/PPS and a fresh IDR is requested from the encoder.

(USB gadget / UVC webcam mode is not possible on this device: Android 9 has no UVC function in UsbDeviceManager, the kernel exposes no configfs gadget, and configuring it would require root on a locked bootloader.)

Smart Camera Kotlin library (smartcamera/)

Drop-in client wrapping the raw binder protocol below. Requires only kotlinx-coroutines and the two manifest permissions.

val camera = SmartCameraController(context, lifecycleScope)
camera.start()                       // binds both services, subscribes mode + crop

camera.state.collect { s ->          // connection, current mode, live crop window
    Log.i("cam", "${s.connection} mode=${s.mode} crop=${s.cropWindow}")
}

camera.desk.activate()               // per-mode sub-controllers
camera.desk.activate(DeskModeController.Tuning(framingTightness = 0.8f))
camera.fixed.setCrop(CropConfig(centerX = 0.3f, scale = 0.5f))  // ~2x zoom, left third
camera.fixed.pan(dx = 0.05f, dy = 0f)
camera.fixed.zoomBy(1.25f)
camera.auto.activate()
camera.meeting.isActive.collect { active -> ... }

camera.stop()

• SmartCameraController owns one control session (the service only honors the top priority-queue connection), keeps death tokens alive, and rebinds on service death. State comes from subscribeModeChanges / subscribeFrameMetadata("crop").
• ModeController subclasses (auto, desk, meeting, fixed) expose isActive (from the mode subscription) plus mode-specific params; fixed also exposes appliedCrop from the crop subscription.
• Only DefaultAuto/Desk/Meeting/Fixed are whitelisted on the external API.

Build and install:

./build-apk.sh
adb install -r -g portal-capability-test.apk

Reaching the private APIs

Reverse engineering aiservice.apk (see jadx-ai/) shows the external services are gated by nothing more than a normal-level manifest permission:

External services: normal-level manifest permission (SOLVED)

SmartCameraControlService / SmartCameraMetadataService are protected only by android:permission="com.facebook.portal.permission.SMART_CAMERA_CONTROL" / ..._METADATA, both defined by aiservice with protectionLevel="0x0" (normal). Any app installed after aiservice that declares the uses-permission is granted it. This app declares them, and all four mode requests are accepted — verified on device. No exploit needed. No runtime FbPermission check exists on the external path (SmartCameraControlService never calls SmartCameraIPCPermissionManager).

Practical notes for the external control service (all learned on device):

• One connection only. Each connect() creates a Connection in a priority queue; only the top connection is enabled and setMode on any other is silently dropped (Enabled: false; returning... in logcat). Priority: the camera-editor package always wins, otherwise the client with the foreground process wins. Reuse a single session — the app caches its IControlSession.
• The crop/zoom applies to Portal's AIService pipeline (calls, photobooth), not to the app's own raw Camera2 preview.

Smart Camera external API reference

All discovered by decompiling aiservice.apk and verified on device. Binder wire format notes: every call writes its interface token first; ModeSetting parcels as writeInt(1) + writeString(name) + writeBundle(params); MetadataBundle parcels as a bare Bundle; replies are readException() then readInt() flag (1 = payload follows).

Services

Service class	Bind action	Permission (normal level)
SmartCameraControlService	com.facebook.portal.SMART_CAMERA_EXTERNAL_CONTROL_SERVICE	com.facebook.portal.permission.SMART_CAMERA_CONTROL
SmartCameraMetadataService	com.facebook.portal.SMART_CAMERA_EXTERNAL_METADATA_SERVICE	com.facebook.portal.permission.SMART_CAMERA_METADATA

Both live in package com.facebook.portal.aiservice (set it explicitly on the bind intent).

Control: ISmartCameraControlService

#	Method	Args	Returns
1	getVersion()	—	int (1)
2	connect(IBinder deathToken)	any Binder (strong-ref it client-side!)	ISmartCameraControlConnection

Control: ISmartCameraControlConnection

#	Method	Notes
1	close()
2	requestControls(IControlStateCallback)	returns IControlSession; pass any Binder as the callback (only used to signal "control revoked")
3/4	subscribeAvailabilityChanges / unsubscribe…	deprecated; immediately reports available
5	setMode(ModeSetting)	deprecated direct path, same whitelist as sessions
6	isSessionPropertySupported(String)	always false
7	setSessionProperties(...)	throws (unimplemented)

Control: IControlSession (from requestControls)

#	Method	Notes
1	close()	closes the whole connection
3	setMode(ModeSetting)	the call this app uses; whitelist enforced
4	isSessionPropertySupported(String)	always false
5	setSessionProperties(...)	delegates, then throws

Modes (ModeSetting_<name> + Bundle params)

Whitelist (all an external caller may set): DefaultAuto, Desk, Meeting, Fixed.

Mode	Bundle params	Effect
DefaultAuto	—	normal auto tracking
Meeting	—	group framing; forces PERFORMANCE_NO_TRACKER resource profile
Desk	all optional floats: additional_stable_framing_tightness (zoom-like framing tightness), tracking_response_delay_percentage, tracking_sensitivity_percentage, transition_speed_percentage	desk framing; mobileconfig defaults when absent; same resource profile as Meeting
Fixed	required: camera.relative_crop_center_x / _y (float, normalized), camera.relative_crop_scale (float, window size vs full frame: 1.0 = full, 0.5 ≈ 2x zoom), camera.relative_exposure_region (RectF, nullable), camera.relative_face_metering_regions (ArrayList, nullable)	static crop, no tracking; missing keys throw Missing key server-side; keep center within [scale/2, 1-scale/2]

Internal-only modes (whitelist-blocked externally, see "gated" section): Spotlight (person_id int — follow one tracked person; reads back as BasicSpotlight), SpotlightStoryTime (person_id + optional target_rect_landscape/portrait RectF + target_regions = head/shoulders list), FullWide, Storytime, NoOp.

Metadata: ISmartCameraMetadataService

#	Method	Args	Returns
1	getVersion()	—	int
2	connect(IBinder deathToken)	any Binder	ISmartCameraMetadataConnection

Metadata: ISmartCameraMetadataConnection

#	Method	Args	Returns
1	close()
2	getMode()	—	current ModeSetting or null
3	subscribeModeChanges(IModeListener)	listener binder	current mode + push on every change
4	unsubscribeModeChanges(...)
5	getFrameMetadata(List<String> topics)	topic names	MetadataBundle snapshot
6	subscribeFrameMetadata(IStreamingMetadataReceiver, List<String> topics, float rateHz)	receiver + topics + rate	initial snapshot + push at rate
7	unsubscribeFrameMetadata(...)

Callback binder stubs (implement onTransact code 1; skip token via readInt(); readString(), then readInt() flag, then payload):

• IModeListener (com.facebook.portal.smartcamera.external.metadata.IModeListener): payload = ModeSetting
• IStreamingMetadataReceiver (com.facebook.portal.smartcamera.metadata.IStreamingMetadataReceiver): payload = MetadataBundle

External frame-metadata topics: crop (RectF — the live crop window; watch it to see tracking happen), frame_orientation, frame_rotation, effective_device_rotation, full_fov_aspect_ratio. Unknown topics throw IllegalArgumentException: Unknown frame metadata topic.

Person-selection modes are gated to Meta-signed apps

The internal-only modes listed in the mode table above are gated two ways:

1. The external control service hardcodes a whitelist in AidlConnection.setMode (ModeSetting.A02 = DefaultAuto/Desk/Fixed/ Meeting) applied to every caller; bytecode inspection confirms the session path (IControlSession.setMode) delegates to that same checked method, so there is no bypass. Other modes fail with IllegalArgumentException: Mode not supported.
2. The internal control service (SMART_CAMERA_INTERNAL_CONTROL_SERVICE) has no whitelist — its setMode accepts every mode — but every method first calls SmartCameraIPCPermissionManager .enforceAccessSmartCameraControlPermission(), which requires the caller's package to declare the com.facebook.portal.smartcamera.fbpermission.ACCESS_SMART_CAMERA_* meta-data and be signed with Meta's key. A sideloaded app cannot satisfy the signature check, so Spotlight/person-selection is unreachable. (Valid person_ids would also require the internal world-metadata topic person_ids.)

(Two apparent Spotlight successes observed during testing were not reproducible under controlled conditions; BasicSpotlight is also what the Portal's own auto-tracking reports as the current mode when it follows someone, which likely explains the readings.)

Tracked-people data is likewise internal-only. The world-model topics — person_count, person_ids, person_biometric_ids, person_distance_ft, person_alignment_ratio, is_hand_waving, is_arm_raised, is_voice_activity_detected, update_time_ms — are served by getWorldMetadata / subscribeWorldMetadata (transacts 8/9), which exist only on the internal metadata connection (signature-gated). The external binder stub only dispatches transactions 1-7, so there is no transaction-number trick to reach them.

Internal services: out of scope

SmartCameraInternalControlService, SmartCameraInternalMetadataService and AiInternalControlService enforce a runtime FbPermission check requiring Meta's signing key, so they are not reachable from a sideloaded app and are not used here.

Notes

• The camera_denied_package=com.portaltv.capability flag seen in platform_state_service output is set by Portal's privacy layer the first time the app opens the camera without the privacy system expecting it; it does not affect the Smart Camera service path.