HTTP-Based Scraper — Architecture & Implementation Plan

Track: 01 — HTTP-Based Scraper
Priority: P0 — Highest impact improvement
Author: Research compiled from the-convocation/twitter-scraper, d60/twikit, elizaOS/agent-twitter-client
Date: 2026-02-26
Status: Architecture Plan (Ready for Implementation)

Table of Contents

1. Executive Summary
2. GraphQL Endpoint Map
3. REST API Endpoints
4. Authentication Flows
5. Request Headers
6. Feature Flags
7. Response Parsing
8. Cursor-Based Pagination
9. Rate Limits
10. Migration Table
11. Architecture Design
12. File Plan
13. Dependency Analysis
14. Risk Assessment
15. Implementation Phases

1. Executive Summary

Twitter/X's web client communicates exclusively via internal GraphQL and REST API endpoints at api.x.com. Every action performed in the browser — viewing profiles, scrolling timelines, searching tweets — is a plain HTTP request with JSON responses. This means all 18 of XActions' Puppeteer-based scraper functions can be replaced with direct HTTP calls, eliminating the need for a headless browser for data retrieval.

Impact

Approach

• Non-breaking: Puppeteer scraper (src/scrapers/twitter/index.js) stays untouched
• Adapter pattern: New http adapter at src/scrapers/adapters/http.js
• Same signatures: All 18 exported functions keep identical call signatures
• Graceful fallback: HTTP failures can optionally fall back to Puppeteer

2. GraphQL Endpoint Map

All GraphQL endpoints follow this URL pattern:

https://api.x.com/graphql/{queryId}/{operationName}

Query parameters are URL-encoded JSON objects: variables, features, and optionally fieldToggles.

2.1 Read Endpoints (GET)

2.2 Write Endpoints (POST)

These endpoints use POST with a JSON body and require full authentication (bearer + auth_token cookie + ct0 CSRF):

2.3 Relationship Write Endpoints (POST — REST v1.1)

2.4 Detailed Endpoint Specifications

UserByScreenName

GET https://api.x.com/graphql/qW5u-DAuXpMEG0zA1F7UGQ/UserByScreenName
  ?variables={"screen_name":"elonmusk"}
  &features={...featureFlags}

Variables:

{
  "screen_name": "elonmusk"
}

Response Path: data.user.result

{
  "data": {
    "user": {
      "result": {
        "__typename": "User",
        "rest_id": "44196397",
        "is_blue_verified": true,
        "legacy": {
          "created_at": "Tue Jun 02 20:12:29 +0000 2009",
          "description": "...",
          "favourites_count": 45231,
          "followers_count": 175000000,
          "friends_count": 823,
          "media_count": 2345,
          "statuses_count": 38000,
          "name": "Elon Musk",
          "screen_name": "elonmusk",
          "profile_image_url_https": "https://pbs.twimg.com/..._normal.jpg",
          "profile_banner_url": "https://pbs.twimg.com/...",
          "location": "...",
          "verified": false,
          "protected": false,
          "can_dm": false,
          "id_str": "44196397"
        }
      }
    }
  }
}

UserTweets

GET https://api.x.com/graphql/E3opETHurmVJflFsUBVuUQ/UserTweets
  ?variables={"userId":"44196397","count":20,"includePromotedContent":false}
  &features={...featureFlags}

Variables:

{
  "userId": "44196397",
  "count": 20,
  "includePromotedContent": false,
  "cursor": "HBaagLy25a..."  // optional, for pagination
}

Response Path: data.user.result.timeline.timeline.instructions[].entries[]

Each entry has:

• entryId: "tweet-{tweetId}" or "cursor-bottom-{value}"
• content.itemContent.tweet_results.result.legacy — the tweet data
• content.cursorType: "Bottom" or "Top" for pagination cursors

Followers / Following

GET https://api.x.com/graphql/P7m4Qr-rJEB8KUluOenU6A/Followers
  ?variables={"userId":"44196397","count":20,"includePromotedContent":false}
  &features={...featureFlags}

Variables:

{
  "userId": "44196397",
  "count": 20,
  "includePromotedContent": false,
  "withGrokTranslatedBio": false,
  "cursor": "..."  // optional
}

Response Path: data.user.result.timeline.timeline.instructions[].entries[]

Each user entry: content.itemContent.user_results.result.legacy

SearchTimeline

GET https://api.x.com/graphql/ML-n2SfAxx5S_9QMqNejbg/SearchTimeline
  ?variables={"rawQuery":"from:elonmusk","count":20,"querySource":"typed_query","product":"Latest"}
  &features={...featureFlags}

Variables:

{
  "rawQuery": "from:elonmusk",
  "count": 20,
  "querySource": "typed_query",
  "product": "Top",       // "Top", "Latest", "Photos", "Videos", "People"
  "cursor": "..."          // optional
}

Response Path: data.search_by_raw_query.search_timeline.timeline.instructions[]

TweetDetail (Thread)

GET https://api.x.com/graphql/nBS-WpgA6ZG0CyNHD517JQ/TweetDetail
  ?variables={"focalTweetId":"1234567890","withRuxInjections":false,"rankingMode":"Relevance"}
  &features={...featureFlags}
  &fieldToggles={"withArticleRichContentState":true,"withArticlePlainText":false}

Response Path: data.threaded_conversation_with_injections_v2.instructions[].entries[]

3. REST API Endpoints

3.1 Guest Token

POST https://api.x.com/1.1/guest/activate.json
Headers:
  authorization: Bearer {bearerToken}
  content-type: application/x-www-form-urlencoded
  
Response: { "guest_token": "1234567890" }

3.2 Trending Topics

GET https://api.x.com/1.1/trends/place.json?id=1
Headers:
  authorization: Bearer {bearerToken}
  x-guest-token: {guestToken}

3.3 User Lookup by ID

GET https://api.x.com/1.1/users/show.json?user_id={id}
Headers:
  authorization: Bearer {bearerToken}
  cookie: auth_token={token}; ct0={csrfToken}
  x-csrf-token: {csrfToken}

4. Authentication Flows

4.1 Bearer Tokens

Two hardcoded bearer tokens used by the Twitter web client:

bearerToken  = AAAAAAAAAAAAAAAAAAAAAFQODgEAAAAAVHTp76lzh3rFzcHbmHVvQxYYpTw%3DckAlMINMjmCwxUcaXbAN4XqJVdgMJaHqNOFgPMK0zN1qLqLQCF
bearerToken2 = AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs%3D1Zv7ttfk8LF81IUq16cHjhLTvJu4FA33AGWWjCpTnA

4.2 Auth Flow Diagrams

Guest Token Flow (Unauthenticated)

sequenceDiagram
    participant C as Client
    participant A as api.x.com

    C->>A: POST /1.1/guest/activate.json
    Note right of C: Header: Authorization: Bearer {bearerToken}
    A-->>C: { "guest_token": "..." }
    
    C->>A: GET /graphql/{qid}/UserByScreenName
    Note right of C: Headers: Authorization: Bearer {bearerToken}<br/>x-guest-token: {guestToken}
    A-->>C: Profile JSON response

Limitations: Guest tokens are rate-limited aggressively. If x-rate-limit-incoming header returns 0, the guest token is burned — delete it and acquire a new one.

Cookie-Based Auth (Primary Method for XActions)

sequenceDiagram
    participant U as User Browser
    participant X as XActions
    participant A as api.x.com

    U->>X: Provide auth_token cookie from browser
    Note over X: Extract ct0 from cookie jar<br/>(set by x.com on first visit)
    
    X->>A: GET /graphql/{qid}/UserTweets
    Note right of X: Headers:<br/>Authorization: Bearer {bearerToken2}<br/>Cookie: auth_token={token}; ct0={csrf}<br/>x-csrf-token: {csrf}
    A-->>X: Timeline JSON response

Cookie Requirements:

• auth_token — Session token (from browser DevTools → Application → Cookies → x.com)
• ct0 — CSRF token (auto-set by Twitter when visiting x.com, sent back as x-csrf-token header)

Username/Password Login Flow

sequenceDiagram
    participant C as Client
    participant A as api.x.com

    C->>A: POST /1.1/guest/activate.json
    A-->>C: guest_token

    C->>A: POST /i/api/1.1/onboarding/task.json
    Note right of C: flow_token: null<br/>subtask_inputs: [{subtask_id: "LoginJsInstrumentationSubtask"}]
    A-->>C: flow_token + next subtask

    C->>A: POST /i/api/1.1/onboarding/task.json
    Note right of C: flow_token: {token}<br/>subtask_inputs: [{subtask_id: "LoginEnterUserIdentifierSSO",<br/>settings_list: {setting: {key: "user_identifier", value: "username"}}}]
    A-->>C: flow_token + next subtask

    C->>A: POST /i/api/1.1/onboarding/task.json
    Note right of C: flow_token: {token}<br/>subtask_inputs: [{subtask_id: "LoginEnterPassword",<br/>enter_password: {password: "..."}}]
    A-->>C: flow_token + next subtask

    alt 2FA Required
        C->>A: POST /i/api/1.1/onboarding/task.json
        Note right of C: subtask_id: "LoginTwoFactorAuthChallenge"<br/>enter_text: {text: "{totp_code}"}
        A-->>C: flow_token + success
    end

    alt Email Verification Required
        C->>A: POST /i/api/1.1/onboarding/task.json
        Note right of C: subtask_id: "LoginAcid"<br/>enter_text: {text: "email@example.com"}
        A-->>C: flow_token + success
    end

    Note over C: Extract auth_token + ct0 from Set-Cookie headers

Important: The login flow requires 1-3 second delays between steps to avoid bot detection (error 399). The the-convocation/twitter-scraper implements this with configurable flowStepDelay.

5. Request Headers

5.1 Required Headers for All API Requests

const headers = {
  // Authentication
  'authorization': 'Bearer {bearerToken2}',
  'cookie': 'auth_token={authToken}; ct0={csrfToken}',
  'x-csrf-token': '{csrfToken}',
  
  // Browser fingerprint (required by Cloudflare)
  'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36',
  'sec-ch-ua': '"Chromium";v="131", "Not_A Brand";v="24"',
  'sec-ch-ua-mobile': '?0',
  'sec-ch-ua-platform': '"Windows"',
  
  // Standard browser headers
  'accept': '*/*',
  'accept-language': 'en-US,en;q=0.9',
  'referer': 'https://x.com/',
  'origin': 'https://x.com',
  
  // Sec-Fetch headers (browser sends these automatically)
  'sec-fetch-site': 'same-site',
  'sec-fetch-mode': 'cors',
  'sec-fetch-dest': 'empty',
  
  // Priority hint
  'priority': 'u=1, i',
  
  // For GraphQL requests
  'content-type': 'application/json',
  
  // For authenticated users
  'x-twitter-auth-type': 'OAuth2Session',
  'x-twitter-active-user': 'yes',
};

5.2 Guest Token Headers

const guestHeaders = {
  'authorization': 'Bearer {bearerToken}',
  'x-guest-token': '{guestToken}',
  // ... same browser fingerprint headers as above
};

5.3 Optional Advanced Headers

6. Feature Flags

Every GraphQL request requires a features JSON object. These are boolean flags that Twitter uses to gate features. Sending the wrong features will cause 400 errors.

6.1 Standard Feature Flags (Current as of Feb 2026)

const FEATURES = {
  rweb_video_screen_enabled: false,
  profile_label_improvements_pcf_label_in_post_enabled: true,
  responsive_web_profile_redirect_enabled: false,
  rweb_tipjar_consumption_enabled: false,
  verified_phone_label_enabled: false,
  creator_subscriptions_tweet_preview_api_enabled: true,
  responsive_web_graphql_timeline_navigation_enabled: true,
  responsive_web_graphql_skip_user_profile_image_extensions_enabled: false,
  premium_content_api_read_enabled: false,
  communities_web_enable_tweet_community_results_fetch: true,
  c9s_tweet_anatomy_moderator_badge_enabled: true,
  responsive_web_grok_analyze_button_fetch_trends_enabled: false,
  responsive_web_grok_analyze_post_followups_enabled: true,
  responsive_web_jetfuel_frame: true,
  responsive_web_grok_share_attachment_enabled: true,
  responsive_web_grok_annotations_enabled: true,
  articles_preview_enabled: true,
  responsive_web_edit_tweet_api_enabled: true,
  graphql_is_translatable_rweb_tweet_is_translatable_enabled: true,
  view_counts_everywhere_api_enabled: true,
  longform_notetweets_consumption_enabled: true,
  responsive_web_twitter_article_tweet_consumption_enabled: true,
  tweet_awards_web_tipping_enabled: false,
  responsive_web_grok_show_grok_translated_post: true,
  responsive_web_grok_analysis_button_from_backend: true,
  post_ctas_fetch_enabled: true,
  freedom_of_speech_not_reach_fetch_enabled: true,
  standardized_nudges_misinfo: true,
  tweet_with_visibility_results_prefer_gql_limited_actions_policy_enabled: true,
  longform_notetweets_rich_text_read_enabled: true,
  longform_notetweets_inline_media_enabled: true,
  responsive_web_grok_image_annotation_enabled: true,
  responsive_web_grok_imagine_annotation_enabled: true,
  responsive_web_grok_community_note_auto_translation_is_enabled: false,
  responsive_web_enhance_cards_enabled: false,
};

6.2 TweetDetail Extra Field Toggles

const FIELD_TOGGLES = {
  withArticleRichContentState: true,
  withArticlePlainText: false,
};

Note: Feature flags change every few weeks when Twitter deploys updates. The implementation should store these in a config file (src/scrapers/twitter/http/features.json) for easy updates without code changes.

7. Response Parsing

7.1 User Object (from legacy field)

Response path: data.user.result.legacy

Additional fields from parent:

• data.user.result.rest_id → userId
• data.user.result.is_blue_verified → isBlueVerified

7.2 Tweet Object (from legacy field)

Response path: instructions[].entries[].content.itemContent.tweet_results.result.legacy

View count from parent:

• result.views.count → views

Author info from parent:

• result.core.user_results.result.legacy.screen_name → username
• result.core.user_results.result.legacy.name → name

7.3 Timeline Entry Types

// Entry ID patterns determine content type
'tweet-{id}'           → Tweet item
'user-{id}'            → User item (in Followers/Following)
'cursor-bottom-{val}'  → Bottom pagination cursor
'cursor-top-{val}'     → Top pagination cursor
'promoted-tweet-{id}'  → Ad (skip)
'who-to-follow-{n}'    → Suggestion (skip)

8. Cursor-Based Pagination

8.1 How Cursors Work

1. First request: Send without cursor variable
2. Extract cursors: From response entries with cursorType: "Bottom" or cursorType: "Top"
3. Next page: Send with cursor variable set to the bottom cursor value
4. Stop condition: When entries array is empty or contains only cursor entries

8.2 Cursor Format

Cursor values are opaque base64-like strings:

HBaagLy25a3g8igAAA==
DAACDAABCgABGA0XC78J_BIKAAIYDRcLiGT0FAAIAAEAAAAA

8.3 Pagination Implementation Pattern

async function* paginateEndpoint(endpoint, variables, auth, maxItems) {
  let cursor = undefined;
  let total = 0;
  
  while (total < maxItems) {
    const vars = { ...variables };
    if (cursor) vars.cursor = cursor;
    
    const response = await fetchGraphQL(endpoint, vars, auth);
    const { items, nextCursor } = parseTimeline(response);
    
    for (const item of items) {
      yield item;
      total++;
      if (total >= maxItems) break;
    }
    
    if (!nextCursor || items.length === 0) break;
    cursor = nextCursor;
  }
}

9. Rate Limits

9.1 Known Rate Limits (per 15-minute window)

9.2 Rate Limit Headers

x-rate-limit-limit: 50
x-rate-limit-remaining: 48
x-rate-limit-reset: 1709000000
x-rate-limit-incoming: 1      // When 0, guest token is burned

9.3 Rate Limit Strategy

class RateLimitManager {
  // Track remaining calls per endpoint
  // When 429 received:
  //   1. Read x-rate-limit-reset header
  //   2. Wait until reset time
  //   3. Retry request
  //   4. For guest tokens: acquire new token if incoming=0
  // When approaching limit:
  //   1. Slow down requests with progressive delays
  //   2. Optionally rotate to Puppeteer fallback
}

10. Migration Table

10.1 Function-by-Function Migration Plan

Result: 16 of 18 scraper functions can be replaced with HTTP calls. exportToJSON/exportToCSV stay as-is (file I/O only). createBrowser/createPage are replaced by HTTP session/client initialization.

10.2 Functions That Must Stay Puppeteer

11. Architecture Design

11.1 Module Dependency Graph

graph TB
    subgraph "Public API (unchanged)"
        A[src/scrapers/twitter/index.js<br>Puppeteer scraper — UNTOUCHED]
        B[src/scrapers/index.js<br>Multi-platform entry point]
    end

    subgraph "Adapter Layer"
        C[src/scrapers/adapters/index.js<br>Registry + factory]
        D[src/scrapers/adapters/http.js<br>NEW — HTTP adapter]
        E[src/scrapers/adapters/puppeteer.js]
        F[src/scrapers/adapters/playwright.js]
    end

    subgraph "HTTP Scraper Layer (NEW)"
        G[src/scrapers/twitter/http/index.js<br>Public API — same signatures]
        H[src/scrapers/twitter/http/client.js<br>HTTP client + session]
        I[src/scrapers/twitter/http/auth.js<br>Auth token management]
        J[src/scrapers/twitter/http/endpoints.js<br>GraphQL query definitions]
        K[src/scrapers/twitter/http/parsers.js<br>Response → XActions format]
        L[src/scrapers/twitter/http/pagination.js<br>Cursor-based pagination]
        M[src/scrapers/twitter/http/rate-limiter.js<br>Rate limit manager]
        N[src/scrapers/twitter/http/features.json<br>Feature flags config]
    end

    B --> C
    C --> D
    C --> E
    C --> F
    D --> G
    G --> H
    G --> K
    G --> L
    H --> I
    H --> J
    H --> M
    H --> N
    J --> N

11.2 Adapter Integration

The existing adapter system at src/scrapers/adapters/ already supports registration of new adapters. The HTTP adapter will be registered as:

// In src/scrapers/adapters/index.js — ADD this line:
registerBuiltin('http', () => import('./http.js'));

Usage:

import { createBrowser, scrapeProfile } from 'xactions/scrapers/twitter';

// Option A: Direct HTTP (new)
const session = await createBrowser({ adapter: 'http' });
const profile = await scrapeProfile(session, 'elonmusk');

// Option B: Puppeteer (existing, unchanged)
const browser = await createBrowser({ adapter: 'puppeteer' });
const page = await createPage(browser);
await loginWithCookie(page, authToken);
const profile = await scrapeProfile(page, 'elonmusk');

// Option C: HTTP with Puppeteer fallback
const session = await createBrowser({ adapter: 'http', fallback: 'puppeteer' });

11.3 Fallback Strategy

flowchart TD
    A[Scraper function called] --> B{Adapter = http?}
    B -->|Yes| C[Make HTTP API request]
    B -->|No| D[Use Puppeteer as-is]
    C --> E{Response OK?}
    E -->|200| F[Parse JSON → return data]
    E -->|429| G[Rate limit → wait + retry]
    E -->|403/401| H{Fallback enabled?}
    H -->|Yes| I[Fall back to Puppeteer]
    H -->|No| J[Throw RateLimitError]
    G --> C
    I --> D

12. File Plan

12.1 New Files to Create

Total: ~2,580 lines of new code + tests

12.2 Files to Modify

13. Dependency Analysis

13.1 HTTP Client

Decision: Use Node.js native fetch (available since Node 18+). XActions already targets Node 18+. For cookie management, use tough-cookie.

13.2 Cookie Management

Decision: Use tough-cookie — it handles domain matching, expiry, secure flags, and SameSite correctly. Critical for Twitter's multi-domain cookie setup (x.com, api.x.com, abs.twimg.com).

13.3 Proxy Support

Decision: Support proxies via an optional proxy config. Use undici's ProxyAgent for HTTP proxies (available as part of Node.js). Add socks-proxy-agent only if SOCKS5 is needed.

13.4 Response Parsing

Decision: No extra libraries needed. Use native JSON.parse() + custom lightweight parsers. Twitter's responses are standard JSON — the complexity is in the deeply nested structure, not the parsing itself. The parsers.js module will contain focused functions:

// parsers.js excerpts
export function parseUserResponse(json) { /* extract from data.user.result.legacy */ }
export function parseTweetResponse(json) { /* extract from timeline entries */ }
export function parseTimelineCursors(json) { /* extract bottom/top cursors */ }

13.5 Final Dependency Summary

Total new dependencies: 1 package, ~60 KB

14. Risk Assessment

14.1 Technical Risks

14.2 Query ID Auto-Update Strategy

Twitter embeds query IDs in their webpack bundles. The auto-update flow:

flowchart LR
    A[Fetch x.com HTML] --> B[Find main.js bundle URL]
    B --> C[Fetch main.js]
    C --> D[Regex extract queryId-operationName pairs]
    D --> E[Compare with stored IDs]
    E --> F{Changed?}
    F -->|Yes| G[Update endpoints.js config]
    F -->|No| H[No action]

This can be run:

• On startup (optional, adds ~2s)
• As a scheduled job
• Manually, by following Query not found

14.3 Cloudflare/TLS Mitigation

Twitter uses Cloudflare and validates TLS fingerprints. Mitigations:

1. Send all browser headers (sec-ch-ua, sec-fetch-*, etc.) — already documented above
2. Use Node.js native fetch which has a neutral TLS fingerprint
3. Set realistic User-Agent matching the sec-ch-ua values
4. Include Referer and Origin headers (Cloudflare rejects requests without these)

15. Implementation Phases

Phase 1: Core HTTP Client (Week 1)

Goal: Establish the HTTP request foundation.

1. Create src/scrapers/twitter/http/client.js — fetch wrapper with headers, cookie jar
2. Create src/scrapers/twitter/http/auth.js — cookie import (auth_token + ct0)
3. Create src/scrapers/twitter/http/endpoints.js — GraphQL endpoint definitions
4. Create src/scrapers/twitter/http/features.json — feature flags
5. Create src/scrapers/twitter/http/rate-limiter.js — basic rate limit tracking
6. Test with manual scrapeProfile call

Deliverable: Can make authenticated API requests and get raw JSON back.

Phase 2: Parsers + Profile/Tweets (Week 2)

Goal: Implement the most-used scraper functions.

1. Create src/scrapers/twitter/http/parsers.js — user + tweet response parsers
2. Create src/scrapers/twitter/http/pagination.js — cursor-based async generators
3. Implement: scrapeProfile, scrapeTweets, searchTweets, scrapeHashtag
4. Write unit tests with fixture data

Deliverable: Profile and tweet scraping works end-to-end via HTTP.

Phase 3: Relationships + Adapter (Week 3)

Goal: Complete the core scrapers and integrate with adapter system.

1. Implement: scrapeFollowers, scrapeFollowing, scrapeLikes, scrapeThread
2. Create src/scrapers/adapters/http.js — BaseAdapter conformance
3. Create src/scrapers/twitter/http/index.js — unified public API
4. Register HTTP adapter in src/scrapers/adapters/index.js
5. Add fallback mechanism to Puppeteer on failure

Deliverable: HTTP adapter is fully functional as a drop-in replacement.

Phase 4: Remaining Endpoints + Polish (Week 4)

Goal: Complete coverage and production-hardening.

1. Implement: scrapeMedia, scrapeListMembers, scrapeBookmarks, scrapeNotifications, scrapeTrending, scrapeCommunityMembers, scrapeSpaces
2. Add Guest token flow for unauthenticated operations
3. Add proxy support
4. Add query ID auto-update mechanism
5. Integration tests (requires live auth token)
6. Documentation + examples

Deliverable: Feature-complete HTTP scraper with 100% function coverage.

Appendix A: Bearer Token Values

# Token 1 — Guest operations (login flow, guest token acquisition)
AAAAAAAAAAAAAAAAAAAAAFQODgEAAAAAVHTp76lzh3rFzcHbmHVvQxYYpTw%3DckAlMINMjmCwxUcaXbAN4XqJVdgMJaHqNOFgPMK0zN1qLqLQCF

# Token 2 — Authenticated API operations (all GraphQL endpoints)
AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs%3D1Zv7ttfk8LF81IUq16cHjhLTvJu4FA33AGWWjCpTnA

Appendix B: Complete URL Examples

Profile Request

https://api.x.com/graphql/qW5u-DAuXpMEG0zA1F7UGQ/UserByScreenName?variables=%7B%22screen_name%22%3A%22elonmusk%22%7D&features=%7B%22rweb_video_screen_enabled%22%3Afalse%2C...%7D

User Tweets Request

https://api.x.com/graphql/E3opETHurmVJflFsUBVuUQ/UserTweets?variables=%7B%22userId%22%3A%2244196397%22%2C%22count%22%3A20%2C%22includePromotedContent%22%3Afalse%7D&features=%7B...%7D

Appendix C: REST API Params for v1.1 Endpoints

When using REST v1.1 endpoints (e.g., follow/unfollow), these query params improve response completeness:

include_profile_interstitial_type=1
include_blocking=1
include_blocked_by=1
include_followed_by=1
include_want_retweets=1
include_mute_edge=1
include_can_dm=1
include_can_media_tag=1
include_ext_has_nft_avatar=1
include_ext_is_blue_verified=1
include_ext_verified_type=1
skip_status=1
cards_platform=Web-12
include_cards=1
include_ext_alt_text=true
include_quote_count=true
include_reply_count=1
tweet_mode=extended
include_entities=true
include_user_entities=true
include_ext_media_color=true
include_ext_media_availability=true
send_error_codes=true
simple_quoted_tweet=true

Success Criteria Checklist

• Every GraphQL query ID documented with parameters and response shape
• Auth token acquisition flow documented step-by-step (guest, cookie, login)
• Architecture plan covers all 18 exported scraper functions
• No function signature changes from current API
• Fallback strategy defined for every failure mode
• Dependency choices justified with size/performance comparison