This document explains how to set up the scheduler for the Twitter data ingestion process in Val Town.

The Twitter data ingestion process is implemented as a cron job in Val Town. This job fetches tweets from the Twitter API, processes them through our filtering pipeline, and stores them in the database. The scheduler runs this job at regular intervals to ensure timely and consistent data updates.

As per the Val Town documentation, cron schedules cannot be configured programmatically via code. Instead, they must be set up through the Val Town web GUI.

1. Upload the Cron File:

   • Ensure the twitter-fetch.cron.ts file is uploaded to your Val Town account
   • The file should have the .cron.ts extension to indicate it's a cron job

2. Configure Schedule in Val Town Web GUI:

   • Log in to your Val Town account
   • Navigate to the twitter-fetch.cron.ts val
   • Look for the "Schedule" or "Cron" settings in the val's interface
   • Set the desired schedule using cron syntax (e.g., 0 */3 * * * for every 3 hours)

3. Recommended Schedule:

   • For production use, we recommend running the job every 1-3 hours (0 */1 * * * to 0 */3 * * *)
   • For testing or development, you might want to run it more frequently
   • Consider Twitter API rate limits when setting the schedule

The cron job requires the following environment variables to be set in your Val Town environment:

• TWITTER_BEARER_TOKEN: Your Twitter API bearer token
• EMAIL_TO (optional): Email address to send notifications to
• AUTH_USERNAME and AUTH_PASSWORD: Credentials for accessing the test endpoints
• TWITTER_FILTER_* (optional): Various filter configuration options

• The cron job logs its activity to the Val Town console
• If configured, it sends email notifications about successful runs and errors
• You can view the logs in the Val Town interface for the val

• If the job fails due to rate limiting, it will automatically retry with exponential backoff
• Error emails include detailed information about the failure
• Check the Val Town logs for more detailed error messages

For testing or one-off runs, you can manually trigger the cron job using the test endpoint:

https://your-val-town-username.web.val.run/twitter-test

This endpoint requires basic authentication using the AUTH_USERNAME and AUTH_PASSWORD environment variables.