[← Back to Chapter 7: Testing Your Evaluation](./07_TESTING_YOUR_EVALUATION.md) # Chapter 8: Submitting Your Implemented Task Once your task implementation is complete (your `main.py` is implemented and your remote environment is set up), follow the steps below to submit your work. --- ## What You Need to Submit You will submit: - Your **task folder** under `tasks///` (including your implemented `main.py`) - Any updates needed to support your task (if you changed shared utilities) - Enough information for reviewers to reproduce your setup and validate your evaluator --- ## Step 1 — Validate Locally: `test_eval_local.sh` and `test_launch_local.sh` Before opening a pull request, you must confirm both scripts run successfully. > **Important**: Do **not** edit `test_eval.sh` or `test_launch.sh` directly. Copy them to `test_eval_local.sh` and `test_launch_local.sh` respectively, and make your changes in those local copies. **Do not commit or push `test_eval_local.sh` or `test_launch_local.sh` to GitHub** — they contain your personal credentials and local paths. ### A) Evaluation-only test: `agenthle-base/test_eval_local.sh` This runs the evaluation pipeline end-to-end (without requiring the agent to finish the task) and is the fastest sanity check. 1) Create your local copy (if you haven't already): ```bash cp agenthle-base/test_eval.sh agenthle-base/test_eval_local.sh ``` Then open `agenthle-base/test_eval_local.sh` and set: - `LOCAL_TASK_DIR`: local path to your implemented task folder Example: `./tasks/game/mota_24` - `OPENAI_API_KEY`: your API key (If you were assigned a LiteLLM key, set `OPENAI_API_BASE` accordingly) - `CUA_ENV_API_URL`: your remote environment service URL Example: `http://:8000` - `REMOTE_OUTPUT_DIR`: which **remote** output folder your evaluator should read Example values: `output_test_pos`, `output_test_neg`, `output` (depending on your setup) 2) Run the script: ```bash bash agenthle-base/test_eval_local.sh ``` 3) Confirm behavior: - **Positive** test outputs should score **1.0** - **Negative** test outputs should score **0.0** If this fails, fix your evaluation logic before proceeding. ### B) Full pipeline launch: `agenthle-base/test_launch_local.sh` This launches the solver/agent and runs the full pipeline. 1) Create your local copy (if you haven't already): ```bash cp agenthle-base/test_launch.sh agenthle-base/test_launch_local.sh ``` Then open `agenthle-base/test_launch_local.sh` and set: - `LOCAL_TASK_DIR` - `OPENAI_API_KEY` (and optionally `OPENAI_API_BASE`) - `CUA_ENV_API_URL` 2) Run the script: ```bash bash agenthle-base/test_launch_local.sh ``` 3) Confirm: - The run starts successfully (no config errors) - The remote environment is reachable - Outputs can be written to the remote `output/` folder > Note: `test_launch_local.sh` does **not** need to finish. As long as the run is started and progressing normally, you can stop it (e.g., press `Ctrl+C`) after you've confirmed it's running. > Tip: If your task expects outputs in a folder controlled by `REMOTE_OUTPUT_DIR`, ensure you set it consistently in your task code and test scripts. --- ## Step 2 — Fork `cua-verse/agenthle-base` and Open a Pull Request You will submit code via GitHub. ### A) Fork the repo Fork: [cua-verse/agenthle-base](https://github.com/cua-verse/agenthle-base) Then clone **your fork** locally: ```bash git clone cd agenthle-base ``` ### B) Create a branch ```bash git checkout -b / ``` ### C) Add your implemented task folder Copy or implement your task under: - `tasks///` Make sure the task folder includes: - `main.py` with `load()`, `start()`, `evaluate()` - Any additional local assets needed by the task (if required by the task design) > **Important**: Only push `main.py` and any small local assets to GitHub. **Do not** include large files from `input/`, `output/`, `reference/`, or `software/` folders (such as installation packages, game files, or screenshots) — these should remain on your VM for evaluation. The GitHub repository should contain only the code, not large data files. ### D) Run your local checks again Re-run: ```bash bash test_eval_local.sh bash test_launch_local.sh ``` ### E) Push and open a PR ```bash git add -A git commit -m "Add task: " git push -u origin / ``` Open a pull request from your fork/branch into `cua-verse/agenthle-base`. In the PR description, include: - Task category/name: `tasks//` - Remote VM name used for development - Required software installed on the VM (and version if relevant) - include download instructions and installation packages in the software folder - How to run `test_eval_local.sh` successfully (expected scores for pos/neg) - Any special setup assumptions --- ## Step 3 — Notify the Manager on Discord After your PR is opened, notify the manager in Discord with: - **VM name** - **Task folder path** (e.g., `tasks/game/mota_24`) - **PR link** Example message template: ```text Submitted task PR VM: Task: tasks// PR: ``` --- **Next:** [Appendix: Session Commands Reference](./appendix_SESSION_COMMANDS.md) →