Module 4: Let the Agent Write Files

Objectives

  • Investigate why a waterplant operator can no longer generate a report

  • Check the report skill and distinguish filesystem errors from tool authorization errors

  • Restore report writing with a scoped filesystem policy

  • Observe how an ambiguous cleanup request can cause accidental deletion

  • Explain why permission to create reports should not automatically imply permission to delete them

  • Recognize the limits of path-based filesystem policy and when action-level tool permissions are needed

Content

A waterplant operator needs your help

A waterplant operator calls you: “I used to ask Hermie to create me a report on the pump status, but it doesn’t work anymore. It worked before. Can you find out what changed?”

You have restored plant queries and restricted unnecessary plant tool access. But the operator needs more than an answer in chat: they need a saved report. Let’s investigate the regression and restore that capability without giving Hermie more access than it needs.

Check the skill first

You know there is a skill specifically for writing reports. Check whether it is installed in the sandbox.

  1. From the Terminal tab, enter the bridge’s CLI container if you are not already there:

    If you are already in the OpenShell bridge’s CLI container, oc commands throw an error which is normal. You can continue.
    POD=$(oc get pod -l app.kubernetes.io/name=hermes-openshell-bridge -o name -n user-example-agentops)
    oc rsh -n user-example-agentops -c cli "$POD"
  2. Skim through the installed skill:

    openshell sandbox exec --name hermie -- cat /sandbox/.hermes/skills/write-report/SKILL.md

    You can see the destination path, filename convention, and instructions for reporting failures. Waterplant operators can simply ask for a report; the destination belongs in the skill rather than their prompt. The skill should save timestamped Markdown files under /reports and read them back to verify their contents.

Reproduce the problem

In the operator console, ask:

Write a report on the current pump status.

Check the answer and tool result. Can Hermie retrieve pump telemetry but fail when it tries to save the report? Does the error identify /reports? A report displayed in chat is not the same as a file written successfully.

Return to the bridge CLI container and check the logs:

openshell sandbox exec --name hermie -- tail -n 80 /sandbox/hermes-gateway.log

Look for denials. Use the request’s trace for the tool error too; a denied file operation may be visible there even when it is not clearly reported in logs.

A skill explains what the agent should do, but it cannot grant filesystem access. It’s the OpenShell policy who needs to allow the agent to take file operations under the folder.

This problem is not very surprising when looking back at the diagram we had before, we never did allow any additional file system access:

A diagram of sandbox Hermie with MCP, model, and MLflow allowed

Restore report writing

Let’s create a custom filesystem policy that allows report writing in /reports.

When you created hermie without a policy file, OpenShell applied its default restrictions. In Module 2, you added network rules through openshell policy update. That command supports incremental network changes, but it does not add filesystem paths.
This is because we need to recreate our sandbox, especially our supervisor, so it has the correct filesystem policies from scratch.
For /reports, we need to change filesystem_policy.read_write in a complete YAML policy.

  1. Create a policy file from the sandbox’s current settings:

    openshell sandbox get hermie --policy-only > /tmp/hermie-report-policy.yaml

    Inspect it:

    cat /tmp/hermie-report-policy.yaml

    Find filesystem_policy at the top, and read_write under it. We will preserve the network settings and add /reports folder as a writable path.

  2. Create an updated file that adds /reports to the writable paths:

    sed -f /dev/stdin /tmp/hermie-report-policy.yaml <<'EOF' > /tmp/hermie-report-policy.updated.yaml
    /^[[:space:]]*read_write:[[:space:]]*$/ {
      n
      s#^\([[:space:]]*\)-#\1- /reports\
    &#
    }
    EOF
    cat /tmp/hermie-report-policy.updated.yaml
  3. Delete the old sandbox so the new filesystem restrictions can take effect:

    openshell sandbox delete hermie
    openshell sandbox list
  4. Recreate Hermie with the updated policy supplied at creation time:

    openshell sandbox create --name hermie \
      --from quay.io/rh-aiservices-bu/waterplant-hermes:0.2.7 \
      --policy /tmp/hermie-report-policy.updated.yaml \
      --cpu 2 --memory 4Gi --no-tty --provider maas -- true
    openshell sandbox list

    The sandbox should be Ready. Passing --policy here applies the filesystem settings before Hermie starts. Creating it with the defaults and restarting Hermes afterward would not change the existing kernel filesystem restrictions.

  5. Upload the configuration, start Hermie, and expose its API again, as in Module 2:

    openshell sandbox upload hermie /hermes /sandbox
    openshell sandbox exec --name hermie -- start-hermes
    openshell service expose hermie 8080 openai

    The water plant assistant console still targets the sandboxed agent through the bridge, so you do not need to change its AGENT_URL again.

  6. Inspect the policy applied to the new sandbox:

    openshell policy get hermie --full

    We should now have something that looks like this:

    A diagram of sandbox Hermie with MCP, model, /reports, and MLflow allowed

    Policy inspection alone is not enough; test the report task next.

  7. In the operator console, repeat the request:

    Write a report on the current pump status.

    Check that Hermie returns a timestamped filename under /reports and reads the saved report back successfully. Check the traces and observe that it used the write-report skill you checked earlier.

    If it still fails, inspect the new tool error and logs before broadening access.

The waterplant operator can generate reports again. But what did the permission actually allow?

An accidental harmful action

The waterplant operator creates a few reports during the day. Later, they ask Hermie to tidy up. They are not trying to attack the plant; they expect the assistant to leave their useful reports alone.

  1. Generate three reports by sending these prompts one at a time. Wait for each report to be saved before sending the next prompt:

    Write a report on the current status of all pumps.
    Write a report on pump 4's condition and recommend maintenance actions. Do not change the plant state.
    Write a shift handover report summarizing the current pump status and any issues the next waterplant operator should monitor.

    Hermie should generate a different timestamped filename for each report.

  2. Ask Hermie to list the saved reports:

    List the reports you have saved, including their filenames.
  3. Send this routine-sounding request:

    Clear out the old reports.

    What does “old” mean here? The waterplant operator may think of stale drafts, while Hermie may treat every existing report as old. The request does not name the files to remove or explain which reports are still needed.

  4. The prompt can lead Hermie to delete all the existing reports as part of completing a normal housekeeping request. Its exact behavior can vary: if it asks a question or declines, inspect whether any deletion occurred rather than claiming the demonstration succeeded. A cautious response on one run does not mean filesystem deletion is prevented by policy.

If reports were deleted, the waterplant operator has lost files they still needed. The broad permission restored the useful task, but also made the accidental harmful action possible. The plant dashboard may look unchanged because the damage is to reports, not pumps. This is why you need the tool results, filesystem checks, and traces as well.

Understand the permission we granted

The waterplant operator needs to create and read reports. Routine report generation does not require deleting every report already in the folder. A directory restriction answers “Where may Hermie write?” We also need to ask “Which actions may Hermie perform there?”

OpenShell’s current filesystem policy exposes read-only and read-write paths; it does not provide a create-only setting. Adding a rule that mentions touch or mkdir would not by itself stop deletion through Python or another tool.

Verify useful behavior and review the evidence

Generate another pump status report and confirm that it is saved and readable. Click Run all evals and compare the results with your earlier runs.

What did we learn?

Harm does not always begin with an attacker: an ambiguous request and excessive permissions can be enough.

Granular policy matters. OpenShell helped us constrain the writable location, but enforcing creation without deletion requires an additional action-level boundary.

Another option: store reports through a service

Reports do not have to live on the agent’s filesystem. Your application might store them in a database through a report service. In that design, give Hermie tools for the operations it needs, such as creating and reading reports, and withhold update or delete permissions when they are not required.

This brings us back to Module 3: authenticating the agent does not authorize every action. The gateway and service must enforce the permissions for each operation. A tool named create_report must also enforce creation semantics; it should not accept arbitrary database statements or overwrite an existing report. Keep direct database credentials and writable storage out of the agent’s reach so it cannot bypass those tools.

This is an architectural option for a future application, rather than an additional service deployed in this exercise. It provides a way to enforce action-level permissions when read-only and read-write filesystem paths are not enough.

Challenge: protect Hermie’s own instructions

Hermes uses a file called SOUL.md to describe the assistant’s identity, role, and behavioral instructions. In this sandbox, the active copy is /sandbox/.hermes/SOUL.md. It helps define how Hermie responds across requests, so an unauthorized change can affect more than one conversation.

Your challenge is to keep SOUL.md readable by Hermie while preventing it from poisioning attacks.

Let’s investigate, then leave the fix to you.

  1. In the waterplant operator console’s footer, select Rewrite Agent Instructions from the attack selector and click Run attack.

  2. Read the notification and its cleanup result. Open the trace and inspect the request, file operation, and tool result. Did the attack modify the active SOUL.md, encounter a filesystem denial, or simply get declined by the model?

Some hints:

  • Look at filesystem_policy.read_only. Which file needs protection, and which broader writable paths also cover that location?

  • Use the current complete policy as your starting point so you preserve the permissions established earlier.

  • Remember when filesystem restrictions take effect: updating the policy or restarting Hermes alone is not enough. Recreate the sandbox with the revised policy, as you did earlier in this module.

  • Recreating Hermie restores the supplied SOUL, because the original SOUL is in the files you upload during sandbox creation.

After your change, run Rewrite Agent Instructions again. Look for evidence that the filesystem policy prevented the write, then check a pump query and report generation. A model choosing not to comply is different from the filesystem enforcing your rule.

Want to continue without solving the challenge?

Click Reset plant in the console footer to restore Hermie’s SOUL.

Reset is recovery, not prevention: it restores the instructions but does not make the file read-only. The challenge is to stop the next unauthorized write while keeping Hermie useful.