Upload Files in Pipedream Workflows - Get Public URLs Instantly

Pipedream is one of the best platforms for code-first workflow automation. You get real Node.js and Python steps, access to npm and PyPI packages, built-in triggers for hundreds of services, and the ability to write custom logic wherever you need it. If you have ever felt limited by drag-and-drop automation tools, Pipedream is the answer.

But there is one gap that trips up developers constantly: file hosting. Pipedream can receive files, process files, and pass files between steps, but it has no built-in way to host a file and give you a permanent, public URL. You get a /tmp directory that vanishes between executions. You get binary data that exists only during the workflow run. What you do not get is a URL you can store in a database, send in a Slack message, or return in an API response.

This guide covers how to upload files from Pipedream workflows and get permanent CDN URLs using the FilePost API. We will walk through complete Node.js and Python step code, two real-world workflow examples, and practical tips for handling binary data in Pipedream.

Why Pipedream Doesn't Host Files

Pipedream is a workflow execution engine, not a file storage service. Understanding its file handling limitations helps explain why you need an external solution.

• /tmp is ephemeral. Pipedream gives each workflow execution access to a /tmp directory with 512 MB of space. But this storage is not persistent. Files written to /tmp may survive briefly across executions running on the same worker, but there is zero guarantee. Pipedream's own documentation warns against relying on /tmp for persistence.
• No CDN, no public URLs. Even if a file exists in /tmp during execution, there is no way to generate a public URL for it. Pipedream does not serve static files. The /tmp directory is local to the execution environment and invisible to the outside world.
• Binary data lives in memory. When a trigger receives a file (via webhook, email, or another service), the file data exists as a Buffer in the step's execution context. Once the workflow finishes, that data is gone.
• No built-in storage integration. Unlike some platforms that bundle S3 or similar storage, Pipedream requires you to bring your own storage solution. You need to explicitly upload files somewhere if you want them to persist.

This is not a criticism of Pipedream. It is a design decision that keeps the platform fast and focused. But it means you need a file hosting API to close the gap.

The Solution: FilePost API

FilePost is a file upload API built for exactly this use case. You POST a file, you get back a permanent CDN URL. No buckets to configure, no IAM policies, no region selection. One HTTP request, one URL.

Here is what the API call looks like:

curl -X POST https://filepost.dev/v1/upload \
  -H "X-API-Key: YOUR_API_KEY" \
  -F "file=@photo.jpg"

Response:

{
  "url": "https://cdn.filepost.dev/abc123/photo.jpg",
  "file_id": "abc123",
  "name": "photo.jpg",
  "size": 45321
}

The URL is permanent, served through Cloudflare's global CDN, and works until you explicitly delete the file. That is exactly what a Pipedream workflow needs: a way to take an in-memory file and turn it into a stable URL.

Node.js Step: Upload a File

Pipedream's Node.js steps have full access to npm packages. Here is a complete step that uploads a file to FilePost using axios and FormData.

import axios from "axios";
import FormData from "form-data";

export default defineComponent({
  props: {
    filepost_api_key: {
      type: "string",
      label: "FilePost API Key",
      secret: true,
    },
  },
  async run({ steps, $ }) {
    // Get the file buffer from a previous step
    // Adjust steps.trigger.event based on your trigger type
    const fileBuffer = steps.trigger.event.body;
    const fileName = steps.trigger.event.headers["x-filename"] || "upload.bin";

    // Build the multipart form data
    const form = new FormData();
    form.append("file", Buffer.from(fileBuffer), {
      filename: fileName,
      contentType: steps.trigger.event.headers["content-type"] || "application/octet-stream",
    });

    // Upload to FilePost
    const response = await axios.post(
      "https://filepost.dev/v1/upload",
      form,
      {
        headers: {
          "X-API-Key": this.filepost_api_key,
          ...form.getHeaders(),
        },
        maxContentLength: Infinity,
        maxBodyLength: Infinity,
      }
    );

    // Export the URL for downstream steps
    $.export("file_url", response.data.url);
    $.export("file_id", response.data.file_id);
    $.export("file_name", response.data.name);
    $.export("file_size", response.data.size);

    return response.data;
  },
});

A few things to note about this step:

• Props for secrets. The API key is defined as a prop with secret: true, so Pipedream stores it encrypted and never logs it.
• $.export() for downstream access. Exporting the URL means any subsequent step can access it via steps.upload_file.file_url (where upload_file is the step name).
• maxContentLength: Infinity. This prevents axios from rejecting large file uploads. Without it, files over a few megabytes may fail.
• Buffer.from() wrapping. Depending on your trigger, the file data may arrive as a Buffer, a string, or a Uint8Array. Wrapping it in Buffer.from() normalizes it.

Python Step: Upload a File

Pipedream also supports Python steps with access to PyPI packages. Here is the equivalent upload step in Python.

import requests

def handler(pd: "pipedream"):
    # Get file data from a previous step
    file_data = pd.steps["trigger"]["event"]["body"]
    file_name = pd.steps["trigger"]["event"]["headers"].get("x-filename", "upload.bin")
    content_type = pd.steps["trigger"]["event"]["headers"].get("content-type", "application/octet-stream")

    # Upload to FilePost
    response = requests.post(
        "https://filepost.dev/v1/upload",
        headers={"X-API-Key": pd.inputs["filepost_api_key"]},
        files={"file": (file_name, file_data, content_type)},
    )

    result = response.json()

    # Export for downstream steps
    pd.export("file_url", result["url"])
    pd.export("file_id", result["file_id"])
    pd.export("file_name", result["name"])
    pd.export("file_size", result["size"])

    return result

The Python step follows the same pattern: pull the file data from a previous step, POST it to FilePost, and export the resulting URL. The requests library handles multipart encoding automatically when you use the files parameter.

Example Workflow: Webhook to Public File URL

This is the simplest practical workflow. An HTTP webhook receives a file, uploads it to FilePost, and returns the permanent URL in the HTTP response. Four steps, zero external dependencies beyond FilePost.

Step 1: HTTP Webhook Trigger

Create a new workflow and select the HTTP / Webhook trigger. Pipedream gives you a unique URL like https://eo1234abc.m.pipedream.net. Any file POSTed to this URL will trigger the workflow.

Step 2: Upload to FilePost

Add a Node.js step with the upload code from the section above. The webhook trigger passes the file body through steps.trigger.event.body and the content type through the headers.

Step 3: Return the URL

Add a final step that sends the URL back in the HTTP response:

export default defineComponent({
  async run({ steps, $ }) {
    await $.respond({
      status: 200,
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({
        url: steps.upload_to_filepost.file_url,
        file_id: steps.upload_to_filepost.file_id,
        size: steps.upload_to_filepost.file_size,
      }),
    });
  },
});

Testing the Workflow

Send a file to your webhook and you get a permanent URL back:

curl -X POST https://eo1234abc.m.pipedream.net \
  -H "Content-Type: image/png" \
  -H "X-Filename: screenshot.png" \
  --data-binary @screenshot.png

Response:

{
  "url": "https://cdn.filepost.dev/abc123/screenshot.png",
  "file_id": "abc123",
  "size": 84210
}

You now have a webhook that turns any file into a permanent CDN URL. You can call this endpoint from any app, script, or service that can make HTTP requests.

Example Workflow: Process Email Attachments

This workflow monitors a Gmail inbox for emails with attachments, uploads each attachment to FilePost, and logs the file URLs to a Google Sheet or Airtable base. It is useful for teams that receive documents by email and need them accessible via URL.

Step 1: Gmail Trigger

Use Pipedream's Gmail - New Email trigger. Configure it to watch a specific label or your entire inbox. When an email arrives with attachments, Pipedream parses the email and makes the attachments available as base64-encoded data.

Step 2: Loop Over Attachments

Emails can have multiple attachments. Use a Node.js step to iterate over them and upload each one:

import axios from "axios";
import FormData from "form-data";

export default defineComponent({
  props: {
    filepost_api_key: {
      type: "string",
      label: "FilePost API Key",
      secret: true,
    },
  },
  async run({ steps, $ }) {
    const attachments = steps.trigger.event.attachments || [];
    const results = [];

    for (const attachment of attachments) {
      // Decode base64 attachment data
      const fileBuffer = Buffer.from(attachment.data, "base64");

      const form = new FormData();
      form.append("file", fileBuffer, {
        filename: attachment.filename,
        contentType: attachment.mimeType,
      });

      const response = await axios.post(
        "https://filepost.dev/v1/upload",
        form,
        {
          headers: {
            "X-API-Key": this.filepost_api_key,
            ...form.getHeaders(),
          },
        }
      );

      results.push({
        original_name: attachment.filename,
        url: response.data.url,
        file_id: response.data.file_id,
        size: response.data.size,
      });
    }

    $.export("uploaded_files", results);
    return results;
  },
});

Step 3: Store URLs in Airtable or Google Sheets

Add a Google Sheets or Airtable step that writes a row for each uploaded file. Use the exported uploaded_files array from the previous step. Each row should include the sender's email, the original filename, the FilePost URL, and a timestamp.

The result is a fully automated pipeline: someone sends you an email with attachments, and within seconds those files are permanently hosted with URLs logged in your spreadsheet. No manual downloading, no forwarding, no drag-and-drop into cloud storage.

Working with Binary Data in Pipedream

File handling in Pipedream has some quirks. Here are practical tips for dealing with binary data across steps.

Accessing Data from $.steps

Every step's output is available to subsequent steps via steps.step_name. When a trigger receives binary data, the location depends on the trigger type:

• HTTP webhook: Raw body is in steps.trigger.event.body. For multipart form uploads, individual files are in steps.trigger.event.body.file (or whatever field name was used).
• Gmail trigger: Attachments are in steps.trigger.event.attachments as an array of objects with data (base64), filename, and mimeType.
• S3 / Dropbox trigger: These triggers typically provide a file path or download URL. You need an intermediate step to download the file content before uploading to FilePost.

Buffer Handling in Node.js Steps

Pipedream Node.js steps run in a standard Node.js environment, so all Buffer methods work as expected. Common patterns:

// From base64 string
const buffer = Buffer.from(base64String, "base64");

// From a URL (download first, then upload)
const download = await axios.get(fileUrl, { responseType: "arraybuffer" });
const buffer = Buffer.from(download.data);

// From a previous step's raw body
const buffer = Buffer.from(steps.trigger.event.body);

// Check the buffer size before uploading
console.log(`File size: ${buffer.length} bytes`);

Returning Files in $.send.http()

If your workflow is triggered by an HTTP webhook and you want to return the FilePost URL in the response, use $.respond() at the end of your workflow. Important: you can only call $.respond() once per execution, and it must be in the last step that interacts with the HTTP response.

// Return the CDN URL as a JSON response
await $.respond({
  status: 200,
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    success: true,
    url: steps.upload_to_filepost.file_url,
  }),
});

Handling Large Files

Pipedream workflows have memory limits (256 MB default). If you are working with large files, keep these tips in mind:

• Avoid loading the entire file into memory twice. Do not copy a Buffer unnecessarily.
• If a previous step provides a download URL instead of raw data, stream the download directly into the FormData object rather than buffering the entire file first.
• Check your Pipedream plan's execution time limit. Free plans have a 30-second timeout, which may not be enough for large file uploads. Paid plans extend this to 12 minutes.
• Monitor the /tmp directory size if you are writing files to disk during processing. The 512 MB limit is shared across all files.

FAQ

Does Pipedream have built-in file hosting?

No. Pipedream provides a /tmp directory during workflow execution, but it is ephemeral. Files stored in /tmp are wiped between executions and are not accessible via URL. To host files permanently and get public URLs, you need an external file hosting API like FilePost.

How do I get a public URL for a file in Pipedream?

Upload the file to a file hosting API from within your Pipedream step. With FilePost, you POST the file to https://filepost.dev/v1/upload with your API key, and the response includes a permanent CDN URL. You can export this URL with $.export() and use it in any downstream step, whether that is returning it in an HTTP response, storing it in a database, or sending it in a notification.

What happens to /tmp files in Pipedream?

Pipedream's /tmp directory provides 512 MB of temporary storage during a single workflow execution. Files written to /tmp may persist briefly across executions running on the same worker, but this is not guaranteed and should not be relied on. If you need a file to survive beyond the current execution, upload it to a permanent hosting service like FilePost before the workflow ends.

Can I upload large files from Pipedream?

Yes, but be aware of limits on both sides. Pipedream workflows have memory limits (typically 256 MB) and execution time limits (30 seconds on free, up to 12 minutes on paid plans). FilePost supports files up to 50 MB on the free tier, 200 MB on Starter ($9/mo), and 500 MB on Pro ($29/mo). For large file uploads, make sure your Pipedream plan has sufficient execution time, and avoid buffering the file in memory more than once.