Onchain Storage

Chunked Uploader

Overview

The Chunked Uploader is a fault-tolerant, resumable, stream-based signer and uploader. It allows you to pause and resume uploads, and to do things like create progress bars that show upload progress.

Batch size
Chunk size

For those with slower or unstable connections, reducing both should lead to improved reliability. For those with faster connections, increasing both will lead to higher throughput, at the cost of more memory and CPU.

Info

The default uploader (accessed via irys.upload()) does chunking automatically. You only need to use the Chunked Uploader to access advanced features like pausing and resuming uploads, changing batch size, and changing chunk size.

Connecting

When using the chunking uploader, first create an Irys object and then request the chunked uploader using irys.uploader.chunkedUploader.

Warning

The chunkedUploader object reference must be updated before each subsequent upload. It cannot be reused.

const irys = await getIrysUploader();

const uploader = irys.uploader.chunkedUploader;

Note

Use a token-specific version of getIrysUploader() to connect to an Irys Bundler before uploading. Choose one from the setup documentation.

Setting Upload Parameters

The default batch size (number of chunks uploaded simultaneously) is 5, while the default chunk size (maximum chunk size) is 25MB. These parameters can be customized using uploader.setBatchSize(size) and uploader.setChunkSize(size).

uploader.setBatchSize(10);

// Value is in bytes
uploader.setChunkSize(500000);

Data Mode

The chunked uploader operates in two distinct modes: Data Mode and Transaction Mode. When using Data Mode, you should not create a transaction manually -- this will be done automatically for you.

Within Data Mode, you can upload data using either a buffer containing the data you want to upload, or a readable stream pointing to the data you want to upload.

uploader.uploadData()

const transactionOptions = { tags: [{ name: "Content-Type", value: "text/plain" }] };

// 1. Upload a Buffer containing just the data you want to upload.
const dataBuffer = Buffer.from("Hirys, world!");
const response = await uploader.uploadData(dataBuffer, transactionOptions);
// The transaction ID (used to query the network) is found in response.data.id
console.log(`Data buffer uploaded ==> https://gateway.irys.xyz/${response.data.id}`);

// 2. OR Upload a Readable (stream) pointing to the data
uploader = irys.uploader.chunkedUploader; // Recreate for each transaction
const dataStream = fs.createReadStream("./data.txt");
response = await uploader.uploadData(dataStream, transactionOptions);
console.log(`Read Stream uploaded ==> https://gateway.irys.xyz/${response.data.id}`);

Transaction Mode

Transaction Mode gives you more fine-grained control over the upload workflow. You can create and sign your transaction first, store it, and then upload when it makes the most sense for your application.

uploader.uploadTransaction()

uploader = irys.uploader.chunkedUploader; // Recreate for each transaction
const transaction = irys.createTransaction("Hello, world!");
await transaction.sign();
response = await uploader.uploadTransaction(transaction);
console.log(`Transaction mode uploaded ==> https://gateway.irys.xyz/${response.data.id}`);

The workflow involves three key steps:

Create a transaction using irys.createTransaction().
Sign the transaction with transaction.sign().
Upload using uploader.uploadTransaction(transaction).

Controlling the Upload

Uploads created with the chunked uploader can be paused and resumed at any time using uploader.pause() and uploader.resume().

Warning

The initial call to uploader.uploadData() or uploader.uploadTransaction() must not use the await keyword for pause/resume to work properly.

To resume an upload from a new uploader instance, maintain consistency with: token, bundler network (mainnet/devnet), input data, and configured chunk size.

const irys = await getIrysUploader();

// When uploading smaller files, it's common to use the await keyword before
// uploadData() or uploadTransaction(). This causes execution to pause until the file
// is fully uploaded. If you omit await, the upload happens in the background
// and you can use pause and resume as needed.
transaction = irys.createTransaction("Hello, world!");
uploader = irys.uploader.chunkedUploader; // Recreate for each transaction
const upload = uploader.uploadTransaction(transaction);
uploader.pause(); // Pauses the upload
console.log("Upload paused");
uploader.resume(); // Resumes the upload
console.log("Upload resumed");

While the initial upload call should not use await, you can apply it later at any time to verify completion:

response = await upload;

Expired Uploads

Paused uploads will expire after a period of inactivity. To recover an expired upload, use the following approach:

const resumeData = uploader.getResumeData();
uploader.setResumeData(resumeData);
await uploader.uploadTransaction(dataItem);

To restore a paused upload that has expired:

Retrieve the resume data using getResumeData().
Reinitialize the uploader with that data via setResumeData().
Continue the upload by calling uploadTransaction() with the data item.

Upload Events

The uploader emits three events during each upload. These can be subscribed to for any use case when tracking upload progress is needed.

chunkUpload
chunkError
done

uploader.on("chunkUpload", (chunkInfo) => {
  console.log(
    `Uploaded Chunk number ${chunkInfo.id}, offset of ${chunkInfo.offset}, size ${chunkInfo.size} Bytes, with a total of ${chunkInfo.totalUploaded} bytes uploaded.`,
  );
});

uploader.on("chunkError", (e) => {
  console.error(`Error uploading chunk number ${e.id} - ${e.res.statusText}`);
});

uploader.on("done", (finishRes) => {
  console.log(`Upload completed with ID ${finishRes.id}`);
});

Previousutils.verifyReceipt()NextIrys in the Browser