6. Accelerate data transfers with Direct Memory Access (DMA)#
6.1. Required files#
SpaceStudio Project
6.2. Introduction#
In data-centric applications with high bandwidth requirements, processing the data stream at real-time speed can be challenging. Memories are often used to buffer the stream for processing by algorithmic blocks, but such memory accesses can become a bottleneck.
This tutorial introduces the SpaceStudio Direct Memory Access (DMA) capabilities. The attendee will move a block of data from software to a hardware co-processor over three different communication mechanisms — a plain hardware FIFO, a DMA-configured channel, and a DMA fed straight from a memory — and measure, on real hardware, how each one performs as the size of a transfer grows.
Note
The communication mechanisms are compared with an on-target measurement: each architecture runs on a real board through FPGA-in-the-loop (FIL) and the controller times its own transfers. A SystemC simulation time would instead say how fast the host runs the simulator, not how the SoC behaves; timing on the target is reproducible, directly meaningful, and much quicker to obtain. The tutorial also covers the non-blocking (asynchronous) DMA API, where DMA’s real payoff becomes visible: the processor is free to do other work while the transfer runs.
6.2.1. Efficient Memory Data Transfers#
General-purpose processors can perform memory transfers, but this implies a performance hit. Hardware IPs can access memories as bus masters using their own custom logic, but this incurs a development and maintenance effort that is not part of the IP’s core functionality.
A better solution is to use Direct Memory Access (DMA). A DMA IP can transfer large amounts of data with minimal interaction with the software. In a nutshell, the software indicates to the DMA the source and destination addresses along with the number of bytes to be transferred. The main advantage is offloading the actual data transfer to the DMA, freeing the processor to work on other parts of the algorithm. DMA IPs use various techniques to achieve high throughput such as burst bus operations, wide data paths, higher frequency, circular mode, scatter and gather, etc.
There are three types of DMA operation: 1) memory to memory, 2) memory to stream and 3) stream to memory. Whichever one a design uses, the DMA IP is configured by a controller module running on a processor; that controller can be dedicated to driving the DMA, or it can be an ordinary task of the application. SpaceStudio instantiates the DMA IP, connects it to all the relevant interconnects and generates an optimized software driver for you.
The two sections below introduce the three calls that express these transfers, then show how a DMA ends up behind one.
6.2.1.1. The communication API#
A transfer between two component instances crosses a channel, and both of its ends are named the same way: StreamWrite to put data on the channel, StreamRead to take data off it. The first argument is always the instance at the other end — never a memory, never a transport. The last argument is the timeout: SPACE_BLOCKING returns only once the transfer is complete (SpaceStudio handles the interrupt for you), SPACE_NON_BLOCKING returns immediately with a space_transfer_t handle to join on later.
Only one thing varies between the three calls: what the caller hands over.
1. Buffer transfer — the caller passes its own array. It produced the data, or it wants the data:
uint32_t block[256];
for (int i = 0; i < 256; ++i)
block[i] = compute(i);
StreamWrite(CHECKSUM0_ID, SPACE_BLOCKING, block, 256); // 256 *elements*
uint32_t sum;
StreamRead(CHECKSUM0_ID, SPACE_BLOCKING, sum); // read the result back
2. Region transfer — the caller passes a memory_region_t instead: which memory, at what byte offset, how many bytes. It never touches the data; it wires the channel to a memory and lets a DMA engine move the bytes:
// 1 KB sitting at offset 0 of ZYNQ_DDR0 goes out on the channel to CHECKSUM0.
StreamWrite(CHECKSUM0_ID, SPACE_BLOCKING,
memory_region_t{ZYNQ_DDR0_ID, 0x0, 1024}); // 1024 *bytes*
// ... and the same in reverse: what CHECKSUM0 sends lands in the DDR.
StreamRead(CHECKSUM0_ID, SPACE_BLOCKING,
memory_region_t{ZYNQ_DDR0_ID, 0x0, 1024});
3. Memory-to-memory copy — no channel and no component at the far end, just two memories, so this call is named after its transport rather than after a peer:
DmaCopy(memory_region_t{VIDEO_RAM0_ID, 0x0, 0x100000}, // destination
memory_region_t{ZYNQ_DDR0_ID, 0x0, 0x100000}, // source
SPACE_BLOCKING);
Note the unit: the buffer form counts elements, the memory_region_t form counts bytes.
6.2.1.2. How a DMA gets involved#
A DMA appears for one of two reasons:
The channel is configured to use one. A buffer transfer looks exactly like a plain hardware FIFO in the source. The designer marks a specific channel — a matching StreamWrite / StreamRead pair — to be realized by a DMA IP in the Communication tab of the Properties view, and the same unchanged code now runs over a DMA. The decision lives in the architecture, not at the call site, which is what makes the mechanism swappable without rewriting the application.
The data is already in a memory. A region transfer names operands the caller only points at, so a DMA is the only thing that can move them; likewise for DmaCopy. Here the choice is the call site.
In this tutorial the region transfers use a scratch window of the Zynq PS DDR (ZYNQ_DDR0) rather than a dedicated PL block RAM: the same DDR the processor already runs from. The controller stages the block into that window itself before launching the DMA, mirroring how a real system stages data in DDR and then offloads the movement.
On Linux, a region transfer is faster than a DMA-configured buffer transfer, because the kernel driver does not have to copy the user-space buffer into a kernel DMA buffer first.
6.3. Manipulation#
6.3.1. The reduction application#
To compare the communication mechanisms fairly, the application is deliberately tiny. It has just two components: the controller (software, on zynq0) and a checksum co-processor. The controller sends one contiguous block of 32-bit words to checksum; the co-processor sums the block (a reduction) and returns the single scalar. The controller measures how long the round trip takes, checks the result, and reports the throughput. It repeats this over a sweep of transfer sizes, from 16 B up to 1024 KB, doubling at each step — the sweep is what makes the FIFO → DMA crossover visible in a single run.
The sweep is described in application_definitions.h:
// Only the *ceiling* is fixed at build time: SpaceStudio sizes the buffers
// and the communication channel from the statically known maximum, so the
// length passed to an individual transfer may vary at run time as long as
// it never exceeds SWEEP_MAX_BYTES.
#define SWEEP_MIN_BYTES 16u
#define SWEEP_MAX_BYTES (1024u * 1024u)
#define SWEEP_POINTS 17u // one per power of two
#define SWEEP_MIN_WORDS (SWEEP_MIN_BYTES / 4u) // 4 words
#define MAX_BLOCK_WORDS (SWEEP_MAX_BYTES / 4u) // 262144 words
// Words transferred at sweep point `i` (0 .. SWEEP_POINTS-1).
#define SWEEP_WORDS(i) (SWEEP_MIN_WORDS << (i))
// Bounded on-chip buffer through which the co-processor drains the stream.
#define CHUNK_WORDS 1024u
// Back-to-back transfers per sweep point: the first is warm-up; the rest
// show the steady-state throughput.
#define BENCH_ITERS 8u
The checksum co-processor drains the stream through a bounded buffer of CHUNK_WORDS words, so its on-chip memory footprint stays at 4 KB even at the 1024 KB sweep point — it never has to hold a whole block. It walks the same sweep in lockstep with the controller, which is how it knows how many words to expect without a length header travelling on the channel and distorting the measurement:
void checksum::thread(SPACECOMP_THREAD_PARAMS(checksum, INDEX, thread)) {
spacecomp_thread_initialize();
uint32_t data[CHUNK_WORDS];
unsigned point = 0, iter = 0;
while (1) {
const bool initializing = spacecomp_thread_loop_start();
if (initializing) { point = 0; iter = 0; }
send_sum(sum_block(data, SWEEP_WORDS(point)));
if (++iter == BENCH_ITERS) {
iter = 0;
if (++point == SWEEP_POINTS)
point = 0;
}
}
}
// Drain `words` words through the bounded buffer, accumulating as we go.
// The last chunk is short whenever `words` is not a multiple of CHUNK_WORDS
// -- and at the small end of the sweep it is the only chunk.
uint32_t checksum::sum_block(uint32_t* data, uint32_t words) {
uint32_t sum = 0;
for (uint32_t done = 0; done < words; done += CHUNK_WORDS) {
const uint32_t remaining = words - done;
const uint32_t n = (remaining < CHUNK_WORDS) ? remaining : CHUNK_WORDS;
read_chunk(data, n);
sum += reduce(data, n);
}
return sum;
}
In all three architectures checksum reads and writes with plain buffer transfers — its source is byte-for-byte identical across the three, and it never learns that a memory or a DMA is involved. The controller calls the same verb pair throughout; all that changes is its data argument, which becomes a memory_region_t in the third architecture.
6.3.2. Communication performance#
The controller measures the elapsed time of one block transfer on the target and prints it. The measurement is taken with std::chrono::steady_clock around the send/reduce/receive sequence:
const uint32_t words = SWEEP_WORDS(point);
const uint64_t t_start = now_ns();
send_block(m_buffer, words);
const uint32_t sum = read_sum();
const uint64_t t_end = now_ns();
const bool valid = (sum == cpu_sum(m_buffer, words));
// Print in the most readable unit (ns/us/ms/s) for the magnitude.
const uint64_t ns = t_end - t_start;
double value; const char* unit;
if (ns < 1000ULL) { value = ns; unit = "ns"; }
else if (ns < 1000000ULL) { value = ns / 1e3; unit = "us"; }
else if (ns < 1000000000ULL) { value = ns / 1e6; unit = "ms"; }
else { value = ns / 1e9; unit = "s"; }
const double mbps = (double)words * 4.0 / ((double)ns / 1e9) / 1e6;
printf("block %6u words (%8u B) iter %u: %8.3f %s %8.1f MB/s [%s]\n",
words, words * 4u, iter, value, unit, mbps,
valid ? "OK" : "FAIL");
now_ns() reads std::chrono::steady_clock on the target. Each line also ends with [OK] or [FAIL]: the controller re-computes the sum on the ARM core (in uint32_t, matching the co-processor’s wrap-around arithmetic) and compares it against the value the hardware returned. This confirms the transport actually moved the right data: a mis-programmed DMA or dropped word shows up immediately as FAIL rather than a plausible-looking wrong number.
6.3.2.1. Hardware FIFO#
As delivered, the controller moves the block over a plain hardware FIFO. It sends the block with StreamWrite and reads the scalar back with StreamRead:
void controller::send_block(const uint32_t* v, uint32_t words) {
StreamWrite(CHECKSUM0_ID, SPACE_BLOCKING, v, words);
}
uint32_t controller::read_sum() {
uint32_t value;
StreamRead(CHECKSUM0_ID, SPACE_BLOCKING, value);
return value;
}
This is the buffer form, so the count is in elements, not bytes. Build the hardware_fifo architecture, run it on the board, and note its throughput as the baseline.
6.3.2.2. DMA on a configured channel#
Nothing changes in the source: the controller keeps the buffer transfer above. The DMA is selected purely by configuring the channel:
Select the
controllermodule in the diagramIn the Properties view, select the Communication tab
Double-click the Channel type cell of the channel whose reader is
checksum; a combo box appears — select DMASelect the
checksummodule and repeat for the channel whose reader iscontroller
Both directions must be changed: a channel type applies to one direction only, and the channel type defaults to Hardware FIFO.
The dma_channel architecture already has both channels set to DMA. Build it, run it, and note its throughput.
6.3.2.3. DMA from a memory region#
The third architecture changes where the data lives: in a memory the controller names with a memory_region_t, rather than in an application array. In this tutorial that memory is the Zynq PS DDR (ZYNQ_DDR0). The controller stages the block into a reserved DDR window with DeviceWrite, then streams it out with a DMA and reads the result back:
#define STAGE_BLOCK_OFFSET ((uintptr_t)0)
#define STAGE_RESULT_OFFSET ((uintptr_t)MAX_BLOCK_WORDS * 4)
void controller::send_block(const uint32_t* v, uint32_t words) {
DeviceWrite(ZYNQ_DDR0_ID, STAGE_BLOCK_OFFSET, v, words);
space_transfer_t transfer = StreamWrite(
CHECKSUM0_ID, SPACE_NON_BLOCKING,
memory_region_t{ZYNQ_DDR0_ID, STAGE_BLOCK_OFFSET, words * 4});
space::wait(transfer);
}
DeviceWrite(ZYNQ_DDR0_ID, offset, ...) addresses the DDR by memory id and byte offset. The offset is relative to ZYNQ_DDR0’s base — offset 0 is the start of the staging window — and the DMA reads back the very same offset, so software pre-load and DMA stay in sync in both simulation and hardware.
Important
The DDR region backing ZYNQ_DDR0 that is used as the DMA scratch must be reserved from Linux. Reserve it through the board’s device tree (or reduce the amount of DDR handed to Linux) so the kernel never allocates over the DMA scratch region. The staging window is sized for the largest sweep point, so it must be at least SWEEP_MAX_BYTES + 4 bytes — 1 MB + 4 B with the values above.
For simplification, the controller exposes a single user-defined parameter, REGION_TRANSFER, which switches between the buffer-transfer path (hardware FIFO and DMA-configured channel) and the region-transfer path. The three architectures set it for you: hardware_fifo and dma_channel (REGION_TRANSFER = 0), and dma_region (REGION_TRANSFER = 1).
6.3.2.4. Non-blocking (asynchronous) transfers#
A blocking DMA call does not return until the transfer completes, so the processor gains nothing but a free bus master. The real advantage of a DMA is that the processor can keep working while the transfer is in flight. The asynchronous API returns a space_transfer_t handle immediately when SPACE_NON_BLOCKING is passed as the timeout; the caller then does useful work and later joins on the transfer:
space_transfer_t transfer = StreamWrite(
CHECKSUM0_ID, SPACE_NON_BLOCKING,
memory_region_t{ZYNQ_DDR0_ID, STAGE_BLOCK_OFFSET, words * 4});
m_overlap_checksum += overlap_work(); // work done while the DMA runs
space::wait(transfer); // join
The space namespace provides four mutually-exclusive predicates to query a transfer without blocking: space::is_pending, space::is_busy, space::is_error and space::is_completed. A polling loop should be written while (space::is_pending(transfer)) { ... }. The dma_region architecture uses this non-blocking path.
6.3.2.5. Where DMA pays off: sweeping the transfer size#
Running the benchmark reveals the key lesson about DMA. A DMA has a fixed per-transfer cost (descriptor setup, a completion interrupt, and — for a DMA-configured channel on Linux — a copy of the application buffer into a kernel DMA buffer). For a tiny transfer that fixed cost dwarfs the few bytes moved, so a plain FIFO, which just pushes words with no setup, wins. A DMA only pays off once a single transfer is large enough to amortize its setup cost. The lever is therefore the size of one transfer.
Run the sweep once for each of hardware_fifo, dma_channel and dma_region, then plot throughput against transfer size. You should see:
for small blocks, the hardware FIFO has the highest throughput (no setup cost);
past a threshold, the region transfer overtakes it;
at a larger threshold, the DMA-configured channel overtakes the FIFO too — its threshold is higher because of the extra Linux user-to-kernel copy.
The lesson is not “DMA is always faster”; it is “DMA wins above a transfer-size threshold, and a region transfer’s threshold is lower than a DMA-configured channel’s on Linux.”
6.3.3. Running on hardware (FPGA-in-the-loop)#
To obtain the on-target numbers, each architecture is deployed to a ZedBoard through the FPGA-in-the-loop flow introduced in Tutorial 4. For a given architecture, SpaceStudio exports the virtual platform to Vivado, builds the bitstream and the Linux boot image, programs the board, and runs the application while the controller prints its timing in the SpaceStudio console.
Because the sweep runs inside a single build, only the three architectures have to be deployed — but doing even that by hand is tedious and error-prone. Use the Architect Dashboard from Tutorial 5 to automate the deployment: queue the hardware_fifo, dma_channel and dma_region architectures, let the dashboard drive build → program → run for each, and collect the reported throughput.
6.4. Conclusion#
The attendee moved the same block of data to the same co-processor over three communication mechanisms, changed mechanism without rewriting the application — a channel-type setting in one case, a memory_region_t argument in the other — and measured the outcome on a real board.
What the measurements show:
A DMA is not free. Descriptor setup, the completion interrupt, and (for a DMA-configured channel on Linux) the user-to-kernel copy are a fixed cost paid on every transfer. Below a transfer-size threshold, the plain hardware FIFO wins.
The lever is the size of one transfer, not the total amount of data moved. Batching a stream into fewer, larger transfers is what makes a DMA pay off.
A region transfer crosses over earlier than a DMA-configured channel on Linux, because the data already sits in a memory the DMA can read and no buffer copy is needed.
Throughput is only half of the payoff. With
SPACE_NON_BLOCKING, the transfer time is removed from the processor’s critical path entirely: thecontrollerkeeps working while the DMA moves the block, which a throughput figure alone does not show.
The design lesson generalizes beyond this application: choose the communication mechanism from the transfer size the application actually produces, and prefer the non-blocking API whenever the processor has other work available. Since the mechanism is an architecture-level choice in SpaceStudio, the cheapest way to settle it is the one used here — build the candidate architectures, run them on the target, and compare.
6.5. Result files#
SpaceStudio Project