A huge variety of servers around all sorts of communities, efforts, and initiatives have been popping up and are still appearing. And they're largely taking the place of forums, chat rooms and, at times, documentation sites and news boards.

To what degree this is a good thing is up for debate, but the long and short of it is that Discord is likely here to stay.

When it comes to accessibility, Discord's had a bit of a rocky road. For years, it was very painful to use for screen reader users due to an apparent lack of forethought regarding accessibility.

Over the last few years, this situation has improved substantially. While it's by no means fully accessible in 2026, it can be used relatively efficiently once you know the tricks of the trade.

freeCodeCamp uses Discord, and the community has often seen that screen reader users struggle to use this communication tool comfortably. This is where this article comes in.

In this article, I'll go over the basics you'll need to use Discord as a platform to send and receive messages and contribute to communities that use Discord as a communication platform. It's relatively easy to learn but can be difficult to fully grasp due to the various interwoven things it does. Still, these basics should get you up and running and will equip you to learn about all the other features it offers by yourself going forward.

Table of Contents

• Screen Reader Prerequisites

• The Broad Strokes: How to Navigate Efficiently

• Speeding Things Up: How to Navigate and Interact Efficiently

• When is Browse Mode More Efficient?

• Conclusion

Screen Reader Prerequisites

Discord is an application that, because of the way it was built, has both web app and desktop app aspects to it. I think this is one of the main issues people run into, so I'll give a brief rundown about why this matters for screen reader users.

Particularly on Windows, screen readers tend to operate in two modes when a web app is encountered: you're either in browse/virtual mode or you're in forms/focus mode.

Discord, being essentially a web app, gives you both of these modes as well. In "browse" mode, you have access to keys that navigate by heading, form field, button, and so on. In focus mode, you only have access to the keyboard shortcuts that work with the app. By default, these are tab/shift+tab, the arrow keys, space, enter, and escape.

Knowing when to use which mode is a bit of a mine field, but in general, a good rule of thumb is that you read/browse in browse mode, and you act/type in forms/focus mode.

Many web apps, Discord included, generally gracefully switch you between modes when required, but this isn't always the case. When this doesn't happen, you need to know how and when to switch, which I'll point out when required in the upcoming sections of this article.

The Broad Strokes: How to Navigate Efficiently

The tricky bit with learning an app like this is generally that documentation can be really scarce and hard to find. Discord probably has articles on this, but you'd need to know what to look for. And at the end of the day, all we want to do is use the app for what we're trying to do and move on with our lives. So here's the Cliff's notes.

Discord has a pretty rich list of keyboard shortcuts that allow you to do all sorts of things, but some of the keys aren't super obvious, and some of the navigation patterns are inconsistent.

In general, when navigating or skimming, you want to be in your screen reader's focus or forms mode. NVDA toggles this with NVDA+space, JAWS with JAWS+z. This allows Discord's own navigation keys to work correctly, which makes things go a little faster.

A key that works in a lot of applications on Windows is f6. This jumps you between specific regions of an application and is a bit of a hold-over from Ye olden Days of Windows 98 and XP. File explorer, office apps and browsers generally use this key in this way, but other apps like VS Code, Slack, and Discord do as well. It's a bit of a super power for keyboard-only navigation, as it gets you places a lot quicker.

In Discord, it bounces you between the server list, the message list, and a few other spots. But, oddly enough, it doesn't take you to the actual message entry field. To go there, the quickest way is to just start typing. This will zip your cursor to the right place.

With Discord having essentially its own keyboard navigation layer, I would generally recommend that you stay in focus/forms mode for most interactions, unless you need the conveniences of browse mode. More on that below.

When in focus mode, f6 moves you between the various regions of the screen, as I mentioned above. Tab will navigate you between those regions, as well as the interactable elements within those regions.

This, together with the arrow keys to navigate lists of channels, messages, and servers, will get you to most places you'll need to go for basic Discord usage.

Speeding Things Up: How to Navigate and Interact Efficiently

If you know where you're going, the best way to get there quickly is the ctrl+k or cmd+k hotkeys, depending on your operating system. This hotkey will bring up a search field where you can type part of a channel name, user name, or server name to search for it. A list of results will be focused after a brief pause, and then you can use the arrow keys to move to the correct result. Pressing enter takes you straight to the server, channel, or user you selected.

On Discord servers, it can be difficult to keep up with everything happening, given how many channels a lot of servers have. Here's a few tips to keep track of it all:

• Ctrl+i / cmd+i will open your so-called "inbox". This is where you can get a list of your mentions on the various servers you're in, which can be a great way to keep track of people trying to get your attention.

• There are a number of hotkeys that also let you cycle between various types of channels. For example, shift+alt+up and down will navigate between channels with unread messages. This is another quick way to get through a large amount of unread channels efficiently.

When is Browse Mode More Efficient?

I'd say there are two scenarios in which browse mode may be beneficial.

First, if you want to look at a message more granularly (for example to see how a word is spelled), dropping into browse mode will let you do that.

You can also use review cursors/object navigation shortcuts if you want, but browse mode tends to be easier in these cases. It also allows for more convenient copying of text if you need to do that.

To my knowledge, there's no quick way to jump to the earliest unread message in focus mode, which can be a bit annoying if you're scrolling back to see what messages you did and didn't read yet.

There is a "NEW" indicator above that message, which you can find with a reverse NVDA search, or by scrolling and listening out for it. Discord does have a hotkey (shift+page-up) to do this as well, but in my testing it doesn't always work as advertised.

Apart from that, barring a few niche edge cases, you can generally stay in focus mode unless you want to NVDA or JAWS search for button labels. This can be useful to quickly, say, find the disconnect button in a voice channel, which currently doesn't appear to have a hotkey associated with it.

Conclusion

Discord is a pretty complicated app, and this guide touches on the basics to get you started. With these general strategies, you should be able to deal with day-to-day messaging, channel management, and voice channels.

I always encourage anyone to just explore and work out your own strategies though. You can rarely break things irreparably, so see how things work for you personally and develop a way of working that works for you, specifically. This is always the best strategy for any kind of app, particularly when using assistive technology.

If you want to learn about all the other hotkeys Discord offers, you can find a keyboard reference on the Discord website. They offer both visual charts, which are inaccessible for screen reader users in this instance, or a readable table.

I hope this was helpful. Go forth and Discord!

Historically, accessibility has often been treated as an afterthought. Something to consider only if time permitted before a release.

Today, that perspective has fundamentally changed. With regulations like the European Accessibility Act (EAA) taking effect across European Union markets, building accessible software is increasingly a legal and business requirement.

More importantly, building accessible applications is just good engineering. Accessible apps improve usability for all users and help make sure that you're not excluding a substantial portion of your potential audience.

Jetpack Compose simplifies accessibility compared to the traditional Android View system. Because Compose is declarative and state-driven, it manages semantic metadata alongside your visual user interface.

In this comprehensive guide, you will learn how accessibility works in Jetpack Compose, how to use the Semantics tree, how to apply essential and advanced accessibility patterns, and how to thoroughly test your apps using automated tests, Google's Accessibility Scanner, Android Studio's Layout Inspector, and TalkBack.

Prerequisites

To follow along with this guide, you'll need:

• Basic familiarity with Kotlin and Jetpack Compose (such as Composables, Modifiers, and State).

• Android Studio (Hedgehog, Iguana, Jellyfish, Koala, Ladybug, or newer).

• An Android device or emulator running Android 9.0 (API 28) or higher with access to the Google Play Store (to install and run the current version of Google Accessibility Scanner).

Table of Contents

• Prerequisites

• Why Accessibility Matters in Modern Android Development

• Core Concepts of Android Accessibility

• Semantics in Jetpack Compose

• Essential Accessibility Practices

• Advanced Accessibility Techniques

• How to Test and Debug Accessibility

• Real-World Accessible UI Patterns

• Accessibility Audit Checklist

• Conclusion and Next Steps

• Essential Resources

Why Accessibility Matters in Modern Android Development

According to the World Health Organization (WHO), more than 1.3 billion people (representing roughly 16% of the global population) live with a significant disability. This includes people with permanent visual impairments, hearing loss, physical motor limitations, and cognitive or neurological differences.

In addition to permanent disabilities, users can often experience temporary or situational limitations. A person holding a baby with one hand has temporary motor constraints. Someone using their phone in harsh midday sunlight experiences situational visual impairment. And someone in a loud airport terminal experiences situational hearing limitations.

Designing an accessible app ensures that your interface remains resilient and usable across all of these scenarios and more.

Beyond empathy and inclusive design, accessibility is increasingly governed by international law.

In the European Union, the European Accessibility Act (EAA) mandates accessibility for a defined list of products and services, including smartphones and computers, e-commerce, consumer banking services, e-books, and passenger transport services. In practice, conformity is typically demonstrated against EN 301 549, the European accessibility standard for digital products and services.

In the United States, the Americans with Disabilities Act (ADA) applies to state and local governments (Title II, which, since 2024, has an explicit web and mobile app rule requiring compliance with WCAG 2.1 Level AA) and to public accommodations (Title III). Separately, Section 508 of the Rehabilitation Act requires federal agencies' information and communications technology to be accessible.

Failing to meet these standards can result in legal penalties and brand damage. Conversely, prioritizing accessibility expands your total addressable market and boosts user retention.

Quality also affects distribution: Google's Android vitals documentation notes that a high user-perceived crash rate hurts your app's discoverability on Google Play.

Core Concepts of Android Accessibility

To design accessible applications in Jetpack Compose, you must understand how the underlying Android operating system communicates with assistive technologies.

Understanding Android Accessibility Services

Android includes several built-in accessibility services that run as background processes. These services intercept the app's user interface and translate it into alternative sensory feedback or alternative input mechanisms:

• TalkBack (Screen Reader): TalkBack is Android's built-in screen reader designed for blind and low-vision users. It inspects your interface elements, converts visual content and semantic descriptions into synthesized speech and haptic vibrations, and enables non-visual navigation using linear swipe gestures.

• Switch Access: Designed for users with severe motor impairments who can't use a physical touch screen, Switch Access allows users to control their device using one or more physical switches (such as foot pedals, sip-and-puff devices, or single buttons) or a connected keyboard. It scans through focusable screen elements sequentially, allowing the user to select items when highlighted.

• Voice Access: Allows users to control their entire phone using spoken commands (such as "Tap Open," "Scroll down," or "Type hello"). It assigns numeric badges to interactive elements based on their accessibility labels so users can trigger actions by number.

• Select to Speak: Allows users to highlight specific paragraphs, buttons, or icons on the screen to hear them spoken aloud without turning on full TalkBack navigation.

All of these accessibility services share one common requirement: they don't interact directly with visual pixels. Instead, they read the Semantics Tree generated by your application.

How the Compose Semantics Tree Works

When you build a user interface in Jetpack Compose, Compose generates two separate but linked internal tree structures:

1. The Layout (UI) Tree: This tree contains the visual rendering nodes that measure, place, and draw pixels on the screen (such as Canvas, Box, Row, Column, Text, and Image).

2. The Semantics Tree: This tree runs in parallel with the Layout tree. It contains metadata that describes the meaning, purpose, state, and interactive capabilities of each element.

Standard Material Compose components (such as Button, Checkbox, Slider, Switch, and Text) automatically populate appropriate semantic properties into the Semantics tree. For example, a Button composable automatically sets its role to Role.Button and attaches a click action.

But when you build custom layouts, canvas drawings, or non-standard interactive widgets, Compose can't automatically infer your design intent. In those cases, you must use Compose's semantic modifiers to enrich the Semantics tree manually.

The WCAG 2.1 POUR Principles

The international benchmark for digital accessibility is the Web Content Accessibility Guidelines (WCAG) 2.1, published by the World Wide Web Consortium (W3C). These guidelines are organized around four core principles known by the acronym POUR:

• Perceivable: Information and UI components must be presentable to users in ways they can perceive. Content can't be invisible to all of a user's senses. In Android apps, this means providing descriptive text alternatives for all visual media, supporting dynamic font scaling, and maintaining high color contrast.

• Operable: User interface components and navigation must be operable. Users must be able to perform all interactions regardless of whether they use a touchscreen, a hardware keyboard, voice commands, or a switch device. This requires adequate touch targets (Android's guideline is at least 48dp × 48dp) and logical navigation orders.

• Understandable: Users must be able to understand both the information and the interface's operation. This means writing clear labels, providing meaningful validation error messages, avoiding sudden unexpected layout shifts, and structuring forms predictably.

• Robust: Content must be sufficiently robust to be reliably interpreted by a wide variety of user agents and assistive technologies. In Compose, this means avoiding hacky workarounds, using standard semantic roles, and honoring system-level user preferences.

Semantics in Jetpack Compose

In this section, we'll examine how Compose exposes and manipulates semantic metadata.

What Are Semantics?

Semantics in Jetpack Compose are key-value properties attached to layout nodes using the Modifier.semantics modifier. They convey information such as:

• What an element is called (contentDescription)

• What kind of UI control it represents (role = Role.Checkbox, Role.Button, Role.Tab)

• What state it currently holds (stateDescription = "Checked", progressBarRangeInfo)

• What actions the user can perform (onClick, onLongClick, customActions)

Basic Semantic Properties and Content Descriptions

The most widely used semantic property is contentDescription. It provides a localized textual representation of non-textual UI elements, such as icons, photographs, and vector graphics.

Here's how you provide content descriptions for visual components:

// A functional icon button that performs an action
IconButton(onClick = { /* Open camera */ }) {
    Icon(
        painter = painterResource(id = R.drawable.ic_camera),
        contentDescription = "Open camera"
    )
}

// A decorative background illustration
Image(
    painter = painterResource(id = R.drawable.decorative_pattern),
    contentDescription = null, // Informs TalkBack to skip this node completely
    modifier = Modifier.fillMaxWidth()
)

How It Works Under the Hood

When TalkBack navigates to the Icon, it reads the contentDescription aloud (something like "Open camera, button, double tap to activate"). TalkBack automatically appends the word "button" because IconButton exposes Role.Button. (Exact TalkBack phrasing varies by version and locale. The spoken examples throughout this guide are illustrative.)

For decorative elements (such as subtle background shapes, divider lines, or visual illustrations that accompany adjacent descriptive text), you should explicitly set contentDescription = null. Setting contentDescription = null instructs Compose to omit that element from the Semantics tree, preventing screen readers from stopping on meaningless visual noise.

Custom Semantics and State Descriptions

When an element changes its state dynamically (such as toggling between playing and paused, or expanded and collapsed), screen reader users need to be notified of the current state before they interact with it.

You can set stateDescription and role inside the Modifier.semantics block:

var isPlaying by remember { mutableStateOf(false) }

IconButton(
    onClick = { isPlaying = !isPlaying },
    modifier = Modifier.semantics {
        // Explicitly declare what state the media player is in
        stateDescription = if (isPlaying) "Playing audio" else "Audio paused"
        role = Role.Button
    }
) {
    Icon(
        imageVector = if (isPlaying) Icons.Default.Pause else Icons.Default.PlayArrow,
        contentDescription = if (isPlaying) "Pause" else "Play"
    )
}

How It Works Under the Hood

Without stateDescription, TalkBack only reads the action ("Play, button"). The user has to guess whether the audio is currently playing or stopped.

By adding stateDescription, TalkBack announces: "Playing audio, Pause, button, double-tap to toggle." This gives the user immediate confirmation of the current state followed by the action that activating the control will perform.

How to Merge Semantics with mergeDescendants

In complex layouts, multiple individual composables often combine to represent a single logical entity. For instance, a user profile row might contain an avatar image, a username, an online badge, and a timestamp.

By default, TalkBack will treat every child Text and Image node as an independent stop, forcing the user to swipe four or five times just to move past a single list item.

You can use Modifier.semantics(mergeDescendants = true) to combine all descendant nodes into a single, cohesive accessibility node:

// Bad: TalkBack focuses three separate times: "Star icon", "4.5", "Customer rating"
Row(modifier = Modifier.clickable { /* Navigate to reviews */ }) {
    Icon(
        imageVector = Icons.Default.Star,
        contentDescription = "Star icon"
    )
    Text(text = "4.5")
    Text(text = "Customer rating")
}

// Good: Merged into one single stop: "Customer rating: 4.5 out of 5 stars, button"
Row(
    modifier = Modifier
        .clickable(onClickLabel = "View all reviews") { /* Navigate to reviews */ }
        .semantics(mergeDescendants = true) {
            contentDescription = "Customer rating: 4.5 out of 5 stars"
        }
) {
    Icon(
        imageVector = Icons.Default.Star,
        contentDescription = null // Suppressed because parent provides full description
    )
    Text(text = "4.5")
    Text(text = "Customer rating")
}

How It Works Under the Hood

When mergeDescendants = true is set, Compose collapses the accessibility boundaries of all child composables. Instead of generating multiple stops in the Semantics tree, Compose exposes a single node to the accessibility framework. TalkBack focuses the entire Row as a single bounding box and reads the parent's contentDescription.

How to Clear Semantics with clearAndSetSemantics

In certain situations, a standard composable may include default semantic behaviors that interfere with your intended accessibility experience.

The Modifier.clearAndSetSemantics modifier clears all semantic properties that would otherwise be inherited from child composables or default implementations, allowing you to define a clean, custom semantic definition:

// A composite badge displaying notification count
Box(
    modifier = Modifier.clearAndSetSemantics {
        contentDescription = "3 unread messages"
        role = Role.Button
    }
) {
    Icon(
        imageVector = Icons.Default.Email,
        contentDescription = "Email icon" // Cleared and ignored
    )
    Text(text = "3") // Cleared and ignored
}

How It Works Under the Hood

Unlike Modifier.semantics, which adds or overrides specific keys, clearAndSetSemantics discards all existing keys generated by the composable subtree. In the example above, the separate "Email icon" and "3" text nodes are completely erased from the Semantics tree and replaced with the single announcement: "3 unread messages, button."

Essential Accessibility Practices

Now we'll cover the core accessibility practices you should implement in your applications, along with explanations of why each practice matters.

How to Write Meaningful Content Descriptions

A content description should concisely explain the purpose or action of an element rather than its visual appearance.

What to Do (Recommended Practice)

// Descriptive, action-oriented label
IconButton(onClick = { deleteDraft() }) {
    Icon(
        imageVector = Icons.Default.Delete,
        contentDescription = "Delete draft message"
    )
}

// Contextual weather information
Image(
    painter = painterResource(R.drawable.weather_sunny),
    contentDescription = "Current weather: Sunny, 75 degrees Fahrenheit"
)

The icon description tells the user exactly what will happen when they tap the button ("Delete draft message"). The weather image communicates the actual underlying data rather than describing the art style.

What Not to Do (Antipattern)

// Avoid generic or redundant descriptions
IconButton(onClick = { deleteDraft() }) {
    Icon(
        imageVector = Icons.Default.Delete,
        contentDescription = "Trash can icon button" // Bad: includes visual style and element type
    )
}

// Avoid empty strings on interactive elements
IconButton(onClick = { openSettings() }) {
    Icon(
        imageVector = Icons.Default.Settings,
        contentDescription = "" // Bad: leaves the button with no accessible label
    )
}

Including words like "icon" or "button" is redundant because TalkBack already announces the component's role. Using an empty string ("") on an interactive element leaves TalkBack with nothing meaningful to announce (typically read out as an unlabeled button), giving screen reader users zero context about what the button does. Compose's accessibility checks treat an empty content description the same as a missing one.

How to Ensure Minimum Touch Target Sizes

Google's Material Design and Android accessibility guidelines require all interactive elements to have a minimum touch target size of at least 48dp × 48dp, which corresponds to a physical size of about 9mm. This is squarely within the 7–10mm range Google recommends for touchscreen targets.

For comparison, WCAG itself sets lower web-oriented baselines: WCAG 2.1 Success Criterion 2.5.5 requires 44×44 CSS pixels at Level AAA, and WCAG 2.2 Success Criterion 2.5.8 requires 24×24 CSS pixels at Level AA. The 48dp rule is Android's stricter platform standard.

What to Do (Recommended Practice)

// Method A: Using minimumInteractiveComponentSize()
Box(
    modifier = Modifier
        .minimumInteractiveComponentSize() // Expands touch area to 48dp x 48dp
        .clickable { /* Toggle bookmark */ }
) {
    Icon(
        imageVector = Icons.Default.Bookmark,
        contentDescription = "Bookmark article",
        modifier = Modifier.size(24.dp) // Visual size is 24dp, touch target is 48dp
    )
}

// Method B: Standard IconButton (built-in 48dp target)
IconButton(onClick = { /* Toggle bookmark */ }) {
    Icon(
        imageVector = Icons.Default.Bookmark,
        contentDescription = "Bookmark article"
    )
}

Modifier.minimumInteractiveComponentSize() allows your visual design to remain compact (for instance, a 24dp icon) while ensuring that the invisible, clickable hit box expands to 48dp × 48dp. This prevents touch misses for users with motor tremors or limited fine motor control.

What Not to Do (Antipattern)

// Bad: Tightly constrained 20dp clickable area
Icon(
    imageVector = Icons.Default.Close,
    contentDescription = "Close dialog",
    modifier = Modifier
        .size(20.dp)
        .clickable { dismissDialog() } // Touch target is strictly 20dp x 20dp!
)

A 20dp touch target is nearly impossible to tap reliably, especially on high-density displays or while walking. Google's Accessibility Scanner will flag it as a touch target issue.

How to Maintain Accessible Color Contrast Ratios

Color contrast measures the difference in luminance between foreground text and its background. If the contrast ratio is too low, people with low vision, color blindness, or older eyes will likely not be able to read your text.

WCAG 2.1 defines the following minimum contrast ratios (Android's accessibility documentation maps WCAG's point sizes to sp):

• Normal text (smaller than 18sp, or smaller than 14sp bold): Minimum 4.5:1

• Large text (18sp or larger, or 14sp bold or larger): Minimum 3.0:1

• UI components and graphical objects: Minimum 3.0:1

What to Do (Recommended Practice)

// Using Material 3 color tokens (paired roles maintain usable contrast)
Text(
    text = "Account Overview",
    color = MaterialTheme.colorScheme.onPrimaryContainer,
    modifier = Modifier.background(MaterialTheme.colorScheme.primaryContainer)
)

// Explicit high-contrast colors (for example, Black on White = 21:1 ratio)
Text(
    text = "Order Confirmed",
    color = Color(0xFF1B5E20), // Dark green
    modifier = Modifier.background(Color(0xFFE8F5E9)) // Light green tint
)

Material 3 color tokens (such as onPrimaryContainer paired with primaryContainer) are generated by the Material color system so that "on" colors maintain usable contrast against their paired container colors across both light and dark themes. You should still verify contrast for any custom brand colors you override.

What Not to Do (Antipattern)

// Bad: Light gray on pure white (Contrast ratio ~ 1.6:1 - Fails WCAG)
Text(
    text = "Terms and Conditions apply",
    color = Color(0xFFB0B0B0),
    modifier = Modifier.background(Color.White)
)

Light gray text on a white background is one of the most common accessibility violations on the mobile web and in native apps. Sighted users in bright ambient light and users with low vision can't read this text.

How to Provide Custom Clickable Labels

When a user navigates to a clickable element using TalkBack, the screen reader appends default instructions such as "Double-tap to activate."

You can customize this instruction using the onClickLabel parameter in Modifier.clickable to describe the precise outcome of the interaction.

What to Do (Recommended Practice)

ListItem(
    headlineContent = { Text("Wi-Fi Networks") },
    supportingContent = { Text("Connected to Office_5G") },
    modifier = Modifier.clickable(
        onClickLabel = "Open Wi-Fi network settings"
    ) {
        navigateToWifiSettings()
    }
)

TalkBack will announce: "Wi-Fi Networks, Connected to Office_5G, double-tap to open Wi-Fi network settings." The user knows exactly what will happen before committing to the action.

What Not to Do (Antipattern)

// Bad: Generic clickable element without context
Card(
    modifier = Modifier.clickable { openInvoiceDetails() }
) {
    Text("Invoice #4092")
}

TalkBack announces: "Invoice #4092, double-tap to activate." The user is left uncertain whether tapping will pay the invoice, download a PDF, or open an edit screen.

How to Establish Heading Hierarchies

Visual readers scan large, bold text headings to understand a screen's layout and hierarchy. Screen reader users need the exact same structural overview.

By applying the heading() semantic modifier, you designate a text node as an accessibility heading. TalkBack users can change their navigation mode to "Headings" and quickly swipe up or down to jump between sections.

What to Do (Recommended Practice)

Column(modifier = Modifier.padding(16.dp)) {
    // Screen title heading
    Text(
        text = "Security Settings",
        style = MaterialTheme.typography.headlineMedium,
        modifier = Modifier.semantics { heading() }
    )

    Spacer(modifier = Modifier.height(16.dp))

    // Section 1 heading
    Text(
        text = "Two-Factor Authentication",
        style = MaterialTheme.typography.titleMedium,
        modifier = Modifier.semantics { heading() }
    )
    
    // Section 1 content...

    Spacer(modifier = Modifier.height(16.dp))

    // Section 2 heading
    Text(
        text = "Connected Devices",
        style = MaterialTheme.typography.titleMedium,
        modifier = Modifier.semantics { heading() }
    )
    
    // Section 2 content...
}

TalkBack users can bypass dozens of individual switches and descriptions to jump directly to "Connected Devices" in seconds.

What Not to Do (Antipattern)

// Bad: Visual heading without semantic markup
Text(
    text = "Two-Factor Authentication",
    style = MaterialTheme.typography.titleLarge // Visually large, but invisible as a heading to TalkBack
)

Without semantics { heading() }, TalkBack treats the text as regular body copy, preventing users from navigating by headings.

How to Announce Dynamic Updates with Live Regions

When content on the screen updates asynchronously (such as a timer countdown, a file upload completion message, or a real-time validation banner), sighted users see the change immediately.

But screen reader users won't know that anything changed unless you mark the changing container as a live region.

Compose provides liveRegion = LiveRegionMode.Polite and liveRegion = LiveRegionMode.Assertive:

• LiveRegionMode.Polite: TalkBack waits until the current audio announcement is finished before reading the update. Use this for almost all status updates.

• LiveRegionMode.Assertive: TalkBack announces the change immediately, ahead of other feedback. Reserve this strictly for time-sensitive, high-urgency alerts (such as emergency warnings).

What to Do (Recommended Practice)

var uploadStatus by remember { mutableStateOf("Ready to upload") }

Text(
    text = uploadStatus,
    modifier = Modifier.semantics {
        liveRegion = LiveRegionMode.Polite
    }
)

// Later, when upload finishes:
LaunchedEffect(Unit) {
    performUpload()
    uploadStatus = "Upload complete! 12 files saved."
}

As soon as uploadStatus changes, TalkBack automatically speaks: "Upload complete! 12 files saved," keeping non-sighted users fully informed without requiring them to search the screen.

How to Support Dynamic Text Scaling

Users with low vision often increase their system font size in Android Settings (under Display or the accessibility settings, depending on the device). Android 14 and newer support non-linear font scaling up to 200%.

To respect this user preference, you must define all text sizes in sp (scale-independent pixels), never in dp or raw pixels.

What to Do (Recommended Practice)

// Good: Using Material Typography (uses sp automatically)
Text(
    text = "Dashboard Summary",
    style = MaterialTheme.typography.titleMedium
)

// Good: Explicit sp sizing
Text(
    text = "Custom Label",
    fontSize = 18.sp
)

When a user sets their font scale to 1.5×, an 18sp font cleanly renders at 27sp, ensuring readable text.

What Not to Do (Antipattern)

// Bad: Fixed dp size converted to sp (does not scale with system settings)
Text(
    text = "Fixed Size Warning",
    fontSize = with(LocalDensity.current) { 16.dp.toSp() } // Will not scale!
)

Sizing text with fixed dp measurements prevents the text from enlarging when users increase their system font size, directly violating accessibility standards.

Advanced Accessibility Techniques

For complex, production-grade applications, you can apply advanced accessibility techniques to handle non-trivial user interactions.

How to Control Traversal and Focus Order

By default, accessibility services traverse screen elements based on their visual and structural coordinates (top-to-bottom, start-to-end). In multi-column layouts, financial ledgers, or custom grids, this default order can create confusing announcements.

You can customize the navigation sequence using traversalIndex and isTraversalGroup:

Column(
    modifier = Modifier.semantics { isTraversalGroup = true }
) {
    Text(
        text = "Step 1: Account Info",
        modifier = Modifier.semantics { traversalIndex = 1f }
    )
    
    Text(
        text = "Step 3: Confirmation",
        modifier = Modifier.semantics { traversalIndex = 3f }
    )
    
    Text(
        text = "Step 2: Payment Details",
        modifier = Modifier.semantics { traversalIndex = 2f }
    )
}

How the code works under the hood:

1. Setting isTraversalGroup = true creates a self-contained accessibility boundary, ensuring TalkBack reads all elements inside this container before moving to other screen elements.

2. The traversalIndex floating-point value establishes the exact relative order (lowest index first). TalkBack will read Step 1, then Step 2, and finally Step 3, regardless of their visual placement in the layout.

How to Create Custom Accessibility Actions

Consider an e-commerce product card that contains "Quick View," "Add to Wishlist," and "Add to Cart" buttons. For a screen reader user, tabbing through three separate buttons on twenty consecutive cards is tedious.

With customActions, you can attach secondary actions directly to the parent card node. When the card gains focus, TalkBack tells the user that actions are available, and the user opens the TalkBack menu to pick one.

var isBookmarked by remember { mutableStateOf(false) }

Card(
    modifier = Modifier
        .fillMaxWidth()
        .semantics {
            customActions = listOf(
                CustomAccessibilityAction(
                    label = if (isBookmarked) "Remove from bookmarks" else "Add to bookmarks",
                    action = {
                        isBookmarked = !isBookmarked
                        true // Return true to indicate the action was handled
                    }
                ),
                CustomAccessibilityAction(
                    label = "Share article link",
                    action = {
                        shareArticle()
                        true
                    }
                )
            )
        }
) {
    // Visual card content...
}

When a TalkBack user focuses on the card, TalkBack informs them that custom actions are available. The user opens the TalkBack menu (or uses accessibility gestures) to see a clean list containing "Add to bookmarks" and "Share article link." This keeps your visual design streamlined while providing direct, high-efficiency navigation for power users.

How to Design Accessible Form Inputs and Text Fields

Accessible form inputs require more than just a visual placeholder. They need explicit labels, clear input type hints, and logical keyboard navigation actions:

var emailValue by remember { mutableStateOf("") }

OutlinedTextField(
    value = emailValue,
    onValueChange = { emailValue = it },
    label = { Text("Work Email") },
    placeholder = { Text("alex@example.com") },
    singleLine = true,
    keyboardOptions = KeyboardOptions(
        keyboardType = KeyboardType.Email,
        imeAction = ImeAction.Next
    ),
    keyboardActions = KeyboardActions(
        onNext = { /* Move focus to password field */ }
    ),
    modifier = Modifier
        .fillMaxWidth()
        .semantics {
            contentDescription = "Work email address input field"
        }
)

How it works under the hood:

• The label composable provides persistent context that remains visible even after the user types.

• keyboardType = KeyboardType.Email instructs Android to display an email-optimized keyboard (including @ and .com shortcuts) and hints to accessibility services that standard email syntax is expected.

• imeAction = ImeAction.Next ensures hardware keyboard users and switch device users can smoothly advance through form fields using the keyboard Enter key.

How to Communicate Dynamic Error States

When client-side validation fails, setting the visual border to red isn't enough. You must attach semantic error metadata so screen readers announce the validation failure immediately.

var password by remember { mutableStateOf("") }
val isPasswordInvalid = password.isNotEmpty() && password.length < 8

OutlinedTextField(
    value = password,
    onValueChange = { password = it },
    label = { Text("Password") },
    isError = isPasswordInvalid,
    supportingText = {
        if (isPasswordInvalid) {
            Text(
                text = "Password must be at least 8 characters long",
                color = MaterialTheme.colorScheme.error
            )
        }
    },
    modifier = Modifier
        .fillMaxWidth()
        .semantics {
            if (isPasswordInvalid) {
                error("Password must be at least 8 characters long")
            }
        }
)

How it works under the hood:

The isError = isPasswordInvalid parameter handles the visual styling (red outline and error icons).

• The error("...") semantic property tells TalkBack to treat this element as invalid and announce the specific validation error message as soon as the user focuses the text field.

How to Manage Progress Indicators and Asynchronous Loading

Loading states must clearly convey whether a task is indeterminate (in progress with unknown duration) or determinate (progressing toward 100%).

// Indeterminate Progress (such as initial network fetch)
CircularProgressIndicator(
    modifier = Modifier.semantics {
        contentDescription = "Syncing messages, please wait"
    }
)

// Determinate Progress (such as file download)
val downloadProgress = 0.65f // 65%

LinearProgressIndicator(
    progress = { downloadProgress },
    modifier = Modifier.semantics {
        progressBarRangeInfo = ProgressBarRangeInfo(
            current = downloadProgress,
            range = 0f..1f
        )
        contentDescription = "Downloading update: ${(downloadProgress * 100).toInt()}% completed"
    }
)

For determinate progress indicators, ProgressBarRangeInfo tells assistive technologies the minimum, maximum, and current values. TalkBack interprets this information and provides periodic spoken and haptic updates as progress increases.

How to Handle Expandable and Collapsible Content

Accordion menus and expandable FAQ cards must clearly indicate whether they're open or closed, and what action tapping them will trigger.

var isExpanded by remember { mutableStateOf(false) }

Column(modifier = Modifier.fillMaxWidth()) {
    Row(
        verticalAlignment = Alignment.CenterVertically,
        modifier = Modifier
            .fillMaxWidth()
            .clickable(
                onClickLabel = if (isExpanded) "Collapse section" else "Expand section"
            ) {
                isExpanded = !isExpanded
            }
            .semantics {
                stateDescription = if (isExpanded) "Expanded" else "Collapsed"
            }
            .padding(16.dp)
    ) {
        Text(
            text = "Frequently Asked Questions",
            style = MaterialTheme.typography.titleMedium,
            modifier = Modifier.weight(1f)
        )
        Icon(
            imageVector = if (isExpanded) Icons.Default.ExpandLess else Icons.Default.ExpandMore,
            contentDescription = null // Decorative: parent Row communicates full state
        )
    }

    AnimatedVisibility(visible = isExpanded) {
        Text(
            text = "Our refund policy allows returns within 30 days of purchase...",
            modifier = Modifier.padding(16.dp)
        )
    }
}

The stateDescription announces whether the section is currently open or closed ("Expanded" or "Collapsed"). Simultaneously, onClickLabel clarifies the upcoming action ("Collapse section" or "Expand section"). The icon has contentDescription = null to avoid redundant announcements.

How to Enhance Lazy Lists and Large Collections

When users navigate a long feed or list, they need context regarding where they are and how many items exist.

val messages = remember { listOf("Order Shipped", "Delivery Delayed", "Payment Received") }

LazyColumn(
    modifier = Modifier.semantics {
        contentDescription = "Notifications list, ${messages.size} total alerts"
    }
) {
    itemsIndexed(messages) { index, messageText ->
        Card(
            modifier = Modifier
                .fillMaxWidth()
                .padding(vertical = 4.dp)
                .semantics {
                    contentDescription = "Notification ${index + 1} of ${messages.size}: $messageText"
                }
        ) {
            Text(
                text = messageText,
                modifier = Modifier.padding(16.dp)
            )
        }
    }
}

TalkBack announces positional indexes ("Notification 1 of 3: Order Shipped"), allowing screen reader users to track their progress through lists without losing their place.

How to Test and Debug Accessibility

Building accessible software requires a multi-layered testing strategy: automated regression checks, on-device diagnostic tools, semantics tree inspection, and manual verification with TalkBack.

How to Test Manually with TalkBack

Nothing replaces manually navigating your application using TalkBack.

How to Enable and Use TalkBack

1. Open your device's Settings app and navigate to Accessibility and then TalkBack.

2. Toggle the switch to On and accept the system permissions. (You can also use the volume key shortcut, if enabled: hold both volume keys for a few seconds to toggle TalkBack).

3. Core TalkBack gestures:

   • Swipe Right: Move accessibility focus to the next element.

   • Swipe Left: Move accessibility focus to the previous element.

   • Double Tap: Activate the currently focused element.

   • Two-Finger Swipe: Scroll lists or pages.

   • Three-Finger Tap (or swipe down then right in one motion): Open the TalkBack menu.

What to check during your TalkBack walkthrough:

• Can you complete critical user journeys (such as registration, login, searching, and checkout) with your eyes closed?

• Are all buttons and interactive controls clearly announced with meaningful names?

• Does focus move in a logical, expected reading order?

• Are error messages and dynamic state changes announced automatically?

How to Write Automated Compose Accessibility Tests

You can integrate automated accessibility assertions into your JUnit instrumented tests to catch missing descriptions and undersized touch targets on continuous integration (CI) servers.

Add Dependencies to build.gradle.kts:

androidTestImplementation("androidx.compose.ui:ui-test-junit4")
androidTestImplementation("androidx.compose.ui:ui-test-manifest")

Writing Compose Accessibility Tests:

@RunWith(AndroidJUnit4::class)
class AccessibilityTest {

    @get:Rule
    val composeTestRule = createAndroidComposeRule<ComponentActivity>()

    @Test
    fun testLoginButton_hasProperTouchTargetAndLabel() {
        composeTestRule.setContent {
            MaterialTheme {
                Button(
                    onClick = { /* Submit login */ },
                    modifier = Modifier.minimumInteractiveComponentSize()
                ) {
                    Text("Sign In")
                }
            }
        }

        // Verify the node exists with correct text and semantics
        composeTestRule
            .onNodeWithText("Sign In")
            .assertExists()
            .assertHasClickAction()
            .assertHeightIsAtLeast(48.dp)
            .assertWidthIsAtLeast(48.dp)
    }

    @Test
    fun testIconButton_containsContentDescription() {
        composeTestRule.setContent {
            MaterialTheme {
                IconButton(onClick = {}) {
                    Icon(
                        imageVector = Icons.Default.Favorite,
                        contentDescription = "Add to favorites"
                    )
                }
            }
        }

        // Verify content description is accurately exposed in semantics tree
        composeTestRule
            .onNode(hasContentDescription("Add to favorites"))
            .assertExists()
            .assertHasClickAction()
    }
}

Step-by-Step Guide to Google Accessibility Scanner

Google Accessibility Scanner is an Android diagnostic tool that inspects your app's rendered UI and flags accessibility violations.

How to Install and Set Up Accessibility Scanner

1. Install from Google Play: Open the Google Play Store on your test device and install Accessibility Scanner (published by Google LLC).

2. Enable in Accessibility Settings:

   • Go to Settings then Accessibility and then Accessibility Scanner.

   • Toggle the switch to On and grant the required screen-reading permissions.

3. Locate the Floating Button: A blue floating action button with a checkmark icon (✓) will appear overlaid on your screen.

Running a Scan and Interpreting Results

Open your Android application and navigate to the screen you want to audit. Tap the floating blue Scanner button.

Then tap the Snapshot (camera) icon to analyze the static screen, or tap Record to audit a multi-step user flow.

The Scanner outlines each flagged UI component with an orange rectangle. Typical findings include interactive touch targets smaller than 48dp, missing labels on buttons and images, insufficient text or image contrast, and duplicate or redundant descriptions.

Tap any highlighted result to view a detailed breakdown explaining the problem, a link to the relevant accessibility guidance, and suggested remediation steps. Keep in mind that Scanner is a diagnostic aid. A clean scan doesn't guarantee that your app is fully accessible.

Figure: Google Accessibility Scanner auditing the AccessibilityDemo app, flagging an undersized 24dp touch target and displaying remediation guidance.

How to Inspect Semantics with Android Studio Layout Inspector

Android Studio's Layout Inspector allows you to inspect your running Compose hierarchy in real time and view the exact Semantics Tree exposed to the operating system.

How to Inspect Semantics Step-by-Step

1. Run your Compose application on an emulator or physical device connected via USB debugging.

2. In Android Studio, open View then Tool Windows and then Layout Inspector.

3. In the process selector dropdown, select your application's process (such as com.example.accessibilitydemo).

4. In the Layout Inspector component tree panel on the left, navigate to the composable you want to inspect (such as the Card under Semantic Merging).

5. Look at the Attributes panel on the right. You'll see dedicated sections for:

   • Merged Semantics: Displays the collapsed semantic metadata exposed to accessibility services when mergeDescendants = true is used (including merged ContentDescription, Text lists, OnClick actions, and container flags).

   • Declared Semantics: Shows the explicit semantic properties directly attached to that specific composable node.

Figure: Android Studio Layout Inspector inspecting the AccessibilityDemo app, displaying the Component Tree on the left, visual wireframes in the center, and the Merged Semantics attributes panel on the right.

Using Layout Inspector provides visual confirmation that:

• semantics(mergeDescendants = true) is properly grouping disparate children into a single node.

• Decorative icons are successfully excluded from the accessibility tree.

• Click actions (OnClick: AccessibilityAction) and custom actions are correctly wired to the composable.

Real-World Accessible UI Patterns

Here are complete, production-ready implementations of common UI patterns:

Fully Accessible Login Form

Forms are critical entry points. This implementation combines heading semantics, email input validation, dynamic error announcements, and proper touch target sizing:

@Composable
fun AccessibleLoginForm(
    onLoginSubmitted: (String, String) -> Unit
) {
    var email by remember { mutableStateOf("") }
    var password by remember { mutableStateOf("") }
    var isSubmitted by remember { mutableStateOf(false) }

    val isEmailInvalid = isSubmitted && !android.util.Patterns.EMAIL_ADDRESS.matcher(email).matches()
    val isPasswordInvalid = isSubmitted && password.length < 8

    Column(
        modifier = Modifier
            .fillMaxSize()
            .padding(24.dp)
    ) {
        // 1. Designated Screen Title Heading
        Text(
            text = "Welcome Back",
            style = MaterialTheme.typography.headlineLarge,
            modifier = Modifier.semantics { heading() }
        )

        Text(
            text = "Sign in to access your account",
            style = MaterialTheme.typography.bodyMedium,
            color = MaterialTheme.colorScheme.onSurfaceVariant
        )

        Spacer(modifier = Modifier.height(24.dp))

        // 2. Accessible Email Input
        OutlinedTextField(
            value = email,
            onValueChange = { email = it },
            label = { Text("Email Address") },
            isError = isEmailInvalid,
            singleLine = true,
            keyboardOptions = KeyboardOptions(
                keyboardType = KeyboardType.Email,
                imeAction = ImeAction.Next
            ),
            supportingText = {
                if (isEmailInvalid) {
                    Text(
                        text = "Please enter a valid email address",
                        color = MaterialTheme.colorScheme.error
                    )
                }
            },
            modifier = Modifier
                .fillMaxWidth()
                .semantics {
                    if (isEmailInvalid) {
                        error("Please enter a valid email address")
                    }
                }
        )

        Spacer(modifier = Modifier.height(16.dp))

        // 3. Accessible Password Input
        OutlinedTextField(
            value = password,
            onValueChange = { password = it },
            label = { Text("Password") },
            isError = isPasswordInvalid,
            singleLine = true,
            visualTransformation = PasswordVisualTransformation(),
            keyboardOptions = KeyboardOptions(
                keyboardType = KeyboardType.Password,
                imeAction = ImeAction.Done
            ),
            supportingText = {
                if (isPasswordInvalid) {
                    Text(
                        text = "Password must be at least 8 characters long",
                        color = MaterialTheme.colorScheme.error
                    )
                }
            },
            modifier = Modifier
                .fillMaxWidth()
                .semantics {
                    if (isPasswordInvalid) {
                        error("Password must be at least 8 characters long")
                    }
                }
        )

        Spacer(modifier = Modifier.height(24.dp))

        // 4. Accessible Submit Button
        Button(
            onClick = {
                isSubmitted = true
                if (!isEmailInvalid && !isPasswordInvalid) {
                    onLoginSubmitted(email, password)
                }
            },
            modifier = Modifier
                .fillMaxWidth()
                .minimumInteractiveComponentSize()
                .semantics {
                    contentDescription = "Sign in to your account"
                }
        ) {
            Text("Sign In")
        }
    }
}

Why this pattern works:

1. Screen Heading: TalkBack users can instantly jump to "Welcome Back" when entering the screen.

2. Error Semantics: If validation fails, error("...") ensures TalkBack immediately speaks the validation requirement when the text field receives focus.

3. Keyboard Routing: ImeAction.Next and ImeAction.Done guide keyboard and switch users smoothly between input fields.

E-Commerce Product Card with Custom Actions

This pattern demonstrates how to combine mergeDescendants = true with customActions to create a concise, power-user-friendly card:

data class Product(
    val id: String,
    val title: String,
    val priceFormatted: String,
    val rating: Float,
    val imageRes: Int
)

@Composable
fun AccessibleProductCard(
    product: Product,
    onCardClick: () -> Unit,
    onToggleFavorite: () -> Unit,
    onAddToCart: () -> Unit
) {
    Card(
        modifier = Modifier
            .fillMaxWidth()
            .clickable(onClickLabel = "View product details") { onCardClick() }
            .semantics(mergeDescendants = true) {
                // Group all descriptive properties into one coherent announcement
                contentDescription = "${product.title}, Price: ${product.priceFormatted}, Rated ${product.rating} out of 5 stars"
                
                // Expose secondary operations as custom accessibility actions
                customActions = listOf(
                    CustomAccessibilityAction("Add to shopping cart") {
                        onAddToCart()
                        true
                    },
                    CustomAccessibilityAction("Save to favorites") {
                        onToggleFavorite()
                        true
                    }
                )
            }
    ) {
        Row(
            modifier = Modifier.padding(16.dp),
            verticalAlignment = Alignment.CenterVertically
        ) {
            Image(
                painter = painterResource(product.imageRes),
                contentDescription = null, // Decorative: covered by merged description
                modifier = Modifier
                    .size(80.dp)
                    .clip(RoundedCornerShape(8.dp))
            )

            Spacer(modifier = Modifier.width(16.dp))

            Column(modifier = Modifier.weight(1f)) {
                Text(
                    text = product.title,
                    style = MaterialTheme.typography.titleMedium
                )
                Text(
                    text = product.priceFormatted,
                    style = MaterialTheme.typography.bodyLarge,
                    fontWeight = FontWeight.Bold
                )
                Text(
                    text = "★ ${product.rating}",
                    style = MaterialTheme.typography.bodySmall,
                    color = MaterialTheme.colorScheme.onSurfaceVariant
                )
            }
        }
    }
}

Why this pattern works:

1. Single Semantic Stop: Instead of swiping four times through separate image and text views, TalkBack reads the entire card in one unified statement.

2. Action Shortcuts: Users can add items to their cart or favorites directly from the TalkBack actions menu without navigating into the details page.

Tab Navigation with Selection State

This pattern demonstrates accessible tab bar navigation using stateDescription and custom tab announcements:

@Composable
fun AccessibleTabNavigation(
    tabs: List<String>,
    selectedTabIndex: Int,
    onTabSelected: (Int) -> Unit
) {
    TabRow(
        selectedTabIndex = selectedTabIndex,
        modifier = Modifier.semantics {
            contentDescription = "Navigation tabs, ${tabs[selectedTabIndex]} selected"
        }
    ) {
        tabs.forEachIndexed { index, title ->
            val isSelected = selectedTabIndex == index
            Tab(
                selected = isSelected,
                onClick = { onTabSelected(index) },
                modifier = Modifier.semantics {
                    role = Role.Tab
                    stateDescription = if (isSelected) "Selected" else "Not selected"
                    contentDescription = "$title tab"
                }
            ) {
                Text(
                    text = title,
                    modifier = Modifier.padding(vertical = 16.dp)
                )
            }
        }
    }
}

Why this pattern works:

1. Explicit Role: Marking each item with Role.Tab informs accessibility services that this is a selectable tab container.

2. Selection Feedback: The stateDescription tells the user whether a tab is currently active before they tap it.

Confirmation Dialog with Action Descriptions

This pattern shows how to structure an accessible confirmation dialog:

@Composable
fun AccessibleDeleteConfirmationDialog(
    itemName: String,
    onDismissRequest: () -> Unit,
    onConfirmDelete: () -> Unit
) {
    AlertDialog(
        onDismissRequest = onDismissRequest,
        title = {
            Text(
                text = "Delete Item?",
                style = MaterialTheme.typography.headlineSmall,
                modifier = Modifier.semantics { heading() }
            )
        },
        text = {
            Text("Are you sure you want to permanently delete \"$itemName\"? This action cannot be undone.")
        },
        confirmButton = {
            TextButton(
                onClick = onConfirmDelete,
                modifier = Modifier.semantics {
                    contentDescription = "Confirm deletion of $itemName"
                }
            ) {
                Text("Delete", color = MaterialTheme.colorScheme.error)
            }
        },
        dismissButton = {
            TextButton(
                onClick = onDismissRequest,
                modifier = Modifier.semantics {
                    contentDescription = "Cancel deletion and close dialog"
                }
            ) {
                Text("Cancel")
            }
        },
        modifier = Modifier.semantics {
            contentDescription = "Delete item confirmation dialog"
        }
    )
}

Why this pattern works:

1. Clear Focus: When the dialog appears, TalkBack automatically traps focus within the dialog bounds so users can't accidentally click background elements.

2. Context-Rich Buttons: Button descriptions explain the exact consequence of clicking ("Confirm deletion of Shopping List" rather than just "Delete").

Accessibility Audit Checklist

Before releasing your app to production or submitting it to app stores, run through this accessibility audit checklist:

Visual and Typography

• Scalable Text: Define all font sizes in sp (never fixed dp) so text scales cleanly up to 200%.

• Color Contrast: Maintain at least a 4.5:1 contrast ratio for normal text and 3.0:1 for large text and essential UI components against their backgrounds.

• Color Independence: Never convey information through color alone. Always pair color cues with text labels or distinct icons.

Interactive Elements and Touch Targets

• Touch Target Sizing: Ensure every clickable or interactive component has a minimum hit target of 48dp × 48dp using Modifier.minimumInteractiveComponentSize().

• Actionable Descriptions: Provide descriptive, action-oriented contentDescription strings on all functional icon buttons and interactive images.

• Decorative Elements: Set contentDescription = null on purely decorative icons and illustrations to keep TalkBack feedback concise.

• Click Context: Provide custom onClickLabel parameters on cards, list items, and custom buttons to clarify the outcome before activation.

Screen Structure and Navigation

• Accessibility Headings: Mark major section titles and screen headers with Modifier.semantics { heading() } for rapid heading navigation.

• Semantic Merging: Group related visual sub-elements (such as ratings or multi-text card headers) using Modifier.semantics(mergeDescendants = true).

• Live Announcements: Designate asynchronous UI updates with liveRegion = LiveRegionMode.Polite so TalkBack announces background changes.

• Form Optimization: Specify appropriate keyboardType and imeAction configurations on all text inputs.

• Validation Feedback: Declare active input error states using Modifier.semantics { error("...") }.

Testing and Verification

• Automated Unit Tests: Add Compose accessibility assertions in continuous integration to catch touch target and missing label regressions.

• On-Device Diagnostic Audit: Run Google Accessibility Scanner across all key app screens to catch contrast or sizing defects.

• Semantics Tree Inspection: Use Android Studio Layout Inspector to verify merged semantics and declared accessibility actions.

• Manual Screen Reader Walkthrough: Complete end-to-end user journeys (login, checkout, navigation) with TalkBack enabled.

Conclusion and Next Steps

Building accessible applications in Jetpack Compose doesn't mean just retrofitting code at the end of a project. You need to understand Compose's dual-tree architecture and design your user interface so that both visual pixels and semantic metadata accurately represent your application's intent.

By applying semantic properties, ensuring 48dp touch targets, maintaining WCAG contrast ratios, and verifying your work with TalkBack and Google Accessibility Scanner, you ensure that your Android applications are welcoming, compliant, and intuitive for all users worldwide.

Essential Resources

To deepen your understanding of Android accessibility, consult these essential references:

• Android Developers Official Guide: Accessibility in Jetpack Compose

• Android Developers Official Guide: Semantics in Compose

• W3C Web Content Accessibility Guidelines (WCAG) 2.1 Quick Reference

• Google Material Design 3: Accessible Design Foundations

A form might use colour as the only indication that something went wrong. A sticky header might completely cover the element that currently has keyboard focus. A login form might prevent users from pasting a password from their password manager. Or a custom button might work when clicked with a mouse but do nothing when someone uses a keyboard.

These are development decisions, not problems that only appear during an accessibility audit.

The Web Content Accessibility Guidelines (WCAG) provide a common standard for identifying and reducing many of these barriers. WCAG 2.2 is the latest WCAG 2 Recommendation, and the World Wide Web Consortium (W3C) advises developers and organisations to use WCAG 2.2 whenever possible.

This article focuses on the WCAG 2.2 Level A and AA requirements that frequently affect frontend development. The aim is to show how accessibility requirements connect to frontend development decisions.

Table of Contents

• What Is WCAG 2.2?

• How WCAG Conformance Works

• How the Four WCAG Principles Work

• How to Start with Semantic HTML

• How to Write Useful Text Alternatives for Images

• How to Handle Colour, Contrast, Text Resizing, and Reflow

• How to Make an Interface Work with a Keyboard

• How to Keep Keyboard Focus Visible

• How to Design Pointer Targets and Dragging Interactions

• How to Build More Accessible Forms

• How to Avoid Redundant Entry

• How to Keep Help Consistent

• How WCAG 2.2 Affects Authentication

• How to Use ARIA Without Replacing HTML

• How to Make Dynamic Status Messages Accessible

• How to Test Your Website for Accessibility

• Conclusion

What Is WCAG 2.2?

WCAG stands for Web Content Accessibility Guidelines. W3C develops the standard to describe how web content can be made more accessible to people with disabilities.

WCAG 2.2 organises its requirements into principles, guidelines, and testable success criteria. The success criteria are technology-independent, which is important because WCAG doesn't exist specifically for HTML, React, WordPress, or any other implementation technology.

Consider this hierarchy:

Principle: Operable

    Guideline 2.1: Keyboard Accessible

        Success Criterion 2.1.1: Keyboard

The principle gives you the broad accessibility objective. The guideline narrows that objective, while the success criterion provides the testable requirement.

W3C also publishes resources such as Understanding WCAG 2.2, How to Meet WCAG 2.2, and Techniques for WCAG 2.2. These resources explain the success criteria and provide implementation approaches, examples, and known failures. They're informative rather than part of the normative WCAG requirements.

A W3C technique can show one recognised way to satisfy a criterion, but WCAG generally doesn't require you to use that exact technique. Another implementation can also be valid if it meets the actual success criterion.

How WCAG Conformance Works

WCAG defines three conformance levels: A, AA, and AAA.

The levels build on one another. A page can't claim Level AA conformance by satisfying only the criteria labelled AA. It must satisfy all applicable Level A and Level AA success criteria. Level AAA similarly includes A, AA, and AAA requirements.

This distinction is important because accessibility discussions sometimes reduce WCAG to individual checks.

You might fix the keyboard interaction on a menu, add alternatives to your images, and correct several contrast problems. Those are useful accessibility improvements, but they don't automatically make the entire website "WCAG AA compliant".

WCAG conformance applies to complete web pages. When a process requires several pages to complete, such as a checkout process, all pages in that process must conform at the claimed level.

The examples in this article therefore demonstrate ways to address particular accessibility requirements. They don't constitute a conformance claim for an entire application.

How the Four WCAG Principles Work

WCAG groups its guidelines under four principles commonly remembered with the acronym POUR: Perceivable, Operable, Understandable, and Robust.

Perceivable means users need to be able to perceive the information you provide. Text alternatives, captions, contrast, and adaptable layouts fall under this principle.

Operable concerns how people interact with the interface. Keyboard operation, focus behaviour, navigation, pointer interactions, and timing are examples.

Understandable deals with whether users can understand the information and the way the interface behaves. Form instructions, useful error messages, predictable interfaces, and accessible authentication are relevant here.

Robust concerns whether browsers and assistive technologies can correctly interpret the content. Semantic HTML, accessible names, roles, values, and states are central to this principle.

These categories are useful, but accessibility problems rarely respect the boundary between HTML, CSS, and JavaScript.

A custom dropdown, for example, might need semantic information in the markup, visible focus styling in CSS, and correct keyboard behaviour in JavaScript.

Accessibility therefore works best when it forms part of the implementation itself rather than becoming a separate task at the end of development.

How to Start with Semantic HTML

One of the most useful accessibility decisions happens before you write any ARIA: choosing the correct HTML element.

Consider this:

<div onclick="submitForm()">Submit</div>

A mouse user may be able to click the element, but a div doesn't automatically behave like a button.

Compare it with this:

<button type="submit">Submit</button>

The native button already communicates its role to the browser and provides the expected keyboard behaviour.

This relates to Success Criterion 4.1.2 Name, Role, Value, which requires user interface components to expose information such as their name and role programmatically. W3C notes that standard controls already provide much of this information when developers use them according to their specification.

The practical implication is simple: don't recreate browser behaviour unless you need to.

How Semantic HTML Communicates Page Structure

Semantic HTML also helps expose relationships between parts of a page.

You could build a page like this:

<div class="top">
  ...
</div>

<div class="navigation">
  ...
</div>

<div class="content">
  <div class="title">Account Settings</div>
  ...
</div>

The classes may create the visual layout you want, but they don't necessarily communicate the same structure programmatically.

A more meaningful structure could be:

<header>
  ...
</header>

<nav aria-label="Primary">
  ...
</nav>

<main id="main-content">
  <h1>Account Settings</h1>
  ...
</main>

Success Criterion 1.3.1 Info and Relationships requires structure and relationships communicated visually to also be programmatically determinable or available in text. Semantic markup can provide this information without requiring developers to recreate it with additional accessibility attributes.

This doesn't mean that using <main>, <nav>, and <h1> automatically makes a page accessible. It means you're giving the browser more accurate information about what the content represents.

How to Add a Skip Link

Repeated page navigation creates another issue.

If a page has a large navigation menu, a keyboard user may otherwise need to move through those links every time before reaching the main content.

A skip link provides another path:

<a class="skip-link" href="#main-content">
  Skip to main content
</a>

<header>
  ...
</header>

<nav aria-label="Primary">
  ...
</nav>

<main id="main-content">
  ...
</main>

You can position the link outside the normal view until it receives keyboard focus:

.skip-link {
  position: absolute;
  top: -4rem;
  left: 1rem;
}

.skip-link:focus {
  top: 1rem;
}

This is one recognised way to support Success Criterion 2.4.1 Bypass Blocks, which requires a mechanism for bypassing blocks of repeated content. WCAG requires the outcome rather than this exact CSS implementation.

The broader principle is worth keeping: use HTML's existing semantics before adding custom semantics yourself.

How to Write Useful Text Alternatives for Images

Adding alt text is one of the best-known accessibility practices, but the rule is often oversimplified.

Success Criterion 1.1.1 Non-text Content requires non-text content to have a text alternative that serves an equivalent purpose, subject to several exceptions. Decorative content, for example, should be implemented so assistive technologies can ignore it.

The important word is purpose.

Consider this image:

<img src="revenue-chart.png" alt="Chart">

The alternative tells the user that the page contains a chart. It doesn't communicate anything the chart actually tells a sighted user.

If the main message is the change in revenue, an alternative could be:

<img
  src="revenue-chart.png"
  alt="Revenue increased from £1.2 million in 2024 to
       £1.8 million in 2025."
>

That doesn't mean every chart can be reduced to one sentence.

If the chart contains several data series or detailed values that readers need, you may also need a nearby explanation, accessible table, or another way of communicating the underlying information.

The alternative should reflect what the image contributes in its context.

How to Handle Decorative Images

A decorative image serves a different purpose.

Consider a visual divider:

<img src="decorative-line.svg" alt="">

The empty alt value indicates that the image doesn't contribute information that needs to be announced.

A missing alt attribute and alt="" are therefore not interchangeable. The empty alternative is an intentional decision.

How to Handle Icons Inside Controls

Now consider a search button containing a magnifying-glass SVG.

The relevant information isn't that the user is looking at a magnifying glass. The important information is that the control starts a search.

<button type="submit" aria-label="Search">
  <svg aria-hidden="true" viewBox="0 0 24 24">
    <path d="M10 4a6 6 0 1 0 0 12a6 6 0 0 0 0-12Z"></path>
    <path d="m14.5 14.5 5 5"></path>
  </svg>
</button>

The button receives the accessible name Search, while the SVG itself doesn't add duplicate information.

When deciding what alternative to provide, ask a more useful question than "What does this image look like?"

Ask: What information or function would the user lose if they couldn't perceive this image visually? That distinction matters when implementing accessibility.

How to Handle Colour, Contrast, Text Resizing, and Reflow

Accessibility also affects ordinary CSS decisions.

A layout may look correct at your preferred viewport size and still become difficult to use when somebody changes the way content is displayed.

How to Avoid Relying Only on Colour

Imagine a form that changes an input border from grey to red when validation fails:

.input {
  border: 1px solid #777;
}

.input.error {
  border-color: red;
}

The colour communicates that something changed, but a user needs to perceive that colour difference to understand the state.

Success Criterion 1.4.1 Use of Color requires colour not to be the only visual means used to convey information, indicate an action, prompt a response, or distinguish a visual element.

An improved implementation can combine styling with actual text:

<label for="email">Email address</label>

<input
  id="email"
  name="email"
  type="email"
  aria-invalid="true"
  aria-describedby="email-error"
>

<p id="email-error">
  Enter an email address in the format name@example.com.
</p>

You can still use a red border. It just shouldn't carry the message alone.

How to Check Text Contrast

Success Criterion 1.4.3 Contrast (Minimum) requires regular text to have a contrast ratio of at least 4.5:1. Qualifying large-scale text has a minimum ratio of 3:1, subject to the criterion's exceptions.

Don't judge contrast only by looking at the colours. Two colours can look sufficiently different on your display while still falling below the required ratio. Use a contrast-testing tool as part of your design and development process.

How Non-Text Contrast Differs from Text Contrast

Success Criterion 1.4.11 Non-text Contrast deals with visual information needed to identify interface components, states, and meaningful graphical objects. The required ratio is generally 3:1 against adjacent colours, subject to the criterion's scope and exceptions.

This can affect things such as custom form controls, meaningful icons, component boundaries, selected states, and graphical information.

Passing the text contrast requirement therefore doesn't automatically mean the rest of the interface has sufficient contrast.

How to Support Text resizing

Success Criterion 1.4.4 Resize Text requires text, with specified exceptions, to be resizable up to 200% without loss of content or functionality.

Fixed dimensions often expose problems here.

Consider:

.card {
  height: 180px;
  overflow: hidden;
}

If text grows beyond the space the developer assumed it would need, some content can disappear.

Where the design doesn't genuinely require a fixed height, allowing the component to grow is safer:

.card {
  min-height: 180px;
}

This doesn't prove that the component passes the criterion. You still need to resize the text and inspect the result.

The CSS simply removes one common source of failure.

How to Design for Reflow

Success Criterion 1.4.10 Reflow addresses the ability to use content at narrow equivalent dimensions without losing information or functionality or requiring prohibited two-dimensional scrolling.

For vertically scrolling content, the criterion uses a width equivalent to 320 CSS pixels. Certain content, such as some maps and data tables, may genuinely require two-dimensional layout and falls under the criterion's exceptions.

A flexible layout can help ordinary content adapt:

.settings-grid {
  display: grid;
  grid-template-columns:
    repeat(auto-fit, minmax(min(100%, 18rem), 1fr));
  gap: 1rem;
}

As space decreases, the cards move onto new rows rather than forcing the entire page to remain wide.

Responsive design helps here, but "responsive" and "accessible" aren't synonyms.

A responsive page can still hide controls, clip text, overlap content, or remove functionality at high zoom. Test the behaviour rather than assuming a media query solves the accessibility requirement.

How to Make an Interface Work with a Keyboard

One of the simplest manual accessibility tests is to put the mouse aside and use the application with a keyboard.

Success Criterion 2.1.1 Keyboard requires functionality to be operable through a keyboard interface except where the underlying function genuinely depends on the path of the user's movement. WCAG doesn't prevent the interface from also supporting mouse, touch, voice, or other forms of input.

Let's return to our custom control:

<div onclick="saveSettings()">Save</div>

Making the div look like a button doesn't give it button behaviour.

You could begin rebuilding that behaviour yourself:

<div
  role="button"
  tabindex="0"
>
  Save
</div>

But now your JavaScript must also provide the appropriate keyboard interaction.

In most cases, this is unnecessary:

<button type="button">
  Save
</button>

Native controls reduce the amount of interaction behaviour you need to reproduce.

W3C's ARIA Authoring Practices Guide makes this distinction explicit: ARIA roles don't cause browsers to add the keyboard behaviour that comes with native HTML controls. If you create a custom ARIA widget, you're responsible for implementing those interactions.

How to Check for Keyboard Traps

Success Criterion 2.1.2 No Keyboard Trap addresses situations where keyboard focus enters a component but can't leave through a keyboard interface.

This is particularly relevant to custom editors, dialogs, embedded widgets, and other complex controls.

Keyboard testing should go beyond asking whether you can press Tab until an element receives focus.

Try to complete the actual task. If you open a dialog, can you use its controls and close it? If you enter a custom widget, can you leave it? If a menu opens, can you operate it using its expected keyboard pattern?

Keyboard accessibility concerns the whole interaction, not simply whether an element appears in the tab order.

How to Keep Keyboard Focus Visible

Keyboard navigation becomes difficult when users can't tell which element currently has focus.

This CSS is therefore risky:

*:focus {
  outline: none;
}

It removes the browser's default focus indication without providing an alternative.

Success Criterion 2.4.3 Focus Order requires a keyboard-operable interface to provide a mode in which the keyboard focus indicator is visible.

If the default outline doesn't fit your design, replace it with another visible focus treatment rather than simply removing it:

button:focus-visible,
a:focus-visible,
input:focus-visible,
select:focus-visible,
textarea:focus-visible {
  outline: 3px solid currentColor;
  outline-offset: 3px;
}

This is an example, not a guarantee of conformance. Your chosen indicator still needs to remain visible against the colours surrounding the component.

How WCAG 2.2 Deals with Obscured Focus

WCAG 2.2 added Success Criterion 2.4.11 Focus Not Obscured (Minimum) at Level AA.

When a user interface component receives keyboard focus, author-created content must not completely hide it. The Level AA criterion requires at least part of the focused component to remain visible.

A sticky header illustrates the problem:

.site-header {
  position: sticky;
  top: 0;
  height: 5rem;
}

There's nothing inherently inaccessible about a sticky header. The problem occurs if the page scrolls a focused link or control entirely behind that header.

CSS such as this can help when scroll positioning is involved:

html {
  scroll-padding-top: 6rem;
}

But don't treat it as a complete fix.

Test the real keyboard interaction because cookie notices, fixed bottom navigation, chat windows, sticky toolbars, and other overlays can create similar problems.

The important requirement is the outcome: when focus moves, the user should still be able to see the focused component.

How to Design Pointer Targets and Dragging Interactions

Keyboard support doesn't cover every interaction barrier.

WCAG 2.2 introduced additional requirements that are particularly relevant to touchscreens, drag-and-drop interfaces, and compact controls.

How to Provide an Alternative to Dragging

Imagine a task board where users reorder cards only by dragging them.

Dragging may work well for many users, but it depends on pressing a pointer, moving it while maintaining that interaction, and releasing it in the correct place.

Success Criterion 2.5.7 Dragging Movements requires functionality that uses a dragging movement to also be achievable without dragging through a single-pointer operation, unless dragging is essential to the function.

You can keep drag-and-drop while providing another control:

<article class="task">
  <h3>Prepare monthly report</h3>

  <button type="button">
    Move up
  </button>

  <button type="button">
    Move down
  </button>
</article>

The exact reorder logic depends on your application.

The important part is that the user has another pointer-based way to perform the same function without having to drag the card.

WCAG isn't saying "don't use drag-and-drop". It's saying that dragging shouldn't unnecessarily become the only route to the functionality.

How to Think About Target Size

Success Criterion 2.5.8 Target Size (Minimum) is another WCAG 2.2 Level AA addition.

The criterion uses a minimum target size of 24 by 24 CSS pixels or a defined spacing alternative and contains several exceptions. It therefore should not be simplified to "every clickable element must always be at least 24 pixels wide and high".

For an isolated icon button, you can choose to provide an even larger target:

.icon-button {
  min-width: 2.75rem;
  min-height: 2.75rem;

  display: inline-grid;
  place-items: center;
}

At a typical root font size, this deliberately creates a target larger than the WCAG minimum.

The visible icon can remain smaller:

<button
  class="icon-button"
  type="button"
  aria-label="Delete invoice"
>
  <svg
    width="16"
    height="16"
    aria-hidden="true"
    viewBox="0 0 16 16"
  >
    <path d="M3 4h10M6 4V2h4v2M5 6v7M8 6v7M11 6v7"></path>
  </svg>
</button>

The size of the icon and the size of the interactive target are not the same thing.

That distinction is useful when designing dense interfaces.

How to Keep the Accessible Name Aligned with the Visible Label

Success Criterion 2.5.3 Label in Name concerns controls that have a visible text label.

The accessible name should contain the visible label text. This is particularly important for users who operate interfaces using speech and refer to controls by the words they can see.

Avoid this:

<button aria-label="Find products">
  Search
</button>

The visible label is Search, but the accessible name is Find products.

In this case, the simplest version is better:

<button>
  Search
</button>

If additional accessible context is genuinely necessary, retain the visible wording:

<button aria-label="Search products">
  Search
</button>

Before adding an aria-label, check whether the visible text already gives the control an adequate accessible name.

How to Build More Accessible Forms

Forms combine several areas of accessibility: structure, instructions, errors, input purpose, and status changes.

Start with the field itself.

How to Label Form Controls

This pattern is common:

<input
  type="email"
  name="email"
  placeholder="Email address"
>

The placeholder provides a visual hint, but it's not a good replacement for a proper label.

Use:

<label for="email">
  Email address
</label>

<input
  id="email"
  name="email"
  type="email"
>

Success Criterion 3.3.2 Labels or Instructions requires labels or instructions when content requires user input.

The for and id values also create a programmatic relationship between the label and the field.

How to Identify Common Input Purposes

Success Criterion 1.3.5 Identify Input Purpose applies to fields collecting certain types of information about the user. Their purpose needs to be programmatically determinable when the technology supports it.

HTML's autocomplete tokens help communicate common purposes:

<label for="full-name">
  Full name
</label>

<input
  id="full-name"
  name="full-name"
  type="text"
  autocomplete="name"
>

<label for="email">
  Email address
</label>

<input
  id="email"
  name="email"
  type="email"
  autocomplete="email"
>

This also allows browsers and other tools to provide useful input assistance.

How to Write Useful Validation Errors

Now consider an error message:

Invalid input.

The message tells the user almost nothing: Which input is invalid? What's wrong with it? What needs to change?

Success Criterion 3.3.1 Error Identification requires an automatically detected input error to identify the item in error and describe the error in text.

An implementation might look like this:

<label for="email">
  Email address
</label>

<input
  id="email"
  name="email"
  type="email"
  aria-invalid="true"
  aria-describedby="email-error"
>

<p id="email-error">
  Enter an email address in the format name@example.com.
</p>

aria-invalid="true" exposes the invalid state. aria-describedby associates the explanation with the field.

More importantly, the message tells the user what needs correcting.

Success Criterion 3.3.3 Error Suggestion goes further at Level AA. When the system detects an input error and knows how it can be corrected, it should provide an appropriate suggestion unless doing so would compromise the security or purpose of the content.

The aim isn't to make every error message long. The aim is to make it actionable.

How to Avoid Redundant Entry

Consider a checkout process.

The user enters a delivery address on one step. The next step asks them to type exactly the same address again for billing.

WCAG 2.2 introduced Success Criterion 3.3.7 Redundant Entry at Level A.

When information previously entered by or provided to the user is required again during the same process, the information must generally be auto-populated or available for the user to select. The criterion includes exceptions where re-entry is essential, necessary for security, or where the previous information is no longer valid.

A checkout might offer:

<label>
  <input
    type="checkbox"
    name="billing-same-as-delivery"
  >
  Use my delivery address as my billing address
</label>

Notice that the requirement concerns information within the same process. It doesn't mean every website has to remember every value a user entered during earlier visits.

This criterion also shows why accessibility extends beyond screen-reader support.

Reducing unnecessary repetition can lower the cognitive and interaction effort required to complete a task.

How to Keep Help Consistent

WCAG 2.2 also added Success Criterion 3.2.6 Consistent Help at Level A.

If certain help mechanisms appear repeatedly across a set of pages, they need to appear in the same relative order unless the user initiates a change. These mechanisms can include human contact details, contact mechanisms, self-help options, and automated contact mechanisms.

Suppose your account pages all provide a support link in the header:

<header>
  <a href="/">Acme</a>

  <nav aria-label="Primary">
    <!-- Navigation links -->
  </nav>

  <a href="/support">Support</a>
</header>

Do not move that support mechanism unpredictably between otherwise related pages.

A key nuance is that WCAG 2.2 does not require every website to introduce one of these help mechanisms.

The criterion applies when qualifying help is already available and repeated across multiple pages in the same set.

The development implication is therefore mostly about consistency.

If users learn where help appears on one page, avoid making them search for it again on the next.

How WCAG 2.2 Affects Authentication

Authentication is another area that changed in WCAG 2.2.

Consider a login form that deliberately blocks paste:

passwordInput.addEventListener("paste", (event) => {
  event.preventDefault();
});

That may appear to encourage users to type a password manually, but it can also interfere with mechanisms that reduce the need to remember or transcribe credentials.

Success Criterion 3.3.8 Accessible Authentication (Minimum) addresses authentication steps that require cognitive function tests.

The Level AA requirement allows such tests when an accepted alternative or assistance mechanism is available. W3C specifically identifies password-manager support and copy-and-paste as mechanisms that can reduce the cognitive burden involved in authentication.

A conventional login form can allow these tools to work:

<label for="username">
  Email address
</label>

<input
  id="username"
  name="username"
  type="email"
  autocomplete="username"
>

<label for="password">
  Password
</label>

<input
  id="password"
  name="password"
  type="password"
  autocomplete="current-password"
>

It would be inaccurate to simplify this criterion to "WCAG 2.2 prohibits passwords". It does not.

A password is a cognitive function test, but the criterion permits it when the user has a mechanism that assists with completing that test, such as a password manager that can fill the field.

The same reasoning becomes relevant to multi-factor authentication.

If a process requires a user to read a code on one device and manually transcribe it to another, consider whether the authentication flow offers a path that avoids that cognitive burden. W3C's guidance explicitly discusses authentication processes with several steps and the need for an accessible path through them.

This is a good example of why the exact criterion matters more than a simplified accessibility checklist.

How to Use ARIA Without Replacing HTML

ARIA stands for Accessible Rich Internet Applications.

It provides roles, states, and properties that help web applications communicate information that may not otherwise be available to assistive technologies.

ARIA is useful. It's also easy to misuse.

Consider this example:

<div role="button">
  Place order
</div>

The role tells accessibility APIs that the element represents a button. It doesn't make the element behave like a button.

W3C ARIA Authoring Practices Guide describes an ARIA role as a promise. When you use role="button", you take responsibility for providing the expected keyboard and interaction behaviour yourself. ARIA doesn't cause the browser to add that behaviour automatically.

Where a native HTML element already exists, prefer it:

<button type="button">
  Place order
</button>

How ARIA Can Communicate State

ARIA becomes useful when HTML alone doesn't communicate enough about a component's current state.

Consider a disclosure control:

<button
  id="account-options-trigger"
  type="button"
  aria-expanded="false"
  aria-controls="account-options"
>
  Account options
</button>

<div id="account-options" hidden>
  <a href="/profile">Profile</a>
  <a href="/security">Security</a>
</div>

You can keep aria-expanded in sync with the visible state:

const trigger = document.querySelector(
  "#account-options-trigger"
);

const panel = document.querySelector(
  "#account-options"
);

trigger.addEventListener("click", () => {
  const isExpanded =
    trigger.getAttribute("aria-expanded") === "true";

  trigger.setAttribute(
    "aria-expanded",
    String(!isExpanded)
  );

  panel.hidden = isExpanded;
});

The JavaScript does two related things.

It changes whether the panel is hidden, and it updates the accessibility state exposed by the trigger.

If the panel opens visually but aria-expanded remains false, the interface now communicates two conflicting states.

This illustrates a useful ARIA rule: ARIA state must describe the interface that actually exists.

For more complex patterns such as dialogs, comboboxes, tabs, menus, and grids, the W3C ARIA Authoring Practices Guide provides documented interaction patterns and examples. W3C also makes clear that APG is implementation guidance rather than a normative accessibility standard.

How to Make Dynamic Status Messages Accessible

Modern interfaces frequently update without loading a new page.

A user might save a profile and see:

Your settings were saved.

Or run a search and see:

18 results found.

A sighted user can often notice these updates without moving away from the current control.

Assistive technology also needs a programmatic way to identify relevant status messages.

Success Criterion 4.1.3 Status Messages requires qualifying status messages to be programmatically determinable so assistive technologies can present them without requiring the message itself to receive focus.

For a routine save confirmation, you can use role="status":

<button id="save-settings" type="button">
  Save settings
</button>

<p id="save-status" role="status"></p>

Then update its content:

const saveButton = document.querySelector(
  "#save-settings"
);

const saveStatus = document.querySelector(
  "#save-status"
);

saveButton.addEventListener("click", () => {
  saveStatus.textContent =
    "Your settings were saved.";
});

The browser can expose that status change to supporting assistive technologies without moving keyboard focus away from the Save button.

Not every dynamic DOM change is a status message.

WCAG defines the term more narrowly. It includes information about the result or success of an action, an application's waiting state, the progress of a process, or the existence of errors when that update doesn't itself constitute a change of context.

Don't make every changing piece of content a live announcement. An excessively chatty interface can create a different usability problem.

Use status semantics for information users need to receive while continuing their current task.

How to Test Your Website for Accessibility

Accessibility testing works best as a combination of methods.

WCAG itself is designed to support testing through both automated tools and human evaluation. An automated scanner can identify many technical problems, but it can't reliably judge every accessibility requirement or determine whether an entire user journey makes sense.

How to Start with Automated Testing

Automated tools are useful for repeatable technical checks.

They can identify many problems involving accessible names, some contrast failures, invalid ARIA usage, form relationships, and other machine-detectable conditions.

The limitation appears when correctness depends on meaning.

A tool can tell you that an image has an alt attribute. It can't always determine whether the text accurately communicates the purpose of the image.

Automation should therefore start the evaluation, not end it.

How to Perform Keyboard Testing

Open the page, put the mouse aside, and try to complete an actual task using only your keyboard.

Start with Tab to move forwards through interactive elements and Shift + Tab to move backwards.

Use Enter and Space to activate controls where appropriate. Custom widgets may also use arrow keys or Escape depending on their interaction pattern. The ARIA Authoring Practices keyboard guidance documents expected behaviour for common widget patterns.

Don't simply press Tab a few times and stop.

For example, if you're testing a checkout flow:

1. Navigate to the basket.

2. Change a quantity.

3. Continue to checkout.

4. Move through the form.

5. Submit it.

6. Correct an error.

7. Complete the process.

As you do this, check whether you can reach and operate every required control, move away from every component, follow a sensible focus sequence, see where focus currently is, and avoid having focused content hidden by an overlay.

If something goes wrong, the relevant requirements include Success Criterion 2.1.1 Keyboard, Success Criterion 2.1.2 No Keyboard Trap, Success Criterion 2.4.3 Focus Order, Success Criterion 2.4.7 Focus Visible, and Success Criterion 2.4.11 Focus Not Obscured (Minimum).

How to Test Zoom, Resizing, and Reflow

You can perform a basic zoom test directly in your browser.

In most browsers:

• use Ctrl + + on Windows or Linux

• use Cmd + + on macOS

• use Ctrl/Cmd + 0 to return to the default zoom

W3C's Zoom Easy Check suggests testing at 200%.

As you increase zoom, work through the page and look for:

• clipped text

• overlapping elements

• controls that disappear

• navigation that stops working

• content hidden behind other content

• horizontal scrolling across ordinary page content

Also use a narrow browser window or responsive browser tools to inspect how the content reflows.

This is to verify the behaviour discussed under Success Criterion 1.4.4 Resize Text and Success Criterion 1.4.10 Reflow.

How to Test Colour and Contrast

Use a contrast checker or the colour information available in your browser developer tools to measure foreground and background combinations.

Do this for ordinary text as well as important non-text elements such as custom control borders, icons, and state indicators.

Then test colour-dependent information separately.

For example, if an error field turns red, temporarily ignore the colour change and ask whether another visible indication still communicates the error.

These checks correspond to Success Criterion 1.4.1 Use of Color, Success Criterion 1.4.3 Contrast (Minimum), and Success Criterion 1.4.11 Non-text Contrast.

How to Test Forms Manually

Don't test a form only with valid information. You should deliberately make mistakes to test as many cases as possible.

Leave a required field empty. Enter an incorrectly formatted email address. Submit a value the form should reject.

Then check whether you can:

• identify the field that has a problem

• understand the error message

• determine how to correct it

• reach the error using the keyboard

• correct the information and continue

Also inspect form controls in your browser developer tools to confirm that visible labels and descriptions are associated with the correct fields.

These tests help you verify Success Criterion 3.3.1 Error Identification, Success Criterion 3.3.2 Labels or Instructions, and Success Criterion 3.3.3 Error Suggestion.

How to Test Pointer and Dragging Interactions

If your interface contains drag-and-drop, complete the action normally first. Then try to perform the same function without dragging.

For example, if you can drag a task into a new position, check whether another pointer-operated control lets you move it as well.

That gives you a practical test for Success Criterion 2.5.7 Dragging Movements.

For small controls, use browser developer tools to inspect the rendered interactive area rather than judging only the visible icon.

Pay particular attention to close buttons, carousel controls, pagination items, icon buttons, and densely packed toolbars when checking Success Criterion 2.5.8 Target Size (Minimum).

How to Inspect the Accessibility Tree

Modern browser developer tools expose accessibility information associated with elements.

Inspect important controls and compare what the accessibility tree reports with what the interface shows.

A button might visually say Search while its accessible name says something completely different. A disclosure may look open while its aria-expanded state remains false.

Inspecting the accessibility tree helps expose these mismatches.

How to Test with Assistive Technology

When testing with a screen reader, focus on complete tasks.

For a form, navigate to the fields, identify their labels, enter incorrect information, submit it, locate and understand the errors, correct them, and confirm the successful state.

The question isn't simply:

Can the screen reader read this page?

The more useful question is:

Can the user complete the task and understand what happened?

Testing with disabled users can reveal additional usability barriers that automated and standards-based evaluation may not expose.

Conclusion

WCAG becomes easier to understand when you connect its requirements to normal development decisions. The important shift is to stop treating accessibility as a final audit. Build it into the interface while you build everything else.

But the real challenge comes when you want to make that modal or pop-up accessible.

Today, I'll show you how you can use <dialog> to create an accessible modal/pop-up with minimal setup using HTML, CSS and JavaScript.

Prerequisites

Before diving in, it helps if you’re comfortable with a few basics:

• CSS fundamentals: You should be familiar with common CSS terminology and concepts (selectors, properties, positioning, and so on).

• HTML structure: A working knowledge of how elements are organized in the DOM will make the examples easier to follow.

• JavaScript DOM basics: While this guide uses only minimal JavaScript, understanding how to query and manipulate DOM elements will give you more confidence as you experiment.

That’s all you need. No frameworks, no heavy boilerplate. Just a foundation in the core web technologies.

Table of Contents

• Prerequisites

• How to Create Pop-ups with popover

• How to Create a Modal Using the dialog Tag

• Gotchas to Pay Attention to

• The ::backdrop pseudo class

• Final Thoughts

How to Create Pop-ups with popover

Creating a pop-up from scratch using HTML, CSS and JavaScript can be quite challenging. A pop-up is a small dialog box that appears on the screen to show extra information or ask for input. Because it’s temporary by design, users expect it to disappear once they’re done interacting with it, like when they press Escape or click outside of it.

Using the HTML popover and popovertarget attributes, you can have several built-in accessibility and interaction behaviors, such as Escape-to-close and light dismiss, but you still need to choose the right semantic element and test keyboard and screen reader behavior.

Setting Up the Pop-up with HTML

To be practical, let's create a common example and use case for pop-ups on a webpage. We'll create a nav element that displays when a user clicks a hamburger menu or the menu list.

First, you'll set the popover attribute on the pop-up container. This tells HTML to treat the containing block as a pop-up and hide it from the screen by default.

Then you set the popovertarget attribute on the element that will trigger the pop-up (like a button element or something else) to unhide the hidden element with an attribute of popover.

Example:

<button popovertarget="navbar-menu" id='nav-btn'>open</button>

<nav id="navbar-menu" popover>
  <a href="#">Home</a>
  <a href="#">About</a>
  <a href="#">Address</a>
</nav>

With the above setup, you have a pop-up with useful built-in interaction behaviors, including Escape-to-close and light dismiss. You can hide it from the screen by pressing the ESC key on the keyboard or when you click anywhere else on the page (as long as it's not inside the pop-up section).

Remember that the popover attribute alone doesn't automatically make a pop-up accessible. You still need to use the appropriate semantic elements, provide accessible labels where needed, and test keyboard and screen reader behavior.

How to Align the Pop-up

Now you'll want to align the pop-up and place it where you want it to be. By default, the pop-up (or modal) that's created using either the popover attribute or the dialog tag will be centered on the page.

This is because by default elements with popover have a position of fixed and the inset of 0, which centers the pop-up box and a margin of auto.

Note: inset is the shorthand for top, bottom, left and right of an element's position on the page. If you want to have the same size on all of sides of an element, use inset.

If you don't want your pop-up in the center, you can start by setting the element with popover position to absolute to isolate it from the page's flow:

#navbar-menu {
  position: absolute;
}

You can then disable the margin to 0 and positions (inset) to unset:

#navbar-menu {
  position: absolute;
  margin: 0;
  inset: unset;
}

After this, you can then place the popover element on the side of the page you want.

How to Use the position-anchor Property

Note: CSS Anchor Positioning is a newer feature. Check browser support before relying on it in production and provide a fallback for browsers that don't support it yet.

The next step is to position the element close to the element that triggers it. For it we can use the position-anchor property of CSS.

The position-anchor property in CSS specifies a default anchor element that an absolutely or fixed-positioned element will "tether" or snap to. It lets you link a floating target (like our pop-up here) to another element on the page using only CSS.

In our example, we have a menu list icon or a hamburger menu that will open and close the nav element as a pop-up. We want the nav bar menu to be attached to/near the menu list that opens it.

So, you can add the anchor-name property to a menu icon. The name must be prefixed with double-dashes (and the name can be anything you want).

#nav-btn {
  anchor-name: --nav;
}

The position-anchor property lets you attach an element to another element identified by an anchor name. Once anchored, you can use the anchor() function to position the element relative to that anchor, like aligning it to the anchor’s top, bottom, or center.

#navbar-menu {
  position: absolute;
  margin: 0;
  inset: auto;
  position-anchor: --nav;
  top: anchor(bottom);
  right: anchor(right);
}

Pass the anchor name you give your anchor as value of position-anchor to align the nav element (#navbar-menu in our example) on the page around the anchor-name of nav which is #nav-btn in our example.

Then the anchor()positions the top of nav element on the bottom side and right side to the menu's right side.

How to Create a Modal Using the dialog Tag

Using the command and commandfor attributes on a button, you can declaratively control a <dialog>. For example, command="show-modal" opens it as a modal, while command="close" closes it.

Pop-up modals and pop-ups are different: with a pop-up, you can still interact with the page when the pop-up is active. But with modals or pop-up modals you can't interact with the page – modals lock the screen until they're ignored or confirmed.

When a <dialog> is opened as a modal, it's placed in the browser's top layer so that it appears above the rest of the page content. The rest of the document becomes inert, meaning users can't interact with it while the modal is open. The browser also creates a ::backdrop behind the modal, which you can style to provide a visual overlay.

Here's an example:

<button command="show-modal" commandfor="contact-dialog">
    open modal
</button>

 <dialog id="contact-dialog">
    <button command="close" commandfor="contact-dialog" aria-label="close modal">
      close modal
    </button>
    <!--modal contents goes here-->
</dialog>

You can use the command attribute to close and show the modal and the commandfor attribute to reference the modal that's being targeted using the modal's id.

The command attribute can receive one of the following options:

• show-modal: This option is used on the element that will trigger the modal or the dialog box to open it.

• close: This option is passed to an element and will close the modal when it's open.

In the code snippet example, you can see that the button with label open modal has a command attribute with the option to show-modal and the commandfor attribute that's targeting the dialog element by its id.

The close modal button is using the close option on the command attribute to close the modal when clicked. It also uses commandfor to indicate which modal should be closed when the close button is clicked.

Below you'll find a demo of a modal that's created using the command and commandfor attributes:

Try clicking the "click to pop a modal!" button on the Code Pen above to see the modal you can create by just using the command and commandfor attributes on a dialog tag.

Create modal using showModal()

The command API may not be supported in older browsers. Alternatively you can use JavaScript's showModal() method, as it's widely available in many browsers compared to command and commandfor.

const btn = document.querySelector("button");
const dialog = document.querySelector("dialog");

btn.addEventListener("click", () => {
  dialog.showModal();
});

How to Open a Non-Modal Dialog with show()

Sometimes you may want to have an element like modal that stays interactive and doesn't block the other pages content like the command and commandfor attributes do. For this, you can use show() on dialog using JavaScript.

const btn = document.querySelector("button");
const dialog = document.querySelector("dialog");

btn.addEventListener("click", () => {
  dialog.show();
});

The above snippet keeps the rest of the page interactive while the dialog is open. This differs from opening a dialog with showModal() or using command="show-modal", which opens the dialog as a modal and makes the rest of the document inert.

Note: Keep in mind that show() isn't considered a modal but more of a dialog-like element that doesn't block interaction with the rest of the page.

Gotchas to Pay Attention to

When you're using element popover, the command and commandfor attributes, or the show() method, there are some "gotchas" to watch out for. Paying attention to these will help you stick to best practices.

Don't Use Flex or Grid

Directly applying layout-related styles like Flex or Grid is highly discouraged on both elements with popover and modal elements.

By default, elements with popover attribute, command and commandfor button attributes, and show() have a display of none. This basically means the pop-up or modal is hidden from the screen.

When you add flex or grid directly on element with the popover attribute or on a modal element, you're rewriting the modal or popover element's default behavior and they will be always visible on the screen. This means that you won't be able to hide the modal from the screen.

Check out the below example in the CodePen demo:

#navbar-menu {
  display: grid;
/* other styles definition*/
}

You can see from the example that the nav element is always visible even if we click on the menu button to hide it again.

Suggested Approach

In this situation, you can either use the popover-open pseudo-class on an element on a property that has popover attributes, or you can use dialog[open] on the dialog element.

The pseudo-class styles the element based on its state when interacting with the page. So, in this case we want to give the pop-up or a modal a different layout when it's in the open state.

Example:

/* element with a popover*/
#navbar-menu:popover-open {
  display: grid;
  gap: 2rem;
}

/* using a dialog element*/
dialog[open] {
  display: grid;
  gap: 2rem;
}

Background Scrolling

Another thing to consider when using the <dialog> element is background scrolling. Depending on the browser and platform, the underlying page may still be scrollable while a modal dialog is open. If you want to prevent this behavior, you can explicitly disable scrolling while the dialog is open.

Take a look at this CodePen example:

When you click on the button to show the modal, you can still see the scrollbar on the page, and you can scroll around the page.

Suggested Approach

To deal with this issue we can use the has pseudo class. has helps select the parent element based on its children's state. On the root element or HTML element, you can check if it has an open modal. If so, you can set the root element or HTML element overflow to hidden.

For example:

html:has(dialog[open]) {
  overflow: hidden;
}

This will hide the scrollbar on the page when the modal is open. Keep in mind that you can also use any parent element to wrap the dialog tag with the has pseudo class. It doesn't always have to be the root element or HTML element.

The ::backdrop Pseudo Class

If you want to use a customized overlay color on the popover element or modal element, you can use the ::backdrop pseudo class to customize the appearance for the modal overlay color.

Example:

dialog::backdrop {
  background: rgba(43, 50, 200, 0.4);
}

This will apply the overlay with defined RGB colors and the opacity of 0.4 on the overlay to have a little transparent on the overlay.

Final Thoughts

I hope you've gained something new from this article, and that it will help you start using these techniques in your projects.

VoiceOver speaks aloud the text on the screen, app names, icons, buttons, menus, links, and notifications and alerts.

These accessibility features are crucial for users who may be blind, have low vision, or have reading differences.

As a developer, it's always important to manually test your website for accessibility. A screen reader is one of the most important tools to add to your testing process. Even small issues, like a button with no label or an image with no alt text, can make a page completely unusable for someone relying on VoiceOver.

As you test on an actual device with VoiceOver, you may find accessibility issues you didn’t know you had. In this tutorial, we'll cover how to turn VoiceOver on, the basic gestures to know, and how to adjust its settings to fit your needs.

What We'll Cover:

• How to turn on VoiceOver

  • Option 1: Use Settings

  • Option 2: Use Siri

  • Option 3: Set up the Accessibility Shortcut

• Basic Gestures to Know

• How to Adjust VoiceOver Settings

  • Change the Speaking Rate

  • Change the Voice

• Conclusion

How to Turn On VoiceOver

There are a few ways to turn VoiceOver on or off. As you practice, you might lean toward one option over the others.

Option 1: Use Settings

1. Open the Settings app.

2. Tap Accessibility.

3. Tap VoiceOver.

4. Toggle it on or off.

In the Accessibility Settings section, you can also find other accessible settings to explore — including Display & Text Size, Motion, and Spoken Content. It's worth browsing through to understand what tools are available to users.

Option 2: Use Siri

To turn on VoiceOver using Siri, say: "Hey Siri, turn on VoiceOver."

To turn it off, say: "Hey Siri, turn off VoiceOver."

Using Siri might be the easiest way to turn VoiceOver on or off for beginners. If you accidentally turn VoiceOver on and don't yet know any shortcuts or gestures, just ask Siri. Siri works independently of VoiceOver gestures, so it's a reliable fallback when you feel stuck.

Option 3: Set Up the Accessibility Shortcut

If you find yourself turning VoiceOver on and off frequently, setting up the Accessibility Shortcut makes sense. This is the quickest method for developers who are regularly switching VoiceOver on to test and off to work. It lets you toggle VoiceOver by pressing the side button three times.

1. Go to Settings > Accessibility.

2. Scroll down and tap Accessibility Shortcut.

3. Select VoiceOver.

After that, press the side button (or Home button on older iPhones) three times to toggle VoiceOver on or off. If you have more than one accessibility feature enabled in the shortcut, your iPhone will show a menu to pick from instead of toggling automatically.

Basic Gestures to Know

When VoiceOver is on, the way you touch the screen changes what the phone interprets. The same swipe or tap that normally opens an app does something different with VoiceOver active.

Here are the five core gestures every developer should learn first:

• Swipe right: Move to the next item on screen

• Swipe left: Move to the previous item on screen

• Swipe up or down with three fingers: Scroll up or down the page

• One tap: Hear VoiceOver read the item aloud

• Double-tap: Open an app or activate a button

Tip: When you first tap an item, VoiceOver reads it to you. Then you can double-tap to actually open or activate it. This two-step process helps you confirm you're on the right element before you act. This is especially useful when testing unfamiliar interfaces.

These five gestures are the foundation. As you use VoiceOver more frequently, they'll become second nature. Once you're comfortable, you can explore more advanced gestures like the VoiceOver rotor, which lets you navigate by headings, links, form fields, and more.

For the time being, if you're comfortable with these five gestures, you’ll be able to test mobile accessibility issues for your products.

How to Adjust VoiceOver Settings

You can change how VoiceOver sounds and behaves to better suit your testing workflow or personal preferences.

Change the Speaking Rate

1. Go to Settings > Accessibility > VoiceOver.

2. Use the Speaking Rate slider to make it faster or slower.

You can also adjust the speaking rate with the VoiceOver rotor. Rotate two fingers on the screen until you hear “Speaking Rate,” then swipe up or down with one finger to make VoiceOver faster or slower.

This changes the rate on the fly without going into Settings. It's handy when you want to slow down while exploring a complex page or speed up when navigating familiar content.

Experienced VoiceOver users often run the speaking rate very fast. Don't be surprised if the default speed feels quick. You can always slow it down while you're learning.

Change the Voice

If you want to change the voice or language, Go to Settings > Accessibility > VoiceOver > Speech. From there, you can choose a different VoiceOver voice, add rotor voices for other languages, or enable language detection.

Apple offers multiple voice options across many languages. This is useful when testing multilingual content, but make sure your site also uses correct lang attributes so screen readers can switch pronunciation appropriately.

Conclusion

As a developer, it's always important to manually test your website for accessibility. Testing it with the accessibility features your users use is crucial to access the product through their lens and fix accessibility bugs the product might have.

Simply turning the VoiceOver on and learning about these five simple gestures will give you the tools to audit and test your website for accessibility issues.

It started with deciphering some distorted text, then evolved into checking a box or boxes where an element is in an image. While CAPTCHAs can help websites determine whether the user is a human or a bot, they often present challenges to many real humans, especially those with disabilities.

Photo by Karen Grigorean on Unsplash

Table of Contents

• What is CAPTCHA?

• What Are the Issues with Visual CAPTCHAs?

• What Are the Issues with Auditory CAPTCHAs?

• What Are the Issues with Time-based CAPTCHAs?

• Common Workarounds and Accessible Alternatives

  • Risk-Based Authentication

  • Device-Based Trust

  • Human-Friendly Verification

• Conclusion

What is CAPTCHA?

CAPTCHA is an acronym that stands for Completely Automated Public Turing test to tell Computers and Humans Apart. CAPTCHA was originally created to prevent automated systems from abusing websites. For example, an automated system creating thousands of fake accounts in a short amount of time.

CAPTCHA can be a visual recognition test, like selecting all images with traffic lights. CAPTCHA can be audio challenge, which can be a test to type from hearing an audio. CAPTCHA can also be time-based or interaction-based behavior tracking. Visual, audio and time-based CAPTCHAs all have their own unique set of usability and accessibility issues.

What Are the Issues with Visual CAPTCHAs?

Image-based CAPTCHAs are one of the most common formats today. Visual CAPTCHA’s are the tests where we you'd see nine square boxes and have to select all of the boxes with traffic lights. They are also tested with one large image broken down into nine squares, and the user has to select the boxes where traffic lights appear. The visual CAPTCHAs create barriers for many people, such as people with visual impairments like blindness, low vision, color blindness.

First, the visual tests are often poorly compatible with screen readers as they are often not properly labeled for assistive technologies like screen readers, which creates an issue for users who rely on them.

In addition to issues with screen readers, how many times have you tried to complete a CAPTCHA where a sliver of a traffic light was in another square and you didn’t know if the test wanted you to select that box as well? I personally had issues verifying if the CAPTCHA test wanted me to select a square with a sliver of an image. Some CAPTCHAs have extremely low color contrast where I had issues figuring out the distorted text.

Some of the tests and images are ambiguous and create confusion for users. Audio CAPTCHA is an alternative approach to those that struggle with visual CAPTCHAs. Audio also comes with it's own set of usability and accessibility issues.

What Are the Issues with Auditory CAPTCHAs?

The visual CAPTCHA tests often come with an auditory equivalent for users who want to complete the test with auditory clues. The tests often have a button for the user to click and hear the visual clues. The user needs to enter the correct clue they hear in order to complete the CAPTCHA.

While the auditory CAPTCHA provides an alternative way for a user to complete the assignment, these often come with their own challenges. For example, they are frequently hard to understand due to distortion. Also, what if the user tries to complete these in a loud environment? Not only will the audio be distorted, the user will face a hard time listening when their environment is loud. In addition, the user might be hard of hearing, which will make the auditory CAPTCHA even harder to complete.

CAPTCHAs may put users in two difficult situations. Either struggle with a visual interface you cannot see, or attempt an audio alternative that may be equally unusable.

In addition to these struggles, CAPTCHAs can be time-based, giving you a test to complete in a specific amount of time. Time-based limitations can create another set of problems.

What Are the Issues with Time-based CAPTCHAs?

Some CAPTCHA challenges might need to be completed in a specific period of time. If the user takes longer, CAPTCHA test is voided.

The time-based CAPTCHA can create issues for users with cognitive disabilities, like memory, attention, or processing challenges or motor disabilities, those who have difficulty using a mouse or precise interactions. These users might need to spend more time completing the CAPTCHA, more time than the time limit set by CAPTCHA. In addition, users with anxiety disorders, or simply low bandwidth connections can also face the same issue.

Common Workarounds and Accessible Alternatives

Some websites try to improve accessibility by offering invisible CAPTCHA, checkbox verification, or verification through email or SMS. However, these solutions are inconsistent and users may still face challenges when the system is uncertain.

Improving accessibility in bot prevention does not mean removing security - it means reducing unnecessary barriers. We can try to implement Risk-Based Authentication, Device-Based Trust or move toward a human friendly approach.

Risk-Based Authentication

Instead of challenging every user, websites can analyze signals such as device history, location, login patterns, and behavior in the background. This allows the low-risk users proceed normally while triggering additional verification for suspicious activity. This reduces interruptions for legitimate users while maintaining security.

Device-Based Trust

Websites can recognize trusted devices after a successful login using secure tokens, passkeys, or multi-factor authentication. For example:

1. User logs in successfully

2. Device is marked as trusted

3. Future visits bypass CAPTCHA unless unusual activity is detected

However, users should still have clear opt-outs and accessible fallback options for the first time they log in.

Human-Friendly Verification

When verification is necessary, websites can use methods that are easier to access than visual or audio puzzles, such as, email confirmation links, one-time codes and push notifications. These methods are often more compatible with screen readers and assistive technologies.

The goal is to move from “prove you are human by solving a puzzle” to “verify legitimacy with minimal friction for all users.”

Conclusion

CAPTCHAs highlight a broader tension in web design: security versus accessibility. While they solve a real problem—automated abuse—they often introduce barriers that disproportionately affect users with disabilities.

As accessibility standards evolve and awareness increases, the challenge is not just building systems that stop bots, but ensuring that legitimate users are not excluded in the process.

By combining semantic HTML, responsive design techniques, and accessibility best practices (like ARIA roles and keyboard navigation), developers can create interfaces that work across devices and for all users, including those with disabilities.

This article shows how to design scalable, inclusive React UIs using real-world patterns and code examples.

Table of Contents

• Prerequisites

• Overview

• Why Accessibility and Responsiveness Matter

• Core Principles of Accessible and Responsive Design

• Using Semantic HTML in React

• Structuring a Page with Semantic Elements

• Building Responsive Layouts

• Accessibility with ARIA

• Keyboard Navigation

• Focus Management

• Forms and Accessibility

• Responsive Typography and Images

• Building a Fully Accessible Responsive Component (End-to-End Example)

• Testing Accessibility

• Best Practices

• When NOT to Overuse Accessibility Features

• Future Enhancements

• Conclusion

Prerequisites

Before following along, you should be familiar with:

• React fundamentals (components, hooks, JSX)

• Basic HTML and CSS

• JavaScript ES6 features

• Basic understanding of accessibility concepts (helpful but not required)

Overview

Modern web applications must serve a diverse audience across a wide range of devices, screen sizes, and accessibility needs. Users today expect seamless experiences whether they are browsing on a desktop, tablet, or mobile device – and they also expect interfaces that are usable regardless of physical or cognitive limitations.

Two essential principles help achieve this:

• Responsive design, which ensures layouts adapt to different screen sizes

• Accessibility, which ensures applications are usable by people with disabilities

In React applications, these principles are often implemented incorrectly or treated as afterthoughts. Developers may rely heavily on div-based layouts, ignore semantic HTML, or overlook accessibility features such as keyboard navigation and screen reader support.

This article will show you how to build responsive and accessible UI designs in React using semantic HTML. You'll learn how to:

• Structure components using semantic HTML elements

• Build responsive layouts using modern CSS techniques

• Improve accessibility with ARIA attributes and proper roles

• Ensure keyboard navigation and screen reader compatibility

• Apply best practices for scalable and inclusive UI design

By the end of this guide, you'll be able to create React interfaces that are not only visually responsive but also accessible to all users.

Why Accessibility and Responsiveness Matter

Responsive and accessible design isn't just about compliance. It directly impacts usability, performance, and reach.

Accessibility benefits:

• Supports users with visual, motor, or cognitive impairments

• Improves SEO and content discoverability

• Enhances usability for all users

Responsiveness benefits:

• Ensures consistent UX across devices

• Reduces bounce rates on mobile

• Improves performance and scalability

Ignoring these principles can result in broken layouts on smaller screens, poor screen reader compatibility, and limited reach and usability.

Core Principles of Accessible and Responsive Design

Before diving into the code, it’s important to understand the foundational principles.

1. Semantic HTML First

Semantic HTML refers to using HTML elements that clearly describe their meaning and role in the interface, rather than relying on generic containers like <div> or <span>.These elements provide built-in accessibility, improve SEO, and make code more readable.

For example:

Non-semantic:

<div onClick={handleClick}>Submit</div>

Semantic:

<button type="button" onClick={handleClick}>Submit</button>

Another example:

Non-semantic:

<div className="header">My App</div>

Semantic:

<header>My App</header>

Using semantic elements such as <header>, <nav>, <main>, <section>, <article>, and <button> helps browsers and assistive technologies (like screen readers) understand the structure and purpose of your UI without additional configuration.

Why this matters:

• Screen readers understand semantic elements automatically

• It supports built-in accessibility (keyboard, focus, roles)

• There's less need for ARIA attributes

• It gives you better SEO and maintainability

2. Mobile-First Design

Mobile-first design means starting your UI design with the smallest screen sizes (typically mobile devices) and progressively enhancing the layout for larger screens such as tablets and desktops.

This approach makes sure that core content and functionality are prioritized, layouts remain simple and performant, and users on mobile devices get a fully usable experience.

In practice, mobile-first design involves:

• Using a single-column layout initially

• Applying minimal styling and spacing

• Avoiding complex UI patterns on small screens

Then, you scale up using CSS media queries:

.container {
  display: flex;
  flex-direction: column;
}
@media (min-width: 768px) {
  .container {
    flex-direction: row;
  }
}

Here, the default layout is optimized for mobile, and enhancements are applied only when the screen size increases.

Why this approach works:

• Prioritizes essential content

• Improves performance on mobile devices

• Reduces layout bugs when scaling up

• Aligns with how most users access web apps today

3. Progressive Enhancement

Progressive enhancement is the practice of building a baseline user experience that works for all users (regardless of their device, browser capabilities, or network conditions) and then layering on advanced features for more capable environments.

This approach ensures that core functionality is always accessible, users on older devices or slow networks aren't blocked, and accessibility is preserved even when advanced features fail.

In practice, this means:

• Start with semantic HTML that delivers content and functionality

• Add basic styling with CSS for layout and readability

• Enhance interactivity using JavaScript (React) only where needed

For example, a form should still be usable with plain HTML:

<form>
  <label htmlFor="email">Email</label>
  <input id="email" type="email" />
  <button type="submit">Submit</button>
</form>

Then, React can enhance it with validation, dynamic feedback, or animations.

By prioritizing functionality first and enhancements later, you ensure your application remains usable in a wide range of real-world scenarios.

4. Keyboard Accessibility

Keyboard accessibility ensures that users can navigate and interact with your application using only a keyboard. This is critical for users with motor disabilities and also improves usability for power users.

Key aspects of keyboard accessibility include:

• Ensuring all interactive elements (buttons, links, inputs) are focusable

• Maintaining a logical tab order across the page

• Providing visible focus indicators (for example, outline styles)

• Supporting keyboard events such as Enter and Space

Bad Example (Not Accessible)

<div onClick={handleClick}>Submit</div>

This element:

• Cannot be focused with a keyboard

• Does not respond to Enter/Space

• Is invisible to screen readers

Good Example

<button type="button" onClick={handleClick}>Submit</button>

This automatically supports:

• Keyboard interaction

• Focus management

• Screen reader announcements

Custom Component Example (if needed)

<div
  role="button"
  tabIndex={0}
  onClick={handleClick}
  onKeyDown={(e) => {
    if (e.key === 'Enter' || e.key === ' ') {
      e.preventDefault();
      handleClick();
    }
  }}
>
  Submit
</div>

But only use this when native elements aren't sufficient.

These principles form the foundation of accessible and responsive design:

• Use semantic HTML to communicate intent

• Design for mobile first, then scale up

• Enhance progressively for better compatibility

• Ensure full keyboard accessibility

Applying these early prevents major usability and accessibility issues later in development.

Using Semantic HTML in React

As we briefly discussed above, semantic HTML plays a critical role in both accessibility (a11y) and code readability. Semantic elements clearly describe their purpose to both developers and browsers, which allows assistive technologies like screen readers to interpret and navigate the UI correctly.

For example, when you use a <button> element, browsers automatically provide keyboard support, focus behavior, and accessibility roles. In contrast, non-semantic elements like <div>require additional attributes and manual handling to achieve the same functionality.

From a readability perspective, semantic HTML makes your code easier to understand and maintain. Developers can quickly identify the structure and intent of a component without relying on class names or external documentation.

Bad Example (Non-semantic)

<div onClick={handleClick}>Submit</div>

Why this is problematic:

• The <div>element has no inherent meaning or role

• It is not focusable by default, so keyboard users can't access it

• It does not respond to keyboard events like Enter or Space unless explicitly coded

• Screen readers do not recognize it as an interactive element

To make this accessible, you would need to add:

role="button"

tabIndex="0"

Keyboard event handlers

Good Example (Semantic)

<button type="button" onClick={handleClick}>Submit</button>

Why this is better:

• The <button> element is inherently interactive

• It is automatically focusable and keyboard accessible

• It supports Enter and Space key activation by default

• Screen readers correctly announce it as a button

This reduces complexity while improving accessibility and usability.

Why all this matters:

There are many reasons to use semantic HTML.

First, semantic elements like <button>, <a>, and <form> come with default accessibility behaviors such as focus management and keyboard interaction

It also reduces complexity: you don’t need to manually implement roles, keyboard handlers, or tab navigation

They provide better screen reader support as well. Assistive technologies can correctly interpret the purpose of elements and announce them appropriately

Semantic HTML also improves maintainability and helps other developers quickly understand the intent of your code without reverse-engineering behavior from event handlers

Finally, you'll generally have fewer bugs in your code. Relying on native browser behavior reduces the risk of missing critical accessibility features

Here's another example:

Non-semantic:

<div className="nav">
  <div onClick={goHome}>Home</div>
</div>

Semantic:

<nav>
  <a href="/">Home</a>
</nav>

Here, <nav> clearly defines a navigation region, and <a> provides built-in link behavior, including keyboard navigation and proper screen reader announcements.

Structuring a Page with Semantic Elements

When building a React application, structuring your layout with semantic HTML elements helps define clear regions of your interface. Instead of relying on generic containers like <div>, semantic elements communicate the purpose of each section to both developers and assistive technologies.

In the example below, we're creating a basic page layout using commonly used semantic elements such as <header>, <nav>, <main>, <section>, and <footer>. Each of these elements represents a specific part of the UI and contributes to better accessibility and maintainability.

function Layout() {
  return (
    <>
      {/* Skip link for keyboard and screen reader users */}
      <a href="#main-content" className="skip-link">
        Skip to main content
      </a>

      <header>
        <h1>My App</h1>
      </header>

      <nav>
        <ul>
          <li><a href="/">Home</a></li>
        </ul>
      </nav>

      <main id="main-content">
        <section>
          <h2>Dashboard</h2>
        </section>
      </main>

      <footer>
        <p>© 2026</p>
      </footer>
    </>
  );
}

Each element in this layout has a specific role:

• The skip link allows screen reader users to skip to the main content

• <header>: Represents introductory content or branding

• <nav>: Contains navigation links

• <main>: Holds the primary content of the page

• <section>: Groups related content within the page

• <footer>: Contains closing or supplementary information

Using these elements correctly ensures your UI is both logically structured and accessible by default.

Why this structure is important:

Properly structuring a page like this brings with it many benefits.

For example, it gives you Improved screen reader navigation. This is because semantic elements allow screen readers to identify different regions of the page (for example, navigation, main content, footer). Users can quickly jump between these sections instead of reading the page linearly

It also gives you better document structure. Elements like <main> and <section> define a logical hierarchy, making content easier to parse for both browsers and assistive technologies

Search engines also use semantic structure to better understand page content and prioritize important sections, resulting in better SEO.

It also makes your code more readable, so other devs can immediately understand the layout and purpose of each section without relying on class names or comments

And it provides built-in accessibility landmarks using elements like <nav> and <main>, allowing assistive technologies to provide shortcuts for users.

Building Responsive Layouts

Responsive layouts ensure that your UI adapts smoothly across different screen sizes, from mobile devices to large desktop displays. Instead of building separate layouts for each device, modern CSS techniques like Flexbox, Grid, and media queries allow you to create flexible, fluid designs.

In this section, we’ll look at how layout behavior changes based on screen size, starting with a mobile-first approach and progressively enhancing the layout for larger screens.

Using CSS Flexbox:

.container {
  display: flex;
  flex-direction: column;
}

@media (min-width: 768px) {
  .container {
    flex-direction: row;
  }
}

On smaller screens (mobile), elements are stacked vertically using flex-direction: column, making content easier to read and scroll.

On larger screens (768px and above), the layout switches to a horizontal row, utilizing available screen space more efficiently.

Why this helps:

• Ensures content is readable on small devices without horizontal scrolling

• Improves layout efficiency on larger screens

• Supports a mobile-first design strategy by defining the default layout for smaller screens first and enhancing it progressively

Using CSS Grid:

.grid {
  display: grid;
  grid-template-columns: 1fr;
  gap: 16px;
}

@media (min-width: 768px) {
  .grid {
    grid-template-columns: repeat(3, 1fr);
  }
}

On mobile devices, content is displayed in a single-column layout (1fr), ensuring each item takes full width.

On larger screens, the layout shifts to three equal columns using repeat(3, 1fr), creating a grid structure.

Why this helps:

• Provides a clean and consistent way to manage complex layouts

• Makes it easy to scale from simple to multi-column designs

• Improves visual balance and spacing across different screen sizes

React Example:

function CardGrid() {
  return (
    <div className="grid">
      <div className="card">Item 1</div>
      <div className="card">Item 2</div>
      <div className="card">Item 3</div>
    </div>
  );
}

The React component uses the .grid class to apply responsive Grid behavior. Each card automatically adjusts its position based on screen size.

Why this is effective:

• Separates structure (React JSX) from layout (CSS)

• Allows you to reuse the same component across different screen sizes without modification

• Ensures consistent responsiveness across your application with minimal code

By combining Flexbox for one-dimensional layouts and Grid for two-dimensional layouts, you can build highly adaptable interfaces that respond efficiently to different devices and screen sizes.

Accessibility with ARIA

ARIA (Accessible Rich Internet Applications) is a set of attributes that enhance the accessibility of web content, especially when building custom UI components that cannot be fully implemented using native HTML elements.

ARIA works by providing additional semantic information to assistive technologies such as screen readers. It does this through:

• Roles, which define what an element is (for example, button, dialog, menu)

• States and properties, which describe the current condition or behavior of an element (for example, expanded, hidden, live updates)

For example, when you create a custom dropdown using <div> elements, browsers don't inherently understand its purpose. By applying ARIA roles and attributes, you can communicate that this structure behaves like a menu and ensure it is interpreted correctly.

Just make sure you use ARIA carefully. Incorrect or unnecessary usage can reduce accessibility. Here's a key rule to follow: use native HTML first. Only use ARIA when necessary.

ARIA is especially useful for:

• Custom UI components (modals, tabs, dropdowns)

• Dynamic content updates

• Complex interactions not covered by standard HTML

Something to note before we get into the examples here: real-world accessibility is complex. For production apps, you should typically prefer well-tested libraries like react-aria, Radix UI, or Headless UI. These examples are primarily for educational purposes and aren't production-ready.

Example: Accessible Modal

function Modal({ isOpen, onClose }) {
  const dialogRef = React.useRef();

  React.useEffect(() => {
    if (isOpen) {
      dialogRef.current?.focus();
    }
  }, [isOpen]);

  if (!isOpen) return null;

  return (
    <div
      role="dialog"
      aria-modal="true"
      aria-labelledby="modal-title"
      tabIndex={-1}
      ref={dialogRef}
      onKeyDown={(e) => {
        if (e.key === 'Escape') onClose();
      }}
    >
      <h2 id="modal-title">Modal Title</h2>
      <button type="button" onClick={onClose}>Close</button>
    </div>
  );
}

How this works:

• role="dialog" identifies the element as a modal dialog

• aria-modal="true" indicates that background content is inactive

• aria-labelledby connects the dialog to its visible title for screen readers

• tabIndex={-1} allows the dialog container to receive focus programmatically

• Focus is moved to the dialog when it opens

• Pressing Escape closes the modal, which is a standard accessibility expectation

This ensures that users can understand, navigate, and exit the modal using both keyboard and assistive technologies.

Key ARIA Attributes

1. role

Defines the type of element and its purpose. For example, role="dialog" tells assistive technologies that the element behaves like a modal dialog.

2. aria-label

Provides an accessible name for an element when visible text is not sufficient. Screen readers use this label to describe the element to users.

3. aria-hidden

Indicates whether an element should be ignored by assistive technologies. For example, aria-hidden="true" hides decorative elements from screen readers.

4. aria-live

Used for dynamic content updates. It tells screen readers to announce changes automatically without requiring user interaction (for example, form validation messages or notifications).

Example: Accessible Dropdown (Custom Component)

function Dropdown({ isOpen, toggle }) {
  return (
    <div>
      <button
        type="button"
        aria-expanded={isOpen}
        aria-controls="dropdown-menu"
        onClick={toggle}
      >
        Menu
      </button>

      {isOpen && (
        <ul id="dropdown-menu">
          <li>
            <button type="button" onClick={() => console.log('Item 1')}>
              Item 1
            </button>
          </li>
          <li>
            <button type="button" onClick={() => console.log('Item 2')}>
              Item 2
            </button>
          </li>
        </ul>
      )}
    </div>
  );
}

How this works:

• aria-expanded indicates whether the dropdown is open or closed

• aria-controls links the button to the dropdown content via its id

• The <button> element acts as the trigger and is fully keyboard accessible

• The <ul> and <li> elements provide a natural list structure

• Using <a> elements ensures proper navigation behavior and accessibility

Why this approach is correct:

• It follows standard web patterns instead of application-style menus

• It avoids misusing ARIA roles like role="menu", which require complex keyboard handling

• Screen readers can correctly interpret the structure without additional roles

• It keeps the implementation simple, accessible, and maintainable

If you need advanced menu behavior (like arrow key navigation), then ARIA menu roles may be appropriate – but only when fully implemented according to the ARIA Authoring Practices.

Note: Most dropdowns in web applications are not true "menus" in the ARIA sense. Avoid using role="menu" unless you are implementing full keyboard navigation (arrow keys, focus management, and so on).

Keyboard Navigation

Keyboard navigation ensures that users can fully interact with your application using only a keyboard, without relying on a mouse. This is essential for users with motor disabilities, but it also benefits power users and developers who prefer keyboard-based workflows.

In a well-designed interface, users should be able to:

• Navigate through interactive elements using the Tab key

• Activate buttons and links using Enter or Space

• Clearly see which element is currently focused

In the example below, we’ll look at common mistakes in keyboard handling and why relying on native HTML elements is usually the better approach.

Example:

Avoid adding custom keyboard handlers to native elements like <button>, as they already support keyboard interaction by default.

For example, this is all you need:

<button type="button" onClick={handleClick}>Submit</button>

This automatically supports:

• Enter and Space key activation

• Focus management

• Screen reader announcements

Adding manual keyboard event handlers here is unnecessary and can introduce bugs or inconsistent behavior.

What this example shows:

Avoid manually handling keyboard events for native interactive elements like <button>. These elements already provide built-in keyboard support and accessibility features.

For example:

<button type="button" onClick={handleClick}>Submit</button>

Why this works:

• Supports both Enter and Space key activation by default

• Is focusable and participates in natural tab order

• Provides built-in accessibility roles and screen reader announcements

• Reduces the need for additional logic or ARIA attributes

Adding custom keyboard handlers (like onKeyDown) to native elements is unnecessary and can introduce bugs or inconsistent behavior. Always prefer native HTML elements for interactivity whenever possible.

Avoiding Common Keyboard Traps

One of the most common keyboard accessibility issues is “trapping users inside interactive components”, such as modals or custom dropdowns. This happens when focus is moved into a component but can't escape using Tab, Shift+Tab, or other keyboard controls. Users relying on keyboards may become stuck, unable to navigate to other parts of the page.

In the example below, you'll see a simple modal that tries to set focus, but doesn’t manage Tab behavior properly.

function Modal({ isOpen }) {
  const ref = React.useRef();

  React.useEffect(() => {
    if (isOpen) ref.current?.focus();
  }, [isOpen]);

  return (
    <div role="dialog">
      <button type="button" ref={ref}>Close</button>
    </div>
  );
}

What this code shows:

• When the modal opens, focus is moved to the Close button using ref.current.focus()

• The modal uses role="dialog" to communicate its purpose

There are some issues with this code that you should be aware of. First, tabbing inside the modal may allow focus to move outside the modal if additional focusable elements exist.

Users may also become trapped if no mechanism returns focus to the triggering element when the modal closes.

There's also no handling of Shift+Tab or cycling focus is present.

This demonstrates a partial focus management, but it’s not fully accessible yet.

To improve focus management, you can trap focus within the modal by ensuring that Tab and Shift+Tab cycle only through elements inside the modal.

You can also return focus to the trigger: when the modal closes, return focus to the element that opened it.

Example improvement (conceptual):

function Modal({ isOpen, onClose, triggerRef }) {
  const modalRef = React.useRef();

  React.useEffect(() => {
    if (isOpen) {
      modalref.current?.focus();
      // Add focus trap logic here
    } else {
      triggerref.current?.focus();
    }
  }, [isOpen]);

  return (
    <div role="dialog" ref={modalRef} tabIndex={-1}>
      <button type="button" onClick={onClose}>Close</button>
    </div>
  );
}

Remember that this modal is not fully accessible without focus trapping. In production, use a library like focus-trap-react, react-aria, or Radix UI.

Key points:

• tabIndex={-1} allows the div to receive programmatic focus

• Focus trap ensures users cannot tab out unintentionally

• Returning focus preserves context, so users can continue where they left off

Best practices:

• Always move focus into modals

• Return focus to the trigger element when closed

• Ensure Tab cycles correctly

As a general rule, always prefer native HTML elements for interactivity. Only implement custom keyboard handling when building advanced components that cannot be achieved with standard elements.

Focus Management

Focus management is the practice of controlling where keyboard focus goes when users interact with components such as modals, forms, or interactive widgets. Proper focus management ensures that:

• Users relying on keyboards or assistive technologies can navigate seamlessly

• Focus does not get lost or trapped in unexpected places

• Users maintain context when content updates dynamically

The example below shows a common approach that only partially handles focus:

Bad Example:

// Bad Example: Automatically focusing input without context
const ref = React.useRef();
React.useEffect(() => {
  ref.current?.focus();
}, []);
<input ref={ref} placeholder="Name" />

In the above code, the input receives focus as soon as the component mounts, but there’s no handling for returning focus when the user navigates away.

If this input is inside a modal or dynamic content, users may get lost or trapped. There aren't any focus indicators or context for assistive technologies.

This is a minimal solution that can cause confusion in real applications.

Improved Example:

// Improved Example: Managing focus in a modal context
function Modal({ isOpen, onClose, triggerRef }) {  
const dialogRef = React.useRef();

  React.useEffect(() => {
    if (isOpen) {
      dialogRef.current?.focus();
    } else if (triggerRef?.current) {
      triggerref.current?.focus();
    }
  }, [isOpen]);

  React.useEffect(() => {
    function handleKeyDown(e) {
      if (e.key === 'Escape') {
        onClose();
      }
    }

    if (isOpen) {
      document.addEventListener('keydown', handleKeyDown);
    }

    return () => {
      document.removeEventListener('keydown', handleKeyDown);
    };
  }, [isOpen, onClose]);

  if (!isOpen) return null;

  return (
    <div
      role="dialog"
      aria-modal="true"
      aria-labelledby="modal-title"
      tabIndex={-1}
      ref={dialogRef}
    >
      <h2 id="modal-title">Modal Title</h2>
      <button type="button" onClick={onClose}>Close</button>
      <input type="text" placeholder="Name" />
    </div>
  );
}

Explanation:

• tabIndex={-1} enables the dialog container to receive focus

• Focus is moved to the modal when it opens, ensuring keyboard users start in the correct context

• Focus is returned to the trigger element when the modal closes, preserving user flow

• aria-labelledby provides an accessible name for the dialog

• Escape key handling allows users to close the modal without a mouse

Note: For full accessibility, you should also implement focus trapping so users cannot tab outside the modal while it is open.

Tip: In production applications, use libraries like react-aria, focus-trap-react, or Radix UI to handle focus trapping and accessibility edge cases reliably.

Also, keep in mind here that the document-level keydown listener is global, which affects the entire page and can conflict with other components.

document.addEventListener('keydown', handleKeyDown);

A safer alternative is to scope it to the modal:

<div
  onKeyDown={(e) => {
    if (e.key === 'Escape') onClose();
  }}
>

For simple cases, attach onKeyDown to the dialog instead of the document.

Best Practice:

For complex components, use libraries like focus-trap-react or react-aria to manage focus reliably, especially for modals, dropdowns, and popovers.

Forms and Accessibility

Forms are critical points of interaction in web applications, and proper accessibility ensures that all users – including those using screen readers or other assistive technologies – can understand and interact with them effectively.

Proper labeling means that every input field, checkbox, radio button, or select element has an associated label that clearly describes its purpose. This allows screen readers to announce the input meaningfully and helps keyboard-only users understand what information is expected.

In addition to labeling, form accessibility includes:

• Providing clear error messages when input is invalid

• Ensuring error messages are announced to assistive technologies

• Maintaining logical focus order so users can navigate inputs easily

Bad Example:

<input type="text" placeholder="Name" />

Why this isn't good:

• This input relies only on a placeholder for context

• Screen readers may not announce the purpose of the field clearly

• Once a user starts typing, the placeholder disappears, leaving no guidance

• Keyboard-only users may not have enough context to know what to enter

Good Example:

<label htmlFor="name">Name</label>
<input id="name" type="text" />

Why this is better:

• The <label> is explicitly associated with the input via htmlFor / id

• Screen readers announce "Name" before the input, providing clear context

• Users navigating with Tab understand the field’s purpose

• The label persists even when the user types, unlike a placeholder

Error Handling:

<label htmlFor="name">Name</label>
<input
  id="name"
  type="text"
  aria-describedby="name-error"
  aria-invalid="true"
/>

<p id="name-error" role="alert">
  Name is required
</p>

Explanation

• aria-describedby links the input to the error message using the element’s id

• Screen readers announce the error message when the input is focused

• aria-invalid="true" indicates that the field currently contains an error

• role="alert" ensures the error message is announced immediately when it appears

This creates a clear relationship between the input and its validation message, improving usability for screen reader users.

Tip: Only apply aria-invalid and error messages when validation fails. Avoid marking fields as invalid before user interaction.

Responsive Typography and Images

Responsive typography and images ensure that your content remains readable and visually appealing across a wide range of devices, from small smartphones to large desktop monitors.

This is important, because text should scale naturally so it remains legible on all screens, and images should adjust to container sizes to avoid layout issues or overflow. Both contribute to a better user experience and accessibility

In this section, we’ll cover practical ways to implement responsive typography and images in React and CSS.

h1 {
  font-size: clamp(1.5rem, 2vw, 3rem);
}

In this code:

• The clamp() function allows text to scale fluidly:

• The first value (1.5rem) is the “minimum font size”

• The second value (2vw) is the “preferred size based on viewport width”

• The third value (3rem) is the “maximum font size”

• This ensures headings are “readable on small screens” without becoming too large on desktops

Alternative methods include using media queries to adjust font sizes at different breakpoints

Responsive Images:

<img src="image.jpg" alt="Description" loading="lazy" />

In this code, responsive images adapt to different screen sizes and resolutions to prevent layout issues or slow loading times. Key techniques include:

1. Fluid images using CSS:

img {
     max-width: 100%;
     height: auto;
   }

This makes sure that images never overflow their container and maintains aspect ratio automatically.

2. Using srcset for multiple resolutions:

<img src="image-small.jpg"
     srcset="image-small.jpg 480w,
             image-medium.jpg 1024w,
             image-large.jpg 1920w"
     sizes="(max-width: 600px) 480px,
            (max-width: 1200px) 1024px,
            1920px"
     alt="Description">

This provides different image files depending on screen size or resolution and reduces loading times and improves performance on smaller devices.

3. Always include descriptive alt text

This is critical for screen readers and accessibility. It also helps users understand the image if it cannot be loaded.

Tip: Combine responsive typography, images, and flexible layout containers (like CSS Grid or Flexbox) to create interfaces that scale gracefully across all devices and maintain accessibility.

4. Ensure Sufficient Color Contrast

Low contrast text can make content unreadable for many users.

.bad-text {
  color: #aaa;
}

.good-text {
  color: #222;
}

Use tools like WebAIM Contrast Checker and Chrome DevTools Accessibility panel to check your color contrasts. Also note that WCAG AA requires 4.5:1 contrast ratio for normal text.

Building a Fully Accessible Responsive Component (End-to-End Example)

To understand how responsiveness and accessibility work together in practice, let’s build a reusable accessible card component that adapts to screen size and supports keyboard and screen reader users.

Step 1: Component Structure (Semantic HTML)

function ProductCard({ title, description, onAction }) {
  return (
    <article className="card">
      <h3>{title}</h3>
      <p>{description}</p>
      <button type="button" onClick={onAction}>
        View Details
      </button>
    </article>
  );
}

Why This Works

• <article> provides semantic meaning for standalone content

• <h3> establishes a proper heading hierarchy

• <button> ensures built-in keyboard and accessibility support

Step 2: Responsive Styling

.card {
  padding: 16px;
  border: 1px solid #ddd;
  border-radius: 8px;
}

@media (min-width: 768px) {
  .card {
    padding: 24px;
  }
}

This ensures comfortable spacing on mobile and improved readability on larger screens.

Step 3: Accessibility Enhancements

<button type="button" onClick={onAction}>
  View Details
</button>

The visible button text provides a clear and accessible label, so no additional ARIA attributes are needed.

Step 4: Keyboard Focus Styling

button:focus {
  outline: 2px solid blue;
  outline-offset: 2px;
}

Focus indicators are essential for keyboard users.

Step 5: Using the Component

function App() {
  return (
    <div className="grid">
      <ProductCard
        title="Product 1"
        description="Accessible and responsive"
        onAction={() => alert('Clicked')}
      />
    </div>
  );
}

Key Takeaways

This simple component demonstrates:

• Semantic HTML structure

• Responsive design

• Built-in accessibility via native elements

• Minimal ARIA usage

In real-world applications, this pattern scales into entire design systems.

Testing Accessibility

Accessibility should be validated continuously, not just at the end of development. There are various automated tools you can use to help you with this process:

• Lighthouse (built into Chrome DevTools)

• axe DevTools for detailed audits

• ESLint plugins for accessibility rules

Manual Testing

But automated tools cannot catch everything. Manual testing is essential to make sure users can navigate using only the keyboard and use a screen reader (NVDA or VoiceOver. You should also test zoom levels (up to 200%) and check the color contrast manually.

Example: ESLint Accessibility Plugin

npm install eslint-plugin-jsx-a11y --save-dev

This helps catch accessibility issues during development.

Best Practices

• Use semantic HTML first

• Avoid unnecessary ARIA

• Test keyboard navigation

• Design mobile-first

• Ensure color contrast

• Use consistent spacing

When NOT to Overuse Accessibility Features

• Avoid adding ARIA when native HTML works

• Do not override browser defaults unnecessarily

• Avoid complex custom components without accessibility support

Future Enhancements

• Design systems with accessibility built-in

• Automated accessibility testing in CI/CD

• Advanced focus management libraries

• Accessibility-first component libraries

Conclusion

Building responsive and accessible React applications is not a one-time effort—it is a continuous design and engineering practice. Instead of treating accessibility as a checklist, developers should integrate it into the core of their component design process.

If you are starting out, focus on using semantic HTML and mobile-first layouts. These two practices alone solve a large percentage of accessibility and responsiveness issues. As your application grows, introduce ARIA enhancements, keyboard navigation, and automated accessibility testing.

The key is to build interfaces that work for everyone by default. When responsiveness and accessibility are treated as first-class concerns, your React applications become more usable, scalable, and future-proof.

The process in this article needs to be done once per platform. Once you have the code, you can apply it every time to create a ToC. Note that if the platform changes something, you may need to adjust the script.

Table of Contents

• Browser Dev Tools

• JavaScript Console

• Understanding the DOM Structure

• Creating TOC in Markdown

• How to create an HTML TOC?

• Copy the HTML code for the editor

• What to do if I don’t have headers?

  • Create Table of Contents for DEV.to

• Conclusion

Browser Dev Tools

Dev Tools is an extension to the browser that can allow you to inspect and manipulate the DOM (Document Object Model), which is a representation of the HTML the browser keeps in memory in the form of a tree. It also gives access to the JavaScript console, where you can write short code snippets to test something. It has a lot more features, but we'll only use those two.

To open Dev Tools (in Google Chrome), you can press F12 or right-click on the page with your mouse and click Inspect.

Above is the screenshot of DevTools with a preview of this article. On the right, you can see a selected h1 HTML tag (the title) and CSS applied to that tag. The tree structure you see is the DOM.

JavaScript Console

We will need to have access to the JavaScript console. To open the console in Google Chrome, you can use F12, right-click on the page and select Inspect from the context menu, or use the shortcut CTRL+SHIFT+C (Windows, Linux) or CMD+OPTION+C (Mac).

In Chrome DevTools, you can pick the Console tab at the top of the DevTools. But this will hide the DOM tree. It’s better to open the bottom drawer. You need to click the 3 dots in the top right corner and pick “show console drawer”.

The Dev Tools will look like this:

The console is a so-called Read-Eval-Print-Loop. A classic interface, where you type some commands, here JavaScript code, and when you press enter, the code is executed in the context of the page the DevTools is on.

Above, you can see a page alert executed from the console.

Understanding the DOM Structure

The first step to create a ToC is to inspect the DOM and find the headers. They are usually H1…H6 tags. H1 is often the title of the page. In an ideal world, it would always be.

In my case, the header looks like this:

<h2 id="heading-dev-tools">Dev Tools</h2>

The article only has H2 tags, but later in the article, I will also explain how to create a nested ToC.

Now with DevTools, we can write code that will find every header:

document.querySelectorAll('h2[id], h3[id], main h4[id]');

In the case of my article on freeCodeCamp, it returned this output:

NodeList(5) [h2#heading-dev-tools, h2#heading-javascript-console, h2#heading-understanding-the-dom-structure, h2#trending-guides.col-header, h2#mobile-app.col-header]

First, it’s a NodeList that we need to convert to an Array. Second is that besides our headers that we have so far, we also have two headers that are part of the website and not the main content. So we need to find out the single element that is the parent of the headers we need.

You can right-click on the white page that contains the article and pick Inspect Element. In our case, it found an element <main>. So we can rewrite our selector as:

document.querySelectorAll('main h2[id], main h3[id], main h4[id]');

And now it returns our headers and nothing more.

How to Create the ToC in Markdown

A lot of blogging platforms support Markdown, so it'll be the first thing we'll create.

First, we'll convert the Node list to an array. We can use the spread operator:

[...document.querySelectorAll('main h2[id], main h3[id], main h4[id]')];

Then we can map over the array and create the Markdown links that point to the given header.

const headers = [...document.querySelectorAll('main h2[id], main h3[id], main h4[id]')];

headers.map(function(node) {
    // H2 header should have 0 indent
    const level = parseInt(node.nodeName.replace('H', '')) - 2;
    const hash = node.getAttribute('id');
    const indent = ' '.repeat(level * 2);
    return `${indent}* [{node.innerText}](#${hash})`;
});

The output looks like this:

[
  '* [Dev Tools](#heading-dev-tools)',
  '* [JavaScript Console](#heading-javascript-console)',
  '* [Understanding the DOM Structure](#heading-understanding-the-dom-structure)',
  '* [What to do if I don’t have headers?](#heading-what-to-do-if-i-dont-have-headers)'
]

To get the text, we can join the array with a newline character and use console.log to display the output. If we don’t use console.log, it will show a string with \n characters.

const headers = [...document.querySelectorAll('main h2[id], main h3[id], main h4[id]')];

console.log(headers.map(function(node) {
    // H2 header should have 0 indent
    const level = parseInt(node.nodeName.replace('H', '')) - 2;
    const hash = node.getAttribute('id');
    const indent = ' '.repeat(level * 2);
    return `${indent}* [{node.innerText}](#${hash})`;
}).join('\n'));

The output for this article will look like this:

* [Dev Tools](#heading-dev-tools)
* [JavaScript Console](#heading-javascript-console)
* [Understanding the DOM Structure](#heading-understanding-the-dom-structure)
* [Creating TOC in Markdown](#heading-creating-toc-in-markdown)
  * [This is fake header](#heading-this-is-fake-header)

I created one fake subheader. Platforms, even when not supporting Markdown when writing articles, often support Markdown when copy-pasted. The ToC at the top of the article was created by copying and pasting markdown generated with the last JavaScript snippet.

How to Create an HTML ToC

If your platform doesn’t support Markdown (like Medium), you can create HTML, preview that HTML, and copy the output to the clipboard. Pasting that into the editor of the platform you're using should keep the formatting.

To convert Markdown to HTML, you can use any online tool, but you'll see how to create it yourself in the snippet. It will be faster after you create the code.

const headers = [...document.querySelectorAll('main h2[id], main h3[id], main h4[id]')]

function indent(state) {
    return ' '.repeat((state.level - 1) * 2);
}

function closeUlTags(state, targetLevel) {
    while (state.level > targetLevel) {
        state.level--;
        state.lines.push(`${indent(state)}</ul>`);
    }
}

function openUlTags(state, targetLevel) {
    while (state.level < targetLevel) {
        state.lines.push(`${indent(state)}<ul>`);
        state.level++;
    }
}

const result = headers.reduce((state, node) => {
    const level = parseInt(node.nodeName.replace('H', ''));

    closeUlTags(state, level);
    openUlTags(state, level);
    
    const hash = node.getAttribute('id');
    state.lines.push(`${indent(state)}<li><a href="#${hash}">${node.innerText}</a></li>`);
    return state;
}, { lines: [], level: 1 });

closeUlTags(result, 1);

console.log(result.lines.join('\n'));

This is the output of the code in this article:

<ul>
  <li><a href="#heading-table-of-contents">Table of Contents</a></li>
  <li><a href="#heading-dev-tools">Dev Tools</a></li>
  <li><a href="#heading-javascript-console">JavaScript Console</a></li>
  <li><a href="#heading-understanding-the-dom-structure">Understanding the DOM Structure</a></li>
  <li><a href="#heading-creating-toc-in-markdown">Creating TOC in Markdown</a></li>
  <li><a href="#heading-how-to-create-html-toc">How to create HTML TOC</a></li>
  <ul>
    <li><a href="#heading-level-3">Level 3</a></li>
    <ul>
      <li><a href="#heading-level-4">Level 4</a></li>
    </ul>
  </ul>
  <li><a href="#heading-what-to-do-if-i-dont-have-headers">What to do if I don’t have headers?</a></li>
</ul>

I added a few headers at the end, so you can see that it will work for any level of nested headers. Note that we also have the ToC as the first element on the list.

Copy the HTML code for the editor

Most so-called WYSIWYG editors are using HTML, and you should be able to copy the output of HTML code with formatting and paste it into that editor. The easiest is to just save that into a file, open that file, and select the text:

What to Do If I Don’t Have Headers?

You need to find anything that can be targeted with CSS. If they are p tags with a specific class (like header), you can use p.header instead of h2.

How to Create a Table of Contents for DEV.to

If you have a different DOM structure, you can use different DOM methods to extract the element you need. For example, on DEV.to, the headers look like this:

<h2>
  <a name="overview" href="#overview">
  </a>
  Overview
</h2>

So the selector needs to be just main h2. But when you execute this code:

[...document.querySelectorAll('main h2, main h3, main h4')];

You will see that there are way more headers than the content of the document. Luckily, we can use a new selector in CSS :has(). The final selector for one header can look like this: main h2:has(a[name]).

Here is the full code:

const selector = 'main h2:has(a[name]), main h3:has(a[name]), main h4:has(a[name])';
const headers = [...document.querySelectorAll(selector)];

console.log(headers.map(function(node) {
    // H2 header should have 0 indent
    const level = parseInt(node.nodeName.replace('H', '')) - 2;
    // this is how you get the hash
    // you can also access href attribute and remove # from the output string
    const hash = node.querySelector('a').getAttribute('name');
    const indent = ' '.repeat(level);
    return `${indent}* [{node.innerText}](#${hash})`;
}).join('\n'));

Conclusion

Creating a table of contents can help your readers digest your article. Since most people don’t read the whole article, they only scan for what they need. You can also find a lot of articles about its impact on SEO. So it’s always worth adding one if the article is longer.

And as you can see, creating a ToC is not that hard with a bit of web development knowledge.

If you like this article, you may want to follow me on Social Media: (Twitter/X, GitHub, and/or LinkedIn). You can also check my personal website and my new blog.

This article is intentionally vendor-neutral. You can implement these patterns using any AI voice platform that supports WebRTC (directly or via an SFU, selective forwarding unit) and server-side token minting. The goal is to help you ship a voice agent architecture that is secure, observable, and operable in production.

Disclosure: This article reflects my personal views and experience. It does not represent the views of my employer or any vendor mentioned.

Table of Contents

• What You'll Build

• How to Avoid Common Production Failures in Voice Agents

• How to Design a Latency Budget for a Real-Time Voice Agent

• Production Voice Agent Architecture (Vendor-Neutral)

  • Step 0: Set Up the Project

  • Step 1: Keep Credentials Server-side

  • Step 2: Build a Backend Token Endpoint

  • Step 3: Connect from the Web Client (WebRTC + SFU)

  • Step 4: Add Client Actions (Agent Suggests, App Executes)

  • Step 5: Add Tool Integrations Safely

  • Step 6: Add post-call processing (where durable value appears)

• Production readiness checklist

• Closing

What You'll Build

By the end, you'll have:

• A web client that streams microphone audio and plays agent audio.

• A backend token endpoint that keeps credentials server-side.

• A safe coordination channel between the agent and the application.

• Structured messages between the application and the agent.

• A production checklist for security, reliability, observability, and cost control.

Prerequisites

You should be comfortable with:

• JavaScript or TypeScript

• Node.js 18+ (so fetch works server-side) and an HTTP framework (Express in examples)

• Browser microphone permissions

• Basic WebRTC concepts (high level is fine)

TL;DR

A production-ready voice agent needs:

• A server-side token service (no secrets in the browser)

• A real-time media plane (WebRTC) for low-latency audio

• A data channel for structured messages between your app and the agent

• Tool guardrails (allowlists, confirmations, timeouts, audit logs)

• Post-call processing (summary, actions, CRM (Customer Relationship Management), tickets)

• Observability-first implementation (state transitions + metrics)

How to Avoid Common Production Failures in Voice Agents

If you've operated distributed systems, you've seen most failures happen at boundaries:

• timeouts and partial connectivity

• retries that amplify load

• unclear ownership between components

• missing observability

• “helpful automation” that becomes unsafe

Voice agents amplify those risks because:

Latency is User Experience: A slow agent feels broken. Conversational UX is less forgiving than web UX.

Audio + UI + Tools is a Distributed System: You coordinate browser audio capture, WebRTC transport, STT (speech-to-text), model reasoning, tool calls, TTS (text-to-speech), and playback buffering. Each stage has different clocks and failure modes.

Security Boundaries are Non-negotiable: A leaked API key is catastrophic. A tool misfire can trigger real-world side effects.

Debuggability determines whether you can ship: If you don't log state transitions and capture post-call artifacts, you can't operate or improve the system safely.

How to Design a Latency Budget for a Real-Time Voice Agent

Conversations have a “feel.” That feel is mostly latency.

A practical guideline:

• Under ~200ms feels instant

• 300–500ms feels responsive

• Over ~700ms feels broken

Your end-to-end latency is the sum of mic capture, network RTT (round-trip time), STT, reasoning, tool execution, TTS, and playback buffering. Budget for it explicitly or you’ll ship a technically correct system that users perceive as unintelligent.

How to Design a Production Voice Agent Architecture (Vendor-Neutral)

A scalable voice agent architecture typically has these layers:

1. Web client: mic capture, audio playback, UI state

2. Token service: short-lived session tokens (secrets stay server-side)

3. Real-time plane: WebRTC media + a data channel

4. Agent runtime: STT → reasoning → TTS, plus tool orchestration

5. Tool layer: external actions behind safety controls

6. Post-call processor: summary + structured outputs after the session ends

This separation makes failure domains and trust boundaries explicit.

Step 0: Set Up the Project

Create a new project directory:

mkdir voice-agent-app
cd voice-agent-app
npm init -y
npm pkg set type=module
npm pkg set scripts.start="node server.js"

Install dependencies:

npm install express dotenv

Create this folder structure:

voice-agent-app/
├── server.js
├── .env
└── public/
    ├── index.html
    └── client.js

Add a .env file:

VOICE_PLATFORM_URL=https://your-provider.example
VOICE_PLATFORM_API_KEY=your_api_key_here

Now you’re ready to implement each part of the system.

Step 1: Keep Credentials Server-side

Treat every API key like production credentials:

• store it in environment variables or a secrets manager

• rotate it if exposed

• never embed it in browser or mobile apps

• avoid logging secrets (log only a short suffix if necessary)

Even if a vendor supports CORS, the browser is not a safe place for long-lived credentials.

Step 2: Build a Backend Token Endpoint

Your backend should:

• authenticate the user

• mint a short-lived session token using your platform API

• return only what the client needs (URL + token + expiry)

Create server.js (Node.js + Express)

import express from "express";
import dotenv from "dotenv";
import path from "path";
import { fileURLToPath } from "url";

dotenv.config();

const app = express();
app.use(express.json());

// Serve the web client from /public
const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);
app.use(express.static(path.join(__dirname, "public")));

const VOICE_PLATFORM_URL = process.env.VOICE_PLATFORM_URL;
const VOICE_PLATFORM_API_KEY = process.env.VOICE_PLATFORM_API_KEY;

app.post("/api/voice-token", async (req, res) => {
  res.setHeader("Cache-Control", "no-store");

  try {
    if (!VOICE_PLATFORM_URL || !VOICE_PLATFORM_API_KEY) {
      return res.status(500).json({
        error: "Missing VOICE_PLATFORM_URL or VOICE_PLATFORM_API_KEY in .env",
      });
    }

    // TODO: Authenticate the caller before minting tokens.

    const r = await fetch(`${VOICE_PLATFORM_URL}/api/v1/token`, {
      method: "POST",
      headers: {
        "X-API-Key": VOICE_PLATFORM_API_KEY,
        "Content-Type": "application/json",
      },
      body: JSON.stringify({ participant_name: "Web User" }),
    });

    if (!r.ok) {
      const detail = await r.text().catch(() => "");
      return res.status(r.status).json({ error: "Token request failed", detail });
    }

    const data = await r.json();

    res.json({
      rtc_url: data.rtc_url || data.livekit_url,
      token: data.token,
      expires_in: data.expires_in,
    });
  } catch (err) {
    res.status(500).json({ error: "Failed to mint token" });
  }
});

app.listen(3000, () => console.log("Open http://localhost:3000"));

Run the server

npm start

Then open: http://localhost:3000

How this code works

• You load credentials from environment variables so secrets never enter the browser.

• The /api/voice-token endpoint calls the voice platform’s token API.

• You return only the rtc_url, token, and expiration time.

• The browser never sees the API key.

• If the provider returns an error, you forward a structured error response.

Production Notes

• rate-limit /api/voice-token (cost + abuse control)

• instrument token mint latency and error rate

• keep TTL short and handle refresh/reconnect

• return minimal fields

Step 3: Connect from the Web Client (WebRTC + SFU)

In this step, you'll build a minimal web UI that:

• Requests a short-lived token from your backend

• Connects to a real-time WebRTC room (often via an SFU)

• Plays the agent's audio track

• Captures and publishes microphone audio

Create public/index.html

<!doctype html>
<html>
  <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width,initial-scale=1" />
    <title>Voice Agent Demo</title>
  </head>
  <body>
    <h1>Voice Agent Demo</h1>

    <button id="startBtn">Start Call</button>
    <button id="endBtn" disabled>End Call</button>

    <p id="status">Idle</p>

    <script type="module" src="/client.js"></script>
  </body>
</html>

Create public/client.js

Note: This uses a LiveKit-style client SDK to demonstrate the pattern. If you're using a different provider, swap this import and the connect/publish calls for your provider's WebRTC client.

import { Room, RoomEvent, Track } from "https://unpkg.com/livekit-client@2.10.1/dist/livekit-client.esm.mjs";

const startBtn = document.getElementById("startBtn");
const endBtn = document.getElementById("endBtn");
const statusEl = document.getElementById("status");

let room = null;
let intentionallyDisconnected = false;
let audioEls = [];

function setStatus(text) {
  statusEl.textContent = text;
}

function detachAllAudio() {
  for (const el of audioEls) {
    try { el.pause?.(); } catch {}
    el.remove();
  }
  audioEls = [];
}

async function mintToken() {
  const res = await fetch("/api/voice-token", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ participant_name: "Web User" }),
    cache: "no-store",
  });

  if (!res.ok) {
    const detail = await res.text().catch(() => "");
    throw new Error(`Token request failed: ${detail || res.status}`);
  }

  const { rtc_url, token } = await res.json();
  if (!rtc_url || !token) throw new Error("Token response missing rtc_url or token");
  return { rtc_url, token };
}

function wireRoomEvents(r) {
  // 1) Play the agent audio track when subscribed
  r.on(RoomEvent.TrackSubscribed, (track) => {
    if (track.kind !== Track.Kind.Audio) return;

    const el = track.attach();
    audioEls.push(el);
    document.body.appendChild(el);

    // Autoplay restrictions vary by browser/device.
    el.play?.().catch(() => {
      setStatus("Connected (audio may be blocked — click the page to enable)");
    });
  });

  // 2) Reconnect on disconnect (token expiry often shows up this way)
  r.on(RoomEvent.Disconnected, async () => {
    if (intentionallyDisconnected) return;
    setStatus("Disconnected (reconnecting...)");
    await attemptReconnect();
  });
}

async function connectOnce() {
  const { rtc_url, token } = await mintToken();

  const r = new Room();
  wireRoomEvents(r);

  await r.connect(rtc_url, token);

  // Mic permission + publish mic
  try {
    await r.localParticipant.setMicrophoneEnabled(true);
  } catch {
    try { r.disconnect(); } catch {}
    throw new Error("Microphone access denied. Allow mic permission and try again.");
  }

  return r;
}

async function startCall() {
  if (room) return;

  intentionallyDisconnected = false;
  setStatus("Connecting...");

  room = await connectOnce();

  setStatus("Connected");
  startBtn.disabled = true;
  endBtn.disabled = false;
}

async function stopCall() {
  intentionallyDisconnected = true;

  try {
    await room?.localParticipant?.setMicrophoneEnabled(false);
  } catch {}

  try {
    room?.disconnect();
  } catch {}

  room = null;
  detachAllAudio();

  setStatus("Disconnected");
  startBtn.disabled = false;
  endBtn.disabled = true;
}

async function attemptReconnect() {
  // Simplified exponential backoff reconnect.
  // In production, add jitter, max attempts, and better error classification.
  const delaysMs = [250, 500, 1000, 2000];

  for (const delay of delaysMs) {
    if (intentionallyDisconnected) return;

    try {
      // Tear down current state before reconnecting
      try { room?.disconnect(); } catch {}
      room = null;
      detachAllAudio();

      await new Promise((r) => setTimeout(r, delay));

      room = await connectOnce();
      setStatus("Reconnected");
      startBtn.disabled = true;
      endBtn.disabled = false;
      return;
    } catch {
      // keep retrying
    }
  }

  setStatus("Disconnected (reconnect failed)");
  startBtn.disabled = false;
  endBtn.disabled = true;
}

startBtn.addEventListener("click", async () => {
  try {
    await startCall();
  } catch (err) {
    setStatus(err?.message || "Connection failed");
    startBtn.disabled = false;
    endBtn.disabled = true;
    room = null;
    detachAllAudio();
  }
});

endBtn.addEventListener("click", async () => {
  await stopCall();
});

How this Step works (and why these details matter)

• The Start button gives you a user gesture so browsers are more likely to allow audio playback.

• Mic permission is handled explicitly: if the user denies access, you show a clear error and avoid a half-connected session.

• Disconnect cleanup removes audio elements so you don't leak resources across retries.

• The reconnect loop demonstrates the production pattern: if a disconnect happens (often due to token expiry or network churn), the client re-mints a token and reconnects.

In the next step, you'll add a structured data-channel handler to safely process agent-suggested “client actions”.

Handle These Explicitly

Autoplay Restriction Example

Add this to index.html:

<button id="startBtn">Start Call</button>
<button id="endBtn" disabled>End Call</button>
<div id="status"></div>

In client.js:

const startBtn = document.getElementById("startBtn");
const endBtn = document.getElementById("endBtn");
const statusEl = document.getElementById("status");

let room;

startBtn.addEventListener("click", async () => {
  try {
    room = await connectVoice();
    statusEl.textContent = "Connected";
    startBtn.disabled = true;
    endBtn.disabled = false;
  } catch (err) {
    statusEl.textContent = "Connection failed";
  }
});

Microphone denial

try {
  await navigator.mediaDevices.getUserMedia({ audio: true });
} catch (err) {
  statusEl.textContent = "Microphone access denied";
  throw err;
}

Disconnect cleanup

endBtn.addEventListener("click", () => {
  if (room) {
    room.disconnect();
    statusEl.textContent = "Disconnected";
    startBtn.disabled = false;
    endBtn.disabled = true;
  }
});

Token refresh (simplified)

room.on(RoomEvent.Disconnected, async () => {
  const res = await fetch("/api/voice-token");
  const { rtc_url, token } = await res.json();
  await room.connect(rtc_url, token);
});

Step 4: Add Client Actions (Agent Suggests, App Executes)

A production voice agent often needs to:

• open a runbook/dashboard URL

• show a checklist in the UI

• request confirmation for an irreversible action

• receive structured context (account, region, incident ID)

The key safety rule:

The agent suggests actions. The application validates and executes them.

Use structured messages over the data channel:

{
  "type": "client_action",
  "action": "open_url",
  "payload": { "url": "https://internal.example.com/runbook" },
  "id": "action_123"
}

Add guardrails:

• allowlist permitted actions

• validate payload shape

• confirmation gates for irreversible actions

• idempotency via id

• audit logs for every request and outcome

This boundary limits damage from hallucinations or prompt injection.

// Guardrails: allowlist + validation + idempotency + confirmation

const ALLOWED_ACTIONS = new Set(["open_url", "request_confirm"]);
const EXECUTED_ACTION_IDS = new Set();
const ALLOWED_HOSTS = new Set(["internal.example.com"]);

function parseClientAction(text) {
  let msg;
  try {
    msg = JSON.parse(text);
  } catch {
    return null;
  }

  if (msg?.type !== "client_action") return null;
  if (typeof msg.id !== "string") return null;
  if (!ALLOWED_ACTIONS.has(msg.action)) return null;

  return msg;
}

async function handleClientAction(msg, room) {
  if (EXECUTED_ACTION_IDS.has(msg.id)) return; // idempotency
  EXECUTED_ACTION_IDS.add(msg.id);

  console.log("[client_action]", msg); // audit log (demo)

  if (msg.action === "open_url") {
    const url = msg.payload?.url;
    if (typeof url !== "string") return;

    const u = new URL(url);
    if (!ALLOWED_HOSTS.has(u.host)) {
      console.warn("Blocked navigation to:", u.host);
      return;
    }

    window.open(url, "_blank", "noopener,noreferrer");
    return;
  }

  if (msg.action === "request_confirm") {
    const prompt = msg.payload?.prompt || "Confirm this action?";
    const ok = window.confirm(prompt);

    // Send confirmation back to agent/app
    room.localParticipant.publishData(
  new TextEncoder().encode(
    JSON.stringify({ type: "user_confirmed", id: msg.id, ok })
  ),
  { topic: "client_events", reliable: true }
);
  }
}

room.on(RoomEvent.DataReceived, (payload, participant, kind, topic) => {
  if (topic !== "client_actions") return;

  const text = new TextDecoder().decode(payload);
  const msg = parseClientAction(text);
  if (!msg) return;

  handleClientAction(msg, room);
});

Step 5: Add Tool Integrations Safely

Tools turn a voice agent into automation. Regardless of vendor, enforce these rules:

• timeouts on every tool call

• circuit breakers for flaky dependencies

• audit logs (inputs, outputs, duration, trace IDs)

• explicit confirmation for destructive actions

• credentials stored server-side (never in prompts or clients)

If tools fail, degrade gracefully (“I can’t access that system right now, here’s the manual fallback.”). Silence reads as failure.

Create a server-side tool runner (example)

Paste this into server.js:

const TOOL_ALLOWLIST = {
  get_status: { destructive: false },
  create_ticket: { destructive: true },
};

let failures = 0;
let circuitOpenUntil = 0;

function circuitOpen() {
  return Date.now() < circuitOpenUntil;
}

async function withTimeout(promise, ms) {
  return Promise.race([
    promise,
    new Promise((_, reject) => setTimeout(() => reject(new Error("timeout")), ms)),
  ]);
}

async function runToolSafely(tool, args) {
  if (circuitOpen()) throw new Error("circuit_open");

  try {
    const result = await withTimeout(Promise.resolve({ ok: true, tool, args }), 2000);
    failures = 0;
    return result;
  } catch (err) {
    failures++;
    if (failures >= 3) circuitOpenUntil = Date.now() + 10_000;
    throw err;
  }
}

app.post("/api/tools/run", async (req, res) => {
  const { tool, args, user_confirmed } = req.body || {};

  if (!TOOL_ALLOWLIST[tool]) return res.status(400).json({ error: "Tool not allowed" });

  if (TOOL_ALLOWLIST[tool].destructive && user_confirmed !== true) {
    return res.status(400).json({ error: "Confirmation required" });
  }

  try {
    const started = Date.now();
    const result = await runToolSafely(tool, args);
    console.log("[tool_call]", { tool, ms: Date.now() - started }); // audit log
    res.json({ ok: true, result });
  } catch (err) {
    console.log("[tool_error]", { tool, err: String(err) });
    res.status(500).json({ ok: false, error: "Tool call failed" });
  }
});

Step 6: Add post-call processing (where durable value appears)

After a call ends, generate structured artifacts:

• summary

• action items

• follow-up email draft

• CRM entry or ticket creation

A production pattern:

• store transcript + metadata

• enqueue a background job (queue/worker)

• produce outputs as JSON + a human-readable report

• apply integrations with retries + idempotency

• store a “call report” for audits and incident reviews

Create a post-call webhook endpoint (example)

Paste into server.js:

app.post("/webhooks/call-ended", async (req, res) => {
  const payload = req.body;

  console.log("[call_ended]", {
    call_id: payload.call_id,
    ended_at: payload.ended_at,
  });

  setImmediate(() => processPostCall(payload));
  res.json({ ok: true });
});

function processPostCall(payload) {
  const transcript = payload.transcript || [];
  const summary = transcript.slice(0, 3).map(t => `- \({t.speaker}: \){t.text}`).join("\n");

  const report = {
    call_id: payload.call_id,
    summary,
    action_items: payload.action_items || [],
    created_at: new Date().toISOString(),
  };

  console.log("[call_report]", report);
}

Test it locally

curl -X POST http://localhost:3000/webhooks/call-ended \
  -H "Content-Type: application/json" \
  -d '{
    "call_id": "call_123",
    "ended_at": "2026-02-26T00:10:00Z",
    "transcript": [
      {"speaker": "user", "text": "I need help resetting my password."},
      {"speaker": "agent", "text": "Sure — I can help with that."}
    ],
    "action_items": ["Send password reset link", "Verify account email"]
  }'

Production readiness checklist

Security

• no API keys in the browser

• strict allowlist for client actions

• confirmation gates for destructive actions

• schema validation on all inbound messages

• audit logging for actions and tool calls

Reliability

• reconnect strategy for expired tokens

• timeouts + circuit breakers for tools

• graceful degradation when dependencies fail

• idempotent side effects

Observability

Log state transitions (for example):
listening → thinking → speaking → ended

Track:

• connect failure rate

• end-to-end latency (STT + reasoning + TTS)

• tool error rate

• reconnect frequency

Cost control

• rate-limit token minting and sessions

• cap max call duration

• bound context growth (summarize or truncate)

• track per-call usage drivers (STT/TTS minutes, tool calls)

Optional resources

How to Try a Managed Voice Platform Quickly

If you want a managed provider to test quickly, you can sign up for a Vocal Bridge account and implement these steps using their token minting + real-time session APIs.

But the core production voice agent architecture in this article is vendor-agnostic. You can replace any component (SFU, STT/TTS, agent runtime, tool layer) as long as you preserve the boundaries: secure token service, real-time media, safe tool execution, and strong observability.

Watch a full demo and explore a complete reference repo

If you'd like to see these patterns working together in a realistic scenario (incident triage), here are two optional resources:

- Demo video: Voice-First Incident Triage (end-to-end run)
This is a hackathon run-through showing client actions, decision boundaries for irreversible actions, and a structured post-call summary.

- GitHub repo (architecture + design + working code): https://github.com/natarajsundar/voice-first-incident-triage

These links are optional, you can follow the tutorial end-to-end without them.

Closing

Production-ready voice agents work when you treat them like real-time distributed systems.

Start with the baseline:

• token service + web client + real-time audio

Then layer in:

• controlled client actions

• safe tools

• post-call automation

• observability and cost controls

That’s how you ship a voice agent architecture you can operate. You now have a vendor-neutral reference architecture you can adapt to your stack, with clear trust boundaries, safe tool execution, and operational visibility.

If you’re shipping real-time AI systems, what’s been your biggest production bottleneck so far: latency, reliability, or tool safety? I’d love to hear what you’re seeing in the wild. Connect with me on LinkedIn.

This article provides a comprehensive, production-focused guide to supporting multiple languages in a Flutter application using Flutter’s localization system, the intl package, and Bloc for state management. We’ll support English, French, and Spanish, implement automatic language detection, and allow users to manually switch languages from settings, while also exploring the use of AI to automate text translations.

Table of Contents

• Prerequisites

• Why Localization Matters in Flutter Applications

• Flutter Localization Architecture Overview

• How to Set Up Dependencies

• How to Define Supported Languages

• How to Add Localized Text with ARB Files

• How to Generate Localization Code

• How to Configure MaterialApp for Localization

• Auto-Detecting the User’s Device Language

• How to Manage Localization with Bloc

• How to Display Localized Text in Widgets

• Language Switching from Settings

• How to Add Parameters to Localized Strings

• Pluralization and Quantities

• How to Format Dates, Numbers, and Currency

• Localization Data Flow

• Common Pitfalls and How to Avoid Them

• How to Automate Translations with AI

• Best Practices and Considerations

• Conclusion

• References

Prerequisites

Before proceeding, you should be comfortable with the following concepts:

• Dart programming language: variables, classes, functions, and null safety

• Flutter fundamentals: widgets, BuildContext, and widget trees

• State management basics: familiarity with Bloc or similar patterns

• Terminal usage: running Flutter CLI commands

If you have prior experience working with Flutter widgets and basic app architecture, you are well prepared to follow along.

Why Localization Matters in Flutter Applications

Localization (often abbreviated as l10n) is the process of adapting an application for different languages and regions, going beyond simple text translation to influence accessibility, user trust, and overall usability. From a technical perspective, localization introduces several challenges: text must be dynamically resolved at runtime, the UI must update instantly when the language changes, language preferences must persist across sessions, and device locale detection must gracefully fall back when a language is unsupported.

Flutter’s localization framework, when combined with intl and Bloc, solves these challenges cleanly and predictably.

Flutter Localization Architecture Overview

Flutter localization is built around three key ideas:

1. ARB files as the source of truth for translated strings

2. Code generation to provide type-safe access to translations

3. Locale-driven rebuilds of the widget tree

At runtime, the active Locale determines which translation file is used. When the locale changes, Flutter automatically rebuilds dependent widgets.

How to Set Up Dependencies

Add the required dependencies to your pubspec.yaml:

dependencies:
  flutter:
    sdk: flutter

  flutter_localizations:
    sdk: flutter

  intl: ^0.20.2
  flutter_bloc: ^8.1.3
  arb_translate: ^1.1.0

Enable localization code generation:

flutter:
  generate: true

This instructs Flutter to generate localization classes from ARB files.

How to Define Supported Languages

For this guide, the application will support:

• English (en)

• French (fr)

• Spanish (es)

These locales will be declared centrally and used throughout the app.

How to Add Localized Text with ARB Files

Flutter uses Application Resource Bundle (ARB) files to store localized strings. Each supported language has its own ARB file.

English – app_en.arb

{
  "@@locale": "en",
  "enter_email_address_to_reset": "Enter your email address to reset"
}

French – app_fr.arb

{
  "@@locale": "fr",
  "enter_email_address_to_reset": "Entrez votre adresse e-mail pour réinitialiser"
}

Spanish – app_es.arb

{
  "@@locale": "es",
  "enter_email_address_to_reset": "Ingrese su dirección de correo electrónico para restablecer"
}

Each key must be identical across files. Only the values change per language.

How to Generate Localization Code

Run the following command in your terminal:

flutter gen-l10n

Flutter generates a strongly typed localization class, typically located at:

.dart_tool/flutter_gen/gen_l10n/app_localizations.dart

This file exposes getters such as:

AppLocalizations.of(context)!.enter_email_address_to_reset

How to Configure MaterialApp for Localization

The MaterialApp widget must be configured with localization delegates and supported locales:

MaterialApp(
  localizationsDelegates: const [
    AppLocalizations.delegate,
    GlobalMaterialLocalizations.delegate,
    GlobalWidgetsLocalizations.delegate,
    GlobalCupertinoLocalizations.delegate,
  ],
  supportedLocales: const [
    Locale('en'),
    Locale('fr'),
    Locale('es'),
  ],
  locale: state.locale,
  home: const MyHomePage(),
)

The locale property is controlled by Bloc, allowing dynamic updates at runtime.

Auto-Detecting the User’s Device Language

Flutter exposes the device locale via PlatformDispatcher. We can use this to automatically select the most appropriate supported language.

void detectLanguageAndSet() {
  Locale deviceLocale = PlatformDispatcher.instance.locale;

  Locale selectedLocale = AppLocalizations.supportedLocales.firstWhere(
    (supported) => supported.languageCode == deviceLocale.languageCode,
    orElse: () => const Locale('en'),
  );

  print('Using Locale: ${selectedLocale.languageCode}');

  GlobalConfig.storageService.setStringValue(
    AppStrings.DETECTED_LANGUAGE,
    selectedLocale.languageCode,
  );

  context.read<AppLocalizationBloc>().add(
    SetLocale(locale: selectedLocale),
  );
}

This approach reads the device language, matches it against supported locales, falls back to English when the language is unsupported, persists the detected language, and updates the UI instantly.

How to Manage Localization with Bloc

Bloc provides a predictable and testable way to manage application-wide locale changes.

Localization State

class AppLocalizationState {
  final Locale locale;
  const AppLocalizationState(this.locale);
}

Localization Event

abstract class AppLocalizationEvent {}

class SetLocale extends AppLocalizationEvent {
  final Locale locale;
  SetLocale({required this.locale});
}

Localization Bloc

class AppLocalizationBloc
    extends Bloc<AppLocalizationEvent, AppLocalizationState> {
  AppLocalizationBloc()
      : super(const AppLocalizationState(Locale('en'))) {
    on<SetLocale>((event, emit) {
      emit(AppLocalizationState(event.locale));
    });
  }
}

The AppLocalizationBloc manages the app’s language state. It starts with English (Locale('en')) as the default, and when it receives a SetLocale event, it updates the state to the new locale provided in the event, causing the app’s UI to switch to that language. Whenever SetLocale is dispatched, the entire app rebuilds using the new locale.

How to Display Localized Text in Widgets

Once localization is configured, using translated text is straightforward:

Text(
  AppLocalizations.of(context)!.enter_email_address_to_reset,
  style: getRegularStyle(
    color: Colors.white,
    fontSize: FontSize.s16,
  ),
)

AppLocalizations.of(context)!.enter_email_address_to_reset retrieves the localized string enter_email_address_to_reset for the current app locale from the generated localization resources. The correct translation is resolved automatically based on the active locale.

Language Switching from Settings

Users should always be able to override automatic language detection.

ListTile(
  title: const Text('French'),
  onTap: () {
    context.read<AppLocalizationBloc>().add(
      SetLocale(locale: const Locale('fr')),
    );
  },
)

This ListTile displays the text "French", and when tapped, it triggers the AppLocalizationBloc to change the app’s locale to French ('fr') by dispatching a SetLocale event and it persists the selected language so it can be restored on the next app launch.

How to Add Parameters to Localized Strings

Real-world applications rarely display static text. Messages often include dynamic values such as user names, counts, dates, or prices. Flutter’s localization system, powered by intl, supports parameterized (interpolated) strings in a type-safe way.

Where Parameters Are Defined

Parameters are defined inside ARB files alongside the localized string itself, with each parameterized message consisting of the message string containing placeholders and a corresponding metadata entry that describes those placeholders.

Example: Parameterized Text

Suppose we want to display a greeting message that includes a user’s name.

English – app_en.arb

{
  "@@locale": "en",
  "greetingMessage": "Hello {username}!",
  "@greetingMessage": {
    "description": "Greeting message shown on the home screen",
    "placeholders": {
      "username": {
        "type": "String"
      }
    }
  }
}

This defines a parameterized localized message for English, indicated by "@@locale": "en". The "greetingMessage" key contains the string "Hello {username}!", where {username} is a placeholder that will be dynamically replaced with the user’s name at runtime. The "@greetingMessage" entry provides metadata for the message, including a description that explains the string is shown on the home screen, and a "placeholders" section that specifies "username" is of type String. When the app runs, this structure allows the message to display dynamically—for example, if the username is "Alice", the message would appear as "Hello Alice!".

French – app_fr.arb

{
  "@@locale": "fr",
  "greetingMessage": "Bonjour {username} !"
}

Spanish – app_es.arb

{
  "@@locale": "es",
  "greetingMessage": "¡Hola {username}!"
}

The placeholder name ({username}) must be identical across all ARB files.

Generated Dart API

After running:

flutter gen-l10n

Flutter generates a strongly typed method instead of a simple getter:

String greetingMessage(String username)

This prevents runtime errors and ensures compile-time safety.

How to Use Parameterized Strings in Widgets

Text(
  AppLocalizations.of(context)!.greetingMessage('Tony'),
)

If the locale is set to French, the output becomes:

Bonjour Tony !

Pluralization and Quantities

Another common localization requirement is pluralization. Languages differ significantly in how they express quantities, and hardcoding plural logic in Dart quickly becomes error-prone.

Defining Plural Messages in ARB

{
  "itemsCount": "{count, plural, =0{No items} =1{1 item} other{{count} items}}",
  "@itemsCount": {
    "description": "Displays the number of items",
    "placeholders": {
      "count": {
        "type": "int"
      }
    }
  }
}

This defines a pluralized message for itemsCount. The string {count, plural, =0{No items} =1{1 item} other{{count} items}} dynamically changes based on the value of count: it shows "No items" when count is 0, "1 item" when count is 1, and "{count} items" for all other values. The metadata entry "@itemsCount" provides a description and specifies that the placeholder count is of type int.

Each language can define its own plural rules while sharing the same key.

Using Pluralized Messages

Text(
  AppLocalizations.of(context)!.itemsCount(3),
)

Flutter automatically applies the correct plural form based on the active locale.

How to Format Dates, Numbers, and Currency

The intl package also provides locale-aware formatting utilities. These should be used in combination with localized strings, not as replacements.

Date Formatting Example

final formattedDate = DateFormat.yMMMMd(
  Localizations.localeOf(context).toString(),
).format(DateTime.now());

Text(
  AppLocalizations.of(context)!.lastLoginDate(formattedDate),
)

This ensures that both language and formatting rules align with the user’s locale.

Localization Data Flow

Localization is handled as an explicit data flow, with locale resolution modeled as application state rather than a static configuration passed into MaterialApp.

The process starts with the device locale, obtained from the platform layer at startup. This value represents the system’s preferred language and region but is not applied directly to the UI.

Instead, it flows through a detectLanguageAndSet step responsible for applying application-specific rules. This layer typically handles locale normalization and fallback logic, such as mapping unsupported locales to supported ones, restoring a user-selected language from persistent storage, or enforcing product constraints around available translations.

The resolved locale is then emitted into a Localization Bloc, which acts as the single source of truth for localization state. By centralizing locale management, the application can support runtime language changes, ensure predictable rebuilds, and keep localization logic decoupled from both the widget tree and platform APIs.

The Bloc feeds into the locale property of MaterialApp, which is the integration point with Flutter’s localization system. Updating this value triggers a rebuild of the Localizations scope and causes all dependent widgets to resolve strings for the active locale.

At the edge of the system, localized widgets consume the generated localization classes produced by flutter gen-l10n. These widgets remain agnostic to how the locale was selected or updated. They simply react to the localization context provided by the framework.

This architecture cleanly separates:

• Locale detection

• Business logic and state management

• Framework-level localization

• UI rendering

As a result, localization behavior remains explicit, maintainable, and compatible with automated translation workflows and CI-driven localization updates.

Common Pitfalls and How to Avoid Them

1. Avoid manual string concatenation. For example, do not use 'Hello ' + name. You should rely on localized templates instead.

2. Never hardcode plural logic in Dart. Always use intl’s pluralization features to handle different languages correctly.

3. Avoid locale-specific formatting outside intl utilities. Dates, numbers, and currencies should be formatted using the proper localization tools.

4. Always regenerate localization files after updating ARB files. This ensures the app reflects all the latest translations.

How to Automate Translations with AI

In Flutter applications that rely on ARB files for localization, translation maintenance becomes increasingly costly as the application grows. Each new message must be manually propagated across locale files, often resulting in missing keys, inconsistent phrasing, or delayed updates. This problem is amplified in projects that do not use a Translation Management System (TMS) and instead keep ARB files directly in the repository.

While many TMS platforms have begun adding AI-assisted translation features, not all projects use a TMS at all, particularly small teams, internal tools, or personal projects. In these cases, developers frequently resort to copying strings into AI chat tools and pasting results back into ARB files, which is inefficient and difficult to scale.

To address this workflow gap, Leen Code published arb_translate package, a Dart-based CLI tool that automates missing ARB translations using large language models.

Design Approach

The model behind arb_translate aligns with Flutter’s existing localization pipeline rather than replacing it:

• English ARB files remain the source of truth

• Only missing keys are translated

• Output is written back as standard ARB files

• flutter gen-l10n is still responsible for code generation

This design makes the tool suitable for both local development and CI usage, without introducing new runtime dependencies or localization abstractions.

At a high level, the flow is:

1. Parse the base (typically English) ARB file

2. Identify missing keys in target locale ARB files

3. Send key–value pairs to an LLM via API

4. Receive translated strings

5. Update or generate locale-specific ARB files

6. Run flutter gen-l10n to regenerate localized resources

Gemini-Based Setup

To use Gemini for ARB translation:

1. Generate a Gemini API key
   https://ai.google.dev/tutorials/setup

   

2. Install the CLI:

dart pub global activate arb_translate

3. Export the API key:

export ARB_TRANSLATE_API_KEY=your-api-key

4. Run the tool from the Flutter project root:

arb_translate

The tool scans existing ARB files, generates missing translations, and writes them back to disk.

OpenAI/ChatGPT Support

As of version 1.0.0, arb_translate also supports OpenAI ChatGPT models. This allows teams to standardize on OpenAI infrastructure or switch providers without changing their localization workflow.

1. Generate an OpenAI API key
   https://platform.openai.com/api-keys

   

2. Install the tool:

dart pub global activate arb_translate

3. Export the API key:

export ARB_TRANSLATE_API_KEY=your-api-key

4. Select OpenAI as the provider:

Via l10n.yaml:

arb-translate-model-provider: open-ai

Or via CLI:

arb_translate --model-provider open-ai

5. Execute:

arb_translate

Practical Use Cases

This approach is not intended to replace professional translation or review workflows. Instead, it serves as a deterministic automation layer that:

• Eliminates manual copy-paste workflows

• Keeps ARB files structurally consistent

• Enables translation generation in CI

• Allows downstream review in a TMS if required

For content-heavy Flutter applications or teams without a dedicated localization platform, this provides a pragmatic and maintainable solution.

Best Practices and Considerations

1. Always define a fallback locale to ensure the app remains usable.

2. Avoid hardcoding user-facing strings; rely on localized resources.

3. Use semantic and stable ARB keys for maintainability.

4. Persist user language preferences to provide a consistent experience.

5. Test your app with long translations and multiple locales to catch layout or UI issues.

Conclusion

Localization is a foundational requirement for modern Flutter applications. By combining Flutter’s built-in localization framework, the intl package, and Bloc for state management, you gain a robust and scalable solution.

With automatic device language detection, runtime switching, and clean architecture, your application becomes globally accessible without sacrificing maintainability.

References

Here are official links you can use as references for Flutter localization:

• Flutter Internationalization Guide – Official Flutter guide on how to internationalize your app:
  https://docs.flutter.dev/ui/accessibility-and-internationalization/internationalization

• Dart intl Package Documentation – API reference for the intl library used for formatting and localization utilities:
  https://api.flutter.dev/flutter/package-intl_intl/index.html

• Flutter flutter_localizations API – API docs for the flutter_localizations library that provides localized strings and resources for Flutter widgets:
  https://api.flutter.dev/flutter/flutter_localizations/

• Flutter App Localization with AI (LeanCode) – A guide on speeding up Flutter localization using AI and tools like Gemini or ChatGPT, including details on the arb_translate package.
  https://leancode.co/blog/flutter-app-localization-with-ai

• arb_translate package (pub.dev) – A tool for automating ARB file translations in Flutter:
  https://pub.dev/packages/arb_translate

So I decided to change the format instead of giving up. I took a few tech blogs I already enjoy reading, picked the best articles, converted them to audio using my own voice, and turned the result into a private podcast. Now I can stay up to date while walking, running, or driving.

In this tutorial, you’ll learn how to build a simplified version of that pipeline step by step.

Table of Contents

• What You Are Going to Build

• Prerequisites

• Project Overview

• Getting Started

• How to Get the Content

• How to Filter the Content

• How to Clean Up the Content

• How to Convert Content to Audio

• How to Upload the Audio to Cloudflare R2

• How to Make the Podcast

• How to Automate the Pipeline

• Conclusion

What You Are Going to Build

You will build a Node.js script that does the following:

• Fetches articles from RSS feeds.

• Extracts clean, readable text from each article.

• Filters out content you do not want to listen to.

• Cleans the text so it sounds good when spoken.

• Converts the text to natural-sounding audio using your own voice.

• Uploads the audio to Cloudflare R2.

• Generates a podcast RSS feed.

• Runs automatically on a schedule.

At the end, you will have a real podcast feed you can subscribe to on your phone.

If you want to skip the tutorial and jump straight into using the finished tool, you can find the complete version and instructions on GitHub.

Prerequisites

To follow along, you need basic JavaScript knowledge.

You also need:

• Node.js 22 or newer.

• A place to store audio files (Cloudflare R2 in this tutorial).

• A text-to-speech API (OrangeClone in this tutorial).

Project Overview

Before writing code, it helps to understand the idea clearly.

This project is a pipeline:

Fetch content -> Filter content -> Clean up content -> Convert to audio -> Repeat

Each step takes the output of the previous one. Keeping the flow linear makes the project easier to reason about, debug, and automate.

All code in this tutorial lives in a single file called index.js.

Getting Started

Create a new project folder and your main file.

mkdir podcast-pipeline
cd podcast-pipeline
touch index.js

Initialize the project and install dependencies.

npm init -y
npm install rss-parser @mozilla/readability jsdom node-fetch uuid xmlbuilder @aws-sdk/client-s3

Enable ESM so import syntax works in Node 22.

npm pkg set type=module

Here is what each dependency is used for:

• rss-parser reads RSS feeds.

• @mozilla/readability extracts readable article text.

• jsdom provides a DOM for Readability.

• node-fetch fetches remote content.

• uuid generates unique filenames.

• xmlbuilder creates the podcast RSS feed.

• @aws-sdk/client-s3 uploads audio to Cloudflare R2.

How to Get the Content

The first decision is where your content comes from.

Avoid scraping websites directly. Scraped HTML is noisy and inconsistent. RSS feeds are structured and reliable. Most serious blogs provide one.

Open index.js and define your sources.

import Parser from "rss-parser";
import fetch from "node-fetch";
import { JSDOM } from "jsdom";
import { Readability } from "@mozilla/readability";

const parser = new Parser();

const NUMBER_OF_ARTICLES_TO_FETCH = 15;

const SOURCES = [
  "https://www.freecodecamp.org/news/rss/",
  "https://hnrss.org/frontpage",
];

Now fetch articles and extract readable content.

async function fetchArticles() {
  const articles = [];

  for (const source of SOURCES) {
    const feed = await parser.parseURL(source);

    for (const item of feed.items.slice(0, NUMBER_OF_ARTICLES_TO_FETCH)) {
      if (!item.link) continue;

      const response = await fetch(item.link);
      const html = await response.text();

      const dom = new JSDOM(html, { url: item.link });
      const reader = new Readability(dom.window.document);
      const content = reader.parse();

      if (!content) continue;

      articles.push({
        title: item.title,
        link: item.link,
        content: content.content,
        text: content.textContent,
      });
    }
  }

  return articles.slice(0, NUMBER_OF_ARTICLES_TO_FETCH);
}

This function:

• Reads RSS feeds.

• Downloads each article.

• Extracts clean text using Readability.

• Returns a list of articles ready for processing.

How to Filter the Content

Not every article deserves your attention. Start by filtering out topics you do not want to hear about.

const BLOCKED_KEYWORDS = ["crypto", "nft", "giveaway"];

function filterByKeywords(articles) {
  return articles.filter(
    (article) =>
      !BLOCKED_KEYWORDS.some((keyword) =>
        article.text.toLowerCase().includes(keyword)
      )
  );
}

Next, remove promotional content.

function removePromotionalContent(articles) {
  return articles.filter(
    (article) => !article.text.toLowerCase().includes("sponsored")
  );
}

Finally, remove articles that are too short.

function filterByWordCount(articles, minWords = 700) {
  return articles.filter(
    (article) => article.text.split(/\s+/).length >= minWords
  );
}

After these steps, you are left with articles you actually want to listen to.

How to Clean Up the Content

Raw articles text still need to be cleaned up to sound good when spoken. First, replace images with spoken placeholders.

function replaceImages(html) {
  return html.replace(/<img[^>]*alt="([^"]*)"[^>]*>/gi, (_, alt) => {
    return alt ? `[Image: ${alt}]` : `[Image omitted]`;
  });
}

Next, remove code blocks.

function replaceCodeBlocks(html) {
  return html.replace(
    /<pre><code>[\s\S]*?<\/code><\/pre>/gi,
    "[Code example omitted]"
  );
}

Strip URLs and replace them with spoken text.

function replaceUrls(text) {
  return text.replace(/https?:\/\/\S+/gi, "link removed");
}

Normalize common symbols.

function normalizeSymbols(text) {
  return text
    .replace(/&/g, "and")
    .replace(/%/g, "percent")
    .replace(/\$/g, "dollar");
}

Convert HTML to text so TTS does not read tags.

function stripHtml(html) {
  return html.replace(/<[^>]+>/g, " ");
}

Combine everything into one cleanup step.

function cleanArticle(article) {
  let cleaned = replaceImages(article.content);
  cleaned = replaceCodeBlocks(cleaned);
  cleaned = stripHtml(cleaned);
  cleaned = replaceUrls(cleaned);
  cleaned = normalizeSymbols(cleaned);

  return {
    ...article,
    cleanedText: cleaned,
  };
}

At this point, the text is ready for audio generation.

How to Convert Content to Audio

Browser speech APIs sound robotic. I wanted something that sounded human and familiar. After trying several tools, I settled on OrangeClone. It was the only option that actually sounded like me.

Create a free account and copy your API key from the dashboard.

Record 10 to 15 seconds of clean audio and save it as SAMPLE_VOICE.wav in the project root. Then create a voice character (one-time setup).

import fs from "node:fs/promises";

const ORANGECLONE_API_KEY = process.env.ORANGECLONE_API_KEY;
const ORANGECLONE_BASE_URL =
  process.env.ORANGECLONE_BASE_URL || "https://orangeclone.com/api";

async function createVoiceCharacter({ name, avatarStyle, voiceSamplePath }) {
  const audioBuffer = await fs.readFile(voiceSamplePath);
  const audioBase64 = audioBuffer.toString("base64");

  const response = await fetch(
    `${ORANGECLONE_BASE_URL}/characters/create`,
    {
      method: "POST",
      headers: {
        Authorization: `Bearer ${ORANGECLONE_API_KEY}`,
        "Content-Type": "application/json",
      },
      body: JSON.stringify({
        name,
        avatarStyle,
        voiceSample: {
          format: "wav",
          data: audioBase64,
        },
      }),
    }
  );

  if (!response.ok) {
    const errorText = await response.text();
    throw new Error(`Failed to create character: ${errorText}`);
  }

  const data = await response.json();

  return (
    data.data?.id ||
    data.data?.characterId ||
    data.id ||
    data.characterId
  );
}

Generate audio from text.

async function generateAudio(characterId, text) {
  const response = await fetch(`${ORANGECLONE_BASE_URL}/voices_clone`, {
    method: "POST",
    headers: {
      Authorization: `Bearer ${ORANGECLONE_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      characterId,
      text,
    }),
  });

  return response.json();
}

Wait for the job to complete.

async function waitForAudio(jobId) {
  while (true) {
    const response = await fetch(`${ORANGECLONE_BASE_URL}/voices/${jobId}`);
    const data = await response.json();

    if (data.status === "completed") {
      return data.audioUrl;
    }

    await new Promise((r) => setTimeout(r, 5000));
  }
}

How to Upload the Audio to Cloudflare R2

OrangeClone returns an audio URL, but podcast apps need a stable, public file that will not expire.
That is where Cloudflare R2 comes in.

R2 is S3-compatible storage, which means we can upload files using the AWS SDK and serve them publicly for podcast apps.

How to Set Up Credentials

Create an R2 bucket in your Cloudflare dashboard and set the following environment variables:

• R2_ACCOUNT_ID

• R2_ACCESS_KEY_ID

• R2_SECRET_ACCESS_KEY

• R2_BUCKET_NAME

• R2_PUBLIC_URL

These values allow the script to upload files and generate public URLs for them.

How to Initialize the R2 Client

import { S3Client, PutObjectCommand } from "@aws-sdk/client-s3";

const r2 = new S3Client({
  region: "auto",
  endpoint: `https://${process.env.R2_ACCOUNT_ID}.r2.cloudflarestorage.com`,
  credentials: {
    accessKeyId: process.env.R2_ACCESS_KEY_ID,
    secretAccessKey: process.env.R2_SECRET_ACCESS_KEY,
  },
});

This creates an S3-compatible client that connects directly to your Cloudflare R2 account instead of AWS.

How to Download the Audio

async function downloadAudio(audioUrl) {
  const response = await fetch(audioUrl);
  const buffer = await response.arrayBuffer();
  return Buffer.from(buffer);
}

OrangeClone gives us a URL, not a file.
This function downloads the audio and converts it into a Node.js buffer so it can be uploaded to R2.

How to Upload to R2

import { v4 as uuid } from "uuid";

async function uploadToR2(audioBuffer) {
  const fileName = `${uuid()}.mp3`;

  const command = new PutObjectCommand({
    Bucket: process.env.R2_BUCKET_NAME,
    Key: fileName,
    Body: audioBuffer,
    ContentType: "audio/mpeg",
  });

  await r2.send(command);

  return `${process.env.R2_PUBLIC_URL}/${fileName}`;
}

This function uploads the audio buffer to R2 using a unique filename and returns a public URL that podcast apps can access.

Putting It Together

const audioUrl = await waitForAudio(jobId);
const audioBuffer = await downloadAudio(audioUrl);
const publicAudioUrl = await uploadToR2(audioBuffer);

At the end of this step, publicAudioUrl is the final audio file used in the podcast RSS feed.

How to Make the Podcast

With public audio URLs, you can now generate an RSS feed.

import xmlbuilder from "xmlbuilder";

function generatePodcastFeed(episodes) {
  const feed = xmlbuilder
    .create("rss", { version: "1.0" })
    .att("version", "2.0")
    .ele("channel");

  feed.ele("title", "My Tech Podcast");
  feed.ele("description", "Tech articles converted to audio");
  feed.ele("link", "https://your-site.com");

  episodes.forEach((ep) => {
    const item = feed.ele("item");
    item.ele("title", ep.title);
    item.ele("enclosure", {
      url: ep.audioUrl,
      type: "audio/mpeg",
    });
  });

  return feed.end({ pretty: true });
}

How to Automate the Pipeline

Automation in this project happens in two stages. First, the code itself must be able to process multiple articles in one run. Second, the script must run automatically on a schedule. We’ll start with the code-level automation.

Automating Inside the Code

Earlier, we fetched up to fifteen articles. Now we need to make sure every article that passes our filters goes through the full pipeline.

Add the following function near the bottom of index.js.

async function runPipeline() {
  const rawArticles = await fetchArticles();

  const filteredArticles = filterByWordCount(
    removePromotionalContent(filterByKeywords(rawArticles))
  );

  if (filteredArticles.length === 0) {
    console.log("No articles passed the filters");
    return [];
  }

  const characterId = await createVoiceCharacter({
    name: "My Voice",
    avatarStyle: "realistic",
    voiceSamplePath: "./SAMPLE_VOICE.wav",
  });

  const episodes = [];

  for (const article of filteredArticles) {
    console.log(`Processing: ${article.title}`);

    const cleaned = cleanArticle(article);

    const job = await generateAudio(characterId, cleaned.cleanedText);

    const audioUrl = await waitForAudio(job.id);
    const audioBuffer = await downloadAudio(audioUrl);
    const publicAudioUrl = await uploadToR2(audioBuffer);

    episodes.push({
      title: article.title,
      audioUrl: publicAudioUrl,
    });
  }

  return episodes;
}

This function does all the heavy lifting:

• Fetches articles

• Applies all filters

• Creates the voice character once

• Loops through every valid article

• Converts each article into audio

• Uploads the audio to Cloudflare R2

• Collects podcast episode data

At this point, one script run can generate multiple podcast episodes.

Running the Pipeline and Generating the Feed

Now we need a single entry point that runs the pipeline and writes the podcast feed. Add this below the pipeline function.

import fs from "node:fs/promises";

async function main() {
  const episodes = await runPipeline();

  if (episodes.length === 0) {
    console.log("No episodes generated");
    return;
  }

  const rss = generatePodcastFeed(episodes);

  await fs.mkdir("./public", { recursive: true });
  await fs.writeFile("./public/feed.xml", rss);

  console.log("Podcast feed generated at public/feed.xml");
}

main().catch(console.error);

When you run node index.js, this now:

• Processes all selected articles

• Creates multiple audio files

• Generates a valid podcast RSS feed

This is the core automation.

Scheduling the Pipeline with GitHub Actions

The final step is to make this script run automatically. Create a GitHub Actions workflow file at .github/workflows/podcast.yml.

name: Podcast Pipeline

on:
  schedule:
    - cron: "0 6 * * *"

jobs:
  run:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
      - run: npm install
      - run: node index.js
        env:
          ORANGECLONE_API_KEY: ${{ secrets.ORANGECLONE_API_KEY }}
          R2_ACCOUNT_ID: ${{ secrets.R2_ACCOUNT_ID }}
          R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
          R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
          R2_BUCKET_NAME: ${{ secrets.R2_BUCKET_NAME }}
          R2_PUBLIC_URL: ${{ secrets.R2_PUBLIC_URL }}

This workflow runs the pipeline every morning at 6 AM.

Each run:

• Fetches new articles

• Generates fresh audio

• Updates the podcast feed

Once this is set up, your podcast updates itself without manual work.

Conclusion

This is a basic version of my full production pipeline, PostCast, but the core idea is the same.

You now know how to turn blogs into a personal podcast. Be mindful of copyright and only use content you are allowed to consume.

If you have questions, reach me on X at @sprucekhalifa. I write practical tech articles like this regularly.

With OpenAI’s text-to-speech models, you can build a clean service that takes a blog URL or pasted text and produces a natural-sounding audio file.

In this article, you’ll learn how to build this system end-to-end. You will learn how to fetch blog content, send it to OpenAI’s audio API, save the output as an MP3 file, and serve everything through a small FastAPI app.

At the end, you’ll also build a minimal user interface and deploy it to Sevalla so that anyone can upload text and download audio without touching code.

Table of Contents

• Understanding the Core Idea

• How to Set Up Your Project

• How to Fetch and Clean Blog Content

• How to Send Text to OpenAI for Audio

• How to Build a FastAPI Backend

• How to Add a Simple User Interface

• How to Deploy Your Service to Sevalla

• Conclusion

Understanding the Core Idea

A blog-to-audio service has only three important parts. The first part takes a blog link or text and cleans it. The second part sends the clean text to OpenAI’s text-to-speech model. The third part gives the final MP3 file back to the user.

OpenAI’s speech generation is simple to use. You send text, choose a voice, and get audio back. The quality is high and works well even for long posts. This means you do not need to worry about training models or tuning voices.

The only job left is to make the system easy to use. That is where FastAPI and a small HTML form help. They wrap your code into a web service so anyone can try it.

How to Set Up Your Project

Create a folder for your project. Inside it, create a file called main.py. You will also need a basic HTML file later.

Install the libraries you need with pip:

pip install fastapi uvicorn requests beautifulsoup4 python-multipart

FastAPI gives you a simple backend. Requests module helps download blog pages. BeautifulSoup helps remove HTML tags and extract readable text. Python-multipart helps upload form data.

You must also install the OpenAI client:

pip install openai

Make sure you have your OpenAI API key ready. Set it in your terminal before running the app:

export OPENAI_API_KEY="your-key"

On Windows, you can do:

setx OPENAI_API_KEY "your-key"

How to Fetch and Clean Blog Content

To convert a blog into audio, you must first extract the main article text. You can fetch the page with requests and parse it with BeautifulSoup.

Below is a simple function that does this.

import requests
from bs4 import BeautifulSoup

def extract_text_from_url(url: str) -> str:
    response = requests.get(url, timeout=10)
    html = response.text
    soup = BeautifulSoup(html, "html.parser")
    paragraphs = soup.find_all("p")
    text = " ".join(p.get_text(strip=True) for p in paragraphs)
    return text

Here is what happens step by step.

• The function downloads the page.

• BeautifulSoup reads the HTML and finds all paragraph tags.

• It pulls out the text in each paragraph and joins them into one long string.

• This gives you a clean version of the blog post without ads or layout code.

If the user pastes text instead of a URL, you can skip this part and use the text as it is.

How to Send Text to OpenAI for Audio

OpenAI’s text-to-speech API makes this part of the work very easy. You send a message with text and select a voice such as Alloy or Verse. The API returns raw audio bytes. You can save these bytes as an MP3 file.

Here is a helper function to convert text into audio:

from openai import OpenAI
client = OpenAI()

def text_to_audio(text: str, output_path: str):
    audio = client.audio.speech.create(
        model="gpt-4o-mini-tts",
        voice="alloy",
        input=text
    )
    with open(output_path, "wb") as f:
        f.write(audio.read())

This function calls the OpenAI client and passes the text, model name, and voice choice. The .read() method extracts the binary audio stream. Writing this to an MP3 file completes the process.

If the blog post is very long, you may want to limit text length or chunk the text and join the audio files later. But for most blogs, the model can handle the entire text in one request.

How to Build a FastAPI Backend

Now you can wrap both steps into a simple FastAPI server. This server will accept either a URL or pasted text. It will convert the content into audio and return the MP3 file as a response.

Here is the full backend code:

from fastapi import FastAPI, Form
from fastapi.responses import FileResponse
import uuid
import os

app = FastAPI()
@app.post("/convert")
def convert(url: str = Form(None), text: str = Form(None)):
    if not url and not text:
        return {"error": "Please provide a URL or text"}
    if url:
        try:
            text_content = extract_text_from_url(url)
        except Exception:
            return {"error": "Could not fetch the URL"}
    else:
        text_content = text
    file_id = uuid.uuid4().hex
    output_path = f"audio_{file_id}.mp3"
    text_to_audio(text_content, output_path)
    return FileResponse(output_path, media_type="audio/mpeg")

Here is how it works. The user sends form data with either url or text. The server checks which one exists.

If there is a URL, it extracts text with the earlier function. If there is no URL, it uses the provided text directly. A unique file name is created for every request. Then the audio file is generated and returned as an MP3 download.

You can run the server like this:

uvicorn main:app --reload

Open your browser at http://localhost:8000. You will not see the UI yet, but the API endpoint is working. You can test it using a tool like Postman or by building the front end next.

How to Add a Simple User Interface

A service is much easier to use when it has a clean UI. Below is a simple HTML page that sends either a URL or text to your FastAPI backend. Save this file as index.html in the same folder:

<!DOCTYPE html>
<html>
<head>
    <title>Blog to Audio</title>
    <style>
        body { font-family: Arial, padding: 40px; max-width: 600px; margin: auto; }
        input, textarea { width: 100%; padding: 10px; margin-top: 10px; }
        button { padding: 12px 20px; margin-top: 20px; cursor: pointer; }
    </style>
</head>
<body>
    <h2>Convert Blog to Audio</h2>
    <form action="/convert" method="post">
        <label>Blog URL</label>
        <input type="text" name="url" placeholder="Enter a blog link">
<p>or paste text below</p>
        <textarea name="text" rows="10" placeholder="Paste blog text here"></textarea>
        <button type="submit">Convert to Audio</button>
    </form>
</body>
</html>

This page gives the user two options. They can type a URL or paste text. The form sends the data to /convert using a POST request. The response will be the MP3 file, so the browser will download it.

To serve the HTML file, add this route to your main.py:

from fastapi.responses import HTMLResponse

@app.get("/")
def home():
    with open("index.html", "r") as f:
        html = f.read()
    return HTMLResponse(html)

Now, when you visit the main URL, you will see a clean form.

When you submit a URL, the server will process your request and give you an audio file.

Great. Our text to audio service is working. Now let’s get it into production.

How to Deploy Your Service to Sevalla

You can choose any cloud provider, like AWS, DigitalOcean, or others, to host your service. I will be using Sevalla for this example.

Sevalla is a developer-friendly PaaS provider. It offers application hosting, database, object storage, and static site hosting for your projects.

Every platform will charge you for creating a cloud resource. Sevalla comes with a $50 credit for us to use, so we won’t incur any costs for this example.

Let’s push this project to GitHub so that we can connect our repository to Sevalla. We can also enable auto-deployments so that any new change to the repository is automatically deployed.

You can also fork my repository from here.

Log in to Sevalla and click on Applications -> Create new application. You can see the option to link your GitHub repository to create a new application.

Use the default settings. Click “Create application”. Now we have to add our OpenAI API key to the environment variables. Click on the “Environment variables” section once the application is created, and save the OPENAI_API_KEY value as an environment variable.

Now we are ready to deploy our application. Click on “Deployments” and click “Deploy now”. It will take 2–3 minutes for the deployment to complete.

Once done, click on “Visit app”. You will see the application served via a URL ending with sevalla.app . This is your new root URL. You can replace localhost:8000 with this URL and start using it.

Congrats! Your blog-to-audio service is now live. You can extend this by adding other capabilities and pushing your code to GitHub. Sevalla will automatically deploy your application to production.

Conclusion

You now know how to build a full blog-to-audio service using OpenAI. You learned how to fetch blog text, convert it into speech, and serve it with FastAPI. You also learned how to create a simple user interface, allowing people to try it with no setup.

With this foundation, you can turn any written content into smooth, natural audio. This can help creators reach a wider audience, enhance accessibility, and provide users with more ways to enjoy content.

Hope you enjoyed this article. Signup for my free newsletter TuringTalks.ai for more hands-on tutorials on AI. You can also visit my website.

Whether you are building a pixel-art indie game or a high-fidelity 3D world, understanding how resolution works is essential.

It affects how your art assets scale, how your UI appears, and how your game feels on different screens. Yet, many developers still treat resolution as a simple number instead of a design decision.

Let’s learn what resolutions are and why it matters for game developers.

What we will Cover

• What Resolution Really Means

• The Evolution of Resolution in Gaming

• DPI, Scaling, and Texture Clarity

• Resolution vs. Performance

• Aspect Ratio and Display Diversity

• The Art of Testing in 4K and HDR

• Preparing for Next-Gen Displays

• Conclusion

What Resolution Really Means

Resolution defines how many pixels a screen can display horizontally and vertically.

A monitor labelled 1920x1080 has 1920 pixels across and 1080 down, which equals over two million pixels in total. More pixels mean more visual detail but also more rendering work for the GPU.

In game development, that tradeoff is constant. Rendering at higher resolutions improves clarity but reduces frame rates unless your code and assets are optimized.

Many developers solve this by offering resolution scaling options in their games, letting players balance visual quality and performance.

It’s also important to distinguish between screen size and resolution. A 27-inch monitor and a 15-inch laptop can both run at 1080p, but the larger display will have bigger, less dense pixels.

This is where pixel density comes in. High-density displays pack more pixels per inch, creating smoother edges and sharper textures even at the same resolution.

The Evolution of Resolution in Gaming

Games have evolved alongside display technology.

Early consoles ran at 240p, then 480p during the SD era. The jump to HD with 720p and 1080p transformed game visuals. Suddenly, developers had to think about anti-aliasing, texture resolution, and UI scaling in new ways.

Today, 4K and HDR have become the standard for modern consoles and PCs. Developers now design with higher fidelity in mind, baking in lighting systems, shaders, and art pipelines that scale up to Ultra HD.

That’s why testing on different display resolutions isn’t just good practice, it’s critical for consistent player experience.

If you want to see how your game performs on large high-resolution displays, try testing it on a modern TV for PS5. These screens are optimized for 4K and 120Hz refresh rates, giving you a realistic look at how your game will appear in a living-room setup.

They also help you spot UI scaling issues, frame pacing problems, and HDR color mismatches that might go unnoticed on a typical monitor.

DPI, Scaling, and Texture Clarity

For web developers, DPI mostly affects how images scale. But for game developers, DPI connects directly to texture resolution and how art assets are perceived at different screen sizes.

A sprite that looks crisp on a 1080p monitor might appear tiny or blurry on a 4K display if not properly scaled. Engines like Unity and Unreal handle this with dynamic scaling options, but understanding the underlying math helps.

When your display density doubles, each asset needs four times as many pixels to appear at the same size and sharpness. If you do not plan for this, your carefully crafted textures might look soft or misaligned on higher-resolution displays.

This is why UI systems in modern engines rely on resolution-independent units. In Unity, Canvas Scaler helps ensure your interface looks the same on every device. In Unreal, DPI scaling rules allow developers to maintain consistent HUD layouts. Getting this right means your game remains legible on everything from handhelds to 8K TVs.

Resolution vs Performance

The biggest cost of higher resolution is GPU load. Rendering in 4K means pushing four times as many pixels as 1080p. Without proper optimization, frame rates can drop sharply.

That’s why many AAA games use resolution scaling techniques like temporal upsampling or DLSS. These methods render frames at a lower resolution and then use AI or interpolation to upscale them without losing clarity.

As a developer, you should test your game across multiple resolutions and aspect ratios. This helps ensure your render pipeline, shaders, and assets adapt smoothly. Tools like NVIDIA Nsight or Unreal’s built-in profiler show how resolution affects frame time and GPU usage.

If your game includes video content or cinematic sequences, also remember that video compression behaves differently at higher resolutions. Encoding 4K video requires significantly more bandwidth and storage, which can affect your build size and performance during playback.

Aspect Ratio and Display Diversity

Aspect ratio determines the shape of the display.

Most modern games target 16:9, but 21:9 ultrawide and 32:9 super-ultrawide displays are becoming more popular. Developers must ensure their camera framing and UI layouts adapt accordingly.

When a game is locked to one ratio, black bars or stretching can occur. To fix this, adjust your camera’s field of view dynamically or provide safe viewport settings.

Engines like Unreal let you script these adjustments easily, while Unity’s Cinemachine system handles FOV scaling automatically.

Even TVs now vary in aspect ratio capabilities, especially with new mini LED and OLED technologies. Testing across multiple ratios ensures your game looks balanced and cinematic on every screen.

The Art of Testing in 4K and HDR

4K and HDR introduce new layers of visual complexity. HDR displays show a wider range of brightness and color depth, which means lighting and textures can look completely different compared to SDR monitors. To handle this, calibrate your color grading pipeline and use tone mapping tools within your engine.

When working with HDR assets, always test your output on real hardware. Emulators and monitors often fail to reproduce true HDR contrast. A proper HDR-certified TV helps you identify overexposure, color clipping, and banding issues before release.

Preparing for Next-Gen Displays

The display industry continues to evolve fast. 8K and high refresh rate panels are already entering mainstream markets.

For developers, this means thinking ahead. Designing scalable rendering systems, supporting dynamic resolution, and maintaining flexible UI layouts are now essential parts of modern game design.

As displays get sharper, player expectations rise too. Textures, shaders, and post-processing all need to support higher levels of detail without compromising performance. By understanding how resolution interacts with your pipeline, you can future-proof your games for years to come.

Conclusion

Resolution is more than a number on a settings menu. It is a design constraint, a performance factor, and a creative opportunity. As a game developer, mastering resolution helps you build experiences that look sharp, play smoothly, and scale across every device.

The next time you polish your textures or fine-tune your rendering settings, remember that every pixel counts. Understanding how resolution, scaling, and density interact will not only make your games more beautiful but also more accessible to every player, whether they’re gaming on a laptop, a monitor, or the living-room tv that brings your visuals to life in stunning detail.

Hope you enjoyed this article. Find me on Linkedin or visit my website.

This is where Transformers come in. Originally built for language, they’ve become state-of-the-art in vision tasks thanks to models like the Vision Transformer (ViT) and video-focused variants such as TimeSformer.

In this tutorial, we’ll use a Transformer backbone to create a lightweight real-time gesture recognition tool, optimized for small datasets and deployable on a regular laptop webcam.

Table of Contents

• Why Transformers for Gestures?

• What You’ll Learn

• Prerequisites

• Project Setup

• Generate a Gesture Dataset

• Option 1: Generate a Synthetic Dataset

• Training Script: train.py

• Export the Model to ONNX

• Evaluate Accuracy + Latency

• Option 2: Use Small Samples from Public Gesture Datasets

• Accessibility Notes & Ethical Limits

• Next Steps

• Conclusion

Why Transformers for Gestures?

Transformers are powerful because they use self-attention to model relationships across a sequence. For gestures, this means the model doesn’t just see isolated frames, but also learns how movements evolve over time. A wave, for example, looks different from a raised hand only when viewed as a sequence.

Vision Transformers process images as patches, while video Transformers extend this to multiple frames with temporal attention. Even a simple approach, like applying ViT to each frame and pooling across time, can outperform traditional CNN-based methods for small datasets.

Combined with Hugging Face’s pre-trained models and ONNX Runtime for optimization, Transformers make it possible to train on a modest dataset and still achieve smooth real-time recognition.

What You’ll Learn

In this tutorial, you’ll build a gesture recognition system using Transformers. By the end, you’ll know how to:

• Create (or record) a tiny gesture dataset

• Train a Vision Transformer (ViT) with temporal pooling

• Export the model to ONNX for faster inference

• Build a real-time Gradio app that classifies gestures from your webcam

• Evaluate your model’s accuracy and latency with simple scripts

• Understand the accessibility potential and ethical limits of gesture recognition

Prerequisites

To follow along, you should have:

• Basic Python knowledge (functions, scripts, virtual environments)

• Familiarity with PyTorch (tensors, datasets, training loops) – helpful but not required

• Python 3.8+ installed on your system

• A webcam (for the live demo in Gradio)

• Optionally: GPU access (training on CPU works, but is slower)

Project Setup

Create a new project folder and install the required libraries.

# Create a new project directory and navigate into it
mkdir transformer-gesture && cd transformer-gesture

# Set up a Python virtual environment
python -m venv .venv

# Activate the virtual environment
# Windows PowerShell
.venv\Scripts\Activate.ps1

# macOS/Linux
source .venv/bin/activate

The provided code snippet is a set of commands for setting up a new Python project with a virtual environment. Here's a breakdown of each part:

1. mkdir transformer-gesture && cd transformer-gesture: This command creates a new directory named "transformer-gesture" and then navigates into it.

2. python -m venv .venv: This command creates a new virtual environment in the current directory. The virtual environment is stored in a folder named ".venv".

3. Activating the virtual environment:

   • For Windows PowerShell, you can use .venv\Scripts\Activate.ps1 to activate the virtual environment.

   • For macOS/Linux, use source .venv/bin/activate to activate the virtual environment.

Activating a virtual environment ensures that the Python interpreter and any packages you install are isolated to this specific project, preventing conflicts with other projects or system-wide packages.

Create a requirements.txt file:

torch>=2.0
torchvision
torchaudio
timm
huggingface_hub

onnx
onnxruntime

gradio

numpy
opencv-python
pillow

matplotlib
seaborn
scikit-learn

The list provided is a set of package dependencies typically found in a requirements.txt file for a Python project. Here's a brief explanation of each package:

1. torch>=2.0: PyTorch is a popular open-source deep learning framework that provides a flexible and efficient platform for building and training neural networks. Version 2.0 and above includes improvements in performance and new features.

2. torchvision: This library is part of the PyTorch ecosystem and provides tools for computer vision tasks, including datasets, model architectures, and image transformations.

3. torchaudio: Also part of the PyTorch ecosystem, Torchaudio provides audio processing tools and datasets, making it easier to work with audio data in deep learning projects.

4. timm: The PyTorch Image Models (timm) library offers a collection of pre-trained models and utilities for computer vision tasks, facilitating quick experimentation and deployment.

5. huggingface_hub: This library allows easy access to models and datasets hosted on the Hugging Face Hub, a platform for sharing and collaborating on machine learning models and datasets.

6. onnx: The Open Neural Network Exchange (ONNX) format is used to represent machine learning models, enabling interoperability between different frameworks.

7. onnxruntime: This is a high-performance runtime for executing ONNX models, allowing for efficient deployment across various platforms.

8. gradio: Gradio is a library for creating user interfaces for machine learning models, making them accessible through a web interface for easy interaction and testing.

9. numpy: A fundamental package for numerical computing in Python, providing support for arrays and a wide range of mathematical functions.

10. opencv-python: OpenCV is a library for computer vision and image processing tasks, widely used for real-time applications.

11. pillow: A Python Imaging Library (PIL) fork, Pillow provides tools for opening, manipulating, and saving many different image file formats.

12. matplotlib: A plotting library for Python, Matplotlib is used for creating static, interactive, and animated visualizations in Python.

13. seaborn: Built on top of Matplotlib, Seaborn provides a high-level interface for drawing attractive and informative statistical graphics.

14. scikit-learn: A machine learning library in Python that provides simple and efficient tools for data analysis and modeling, including classification, regression, clustering, and dimensionality reduction.

Install dependencies:

pip install -r requirements.txt

The command pip install -r requirements.txt is used to install all the Python packages listed in a file named requirements.txt. This file typically contains a list of package dependencies required for a Python project, each specified with a package name and optionally a version number.

By running this command, pip, which is the Python package installer, reads the file and installs each package listed, ensuring that the project has all the necessary dependencies to run properly. This is a common practice in Python projects to manage and share dependencies easily.

Generate a Gesture Dataset

To train our Transformer-based gesture recognizer, we need some data. Instead of downloading a huge dataset, we’ll start with a tiny synthetic dataset you can generate in seconds. This makes the tutorial lightweight and ensures that everyone can follow along without dealing with multi-gigabyte downloads.

Option 1: Generate a Synthetic Dataset

We’ll use a small Python script that creates short .mp4 clips of a moving (or still) coloured box. Each class represents a gesture:

• swipe_left – box moves from right to left

• swipe_right – box moves from left to right

• stop – box stays still in the center

Save this script as generate_synthetic_gestures.py in your project root:

import os, cv2, numpy as np, random, argparse

def ensure_dir(p): os.makedirs(p, exist_ok=True)

def make_clip(mode, out_path, seconds=1.5, fps=16, size=224, box_size=60, seed=0, codec="mp4v"):
    rng = random.Random(seed)
    frames = int(seconds * fps)
    H = W = size

    # background + box color
    bg_val = rng.randint(160, 220)
    bg = np.full((H, W, 3), bg_val, dtype=np.uint8)
    color = (rng.randint(20, 80), rng.randint(20, 80), rng.randint(20, 80))

    # path of motion
    y = rng.randint(40, H - 40 - box_size)
    if mode == "swipe_left":
        x_start, x_end = W - 20 - box_size, 20
    elif mode == "swipe_right":
        x_start, x_end = 20, W - 20 - box_size
    elif mode == "stop":
        x_start = x_end = (W - box_size) // 2
    else:
        raise ValueError(f"Unknown mode: {mode}")

    fourcc = cv2.VideoWriter_fourcc(*codec)
    vw = cv2.VideoWriter(out_path, fourcc, fps, (W, H))
    if not vw.isOpened():
        raise RuntimeError(
            f"Could not open VideoWriter with codec '{codec}'. "
            "Try --codec XVID and use .avi extension, e.g. out.avi"
        )

    for t in range(frames):
        alpha = t / max(1, frames - 1)
        x = int((1 - alpha) * x_start + alpha * x_end)
        # small jitter to avoid being too synthetic
        jitter_x, jitter_y = rng.randint(-2, 2), rng.randint(-2, 2)
        frame = bg.copy()
        cv2.rectangle(frame, (x + jitter_x, y + jitter_y),
                      (x + jitter_x + box_size, y + jitter_y + box_size),
                      color, thickness=-1)
        # overlay text
        cv2.putText(frame, mode, (8, 24), cv2.FONT_HERSHEY_SIMPLEX, 0.7, (0, 0, 0), 2, cv2.LINE_AA)
        cv2.putText(frame, mode, (8, 24), cv2.FONT_HERSHEY_SIMPLEX, 0.7, (255, 255, 255), 1, cv2.LINE_AA)
        vw.write(frame)

    vw.release()

def write_labels(labels, out_dir):
    with open(os.path.join(out_dir, "labels.txt"), "w", encoding="utf-8") as f:
        for c in labels:
            f.write(c + "\n")

def main():
    ap = argparse.ArgumentParser(description="Generate a tiny synthetic gesture dataset.")
    ap.add_argument("--out", default="data", help="Output directory (default: data)")
    ap.add_argument("--classes", nargs="+",
                    default=["swipe_left", "swipe_right", "stop"],
                    help="Class names (default: swipe_left swipe_right stop)")
    ap.add_argument("--clips", type=int, default=16, help="Clips per class (default: 16)")
    ap.add_argument("--seconds", type=float, default=1.5, help="Seconds per clip (default: 1.5)")
    ap.add_argument("--fps", type=int, default=16, help="Frames per second (default: 16)")
    ap.add_argument("--size", type=int, default=224, help="Frame size WxH (default: 224)")
    ap.add_argument("--box", type=int, default=60, help="Box size (default: 60)")
    ap.add_argument("--codec", default="mp4v", help="Codec fourcc (mp4v or XVID)")
    ap.add_argument("--ext", default=".mp4", help="File extension (.mp4 or .avi)")
    args = ap.parse_args()

    ensure_dir(args.out)
    write_labels(args.classes, ".")  # writes labels.txt to project root

    print(f"Generating synthetic dataset -> {args.out}")
    for cls in args.classes:
        cls_dir = os.path.join(args.out, cls)
        ensure_dir(cls_dir)
        mode = "stop" if cls == "stop" else ("swipe_left" if "left" in cls else ("swipe_right" if "right" in cls else "stop"))
        for i in range(args.clips):
            filename = os.path.join(cls_dir, f"{cls}_{i+1:03d}{args.ext}")
            make_clip(
                mode=mode,
                out_path=filename,
                seconds=args.seconds,
                fps=args.fps,
                size=args.size,
                box_size=args.box,
                seed=i + 1,
                codec=args.codec
            )
        print(f"  {cls}: {args.clips} clips")

    print("Done. You can now run: python train.py, python export_onnx.py, python app.py")

if __name__ == "__main__":
    main()

The script generates a synthetic gesture dataset by creating video clips of a moving or stationary coloured box, simulating gestures like "swipe left," "swipe right," and "stop," and saves them in a specified output directory.

Now run it inside your virtual environment:

python generate_synthetic_gestures.py --out data --clips 16 --seconds 1.5

The command above runs a Python script named generate_synthetic_gestures.py, which generates a synthetic gesture dataset with 16 clips per gesture, each lasting 1.5 seconds, and saves the output in a directory named "data".

This creates a dataset like:

data/
  swipe_left/*.mp4
  swipe_right/*.mp4
  stop/*.mp4
labels.txt

Each folder contains short clips of a moving (or still) box that simulate gestures. This is perfect for testing the pipeline.

Training Script: train.py

Now that we have our dataset, let’s fine-tune a Vision Transformer with temporal pooling. This model applies ViT frame-by-frame, averages embeddings across time, and trains a classification head on your gestures.

Here’s the full training script:

# train.py
import torch, torch.nn as nn, torch.optim as optim
from torch.utils.data import DataLoader
import timm
from dataset import GestureClips, read_labels

class ViTTemporal(nn.Module):
    """Frame-wise ViT encoder -> mean pool over time -> linear head."""
    def __init__(self, num_classes, vit_name="vit_tiny_patch16_224"):
        super().__init__()
        self.vit = timm.create_model(vit_name, pretrained=True, num_classes=0, global_pool="avg")
        feat_dim = self.vit.num_features
        self.head = nn.Linear(feat_dim, num_classes)

    def forward(self, x):  # x: (B,T,C,H,W)
        B, T, C, H, W = x.shape
        x = x.view(B * T, C, H, W)
        feats = self.vit(x)                  # (B*T, D)
        feats = feats.view(B, T, -1).mean(dim=1)  # (B, D)
        return self.head(feats)

def train():
    device = "cuda" if torch.cuda.is_available() else "cpu"
    labels, _ = read_labels("labels.txt")
    n_classes = len(labels)

    train_ds = GestureClips(train=True)
    val_ds   = GestureClips(train=False)
    print(f"Train clips: {len(train_ds)} | Val clips: {len(val_ds)}")

    # Windows/CPU friendly
    train_dl = DataLoader(train_ds, batch_size=2, shuffle=True,  num_workers=0, pin_memory=False)
    val_dl   = DataLoader(val_ds,   batch_size=2, shuffle=False, num_workers=0, pin_memory=False)

    model = ViTTemporal(num_classes=n_classes).to(device)
    criterion = nn.CrossEntropyLoss()
    optimizer = optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.05)

    best_acc = 0.0
    epochs = 5
    for epoch in range(1, epochs + 1):
        # ---- Train ----
        model.train()
        total, correct, loss_sum = 0, 0, 0.0
        for x, y in train_dl:
            x, y = x.to(device), y.to(device)
            optimizer.zero_grad()
            logits = model(x)
            loss = criterion(logits, y)
            loss.backward()
            optimizer.step()

            loss_sum += loss.item() * x.size(0)
            correct += (logits.argmax(1) == y).sum().item()
            total += x.size(0)

        train_acc = correct / total if total else 0.0
        train_loss = loss_sum / total if total else 0.0

        # ---- Validate ----
        model.eval()
        vtotal, vcorrect = 0, 0
        with torch.no_grad():
            for x, y in val_dl:
                x, y = x.to(device), y.to(device)
                vcorrect += (model(x).argmax(1) == y).sum().item()
                vtotal += x.size(0)
        val_acc = vcorrect / vtotal if vtotal else 0.0

        print(f"Epoch {epoch:02d} | train_loss {train_loss:.4f} "
              f"| train_acc {train_acc:.3f} | val_acc {val_acc:.3f}")

        if val_acc > best_acc:
            best_acc = val_acc
            torch.save(model.state_dict(), "vit_temporal_best.pt")

    print("Best val acc:", best_acc)

if __name__ == "__main__":
    train()

Running the command python train.py initiates the training process for your gesture recognition model. Here's a breakdown of what happens:

1. Load your dataset from data/: The script will access and load the gesture dataset stored in the "data" directory. This dataset is used to train the model.

2. Fine-tune a pre-trained Vision Transformer: The training script will take a Vision Transformer model that has been pre-trained on a larger dataset and fine-tune it using your specific gesture dataset. Fine-tuning helps the model adapt to the nuances of your data, improving its performance on the specific task of gesture recognition.

3. Save the best checkpoint as vit_temporal_best.pt: During training, the script will evaluate the model's performance on a validation set. The best-performing version of the model (based on some metric like accuracy) will be saved as a checkpoint file named "vit_temporal_best.pt". This file can later be used for inference or further training.

What Training Looks Like

You should see logs similar to this:

Train clips: 38 | Val clips: 10
Epoch 01 | train_loss 1.4508 | train_acc 0.395 | val_acc 0.200
Epoch 02 | train_loss 1.2466 | train_acc 0.263 | val_acc 0.200
Epoch 03 | train_loss 1.1361 | train_acc 0.368 | val_acc 0.200
Best val acc: 0.200

Don’t worry if your accuracy is low at first, as with the synthetic dataset that’s normal. The key is proving that the Transformer pipeline works. You can boost results later by:

• Adding more clips per class

• Training for more epochs

• Switching to real recorded gestures

Figure 1. Example training logs from train.py, where the Vision Transformer with temporal pooling is fine-tuned on a tiny synthetic dataset.

Export the Model to ONNX

To make our model easier to run in real time (and lighter on CPU), we’ll export it to the ONNX format.

Note: ONNX, which stands for Open Neural Network Exchange, is an open-source format designed to facilitate the interchange of deep learning models between different frameworks. It lets you train a model in one framework, such as PyTorch or TensorFlow, and then deploy it in another, like Caffe2 or MXNet, without needing to completely rewrite the model. This interoperability is achieved by providing a standardized representation of the model's architecture and parameters.

ONNX supports a wide range of operators and is continually updated to include new features, making it a versatile choice for deploying machine learning models across various platforms and devices.

Create a file called export_onnx.py:

import torch
from train import ViTTemporal
from dataset import read_labels

labels, _ = read_labels("labels.txt")
n_classes = len(labels)

# Load trained model
model = ViTTemporal(num_classes=n_classes)
model.load_state_dict(torch.load("vit_temporal_best.pt", map_location="cpu"))
model.eval()

# Dummy input: batch=1, 16 frames, 3x224x224
dummy = torch.randn(1, 16, 3, 224, 224)

# Export
torch.onnx.export(
    model, dummy, "vit_temporal.onnx",
    input_names=["video"], output_names=["logits"],
    dynamic_axes={"video": {0: "batch"}},
    opset_version=13
)

print("Exported vit_temporal.onnx")

Run it with python export_onnx.py.

This generates a file vit_temporal.onnx in your project folder. ONNX lets us use onnxruntime, which is much faster for inference.

Create a file called app.py:

import os, tempfile, cv2, torch, onnxruntime, numpy as np
import gradio as gr
from dataset import read_labels

T = 16
SIZE = 224
MODEL_PATH = "vit_temporal.onnx"

labels, _ = read_labels("labels.txt")

# --- ONNX session + auto-detect names ---
ort_session = onnxruntime.InferenceSession(MODEL_PATH, providers=["CPUExecutionProvider"])
# detect first input and first output names to avoid mismatches
INPUT_NAME = ort_session.get_inputs()[0].name   # e.g. "input" or "video"
OUTPUT_NAME = ort_session.get_outputs()[0].name # e.g. "logits" or something else

def preprocess_clip(frames_rgb):
    if len(frames_rgb) == 0:
        frames_rgb = [np.zeros((SIZE, SIZE, 3), dtype=np.uint8)]
    if len(frames_rgb) < T:
        frames_rgb = frames_rgb + [frames_rgb[-1]] * (T - len(frames_rgb))
    frames_rgb = frames_rgb[:T]
    clip = [cv2.resize(f, (SIZE, SIZE), interpolation=cv2.INTER_AREA) for f in frames_rgb]
    clip = np.stack(clip, axis=0)                                    # (T,H,W,3)
    clip = np.transpose(clip, (0, 3, 1, 2)).astype(np.float32) / 255 # (T,3,H,W)
    clip = (clip - 0.5) / 0.5
    clip = np.expand_dims(clip, 0)                                   # (1,T,3,H,W)
    return clip

def _extract_path_from_gradio_video(inp):
    if isinstance(inp, str) and os.path.exists(inp):
        return inp
    if isinstance(inp, dict):
        for key in ("video", "name", "path", "filepath"):
            v = inp.get(key)
            if isinstance(v, str) and os.path.exists(v):
                return v
        for key in ("data", "video"):
            v = inp.get(key)
            if isinstance(v, (bytes, bytearray)):
                tmp = tempfile.NamedTemporaryFile(delete=False, suffix=".mp4")
                tmp.write(v); tmp.flush(); tmp.close()
                return tmp.name
    if isinstance(inp, (list, tuple)) and inp and isinstance(inp[0], str) and os.path.exists(inp[0]):
        return inp[0]
    return None

def _read_uniform_frames(video_path):
    cap = cv2.VideoCapture(video_path)
    frames = []
    total = int(cap.get(cv2.CAP_PROP_FRAME_COUNT)) or 1
    idxs = np.linspace(0, total - 1, max(T, 1)).astype(int)
    want = set(int(i) for i in idxs.tolist())
    j = 0
    while True:
        ok, bgr = cap.read()
        if not ok: break
        if j in want:
            rgb = cv2.cvtColor(bgr, cv2.COLOR_BGR2RGB)
            frames.append(rgb)
        j += 1
    cap.release()
    return frames

def predict_from_video(gradio_video):
    video_path = _extract_path_from_gradio_video(gradio_video)
    if not video_path or not os.path.exists(video_path):
        return {}
    frames = _read_uniform_frames(video_path)

    # If OpenCV choked on the codec (common with recorded webm), re-encode once:
    if len(frames) == 0:
        tmp = tempfile.NamedTemporaryFile(delete=False, suffix=".mp4"); tmp_name = tmp.name; tmp.close()
        cap = cv2.VideoCapture(video_path)
        fourcc = cv2.VideoWriter_fourcc(*"mp4v")
        w = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH)) or 640
        h = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT)) or 480
        out = cv2.VideoWriter(tmp_name, fourcc, 20.0, (w, h))
        while True:
            ok, frame = cap.read()
            if not ok: break
            out.write(frame)
        cap.release(); out.release()
        frames = _read_uniform_frames(tmp_name)

    clip = preprocess_clip(frames)
    # >>> use the detected ONNX input/output names <<<
    logits = ort_session.run([OUTPUT_NAME], {INPUT_NAME: clip})[0]  # (1, C)
    probs = torch.softmax(torch.from_numpy(logits), dim=1)[0].numpy().tolist()
    return {labels[i]: float(probs[i]) for i in range(len(labels))}

def predict_from_image(image):
    if image is None:
        return {}
    clip = preprocess_clip([image] * T)
    logits = ort_session.run([OUTPUT_NAME], {INPUT_NAME: clip})[0]
    probs = torch.softmax(torch.from_numpy(logits), dim=1)[0].numpy().tolist()
    return {labels[i]: float(probs[i]) for i in range(len(labels))}

with gr.Blocks() as demo:
    gr.Markdown("# Gesture Classifier (ONNX)\nRecord or upload a short video, then click **Classify Video**.")
    with gr.Tab("Video (record or upload)"):
        vid_in = gr.Video(label="Record from webcam or upload a short clip")
        vid_out = gr.Label(num_top_classes=3, label="Prediction")
        gr.Button("Classify Video").click(fn=predict_from_video, inputs=vid_in, outputs=vid_out)
    with gr.Tab("Single Image (fallback)"):
        img_in = gr.Image(label="Upload an image frame", type="numpy")
        img_out = gr.Label(num_top_classes=3, label="Prediction")
        gr.Button("Classify Image").click(fn=predict_from_image, inputs=img_in, outputs=img_out)

if __name__ == "__main__":
    demo.launch()

Running the command python app.py launches a Gradio application in your web browser. Here's what happens:

1. Webcam feed streams live: The application accesses your webcam to provide a live video feed. This allows you to perform gestures in front of the camera in real-time.

2. Predictions update continuously: As you perform gestures, the model processes the video frames continuously, updating its predictions in real-time.

3. Top 3 gesture classes displayed with probabilities: The application displays the top three predicted gesture classes along with their probabilities, giving you an idea of the model's confidence in its predictions.

When you open the app in your browser, you'll find two tabs. In the Video tab, you can click Record from webcam to capture a short clip of your gesture, typically lasting 2–4 seconds. After recording, click Classify Video. The model will then process the captured frames using the Transformer model and display the predicted gesture probabilities. This setup allows for interactive testing and demonstration of the gesture recognition system.

Here’s an example where I raised my hand for a stop gesture, and the model predicts “stop” as the top class:

Figure 2. The Gradio app running locally. After recording a short clip, the Transformer model predicts the gesture with class probabilities.

Evaluate Accuracy + Latency

Now that the model runs in a demo app, let’s check how well it performs. There are two sides to this:

• Accuracy: does the model predict the right gesture class?

• Latency: how fast does it respond, especially on CPU vs GPU?

1. Quick Accuracy Check

Save this as eval.py in the same folder as your other scripts:

import torch
from dataset import GestureClips, read_labels
from train import ViTTemporal

labels, _ = read_labels("labels.txt")
n_classes = len(labels)

# Load validation data
val_ds = GestureClips(train=False)
val_dl = torch.utils.data.DataLoader(val_ds, batch_size=2, shuffle=False)

# Load trained model
model = ViTTemporal(num_classes=n_classes)
model.load_state_dict(torch.load("vit_temporal_best.pt", map_location="cpu"))
model.eval()

correct, total = 0, 0
all_preds, all_labels = [], []

with torch.no_grad():
    for x, y in val_dl:
        logits = model(x)
        preds = logits.argmax(dim=1)
        correct += (preds == y).sum().item()
        total += y.size(0)
        all_preds.extend(preds.tolist())
        all_labels.extend(y.tolist())

print(f"Validation accuracy: {correct/total:.2%}")

2. Confusion Matrix

Let’s also visualize which gestures are confused. Add this snippet at the bottom of eval.py:

import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.metrics import confusion_matrix

cm = confusion_matrix(all_labels, all_preds)

plt.figure(figsize=(6,6))
sns.heatmap(cm, annot=True, fmt="d", xticklabels=labels, yticklabels=labels, cmap="Blues")
plt.xlabel("Predicted")
plt.ylabel("True")
plt.title("Confusion Matrix")
plt.tight_layout()
plt.show()

When you run python eval.py, a heatmap like this will pop up:

Figure 3. Confusion matrix on the validation set. Correct predictions appear along the diagonal. Off-diagonal counts show gesture confusions.

3. Latency Benchmark

Finally, let’s see how fast inference runs. Save the following as benchmark.py:

import time, numpy as np, onnxruntime
from dataset import read_labels

labels, _ = read_labels("labels.txt")

ort = onnxruntime.InferenceSession("vit_temporal.onnx", providers=["CPUExecutionProvider"])
INPUT_NAME = ort.get_inputs()[0].name
OUTPUT_NAME = ort.get_outputs()[0].name

dummy = np.random.randn(1, 16, 3, 224, 224).astype(np.float32)

# Warmup
for _ in range(3):
    ort.run([OUTPUT_NAME], {INPUT_NAME: dummy})

# Benchmark
t0 = time.time()
for _ in range(50):
    ort.run([OUTPUT_NAME], {INPUT_NAME: dummy})
t1 = time.time()

print(f"Average latency: {(t1 - t0)/50:.3f} seconds per clip")

Run: python benchmark.py

On CPU, you might see ~0.05–0.15s per clip; on GPU it’s much faster.

Note: If latency is high, you can enable quantization in ONNX to shrink the model and speed up inference.

Option 2: Use Small Samples from Public Gesture Datasets

If you’d prefer to see your model trained on real gesture clips instead of synthetic moving boxes, you can grab a handful of videos from open datasets. You don’t need to download the entire dataset (which can be several GB) just a few .mp4 samples are enough to follow along.

Recommended sources

• 20BN Jester Dataset: Contains short clips of hand gestures like swiping, clapping, and pointing.

• WLASL: A large-scale dataset of isolated sign language words.

Both projects provide small .mp4 videos you can use as realistic training examples. I’ve linked them below.

Setting up your dataset folder

Once you download a few clips, place them in the data/ folder under subfolders named after each gesture class. For example:

data/
├── swipe_left/
│   ├── clip1.mp4
│   └── clip2.mp4
├── swipe_right/
│   ├── clip1.mp4
│   └── clip2.mp4
└── stop/
    ├── clip1.mp4
    └── clip2.mp4

And update labels.txt to match the folder names:

swipe_left
swipe_right
stop

Now your dataset is ready, and the same training scripts from earlier (train.py, eval.py) will work without modification.

Why choose this option?

• Gives more realistic results than synthetic coloured boxes

• Lets you see how the model handles actual human hand movements

• It just requires a bit more effort (downloading clips, trimming them if needed)

Tip: If downloading from these datasets feels too heavy, you can also record your own short gestures using your laptop webcam. Just save them as .mp4 files and organize them in the same folder structure.

Accessibility Notes & Ethical Limits

While this project shows the technical workflow for gesture recognition with Transformers, it’s important to step back and consider the human context:

• Accessibility first: Tools like this can help students with speech or motor difficulties, but they should always be co-designed with the people who will use them. Don’t assume one-size-fits-all.

• Dataset sensitivity: Using publicly available sign or gesture datasets is fine for prototyping, but deploying such a system requires careful consideration of consent and representation.

• Error tolerance: Even small misclassifications can have big consequences in accessibility contexts (for example, confusing stop with go). Always plan for fallback options (like manual input or confirmation).

• Bias and inclusivity: Models trained on narrow datasets may fail for different skin tones, lighting conditions, or cultural gesture variations. Broad and diverse training data is essential for fairness.

In other words: this demo is a teaching scaffold, not a production-ready accessibility tool. Responsible deployment requires collaboration with educators, therapists, and end users.

Next Steps

If you’d like to push this project further, here are some directions to explore:

• Better models: Try video-focused Transformers like TimeSformer or VideoMAE for stronger temporal reasoning.

• Larger vocabularies: Add more gesture classes, build your own dataset, or use portions of public datasets like 20BN Jester or WLASL.

• Pose fusion: Combine gesture video with human pose keypoints from MediaPipe or OpenPose for more robust predictions.

• Real-time smoothing: Implement temporal smoothing or debounce logic in the app so predictions are more stable during live use.

• Quantization + edge devices: Convert your ONNX model to an INT8 quantized version and deploy it on a Raspberry Pi or Jetson Nano for classroom-ready prototypes.

Conclusion

In this tutorial, you learned how to create a gesture recognition system using Transformer models, demonstrating the potential of cutting-edge machine learning techniques. By preparing a small dataset, training a Vision Transformer with temporal pooling, exporting the model to ONNX for efficient inference, and deploying a real-time Gradio app, you showcased a practical application of these technologies. The evaluation of accuracy and latency further highlighted the system's effectiveness and responsiveness.

This project illustrates how you can leverage advanced ML methods to enhance accessibility and communication, paving the way for more inclusive learning environments.

Remember: while this demo works with small datasets, real-world applications need larger, more diverse data and careful consideration of accessibility, inclusivity, and ethics.

Here’s the GitHub repo for full source code: transformer-gesture.

Across the world, millions of learners face communication barriers. Some live with apraxia of speech or dysarthria, others with limited mobility, hearing differences, or neurodiverse needs. When speaking, writing, or pointing is unreliable or tiring, participation becomes limited, feedback is lost, and confidence slowly erodes. This is not a rare exception but an everyday reality in classrooms.

These barriers appear in very practical ways. Students are skipped or misunderstood when they cannot respond quickly. Their ability is under-measured because their means of expression are constrained. Teachers struggle to maintain the pace of lessons while making individual accommodations. Peers interact less often, reducing opportunities for social belonging.

Assistive technologies have helped over the years, with tools like text-to-speech, symbol boards, and simple gesture inputs. Yet most of these tools are designed for a single mode of interaction. They assume the learner will either speak, or type, or tap. Real communication, however, is fluid. Learners naturally combine gestures, partial speech, symbols, and context to share meaning, especially when fatigue, anxiety, or motor challenges come into play.

This is where modern AI changes the picture. We are beginning to move beyond single-solution tools into multimodal systems that can understand speech, even when it is disordered, interpret gestures and visual symbols, combine signals to infer intent, and adapt in real time as the learner’s abilities develop or change.

AI is reshaping accessibility in education by shifting from isolated tools to multimodal and adaptive systems. These systems combine gesture, speech, and intelligent feedback to meet learners where they are, while also supporting their growth over time.

In this article, we will explore what this shift looks like in practice, how it can unlock participation, and how adaptive feedback personalises support and we will also build a hands-on multimodal demo that turns these ideas into a classroom-ready tool.

Prerequisites

• An Operating System: Windows, macOS, or Linux

• Python installed (3.9 or later) – Along with pip for installing packages.

• Editor: Visual Studio Code or any Integrated development environment (IDE)

• Basics: Comfortable running commands in a terminal

• Optional hardware: Microphone (speech input), Webcam (single-frame tab), speakers (TTS playback)

• Internet: Required for the default SpeechRecognition (Google Web Speech API) and gTTS

• No dataset/model needed: A stub gesture classifier is provided so the demo runs end-to-end

Table of Contents

• Prerequisites

• What We’ve Achieved So Far

• Case Study 1: Translating Makaton to English

• Case Study 2: AURA Prototype (Adaptive Speech Assistant)

• The Bigger Picture: Multimodal Accessibility Tools

• How to Build a Multimodal Makaton to English Translator (Gesture + Speech)

• Project Overview

• Challenges and Ethical Considerations

• Where We’re Heading Next

• Conclusion: Building an Inclusive Future with AI

What We’ve Achieved So Far

The past few years have shown how AI can make classrooms more inclusive when we focus on accessibility. Developers, educators, and researchers are already experimenting with tools that bridge communication gaps.

In my first freeCodeCamp tutorial, I built a gesture-to-text translator using MediaPipe. This project demonstrated how computer vision can track hand movements and convert them into text in real time. For learners who rely on gestures, this kind of system can provide a bridge to participation.

Here is a simplified example of how MediaPipe detects hand landmarks:

import mediapipe as mp
import cv2

# Initialize MediaPipe Hands
mp_hands = mp.solutions.hands
hands = mp_hands.Hands()

# Start capturing video from the webcam
cap = cv2.VideoCapture(0)

# Capture a frame from the video
ret, frame = cap.read()

# Process the frame to detect hand landmarks
results = hands.process(cv2.cvtColor(frame, cv2.COLOR_BGR2RGB))

# Print the detected hand landmarks
print("Hand landmarks:", results.multi_hand_landmarks)

This small piece of code shows how MediaPipe processes a video frame and extracts hand landmarks. From there, you can classify gestures and map them to text.

👉 You can explore the full project on GitHub or read the complete tutorial on freeCodeCamp.

In another freeCodeCamp article, I demonstrated how to build AI accessibility tools with Python, such as speech recognition and text-to-speech. These projects provided readers with a foundation for building their own inclusive tools, and you can find the full source code in the repository.

Beyond these individual projects, the wider field has also made significant progress. Advances in sign language recognition have improved accuracy in capturing complex hand shapes and movements. Text-to-speech systems have become more natural and adaptive, giving users voices that sound closer to human speech. Mobile and desktop accessibility apps have brought these capabilities into everyday classrooms.

These achievements are encouraging, but they remain limited. Most of today’s tools are still designed for a single mode of communication. A system may work for gestures, or for speech, or for text, but not all of them together.

The next step is clear: we need multimodal, adaptive AI tools that can blend gestures, speech, and feedback into unified systems. This is where the most exciting opportunities in accessibility lie, and it is where we will turn next.

Figure 1: Comparison of isolated single-modality systems with unified multimodal AI systems.

Case Study 1: Translating Makaton to English

One of my first projects in this area focused on translating Makaton into English.

Makaton is a language programme that uses signs and symbols to support people with speech and language difficulties. It is widely used in classrooms where learners may not rely fully on speech. The challenge is that while a learner communicates in Makaton, their teachers and peers often work in English, which creates a communication gap.

The AI Workflow

The system followed a clear pipeline:

Camera Input → Hand Landmark Detection → Gesture Classification → English Translation Output

Figure 2: AI workflow for translating Makaton gestures into English.

• Camera Input: captures the learner’s Makaton sign.

• Hand Landmark Detection: a vision library such as MediaPipe or OpenCV identifies the position of the fingers and hands.

• Gesture Classification: a trained machine learning model classifies which Makaton sign was made.

• English Translation Output: the system maps that gesture to its English word or phrase and displays it.

Example in Python

Here is a simplified version of how this workflow might look in code:

# Step 1: Capture input
frame = camera.read()

# Step 2: Detect hand landmarks
landmarks = mediapipe.process(frame)

# Step 3: Classify gesture
gesture = gesture_model.predict(landmarks)

# Step 4: Translate to English
translation_map = {
    "hello_sign": "Hello",
    "thank_you_sign": "Thank you"
}
text = translation_map.get(gesture, "Unknown sign")

print("Makaton sign:", gesture, " -> English:", text)

This is a simplified example, but it shows the core idea: map gestures to meaning and then bridge that meaning into English.

Why This Matters

Imagine a student signing thank you in Makaton and the system instantly displaying the words on screen. Teachers can check understanding, peers can respond naturally, and the learner’s contribution becomes visible to everyone.

The key takeaway is that AI can bridge symbol and gesture based languages with mainstream spoken and written communication. Instead of forcing learners to adapt to rigid systems, we can design systems that adapt to the way they already communicate.

Case Study 2: AURA Prototype (Adaptive Speech Assistant)

Another project I worked on is called AURA, the Apraxia of Speech Adaptive Understanding and Relearning Assistant. The idea was to design a system that not only recognises speech but also supports learners with speech disorders by detecting errors, adapting feedback, and offering multimodal alternatives.

The Challenge

Most commercial speech recognition systems fail when a person’s speech does not follow typical patterns. This is especially true for people with apraxia of speech, where motor planning difficulties make pronunciation inconsistent. The result is frequent misrecognition, frustration, and exclusion from tools that rely on voice input.

The AI Workflow

The AURA prototype used a layered architecture:

Speech Input → Wav2Vec2 (fine-tuned for disordered speech) → CNN + BiLSTM Error Detection → Reinforcement Learning Feedback → Multimodal Output (Speech + Gesture)

Figure 3: Workflow of the AURA prototype, combining speech, error detection, adaptive feedback, and multimodal outputs.

• Wav2Vec2 Speech Recognition: fine-tuned on disordered speech to improve transcription accuracy.

• CNN + BiLSTM Model: classifies articulation or phonological errors in real time.

• Reinforcement Learning Engine: adapts feedback loops so therapy suggestions improve as the learner progresses.

• Gesture-to-Speech Multimodal Input: when speech is too difficult, MediaPipe gestures can be used to trigger spoken outputs.

• Streamlit Interface: integrates everything into a single accessible app for testing.

Here’s a simplified view of how an error detection module could be structured:

# Example: Error classification using CNN + BiLSTM
import torch
import torch.nn as nn

# Define the ErrorClassifier model
class ErrorClassifier(nn.Module):
    def __init__(self):
        super(ErrorClassifier, self).__init__()
        self.cnn = nn.Conv1d(in_channels=40, out_channels=64, kernel_size=3)
        self.lstm = nn.LSTM(64, 128, batch_first=True, bidirectional=True)
        self.fc = nn.Linear(256, 3)  # Output classes: e.g. correct, substitution, omission

    def forward(self, x):
        x = self.cnn(x)
        x, _ = self.lstm(x)
        return self.fc(x[:, -1, :])

# Instantiate the model
model = ErrorClassifier()

This snippet shows the heart of the error detection pipeline: combining CNN layers for feature extraction with BiLSTMs for sequence modeling. The model can flag articulation errors, which then guide the feedback loop.

Why This Matters

With AURA, the goal was not just to recognise what someone said, but to help them communicate more effectively. The prototype adapted in real time offering corrective feedback, suggesting gestures, or switching modes when speech became difficult.

The takeaway is that AI can evolve from being a passive recognition tool into an active partner in learning and communication.

The Bigger Picture: Multimodal Accessibility Tools

The two projects we explored, translating Makaton into English and building the AURA prototype highlight a much larger transformation underway. Accessibility technology is moving away from isolated, single-purpose applications toward multimodal platforms that bring together speech, gestures, text, and adaptive AI into one seamless system.

Why This Shift Matters

The benefits of this shift are profound:

• Greater inclusivity in classrooms: learners who rely on different modes of communication can participate equally.

• Real-time support: systems that detect errors or adapt to gestures give learners immediate feedback rather than delayed corrections.

• Lower frustration: multimodal options mean if one channel breaks down (for example, speech), others like gesture or text can take over smoothly.

• Confidence and independence: learners express themselves more fully, without depending heavily on support staff or interpreters.

Beyond the Classroom

The impact of multimodal accessibility extends across many sectors:

• In healthcare, patients with communication difficulties can use multimodal AI assistants to express needs clearly, reducing misdiagnosis and stress.

• In the workplace, employees with speech or motor impairments can collaborate effectively using adaptive AI tools.

• In community settings, individuals can participate more freely in conversations, services, and digital platforms, strengthening social inclusion.

Visualising the Shift

How to Build a Multimodal Makaton to English Translator (Gesture + Speech)

This demo combines both use cases: a Makaton to English classroom tool and the AURA assistive speech path. It prioritizes gesture when a sign is detected, falls back to speech when it isn’t, and produces a unified English output (with optional text-to-speech). We’ll focus on the translation layer, multimodal fusion, and a simple Streamlit UI.

Project structure

makaton_multimodal_demo/
├─ .streamlit/
│   └─ config.toml 
├─ assets/
│   └─ README.txt 
├─ tests/
│   └─ test_fuse.py 
└─ streamlit_app.py

The structure provided above outlines the organization of a project directory for a multimodal Makaton to English translator demo using Streamlit. Here's a brief explanation of each component:

• makaton_multimodal_demo/: This is the root directory of the project.

• .streamlit/: This directory contains configuration files for Streamlit, which is a framework used to build web apps in Python. The config.toml file is optional and can be used to customize the Streamlit app's settings.

• assets/: This directory is intended to store models or other necessary files for the project. The README.txt serves as a placeholder to indicate where these files should be placed.

• tests/: This directory is for test scripts. The test_fuse.py file likely contains tests for the fusion function, which is a part of the multimodal translation process.

• streamlit_app.py: This is the main application file where the Streamlit app is implemented. It contains the code that runs the app, handling the user interface and the logic for translating Makaton gestures and speech into English.

Install & run

# (optional) create and activate a virtualenv
python -m venv .venv

# Windows
.\.venv\Scripts\activate

# macOS/Linux
source .venv/bin/activate

The code snippet above provides instructions for creating and activating a Python virtual environment, which is a self-contained directory that contains a Python installation for a particular version of Python, plus several additional packages.

1. python -m venv .venv: This command creates a new virtual environment in a directory named .venv. The venv module is used to create lightweight virtual environments.

2. .\.venv\Scripts\activate (Windows): This command activates the virtual environment on Windows. Once activated, the environment's Python interpreter and installed packages will be used.

3. source .venv/bin/activate (macOS/Linux): This command activates the virtual environment on macOS or Linux. Similar to Windows, activating the environment ensures that the specific Python interpreter and packages within the environment are used.

Install dependencies

pip install streamlit opencv-python mediapipe SpeechRecognition gTTS pydub numpy

The command above is used to install multiple Python packages at once. Here's what each package does:

• streamlit: A framework for building interactive web applications in Python, often used for data science and machine learning projects.

• opencv-python: Provides OpenCV, a library for computer vision tasks such as image processing and video analysis.

• mediapipe: A library developed by Google for building cross-platform, customizable machine learning solutions for live and streaming media, including hand and face detection.

• SpeechRecognition: A library for performing speech recognition, allowing Python to recognize and process human speech.

• gTTS: Google Text-to-Speech, a library and CLI tool to interface with Google Translate's text-to-speech API, enabling text-to-speech conversion.

• pydub: A library for audio processing, allowing manipulation of audio files, such as converting between different audio formats.

• numpy: A fundamental package for scientific computing in Python, providing support for arrays and matrices, along with a collection of mathematical functions.

Create streamlit_app.py

# streamlit_app.py
from io import BytesIO
from typing import Optional
import streamlit as st

# Optional deps (kept optional so readers can still run the core demo)
try:
    import cv2
    import mediapipe as mp
    MP_OK = True
except Exception:
    MP_OK = False

try:
    import speech_recognition as sr
    SR_OK = True
except Exception:
    SR_OK = False

try:
    from gtts import gTTS
    GTTS_OK = True
except Exception:
    GTTS_OK = False

# --- 1) Minimal Makaton dictionary (extend as needed)
MAKATON_DICT = {
    "hello_sign": "Hello",
    "thank_you_sign": "Thank you",
    "help_sign": "Help",
    "toilet_sign": "Toilet",
    "stop_sign": "Stop",
}

# --- 2) Gesture classifier (stub for the demo)
def classify_gesture(landmarks) -> Optional[str]:
    """
    Return a canonical label like 'hello_sign' or None if unknown.
    Replace this stub with your trained model + confidence threshold.
    """
    return "hello_sign" if landmarks else None

# --- 3) Speech recognizer (fallback path)
def transcribe_speech(seconds: int = 3) -> Optional[str]:
    if not SR_OK:
        return None
    r = sr.Recognizer()
    try:
        with sr.Microphone() as source:
            st.info("Listening...")
            audio = r.listen(source, phrase_time_limit=seconds)
        return r.recognize_google(audio)
    except Exception as e:
        st.warning(f"Speech recognition error: {e}")
        return None

# --- 4) Fusion logic (gesture first, speech fallback)
def fuse(gesture_label: Optional[str], speech_text: Optional[str]) -> str:
    if gesture_label and gesture_label in MAKATON_DICT:
        return MAKATON_DICT[gesture_label]
    if speech_text:
        return speech_text
    return "No input detected"

# --- 5) Optional: extract single-frame hand landmarks using MediaPipe
def extract_hand_landmarks_from_image(image_bytes: bytes):
    if not MP_OK:
        return None
    try:
        import numpy as np
        np_arr = np.frombuffer(image_bytes, dtype=np.uint8)
        img = cv2.imdecode(np_arr, cv2.IMREAD_COLOR)
        if img is None:
            return None

        mp_hands = mp.solutions.hands
        with mp_hands.Hands(static_image_mode=True, max_num_hands=1, min_detection_confidence=0.5) as hands:
            img_rgb = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
            result = hands.process(img_rgb)

        if not result.multi_hand_landmarks:
            return None

        hand_landmarks = result.multi_hand_landmarks[0]
        return [(lm.x, lm.y, lm.z) for lm in hand_landmarks.landmark]
    except Exception:
        return None

# --- 6) Streamlit UI
st.set_page_config(page_title="Makaton → English (Multimodal Demo)")
st.title("Makaton → English (Multimodal Demo)")
st.caption("Combines a classroom Makaton translator with an assistive speech path (AURA-style).")

with st.expander("What this demo shows"):
    st.write(
        "- **Translation layer:** small Makaton dictionary you can extend.\n"
        "- **Multimodal fusion:** gesture prioritized, speech as fallback.\n"
        "- **UI:** one page, clear output, optional text-to-speech."
    )

tabs = st.tabs(["Simulated Sign", "Single-Frame Webcam (Optional)", "About"])

# Tab 1: Simulated (no CV model required)
with tabs[0]:
    st.subheader("Simulated Gesture + Speech")
    col1, col2 = st.columns(2)

    with col1:
        simulate = st.selectbox(
            "Pick a sign",
            ["", "hello_sign", "thank_you_sign", "help_sign", "toilet_sign", "stop_sign"],
            index=0
        )
        gesture_label = simulate or None

    with col2:
        speech_text = st.session_state.get("speech_text")
        st.write("Current speech:", speech_text or "None")
        if st.button("Transcribe 3s"):
            if SR_OK:
                speech_text = transcribe_speech(3)
                st.session_state["speech_text"] = speech_text
            else:
                st.warning("SpeechRecognition not installed.")

    output = fuse(gesture_label, st.session_state.get("speech_text"))
    st.markdown(f"### Output: **{output}**")

    if output and output != "No input detected":
        if st.button("Speak output"):
            if GTTS_OK:
                mp3 = BytesIO()
                try:
                    gTTS(output).write_to_fp(mp3)
                    st.audio(mp3.getvalue(), format="audio/mp3")
                except Exception as e:
                    st.warning(f"TTS failed: {e}")
            else:
                st.warning("gTTS not installed.")

# Tab 2: Optional single-frame webcam capture
with tabs[1]:
    st.subheader("Single-Frame Hand Detection (Webcam)")
    if not MP_OK:
        st.warning("Install MediaPipe + OpenCV to enable this tab.")
    else:
        img = st.camera_input("Capture a frame")
        captured_label = None
        if img is not None:
            landmarks = extract_hand_landmarks_from_image(img.getvalue())
            if landmarks:
                captured_label = classify_gesture(landmarks)
                st.success("Hand detected.")
            else:
                st.info("No hand landmarks found. Try better lighting/framing.")

        if st.button("Transcribe 3s (webcam tab)"):
            st.session_state["speech_text2"] = transcribe_speech(3) if SR_OK else None

        speech_text2 = st.session_state.get("speech_text2")
        st.write("Current speech:", speech_text2 or "None")

        output2 = fuse(captured_label, speech_text2)
        st.markdown(f"### Output: **{output2}**")

        if output2 and output2 != "No input detected":
            if st.button("Speak output (webcam tab)"):
                if GTTS_OK:
                    mp3 = BytesIO()
                    try:
                        gTTS(output2).write_to_fp(mp3)
                        st.audio(mp3.getvalue(), format="audio/mp3")
                    except Exception as e:
                        st.warning(f"TTS failed: {e}")
                else:
                    st.warning("gTTS not installed.")

The code above creates a Streamlit application that combines gesture recognition and speech recognition to translate Makaton signs into English. Here's a brief explanation of how it works:

1. Dependencies and Setup: The code attempts to import optional dependencies like OpenCV, MediaPipe, SpeechRecognition, and gTTS. These are used for gesture detection, speech recognition, and text-to-speech functionalities.

2. Makaton Dictionary: A minimal dictionary that maps Makaton signs to English words. This can be extended to include more signs.

3. Gesture Classifier: A placeholder function (classify_gesture) is used to classify hand gestures. In a real application, this would be replaced with a trained model.

4. Speech Recognizer: The transcribe_speech function uses the SpeechRecognition library to convert spoken words into text, serving as a fallback when gestures are not detected.

5. Fusion Logic: The fuse function prioritizes gesture recognition over speech. If a gesture is recognized, it translates it using the dictionary; otherwise, it uses the transcribed speech.

6. Hand Landmark Extraction: The code includes a function to extract hand landmarks from an image using MediaPipe, which is used for gesture classification.

7. Streamlit UI: The user interface is built with Streamlit, featuring tabs for simulated gestures, webcam-based gesture detection, and additional information. Users can simulate gestures, capture gestures via webcam, and use speech input. The output is displayed and can be converted to speech using gTTS.

This application demonstrates a multimodal approach by integrating both gesture and speech recognition to facilitate communication for users who rely on Makaton.

Run

streamlit run .\streamlit_app.py

The command above is used to launch a Streamlit application. When executed, it starts a local web server and opens the specified Python script in a web browser, allowing you to interact with the app's user interface. This command is typically run in a terminal or command prompt.

Figure — App interface: the Simulated Sign tab before any input.

Figure — Selecting hello_sign produces “Output: Hello”.

Project Overview

You have developed a multimodal translator that integrates both gesture recognition (specifically Makaton signs) and speech recognition to produce a unified English output. The system is designed to prioritize gesture input, using speech as a fallback when gestures are not detected.

User Interface

The application is built using Streamlit, featuring two main tabs:

• Simulated Sign Tab: Allows users to simulate gestures without requiring computer vision (CV) capabilities.

• Webcam Single Frame Tab: Optionally uses a webcam to capture and process a single frame for gesture detection.

Use Case Integration

• Makaton to English Translation: In a classroom setting, detected Makaton signs are translated into short English phrases, facilitating communication.

• AURA-style Assistive Path: If no gesture is detected, the system relies on speech input to generate an output, ensuring continuous communication support.

Design Limitations

• The gesture classifier is currently a placeholder and should be replaced with a trained model that includes a confidence threshold for better accuracy.

• The Makaton dictionary is minimal and can be expanded to include more phrases and templates.

• The speech recognition component uses a basic recognizer. For improved robustness, consider using advanced models like Wav2Vec2 or offline automatic speech recognition (ASR) systems.

Suggested Extensions

• Implement a confidence threshold to display both gesture and speech inputs when the system is uncertain.

• Expand the dictionary to support slot templates, such as "I want [item]".

• Introduce a toggle to switch between speech-first and gesture-first input priorities.

• Enable logging of outputs for teachers and provide an option to export these logs as CSV files.

• Consider replacing gTTS with an offline text-to-speech solution for better reliability.

Troubleshooting Tips

• If you encounter microphone errors, ensure that pyaudio is installed. On Windows, use pip install pipwin followed by pipwin install pyaudio.

• If the webcam is not detected, check your browser permissions. The Simulated Sign tab can still be used without a webcam.

• If there are issues with package imports, verify that they are installed in your active virtual environment.

The link to the full code: Multimodal_Makaton

Challenges and Ethical Considerations

While the promise of multimodal accessibility tools is exciting, building them responsibly requires us to confront several challenges. These are not only technical problems but also ethical ones that affect how learners, teachers, and communities experience AI.

Data Scarcity

Training AI systems requires large, diverse datasets. But when it comes to disordered speech or symbol systems like Makaton, the data is limited. Without enough examples, models risk being inaccurate or biased toward a narrow group of users. Collecting more data is essential, but it must be done ethically, with consent and respect for the communities involved.

Fairness and Inclusion

AI systems often work better for some groups than others. A model trained mostly on fluent English speakers may fail to recognise learners with strong accents or speech difficulties. Similarly, gesture recognition may not account for differences in motor ability. Fairness means designing models that work across abilities, accents, and cultures, so that no group is excluded by design.

Privacy and Security

Speech and video data are highly sensitive, especially when collected in schools. Protecting this data is not optional, it is a requirement. Systems must anonymize or encrypt recordings and store them securely. Transparency is also key: learners, parents, and teachers should know exactly how data is being used and who has access to it.

Accessibility of the Tools Themselves

Ironically, many “accessibility tools” remain inaccessible because they are expensive, require powerful hardware, or are too complex to use. For AI to truly reduce barriers, solutions must be affordable, lightweight, and easy for teachers to set up in real classrooms, not just in research labs.

Takeaway

These challenges remind us that accessibility in AI is not only a technical question but also an ethical and social responsibility. To build tools that genuinely help learners, we need collaboration between developers, educators, policymakers, and the communities who will use the systems.

Where We’re Heading Next

The future of AI accessibility tools is speculative, but the possibilities are both exciting and necessary. What we have now are prototypes and early systems. What lies ahead are tools that could reshape how classrooms and society more broadly approach communication and inclusion.

Multilingual Makaton Translation

One promising direction is the ability to translate Makaton across multiple languages. A learner in the UK could sign in Makaton and see their contribution appear not just in English but in French, Spanish, or Yoruba. This would open up international classrooms and give learners access to global opportunities that are often closed off by language barriers.

AI Tutors with Dynamic Adaptation

Imagine a classroom assistant powered by AI that adapts in real time. If a learner struggles with speech, it could switch to gesture recognition. If gestures become tiring, it could prompt the learner with symbol-based options. These AI tutors would not only support communication but also guide learning, adapting to each student’s strengths and challenges over time.

Wearable Multimodal Devices

The rise of lightweight hardware makes it possible to imagine wearable AI assistants that provide instant translation and support. Glasses could capture gestures and overlay text, while earbuds could translate disordered speech into clear audio for peers and teachers. Instead of bulky setups, accessibility would become portable, personal, and ever-present.

A Broader Impact

These innovations go beyond technology alone. They align with the United Nations Sustainable Development Goals (SDGs) especially:

• Quality Education (Goal 4): ensuring that every learner, regardless of ability, has equal access to education.

• Reduced Inequalities (Goal 10): breaking down barriers so that disability or difference is not a cause of exclusion.

The journey from single-modality tools to multimodal, adaptive systems is still in its early stages. But if we continue to push forward with creativity, ethics, and inclusivity at the center, AI accessibility tools will not only change classrooms they will change lives.

Conclusion: Building an Inclusive Future with AI

AI accessibility tools are no longer just optional add-ons for a few learners. They are becoming core enablers of inclusion in education, healthcare, workplaces, and daily life.

The journey from early gesture recognition systems to multimodal, adaptive prototypes like Makaton translation and AURA shows what is possible when technology is designed around people rather than forcing people to adapt to technology. These innovations break down communication barriers and open up new opportunities for learners who have too often been left on the margins.

But the future of accessibility is not automatic. It depends on choices we make now as developers, educators, researchers, and policymakers. Building tools that are open, ethical, and affordable requires collaboration and commitment.

The vision is clear: a world where every learner, regardless of ability, can express themselves fully, be understood by others, and participate with confidence.

The future of education is inclusive and with thoughtful design, AI can help us get there.